# GRADE guidelines and effect estimate & CIs

**URL:** <https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283>\
**Category:** study interpretation\
**Tags:** reporting\
**Created:** [January 25, 2019, 9:52am UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283 "2019-01-25T09:52:08Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![aleksi](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/aleksi/32/425_2.png) [@aleksi](https://discourse.datamethods.org/u/aleksi)\
**Post date:** [January 25, 2019, 9:52am UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/1 "2019-01-25T09:52:09Z")

</div>

The GRADE approach (Grading of Recommendations Assessment, Development and Evaluation) is very widely used tool to evaluate the certainty of evidence in healthcare.

The GRADE approach is usually associated to meta-analyses pooling single point estimates.

The grade and statement goes as follows:  
| **High** | = We are very confident that the true effect lies close to that of the estimate of the effect  
| **Moderate** | = We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different  
| **Low** | = Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect  
| **Very low** | = We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect

As an example from [1](https://onlinelibrary.wiley.com/doi/full/10.1002/jmri.26567)  
_There were no differences in CSA of multifidus at the C2‐3 level between the chronic NSNP and control groups (two studies[38], [47]; SMD –0.30 [95% CI –1.12, 0.51], P = 0.47) ([Fig. 4]. The GRADE quality of evidence was very low._

_There were no differences in disc degeneration between chronic NSNP and control, when assessed using a four‐point ordinal scale[49] and the five‐point Pfirrman scale[50] (two studies[49], [50]; OR 0.84 [95% CI 0.57, 1.24], P = 0.39) ([Fig. 5]. The GRADE quality of evidence was moderate._

(I have not idea about the scales in the second example, but I guess an ordinal scale is dichotomized to binary outcome, why OR is used.)

Stating that there is no difference is obvisouly wrong. In the first one the level of evidence is graded very low, which is reasonable since data is consistent with wide range of plausible values.

But then the second one. Authors claim that the quality of evidence if moderate indicating per definition “_We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different_”.

Again, CI for point estimate is consistent with wide range of values. It is not right to declare “no difference”, it might be there. According to the statement, authors have confidence on “_the effect estimate_”. If the point estimate is not _exactly_ 1.000, doesn´t that mean that they believe that the other group have lower odds for disc degeneration (OR=0.84), ie. they reject the null?

On the other hand, if the _the effect estimate_ includes interpretation of CIs I don´t understand how you can have confidence on a point estimate and CI, since the statement IMO implies “_no further studies needed, this is it for the population value_”, which would again exempt the use of CIs.

To conclude, it seems counter intuitive to state there is “_confident in the effect estimate_” which, however, differs from 1.000 (OR, RR) or 0.000 (SMD, MD) while at the same time the CI for point estimate does not exclude 1 (or 0) indicating that null hypothesis cannot be rejected. This very often of course is considered as “no difference” or “equal” which can be debated.

In other words, is there an inherent flaw in the use of GRADE system or am I missing the point here?

---

<div class="post-metadata">

**Author:** ![Raj](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/raj/32/169_2.png) [@Raj](https://discourse.datamethods.org/u/Raj)\
**Post date:** [January 25, 2019, 12:43pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/2 "2019-01-25T12:43:57Z")

</div>

I had always interpreted the GRADE criteria as providing appraisal for aspects unrelated to confidence intervals, i.e. making recommendation for an intervention is more than just the estimate for risk reduction of primary outcome.

The handbook provides specific examples:

[https://gdt.gradepro.org/app/handbook/handbook.html#h.j8tx801skdu3](https://gdt.gradepro.org/app/handbook/handbook.html#h.j8tx801skdu3)

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [January 25, 2019, 1:14pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/3 "2019-01-25T13:14:28Z")

</div>

Not having read the handbook, I very much resonate with @aleksi. To answer his odds ratio question, a truly ordinal analysis such as the proportional odds model (a generalization of the Wilcoxon two-sample test) results in a single odds ratio. It is the ratio of odds of exceeding a given level of severity, for two treatments. By the proportional odds assumption this ratio is the same no matter which severity threshold you use.

---

<div class="post-metadata">

**Author:** ![pieter](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/pieter/32/278_2.png) [@pieter](https://discourse.datamethods.org/u/pieter)\
**Post date:** [January 25, 2019, 2:14pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/4 "2019-01-25T14:14:21Z")

</div>

Seems they are actually looking for a posterior distribution, where the width of the x% Highest Density Interval (or similar measures) indicate how certain you are.

Under a number of assumptions CIs don’t differ too much numerically from HDIs, you should check it these assumptions hold true.

Added: [https://www.collabra.org/articles/10.1525/collabra.149/](https://www.collabra.org/articles/10.1525/collabra.149/)  
^^ nice article describing that.

---

<div class="post-metadata">

**Author:** ![aleksi](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/aleksi/32/425_2.png) [@aleksi](https://discourse.datamethods.org/u/aleksi)\
**Post date:** [January 26, 2019, 7:22am UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/5 "2019-01-26T07:22:55Z")

</div>

@Raj I understand that they consider all relevant aspects such as bias And heterogeneity and imprecision which contribute to the ”confidence”. But still, to me it seems that it reduces to interpretation of point estimate. And if ”confidence” is high meaning that no further studies needed, I see a conflict interpreting findings failing to reject null.

@pieter I Dont see any mention about HDI or credible intervals in the manual?

---

<div class="post-metadata">

**Author:** ![Raj](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/raj/32/169_2.png) [@Raj](https://discourse.datamethods.org/u/Raj)\
**Post date:** [January 27, 2019, 4:09am UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/6 "2019-01-27T04:09:03Z")

</div>

> [@aleksi](#):
>
> I understand that they consider all relevant aspects such as bias And heterogeneity and imprecision which contribute to the ”confidence”. But still, to me it seems that it reduces to interpretation of point estimate. And if ”confidence” is high meaning that no further studies needed, I see a conflict interpreting findings failing to reject null.

Preface: I’m not familiar enough with GRADE process to comment with authority, but I will offer my thoughts on how I read studies like this. Also I agree that statement of “no difference” is a flawed interpretation given such wide confidence interval.

To go into GRADE “confidence of point estimate” - I had always interpreted this as part of the process to evaluate risk of bias/confounders, especially in observational studies. I.e. if their was a confounder that could not be adjusted from available measures in data, it would throw into doubt accuracy of point estimate.

Specific to your example, the authors discuss in the bias section that their was disagreement on “measurement  
of different levels of exposures” (ie, stratification of WAD or  
NSNP symptoms). If the discrepancy was large, and many patients with more mild symptoms were categorized as more severe in the data (or vice versa), I would be hesitant to accept accuracy of the confidence interval. As it turns out, the authors think the discrepancy was not great, nor did they find other obvious biases. So to me, it seems reasonable to comment on accuracy of the estimate, i.e. “true effect is likely to close the estimate”.

My take away from this nice paper is that correlating MRI findings to neck pain is very difficult, something that is consistent with my real life experiences with patients.

---

<div class="post-metadata">

**Author:** ![pieter](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/pieter/32/278_2.png) [@pieter](https://discourse.datamethods.org/u/pieter)\
**Post date:** [January 28, 2019, 1:56pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/7 "2019-01-28T13:56:21Z")

</div>

You are right. They did this: “By taking the 2.5% and 97.5% percentile points of this distribution, one achieves the 95% credible interval.”

There are several ways to determine 95% credible intervals. Any interval that contains 95% of the posterior mass suffices. It is described clearly in this paper: [http://www.indiana.edu/~kruschke/BEST/BEST.pdf](http://www.indiana.edu/~kruschke/BEST/BEST.pdf)

My point was that it is a fallacy to interpret a narrow confidence interval as precision or certainty of the estimation. The guidelines seem to look for is a Bayesian credible interval (of whatever type, HDI or other)

See here for more on the confidence interval fallacies: [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4742505/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4742505/)

The authors identify three fallacies regarding confidence intervals:  
Fallacy 1 (The Fundamental Confidence Fallacy)  
“If the probability that a random interval contains the true value is X%, then the plausibility or probability that a particular observed interval contains the true value is also X%; or, alternatively, we can have X% confidence that the observed interval contains the true value.” (wrong)

Fallacy 2 (The Precision fallacy)  
“The width of a confidence interval indicates the precision of our knowledge about the parameter. Narrow confidence intervals correspond to precise knowledge, while wide confidence errors correspond to imprecise knowledge.” (wrong)

Fallacy 3 (The Likelihood fallacy)  
“A confidence interval contains the likely values for the parameter. Values inside the confidence interval are more likely than those outside. This fallacy exists in several varieties, sometimes involving plausibility, credibility, or reasonableness of beliefs about the parameter.” (wrong)

---

<div class="post-metadata">

**Author:** ![Raj](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/raj/32/169_2.png) [@Raj](https://discourse.datamethods.org/u/Raj)\
**Post date:** [January 29, 2019, 4:47pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/8 "2019-01-29T16:47:44Z")

</div>

> [@pieter](#):
>
> My point was that it is a fallacy to interpret a narrow confidence interval as precision or certainty of the estimation. The guidelines seem to look for is a Bayesian credible interval (of whatever type, HDI or other)

Genuine question here: if I follow a strict interpretation of confidence intervals, how can I make any meaningful interpretations of study results? Is it really unreasonable to assume a flat prior and interpret confidence intervals like credible intervals?

GRADE system (like any method) is not perfect, but if the average reader uses the folk interpretation of confidence intervals (correct or incorrect), it seems reasonable to add some comments about perceived accuracy of the point estimate.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [January 29, 2019, 4:57pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/9 "2019-01-29T16:57:30Z")

</div>

My personal opinion is that if you are using frequentist methods you should not mix in Bayes. If you want to be Bayesian be Bayesian. And don’t use flat priors. Confidence intervals deal with long-run properties of the _method_.

---

<div class="post-metadata">

**Author:** ![Raj](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/raj/32/169_2.png) [@Raj](https://discourse.datamethods.org/u/Raj)\
**Post date:** [January 29, 2019, 10:22pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/10 "2019-01-29T22:22:55Z")

</div>

I agree with not mixing methods when interpreting results within a study.

But when applying evidence (in my case toward patient care), I also need to assess external validity. And not just “do I think hypothesis is true or false”, but expected relative and absolute effect size for intervention. It seems reasonable to take a Bayesian approach for this process of externalizing validity, even if results are frequentist.

But, If there is another/better approach, I would love to learn!

---

<div class="post-metadata">

**Author:** ![pieter](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/pieter/32/278_2.png) [@pieter](https://discourse.datamethods.org/u/pieter)\
**Post date:** [January 30, 2019, 3:44pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/11 "2019-01-30T15:44:42Z")

</div>

> [@Raj](#):
>
> Genuine question here: if I follow a strict interpretation of confidence intervals, how can I make any meaningful interpretations of study results? Is it really unreasonable to assume a flat prior and interpret confidence intervals like credible intervals?

Very good questions, as matter of fact I don’t have an answer!

---

<div class="post-metadata">

**Author:** ![Raj](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/raj/32/169_2.png) [@Raj](https://discourse.datamethods.org/u/Raj)\
**Post date:** [January 30, 2019, 3:48pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/12 "2019-01-30T15:48:31Z")

</div>

> [@pieter](#):
>
> Very good questions, as matter of fact I don’t have an answer!

I suppose this is what makes us all Bayesians in the end! 🙂

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [January 30, 2019, 4:58pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/13 "2019-01-30T16:58:44Z")

</div>

The interpretation of confidence intervals (better called compatibility intervals) is akin to the interpretation of cross-validation or bootstrap validation of a predictive model. Resampling methods validate the process used to fit the model, not the final fitted model per se. But the likely future performance of the process is still a good estimate of the likely future performance of “the” model.

---

<div class="post-metadata">

**Author:** ![pieter](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/pieter/32/278_2.png) [@pieter](https://discourse.datamethods.org/u/pieter)\
**Post date:** [January 31, 2019, 4:10pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/14 "2019-01-31T16:10:08Z")

</div>

Yes, the value of frequentist methods is in being reasonably correct in the long run. This doesn’t help with the interpretation of single studies (or single confidence intervals) though… @Raj

In the words of Neyman: “_no test based upon a theory of probability can by itself provide any valuable evidence of the truth or falsehood of a hypothesis. But we may look at the purpose of tests from another viewpoint……we may search for rules to govern our behaviour……we insure that, in the long run of experience, we shall not often be wrong._”

If I paraphrase that: “a single confidence interval does not tell you what the population mean and dispersion are. But…we insure that, in the long run of experience, we shall not often be wrong.”

So the price for long run error control (and for not having to pick a prior distribution) is that inference from individual experiments is not possible.

Correct me if I’m wrong.

---

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [March 27, 2019, 11:47am UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/15 "2019-03-27T11:47:50Z")

</div>

Am I the only one who finds it odd, that the guidelines for “evidence based medicine” are based upon a philosophy of decision theory that selects for rule sets that are indifferent to the state of nature?

This discussion of frequentist vs Bayesian analysis was very helpful to me.  
Beyond Bayesians and Frequentists

> **[Beyond Bayesians and Frequentists](https://jsteinhardt.wordpress.com/2012/10/31/beyond-bayesians-and-frequentists/)**
>
> (This is available in pdf form here.) If you are a newly initiated student into the field of machine learning, it won’t be long before you start hearing the words “Bayesian” and &…

The essential difference between the 2 perspectives is the loss function. Bayesians deal with the approximating the average case, Frequentists deal with approximating the worst case.

I don’t dispute that the frequentist attitude has value in many contexts. But that attitude also makes it difficult to accumulate and synthesize knowledge. In the context of research synthesis, I am interested in the average case, or at least creating a defensible, informative prior from previous research reports to decide on whether new research should be done, or if the findings are reasonably consistent (even after adjustment for issues like various types of bias), that an action/policy can be decided upon now.

At least from a research synthesis POV, more work needs to be done from a likelihood perspective. That separates the questions of:

1. What should I do?
2. What should I believe?
3. What do the data say?

It is question 3 that is of interest to the research synthesizer. Ultimately, researchers need a better grounding in decision theory to make sense of these issues, than a mastery of “off the shelf” cookbook techniques, whether Frequentist or Bayesian.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [March 27, 2019, 1:02pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/16 "2019-03-27T13:02:31Z")

</div>

I don’t see a reason to say that Bayes is interested in the average case and frequentist is interested in the worst case.

---

<div class="post-metadata">

**Author:** ![pieter](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/pieter/32/278_2.png) [@pieter](https://discourse.datamethods.org/u/pieter)\
**Post date:** [March 27, 2019, 1:48pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/17 "2019-03-27T13:48:02Z")

</div>

I disagree with this:

> **2.3. When To Use Each Method**  
> The essential difference between Bayesian and frequentist decision theory is that Bayes makes the additional assumption of a prior over ![heta](https://discourse.datamethods.org/uploads/default/original/1X/3cd095732815e3dbdd1db5ef88b015e690c1eef5.png), and optimizes for average-case performance rather than worst-case performance. _It follows, then, that Bayes is the superior method whenever we can obtain a good prior and when good average-case performance is sufficient._ However, if we have no way of obtaining a good prior, or when we need guaranteed performance, frequentist methods are the way to go. For instance, if we are trying to build a software package that should be widely deployable, we might want to use a frequentist method because users can be sure that the software will work as long as some number of easily-checkable assumptions are met.

^this misses the point (and/or is simply wrong imho). Frequentism and Bayesianism answer different questions. Are you ultimately interested in Pr(data | ![heta](https://discourse.datamethods.org/uploads/default/original/1X/3cd095732815e3dbdd1db5ef88b015e690c1eef5.png)) or in Pr( ![heta](https://discourse.datamethods.org/uploads/default/original/1X/3cd095732815e3dbdd1db5ef88b015e690c1eef5.png)|data) ? Are you interested in long term error control or in learning from the specific data at hand?

---

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [March 27, 2019, 2:10pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/18 "2019-03-27T14:10:13Z")

</div>

I agree there are some points to take issue with. I absolutely disagree with his claim that Bayesian techniques are weak in adversarial contexts. That is a very narrow look at Bayesian theory. But the broad point of attitude toward the loss function is viable one to distinguish between the Frequentist philosophy and the Bayesian one. He is not the only one to advocate for it.

As Prof. Harrell disputes this point as well, I am gathering some references in defense of it.

But suffice it to say, statistical philosophy is a continuum, not discrete. Analysis methods can go from extreme emphasis on mini-max decision rules, to a full subjective Bayesian approach that optimizes the subjective expected utility of one decision maker. There are also many points in between, including Empirical Bayes which is a reasonable compromise.

References:  
Michael I. Jordan: Are you A Frequentist or Bayesian  
First 15-20 minutes is most relevant.  
[http://videolectures.net/mlss09uk\_jordan\_bfway/](http://videolectures.net/mlss09uk_jordan_bfway/)

For those pressed for time, the first 13 slides are most relevant.

Old blog post by Larry Wasserman comparing Frequentist “confidence” (aka coverage or compatibility) interval, with a Bayesian credible interval. Suffice it to say, as long as the Frequentist sticks to offering no more than 19 to 1 against the next observation falling outside his interval, there is no way for the Bayesian to take advantage of his superior knowledge of the parameter value (assuming his prior is correct).

> **[WHAT IS BAYESIAN/FREQUENTIST INFERENCE?](https://normaldeviate.wordpress.com/2012/11/17/what-is-bayesianfrequentist-inference/)**
>
> WHAT IS BAYESIAN/FREQUENTIST INFERENCE? When I started this blog, I said I wouldn’t write about the Bayes versus Frequentist thing. I thought that was old news. But many things have changed m…

---

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [July 9, 2019, 4:25pm UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/19 "2019-07-09T16:25:55Z")

</div>

Followers of this topic should find this empirical evaluation of Cochrane reviews using GRADE useful:

> **[Assessing the Predictive Validity of Strength of Evidence Grades: A...](https://www.ncbi.nlm.nih.gov/books/NBK321518/)**
>
> We sought to determine the predictive validity of the U.S. Evidence-based Practice Center (EPC) approach to GRADE (Grading of Recommendations Assessment, Development and Evaluation) by examining how reliably it can predict the likelihood that...

Paraphrasing – “low quality” evidence had less variability than “high quality” evidence, which is just the opposite of what we would expect to see if the rating process was valid.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [July 10, 2019, 11:19am UTC](https://discourse.datamethods.org/t/grade-guidelines-and-effect-estimate-cis/1283/20 "2019-07-10T11:19:44Z")

</div>

> [@R\_cubed](#):
>
> Old blog post by Larry Wasserman

Larry’s blog article is excellent, and so are some of the severe criticisms of it in the discussions that follow.