Many Analysts, One Data Set – many conclusions

[ADAlthousePhD]

I thought we already had a thread that discussed this project…

Nosek et al paper

Would love to get the opinion of experts on the Nosek et al paper showing that 29 research groups analyzed the same data on same research question and found disparate odds ratios depending on analysis type. http://journals.sagepub.com/doi/10.1177/2515245917747646 What are the implications for observational research in cardiology and other areas of medicine? What are the take-homes for the practicing clinician?

I agree that the project was interesting, and highlights how many different analytic decisions are made along the way that can influence the final results.

I think the degree of heterogeneity in the final estimates has been rather overstated. The majority of the effect size estimates are very close. There are a few outliers on each end and a bunch of effect sizes that are all in the same ballpark as one another.

One thing that frustrates me is how this is being interpreted by many readers:

@RonanConroy

For years, when I read a scientific paper, I’ve thought that the data yield the published result. What Nosek and his colleagues have found is that results can be highly contingent on the way the researchers analyze the data. And, get this: There is little agreement on the best way to analyze data.

Yes, the results (or perhaps we should say the conclusions) are contingent on the way researchers analyze the data, but some people seem to think this means we’re doing math differently. This has much more to do with other decisions - what variables should be “adjusted” for (requires causal thinking often in absence…) and what is the proper way to model the outcome - than the actual computations themselves.