Over on LinkedIn, I somehow stumbled upon the work of James Heathers Ph.D, who is reported as being the director of the Medical Evidence Project: https://medicalevidenceproject.org/. They are associated with scientists (ie. Elizabeth Bik) who have been responsible for the retraction of erroneous or fraudulent peer reviewed biomedical research reports.
He authored a very useful document - An Introduction to Forensic Metascience.
The techniques described are especially helpful in reading research reports with a disposition more compatible with a cooperative skeptic, than naively accepting the report as written. Most of the techniques are based on algebraic manipulation of intro stats methods to check whether the reported data makes any sense when compared to the summary statistics. The document presents R code to perform most of the computations.
Related Thread
My old post examining a meta-analysis on predicting ACL injury in the rehab literature uses some closely related techniques, along with my own analysis using the relationship between the Bayesian Rejection Ratio described by Berger et. al and Frequentist error probabilities.
What I didn’t mention in the original thread
My initial analysis assumed the data from the studies were accurately reported. When I attempted to look up the original reports and compare the reported p-value to the implied p-value derived from the confidence interval, I discovered one reported “non-significant” study in the meta-analysis was actually “significant”, with a 2 sided p-value of 0.006.
I quickly lost interest and came to the conclusion that the reported data, even assuming that they were correctly recorded and reported, did not support the claims of the authors.