Should all medical research papers require statistical review?

This tweet from Andrew Althouse (https://twitter.com/ADAlthousePhD/status/1035130907785658370)
spawned me to finally post this question: Should all manuscripts require a statistical review?

I am a surgeon. I have a year of graduate level epi/biostats training and while I feel comfortable with some methods, I always feel like a discussion with a fully trained epi/biostats person helps (at the very worse I come away with no additional info). However, and particularly in the surgical literature, well meaning (and some not so much)surgeons and physicians perform research that is sometimes bad, and at other times woefully negligent.

This obviously probably happens less in some fields where stats is a core part of training, but I think suggesting that having an MD prepares you fully in methodology is laughable. Even my minimal training did more to expose me to possibilities, limitations, and the basics more than it prepared me to take on every possible situation.

So here I am asking if every medical paper should have a statistical review. I am not quite suggesting that statisticians tear everything apart, but even a gross review of major problems would be useful. Perhaps statisticians could also help make a paper more clear by requiring assumptions be clearly stated.

I am sure this has been discussed before in other locals, but really I would love to hear what the community thinks. It is possible this is overkill, but also possible that it would greatly reduce the trash that gets published and help to improve the quality of the literature greatly.

BTW: I think this would be a great idea, and would support it.

2 Likes

Iā€™m glad you opened this topic. Being a statistician Iā€™m biased in favor of much more frequently having statisticians review medical article submissions. The question is which level of statistician. I think that MS equivalent or higher is warranted. But I also feel that many things get by that are really under quantitative reasoning more than they are under statistics. For example, when should percentages be used vs. raw data? If percent change is used should it be computed on raw patient data or only on aggregate summary statistics (usually the latter IMHO). So we really need quantitatively literature reviewers first.

1 Like

Iā€™m surprised how eager authors are to please reviewersā€™ demands, and thus my worry would be that stats reviewers will make unreasonable requests, and authors will fulfill them. Statisticians disagree on many issues, and can appear uninformed etc, and if thereā€™s no statistician on the authorā€™s side who is inclined to push back then the reviewer, as gate-keeper, can impose their own personal habits wholesale. I think you need a stat on both sides and they should maybe bicker with one another to achieve the best results

2 Likes

Absolutely agree that responding to reviewers is difficult. Pushing back can be difficult, especially if unsure. I guess my hope is that knowing every paper you submit will have a statistical review would encourage authors to enlist a statistician at some point (hopefully earlier rather than later).

I would love to hear other thoughts about difficulties with this idea.

1 Like

Iā€™m not familiar with the procedures across all journals but I do know that Lancet journals seek a statistical review for all research articles. This is often the undoing of many papers, even major trials.

1 Like

Since it is much easier to learn what not to do that it is to learn what to do, letā€™s at least make sure clinical reviewers study these statistical errors to avoid.

3 Likes

Iā€™m glad that you brought this up. After stewing further on this paper (and of course the fact that dozens like it appear every week), I too was thinking to myself ā€œWhat can I do besides start angry Twitter threads to actually do something about this?ā€ I have a proposal (at the bottom of post).

The surgical literature often goes there, yes, but it is far from alone.

Oh, no, rest assured, this happens everywhere.

Of course I, too, feel that every medical paper should have a statistical review. I appreciate that statisticians may disagree over what the optimal analytic approach typically is, and therefore my bar for this review is not necessarily ā€œmake the authors do everything the way I would have done itā€ so much as ā€œmake sure that the authors have presented their data in a reasonable away and used a statistical approach that is appropriate to draw the conclusions given.ā€ Others may disagree in that regard, but it would certainly be nice if we could at least get everyone to that baseline and stop the flood of ā€œHereā€™s a logistic regression with all p<0.05 variables in a multivariable modelā€ papers that appear thoughtlessly and apparently uncriticized by the journal editorial staff even when the analyses are clearly absurd.

Which brings me to my suggestion/thought:

Perhaps the major statistical bodies (ASA and RSS for a start) should offer a ā€œcertificationā€ or something of that nature to journals that have statisticians on their editorial staff and guarantee that all papers have a full statistical review. Sure, right now I know that J Thorac Cardiovasc Surg has all papers get a statistical review while J Am Coll Surg doesnā€™t, but your average reader doesnā€™t know that and could easily take the data in JACS papers just as seriously as JTCVS papers (not that JTCVS papers are perfect, mind you, but at least they have had a statistical review!) without considering abominations such as the one which inspired my Twitter rant yesterday. Perhaps offering at least an acknowledgement of journals that have statistical review would give them a slight credibility edge over journals that clearly do not have statistical reviewers.

It may feel like a hollow gesture, and itā€™s certainly not a perfect solution, but we (the statistical-methods-matter-in-academic-medicine crowd) must start with something more than just blog posts and Tweetorials.

2 Likes

The Lancet does; most of the higher-level journals in the US probably do as well; but there is an entire class of second- and third-tier journals, indexed in PubMed with nontrivial impact factors, that do not.

That many journals donā€™t engage stats reviewers is quite alarming, given the quality of the statistical acumen that crosses my desk. I think your proposal above (ie, some sort of certification of stats review) is a good start. But it wouldnā€™t surprise me if high-impact journals are accused of having self-serving motivations behind flagging their ā€œsuperiorā€ approach to peer review

1 Like

Thatā€™s probably true, but there are two obvious retorts to this:

  1. How can one argue that fulfilling a requirement to have a statistician review quantitative papers is a bad thing? If you agree that itā€™s a benefit to have statistical review thenā€¦

  2. Get it yourself.

Iā€™m not saying the ASA or RSS would charge anything. Simply that journals which fulfill a requirement to have at least one statistician on the editorial board as well as a large enough roster of statistical reviewers that all outcomes manuscripts get a statistical review would get a little seal of approval that their journal complies with that minimum reasonable standard. As I allude above, Iā€™m not expecting every paper to be read and scored by Frank Harrellā€™s exacting standard - at least having all papers reviewed by a qualified statistician to flag to ā€œHey, the authors used logistic regression with 15 variables when they have 21 events, they probably shouldnā€™t do thatā€ would be a start.

Perhaps only the upper tier of journals would be able to get statisticians interested. Hey, thatā€™s fine. At least we have taken a first step towards identifying journals whose results are more likely to be credible.

Iā€™m certainly aware that there are possible drawbacks here, too. Just thinking out loud of what we (I) can do other than Tweet snarky GIFs about poor-quality papers that get published and passed around as though they prove something when they do no such thing.

2 Likes

Thank you for your replies and suggestions. Iā€™m always looking to expand my pool of statistical reviewers. At least in other fields,there are reviewer databases, and early career researchers can throw their hats into the ring to be considered as referees. Is there something like that for statistical reviewers that you know of, or some other way to identify those in that community who might be interested in reviewing manuscripts? Thank you!

1 Like

didnā€™t help save this https://www.nejm.org/doi/full/10.1056/NEJMoa1715849?query=featured_home

i have wondered this too. I know only of publons eg here is stephen sennā€™s profile: https://publons.com/author/455068/stephen-senn#profile

Sorry for the delayed reply, have had a bit going on, but I wanted to come back to this.

I think this is very important. I put out a call on the American Statistical Association message boards earlier this year and received over 30 replies, admittedly from people of variable qualifications or interest (i.e. one gentleman replied and said he would be interested in reviewing 1 paper per year, which wasnā€™t especially helpful). But I think we must try harder to build a truly representative database of qualified statisticians who agree to be considered as referees for manuscripts. Some individual journals surely have their own pool, but my dream would be a full list curated by the ASA and/or RSS and/or other governing bodies - and furthermore, some way of assigning credit for completed reviews (I think thatā€™s part of the idea of Publons).

The truth is, I think there are many more statisticians lurking about who could be more engaged in peer review but for whatever reason have demurred. One problem is certainly incentives - there may be a feeling that absent compensation or a role on the editorial board (things that promote our careers) the time required to review papers is better spent elsewhere. I donā€™t feel this way, but Iā€™m sure some of my statistical colleagues do - theyā€™ve already swamped with work and grant deadlines, whatā€™s the benefit for them in reviewing articles in an area outside their clinical specialty that does little to build their CV or promotion file?

If you cannot offer a role on the editorial board and/or compensation for statistical reviewers, an option that I think some journals ought to consider is an ā€œexchangeā€ program of sorts - if a statistical reviewer completes a certain number of high-quality reviews (say, 12 in one year?) you will agree to publish a statistical editorial or column of their choosing (subject to editorial review and approval, of course). That gives a statistician some incentive to complete reviews for your journal because theyā€™ll get a publication out of it at the end, and lord knows we need more statistical-education pieces in the clinical literature.

Curious for othersā€™ thoughts.

2 Likes

The ā€œexchangeā€ idea is really interesting and Iā€™ve not heard that discussed before. I hope we see some discussion about that.

This makes my mind wander to an opposite approach: crowd-sourcing. What if a journal were to describe the study, experimental design, and statistical analysis in generic terms and get comments from lots of statisticians on a web site? I personally think that the discussion would be interesting, instructional to students and biomedical researchers, and would possibly make paper reviews less subject to the ā€˜luck of the reviewer drawā€™.

1 Like

I discussed it on Twitter briefly - not just about statistical reviews, but about finding reviewers in general - since economic realities may make it difficult for journals to pay all reviewers, one way to reward high-quality people who review faithfully is some sort of exchange. At first I suggested that reviewing X papers per year (assume that the editorial staff considers them good-quality reviews) should get you 1 ā€œexpedited reviewā€ for a paper of your choosing. Specifically for the statistician, giving us a platform to publish 1 statistical editorial in a journal for which we review >X papers per year/month would possibly be motivating. We will all naturally gravitate towards things that promote our careers - first author papers, editorial board positions, and money. If the latter two are not an option, maybe giving us a once-a-year platform as a reward for reviewing 12 papers in one year (or maybe 24 - two per month) seems like an attractive option, especially to early-career folks who would benefit from the chance to get a first/sole author paper.

1 Like

In my journal, I have created a Statistics Advisory Board precisely for this, so that these experts are indeed part of the journalā€™s board (and also, to have a small pool of experts who are both interested and willing to review for the journal), and we also reach out to them yearly to ask them about journal strategy/development. I donā€™t know if itā€™s enough of an incentive, but we are happy to offer that (in addition to APC discounts). Iā€™m happy to explore other ways of credit as well.

Regarding your other point, Iā€™d be more than happy to offer publication of such statistical discussions (after proper peer review). Iā€™ve consider it, but thought nobody would be willing to write one.

1 Like

It sounds like you are doing a terrific job, to be honest.

In general, I think three things will attract statisticians to review:

  1. Financial Compensation (obvious reasons)
  2. Editorial Board Membership (we can use this for promotion)
  3. Opportunity to write our own articles as reward for high-quality reviews

As for the last one, if you thought nobody would be willing to write, it will certainly be specific to the individual and some may not want to bother, depending on their career stage and workload. Iā€™ll merely note that at my career stage, getting a couple of extra single-author publications will be valuable in building a file for promotion and tenure, and I suspect many others (in the USA) at the rank of Assistant Professor would feel the same. In a higher position, some folks may feel the additional time to write a piece is not the best use of their effort.

Also, for everyoneā€™s amusement, hereā€™s a Twitter thread on an even worse offense that I turned up yesterday:

1 Like

This is good to know! I could mention in my invitation to join the board, that contributing a piece is part of the offer.

Anyway, if anybody here wants to join the board, let me know :slight_smile: