Use of machine learning experts and statisticians as referees

I have been asked to write an article on machine learning literature in Cardiology. One of my concerns is the limited pool of potential referees who can assess the ML methodology.

Q1: Does anyone know of any literature that looks at refereeing of ML in the medical literature?

Failing that:
Q2: Any literature on the use of statisticians as referees?

Q3: Also, what is your favourite paper reporting on common statistical errors in the medical literature?


1 Like

I don’t have an answer to your questions, but ask for clarification of Q3: do you mean a (statistics) paper “pointing out” some common error, or a (medical) paper in which the authors use incorrect methods?

P.S. She’s a radiologist (not a cardiologist), but Lauren Oakden-Rayner has written quite a lot about the use of AI in medicine. Most of her papers are specific to Radiology/medical imaging but there are some more general ones.

1 Like

RE: Q1. Frank is listed on these 2 recent papers that compare “statistical” vs “machine learning” in a health context. I don’t know of anything specific towards refereeing.

  1. Austin, P. C., Harrell, F. E., Lee, D. S., & Steyerberg, E. W. (2022). Empirical analyses and simulations showed that different machine and statistical learning methods had differing performance for predicting blood pressure. Scientific Reports, 12(1), 1-11. link
  2. Austin, P. C., Harrell Jr, F. E., & Steyerberg, E. W. (2021). Predictive performance of machine and statistical learning methods: Impact of data-generating processes on external validity in the “large N, small p” setting. Statistical methods in medical research, 30(6), 1465-1483. link

I’m reminded of this thread that was critical of using the term ML to make a paper seem more sophisticated.

RE Q3. This data methods thread has lots of relevant citations:


Thanks for the response, I mean the “pointing out” - I see Robert has reminded me of the Wiki with a reference collection of Common Statistical Myths which I will check out.

Thanks Robert - especially the reminder of the wiki! For this review I’m trying to side-step some debate by being “inclusive” of a lot of language to describe models and by recognising that at least some of what we are talking about are simply differences in language arising out of two cultures - Comp Sci and Statistics.

Hi John – the following is all about #2 (benefits of statistical review) and has a few potential leads to follow

This first paper is a survey of practice, but includes a section with several references (#s 10-17) about studies that specifically address impact of statistical review, which is what I think you’re after.

This next paper is about adopting statistical review in Psychology journals, but starts off with a brief list of papers that cover this issue from a (bio)medical perspective that might have some good leads too:

1 Like

Brilliant, thanks James.