Reanalyzing MERIT design using Julia + recent numerical algorithms

davidcnorrismd · April 18, 2025, 11:21am

Here’s an interesting case of a paper that made it through peer review at Statistics in Medicine despite using Monte Carlo estimation without regard to its inherent error. An extensive tabulation and simulation study were presented, with absolutely no mention of MCSE’s! (As one would expect, this vitiates the reported results.)

Two interesting aspects of the above critique are:

Relatively recent numerical work [1] played a crucial role
Julia proved remarkably effective in rendering the reanalysis clearly

Frey J. An algorithm for computing rectangular multinomial probabilities. Journal of Statistical Computation and Simulation. 2009;79(12):1483-1489. doi:10.1080/00949650802286753

Eslam_Maher · April 27, 2025, 1:11pm

Thanks for sharing this feedback. I had read the paper, but these details go over my head.

However, generally speaking, there seems to be a need (following Project Optimus as well) to randomise in dose-optimization oncology studies. Yet, bec this typically requires large N, I don’t think this has happened in oncology (no? except when requested following later confusion in the development pipeline), whereas these dose-ranging studies are pretty typical outside of this domain.

There seems to be a plethora of designs, but most prominent of which are MERIT and DROID (from Ying Yuan’s team, I presume). The former, which you are discussing here, seems to be comparing each arm to predefined thresholds of toxicity & efficacy, rather than actually comparing arms to each other, resulting in an OBD admissable set (Stage 1). Do you find this as another valid point against MERIT?

davidcnorrismd · April 28, 2025, 10:07am

This is a keen observation, Eslam! It had not occurred to me to criticize MERIT on such grounds, since one [valid?] perspective on early-phase trials is that they are to inform the grossest kind of considerations, taking existing therapies as the comparator. (“Is this new drug in a reasonable ballpark to trial further, considering what we already have available? If so, which of several doses should we study?”) But I now think you have indeed identified yet another vulnerability here.

davidcnorrismd · June 13, 2026, 11:10am

After over a year of silence from the authors I’ve revived the critique this morning, posting the Statistics in Medicine DATA AND CODE POLICY,

and following up with an email to authors (cc’ing the Editorial Office) requesting the code for reproducing the Table 2 in question.

davidcnorrismd · July 8, 2026, 6:59pm

How it’s going …

davidcnorrismd · August 11, 2026, 5:47pm

After several attempts, I managed to get through to an actual person in the SIM Editorial Office with the following email, sent July 7:

My previous emails to the Editorial Office seem to have been handled by chatbots or similar automation, or else to have been dealt with by inexperienced personnel.

This email is about a published paper, not a manuscript under review or in production. Please do not ask automatically for a “manuscript ID”.

The DOI for the paper in question is 10.1002/sim.10093. This should enable you to identify the paper uniquely.

I am not asking an Editor to read and understand the PubPeer comments I have posted on this article, nor to form a judgment about the integrity of the work. (But FYI these may be looked up by DOI at pubpeer.com.)

My request to the Editor is simply to uphold your DATA AND CODE POLICY in relation to this paper.

Your policy reads [the bold emphasis is mine]:

Statistics in Medicine expects that data supporting the results reported in the paper will be archived in an appropriate public repository. Whenever possible the scripts and artefacts used to generate the analyses presented in the paper should be publicly archived. Exceptions may be granted for sensitive information such as data which allows the identification of individuals, at the discretion of the Editors. Authors will be able to complete a data accessibility statement which will be published with their paper. Submitting authors should upload their data and code as “Data Files” within the online submission system.

The journal also requires authors to supply any supporting computer code or simulations that allow readers to institute any new methodology proposed in the published article. This may be in any form(s) the authors feel is most accessible for use, including programming code, SAS commands, R functions, packages etc. Submitting authors should upload their code/simulations as “Data Files” within the online submission system.

I request specifically the code to reproduce the paper’s Table 2. To complete my critique of the paper, I will need to inspect this code, and modify it to apply a control variate technique. The web app which the authors reference in the paper is not sufficient for this.

With kind regards to any human who reads this email,

David C. Norris, MD

A July 9 reply informed me,

I have contacted the Production team and Editor regarding your query and am currently awaiting their decision. I kindly ask for your patience, and I will update you as soon as I receive a response.

On Aug 10, I requested an update:

This matter has now been with your Editor for at least 1 month.

Could you inform me whether the paper’s authors have provided code demonstrating the reproducibility of Table 2? If so, when will this code become available to readers?

Today’s (Aug 11) reply is:

Currently, there is no separate Code Availability Statement requirement or question within the submission system. As such, we are unable to confirm whether the authors have provided code demonstrating the reproducibility of Table 2 without reviewing the materials submitted with the manuscript.

If code has been provided, its availability to readers will depend on the authors’ Data Availability Statement and the journal’s data and code sharing policies. Where necessary, the Editors may request additional information or a code availability statement during the revision process.

We hope this clarifies the current process.

So in sum, it looks to me as if the Editorial Office are earnestly pondering whether a paper that slips through the cracks during review and production — managing to get published without adhering to the journal’s CODE POLICY — should get a ‘pass’ against future requests for code.