In the preprint Effect of Convalescent Plasma on Mortality among Hospitalized Patients with COVID-19: Initial Three-Month Experience from Mayo Clinic by Joyner et al, an observational study of the use of convalescent plasma (CP) for treatment of patients with COVID-19 was presented. The FDA has given almost unprecedented emphasis to this paper by using it as the primary basis for granting an emergency use authorization for the use of CP in a highly politicized environment.
The purpose of this topic is to start a discussion about the methods used in the Mayo Clinic paper and how the paper has been interpreted and misinterpreted by the FDA. It is hoped that others will write more in-depth evaluations of the paper’s methods. Here are some initial thoughts.
- The authors provided a list of reasons why it was not possible to randomize patients to CP from the start. These reasons are singularly unconvincing.
- There is no control group in the paper so the authors and FDA relied on evidence for an association between earlier transfusion and waiting time for transfusion <= 3 days having a lower 7d or 30d mortality than transfusion done at 4+ days, and a putative “dose-response” relationship between IgG antibody levels of donor plasma and mortality.
- The study design assumes there are no deaths before 4 days, and I can’t find data on frequencies of death by day. I am also uncertain about the definition of “time zero”.
- By categorizing transfusion time and IgG the authors failed to provide the needed information with enough resolution, and left the analysis open to gaming as discussed by Howard Wainer who showed that from the same set of data cutpoints can be found that yield a positive association and different cutpoints can be found that demonstrate a negative association. It is very unlikely that the authors engaged in such gaming but there was no related documentation in the paper.
- The “low”, “medium”, and “high” IgG level intervals used in the paper involve intervals that are too wide resulting in much heterogeneity within the intervals.
- IgG levels were available in < 0.1 of the patients and the authors did not do a propensity analysis to show predictors of having IgG levels determined as a function of patient baseline characteristics.
- The authors lost a major opportunity to provide the reader and FDA with continuous joint analysis of IgG levels and day of transfusion using e.g. logistic regression with flexible nonlinear effects of IgG and transfusion day.
- The time-dependent nature of the treatment (transfusion) was not explicitly used in the analysis. Transfusion is an “internal time-dependent covariate” which presents special analytical problems. The authors seemed to analyze the data as if transfusion is a baseline covariate.
- As @Biomaven suggested below, the analysis needs to adjust for a secular trend. This can be done by included in a model a restricted cubic spline in days from 2020-01-01 with 7 knots to allow for changing background therapy.
- Likewise, in doing a propensity analysis of IgG being measured, calendar time should be flexibly included in the binary logistic model.
- There is confusion about relative vs. absolute treatment effectiveness estimates. When referring to a 37% reduction in mortality, I recommend that the two absolute risk estimates be emphasized and the 37% be greatly de-emphasized.
- An FDA tweet made the patently false claim that CP benefits 0.35 of patients. As @Stephen has written many times, a parallel-group study simply cannot do what a 6-period randomized crossover study can do: provide an estimate of the proportion of patients who benefit from a treatment. Parallel-group data cannot distinguish all patients getting a small benefit from 0.35 of patients getting full benefit and the remaining 0.65 getting no benefit. There is absolutely no way to get the 0.35 figure from the Mayo Clinic paper.
My Vanderbilt colleague Jill Pulley provided the following question and an interesting way to think about how a study without a control group may mislead:
- The part I don’t understand even from the Mayo paper is how they could say it is consistent with efficacy when it could also be consistent with the low titer units being unsafe. The Mayo study measured donor antibody levels in sera using the Ortho-Clinical Diagnostics VITROS Anti-SARS-CoV-2 IgG chemiluminescent immunoassay. This is qualitative assay based on a recombinant form of the SARS-CoV-2 spike subunit 1 protein. Results of this assay are based on the sample signal-to-cut-off (S/Co) ratio, with values <1.0 and ≥1.00 corresponding to negative and positive results . However, they created a “semi-quantitative” interpretation of this assay and established relative, low, medium and high binding antibody levels by setting thresholds for low and high based on the ~20th and ~80th percentiles of the distribution for the S/Co ratios, respectively. To us, a key issue to consider in this approach is that if their ‘low’ category included sub-therapeutic doses with non-neutralizing concentrations, rather than being removed from the pool of available plasma (as in our trial), these can theoretically trigger antibody-dependent enhancement (ADE) which can facilitate virus uptake and subsequent worsening of symptoms. For this reason, doesn’t it seem the group receiving ‘low’ titer plasma could have actually been made WORSE, causing the ‘high’ group to look relatively efficacious and fostering an erroneous interpretation of the results???
- A dose-response study without a control group is analogous to having an interaction in a regression model without having the main effect. The interaction analysis may show the slope of the dose in the right direction, but the lack of main effects causes the intercepts to be incorrect, so that you don’t know where to vertically position the line.
The redaction of FDA personnel names in the FDA EUA memo is troubling: