mRNA vaccine effectiveness matched case-control design

We have created a preprint assessing over 400000 consecutive cases of COVID-19 using a hard endpoint - admission to ICU - to assess mRNA vaccine effectiveness. We got very different results from the main-stream papers in NEJM and others that used a test negative case-control design or matched cohort studies. These results demonstrate no waning of effectiveness at 12 months after the second dose. We attribute this difference to the designs used previously versus the design we used
Any thoughts?


sorry, i stay away from researchgate. Any reason you dont put it on ? or can you attach the pdf here? have you evaluated different matching strategies? you match 1:1?

Researchgate is easy - one click upload and all authors etc are extracted and put in the right place. Also one click link to published version and unique DOI so better than MedRxiv except for the PubMed link that happens with MedRxiv. To get the PDF just click on “More” on the right side of “references” and then click “download”.

Matched on calendar month of infection (accounts for the circulating variant) and comorbidity burden but not age as categorizing age for matching would lead to residual confounding as it is the most critical factor in severity. We adjusted for age (restricted cubic splines with 4 knots). Matching was 1:30 as we had a large dataset.

Your data would be easier to compare with findings reported in the 2022 systematic review and meta-regression published in the Lancet if Figure 2 presented point estimates and the upper and lower bounds for the 95% CIs and estimates of effectiveness (and their CIs) derived from the ORs and their CI’s. Calculating effectiveness from an OR easy arithmetic, and the estimated effectiveness in preventing hospitalization given infection by time interval after vaccination is about 90%-95% for the intervals after vaccination presented in your study (except the earliest).

In your matched case-control design, cases are defined as positive test for SARS-CoV-2 with severe disease and controls are defined as positive test for SARS-CoV-2 without severe disease. This design is unique among studies of this topic that are included in the 2022 systematic review.

As pointed out in the Discussion section of your paper, the overall conclusion of the authors of the Lancet paper about the effectiveness of COVID-19 vaccinations against severe disease is presented in the last sentence of the Findings section of the Abstract.

“For severe COVID-19 disease, vaccine efficacy or effectiveness decreased by 10·0 percentage points (95% CI 6·1–15·4) in people of all ages and 9·5 percentage points (5·7–14·6) in older people. Most (81%) vaccine efficacy or effectiveness estimates against severe disease remained greater than 70% over time.”

Feikin DR, Higdon MM, Abu-Raddad LJ, et al. Duration of effectiveness of vaccines against SARS-CoV-2 infection and COVID-19 disease: results of a systematic review and meta-regression. Lancet. 2022;399:924-944. doi: 10.1016/S0140-6736(22)00152-0.

The estimate of the decrease in effectiveness is derived from a meta-regression that includes all studies identified in the systematic review and data about the following vaccines: Janssen (Ad26.COV2.S), Vaxzevria (AstraZeneca), Pfizer–BioNTech/Comirnaty and Moderna (mRNA-1273). See Table 2 of the Lancet paper.

A careful look at the individual studies that are the source of data for the meta-regression shows a lot of variation in the estimates of vaccine effectiveness by time since the last vaccine dose. There are at least two studies included in the meta-regression that report estimates of effectiveness in the period up to about 6 months after the last vaccine dose that are in the same range as the estimates from your analysis of data from Qatar:

Skowronski DM, Setayeshgar S, Febriani Y, et al. Two-dose SARS-CoV-2 vaccine effectiveness with mixed schedules and extended dosing intervals: test-negative design studies from British Columbia and Quebec, Canada. medRxiv. 2021 doi: 10.1101/2021.10.26.21265397. published online Oct 26. (preprint)

Lin D-Y, Gu Y, Wheeler B, et al. Effectiveness of covid-19 vaccines in the United States over 9 months: surveillance data from the state of North Carolina. medRxiv. 2021 doi: 10.1101/2021.10.25.21265304. published online Oct 26. (preprint)

Further, as I understand it, reading your paper and a previously published paper about vaccine effectiveness from Qatar (New Engl J Med 2021; 385: e83), virtually all people vaccinated in Qatar received one the two then-approved mRNA vaccines: Pfizer–BioNTech/Comirnaty and Moderna (mRNA-1273). While the data are not completely clear, an inspection of the data in the systematic review and a read of the primary literature suggests that the effectiveness against severe disease for the Janssen (Ad26.COV2.S) and Vaxzevria (AstraZeneca) may fall more with time since vaccination than for the two mRNA vaccines.

US Food and Drug Administration FDA briefing document, EUA amendment request for a booster dose for the Janssen COVID-19 vaccine. Oct 15, 2021.

Andrews N, Tessier E, Stowe J, et al. Vaccine effectiveness and duration of protection of Comirnaty, Vaxzevria and Spikevax against mild and severe COVID-19 in the UK. medRxiv. 2021 doi: 10.1101/2021.09.15.21263583. published online Oct 6. (preprint).

There are, however, some data that do not support this speculation. Thus, the previously cited study by Skowronski reported continued high effectiveness (>90%) in preventing hospitalization 6 months after the last dose for the Vaxzevria (AstraZeneca) vaccine

Skowronski DM, Setayeshgar S, Febriani Y, et al. Two-dose SARS-CoV-2 vaccine effectiveness with mixed schedules and extended dosing intervals: test-negative design studies from British Columbia and Quebec, Canada. medRxiv. 2021 doi: 10.1101/2021.10.26.21265397. published online Oct 26. (preprint)

The problem of confounding by co-morbidity is a serious one for this topic. As your paper points out, it is difficult to address with most “simple” adjustments and data capture in electronic records may be poor. Your study, in which cases and controls are closely matched for the number of co-morbid conditions using a data source that attempts to capture information on co-morbidity uniformly, may have done a better job than others in controlling for differences in co-morbidity between people vaccinated early and late.

IMHO, it seems reasonable to point out that your design might better take into account differences in co-morbidity, but this is only my opinion.


Thanks for your detailed comments and yes this is the only study that affirms robust immunity till 12 months against severe disease.

The Lancet MA was based essentially on test negative CC studies or cohort designs and both are associated with several biases that we got rid of using only an infected cohort from which was extracted a matched CC study. We matched on calendar month of infection eliminating the impact of changing variants over time impacting these results

A new NEJM study that used the same data we used did not concur and their methodology is summarized below:

-The study compared the incidence of symptomatic breakthrough SARS-CoV-2 infection among persons in booster cohort vs non booster cohort

*-Population: All persons who had received at least two doses with No previous documented infection before the start of follow-up. BNT162b2 vaccine between January 5, 2021 and January 26, 2022. *
*mRNA-1273 vaccine between January 24, 2021, and January 26, 2022 *
Study groups: matched cohort study with booster group and non-booster group compared
*Booster group: who had received the booster dose more than 7 days previously *
*Non-booster group: had not received the third dose of vaccine by the beginning of follow-up of the matched pair. *

*Persons in the booster group and those in the non-booster group were matched exactly in a 1:1 ratio according to *
- Sex
- 10-year age group
*- nationality *
*- calendar week of the second-dose vaccination *

Follow up
*Follow-up began on the eighth day after the person in the booster group received the booster dose, provided this day occurred on or after December 19, 2021. *

*- Persons in the booster group who had received the booster dose at least 7 days before December 19, 2021 were followed along with their matched controls in the non-booster group beginning on December 19, 2021. *
*- Controls in the non-booster group who received the booster dose at a future date were eligible for recruitment into the booster group, provided they were alive and infection-free at the start of follow-up. *
- Accordingly, some persons contributed follow-up time both as persons who had received only a two-dose primary series and as persons who had received a booster, but at different times.

*Follow-up continued until the first of one of these events: *
- a documented SARS-CoV-2 infection (defined as the first positive PCR or rapid antigen test after the start of follow-up
*- booster vaccination of the control (with matched pair censoring), Data on both members of each matched pair were censored once the control received the booster dose. *
*- death, *
- the end of study censoring (on January 26, 2022).

- Groups were compared with the use of standardized mean differences: <=0.1 indicating adequate matching.

*- The cumulative incidence of symptomatic infection – *
% of persons at risk in whom symptomatic infection occurred during follow-up
estimated in each cohort using Kaplan–Meier estimator method.

*- The incidence rate of symptomatic infection in each group- *
number of identified symptomatic infections divided by the number of person-weeks contributed by all persons in the group
*estimated using Poisson log-likelihood regression model *
Stata stptime command.

- The hazard ratio –
*for the between-group comparison of the incidence of symptomatic infection *
with adjustment for the matching factors
*using Cox regression *
*Stata stcox command. *

*- The effectiveness of booster vaccination, as compared with that of the two-dose primary series, was estimated with the following equation: *

vaccine effectiveness=
1−adjusted hazard ratio for the incidence of symptomatic infection

The NEJM paper concluded that “mRNA boosters led to strong protection against Covid-19–related hospitalization and death” while we showed that the primary schedule maintained strong protection with a marginal added benefit for boosters given between 6-8 months after the second dose

Outcomes were sparse in the NEJM method for severe disease (15 versus zero with the main vaccine analysis) and I think the main issue here is residual confounding by age (10 year age groups is grossly insufficient for the single most important risk factor
Nationality matching was unnecessary as there is no association between nationality and vaccination in Qatar so cannot be a confounder. The booster analysis took place mainly in the omicron wave that is less associated with severity as well

The main focus was on infection rates and I am not sure that preventing infection can be achieved with vaccination given the rapid waning of effectiveness against infection - policy decisions must prioritize effectiveness against severity

Thanks for sending the new NEJM study.

It is always tough when different analyses of data attempting to address the same questions and using data from the same setting arrive at what appears to be conflicting conclusions. This newest NEJM paper seems to conclude that getting a booster decreases the risk that an infection will lead to severe disease whereas your analysis suggests that two-doses of the mRNA without boosting maintains a very high effectiveness (>90%) against severe disease through a period 12 months after the initial series. Your data are balanced for virus subtype because of matching and the high effectiveness appears to apply to all of the subtypes.

I agree with you that careful and complete control (in an epidemiologic sense) for age is essential to a valid conclusion about prevention of severe disease and that matching in 10-year age buckets is not especially good control for age. The problem would seem particularly serious for individuals at older ages (e.g., >70 years) since these individuals are at high risk of severe disease.

There were 6,219 individuals age >70 in the boosted group and “only” 13,832 in the non-boosted group. With one-to-one matching, it is possible (or even likely) that the study “ran out” of unboosted people (or had to use a lot of unboosted people as matches more than once) using a closer specification of age similarity. Perhaps the authors should be asked why they chose to match +/- ten years and not +/-, for example, 2 years.

At the younger ages, especially the middle ages (age 50-69 years), I would be very concerned about an imbalance in co-morbidity between those boosted and unboosted when considering the risk of severe disease. Without any data, I would guess that getting a booster at these ages is related to a perception that the person is at high risk of severe disease because of co-morbidity. The perception could be the individual’s perception or the perception of their physician. I doubt that rushing to be first in line for a booster was a random event.

I agree with you that matching by nationality probably was not necessary. It probably didn’t hurt.

The apparently conflicting data will surely drive the systematic reviewers and meta-analysts nuts!

Agree with the points you have raised. When I graph the age severity relationship from our model this is what we get (duration held constant):

We got a rejection without review from JAMA

Those manuscripts judged unlikely to succeed through stringent external review or whose subject matter does not meet our current editorial priorities are rejected at that point.

It is at odds with the waning immunity literature but time will tell if this is right or the others are right - we will have to submit elsewhere

1 Like

I have read the methods and results, but I have some queries/points:

  1. All your cases were positive for SARS-CoV-2?
    If so, you aren’t estimating VE but conditional progression to ICU on severe covid (I don’t remember any paper evaluating waning in this specific outcome)
  2. In your stratification for vaccination status, use the period between 0-13 days after the first dose (and consecutive doses) is suitable to detect potential bias (
  3. If you have so many controls, why match by month? One month is a huge time window for COVID.
  4. If you want to evaluate VE for ICU admission, you could match cases and controls (in the test negative design) and only model those sets in which the case progresses to ICU.

PS: One very good protocol/paper is from Thompson: 10.1056/NEJMoa2110362

All your cases were positive for SARS-CoV-2?
If so, you aren’t estimating VE but conditional progression to ICU on severe covid (I don’t remember any paper evaluating waning in this specific outcome)

This may be true if you limit vaccine effectiveness (VE) to its ability to prevent transmission and spread of the virus. However if effectiveness in preventing disease severity is the goal then it matters little if all participants studied were all positive as that is the least biased way of assessing this. VE is indeed vaccine conditional reduction in progression to severity.

In your stratification for vaccination status, use the period between 0-13 days after the first dose (and consecutive doses) is suitable to detect potential bias

This is not a test-negative case-control study so this does not apply

If you have so many controls, why match by month? One month is a huge time window for COVID

Not really - it takes at least a month for a variant of interest to be replaced by another or for a new wave to subside or emerge

If you want to evaluate VE for ICU admission, you could match cases and controls (in the test negative design) and only model those sets in which the case progresses to ICU.

I do not see the advantage of doing so since the test-negative case-control study design is a more potentially biased design than what we used. I think you are assuming that VE implies that prevention of infection be a required component and if so why?

It is not true, once you don’t know how is the true “model” of vaccines (leaky or all-or-none), if they are all-or-nothing or leaky ( 10.1016/j.vaccine.2012.08.045 | 10.1155/2014/813789 | others)
If you follow the WHO guidelines of VE measurement:
#6.2 Case-control studies
In the case-control design, investigators identify individuals who were diagnosed with COVID-19 (i.e.
cases), and a comparison group of individuals who were not diagnosed with COVID-19 (i.e. controls).
Various approaches to selecting controls from the underlying population have been proposed; concurrent
enrolment of cases and controls at the same time is encouraged to minimize time-variant exposure
differences (58). COVID-19 vaccination history is then determined for all cases and controls, and the odds of vaccination in each group is calculated. Control enrolment should be concurrent to cases by time (e.g. onset of symptoms or testing) in a high-incidence setting, as with COVID-19 in most settings currently.

So, your definition falls in the conditional effectiveness: Estimating Conditional Vaccine Effectiveness by John P.A. Ioannidis :: SSRN

The matching by time isn’t useful only for VOCs, it is useful to adjust for unmeasured confounders such NPI / force of infection / others

The TND has many advantages over the traditional case-control in the context of VE studies.
PMID: 31430265/

As you are evaluating only cases already in hospital, there are some concerns between time of infection and hospitalization, so it is possible to people only seek health care after more severe symptoms affecting the Odds of require ICU for different conditions other than vaccination status.

It is due to the Leaky vs All-or-nothing

This paper takes your definition as well and says
"“When the hospital sample was limited to 7283 admissions to an ICU, the effectiveness of full mRNA-based vaccination against laboratory-confirmed SARS-CoV-2 infection leading to ICU admission was 90% (95% CI, 86 to 93)”*

This was a test-negative CC design and so the odds of testing positive for SARS-CoV-2 among vaccinated patients compared with the odds among unvaccinated patients was assessed in a restricted cohort where ALL the test positives led to ICU admission (N=7283). The fundamental problem here is that infections first need to be transmitted and then paired with host factors that lead to ICU admission. Thus this sort of restriction means that there is exaggeration of unmeasured differences between cases and controls such that the restriction has the potential to introduce serious bias

So, your definition falls in the conditional effectiveness

Agree but we called it effectiveness as the condition is crystal clear here and in fact this paper by Ioannidis perfectly illustrates our chain of thought - if we had seen this paper earlier we would have referenced it before the next submission

…NPI / force of infection

We actually also mentioned in the paper that temporal trends in NPI were likely accounted for by matching on calendar month - I am not aware of any important factors that change significantly in days but you may suggest some if you think they exist

As you are evaluating only cases already in hospital…

No we are not - the study base was ALL first detected infection in individuals in the country in the specified period. Those that got hospitalized and into ICU did so later

The TND has many advantages over the traditional case-control

Perhaps but does not have the clear advantages of this CC study derived from all positive tested as the study base and using “conditional” effectiveness (will start to use the terminology of Ioannidis :grinning:)

It is due to the Leaky vs All-or-nothing

Please describe exactly what you mean

1 Like

The leaky vs all-or-nothing type of one vaccine can work (these papers: 10.1016/j.vaccine.2012.08.045 | 10.1155/2014/813789 |), one vaccine can protect each individual by “challenge”, so after X challenges he gets infected or he can always protect some individuals but 0 protection in another.

In this paper he is still comparing cases who progress to ICU vs controls (not covid-19), just look at table s16 and there is a % of positive SARS-CoV-2.

I have seen the paper here and cant see any immediate utility of this information given that all mRNA vaccines are leaky (to use the authors language) and there is no debate over this. Does this matter to policy makers?

More important issues are at stake and the most pressing is that important confounders are poorly accounted for because of sparse data in TND or cohort studies. We completely avoid this in our paper. To take an example here and here we get age related differences in VE. This is not true and simply reflects huge amounts of residual confounding due to age which should really not affect VE (a relative measure) if modeled properly but certainly will impact the NNV (absolute measure). Yet we find very recent papers in the Lancet, NEJM and JAMA analyzing VE in broad age groups and reporting such age differences for VE that simply reflect residual confounding and nothing more - yet these papers get prioritized for publication and when we do not go down this path in our paper and point out this issue, editors seem to miss the point completely

After looking at the paper in more detail, I now understand what you mean by leaky vaccines and the use only of cases as the author says this about the “leaky” vaccines (aka mRNA)

“We showed that associations between subject covariates and vaccine efficacy will be biased unless those covariates are distributed equivalently in all risk groups. A simple diagnostic analysis of the risk of infection among placebo recipients as a function of the covariate could be used to reject the hypothesis of independence that is required for interpreting correlations with vaccine efficacy as indicative of differential efficacy rather than differential baseline risk, but this is not possible in a “case-only” analysis (which evaluates the association only among infected subjects).”

This only applies to “risk of infection” and why would a case only design have risk of infection as the outcome? I don’t think this makes sense and perhaps the author is unaware that covariate independence aka removal of confounding can be achieved by appropriate regression adjustment after accounting for non-linearity which was what I was complaining about in other designs in the previous post. The most important covariate is age when severe COVID is the outcome and age independence has been very poorly achieved in previous studies in Lancet and NEJM to the extent that age-effects on VE are being discussed without realizing that this implies residual confounding

After thinking about this some more, how do we have only cases in a VE study? There have to be controls too. So I implicitly assumed that controls were COVID detected cases in my response above otherwise the whole argument from Edlefsen makes little sense

We finally got a review rather than reject without reason thus far - this time from Intensive Care Med

Not a ICU topic; more public health topic
Does not take into account the different variants; vaccines are not equally effective against different variants
it was also a surprise that the authors did not find a decrease protection over time since that is found in other studies (UK Health Security TechnicalBriefing_31Dec21_Omicron) and SRMA

The last comment is interesting in that we did not conform to findings till date - seems like a bias verdict by convention rather than by science

1 Like