Dear DataMethods users,
In this post, I would like to propose what I consider to be an important attempt in the social and epidemiological domain, in light of the highly polarized climate surrounding COVID-19 vaccines. A new study analyzing adverse effects has recently been published. As often happens, the study has been widely praised and criticized - at least within my information bubble - purely based on personal beliefs. I reached this conclusion because, in my view, there has been no real analysis of strengths and limitations, a task that requires broad multidisciplinary discussion.
Such multidisciplinary discussion is the foundation of this post. The aim is not to blame the authors of the study nor to claim that the study should be retracted, but rather to do the hard work that is expected of a scientist: to doubt (in the sense of analyzing, investigating, questioning). The idea is to adopt an unconditional approach as much as possible. The only non-unconditional premise is related to error-cost sensitivity: my strategy was essentially to “draw every plausible arrow” in the causal diagrams, that is, to list any potential source of bias without presupposing its magnitude or importance in the causal path. In this way, we may initially err on the side of “false positives” and then progressively discard the less plausible options, or at least rank them by relevance. This is consistent with the metal-detector approach, based on the premise that the costs associated with a false negative are lower than those associated with a false positive, as “Safety surveillance systems should be optimized for high sensitivity, erring on the side of caution by ensuring true associations are not missed.”
Therefore, I will now propose a list of possible biases, the assessment of whose plausibility goes beyond my expertise. I have tried to compile this list to the best of my ability to not waste your time. I kindly ask for your help in understanding which ones deserve further investigation and which can instead be considered of secondary importance or even discarded.
Thank you very much to everyone who decides to take part.
===============
A. Confounding and covariate measurement
Residual confounding by health behavior and socioeconomic factors.
Even after IPTW adjustment for age, sex, region, area-level deprivation index, CSS, and 41 comorbidities, important predictors of both vaccination and mortality (education, occupation, smoking and alcohol patterns, physical activity, adherence to medical advice, trust in health care, etc.) may remain insufficiently captured. Given the reported E-values, I consider the scenario still highly compatible with possible unmeasured (or poorly measured) factors of this type, which remain fully consistent with the observed association under alternative model specifications.
Confounding by prior SARS-CoV-2 infection and sequelae.
Prior infection appears to be captured only when it results in hospitalization (“severe COVID”). Non-hospitalized infections, long COVID without hospitalization, and differential testing and diagnosis patterns are not fully accounted for. These factors may be associated with both vaccine uptake and medium-term mortality, so some degree of confounding from prior infection history cannot be ruled out.
Crude socioeconomic surrogates.
The area-based deprivation index and CSS coverage are, at best, partial proxies of individual socioeconomic position. Fine-grained SES (education, income, occupation) and social marginalization may still differ between vaccinated and unvaccinated individuals, potentially in ways that affect mortality risk.
Underascertainment of lifestyle comorbidities.
Obesity, smoking, and alcohol disorders are identified from diagnoses and reimbursement data, which often under-record such conditions. If underascertainment differs by health care use or by vaccination propensity, this could leave residual confounding by lifestyle and risk behaviors.
B. Exposure definition and treatment-regime issues
Ill-defined treatment regime (“real-world package”).
“Vaccinated” is defined as having received a first mRNA dose in May-October 2021; subsequent doses and boosters are not modeled explicitly. The comparison therefore appears closer to two evolving real-world care pathways (mRNA vaccination plus whatever follows vs delayed or absent vaccination) than to a clearly specified intervention such as “maintain vaccination status X over time”. This may complicate causal interpretation of the estimated contrast.
Censoring of unvaccinated individuals at later vaccination (potentially informative).
Unvaccinated individuals who subsequently receive a COVID-19 vaccine are censored at the time of vaccination, apparently without inverse probability of censoring weighting. If the decision to vaccinate later is related to health changes or perceived risk, the resulting censoring mechanism may be informative and could bias hazard ratio estimates, although the direction and magnitude are uncertain.
Potential misclassification of vaccination status (e.g. fraudulent passes).
Misclassification of some unvaccinated persons as vaccinated (or vice versa) due to fraudulent or erroneous records has been discussed in this setting. The extent of such misclassification in this cohort is unclear, but if present and differential by health status, it could distort the estimated association.
C. Time, follow-up, and selection
Synthetic time-zero and remaining time-related biases.
Time zero is defined as 6 months after an “index date”; for unvaccinated individuals, the index date is randomly assigned to mimic the distribution of first-dose dates among the vaccinated. This strategy is intended to address immortal time bias and align calendar time. However, the use of a synthetic time origin for unvaccinated individuals may introduce subtle discrepancies between the biological exposure process and the analytic time scale, and the residual impact of such discrepancies is not fully transparent.
Restriction to individuals with at least one reimbursement in 2020.
Limiting the cohort to persons with at least one health care reimbursement in 2020 selects a subpopulation with some minimal engagement with the health system. This may reduce extreme differences between groups, but it also conditions on health care use, which itself may be related to both vaccination and mortality. Whether this restriction induces additional selection bias is not entirely clear.
Survivor selection at November 1, 2021.
Only individuals alive on November 1, 2021 are included. Any differences in early mortality (due either to vaccination or to infection prior to that date) are, by design, excluded from the analysis. As a result, the causal contrast pertains to a selected group of survivors, and does not address potential early differential survival up to that calendar date.
Limited horizon for cause-specific mortality and long-latency effects.
Cause-of-death data are available only through December 31, 2023, while all-cause mortality is followed to March 31, 2025. More generally, the follow-up horizon is about 4 years from the index date. Thus, any etiologic effects that mainly manifest as mortality beyond this time frame (e.g., subtle acceleration of certain oncologic or neurodegenerative processes) would not be detected. I think this limitation should at least be acknowledged and discussed in terms of biological plausibility and, where possible, in light of historical experience with previous large-scale vaccination/pharmacological campaigns and epidemics.
D. Weighting, model specification, and age effect modification
*
Unreported distribution of IPTW weights and potential instability.*
The authors report good post-weighting balance (absolute standardized mean differences < 0.1), but do not present the distribution of IPTW weights nor describe any trimming or truncation of extreme weights. Without this information, it is difficult to assess whether the estimates depend heavily on a small set of highly weighted individuals, which could make the results fragile or even highly artificial despite apparently good covariate balance.
Coarse age categorization and limited exploration of effect modification.
Age is categorized into four wide bands (18-29, 30-39, 40-49, 50-59) for both adjustment and stratified analyses. These broad groups may obscure more nuanced age-patterns of effect modification, and the choice of cut points does not appear to be motivated by a specific causal model. This could limit the study’s ability to detect heterogeneous effects across narrower age ranges.
**
E. Negative control outcomes and calibration**
Uncertain suitability of chosen negative control outcomes (NCOs).
Traumatic and unintentional injury hospitalizations are used as negative control outcomes under the assumption that they are not directly affected by vaccination status, while sharing similar confounding structures. However, such events are influenced by mobility, occupational and recreational activities, and other social behaviors, which may differ systematically by vaccination status and may themselves have been affected by vaccination policies. It is therefore not obvious that these NCOs share the same confounding structure as all-cause mortality, nor that they are entirely unaffected by the exposure. Calibration based on these outcomes might over- or under-correct for residual confounding.
F. Hazard-ratio–specific concerns
Single long-term average HR in the presence of time-varying effects.
The primary result is a single IPTW Cox hazard ratio over roughly four years of follow-up, despite evidence that the association varies over time. As Hernán has emphasized, such long-term average HRs can be highly sensitive to the length of follow-up and to departures from proportional hazards, and may not correspond to a well-defined causal parameter.
Period-specific HRs potentially affected by depletion of susceptibles.
The reported 3-month period-specific HRs condition on surviving and remaining in one’s exposure category up to each time point. If the vaccinated group experiences earlier events among its most susceptible members, later-period HRs can be attenuated or even fall below 1 purely due to this selection mechanism. Thus, period-specific HRs may be difficult to interpret causally without additional structural modeling.
Absence of standardized survival curves and risk contrasts at prespecified times.
The analysis does not present adjusted survival curves or cumulative risks under explicit vaccination vs non-vaccination strategies, nor risk differences/ratios at prespecified time points. Such summaries could avoid many of the conceptual issues associated with hazard ratios and might provide more directly interpretable causal contrasts.
