Background and Comment
The UK analysis is based on data from the entire population of England. The data used by Xu et al. are from 7 United States health maintenance organizations, five of them organizations that are insured by the Kaiser Foundation Health plan and are integrated health systems (insured people receiving care receive that care in a defined set of hospitals from physicians and other health care providers who are employed or contracted by the hospital).
A priori, one should not assume that selection for vaccination of people in good health (positive selection) versus selection for vaccination of people in bad health (negative selection) would be the same for people in England and people who received their medical care in the seven United States health care entities whose data were used in the Xu et al. analysis.
Selection for COVID-19 vaccination of people in good health (positive selection) versus selection for vaccination of people in bad health (negative selection) may not have been the same in different age groups. For example, a person age 90+ years in bad health (e.g., with severe dementia and multiple co-morbid conditions confined to bed) may have been selected to NOT receive COVID-19 vaccination whereas a young person in bad health (e.g., with multiple sclerosis confined to a wheel chair) might have been selected to receive COVID-19 vaccination (or prioritized to receive it earlier than a person the same age in good health).
Further, selection factors—for and against vaccination based on health–may have changed over the period when COVID-19 vaccinations were being given. The changes in these factors over time would not necessarily be the same in England and in the seven United States entities contributing data to the Xu et al. analysis.
Data From England on Changes in Non-COVID-19 Mortality Over Time
Figure 2 of a report from England shows changes in the relationship between vaccination status and non-COVID mortality by age over time from the availability of vaccine in January 1, 2021 (when vaccine became available) through December 31, 2021.
Considering older people (ages 70-79, 80-89 years and 90+ years), all of whom were eligible for vaccination in England in January 2021 when vaccine became available, there is higher non-COVID-19 mortality in the unvaccinated compared with the vaccinated in first few months after vaccine availability and eligibility, but the non-COVID-19 mortality rates comparing the unvaccinated and the vaccinated converge by the end of 2021. (NOTE: The use of color in this figure is not ideal).
In the data from England, a similar pattern of higher non-COVID-19 mortality in the unvaccinated compared with the vaccinated in the initial period after age-defined eligibility to receive vaccine with convergence in non-COVID-19 mortality by the end of 2021 is also seen for people ages 50-59 and 60-69 years.
Thus, for older adults in England in 2021, there was initial positive selection bias (selection of healthy people for vaccination) considering COVID-19 mortality (i.e., vaccinated people have lower non-COVID-19 mortality—healthy vaccinee bias) with diminution of the positive selection bias over time until the end of 2021.
Finally, it is important to note that, by July 2022, the overall COVID-19 vaccination rates (2 doses) in England became very, very high (over 90% in people age 55+ years) and high (not less than 70%) in people age 18-49 years.
If everyone is vaccinated, it is impossible to study differences in non-COVID-19 mortality between the vaccinated and the unvaccinated because there are no unvaccinated people to be included in a comparison group.
Xu et al.
In the Xu et al. analysis, vaccination status was assessed from December 14, 2020 through June 30, 2021, and non-COVID-19 deaths were assessed until August 31, 2021, allowing at least two months of follow-up for all people included in the study.
No details are provided on eligibility to receive COVID-19 vaccine and how these might have differed between the seven participating health care organizations, which are located in, and provide services in, eight states (California, Oregon, Washington, Hawaii, Colorado, Minnesota, Wisconsin, Michigan). While the Centers for Disease Control and Prevention (CDC), through its Advisory Group on Immunization Practices (ACIP), provided guidance on initial prioritization of people for receipt of COVID-19 vaccinations considering age, status as a health care worker or first responder, residence in a long-term care or congregate facility, and co-morbid conditions. In practice, the priority for administration of vaccine and the timing of vaccine availability to priority groups was controlled by states, which differed in the way they set priorities and timed vaccine availability to priority groups.
Information on how programs to implement the priorities defined at the state level is sparse.
How vaccine was actually delivered to the people in the health plans whose data were included in the Xu et al. analysis was not presented.
However the bias arose, in the Xu et al. study population, there is clearly a positive selection bias operating. That is, the people in these seven health care entities who were at lower risk of non-COVID-19 mortality were more likely to vaccinated by June 30 2021 than people who were, at an identical time, unvaccinated. As Xu et al. acknowledge, the use of propensity score analysis to try to overcome this selection bias was inadequate to the task using the measures of co-morbidity and selection factors available to the authors. Thus, the analysis showcases the limitations of propensity score analysis to overcome selection bias even in a dataset based on high quality, well-documented data and conducted by researchers with considerable experience in analysis of “real world data.”
Positive or Negative Vaccinee Bias Is Not a Universal and Differences Between Populations Can Be Large
Comparison of these two analyses demonstrates that selection bias for (or against) vaccination considering underlying health is not universally positive (healthy people get vaccinated) or negative (unhealthy people get vaccinated). It is specific to the vaccine, the condition to be prevented, and the way that vaccine is delivered and taken up. As a comparison of these two analyses shows, the differences in the magnitude of selection can be large.