Flu Vaccine Studies: Pseudo-Randomized or Untrustworthy Results?

 The validity of a study can be questioned through various methods, including identifying unexpected results. An example used in teaching appears in the discussion of yellow-stained fingers as a marker for lung cancer, where smoking is the real cause behind both issues. If researchers report that an association remains even after adjusting for smoking, it raises concerns about the study's reliability.

The VISION Network, referenced by the CDC, provides yearly estimates on the flu vaccine's effectiveness using a case-control study called the test-negative design. Cases and controls are derived from patients seeking care for respiratory symptoms, where cases test positive for flu and controls test negative. Since the COVID pandemic, VISION has been a key data source.

Reports from VISION consist of observational studies rather than randomized trials, resulting in differences between groups that can introduce confounding variables. Vaccinated individuals tend to be healthier, leading to a bias known as the healthy vaccinee effect, which can inflate the apparent effectiveness of the vaccine. In observational studies, adjusting for certain factors may reveal lower effectiveness estimates. For example, an unadjusted association showing a 50% effectiveness may drop to 25% after adjustments.

In randomized trials, randomization typically ensures that both groups receiving different treatments are comparable, eliminating the need for adjustments that could alter effectiveness estimates. The key takeaway here is that in observational research, adjustments usually result in lower effectiveness, and if estimates remain unchanged, it suggests potential issues with the data or analysis.

Researching year-to-year flu vaccine effectiveness presents challenges due to the complexity and volume of data, making it hard to spot questionable results. To simplify analysis, one should narrow focus to specific expectations and results. In reviewing VISION data from the elderly over the years, certain trends were noted, including unexpected low effectiveness against outpatient visits and consistently similar effectiveness across different clinical settings.

In the 2022–2023 season, effectiveness estimates showed around 40% in both outpatient and inpatient settings despite minimal adjustments. The studies suggested either significant biases were canceled out or that the data bore witness to miraculous adjustments. The subsequent seasons, 2023–2024 and the next two, also showed instances of negative confounding, which contradicts expected biases from the healthy vaccinee effect.

Despite analyzing multiple estimates across various seasons, findings revealed a concerning lack of adjustment methods for confounding. Half of the estimates revealed potential data substitution for randomized trials, while the rest indicated negative confounding. This leads to the need for new studies to assess flu vaccine efficacy, suggesting cohort studies might be more insightful than the existing design. The ongoing struggle with flu vaccine effectiveness raises questions about its consistency and the influence of confounding variables, indicating a potential need for a reevaluation of how vaccine effectiveness is studied.

https://brownstone.org/articles/flu-vaccine-studies-pseudo-randomized-or-untrustworthy-results/