Evidence Based Decision Making and Meta-Analysis -- An Information Theoretic Critique

Andrew Gelman posted an interesting review of a paper that documents the failure of international public health authorities throughout the pandemic.

It might sound silly to say that people are making major decisions based on binary summaries of statistical significance from seriously flawed randomized studies, but that seems to be what’s happening.

After much reading and lurking here, I’ve come to believe that @Sander is right, and statistics needs to be taught in the context of information theory. An information theoretic POV leads to the following observations on what Gelman refers to as “the standard norms of evidence”:

  • Modern applied sciences face new challenges that require greater attention to rigorous study
    design, data collection, processing, and integration than the guidelines of “Evidence Based Practice” currently provides.

  • Evidence based practice instruction provides misleading heuristics in the evaluation of information from primary studies that are magnified at the level of information synthesis (Meta-analysis). It can be shown that these heuristics to evaluate individual studies are contradictory with the placement of information fusion (meta-analysis) at the top of the (categorical) evidence hierarchy. These heuristics are rejected by all philosophies of statistics.

  • The evidence hierarchy has a charitable interpretation of the expected discriminant information
    obtained from the choice of research designs, if the categorical interpretation is rejected, but is not useful for the evaluation of information/evidence in specific cases.

  • An information theoretic approach to evidence leads directly to formal decision theoretic considerations. In a scientific context, maximization of utility is equivalent to maximizing information received via experiment. Meta-analysis is not a separate research design, but a computational procedure to combine available information in order to decide upon the conduct of future studies.

  • A decision theoretic approach to evidence leads towards broadly Bayesian perspective on scientific questions and their applications.

  • Formal cost/benefit considerations that include the value of information (ie. new studies) will improve the allocation of resources to questions of true importance to stakeholders.

  • The frequentist perspective remains valuable; it can be also be modeled as a Bayesian agent with
    skeptical or vague prior beliefs, or a cost function.

  • This dialectic among various Bayesians with different priors (or utility functions) leads to a spectrum of covering priors that present the contextual boundaries of principled, scholarly debate.

  • Constructive debate within this framework is the assertion and critique of regression models to explain and predict future observations.

  • Information theoretic considerations aid in the application and assessment of recent developments in machine learning and theoretical computing science.

  • It can also be shown that the (categorical) hierarchy of evidence undervalues observational evidence.

  • Observational studies can be vastly improved by paying attention to information theoretic considerations, up to including the use of directed acyclic graphs (DAGS) to model causal and confounding factors.

Related papers:

  1. Trisha Greengalagh on mental models and public health
  1. Harry Crane on naive probabilism:
  1. Philip Stark and Andrea Saltelli on Cargo Cult Statistics:
  1. Frank Harrell on Errors in the Medical Literature.