Hi everyone,
I’m a junior statistician at the beginning of my career in medical research, currently working at a small research center. This week I attended a medical congress where I was hoping to learn how more established groups approach study design and data analysis in real-world settings.
One theme that stood out — and left me somewhat discouraged — was the overwhelming reliance on univariate screening followed by stepwise regression, especially in retrospective case-control studies. Many presentations were framed around identifying “clinical factors associated with” some outcome, such as progression vs. non-progression, treatment response or having certain concomitant disease. The common approach seemed to be: perform univariate tests for all candidate predictors, select those below a certain p-value threshold, and then build a multivariable model with the selected variables.
From what I’ve learned, particularly from Harrell’s Regression Modeling Strategies and the BBR course, this approach is statistically unsound and based on inherited but flawed practices. I know there are better ways — more principled and robust modeling strategies — but they can be harder to implement and communicate, especially in applied medical settings.
Here’s where I’d love your input: As someone just beginning my career, I want to suggest better alternatives when working with clinicians who are used to these workflows. But I sometimes struggle to articulate the “how” — particularly in the context of small or moderate-sized retrospective datasets, which are very common in my work.
How do you approach modeling in these “clinical factors associated with” frameworks when dealing with retrospective observational data? What strategies do you recommend — both statistically and in terms of communication — to steer teams away from univariate screening and toward better modeling practices?
Thanks so much in advance for your thoughts. I really appreciate any advice or shared experiences.
Best,
Albert
7 Likes
Our group predominantly uses causal diagrams (directed acyclic graphs; DAGs) to guide multivariable analyses as reviewed here. In addition to explicitly showing the key assumptions guiding our statistical models, DAGs also move us and our readers/reviewers away from the misconception that the data themselves can guide the covariates using a univariable regression or other such strategies. Example applied papers using DAGs in our retrospective medical research can be found here, here, and here.
We also nowadays have started to use more the “target trial” concept (related courses here) whereby we explicitly formulate the ideal RCT we would conduct to answer our causal question, and then use observational data to replicate as closely as possible the key design components of that RCT.
6 Likes
“Target trial emulation” worries me a good deal, only because it has given many authors an excuse for not doing the important DAG step that Pavlos describes so well, and for being subject to data availability bias.
The thought that univariable screening and stepwise variable selection are still being used in 2025 is very depressing to me.
An outstanding example of principled modeling as well as describing what goes wrong with variable selection may be found in @Sander’s incredible 2000 paper. We usually make use of the real data structure without random effects, but this is a great example where random slopes connect a series of related variables (food consumptions of various food constituents) in a beautiful way.
6 Likes
I have this problem frequently when I peer review the veterinary literature. However, most editors think univariable followed by stepwise selection is the preferred method. As a consequence most of the “risk factor” studies they perform have little clinical utility. I was told it may be appropriate for prediction but not explanation (not sure if this is accurate). I refer them to the paper mentioned here Sung G-W, Shook TL, Kay GL. Inappropriate use of bivariable analysis to screen risk factors for
use in multivariable analysis. Journal of Clinical Epidemiology. 1996;49:907-16
Heinze, G. and Dunkler, D. (2017), Five myths about variable selection. Transpl Int, 30: 6-10. https://doi.org/10.1111/tri.12895
4 Likes
Just saw this reference which is very useful. Many studies I review dont explicitly define the question of interest. On the Uses and Abuses of Regression Models: A Call for Reform of Statistical Practice and Teaching - PMC
6 Likes