RMS Introduction

Regression Modeling Strategies: Introduction

This is the first of several connected topics organized around chapters in Regression Modeling Strategies. The purposes of these topics are to introduce key concepts in the chapter and to provide a place for questions, answers, and discussion around the chapter’s topics.

Overview | Course Notes

Key points featured in Chapter 1:

  1. The regression modeling framework for statistical analysis accommodates all the fundamental statistical interests of estimation, hypothesis testing, description, bias mitigation, and prediction;
  2. The ultimate measure of statistical veridicality is “accurate prediction of responses for future observations”—so all inferential statistical objectives are optimized in as much as model predictions are performant;
  3. Every hypothesis test implies a model—and models, moreover, make many of the underlying assumptions of hypothesis tests explicit;
  4. It is advantageous to approach effect estimation from a model as differences in predicted values;
  5. For comparison of responses or outcomes among groups, multivariable models are important—even for randomized designs;
  6. It is important to distinguish prediction and classification—and predictions should be separate from decisions so as not to distort and compromise either the prediction or the decision (see also Classification vs. Prediction and Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules);
  7. Development of the most accurate and reliable predictive model will enhance any/all of an analysts’ research interests;
  8. There are many ways and reasons that developing a model which will yield accurate and reliable predictions can go awry, but a principle-based approach guiding empirical model development can be felicitous;
  9. A principal-principle is, for an efficient and valid analysis, thoughtful selection of a response variable that is faithful to the nature of—and information in—the problem is crucial (see BBR notes 3.4.1. Proper Response Variables);
  10. And, finally, ‘friends don’t let friends’ categorize continuous response variables! :wink:

Written by @Drew_Levy

1 Like

Q&A from the May 2021 RMS course

  1. ? (dgl) for the class: Anything important about Chapter 1 missing from Comment-1? Anything mis-stated or ill-expressed?
  2. With FH’s recent emphasis on Bayesian perspectives, how would the precepts in Chapter 1 from RMS (2015) be amended or revised? What, if anything, would you now put differently? FH: Great question. I can think of two changes: (1) When Bayes is at its best, hypothesis testing plays no role IMHO, so I would de-emphasize hypothesis testing and add decision making/playing the odds. (2) The Bayesian approach has a lot to say about model choice, i.e., in some cases it avoids the need to choose a model by having traditional model components (and non-shrinkage priors for them) supplemented by extensions of the model, with priors that require these extensions to be convincing, i.e., shrinkage priors that wear off as n gets large. A simple example of this is the Bayesian t-test that favors normality with equal variances but has priors for the log variance ratio and the amount of non-normality.
  3. JM: I noted in the pre-course material that “Subgroup analysis is virtually worthless for learning about treatment effects” Why is this true? Nice post from Harrell regarding this topic Statistical Errors in the Medical Literature | Statistical Thinking FH: Some of the issues are imprecision from the smaller sample size, lack of covariate adjustment, and failure to demonstrate interaction.
  4. In line with question 3 I would be interested in whether predicting treatment effects using it as a dependent variable in a regression model makes sense? What would be a good approach to predict treatment effects? FH: Can only estimate a patient’s treatment effect in a cross-over study.
  5. Comment: I fear that the rumor of the death of ML is greatly exaggerated, having just participated in the statistics judging for the ASA Special Award in Statistics at the International Science and Engineering Fair 2021. These are high school students, and over half of the projects used some form of ML, mostly without careful thought. And, sadly, several of the statistics judges recommended many of the worst for the ASA Award for “best use of statistics”. Sigh.
  6. Do you have more information or can you talk about the issue of omitting time in the Probability of Ever having Cancer in 5 years model? FH: The idea is to model time to event as a continuous variable, with proper handling of censoring.
  7. Frank, there is a statement that survival model are essentially binary counting process, is that a true statement? Can we dichotomize the events at certain time point? Say if you have a 10-year survival, you can estimate the model by 1, 2, 4…. Year binary models? FH: Yes there is a counting process formulation but this doesn’t amount to using binary Y. The binary models would each be ineffective but you could combine them in some way to get over some of that problem were there to be no censoring.
  8. Can you explain some real world scenarios to choose between Cox model and accelerated faillure model? FH: We’ll get to that in a case study.
  9. Frank’s objections to stepwise regression are well-known. But, in ecological journals now, investigators are fitting a suite of regression models—even non-hierarchical models—presenting the table of AICs, and selecting for discussion & interpretation the model with the lowest AIC (or lowest group of regression models within 2 AIC units). Isn’t this as bad as stepwise or other automatic selection routines? Yes that’s just stepwise in disguise, but if you only compare 2-3 AICs things work pretty much OK.
  10. 1.3. We discussed how ROC analysis is misleading. In the context of survival analysis, we have the option to report Harrell’s C-statistic or IAUC (Integrated Area Under the Curve). Could you please comment on whether reporting IAUC is appropriate? FH: I don’t have enough experience with IAUC. The ordinary c-index is pretty simple to interpret so I stick to it.
  11. You discussed in this section that in a RCT adjusting by baseline covariates improves efficiency (and would lead to lower sample sizes needed to detect a given effect). How would you factor that adjustment in sample size calculation (since the calculators available that I know do not allow for that)? FH: For linear models it’s not hard. For other models people tend to use simulation. But need to have pilot data for that.