# Is Any Method So Obsolete It Shouldn't Be Used?

**URL:** <https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820>\
**Category:** models\
**Created:** [September 17, 2021, 2:27am UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820 "2021-09-17T02:27:50Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![EpiMD5](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/e/838e76/32.png) [@EpiMD5](https://discourse.datamethods.org/u/EpiMD5)\
**Post date:** [September 17, 2021, 2:27am UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/1 "2021-09-17T02:27:50Z")

</div>

Are there any regression modeling strategies that should be deemed (authoritatively) to be now (2021) “obsolete?” Obsolete means—“there is always a better alternative/do not do this because the strategy cannot reliably yield a valid solution.” Obsolescence would imply "do not teach except in historical context.”

Nominating: Stepwise regression.

How would a regression modeling strategy be deemed “obsolete” given this definition? Can/should stepwise regression be deemed “obsolete” given this definition?

Recognizing that there have been long-standing attempts to “stomp out” stepwise regression that have failed, is there anything else to be done to “stomp out” stepwise regression?

---

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [September 17, 2021, 6:50am UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/2 "2021-09-17T06:50:46Z")

</div>

With out a doubt, fixed level testing at the magical levels of 0.1, 0.05, 0.025, or 0.01 should be relegated to the historical dustbin.

That doesn’t imply any “ban” on testing, however.

It is responsible for the litany of p-value fallacies we see in the scientific journals. It is also responsible for a vast literature on the so-called “Jeffreys-Lindley” paradox. There is no paradox if a declines at a suitable rate as 1-\beta increases.

> **[Almost sure hypothesis testing and a resolution of the Jeffreys-Lindley paradox](https://projecteuclid.org/journals/electronic-journal-of-statistics/volume-10/issue-1/Almost-sure-hypothesis-testing-and-a-resolution-of-the-Jeffreys/10.1214/16-EJS1146.full)**
>
> A new method of hypothesis testing is proposed ensuring that as the sample size grows, the probability of a type I error will become arbitrarily small by allowing the significance level to decrease with the number of observations in the study....

I’m exploring the relationship of this adaptive \alpha to the Bayesian rejection ratio described in this paper I’ve posted on a number of threads.

> **[Rejection odds and rejection ratios: A proposal for statistical practice in...](https://www.sciencedirect.com/science/article/pii/S002224961600002X)**
>
> Much of science is (rightly or wrongly) driven by hypothesis testing. Even in situations where the hypothesis testing paradigm is correct, the common …

Making explicit the important distinction between Fisher’s p value and Neyman’s \alpha would go a long way to improving both the reporting of results as well as the interpretation.

> **[Confusion Over Measures of Evidence (p's) Versus Errors (α's) in Classical...](https://www.tandfonline.com/doi/abs/10.1198/0003130031856)**
>
> (2003). Confusion Over Measures of Evidence (p's) Versus Errors (α's) in Classical Statistical Testing. The American Statistician: Vol. 57, No. 3, pp. 171-178.

**Related to regression:** I’d hope that clarifying the important distinction between \alpha and p would lead to understanding that leaving out terms in a regression model based on a decision rule interpretation of p-values is irrational from an information theoretic and Bayesian POV.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [September 17, 2021, 12:34pm UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/3 "2021-09-17T12:34:19Z")

</div>

Other obsolete methods:

- repeated measures ANOVA
- Mood median test
- sign test
- any method that converts a continuous variable to a categorical variable

---

<div class="post-metadata">

**Author:** ![s\_doi](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/s/ea666f/32.png) [@s\_doi](https://discourse.datamethods.org/u/s_doi)\
**Post date:** [September 17, 2021, 6:24pm UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/4 "2021-09-17T18:24:39Z")

</div>

I am guilty of the last one

Another two obsolete methods are:

- log-binomial regression
- modified Poisson regression with binary outcomes

---

<div class="post-metadata">

**Author:** ![EpiMD5](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/e/838e76/32.png) [@EpiMD5](https://discourse.datamethods.org/u/EpiMD5)\
**Post date:** [September 18, 2021, 4:30am UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/5 "2021-09-18T04:30:25Z")

</div>

Considering the Mood Median Tests.

So, why do various software packages (e.g., R, SAS, STATA, SPSS and even excel–if this is statistical software package) make this test an option. In my opinion, the option “invites” users to use this obsolete test. If this test is “obsolete”, why would R, SAS, STATA, SPSS (and excel) make it easy/possible/acceptable to conduct an analysis using the software.

This post was motivated by a review of a post from CROSS-VALIDATED where what seemed to be a student tried to do an analysis of data using stepwise regression and ended up with a mess. No surprise.

I am happy to know that the Mood median test is something I never need to understand or use. I never even heard of it. But, even in excel, there it is an option to conduct this test as a comparison of medians.

Deeming obsolescent the sign test would likely have many advocates and few detractors.

Deeming obsolescent repeated measures ANOVA should be a special priority for some fields- not limited to psychology, sociology, medicine, economics.

But how to “stomp out” things that are obsolete–regression approaches, tests–others.

Stepwise regression is being TAUGHT in introductory statistics courses.  
The sign test is being TAUGHT in introductory statistics courses.  
These methods have software that “invites” people to use the methods.

Stepwise regression, the Sign Test, the Mood Median Test (and others) could be taught as “these are methods that were developed in the past and advances in knowledge and technology have made them obsolete.”

---

<div class="post-metadata">

**Author:** ![davidcnorrismd](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/davidcnorrismd/32/3502_2.png) [@davidcnorrismd](https://discourse.datamethods.org/u/davidcnorrismd)\
**Post date:** [September 18, 2021, 10:38am UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/6 "2021-09-18T10:38:15Z")

</div>

> [@EpiMD5](#):
>
> If this test is “obsolete”, why would R, SAS, STATA, SPSS (and excel) make it easy/possible/acceptable to conduct an analysis using the software.

I join you in ‘screaming into the void’ here. A crucial function that excellent statistical software serves is to provide expert guidance to the user. (If only Stata refused to compute post hoc power, at least one calamitous episode in the history of statistics could have been avoided!) At the very least, warnings should be issued when bad things are attempted. I once engaged with Stata Journal over Stata’s cheerfully computing cluster-robust variance estimators with as few as 2 clusters, in the context of a case where this happened, resulting in an erratum.

Even if ‘canceling’ bad methods from software is not feasible, I wonder whether one might overlay an advisory layer. It should be possible to write an R package that, when attached, effectively looks over your shoulder while you work, and offers the missing warnings. Presumably, this could be implemented simply by masking every problematic function, and imposing additional quality checks before passing through to the original function or refusing to proceed.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [September 18, 2021, 12:16pm UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/7 "2021-09-18T12:16:07Z")

</div>

Great comments from both of you. I wish that software systems would put warnings every time an obsolete or highly problematic method is used. Or we need to teach everyone to do simulations or to dissect their results, which would expose the problems. Here was one of my attempts to expose the futility of feature/variable selection: [https://discourse.datamethods.org/t/challenges-to-variable-selection](https://discourse.datamethods.org/t/challenges-to-variable-selection)

---

<div class="post-metadata">

**Author:** ![EpiMD5](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/e/838e76/32.png) [@EpiMD5](https://discourse.datamethods.org/u/EpiMD5)\
**Post date:** [September 18, 2021, 2:51pm UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/8 "2021-09-18T14:51:48Z")

</div>

Yes, advisories and warnings and quality checks in the software seem first steps in the right direction.

But I can see from reading the many posts and writings of @f2harrell (and others) about (e.g.) stepwise regression that there is a feeling of “screaming in the wind.” Can’t hurt to talk about it.

But a clear statement that " can be done but shouldn’t be done (or doesn’t need to be done) because is obsolete" wouldn’t hurt. I have a washboard but I don’t need to use it because I have a washing machine, which does a better job. Washboards are obsolete.

---

<div class="post-metadata">

**Author:** ![EpiMD5](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/e/838e76/32.png) [@EpiMD5](https://discourse.datamethods.org/u/EpiMD5)\
**Post date:** [September 18, 2021, 2:52pm UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/9 "2021-09-18T14:52:35Z")

</div>

I hope someone will take up a response to your post.

---

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [September 18, 2021, 3:06pm UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/10 "2021-09-18T15:06:24Z")

</div>

> [@f2harrell](#):
>
> Or we need to teach everyone to do simulations or to dissect their results, which would expose the problems.

I think a great class could be developed by having students do simulations on the large number of flawed methods and proposals that litter the journals and textbooks. This could be done in an entirely [inquiry-based](https://en.wikipedia.org/wiki/Inquiry-based_learning) fashion, after the essentials of programming the simlations were taught. Has anyone tried it?

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [September 18, 2021, 3:11pm UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/11 "2021-09-18T15:11:31Z")

</div>

I’d surely like to know. I think that simulations are great teaching tools, and having a compendium of R Markdown notebooks for helping to jump start them would be a good idea. This could include not only obsolete methods but non-robust methods, e.g., a simple simulation to show that the central limit theorem doesn’t apply.

---

<div class="post-metadata">

**Author:** ![dpananos](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/dpananos/32/1577_2.png) [@dpananos](https://discourse.datamethods.org/u/dpananos)\
**Post date:** [September 22, 2021, 1:35am UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/12 "2021-09-22T01:35:50Z")

</div>

> - modified Poisson regression with binary outcomes

I assume you are referring to the methods described [here](https://academic.oup.com/aje/article/159/7/702/71883). Is this really obsolete? Why? What is to be used instead?

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [September 22, 2021, 2:14am UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/13 "2021-09-22T02:14:51Z")

</div>

The idea of basing models on relative risks is not sensible to me as such models would require more interactions to be included to make up for model restrictions (restrictions that log odds don’t have).

---

<div class="post-metadata">

**Author:** ![s\_doi](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/s/ea666f/32.png) [@s\_doi](https://discourse.datamethods.org/u/s_doi)\
**Post date:** [September 22, 2021, 3:56am UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/15 "2021-09-22T03:56:34Z")

</div>

Yes, that method. As mentioned by Frank above, the problem is that this generates a relative risk which has a different numerical value with changes in baseline risk independent of outcome discrimination. Thus the coefficients are just a rolling number. In addition there will be unnecessary product terms required to keep probabilities bounded between 0,1

Best to use logistic regression and interpret the odds as risks as needed. There is a thread discussing this recently if you scroll down the list. Also see this [paper](https://www.jclinepi.com/article/S0895-4356(21)00241-9/fulltext)

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [September 22, 2021, 11:30am UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/16 "2021-09-22T11:30:26Z")

</div>

Don’t say “interpret the odds as risks” but rather use the logistic regression model to get risks and use those covariate-specific risks to get covariate-specific risk ratios.

---

<div class="post-metadata">

**Author:** ![s\_doi](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/s/ea666f/32.png) [@s\_doi](https://discourse.datamethods.org/u/s_doi)\
**Post date:** [September 22, 2021, 5:14pm UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/17 "2021-09-22T17:14:11Z")

</div>

Agree - I was just thinking in terms of posterior odds from the logistic regression model

---

<div class="post-metadata">

**Author:** ![cvmsteve](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/cvmsteve/32/2024_2.png) [@cvmsteve](https://discourse.datamethods.org/u/cvmsteve)\
**Post date:** [September 24, 2021, 11:09am UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/18 "2021-09-24T11:09:17Z")

</div>

Frank, my turn to ask. Why repeated measures ANOVA? I could understand OLS being obsolete for analyzing repeated measures, but safety testing in preclinical studies or target animal studies are two fields where the MLE/REML method is currently the standard due to the split-plot in time experimental design of the studies, and Bayesian methods with different covariance structures are being explored. Thanks!

SteveDenham

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [September 24, 2021, 1:03pm UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/19 "2021-09-24T13:03:14Z")

</div>

The traditional repeated measures ANOVA that assumes multivariate normality with a very restrictive correlation patter is too restrictive.

---

<div class="post-metadata">

**Author:** ![cvmsteve](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/cvmsteve/32/2024_2.png) [@cvmsteve](https://discourse.datamethods.org/u/cvmsteve)\
**Post date:** [September 24, 2021, 1:23pm UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/20 "2021-09-24T13:23:09Z")

</div>

No kidding. My Linear Models professor in 1977 pointed that out. He also pointed out that no null hypothesis is true, that p values are some sort of random variable with an indeterminate distribution that is dependent on how much faith you can put into the prior supporting it and that someday in the future, we would ALL be using Bayesian and simulation methods. Hats off to Dr. Norm Matloff, at UC Davis then.

---

<div class="post-metadata">

**Author:** ![stanlazic](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/stanlazic/32/2453_2.png) [@stanlazic](https://discourse.datamethods.org/u/stanlazic)\
**Post date:** [October 5, 2021, 4:53pm UTC](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820/21 "2021-10-05T16:53:38Z")

</div>

Newman-Keuls post-hoc test… [doesn’t do what it says on the tin](https://www.graphpad.com/support/faq/why-we-recommend-you-do-not-use-the-newman-keuls-multiple-comparison-test/)

[Next page](https://discourse.datamethods.org/t/is-any-method-so-obsolete-it-shouldnt-be-used/4820.md?page=2)