How to get new statistical methods to be used in practice

data analysis
1

Tim Morris asked about “when a new statistical method is fit-for-use and how we get them used?”

“Fit for use” is a complex topic worth much more discussion than I’m able to provide, but to start we should consider at least these attributes:

  • the analytic method should use all the information in the data it uses
  • the analytical algorithm provides a reproducible numeric solution to within simulation error
  • the method is shown to work either theoretically or by extensive realistic simulation
  • by simulation the method is demonstrated to recover the hidden truth in the data generating mechanism for sample sizes that are typical of those available when the method will be used
  • the method should have Bayesian or frequentist power or precision that is better than currently easily available methods

In terms of how to get a newer statistical method to be used in practice, here are some considerations:

  • make sure that the method addresses a problem that is somewhat frequently encountered
  • emphasize development of methods that have fewer hidden assumptions; prefer full model specification over weighting or estimating equations
  • competently implement the analytical algorithm in a free highly available software platform such as R or python
  • provide good documentation
  • provide a template for the analytical workflow using markdown with ample comments
  • provide a template for the user to easily run alternative simulation studies using markdown with ample comments
  • use social media to publicize the method and make it easy to download free software

Some of the biggest holdups to getting a new method to be used even within the scientific team of which the statistician is a part are

  • conservatism of the non-statisticians in the team (“but we’ve always used method x in our past publications …”)
  • a belief among your co-authors that the subject matter journal will not accept a new method

Here the remedy is

  • persistence of the statistician
  • inclusion of nice graphics and appropriate references in the manuscript
  • the statistician remembering that her job is never to give people what they want to rather to give people what they need
  • never accepting as a reason “We’ve always done it this way”

Note: I have never had a situation where ultimately a journal prevailed in wanting only traditional analyses and graphics to be used in a manuscript.

11 Likes
2

I hope to see lots more discussion here, since the problem seems so rich. As far as I can tell, the term fit for use is colloquial, but it brings to mind general legal notions like warranty of fitness for a particular purpose, as well as domain-specific notions such as airworthiness and even FDA’s (stalled?) Fit-for-Purpose initiative.

A bit closer to this forum’s home territory of statistics, though, fit for use has the ring (to my ears) of ‘validated’, a word usually invoked as if it authoritatively stamps a questionnaire, construct or method as definitively verified and beyond further criticism. (Against that view, contrast the fallabilism in Cronbach & Meehl’s 1955 admonition [1] that “The construct is at best adopted, never demonstrated to be ‘correct’.”)

  1. Cronbach LJ, Meehl PE. Construct validity in psychological tests. Psychological Bulletin. 1955;52(4):281-302. doi:10.1037/h0040957 [PDF]
2 Likes