This is a wonderful question. Unfortunately as the paper being discussed exemplifies, there is analysis/statistician variation in addition to sampling and measurement variation. This leads to a few random observations:
- The best we usually hope for in empirical work (outside of math and physics were equations can be argued from first principles) is to choose a method that is competitive, i.e., that is not easily beat by a method that someone else might choose.
- The method should be chosen on the basis of statistical principles, simulation performance, and suitability for the task/data.
- The method should be general enough to encompass complexities that other statisticians may have chosen a different method to handle.
The frequentist/Bayesian battle has a lot to do with the last point. Frequentist modeling invites a kind of dichotomization, e.g., do we include or exclude interaction terms? Do we allow the conditional variance of Y | X to vary over groups of observations? Do we entertain non-normality of Y | X? These issues create model uncertainty, require huge sample sizes to answer definitively, and result in non-preservation of confidence coverage and type I error. The Bayesian approach is the only way to take analysis uncertainties into account, by having parameters for everything you think may be a problem but don’t know. For example, in a 2-sample t-test you can have a parameter for the ratio of variances of the two groups, and a prior for that that favors 1.0 but allows for ratios between 1/4 and 4.
Explicitly allowing for model uncertainty preserves the operating characteristics of the method and results in just the right amount of uncertainty/conservatism in the final analysis.
Besides choosing between frequentist, Bayesian, and likelihood schools of modeling/inference, statisticians tend to be overly divided by how nonparametric they are. The majority of statisticians use linear models for continuous Y. I don’t, since I believe that semiparametric ordinal response models (e.g., proportional odds or proportional hazard model) are more robust and powerful, and most importantly, free the analyst from having to make a decision about how to transform Y (and how to penalize for any data-based choice of transformation). The proportional odds model has the WIlcoxon and Kruskal-Wallis tests as special cases, and robustly handles bizarre distributions of Y | X including clumping at zero and bimodality.
Then there is the issue of prior beliefs in simplicity, e.g., additivity and linearity vs. routinely allowing effects to be smooth and nonlinear as I do. A side note: I can often predict that an analysis will be shown to have problems by seeing the choice of software the authors used. Papers using SPSS or SAS tend remarkably to show linear effects more often than papers using R or Stata. That’s because SPSS and SAS make it harder to handle nonlinearities, and statistician laziness sets in.