Several of these examples seem to have more to do with reification errors related to (but perhaps not quite the same as?) those discussed by @Sander [1], rather than with mere misinterpretations of statistical technicalities. That is, the problem is not merely technical, but “cognitive” or (I would rather say) “epistemological.” For example, apart from the utter lack of scholarship on FDA’s part which seems to be @Stephen’s main complaint in the linked post, he also is troubled by how artifacts emerging from a statistical analysis are presumed ipso facto to amount to real things (e.g., responders) that exist in the real world. A lot of @llynn’s concerns fall into the same category: artificial thresholds become reified (e.g., SIRS, OSA) as if they were per se the subject of investigation, rather than mere phenomena of the ding an sich. Likewise, the whole surrogate endpoints business is at least complicated by our predilection toward reifying frequently-used terms such as ‘progression-free survival’.
It seems to me that such cognitive/epistemological errors are far more insidious than technical errors in p-value interpretation, and much more capable of insinuating themselves into CPGs.
- Greenland S. Invited Commentary: The Need for Cognitive Science in Methodology. Am J Epidemiol. Published online 2017:1-7. doi:10.1093/aje/kwx259