I was planning a talk on p values and wanted to come up with an analogy for the historical unintended negative consequences of targeting p < .05. I had not heard the parable of the Cobra Effect and thought it might be appropriate. The British Government in order to reduce the number of Cobra Snakes in Delhi offered a bounty for each dead cobra. The enterprising citizens responded by breeding Cobras , killing the offspring and collecting the bounty. Upon hearing this the British government mandated that all Cobras be released. The cobra population in Delhi was then much larger than it was pre-bounty. P values were intended as a method for error control and severe testing. The bounty of grants, publications, research money etc. led to their misuse. I was wondering if this is an appropriate analogy.
I don’t think that that’s an analogy for the problems with p values. The problems are not hard to explain, but it takes more attention than an analogy that is more likely to confuse than to help, IMO. My own attempts to explain the problems are, for example, here, and here, and videos linked here.
There are two distinct angles to the problem, both important:
Misinterpretation of the evidentiary value of obtaining p<0.05 in a study, with regard to scientific inference and subsequent decision-making. The genesis of this misinterpretation probably lies with journals, which have, historically, used p<0.05 as a filter for deciding which studies are worthy of publication; and
Incentivization of bad statistical practices as a direct result of that misinterpretation (e.g., multiple testing, HARK’ing…). If researchers want to get their work published, they feel compelled to bend over backward to obtain p<0.05 (the “Cobra Effect”).
The intrinsically “innocent” p value is caught in the middle of these two wicked problems.
Patient advocate here. Given its faults, why is p value still used to summarize the findings in clinical science? Is it feasible to change the standard --you know, so the primary stakeholders (patients) can understand the meaning? The examples offered by AI remind of consent language used to describe the possible risks of the study drug to participant:
Your take?
Plain Language Replacements
Instead of stating “The difference was significant (p < 0.05),” use these descriptive concepts: [1]
Describe the actual benefit (Effect Size):
Explain the literal difference between the two groups using concrete numbers.
Instead of p-value:
“Patients taking Drug A had 30% fewer asthma attacks than those taking Drug B.” [1, 2]
Describe the range of possibility (Confidence Intervals): Translate a confidence interval into a clear statement of mathematical predictability.
Instead of p-value:
“We are highly confident that the true reduction in attacks is between 20% and 40%.” [1]
Use the “Language of Evidence”:
Shift from black-and-white significance to a scale of certainty.
Instead of p-value:
“The data provide strong evidence that the new treatment lowers blood pressure.” [1, 2]
Number Needed to Treat (NNT): Explain how many people need to use the treatment for one person to benefit.
Instead of p-value:
“For every 10 patients who took this medication, 1 person completely avoided a hospital visit.” [1]
Appreciate your response! Agree … also worth noting that consent language for possible risks also incorporates the uncertainty you speak to here (Less Likely Side Effects --Happening in 1% to 20% of people) .
I believe the current crisis in public health (RFK etc.) speaks to the need for statistical language to foster public understanding as much as possible.
Yes very much. It also speaks to the need for us to clean up our own house. The proportion of biomedical research that is poor quality is quite embarrassing. Increasing rigor and enforcing good statistical practice should be our priorities, in conjunction with finding better ways to describe statistical results. Getting away form things like having to envision repeated studies, and NNT, will help. Consumers want to know exactly what we’ve learned from the study at hand, not what we would be likely to observe in future repetitions of the study.
Could you say something like: If you treat100 patients like those in this trial with B rather than A, and, you prevent a certain number of deaths. Given the trial data, there is an 80% chance that number is greater than 2.
That’s not really what a probability is, and has prompted patients to say, when told that only 1 of 5 such patients survive, “I know I’m that one”. As if patients are competing with each other. We need to educate everyone about probabilities (which have a range of 0-1 not 0%-100%), e.g., what it means for your risk of a bad outcome with treatment to be 0.1 and with treatment to be 0.09. Then we need to accomplish the harder feat: showing uncertainties in the 0.1 and 0.09.
This fallacy of ‘competition’ helps me appreciate your preference for p \in [0,1] over natural number ratios, which was previously obscure to me Frank.