Significance tests, p-values, and falsificationism

That is my question to the Popper proponents.

If I may ask: Which of Popper’s works have you read in order to possibly find an explanation for that?

It should not matter, but assume I am familiar with Logic of Scientific Discovery. A simple point of logic should not require extensive citations from numerous texts. Your absence of a response that addresses a direct question is curious.

+1 for Carnap’s original words. While I’m here, I might as well share something inflammatory …

1 Like

But no +1 for Popper’s original words? I’m offended…

1 Like

Would you put Platt’s (1964) Strong Inference piece into this category? (To me, it seems intensely practical in its orientation.)

It certainly is curious, but I must insist that it does matter. Because you’re not actually that familiar with LoSD.

You see, the thing you said was equivalent to what Carnap said (which you claimed was “fatal” for Popper) and that you would like to have explained why Popper never saw it—that’s a direct quote from LoSD.

Let me add, though, that while I do find it a little bit funny, I said all that in the hope that you (and maybe one or two others) would perhaps start to seriously consider the possibility that you know Popper and his ideas rather less well than you think.

Luke 15:32

I am not familiar with Platt, but I find much merit to the Hintikka game semantic model of logic, where quantifiers of logical statements are interpreted as different players in a game. If we extend propositional logic to probability logic, that can be viewed as a gambling game as in Shafers’ Testing by Betting.

A question is decidable if there exists a proof of A or not A. I’d prefer a constructive proof, but I’d accept a classical proof that uses reductio ad absurdum too.

I’m tired of replying, so I’ll try to be brief:

  • Popper can define his terms in however way he wants. What he cannot do is to equivocate his terms: use a word in his technical specialized sense at certain times and then fall back to the common definition when it’s convenient. Let me make a comparison. Suppose I say that Popper is wrong because universal generalizations can be confirmed. You ask, “how so?”. I answer, “well, to confirm a universal generalization is just to make a decision that it’s true based on pragmatic considerations x, y, z”. Sure, if that’s what I mean, then we can “confirm” theories. But then it’s misleading for me to say that Popper is wrong because of that.
  • Popper does not have a monopoly on fallibilism. Virtually everyone agrees that fallibilism is true. The problem is that, by rejecting induction, he thinks fallibilism can only mean “we can never justifiably think anything (even a basic statement) is probably true or probably reliabe”, which is madness. Suppose I see a black swan. Normally, we’d say that we have evidence against the hypothesis that all swans are white. Of course, I can be wrong because I’m fallible. Perhaps I’m hallucinating; perhaps I didn’t really see a swan, but some other animal; etc. But still, that’s some evidence against the hypothesis, I have good reasons to believe the generalization isn’t true. If others see the swan, the evidence is even greater. But Popper cannot even say this. The scientific community accepts the statement after certain tests and then we hope for the best, the basic statement is still conjectural and ungrounded.
1 Like

Which he doesn’t. As I explained above, including a crucially relevant quote. You could engage with that and spend your time and energy less tiresomely, you know? :slight_smile:

The problem is that you misunderstand all of that. His rejection of induction has nothing to with his rejection of justificationism, which founders on its own illogic. Is also has nothing to do with his rejection of the possibility of gaining certain knowledge. And you misunderstand that because you ignore his definitions—again.

The basic argument is pretty much unchanged for the 40 years since Mulkay and Gilbert wrote “Putting Philosophy to Work” in the early 80’s (Philosophy of the Social Sciences 11 (3):389-407 (1981)). Mulkay and Gilbert actually interviewed scientists and watched what they did in the context of Popper’s theories. A typical theme was that Popper’s work didn’t actually guide the science while it was being done, but might be discussed as a post hoc legitimization. Particular attention was paid to one scientist who claimed to be a Popperian and to use Popper in his daily scientific practice. But other scientists said that this scientist didn’t follow Popper at all, and when an experiment was published that falsified this scientist’s theories, he fought “tooth and nail”. As Mulkay and Gilbert put it, the scientist could “just as easily be seen as contravening Popper’s maxims as exemplifying them”.

This is the critique of Popper from the perspective of the sociology and the history of science: if you say “here is what science is and here is how you should do science”, it is somewhat of a problem if most scientists in contemporary practice and indeed, back through history, didn’t do much of that at all.

I with Frank here (except that History and Philosophy of Science was my major at Cambridge, so I am an interested observer who was trained in philosophy): falsification is an interesting idea in the abstract but isn’t much use in the actual doing of science (the same is true of reliabilism by the way, seeing as how David Papineau was mentioned on this thread). I run around calculating p-values for hypothesis tests all the time and it is hard to see how much of what I do fits in a falsificationist framework. Rejecting or not rejecting the null does have superficial similarities to falsifactionism, but they don’t really go much deeper. For instance, I just looked up my last paper published and, in brief, we were testing whether A predicted B. The p value was 0.3. Our conclusion was that A did probably predict B, but that any effect size was too small to worry about and that any patient who developed A should be reassured about B. I stand by that interpretation, but try linking that to Popper and you’ll just get dizzy.

In brief, I majored in philosophy at a world-renowned institution and I am now a practicing scientist. I’m pretty sure that fact that I’m good at the latter is due to the former. But it is indirect. Philosophy made me a better thinker, and a different type of thinker, and that has made me a better scientist. I’m not a better scientist because I looked up the recipe for better science on page 283 of some philosophy textbook.

6 Likes

Just so that I understand properly what you’re saying: In that example of “Spencer”, it is precisely that he is sticking to a theory as long as possible (in what you might call a “dogmatic attitude”) that runs counter to his professed following of Popperian methodology. Is that correct so far?

I’m not saying anything about Spencer. I never met them. What I’m saying is that when we watch what scientists actually do (in my case, what I do and what my colleagues do), it doesn’t match up at all with what Popper claimed about the scientific method. This is even true of scientists who stated that they followed Popper and put his methods into daily practice.

No, of course. I just meant, since you referenced Mulkay & Gilbert, that you would at least agree that “Spencer”, although professing to follow Popper’s ideas, instead showed a pattern of (dogmatically, one might say) sticking to his theory as long as possible, and thereby undermining his professed adherence to Popperian methodology—“it doesn’t match up”, as you say.

Go read the original paper for the description of Spencer’s behavior: you don’t need my characterization of it over and above that it did not appear to follow his professed beliefs.

I just wanted to double-check that my understanding was correct that you agreed with the paper’s point that “tooth and nail” Spencer’s actions ran counter to his professed Popperian beliefs. Just wanted to be sure I didn’t get that wrong.

Peter: I’m sorry to say that I find your goading tone rather inappropriate. You are clearly trying to get me to agree with some extremely narrow point as a debating technique. As such, you are missing the larger point. Back in the day, philosophers of science would listen to scientists in order to understand more about science. You’ve had three of the world’s top statisticians tell you that falsificationism and significance testing have only superficial similarities. You don’t seem to want to listen. As a result, I actually don’t want to try explaining any more.

2 Likes

A tone troll? Didn’t they teach you in Cambridge that that is the best way to embarrass yourself as a scientist? If only you had made the effort to find out why I kept asking, Andrew. That would have shown some critical thinking.

Because here’s something I find inappropriate: People who don’t know what they’re talking about and instead keep boasting about their degrees from “world-renowned universities”. People who cite an article that is not just all-round laughable but that has, not as “some extremely narrow point ” but as its centerpiece, someone who is reported to stick to his theories in an arguably dogmatic fashion, an attitude the authors portray as incompatible (or in your words: that “doesn’t match up”) with Popperian thought—when that is blatantly false.

You see, unfortunately, neither the authors nor you knew enough about Popper (and didn’t think to ask an actual expert) to realise that the “dogmatic attitude of sticking to a theory as long as possible” is a direct Popper quote—an attitude, he said, was “of considerable significance”. (“What is dialectic?”, in C&R) Why? Because “without it we could never find out what is in a theory—we should give the theory up before we had a real opportunity of finding out its strength”. So Mulkay/Gilbert’s interpretation that Spencer is “contravening Popper’s maxims” is simply ignorant.

I could now condescendingly tell you that back in the day, scientists would listen to philosophers of science to understand more about it, and that you have had an actual Popper expert tell (and show) you that what you think you know about him just isn’t so. But then you don’t seem to want to listen…

This sums up my attitude regarding your understanding of mathematical logic and how it applies to both empirical science and mathematics.

If mathematics is the queen of science, and we apply Poppers ideas to mathematics itself, the TLDR is that Popper has been falsified.

I posed a simple question above: why should one accept, as a matter of logic Poppers’ proposed asymmetry between \forall x and \exists x? Carnap asked this and other scholars as evidenced by the link I posted above, also found it odd this was never answered.

Popper was a deductivist who claimed that only logical derivations can be trusted. That is another philosophical question that is debatable. Quine convincingly refuted it in his Two Dogmas of Empiricism.

As a simple point of classical logic, negation of one quantifier is a synonym of another. So as a point of logic, Popper is wrong.

In a particular context (ie. the free variables are bound to a model) perhaps Popper’s claim has some merit. The computer scientist Edgars Dijkstra correctly stated that testing a program and finding no errors does not demonstrate an absence of them.

A charitable interpretation of Popper leads me to see it as sufficiently close to Feynman’s notion of a scientist with integrity as to be interchangeable.

Blockquote
But there is one feature I notice that is generally missing in cargo cult science. That is the idea that we all hope you have learned in studying science in school–we never say explicitly what this is, but just hope that you catch on by all the examples of scientific investigation. It is interesting, therefore, to bring it out now and speak of it explicitly. It’s a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty–a kind of leaning over backwards. For example, if you’re doing an experiment, you should report everything that you think might make it invalid–not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked–to make sure the other fellow can tell they have been eliminated.

A strict interpretation, using the development of mathematical logic and analysis as an example, leads me to conclude Popper’s philosophy is false.

At the time “Testability and Meaning” was written, a number of fundamental results were discovered in mathematical logic, which I’m certain Carnap was aware of. Popper, by remaining silent, skirted the critical issue.

The incompleteness theorems of Godel get all of the philosophical attention, but equally interesting are the consistency results of Ackermann and Gentzen in the late 20’s thru early 40’s. They independently used a similar method. Ackermann proved the consistency of Primitive Recursive Arithmetic (PRA), while Gentzen used PRA to prove Peano Arithmetic was consistent.

PRA (with transfinite induction) is a formal language that expresses the natural numbers, but does not have quantifiers as Peano Arithmetic does. Yet, the cardinality of the languages are the same. If PRA is the meta-language, and Peano Arithmetic is the object language, we can express meta-statements about Peano in PRA. This notion of “consistency” is a meta-notion of fundamental importance.

Godel produced a slightly modified version of a Gentzen consistency proof that is known as the Dialectica interpretation.

W.W. Tait extended this notion and placed it in a game theory context. In logic this is known as the “no counter-example interpretation.”

If we take a (Popperian) skeptic of mathematics, this proof compels him to admit 1. he has the burden of proof to produce a counter-example, and 2. a counter-example does not exist (up to \epsilon_0, Cantor’s smallest infinity – the set of natural numbers).

Taking this game interpretation further, if we extend the propositional calculus from a binary truth value to a continuous one, we get a system that is consistent with probability theory (see Algebra of Probable Inference by R.T. Cox) with the semantic notion of betting on future observations – first described as an implication of Shannon’s communication theory by J.L Kelly in A new interpretation of the information rate (pdf) and extended by the more recent work of Glen Shafer and Vladimir Vovk on building up probability with game theory as the foundation.

The advantage of this approach is the ability to express Keynesian notions of imprecise probabilities, along with computational considerations, that get swept under the rug in applications that Sander justifiably finds important to mention.

2 Likes