Most reasonably hypothesised effects cannot be exactly zero?

The whole testing approach is not about estimation. Testing is a fall-back if estimation is unfeasable, either because it’s not quite clear or not relevant what the estimates would mean (at leat practically; one might think of statistics like R², Chi², F etc. that may be hard or impossible to interpret in any practically sensible or understandable way), or because the information in the data is insufficient to allow an estimation with a useful accuracy and precision.

If we fall back on testing, the questiion can not be whether or not an effect is precisely zero. This would just mean that we treta a test like an approch to estimate the effect. We often do so anyway, and this is a major source of confusion, imho. The test -specifically the significance Test [calculating a p-value according to RA Fisher] - is, instead, a procedure to judge the amount of information in the data at hand relative to a particular restriction in a particular (statistical) model. It was never, and should never be, the aim of the test to demonstrate an effect. The test is not about the effect (which is, by the way, neccesarily unknown to the test!) but about the data (more precisely about the “relevant” information in the data, w.r.t. to a restriction in a model). Thus, a “significant” test is not a demonstration of a “non-zero” effect. It is rather a demonstration that the data (the information provided by the data w.r.t. the restriction in the given model) is sufficient so that the effect is “visible with good confidence”. If the tested hypothesis is a “zero-difference” hypothesis, then a significant result means that we may place confidence in the interpretation of the sign of the effect.

1 Like