Thanks - I think I know what you mean about “absolute calibration” if that corresponds to “under the stated test (target) hypothesis”. My response however is my version of an idea I see in Fisher’s defenses of P-values (his “significance levels”): There is an absolute calibration to a uniform distribution under all assumptions used to derive a valid P-value (the full test model, which includes the test hypothesis and all auxiliary assumptions). This calibration allows the P-value to be treated as a generic type of information measure for the divergence of the data from the test model in a particular direction (the direction specified by the test hypothesis). Following Box, this absolute calibration is for me an essential component of data analysis and complement to Bayesian modeling.
All this means is that the utility or informativeness of a P-value for the test hypothesis is directly proportional to the percentage of auxiliary assumptions which we are either willing to take as certain, or else have verified that P is insensitive (“robust”) to under the targeted test hypothesis. Not simple to be sure, but then again I see P-values as valuable, sophisticated tools that (like antibiotics) got vastly overdistributed and overused, generating strains of researchers and journal editors resistant to logic and common sense.
For details see “Valid P-Values Behave Exactly as They Should: Some Misleading Criticisms of P-Values and Their Resolution With S-Values”