A physical model is not mandatory for the use of probability modeling and reasoning, though having one provides welcome additional insight.
Regarding “not all uncertainties can be represented as probabilities”, this can easily be shown. An example of a quantitative uncertainty which is non-stochastic is the bound on an approximation error for a Taylor series approximation of a function. In some applications, this bound is treated as an uncertainty, though it is purely deterministic. There are other examples in approximation theory, though I concede that some theorems in that field are probabilistic rather than deterministic, but certainly not all of them. (Taylor’s isn’t.)
More interesting are examples of uncertainties that cannot be quantified, which Kay and King’s book Radical Uncertainty (discussed above) gives many examples of. I cannot hope to summarize their arguments here, but a few quotes might give a flavor of where they are coming from.
The appeal of probability theory is understandable. But we suspect the reason that such mathematics was, as we shall see, not developed until the seventeenth century is that few real-world problems can properly be represented in this way. The most compelling extension of probabilistic reasoning is to situations where the possible outcomes are well defined, the underlying processes which give rise to them change little over time, and there is a wealth of historic information.
And
Resolvable uncertainty is uncertainty which can be removed by looking something up (I am uncertain which city is the capital of Pennsylvania) or which can be represented by a known probability distribution of outcomes (the spin of a roulette wheel). With radical uncertainty, however, there is no similar means of resolving the uncertainty – we simply do not know. Radical uncertainty has many dimensions: obscurity; ignorance; vagueness; ambiguity; ill-defined problems; and a lack of information that in some cases but not all we might hope to rectify at a future date. Those aspects of uncertainty are the stuff of everyday experience.
Radical uncertainty cannot be described in the probabilistic terms applicable to a game of chance. It is not just that we do not know what will happen. We often do not even know the kinds of things that might happen. When we describe radical uncertainty we are not talking about ‘long tails’ – imaginable and well-defined events whose probability can be estimated, such as a long losing streak at roulette. And we are not only talking about the ‘black swans’ identified by Nassim Nicholas Taleb – surprising events which no one could have anticipated until they happen, although these ‘black swans’ are examples of radical uncertainty. We are emphasizing the vast range of possibilities that lie in between the world of unlikely events which can nevertheless be described with the aid of probability distributions, and the world of the unimaginable. This is a world of uncertain futures and unpredictable consequences, about which there is necessary speculation and inevitable disagreement – disagreement which often will never be resolved. And it is that world which we mostly encounter.
I won’t provide a list of their examples, but I gave one of my own on another thread (discussion of the London Metal Exchange trading of Nickel in March of this year).
David A. Freedman’s rejoinder to the discussants of his classic shoe leather paper contains a similar assertion to Stark’s (Stark and Freedman were collaborators, so no suprise).
For thirty years, I have found Bayesian statistics to be a rich source of mathematical questions. However, I no longer see it as the preferred way to do applied statistics, because I find that uncertainty can rarely be quantified as probability.
Source: Freedman (1991): A rejoinder to Berk, Blalock, and Mason. Sociological Methodology, 21: 353-358.