How does Mayo apply severity to Perrin's Experiments?
Mayo divides Perrin's experimental inquiry into two steps: Step 1 consists of checking, for each experiment E, whether the results of the experiment actually performed follow the given statistical distribution; Step 2 involves using estimates of the coefficient of diffusion which is crucial for obtaining Avogadro number. Here let us see what Mayo says on Step 1.
According to Mayo, Perrin was to decide between the following two alternative hypotheses (223; j is the null hypothesis)
j: The data from E approximates a random sample from the hypothesized Normal process M.
j': The sample displacements of data from E are characteristic of systematic (nonchance) effects.
And after examining Perrin's experiments, Mayo concludes as follows:
we can argue that were j', and not j, the case it is extremely improbable that none or even very few of the experiments E1, E2, ..., En would have indicated this. It is very probable that a few would have shown differences statistically significant from what is expected under j. That is to say, the test was severe for j---where "the test" includes the results from several individual experiments. The pattern of arguing from error is clear. The experiments conducted by Perrin and his researchers had a very high probability of detecting a statistically significant difference from j', were there dependencies in the motion, yet such differences were not detected. (231)
So we have to think that this is one of the typical ways she wishes to apply her severity criterion (SC). However, the consideration of severity for the Perrin case is given only informally, in obscure terms, and it does not coform to the manner she proposed; for how can we obtain the probability of the "test results" on hypothesis j', which is not defined at all? No doubt we can obtain such probabilities on the null hypothesis j, but, recall, j' is simply the negation of j (a disjunction of indefinite number of alternatives, a "catchall"), and we do not know which particular hypothesis (among the disjuncts) is the closest rival to j. Notice that the case of Step 1 is not a problem of estimating the value of a parameter; we cannot definitely know the range of possible hypotheses. (See Why this definition of severity?)
Since, according to the preceding formulation of j, randomness and Normal process are the two key words, it seems (according to Mayo's Error Statistics) we have got to consider the four possible combinations (that is, [1] a random sample from the Normal process, [2] random sample from the non-Normal process, [3] a non-random sample from the Normal process, and [4] a non-random sample from the non-Normal process), and to give a probability (even a rough estimation) for each case; otherwise we cannot even define the probability of an outcome on the negation of j, i.e. j'.
Notice that I am not arguing against the validity of Perrin's inference; I am arguing only against Mayo's way of reconstructing the force of Perrin's inference. If she tries to supplement her argument by saying, e.g., that our past experience clearly indicates that the three possibilities for j' are extremely unlikely, many people may agree; but this amounts to introducing the consideration of prior probabilities and hence posterior probabilities (in terms of the frequentist notion of probability, I may grant for the sake of argument): the recourse Mayo consistently refuses to use.
It may well be the case that Perrin's experiments (taken together) reject (with an appropriate significance level) all three possibilities except for [1]. But again, this cannot be covered by Mayo's definition of severity, because such rejection must be, as far as I can tell from Perrin's detailed description in his Atoms, on the basis of (well-defined) probabilities calculated on the null hypothesis, not on the basis of (ill-defined) "probabilities" on the negation of the null-hypothesis; thus, it seems, Mayo's notion of severity is inapplicable to Perrin's case. But I am not saying that Mayo's analysis has no virtue (see Argument from Coincidence; How to reconstruct Perrin's argument?).
Last modified Jan. 27, 2003. (c) Soshichi Uchii