Mayo's error statistical approach to the epistemology of experiment, in
her Error and the Growth of Experimental Knowledge (1996), crucially
depends on her definition of severity, but the reader may be rather puzzled
by the manner she introduces this definition. Her definition of severity
appears rather abruptly (in chapter 6), and with no explicit reference to
her previous arguments or preparations in the previous five chapters. What
she says, immediately before she introduces the definition of severity,
is essentially this much:
The cornerstone of an experiment is to do something to make the
data say something beyond what they would say if one passively came across
them. The goal of this active intervention is to ensure that, with high
probability, erroneous attributions of experimental results are avoided.
The error of concern in passing H is that one will do so while H is not
true. Passing a severe test, in the sense I have been advocating, counts
for hypothesis h because it corresponds to having good reasons for ruling
out specific versions and degrees of this mistake. (Mayo 1996,178)
Then comes her first statement of a severe test; that is,
(S): a passing result is a severe test of hypothesis H just to the
extent that it is very improbable for such a passing result to occur,
were H false. (Mayo 1996, 178)
But our immediate response is: how is this notion of severity related to
statistical tests explained so far? We know what "significance level" is,
or what "experimental distribution" is (see pp. 158-9), but they are essentially
defined in terms of the probability of an outcome, given hypothesis H is
true, not given H is false! And so far, Mayo has not given any hint
as regards how we should compute probabilitites, given hypothesis H is false
(her discussin of type II error--accepting the null hypothesis when
it is in fact false--comes long after, on p. 367) . Thus a strong
uneasiness is produced in the reader's mind; Mayo does not begin to discuss
this until page 195, and the discussion there is frustrating, as we will
see shortly.
In order to make her arguments more intelligible (and, consequently, more
susceptible to criticisms), Mayo should have made use of examples in her
repertoire immediately. Later (192 ff.), she indeed uses the Binomial
Experiment (a lady judging whether the tea or the milk was first added to
the cup, and she claims she can do better than chance) for illustration,
but the reader is told a somewhat different story from what he or
she has expected. In this example we have two hypotheses, H0
and H' (p is the probability of success):
H0: the lady is guessing, p = 0.5
H': the lady does better than guessing, p > 0.5
And given the null hypothesis H0, we could obtain
experimental distribution for 100 trials. All right, this time, let us take
H' as our test (null) hypothesis; then Mayo's definition of severity can
be easily illustrated. Let f signify the observed relative frequency of
"success" (i.e. the lady's judgment is correct) in 100 trials; then we already
know that
P(f≧0.6 | H0) = 0.03.
Since in this context ￢H' = H0, we have an indication
of the severity of test for our hypothesis H', according to Mayo's definition.
Suppose we obtained the result that f≧0.6, which may be abbreviated as e.
Then, the test of H' by means of this result e is as severe as 97
percent (in terms of percentage).
However, it must be noticed that if we take H0 as
our test hypothesis, the situation is not as easy as this, since we have
to be able to calculate the probability of some result conditional on the
falsity of H0. For instance,
P(f≧0.6 | ￢H0) = P(f≧0.6 | H') .
Since H' does not specify the value of p, how can we obtain this probability?
Notice that H' in this case becomes quite similar to the "Bayesian catchall"
as Mayo puts it (a disjunction of an indefinite number of hypotheses). Literally,
the negation of H0 is nothing but a disjunction of
all hypotheses assigning any value (between 0 and 1; which amounts
to saying, informally, that the lady does better or worse than mere guessing)
except for 0.5; so how should Mayo obtain the probability of e on
such a disjunction? I don't see any difference between the Baysian difficulty
and Mayo's difficulty in this regard. Suppose (although I am sure Mayo does
not like this supposition), imitating Laplace (the principle of indifference),
we assume that any value for p is equally possible; then it should
be the case that
P(f≧0.6 | ￢H0) = P(f≧0.6 | H') = P(f≧0.6 | H0)
= 0.03.
This means that the test by e is as severe for H0
as for ￢H0. This seems to be quite disastrous for
Mayo's definition; for the same result e counts as a good evidence
for both H0 and ￢H0!
Notice that this suggests a stronger worry than Earman's worry, treated
and answered by Mayo in 6.3. Earman suspected whether we can obtain a low
probability in case test hypothesis is false; and he presented a case in
terms of higher-level alternatives. But my example suggests a far stronger
worry that both the null hypothesis and its rival hypothesis may give the
same probability to the same evidence, the two hypotheses
being low-level alternatives to each other! If she wants to stick to her
definition (S), she has to show that, on her account of error statistics,
this sort of counterexample never appears.
Beginners may have some difficulty for understanding the preceding result;
so let me give you a simpler version (a classical example, from the history
of probability theory), in terms of finite alternative hypotheses.
Let H0 be the same as before, and suppose there are
4 other alternatives (given the background information, say):
- H1: p=0.00
- H2: p=0.25
- H3: p=0.75
- H4: p=1.00
On our assumption, ￢H0 is equivalent to the disjunction
of these four. Then, given that each alternative hypothesis is equally probable
(according to the Laplacean principle of indifference), the probability
of any result e on H0 is the same as the probability
of e on ￢H0 (i.e., on the disjunction of the
four). It suffices to show this for
e = the Lady's judgment is correct.
Then, clearly,
- P(e | H1) = 0.00
- P(e | H2) = 0.25
- P(e | H3) = 0.75
- P(e | H4) = 1.00.
Since each hypothesis is equiprobable (the experimental situation where
this holds, even for the frequentist, will be shown soon), the probability
of e on the disjunction of the four is clearly the mean of these
predictions, i.e., P(e | ￢H0) = 0.50, which
is exactly the same as P(e | H0) . Given this,
it is easy to see that the same result holds for any evidence statement
e. And, the case where there are infinitely many hypotheses (as regards
the value of p), is not any different, in principle, from this simpler version.
Moreover, this example is not as unrealistic as it may seem at first sight.
For, what is crucial is that the alternative hypotheses are distributed
symmetrically around the null hypothesis; we can find many such cases,
when we wish to ascertain the correct value of a parameter.
Finally, it may be pointed out that Mayo's defence of the error statistical
approach agianst this sort of counterexample, in terms of piecemeal character
of experimental learning, does not help much. She says,
Within an experimental testing model, the falsity of a primary hypothesis
H takes on a specific meaning. If H states that a parameter is greater
than some value c, not-H states that it is less than c;
if H states that factor x is responsible for at least p
percent of an effect, not-H states that it is responsible for less than
p percent; if H states that an effect is caused by factor f,
for example, neutral currents, not-H may say that it is caused by some
other factor possibly operative in the experimental context ...; if H
states that the effect is systematic--of the sort brought about more often
than by chance--then not-H states that it is due to chance. How specific
the question is depends upon what is required to ensure a good chance
of leaning something of interest ... (190-1)
I agree with this; and most other (empiricist) Bayesians will join me in
this. But this does not help in the least for solving the difficulty posed
by my counterexample. Thus, as it turns out, all Mayo suggests later (on
p. 195) is that:
(1) the probability of an outcome conditional on a disjunction of alternatives
is not generally a legitimate quantity for a frequentist; and
(2) the severity criterion (SC) requires that the severity be high against
each of single alternatives (not a disjunction of such).
(1) amounts to saying that her definition of severity is not appropriate
for many of her canonical models of experimental inquiry, and (2)
is nothing but a substantial revision of her definition. If (2) is
what she really wants (and indeed this seems to be the case, judging from
her subsequent discussions, long after, in chapter 11, p. 397), she should
have changed the definition in the first place. It seems to me that,
in short, she has chosen a wrong way to state her crucial definition,
and this may easily give the impression that she simply wishes to evade
the whole question by this maneuvre (1) and (2). It looks quite strange
to demand that you should not ask the severity of the test for H0
(but OK for H'), when you are trying to test H0 against
H' (￢H0), in one of her canonical models.
Moreover, we can easily construct a test situation in which it is
legitimate for the frequentist to obtain P(e | ￢H0).
Suppose there are a small number of (say, 5) coins, biased or unbiased (as
our five hypotheses tell, respectively), and one is chosen at random;
and you are asked to determine by experiment which coin is in fact chosen.
In this case, it is legitimate, even for the frequentist, to ask
the probability P(e | ￢H0), and our counterexample
is fully alive. And our intuition is, if the observed frequency of head
is close to 0.5 and the number of trials is large enough, this is a good
evidence for H0 but not for ￢H0.
But, unfortunately, Mayo's definition of severity does not work for this
case.
Thus she has to abandon her official definition
of severity, (S) above, and reformulate it along the line of (2); but, notice
that in this case all
alternative hypotheses must be known in advance. This point becomes
relevant to my next point.
