1 Bayesian versus Orthodox statistics: Which side are you on? RUNNING HEAD: Bayesian versus orthodox statistics Zoltan Dienes School of Psychology University of Sussex Brighton BN1 9QH UK [email protected] 2 Abstract Some common situations are presented where Bayesian and orthodox approaches to statistics come to different conclusions; you can see where your intuitions initially lie. The approaches are placed in the context of different notions of rationality and I accuse myself and others as having been irrational in the way we have been using statistics, i.e. as orthodox statistics. One notion of rationality is having sufficient justification for one’s beliefs. Assuming one can assign numerical continuous degrees of justification to beliefs, some simple minimal desiderata lead to the “likelihood principle” of inference. Hypothesis testing violates the likelihood principle, indicating that some of the deepest held intuitions we train ourselves to have as orthodox users of statistics are irrational on a key intuitive notion of rationality. I consider practical considerations so people can make a start at being Bayesian, if they so wish: If we want to, we really can change! Keywords: Statistical inference, Bayes, significance testing, evidence 3 Bayesian versus Orthodox statistics: Which side are you on? Introduction Psychology and other disciplines have benefited enormously from having a rigorous procedure for extracting inferences from data. The question this article raises is whether we could not be doing it better. Two main approaches are contrasted, orthodox statistics versus the Bayesian approach. Around the 1940s the heated debate between the two camps was momentarily won in terms of what users of statistics did: Users followed the approach systematised by Jerzy Neyman and Egon Pearson (at least this approach defined norms; in practice researchers followed the somewhat different advice of Ronald Fisher; see e.g. Gigerenzer, 2004). But it wasn’t that the intellectual debate was decisively won. It was more a practical matter: A matter of which approach had the mathematics been most well worked out for detailed application at the time; and which approach was conceptually easier for the researcher to apply. But now the practical problems have been largely solved; there is little to stop researchers being Bayesian in almost all circumstances. Thus the intellectual debate can be opened up again, and indeed it has (e.g. Hoijtink, Klugkist, & Boelen, 2008; Howard, Maxwell, & Fleming, 2000; Rouder et al , 2007; Taper & Lele, 2004). It is time for researchers to consider foundational issues in inference. And it is time to consider whether the fact it takes less thought to calculate p values is really an advantage, or whether it has led us astray in interpreting data (e.g. Harlow, Mulaik, & Steiger, 1997; Meehl, 1967; Royall, 1997; Ziliak & McCloskey, 2008), despite the benefits it has also provided. Indeed, I argue we would be most rational, under one 4 intuitively compelling notion of rationality, to be Bayesians. To see how which side your intuitions fall, at least initially, we next consider some common situations where the approaches come to different conclusions. Bayesian or Orthodox: Where do your intuitions fall? Consider the following scenarios and see what your intuitions tell you. You might reject all the answers or feel attracted to more than one. Real research questions do not have pat answers. See if, nonetheless, you have clear preferences for one or a couple of answers over another. Almost all answers are consistent either with some statistical approach or with what a large section of researchers do in practice, so do not worry about picking out the one ‘right’ answer (though, given certain assumptions, I will argue that there is one right answer!). 1) You have run the 20 subjects you planned and obtain a p value of .08. Despite predicting a difference, you know this won’t be convincing to any editor and run 20 more subjects. SPSS now gives a p of .01. Would you: a) Submit the study with all 40 participants and report an overall p of .01? b) Regard the study as non-significant at the 5% level and stop pursuing the effect in question, as each individual 20-subject study had a p of .08? c) Use a method of evaluating evidence that is not sensitive to your intentions concerning when you planned to stop collecting subjects, and base conclusions on all the data? 5 2) After collecting data in a three-way design you find an unexpected partial two-way interaction, specifically you obtain a two-way interaction (p = .03) for just the males and not the females. After talking to some colleagues and reading the literature you realise there is a neat way of accounting for these results: certain theories can be used to predict the interaction for the males but they say nothing about females. Would you: a) Write up the introduction based on the theories leading to a planned contrast for the males, which is then significant? b) Treat the partial two-way as non-significant, as the three-way interaction was not significant, and the partial interaction won’t survive any corrections for post hoc testing? c) Determine how strong the evidence of the partial two-way interaction is for the theory you put together to explain it, with no regard to whether you happen to think of the theory before seeing the data or afterwards, as all sorts of arbitrary factors could influence when you thought of a theory? 3) You explore five possible ways of inducing subliminal perception as measured with priming. Each method interferes with vision in a different way. The test for each method has a power of 80% for a 5% significance level to detect the size of priming produced by conscious perception. Of these methods, the results for four are non-significant and one, the Continuous Flash Suppression, is significant, p = .03, with a priming effect numerically similar in size to that found with conscious perception. Would you: 6 a) Report the test as p=.03 and conclude there is subliminal perception for this method? b) Note that when a Bonferoni-corrected significance value of .05/5 is used, all tests are non-significant, and conclude subliminal perception does not exist by any of these methods? c) Regard the strength of evidence provided by these data for subliminal perception produced by Continuous Flash Suppression to be the same regardless of whether or not four other rather different methods were tested? 4) A theory predicts a difference in reaction time between two conditions. A previous study finds a significant difference between the conditions of 20 seconds, with a Cohen’s dz of 0.5. You wish to replicate in your lab. In order to obtain a conventional power of 80% you run 35 subjects and find a t of 1.0 and a p of .32. Would you a) conclude that under the conditions of your replication experiment there is no effect? b) conclude that null results are never informative and withhold judgment about whether there is an effect or not? c) realise that while 20 seconds is a likely value given the theory being tested, the difference could in fact be 15 seconds either side of this value and still be consistent with the theory. You treat the evidence as inconclusive; e.g. your certainty in the theory might go down modestly from being about 65% to a bit more than 50%, and so you decide to run more subjects until the evidence more strongly supports the null over the theory or the theory over the null? 7 5) You look up the evidence for a new expensive weight loss pill. Use of the pill resulted in significant weight loss after 3 months daily ingestion with a beforeafter Cohen’s dz of 1.0 with n=10 subjects giving a p of .01. In addition, you accept that there are no adverse side effects. Would you: a) Reject the null hypothesis of no change and buy a 3 month’s supply? b) Decide 10 subjects does not provide enough evidence to base a decision on when it comes to taking a drug, withhold judgment for the time being, and help sponsor a further study? c) Decide that in a 3-month period you would like to loose between 10-15kg. In fact, despite the high standardised effect size, the raw mean weight loss in the study was 2kg. The evidence that the pill uses a mechanism producing 0-10 kg loss (which you are not interested in) rather than 10-15kg (which you are) is overwhelming. You have sufficient data to decide not to buy the pill? We will discuss answers to this quiz below. But first we need to establish what the rational basis for orthodox and Bayesian statistics consists in and why they can produce different answers to the above questions. Rationality What is it to be rational? One answer is it is having sufficient justification for one’s beliefs; another is that it is a matter of having subjected one’s beliefs to critical scrutiny. Popper and others inspired by him took the second option under the name of 8 critical rationalism (e.g. Popper, 1963, Miller, 1994). On this view, there is never a sufficient justification for a given belief because knowledge has no absolute foundation. Propositions can be provisionally accepted as having survived criticism, given other propositions those people in the debate are conventionally and provisionally willing to accept. All we can do is set up (provisional) conventions for accepting or rejecting propositions. An intuition behind this approach is that irrational beliefs are just those not subjected to sufficient criticism. Critical rationalism bears some striking similarities to the orthodox approach to statistical inference, the Neyman Pearson approach (an approach almost universally used by users of statistics, but few would know it by that name, or any other). On this view, statistical inference cannot tell you how confident to be in different hypotheses; it only gives conventions for behavioural acceptance or rejection of different hypotheses, which, given a relevant statistical model (which can itself be subjected to testing), results in pre-set long term error rates being controlled. One cannot say how justified a particular decision is or how probable a hypothesis is; one cannot give a number to how much data supports a given hypothesis (i.e. how justified the hypothesis is, or how much its justification has changed); one can only say that the decision was made by a decision procedure that in the long run controls error probabilities (as objective probabilities in the sense of long run frequencies) (see Dienes, 1008, chapter 3, for a conceptual introduction to this approach; also Oakes, 1986; and Royall, 1997, for why p values do not provide such degrees of support). Note probability is a long run relative frequency so it does not apply to the truth of hypotheses, nor even to particular experiments. It is the long run relative frequency of errors for a given decision procedure. It can be obtained from the tail area of test 9 statistics (e.g. tail area of t-distributions), adjusted for factors that affect long run error rates, like how many other tests are being conducted. These error rates apply to decision procedures not to individual experiments. An individual experiment is a oneoff event, so it does not determine a unique long-run set of events; but a decision procedure can in principle be considered to apply over a long run indefinite number of events (i.e. experiments). Now consider the other approach to rationality, that it is a matter of having sufficient justification for one’s beliefs. If we want to assign numerical degrees of justification (i.e. of belief ) to propositions, what are the rules for logical and consistent reasoning? Cox (1946; see Sivia & Skilling, 2006) took two minimal desiderata, namely that 1. If we specify degree of belief in P we have implicitly specified degree of belief in not-P 2. If we specify degree of belief in P and also specify degree of belief in (Q given P) then we have implicitly specified degree of belief in (P&Q) Cox did not assume in advance what form this specification was nor what the relationships were; just that the relationships existed. Using deductive logic Cox showed that degrees of belief must follow the axioms of probability if we wish to accept the above minimal constraints. Thus, if we want to determine by how much we should revise continuous degrees of belief, we need to make sure our system of inference obeys the axioms of probability. In my experience, researchers think all the time in terms of the degree of support data provide for a hypothesis. If they want to think that way, they should make sure their inferences obey the axioms of probability. 10 One version of such continuous degrees of belief are subjective probabilities, i.e. personal conviction in an opinion (e.g. Howson & Urbach, 2006). One can hone in on one’s initial personal probabilities by various gambling games (see Dienes, 2008, chapter four, for an introductory review of these ideas). This can be a useful idea for how one could have probabilities for different propositions when it is hard to specify clear and full reasons for why the probabilities must have certain values. It is natural that people regard the same theory as being more or less plausible, and that probabilities can be personal. However, when probabilities of different propositions form part of the inferential procedure we use in deriving conclusions from data then we need to make sure that the procedure is fair. Thus, there has been an attempt to specify “objective probabilities” that follow from the informational specification of a problem (e.g. Jaynes, 2003). This will be a useful way of thinking about probabilities for evaluating how much data support different hypotheses. In this sense, probabilities can be normative convictions a person should have given the constraints and information made explicit in the statement of the problem. In this way, the probabilities become an objective part of the problem, whose values can be argued about given the explicit assumptions, and do not depend on any further way on personal idiosyncrasies. Note these sort of probabilities can be regarded as consistent with critical rationalism (despite Popper’s aversion to Bayes): The assumptions defining the problem are without absolute foundation, they are open to criticism, but can be debated until tentatively accepted. In any case, whatever probabilities one starts with (entirely subjective personal ones gathered by reaching deep in one’s soul, or objectively specified ones given stated constraints), Bayesian inference insists that 11 one must revise these initial probabilities in the light of data in ways consistent with the axioms of probability. In the Bayesian approach, probability applies to the truth of theories (the relative frequency notion of probability as used in Neyman Pearson statistics does not apply to theories). Thus we can answer questions about p(H), the probability of a hypothesis being true (our prior probability), and also p(H|D), the probability of a hypothesis given data (our posterior probability), neither of which we can do on the orthodox approach. The probability of obtaining the exact data we got given the hypothesis is the likelihood. From the axioms of probability, it follows directly that: Posterior is given by likelihood times prior From this theorem (Bayes’ theorem) comes the likelihood principle: All information relevant to inference contained in data is provided by the likelihood (e.g. Birnbaum, 1962). When we are determining how given data relatively changes the probability of our different theories, it is only the likelihood that connects the prior to the posterior. The likelihood is the probability of obtaining the exact data obtained given a hypothesis, P(D|H). This is different from a p-value, which is the probability of obtaining the same data or data more extreme given both a hypothesis and a decision rule. Thus, a p-value for a t-test is a tail area of the t-distribution (adjusted according to the decision rule); the corresponding likelihood is the height of the t-distribution at the point representing the data - not an area and certainly not an area adjusted for the 12 decision rule. In orthodox statistics these adjustments must be made because they accurately reflect the factors that affect long term error rates of a decision procedure. The likelihood principle may seem a truism; it seems to just follow from the axioms of probability. But in orthodox statistics, p-values are changed according to the decision rule: How one decided to stop collecting data; whether or not the test is post hoc; how many other tests one conducted. None of these factors influence the likelihood. Thus, orthodox statistics violates the likelihood principle. I will consider each of these cases because they have been used to argue Bayesian inference must be wrong, given that we have been trained as researchers to regard these violations of the likelihood principle to be a normative part of orthodox statistical inference. But these violations of the likelihood principle also lead to bizarre paradoxes. I will argue that when the full context of a problem is taken into account, the arguments against Bayes based on these points fail. On the other hand, where subjective probabilities play a role in the construction of the likelihood, care does need to be taken in establishing inter-subjective consensus for Bayesian inferences to be generally acceptable. The Bayes factor One form of Bayesian analysis pits one theory against another, say theory1 against theory2. Theory1 could be your pet theory put under test in an experiment; theory2 could be the null hypothesis, or some other sort of default position. If your personal probability of theory1 being true before the experiment is P(theory1) and that for theory2 is P(theory2), then your prior odds in favour of theory1 over theory2 is P(theory1)/P(theory2). These prior probabilities and prior odds can be entirely 13 personal or subjective; there is no reason why people should agree about these before data are collected. Once data are collected we can calculate the likelihood for each theory. These likelihoods are things we want people to agree on; thus, any probabilities that contribute to them should be plausibly or simply determined by the specification of the theories. The Bayes factor B is the ratio of the likelihoods. From the axioms of probability, Posterior odds = B*prior odds If B is greater than 1 then the data supported your experimental hypothesis over the null. If B is less than 1, then the data supported the null hypothesis over the experimental one. If B is about 1, then the experiment was not sensitive (Jeffreys, 1961, suggests Bayes factors above 3 - or below 1/3 - are “substantial”). Note that B automatically gives a notion of sensitivity; it directly distinguishes data supporting the null from data uninformative about whether the null or your theory was supported. Contrast this state of affairs with just relying on p values in significance testing. The most common mistake people make is believing they can simply take a nonsignificant p-value, and from this alone decide that the null was supported over the theory. In fact, the Neyman Pearson approach itself proscribes against this. It says one should calculate power, and only accept the null when power was high. If people followed the Neyman Pearson approach as it should be done, they would know when a null result meant one could accept the null and when it meant withholding decision. Unfortunately, one CAN simply calculate p-values without calculating power, this is easier, so that’s what people do. In Bayes one does not have the choice: You get the 14 full answer whether you want it or not. This is a practical reason for preferring Bayes over Neyman Pearson. According to this argument, both approaches would work fine if done correctly; but Bayes forces one to do it correctly and Neyman Pearson obviously does not. Decades of statisticians admonishing psychologists has not made them do it properly (see e.g Cohen, 1977, 1994; Harlow, Mulaik, & Steiger, 1997; Hunter & Schmidt, 2004). Now I wish to consider reasons for preferring Bayes even if Neyman Pearson is done correctly. And those reasons are based on the ways in which Neyman Pearson depart from the likelihood principle. Problems with Neyman Pearson 1) On the Neyman Pearson approach one must specify the “stopping rule” in advance, i.e. the conditions under which one would stop collecting data. Once those conditions are met, there is to be no more data collection. Typically this means you should plan in advance how many subjects you will run (with a power calculation). You are forbidden from running until you get a significant result – because if you try you will always succeed, given sufficient time, even if the null is true. Further, one cannot plan to run 30 subjects, find a p of .06, and then run 10 more, and report the pvalue of .04 SPSS now delivers for the full set of 40 subjects, and declare it significant at the 5% level. Five percent is an inaccurate reflection of the error rate of the decision procedure one used (that procedure being: “Run 30 subjects, test, if nonsignificant, run 10 more subjects and test again”). You cannot test your data once at the .05 level, then run some more subjects, and test again at .05 level. The Type I error rate is no longer .05 because you gave yourself two chances at declaring 15 significance. Remember on the Neyman Pearson approach probabilities are long run relative frequencies – i.e. the long run properties of your decision procedure, and thus do not apply to your individual experiment. You will make a Type I error 5% of the time at the first test; the second test can only increase the percentage of Type I errors. The significance can thus be never less than .05, no matter how many subjects one subsequently runs and how strong the evidence seems against the null and no matter what SPSS tells you the p-value is. Each test must be conducted at a lower significance level for the overall error rate to be kept at .05. This puts one in an impossible moral dilemma having tested once at the 5% level if an experiment after 30 subjects yields a p of .06. One cannot reject the null on that number of subjects; yet one cannot accept it either (no matter what the official rules are, would you accept the null for a p of .06?). One cannot publish the data; yet one cannot in good heart bin the data and waste public resources. Bayes avoids this impossible moral dilemma. When calculating a Bayes factor, it does not matter when you decide to stop running subjects. You can always run more subjects if you think it will help. 16 Figure 1 The importance of the stopping rule for significance testing Consider the cartoon in Figure 1. Two researchers come to the end of running 40 subjects. One calculates the p-value based on the full set of data without correction: he had planned to run that many from the start and only test once at the end of data collection. He gets a significant result and decides to reject the null. The other researcher tested after 20 subjects, found it non-significant, so tested again at the end of 40. But he wasn’t cheating: he made the appropriate corrections (see Armitage, Berry and Mathews, 2002, pp 615-623 for examples of legitimate corrections). Now on the final test, because of the correction, the result is non-significant. Notice how the appropriate inference (to accept or reject the null) depends on more than the likelihood, that is, on more than how likely these precise data are given the null 17 hypothesis or theory under test. It depends not just on what the data are but also on what might have happened, even if it did not. In particular, appropriate inference depends on if the researchers would have stopped after 20 subjects if it had been significant – even though it wasn’t, and they didn’t. Notice in the cartoon that if whether they would have stopped collecting data after 20 subjects, even though they didn’t, depends on who has the better kung fu, then the mathematically correct result depends on whose kung fu is better! And if whether they would have stopped collecting data after 20 subjects depends on who has the strongest unconscious desire to please the other then the mathematically correct answer depends on whose unconscious wish to please the other is strongest! This example might seem deliberately absurd but the point is very real and practical. How many people have looked at the results some way through testing – how many would have stopped collecting if the results had been clear enough then? Would you? (Are you sure?) How many people “top up” with just a few more subjects? I believe the practice to be very common. As I said, there are ethical problems with not doing so in many cases. And maybe many people justify it because in their hearts they believe in the likelihood principle: Surely what subjective intentions are concealed in the mind are irrelevant to drawing inferences from data; what matters is just what the obtained data are. The problem is, one cannot believe the likelihood principle and follow Neyman Pearson techniques. Long term error rates using significance tests are affected by counterfactuals, even if likelihoods are not. The Bayes factor behaves differently from p-values as more data are run (regardless of stopping rule). For a p-value, if the null is true, any value in the interval 0 to 1 is equally likely no matter how much data you collect. For this reason, sooner 18 or later you are guaranteed to get a significant result if you run subjects for long enough. In contrast, as you run more subjects and the null is true, or closer to the truth than your alternative, the Bayes factor is driven towards zero. You could run an infinite number of subjects and never have B achieve a given value, e.g. 4. Hence you can run as many subjects as you like, stopping when you like. Isn’t this what researchers really want of their inferential statistics? And if this sounds more sensible, it is because it is literally more rational. 2) On Neyman Pearson, it matters whether you formulated your hypothesis before or after looking at the data (post hoc vs planned comparisons): Predictions made in advance of rather than before looking at the data are treated differently. In Bayesian inference, it does not matter what day of the week you thought of your theory. The evidence for your theory is just as strong regardless of its timing relative to the data. This is because the likelihood is unaffected by the time the data were collected. Note the likelihood principle contradicts not only Neyman Pearson on this point, but also the advice of e.g. Popper (1963) and Lakatos (1978), who valued the novelty of predictions (though Lakatos later gave up the importance of temporal novelty, Lakatos & Feyerabend, 1999, pp 109-112). Kerr (1998) also criticised the practice of HARKing: Hypothesising After the Results are Known. Indeed, novel predictions are often impressive as support for a theory. But this may be because in making novel predictions the choice of auxiliary hypotheses (i.e. those hypotheses implicitly or explicitly used in connecting theory to specific predictions) was fair and simple. Post hoc fitting can involve preference (for no good reason) for one auxiliary over many others of at least equal plausibility. Thus, a careful consideration of the reason for 19 postulating different auxiliaries should render novelty irrelevant as a factor determining the evidential value of data. That is, the issue is not the timing of the data per se, but the priori probability of the hypotheses involved. And prior probability is something Bayes is uniquely well equipped to deal with! Consider an example that has been used as a counter argument to the likelihood principle. I have a pack of cards face down. I lift up the top card. It is a six of hearts. Call the hypothesis that the pack is a standard pack of playing cards “Hs”. Call the hypothesis that all cards in the pack are six of hearts “H6h”. The likelihood of drawing a six of hearts given the pack is a standard pack is 1/52. So L(Hs) = 1/52. The likelihood of drawing a six of hearts given the hypothesis that the pack consists only of 52 six of hearts is 1. So L(H6h) = 1. So the Bayes factor in favour of the pack consisting only of sixes of hearts versus being a standard pack is 52. If you saw a pack of cards face down your initial prediction would be that they are a standard pack of playing cards. The drawing of a single six of hearts would not change your mind. Surely the hypothesis that they are all sixes of hearts is purely post hoc, a mindless fitting of the data! If Bayesian statistics support the post hoc theory over the theory that it is a standard pack of cards, surely something is wrong with Bayes! If someone could predict in advance that the pack was all sixes of hearts that would be one thing; but that is precisely the point, they wouldn’t. By missing out on the importance of what can be predicted in advance, does not the likelihood principle fail scientists? Remember the likelihood principle follows from the axioms of probability. The axioms of probability are by their nature almost self-evident assumptions. They will not lead to wrong conclusions. Indeed, in this case, if we put the problem in its full context, we see the Bayesian answers are very sensible (see Royall, 1997, p 13, 20 for the following argument). Before we pick up the card there are 52 hypotheses that the pack is all of one sort of card – the hypothesis that it is all aces of hearts, all twos of hearts, and so on. Call the probability that one or other of these hypotheses is true π. So the probability of any one of them being true is π /52, assuming we hold them to all have equal probability. Once we have observed the six of hearts, all these hypotheses go to zero, except for H6h. The probability of that hypothesis goes to π. The probability that the whole pack is all of one suit remains the same – it is still π.. The probability that it is a standard pack of cards remains the same. The axioms of probability and their consequence, Bayes theorem, give us just the right answer. There is no need to introduce an extra concern with ability to predict in advance; that concern is already implicitly covered in the Bayesian approach. It is not the ability to predict in advance per se that is important; that ability is just an (imperfect) indicator of the prior probability of relevant hypotheses. The data provide stronger or weaker evidence for different hypotheses regardless of the day of the week the data were collected. When performing Bayesian inference there is no need to adjust for the timing of predictions per se. Indeed, it would be paradoxical to do so: Adjusting conclusions according to when the hypothesis was thought of would introduce irrelevancies into inference, leading to one conclusion on Tuesday and another on Wednesday for the same data and hypotheses. 3) On Neyman Pearson you must correct for how many tests you conduct in total. For example, if you ran 100 correlations and 4 were just significant at the 5% level, researchers would not try to interpret those significant results. On Bayes, it does not 21 matter how many other statistical hypotheses you investigated. All that matters is the data relevant to each hypothesis under investigation. Consider an example from Dienes (2008) to first pump your intuitions along Neyman Pearson lines; and then, as above, we will see how the axioms of probability do indeed give us the sensible answer, and hopefully your intuitions come to side with Bayes. The example is about searching for the reincarnation of a recently departed lama by a search committee set up by the Tibetan Government-in-exile. The lama’s walking stick is put together with a collection of 20 others. Piloting at a local school shows each stick is picked equally often by children in general. We have now set up a test with a known and acceptable test-wise Type I error probability, i.e. controlled to be less than 5% for each individual test. If a given candidate picks the stick, p = 1/21 < .05. Various omens narrows the search down to 21 candidate children. They are all tested and one of these passes the test. Can the monks conclude that he is the reincarnation? The Neyman Pearson aficionado says, “No! With 21 tests family-wise error rate = 1 – (20/21)21 = 0.64. This is unacceptably high. Of course if you test enough children, sooner or later one of them will pass the test. That proves nothing. As Bayes does not correct for multiple testing, surely the Bayesian approach must be wrong!” The Bayesian argues like this. “Assume the reincarnation will definitely choose the stick. If the 10th child chose the stick, the Bayes factor B for the 10th child = 21. Whatever your prior odds that the 10th child was the reincarnation they should be increased by a factor of 21.” The Neyman Pearson aficionado cannot contain himself. “Ha! You have manufactured evidence out of thin air! By ignoring the issue of multiple testing you have found strong evidence in favour of a child being the reincarnation just because 22 you tested many children!” (see e.g. Mayo, 1996, 2004, for this argument against Bayes). The Bayesian patiently continues with an argument similar to the one in the previous section, “The likelihood of any child who did not choose the stick is 0. Call the prior probability that one or other of these children was the reincarnation π. If prior probabilities for each individual child equal, prior probability that any one is the reincarnation = π/21. After data, twenty of these go to zero. One goes to 21 * π/21 = π. They still sum to π. If you were convinced before collecting the data that the null was false you can pick the reincarnation with confidence; conversely if you were highly confident in the null beforehand you should be every bit as confident afterwards. And this is just as it should be!” The Bayesian answer does not need to correct for multiple testing because if an answer is already right it does not need to be corrected. Once one takes into account the full context, the axioms of probability lead to sensible answers, just as one would expect. As I point out in Dienes (2008), a family of 20 tests in which one is significant at the .05 level typically leads one by Bayesian reasoning to have MORE confidence in the family-wise null hypothesis that ‘all nulls are true’ while decreasing one’s confidence in the one null that was significant. And this fits one’s intuitions that if evidence went against the null in 4 out of 100 correlations, one would be more likely to think the complete null is true, but still find oneself more likely to reject the null for the four specific cases. If all 100 correlations bore on a theory that predicted non-zero correlations in all cases, then one’s confidence in that theory would typically decrease by a Bayesian analysis. 23 The moral is that in assessing the evidence for or against a theory, one should take into account ALL the evidence relevant to the theory, and not cherry pick the cases that seem to support it. Cherry picking is wrong on all statistical approaches. A large number of results showing evidence for the null against a theory still count as against the theory, even if a few of the effects the theory predicted are supported. And Bayes gives one the apparatus for combining such evidence to come to an overall conclusion, an apparatus missing in Neyman Pearson. Thus, it is Bayes, rather than Neyman Pearson, most likely to demand of researchers they draw appropriate conclusions from a body of relevant data involving multiple testing. Bayes factors close to zero count as evidence against the theory; in practice, non-significant values are either left to count or not depending on whim. 4) Finally, because the likelihood tells one by how much to change confidence, and because significance testing violates the likelihood principle, the Neyman Pearson approach does not tell one how to change confidence. A significant difference (i.e. accepting the experimental hypothesis on Neyman Pearson) can mean one should reduce one’s confidence in a theory that predicted the difference…. and a null result can mean confidence in theory should increase (see Dienes 2008, chapter four, and below for examples). Neyman Pearson does not tell you how justification for a theory has changed. No matter what anyone tells you, Neyman-Pearson analyses do not directly license assigning any degree of confidence to one conclusion rather than another. If you are interested in the extent to which you should change your confidence in a theory, you must do a Bayesian analysis. Consider for example if a particular theory has often predicted effects which turn out to be about 100 ms in size. 24 In a new prediction in a domain where one would expect the same size effect, a significant effect of 5 ms is obtained. This may well count strongly against the theory. To summarise the arguments of this section, key differences between the approaches that follow from the likelihood principle are shown in Table 1. Table 1 Contrasts between Bayesian and orthodox statistics following from whether or not the likelihood principle is obeyed. Because orthodox statistics are sensitive to the factors listed, contrary to the likelihood principle, in each case different people with the same data and hypothesis may come to opposite conclusions. Orthodox: Could this factor affect whether or not a null hypothesis is rejected? Bayes: Does this factor ever affect the support of data for a hypothesis? When you initially intended to stop running participants Yes Whether or not you predicted a result in advance of obtaining it Yes The number of tests that can be grouped in a family No. You can always run more participants to acquire clearer evidence if you wish No. No one need ever try to second guess which really came first. No. Please test as many different hypotheses as is worth your time – but you must take into account all evidence relevant to a theory. Your answers to the quiz Yes 25 Now consider the situations we started with. What are your intuitions now? In all cases, answer c) is the Bayesian answer. For question 1, I suspect a majority of researchers have at some time taken a) as their answer in similar cases. They have Bayesian intuitions, but use them with the wrong tools, the only tools apparently available, and tools inappropriate for actually cashing out the intuition. Choice a) is also the answer one might pick by thinking with a meta-analytic mind set, but use of meta-analysis here is complicated by the fact the stopping rule was conditional upon obtaining a significant finding. Thus, the correct orthodox answer is to regard the data non-significant, as in b). Power may be low, but in effect one committed to that level of Type II error in planning the study. Answer c) spells out the intuition, and Bayes provides the tools for implementing it. For question 2, again I suspect many people have decided a) in similar circumstances, because of the Bayesian intuitions in c), and so used the wrong tools for the right reasons. One suspects in many papers the introduction was written entirely in the light of the results. We implicitly accept this as good practice, indeed train our students to do likewise for the sake of the poor reader of our paper. But b) is the correct answer based on the Neyman Pearson approach, and maybe your conscience told you so. But should you be worrying about what might be murky – which really came first, data or hypothesis? – or, rather, about what really matters, whether the predictions really follow from a substantial theory in a clear simple way? For question 3, practice may vary depending on whether the author is a believer or sceptic in subliminal perception. After all, there is no strict standard about what counts as a “family” for the sake of multiple testing. There is a pull between accepting the intuition in c) that surely there is evidence for this method, and the 26 realisation that more tests means more opportunities for inferential mistakes. But one should not confuse strength of evidence with the probability of obtaining it (Royall, 1997). Evidence is evidence even if, as one increases the circle of what tests are in the “family”, the probability that some of the evidence will be misleading increases. For question 4, many researchers may conclude a), and indeed feel that by taking power into account they are morally ahead of the pack of typical researchers who ignore power. Those schooled in Fisherian intuitions may choose b). A Bayesian analysis forces one to consider the range of effect sizes consistent with a theory. As soon as one considers this question, whether as a Bayesian or not, it will become apparent that the effect obtained in a previous study does not define the lowest effect size one is interested in. Thus, even in studies that do take power into account, likely they do not consider a minimally interesting effect, and likely they use rather low power (80%) – rather low that is, for obtaining evidence that could substantially favour a null hypothesis over the theory of interest. Because orthodox methods are not based on how persuasive you should find evidence, they often allow conclusions that in fact should not be persuasive. And without doing a Bayesian analysis, you have no idea how persuasive they should be. Question 5 again illustrates a case where orthodox statistics could produce the same answer as Bayes: One could calculate a confidence interval on raw weight loss and see that it excludes the values one is interested in. In this case, the advantage of the Bayesian approach is that it forces you to consider what range of effects you are really interested in. It forces you to take into account that which is inferentially relevant. Orthodox statistics do not (cf e.g. Kirsch, 2009). 27 Now dear reader and fellow journey person: Are you a closet – or indeed, out Bayesian? What inferential methods seem rational to you? If you want some pointers in bringing our your inner Bayesian, read on! Bayes factors in practice The last example illustrates the importance of considering the size of an effect in using it to evaluate a theory. Effect size is very important in the Neyman Pearson approach: One must specify the sort of effect one predicts in order to calculate power. On the other hand, Fisherian significance testing leads people to ignore effect sizes. People have followed Fisher’s way, while paying lip service to effect sizes. By contrast, to calculate a Bayes factor one must specify what sort of effect sizes the theory predicts. Bayes forces people to think about effect size. Nowadays many journals require one mention standardised effect sizes for each inferential test. But has this led people to either calculate power or use confidence intervals? When confidence intervals are given, do authors argue what size effect would be predicted by the theory and indicate whether the confidence interval includes or excludes effects interesting on the theory? That is what people should be doing (particularly when interpreting null results), but they do not. People may mention effect sizes but they don’t seem to do anything with this information other than state what they are. And when power is calculated, it is often calculated for “a Cohen’s d of 0.5” because, the authors say, they are interested in “medium sized” effects. This is a step above not calculating power at all. But it is still an unthinking mechanical response when we could be doing so much more. What size effect does 28 the literature suggest is interesting for this particular domain? Rather than plucking “0.5” out of thin air we should get to know the data of our field. Often the already existing published data in our field allow us to say in raw units just what sort of effect size a theory deals with. Sometimes one really does not know; then using wild speculations – like a Cohen’s d of 0.5 because that is the sort of effect psychologists in general often deal with – may be the best one can do. But often it will be not far off the least one can do. (For arguments for the frequent relevance of raw rather than standardised effect sizes, see Baguley, 2009; Ziliak & McCloskey, 2008.) It is easy to see why psychologists, and other users of statistics, became Fisherian, i.e. just reported p values and did not consider effect sizes in an inferential way. Reporting p-values requires minimal thought. While that response might sound cynical, it is indeed an advantage of p-values: The result requires minimal assumptions. For this reason, I suspect p-values may be interesting as a side line when reporting many tests. It is the result one gets with minimal assumptions. Nonetheless, this approach has done enough mischief we really have to change. Unless one incorporates effect sizes into one’s inferences, dealing with null results is impossible. How predictably are null results handled without thought, leaving the reader genuinely uninformed about the status of some theory or treatment or danger, whatever the confident assertions of the author. The data has been collected to inform the reader, but the reader is not getting informed. Untold damage has been done to many fields because of this lapse. Despite some attempts to encourage researchers to use confidence intervals their use has not taken off (Fidler et al, 2004). Confidence intervals of some sort would deal with many problems (either confidence, credibility or likelihood intervals; 29 see Dienes 2008 for definitions and comparison). But an approach that has something of a flavour of a t or other inferential test might be taken up more easily. Further, confidence intervals themselves have all the problems enumerated above for Neyman Pearson inference in general (unlike credibility or likelihood intervals). So here I urge the use of the Bayes factor. To calculate a Bayes factor in support of a theory (relative to say the null hypothesis), one has to specify what the probability of different effect sizes are, given the theory. In a sense this is not new: We should have been specifying predicted effect sizes anyway. And if we are going to do it, Bayes gives us the apparatus to flexibly deal with different degrees of uncertainty regarding the predicted effect size. For example consider a theory that predicts a difference will be in one direction. A minimally informative distribution, containing only the information that the difference is positive, is to say all positive differences are equally likely between zero and the maximum difference allowed by the scale used. Such a vague prediction works against finding evidence in favour of the theory. Generally we can do better than that. For example, it seems rather unlikely that the difference will be as large as the maximum allowed by the scale: That requires all subjects in one condition were at one extreme of the scale, and all subjects in the other condition were at the other extreme. In general, smaller effects are more likely than the larger ones. This can be modelled by one half of a normal distribution, with its mode at zero, and its tail dropping away in the positive direction. But how to scale the rate of drop? If similar sorts of effects as those predicted in the past have been on the order of a 5% difference between conditions in classification accuracy, then we can set the standard deviation of the normal to be 5%. This distribution would imply that smaller effects 30 are more likely than bigger ones; and that effects bigger than about 10% are unlikely. If an argument based on the existing literature makes these assumptions plausible, then the Bayes factor based on those assumptions is one that can be accepted generally. To play with how assumptions affect the Bayes factor, see the web site for Dienes (2008) for flash programs and Matlab code for the Bayes factor, and Baguley and Kaye (in press) for corresponding R code; and Rouder et al (2009) for another Bayes factor calculator. Bayes factors vary according to assumptions, but they cannot be made to vary ad lib: Often a wide range of assumptions leads to essentially the same conclusion. The results will depend on the assumptions about likely effect sizes given the theory. But notice these assumptions are open to public scrutiny. They can be debated and other assumptions used according to the debate. In this sense Bayes is objective. In Neyman Pearson inference, the inference depends on how the experimenter decided to stop and when he thought of the hypothesis. These concerns are not open to public scrutiny and may not even be known by the author. All assumptions relevant to Bayesian inference are available to critical debate because they are part of the public problem situation itself and not locked in the head of researchers. Strangely, starting from subjective probabilities leads to more objective conclusions than starting with objective probabilities (as relative frequencies)! A statistical philosophy would not have persisted so long if it did not produce tenable conclusions in many cases. And indeed, in many cases Bayesian and orthodox answers agree, which is reassuring. For example, Dienes et al (2009) investigated possible correlates of hypnotic suggestibility. Previous research had found that a task measuring cognitive inhibition correlated about .40 with hypnotic suggestibility. With 31 180 participants, Dienes et al found a correlation of -.05, with a 95% confidence interval of [-.20. .09]. The null hypothesis was accepted. We can use the software from Dienes (2008) to calculate a Bayes factor by Fisher-z transforming the correlation so that its sampling distribution becomes normal. Based on the previous study alone, a positive correlation would be expected of around .40. However, in light of the larger literature, one can in addition say that smaller correlations are more likely than larger ones: Replicable correlates of hypnotic suggestibility, such as they are, tend to have values closer to .20 or .10 rather than .40 or .50 over many studies (e.g. Kirsch & Council, 1992). Thus, a reasonable distribution of the population (Fisher-z transformed) correlation value is a half normal with its mode at 0 and an SD of .25, effectively ruling out correlations greater than about .50, and allowing any value between 0 and .50, with smaller values more likely. This is a judgment based on knowledge of the literature, and it is of course open to debate. The assumptions are laid bare for anyone to consider. The obtained (Fisher-z transformed) correlation of .05 has a standard error of 1/squareroot(N-3) = .075. Feeding these numbers into the online software, the Bayes factor is .18, i.e. substantial evidence for the null hypothesis over the alternative considered. Note that this does not amount to accepting the null in any absolute sense; there is just more evidence for the null than the alternative considered. If the alternative was a normal centred on zero with a standard deviation of .10 (i.e. if one expected possible correlations between 0 and about .20 in either direction) the Bayes factor is .69, which barely changes one’s prior odds at all: The data do not discriminate the null hypothesis from the hypothesis of a range of small correlations. And of course, this is just as it should be. Finally, one can compare the evidence for a 32 correlation of zero to the specific value of .40 obtained in the prior study (the likelihoods for 0 and .40 can be obtained from the corresponding heights in normal tables). The Bayes factor is .00 to two decimal places, indicating exceptionally strong evidence for a correlation of zero rather than the previously obtained value of .40. The data are sufficiently clear that the precise statistical philosophy does not change the ultimate conclusions, even for a null result. But using Bayes does draw one’s attention to the amount of evidence for one hypothesis relative to a specified other. Orthodox hypothesis testing does not: One accepts or rejects the null outright. The specification of what the theory predicts of course depends on the theory. Bayesians tend at heart to be against mechanical unthinking procedures, and this is both a strength and a weakness. It is ultimately a strength, though it makes it harder for the procedures to be adopted generally. Consider a psychologist who decides to use a conventional expected medium effect size (Cohen’s d) of 0.5 to scale his distribution of the predicted effect, because medium effect sizes are typical for psychologists. He uses a distribution symmetric around zero, dropping off in either direction, scaled by a Cohen’s d of 0.5 (see Rouder et al 2009 for a Bayes factor calculator that works given these assumptions). These may be excellent assumptions to make, but they should not be made as an unthinking default. Another person may convincingly argue that one can be more specific: Based on past literature, this theory typically deals with raw effects of 100 ms. Cohen’s d may change according to what other factors and covariates are in the experiment, but one expects a raw effect of about 100ms in a certain direction. Further, the person might argue that an effect size of say 10ms would actually argue against the theory being the relevant explanation, because the effect would be too small. (It is not so far fetched to think psychologists 33 could say something this exact about effect sizes based on their theories: Consider the difference between pop out and serial search in the attention literature.) Now we could model the predictions of the theory by a normal centred on 100, and we would need to debate its standard deviation. The less information we have, the more we spread the distribution out until limited by known constraints that mean it cannot be spread further (e.g. maybe we want the probability of effects less than 20ms to be very small). The process of making these arguments means getting to know one’s theories and the existing data. In one sense Orthodox and Bayesian answers will never contradict each other because they answer entirely different questions (Royall, 1997). But a Bayes factor can indicate more support for the null hypothesis than a theory that predicts a nonzero effect when significance testing indicates one should reject the null. And vice versa, a Bayes factor can indicate more support for a theory that predicts a nonzero effect than the null hypothesis when a result is non-significant. Because the distribution of effects predicted by a theory depends on the theory, no firm rules can be given for when orthodox and Bayesian answers will differ in this respect. It all depends on the theory considered (cf Berger, 2003). Here we consider some hypothetical examples. Consider the theory that making prejudice between ethnic groups can be reduced by making both racial groups part of the same in-group. A manipulation for reducing prejudice following this idea could consist of imagining being members of the same sports team. A control group could consists of imagining playing a sport with no mention of the ethnic group. A post-manipulation mean difference in prejudice (in the right direction) is obtained with 30 participants of x raw units, equal to the standard error of difference; i.e. a non-significant t value of 1.00 is obtained. 34 What follows from this null result? Should one reduce one’s confidence in the theory (assuming the experiment is regarded as well designed)? It depends. Let us say in previous research, instead of imagining the scenario, participants actually engaged in a common activity. A reduction in prejudice on the same scale was obtained of 2x. It seems unlikely that imagination would reduce prejudice by more than the real thing. If smaller effects are regarded as more likely than larger effects in general, then we may model predictions by half a normal, with its mode on zero, and a standard deviation of x units. In this case, the Bayes factor is 1.38. That is the data are essentially uninformative but if anything we should more confident in the theory after getting these null results. It would be a tragic mistake to reject the usefulness of imagination treatments for prejudice based on this experiment. Indeed, even if the mean difference had been exactly zero, the Bayes factor is 0.71, that is, one’s confidence in the theory relative the null should be barely altered. More strongly, even if the mean difference had been x in the wrong direction, the Bayes factor is still 0.43. This does not count as substantial evidence against the theory by Jeffreys’ (1961) suggested convention of 3 (or a 1/3) for indicating substantial evidence; and indeed if one felt strongly confident of the theory before collecting the data, one could normatively still be very confident afterwards. Determining predicted effect sizes is hard if a motivation for a study is simply “I wonder what would happen if…”. One can only predict effect sizes if a motivation is given for the study that relates to previous work. And the more previous work it relates to, the more informed the prediction of effect sizes can be. Thus, a Bayesian analysis puts pressure on formulating general theories. The more one can specify a mechanism for an effect, the more one can, by reference to past work involving the 35 same putative general mechanism, pin down the expected effect size. And the more one can pin down an expected effect size by reference to other research, the more strongly the data can in principle support one’s theory, or else falsify it. This must be good for theory development. For example, consider obtaining a t-value of 2, a raw mean difference of x, leading to rejection of the null by orthodox statistics. If the experiment was not based on any theory at all, and not related to any past data or expectations, the vaguest prediction is that the difference should be between the extremes possible on the scale, e.g. plus and minus 10x. The Bayes factor is 0.46, marginally counting against the vague expectation of some difference. But if theory led one to predict a difference of around x, and lying between 0 and 2x, we could model the prediction of the theory as a normal with mean x and standard deviation x/2. Now the Bayes factor is 5, strongly supporting the theory. That is, the more researchers can make reasonably precise predictions, the more Bayes can reward them. (And if they do so with implausible assumptions, surely other researchers will correct them, and argue for more reasonable ways of applying the theory, leading to useful debate on plausible auxiliary hypotheses to link theory to data.) In the development of new paradigms, where there may not be past data to refer to, pilot studies can be done on the basic effect to scale the expected size of manipulations of the effect in order to calculate a Bayes factor (as is done in Dienes, Baddeley, & Jansari, submitted). Weaknesses of the Bayesian approach The strengths of Bayesian analyses are also its weaknesses: 36 1. Calculating a Bayes factor depends on answering the following question about which there may be disagreement: What way of assigning probability distributions of effect sizes as predicted by theories would be accepted by protagonists on all sides of a debate? Answering this question might take some arguing. But isn’t this just the sort of argument that psychology has been missing out on and could really do with (cf Meehl, 1967)? People would really have to get to know their data and their theories better to argue what range of effect sizes their theory predicts. This will take effort compared to simply calculating p-values. The very effort of calculating Bayes factors will have a desirable consequence: People will think carefully about what specific (probably onedegree-of-freedom) contrasts actually address the key theoretical questions of the research, and people will not churn out e.g. all the effects of an ANOVA just because they can. Results sections will become focused, concise and more persuasive. But to begin with, psychologists may start using Bayes factors only to support key conclusions, especially based on null results, in papers otherwise based on extensive orthodox statistics. Of course, it would have to be done that way initially because editors and reviewers expect orthodox statistics. And it would in any case be good to explore the use of Bayes factors gradually. Once Bayes factors become part of the familiar tool box of researchers, their proper use can be considered in the light of that experience. An alternative response to the problem of assigning a probability distribution to effect sizes is to not take on the full Bayesian apparatus: One CAN just report likelihoods for the simple hypotheses that the population value is 1, 1.1, ….etc (e.g. 37 consider comparing the likelihood of a correlation of zero with a correlation of .40 in the example above). This is “theory free” in the sense that no prior probabilities are needed for these different hypotheses (see Blume, in press; Dienes, 2008, chapter five; Royall, 1997). This procedure results in a likelihood interval, similar to confidence interval (though one that follows the likelihood principle). The “likelihood approach” has the advantage of not committing to an objective or subjective notion of probability, and not worrying about precisely how to specify prior distributions, while committing to the likelihood principle. On the other hand, if a probability distribution over effect sizes can be agreed on, the full use of Bayes can be obtained (Jaynes, 2003). In particular, one can average out nuisance parameters, and assign relative degrees of support to different theories each consistent with a range of effect sizes (for example, the null hypothesis need not be just the hypothesis of zero, but the range of values too small to be support for a theory). 2. Bayesian procedures, because they are not concerned with long term frequencies, are not guaranteed to control Type I and type II error probabilities (Mayo, 1996). Royall (1997) showed how the probability of making certain errors with a likelihood ratio – or Bayes factor - can be calculated in advance. In particular, for a planned number of subjects, one can determine the probability that the evidence will be weak (Bayes factor close to 1) or misleading (Bayes factor in wrong direction). These error probabilities have interesting properties compared to Type I and II error rates: No matter how many subjects one runs, Type I error is always the same, typically 5%. But for Bayes factors, the more subjects one runs, the smaller the probability of weak or misleading evidence. Further, these probabilities decrease as 38 one runs more subjects no matter what one’s stopping rule. One can always decide to run some more subjects to firm up the evidence. Ultimately, however, the issue is about what is more important to us: To use a procedure with known long term error rates or to know the degree of support for our theory (the amount by which we should change our conviction in a theory)? If we want to know the degree of evidence or support for our theory, we are being irrational in relying on orthodox statistics. Conclusion I suggest that the arguments for Bayes are sufficiently compelling, even if not completely persuasive, that psychologists should be aware of the debates at the logical foundations of their statistics and make an informed choice between approaches for particular research questions. The choice is not just academic; it would profoundly affect what we actually do as researchers. References Armitage, P., Berry, G., & Matthews, J. N. S. (2002). Statistical methods in medical research. (4th Edition). Blackwell. Baguley, T. (2009). Standardized or simple effect size: What should be reported? British Journal of Psychology, 100, 603-617. 39 Baguley, T., & Kaye, W.S. (in press). Review of “Understanding psychology as a science: An introduction to scientific and statistical inference”. British Journal of Mathematical & Statistical Psychology. Berger, J. (2003). Could Fisher, Jeffreys and Neyman have agreed on testing? Statistical Science, 18, 1-32. Birnbaum, A. (1962). On the foundations of statistical inference (with discussion). Journal of the American Statistical Association, 53, 259-326. Blume, J. D. (in press). Likelihood and its Evidential Framework. In P. S. Bandyopadhyay & M. Forster (eds), Handbook of the Philosophy of Statistics. Elsevier. Cohen, J. (1977). Statistical power analysis for behavioral sciences. Academic Press. Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49(12), 9971003. Cox, R. T. (1946). Probability, frequency, and reasonable expectation. American Journal of Physics, 14, 1-13. Dienes, Z. (2008). Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference. Palgrave Macmillan 40 Website: http://www.lifesci.sussex.ac.uk/home/Zoltan_Dienes/inference/ Dienes, Z., Brown, E., Hutton, S., Kirsch, I. , Mazzoni, G., & Wright, D. B. (2009). Hypnotic suggestibility, cognitive inhibition, and dissociation. Consciousness & Cognition, 18 , 837-847. Fidler, F., Thomason, N., Cumming, G., Finch, S., & Leeman, J. (2004). Editors Can Lead Researchers to Confidence Intervals, but Can’t Make Them Think. Psychological Science, 15, 119-126. Gigerenzer, G. (2004). Mindless statistics. Journal of Socio-Economics, 33, 587-606. Harlow, L. L., Mulaik, S. A., Steiger, J. H.(Eds) (1997). What if there were no significance tests? Erlbaum. Hoijtink, H., Klugkist, I., & Boelen, P. A. (Eds) (2008). Bayesian evaluation of informative hypotheses. Springer. Howard, G. S., Maxwell, S. E., & Fleming, K. J. (2000). The Proof of the Pudding: An Illustration of the Relative Strengths of Null Hypothesis, Meta-Analysis, and Bayesian Analysis. Psychological Methods, 5 (3), 315-332. Howson, C., & Urbach, P. (2006). Scientific reasoning: The Bayesian approach. (Third edition.) Open Court. 41 Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Sage. Jaynes, E. T. (2003). Probability theory: The logic of science. Cambridge University Press. Jeffreys, H. (1961). The Theory of Probability. Third edition. Oxford University Press. Kerr, N. L. (1998). HARKing: Hypothesizing After the Results are Known. Personality and Social Psychology Review, 2, 196-217. Kirsch, I. (2009). The emperor’s new drugs. The Bodley Head. Kirsch, I., & Council, J. R. (1992). Situational and personality correlates of suggestibility. In E. Fromm & M. Nash (Eds.), Contemporary hypnosis research (pp. 267–291). New York: The Guilford Press. Lakatos. I. (1978). The methodology of scientific research programmes: Philosophical papers, vol 1. Cambridge University Press. Lakatos, I. and Feyerabend, P. (1999). For and against method. University of Chicago Press. 42 Mayo, D. (1996). Error and the growth of experimental knowledge. University of Chicago press. Mayo, D. G. (2004). An error statistical philosophy of evidence. In M. L. Taper& S. R. Lele, The nature of scientific evidence: Statistical, philosophical and empirical considerations. University of Chicago Press (pp 79 – 96). Meehl, P. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103-115. Miller, D. (1994). Critical rationalism: A restatement and defence. Open Court. Oakes, M. (1986). Statistical inference: A commentary for the social and behavioural sciences. Wiley. Popper, K. (1963). Conjectures and refutations. Routledge. Rouder, J. N., Morey, R. D., Speckman, P. L., & Pratte, M. S. (2007). Detecting chance: A solution to the null sensitivity problem in subliminal priming. Psychonomic Bulletin & Review, 14, 597-605. 43 Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t-Tests for Accepting and Rejecting the Null Hypothesis. Psychonomic Bulletin & Review. 16, 225-237. Royall, R. M. (1997). Statistical evidence: A likelihood paradigm. Chapmen & Hall. Sivia, D. S., & Skilling, J. (2006). Data analysis: A Bayesian tutorial, second edition. Oxford University Press. Taper, M. L., & Lele, S. R. (2004). The nature of scientific evidence: Statistical, philosophical and empirical considerations. University of Chicago Press. Ziliak, S. T., & McCloskey, D. N. (2008). The cult of statistical significance: How the standard error cost us jobs, justice and lives. The University of Michigan Press.

© Copyright 2019