Bayesian Repeated Measures ANOVA: What n do you use?

Bayesian statistics are interesting as they give credence to the idea that stats can be interpretable and statistically meaningful without having to do post-hoc corrections and analyses. Two papers (and one review), one from Masson (2011) and one from Wagenmakers (2007) (with a nice review of the current debate by Jarosz & Wiley (2014)), discuss how to compute Bayesian Factors using sums of squares, which were computed through a traditional repeated measures analysis of variance. Three parts of computing a Bayesian Factor are: 1) all of your sums of squares, 2) your k degrees of freedom (for me, this is the amount of different conditions), and 3) your n number of participants/subjects (I study humans, so I use the word "participants").

Wagenmakers makes the argument for simply  using your n value in the Bayesian rmANOVA. Masson believes that you need to account for each participant in every free condition (aka n X - 1, where k is the amount of conditions in your experiment), which increases your n value.

Now, there is no set rule as to how to compute a Bayesian rmANOVA. So I tried both methods using the values from Masson's paper example 1. Here's a screenshot:

Capture

Using Masson's n, I end up with something similar to his result: BIC = -18.5; BF = 0.000096 (Masson reports BIC = -14.1, BF = 0.00086). When I convert this into probability, I get p(Ha) > 0.99 (Masson reports the same probability).

Using Wagenmakers' n, I get BIC = -7.75; BF = 0.021; p(Ha) > 0.98

In this example, the probabilities are so near each other, this particular example doesn't really argue for or against either Masson's or Wagenmakers' n value.

So I tried doing this with some of my own data. I have 30 participants going through 2 tasks divided at 2 different time epochs. If I am doing this right, k - 1 = 3. For reference: F = 10.27, p < .005, partial η^2 = 0.268, SSwithin = 0.00000082, SSbetween = 0.0012, SSresidual = 0.0033 (making SStotal = 0.0046).

Using Masson's n: BIC = -19.12; BF = 0.000071; p(Ha) > 0.99

Using Wagenmaker's n: BIC = -2.57; BF = 0.28; p(Ha) > 0.78

This is definitely different. One of the advantages to Bayesian statistics is the ability to give a more "straight-forward" report of a situation. In my situation, I'm asking are there group differences? and my Bayesian factors are both strong in support of my hypothesis. However, Masson's n indicates the probability of group differences given this experiment are practically 100%, whereas Wagenmaker's n says group differences given this experiment are more like 78%.

Still, both percentages are very high -- especially for fNIRS data. But this is all driven by whether you consider your participants to be in every condition separately (Masson's n) or if you consider your participants to be simply participants (Wagenmaker's n).

Since my traditional statistical value is significant, and my partial η^2 is a moderate-ish 0.27, I should be expecting a strong but moderate Bayes Factor and probability. Since my Wagenmaker's BF = 0.28 (making my inverse BF = 3.61), and since my probability for my alternative hypothesis in this experiment 78%, and since my interpreted support for my alternative hypothesis is "positive" and "substantial" (based on Jarosz & Wiley's interpretation table), I am more comfortable calculating the more conservative of the two n's, the Wagenmaker's n.

On the other hand, the interpretation for Masson's n would be an inverse BF > 100000, with "very strong" and "decisive" support for my alternative hypothesis. Based on my traditional effect size, I can't really see the leap from a moderate-at-best interpretation to a guaranteed-or-your-money-back interpretation.

Really, either way has some strong logic behind it. I personally side on the side of caution when trying to discuss my results. I don't know how kosher it would be to use both of these numbers as a range (e.g. my likelihood ranges between 78% and 99%), but that could be a fair way to incorporate both computations. Again, I'm not a stats expert, nor a Bayesian expert, so I don't really know. I would suggest reading Masson's paper, Wagenmaker's paper, and the review from Jarosz & Wiley if you want a stronger grasp on Bayes Factor computation! Good easy reads!

Leave a Reply