ANOVA Application

One-Way Analysis of Variance (ANOVA)

Introduction

In this lesson, we will be examining one-way analysis of variance (ANOVA). You’ve learned that t-tests compare two group mean in light of group distributions to help us understand if they are significantly different. But many variables contain more than two groups. ANOVA gives us a way to compare group means (similar to the independent samples t-test), but with more groups, and more flexibility.

The downside of this increase in flexibility is that ANOVA is slightly more complicated. Specifically, ANOVA requires an extra step of interpretation for significant results. In the tabs to the right, we will work through an ANOVA example to help you better understand what is required, and how to conduct an ANOVA.

Visualization

As with most types of analysis, it helps to better understand your data before you begin with statistical tests. Not only to visualizations show us relationships, but they can save us from making interpretive errors later in the analysis.

Perhaps the most useful type of visualization for group mean comparisons is a boxplot. You can use the R-base boxplot() function or the geom_boxplot() in ggplot2. For this example, I’m going to be using the National Youth and Family Study (NYFS) data from 1974. The code will be visible in the Rmarkdown file, but I will hide in the dashboard when knitted.

I’m going to focus on minor offending and drug use across these examples, and so am visualizing those comparisons below.

Visualization 1 - Boxplot

Examining the first plot, we can see that most minor offending is near 0. The red line represents the overall mean of offending across all groups. I’m going to eliminate outliers in the next chart by “slicing” the data to only include minor offending values under 50.

Visualization 2 - Boxplot using GGplot2

From the above, we can see that even with much of the “extreme” data taken out of the dataset, the means (as shown by the thick black line within each box) are essentially 0. This gives us a reasonable clue that there’s not going to be much difference between the two groups.

Alternative Example

By way of counter-example, I’ve provided a graph that seems to show large group differences, and though it’s not necessarily relevant to the current question, we can use it for demonstration purposes.

Religious Attendance

In the above chart, we can see that there are a wide range of values for people who never attend religious services, but that for those who attend at all, the average value decreases with higher religious attendance.

ANOVA Assumptions

Like all statistical tests, ANOVA has a set of assumptions that need to be satisfied before we use it.

  1. The responses for each factor level have a normal population distribution.
  2. These distributions have the same variances.
  3. The data are independent.

In reality, we often don’t know if 1 and 2 are satisfied, and we consider these not to be strict assumptions. In terms of the 3rd, however, we need to be quite strict. This means that an individual cannot be a member of more than one group. If that’s violated, you have some serious problems and should not trust the results you get.

It is worth mentioning that there is a simple “rule of thumb” test for the quality of variances assumption. If you look at your standard deviations within groups, compare the smallest and largest, and the ratio between them falls below 2, you can assume equal variances.

If you look at the code in the RMarkdown file (I didn’t include it in the output), you’ll see that we violate the assumption with this data. But, eh. What are you gonna do? As I said, violations to this assumption are not considered serious.

Types of Variables

Remember, like with t-tests, you must have a categorical (in R-language, a factor) variable as your X and your Y must be continuous. This means you must double-check your data types before actually executing the ANOVA test.

ANOVA Test

As you can see in the visualization tab, it doesn’t look like there are many differences between the group means. But, like all good scientists, we should test it. We can’t use a t-test because we have more than one group, so we’ll have to use ANOVA.

However, before we conduct the test, as always, we need to set up our hypotheses. Fortunately, as ANOVA is an extension of t-tests, the hypotheses we generate are largely similar, with the exception of multiple groups.

H0: \(\mu_w = \mu_a = \mu_h = \mu_n\) is the null hypothesis

H1: \(\bar{X}_w \neq \bar{X}_a \neq \bar{X}_h \neq \bar{X}_n\) is the research hypothesis where \(w =\) White non-Hispanic, \(a =\) African American, \(h =\) Hispanic, and \(n =\) Native American.

[Note that you can technically have any number of groups and the research/null hypotheses will be basically the same. However, we only use ANOVA up to about 9 groups for reasons that will be clear when we examine regression.]

model<-aov(nyfs$minor~nyfs$ethnic) # this produces the model
summary(model) # this shows the summary of the model
##              Df Sum Sq Mean Sq F value Pr(>F)
## nyfs$ethnic   3    516     172   0.153  0.928
## Residuals   574 644493    1123

Fortunately, as shown above, the code for ANOVA is straightforward. aov() is the function, and we use the standard formula notation (formulas use the ~ sign). Worth noting is that we put the dependent variable first in that notation style.

The model summary gives us the relevant information. In particular, we are interested in the Pr(>F) value, as well as the F value. The first, Pr(>F) is equivalent to a p-value within a t-test. This means that if the value is \(<\) .05, we must reject the null hypothesis. Usually, like with t-tests, Pr(>F) values will be lower with higher values of F, often acheiving significane when F is around 2. We can see here that \(F = .153\), so it is unlikely that \(Pr(>F) < .05\). And indeed, when we check the value, we can see it is .928, much higher than we can reject the null with.

Thus, we have to conclude that, in terms of minor offending, there is no difference between ethnicities (within this dataset).

Alternative Example

There’s a question that may naturally arise for you, however. What if we did get a p-value of < .05? We would know that there was a difference between groups, but we wouldn’t know which groups the difference was between!

How can we deal with this? Well, the first thing to remember is F is what we call an omnibus test. It tests all of the differences at once. So, if the p-value (Pr(>F)) associated with F is > .05, it means that there are no significant differences between any groups. In other words, our null hypothesis is true.

Let’s go with an example where there is a difference, however. I’ll use our counter example from the first section - alcohol consumption and religous attendance.

##              Df  Sum Sq Mean Sq F value   Pr(>F)    
## nyfs$relatt   4  121123   30281   5.821 0.000135 ***
## Residuals   573 2980789    5202                     
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

We can see here that \(p < .05\), and therefore there is at least one difference between groups, and therefore we can reject the null hypothesis. The question is which group(s) are different from which?

In order to find that out, we need to use what’s called a post-hoc test.

Post-hoc Tests

Before we go into the tests, it’s essential to know that if your p-value is not < .05 you do not do the post-hoc tests. If we can’t trust our answers (which is what a p-value > .5 tells us), then we don’t bother with interpreting anything else. If however, as in our alternative example above, we do have \(p < .05\), we need to understand where that difference is. Post-hoc tests tell us. We will examine 2 post-hoc tests. The first is TukeyHSD (Tukey’s Honestly Significant Difference), which should be used when you have equal variances. The second is the Games-Howell test, which should be used when you do not have equal variances. You should only ever use a single post-hoc test. Do not try multiple tests to “see which is better.” This is called p-hacking and it is unethical.

tuk_model2<-TukeyHSD(model2)
print(tuk_model2)
##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = nyfs$alcohol ~ nyfs$relatt)
## 
## $`nyfs$relatt`
##                                                  diff       lwr         upr
## several times a year-never                  -8.012831 -31.72571  15.7000460
## once or twice a month-never                -17.752408 -43.95858   8.4537691
## once a week-never                          -31.525340 -56.64369  -6.4069901
## several times/week-never                   -48.215701 -82.34262 -14.0887804
## once or twice a month-several times a year  -9.739577 -33.76930  14.2901440
## once a week-several times a year           -23.512509 -46.35097  -0.6740485
## several times/week-several times a year    -40.202870 -72.68841  -7.7173273
## once a week-once or twice a month          -13.772932 -39.19061  11.6447476
## several times/week-once or twice a month   -30.463293 -64.81113   3.8845401
## several times/week-once a week             -16.690361 -50.21560  16.8348747
##                                                p adj
## several times a year-never                 0.8873298
## once or twice a month-never                0.3438251
## once a week-never                          0.0057260
## several times/week-never                   0.0011609
## once or twice a month-several times a year 0.8017479
## once a week-several times a year           0.0399975
## several times/week-several times a year    0.0067564
## once a week-once or twice a month          0.5741081
## several times/week-once or twice a month   0.1096079
## several times/week-once a week             0.6520775

The above table, while initially hard to read, shows the pairwise comparisons between each group mean. While the first 3 columns are informative, it’s the last column that gives us the information we’re looking for. The p adj (adjusted p-value) column tells us which differences are significant. As in most cases, we are looking for \(p < .05\).

Examining that 4th column, we can see that the “once a week - never”1 comparison is significant, as is the “several times/week - never” comparison. “Several times a year - once a week” is also significant, as is “several times/week - several times a year.” To determine which mean is larger, we look back at column 1 for the direction of the difference. We can see that people who attend religious services one a week drink less alcohol than those who never attend, those who attend multiple times a week drink less alcohol than those who never attend, those who attend several times a year drink more than those who attend once a week, and those who attend several times a week drink less than those who attend several times a year.

This paints a (relatively) straightforward picture of how religious attendance affects drinking, and we can address our null hypotheses (which would be that all groups were equal), and reject it.

As with many things, it helps to plot the relationships.

The above is a plot of the confidence intervals of the difference between means of each comparison. We get these from the TukeyHSD test we ran above (check out the code for specifics on how to plot it). If the confidence interval contains 0, we would no conclude that it’s different (at least we can’t be sure - think of it like the inverse of the \(p < .05\)). The significant comparisons from the TukeyHSD post-hoc test (where the differences lie) are also clearly captured in the plot.

Games-Howell Test

The Games-Howell post-hoc test does essentially what Tukey’s HSD does in the above section. However, Games-Howell has fewer assumptions than Tukey’s HSD, and so is more appropriate when you don’t meet the equal variances assumption. Often, you will get very similar results to TukeyHSD with Games-Howell. However, in the case where there are values that are close to significance, we do see differences.

In terms of the current example, we see that with Games-Howell, the only significant difference in alcohol consumption is between those who attend once a week and those who never attend.

Effects Size

Effects sizes within ANOVA are pretty easy to calculate. The formula for \(\eta^{2}\) (pronounced eta-squared) is:

\[ \eta^{2} = \frac{\textrm{Between-groups sums of squares}}{\textrm{Total sums of squares}}\]

Hand calculating this is straightforward with the information provided in the ANOVA table - if you know how to access the right parts of the object. R, unfortunately, does not make these very easy to access, so we can use a library to help us out.2

##                 eta.sq eta.sq.part
## nyfs$relatt 0.03904775  0.03904775

Here we can see that the \(\eta^{2}\) value is only .04. This can be interpreted to mean that we are explaining about 4% of the variance in our dependent variable (alcohol use) with our independent variable (religious attendance). This, in turn, suggests that while the differences between groups is real, it is not strong at all.

Reporting ANOVA Findings

Reporting APA-style findings with ANOVA is pretty straightforward. You need to explain what you found between your groups, as well as the statistics and any visualizations to support it. If your ANOVA was insignificant, you still report your general statistics (F, p), but you don’t worry about the post-hoc tests.

Report of no significant differences

The analysis of variances (ANOVA) showed that the effects of ethnicity on minor offending were insignificant, (\(F(3, 574) = 0.153\), \(p > .05\)). This indicates that the levels of minor offending were statistically equivalent across all 4 groups. It further suggests that we cannot reject the null hypothesis.3

Report on significant differences

The analysis of variances (ANOVA) showed significant differences among alcohol consumption dependent on the level of religious service attendance, \(F(4, 573) = 3.25\), \(p < .05\).4 Post hoc analyses were conducted using Tukey’s HSD post-hoc test. Based on Tukey’s value of \(HSD = -31.525, p < .05\), one-a-week religious service-goers used significant less alcohol than those who never attended. Similarly, those who attended several times a week drank significantly less than those who never attended (\(HSD = -48.216, p < .05\)) and those who attended several times a year drank more than those who attended once a week (\(HSD = 23.513, p < .05\)). Finally, those who attended several times a week drank significantly less on average than those who only attended services several times a year (\(HSD = -40.203, p < .05\)). This indicates, in general, that while not totally consistent, increased church attendance is associated with decreased alcohol consumption.5

However, even given these differences, with an \(\eta^{2}\) value of only .04, the relationship between religious attendance and alcohol consumption is quite weak.

Conclusion

One one hand, One-Way ANOVA is essentially just a generalized t-test. On the other, there are some complications with its use that require us to be careful with how we conduct the test (and post-hoc tests). In total, however, ANOVA gives us another powerful tool for detecting differences in group means. It also, fundamentally, serves as the basis for most of the omnibus tests we will be using in other techniques like regression - which is the workhorse of our field. But before we get there, we need to talk about Two-Way (aka Factorial Design) ANOVA. I’ll address that in another file.

  1. The dash can be read as “compared to.”↩

  2. See the RMarkdown code for the library.↩

  3. Notice we also do not report effects size, as it’s irrelevant if we can’t say the relationship/difference is real.↩

  4. You can also give a specific p-value like \(p = .012\)↩

  5. The reality is that this can get quite a bit more complicated. Here, we’re using a straightforward indicator of relgious attendance, measured at the ordinal level. With nominal groups, it can sometimes get confusing to interpret what, exactly, your findings mean. That’s part of what makes research difficult - and often interesting!↩

Comments

Post a Comment

Popular posts from this blog

ANOVA Theory

Image
ANOVA Theory ANOVA Theory Dr. Hill Why ANOVA? ANOVA in Principle Easy Conceptual Example Degrees of Freedom Conclusion Why ANOVA? Imagine a situation in which you want to compare men and women on some continuous variable - say, height. You have access to a reasonable-sized group of people (say, 1000), with roughly equal numbers of men and women. To test this, you would simply measure everyone’s height, record it along with whether they are a man or woman (I’m assuming two genders for the sake of ease, here), and use a t-test to compare the means of the two groups. Basically this: \(\mu_1 = \mu_2\) But what happens when you have three groups? Couldn’t we just do a set of t-tests? Let’s take an example: Now we have a group of people who score “high”, “medium”, and “low” on an agressiveness scale, and we want to see if that affects whether the speed while driving. We have access, as in the above example,...
Read more

Z-Tests

Image
Z Tests Z Tests Testing Hypotheses So, we’ve learned about what a hypothesis is and why we use null hypotheses. However, we haven’t really used hypotheses for anything - we haven’t conducted any tests. But the reason we set things up this way is because you need to know how to set up the test we want - and what we do in social science is really all about testing! But! We’re not limited to social sciences. Understanding whether something is actually different is really important in practical contexts as well. For instance let’s say we have a measure of the severity of crime within individuals ranging from 1-10. Let’s further assume that this crime severity is normally distributed in the population (e.g. lots of people do midling crimes, very few do really heavy duty crime or no crime at all). We implement a program that is supposed to reduce the severity of crime committed by a sample of 100 people. Did it work? ...
Read more

Factorial ANOVA

Image
Factorial ANOVA Factorial ANOVA Adding Variables to ANOVA - Factorial ANOVA While One-way Analysis of Variance is a powerful tool, it’s rare within social science (and practical matters) that we’re interested only in the dfferences across a single dimension (e.g. differences in only race). More often, we have multiple dimensions that we’re interested in testing, and frequently, we’re interested not just in what we would term their direct effects, but also how those variables interact . The good news is that ANOVA is very flexible, and can handle multiple variables. While it can (technically) handle all kinds of levels of measurement, the version of ANOVA we’re going to work with in this lesson is Factorial ANOVA. That is, ANOVA by more than one factor . This both helps to simplify the explanation (if we used interval/ratio level data, we’d be moving into something caled ANCOVA), but it also will help us understa...
Read more