Z Tests

Testing Hypotheses

So, we’ve learned about what a hypothesis is and why we use null hypotheses. However, we haven’t really used hypotheses for anything - we haven’t conducted any tests. But the reason we set things up this way is because you need to know how to set up the test we want - and what we do in social science is really all about testing!

But! We’re not limited to social sciences. Understanding whether something is actually different is really important in practical contexts as well. For instance let’s say we have a measure of the severity of crime within individuals ranging from 1-10. Let’s further assume that this crime severity is normally distributed in the population (e.g. lots of people do midling crimes, very few do really heavy duty crime or no crime at all). We implement a program that is supposed to reduce the severity of crime committed by a sample of 100 people. Did it work?

Experiment 1

In order to know, we first have to know what we are testing - our hypothesis! In this case we can set it up as directional or non-directional. We’ll do the non-directional hypothesis for simplicity’s sake:

H₀: \(\mu_p = \mu_s\) is the null hypothesis

H₁: \(\mu_p \neq \mu_s\) is the research hypothesis

In other words, if our program has no effect, than the mean of our sample \(\mu_s\) and the mean of our population \(\mu_p\) should not differ - they should be equal. If, on the other hand, our program did have an effect (our hypothosis is that it did), we should see them be unequal.

So now we know what we need to test! But how do we know if they’re unequal? Well, we could do some simple subtraction! \(\mu_p - \mu_s\) would tell us if there’s a difference. But let’s say we have a mean of 5 for the population and a mean of 4.5 for the sample. From the subtraction approach we get \(5-4.5=.5\), so there is a difference. But, is this a real difference? \(.5\) is a pretty small number. Maybe, if we had chosen a different sample, we could have gotten a larger difference? Or a smaller one? It’s hard to say.

So, one way we can solve this knowledge problem is by assuming a normal distribution (remember that?) in the population. Because we know the probability, based on the features of a normal distribution, of where any value will lie, we can use the population mean and standard deviation to figure out if the difference between our sample and our population means is real. In order to do this, we have to follow a couple of steps.

1. Define our null hypothesis and research hypothesis.
2. Define \(\alpha\) (alpha) (the value at which we can say definitively this is not by chance - almost always .05 for us)
3. State decision rule (based on \(\alpha\) - basically you split \(\alpha\) in half and, for non, directional hypotheses, put the 2.5% on each tail of the normal distribution)
4. Define the critical value (this is the z-score asscoiated with the \(\alpha\) - basicaly the number that tells you it’s above or below 95% of the cases and therefore statistically significant)
5. Calculate the z-test
6. State results & Conclusion

We’ve already got our hypotheses above, so what about \(\alpha\)? One way (not technically correct, but works for our purposes) to think of alpha is as the risk we’re willing to take that we’re wrong. In other words, we’re willing to take a \(5%\) chance that we’re wrong if we set \(\alpha\) to \(.05\). We will virtually never choose a different \(\alpha\). \(5%\) chance that we’re wrong is the highest risk we’re willing to take.

(\(\alpha\) is more tecnically the chances out of 100 (or percentage) that we got our answer by statistical chance alone. An \(\alpha\) of .05 implies that we’re willing to take risk that our answer was just pure “luck” 5% of the time.)

Our decision rule is pretty straightforward. Because \(\alpha\) is .05, we need to know where we can be 95% sure that whatever answer we get is truly, statistically different. If you think about a normal distribution:

You know that there are two areas (to the right and left of each of the red lines in the above figure) that represent 2.5% of the population’s potential values (each) because we know that within 2 standard deviations of the mean (\(\mu = 0\) above) is 95% of all the cases. So, if a value falls below the line on the left, or above the line on the right, that it is really different.

The critical value changes dependent upon \(\alpha\) and the number of cases. Fortunately, we don’t have to worry about calcualting these by hand, or even looking them up in a table. R takes care of that. What we have to worry about is calculating the test statistic, which is also quite easy.

In the code chunk below, I’ll finish out the test we’re interested in above. Remember we had a mean of 5 and let’s say we have a standard deviation of 1 for crime severity in the population. The group that got the treatment has a mean of 4.5 and a standard deviation of .5. With those values, we can calculate the test.

## 
##  One-sample z-Test
## 
## data:  treatment
## z = -5.6458, p-value = 1.644e-08
## alternative hypothesis: true mean is not equal to 5
## 95 percent confidence interval:
##  4.239421 4.631414
## sample estimates:
## mean of x 
##  4.435418

Our calucalted test statistic is \(z = -5.6458\). The direction tells us our sample mean is less than our population mean. Even though .5 is a very small difference, we can see that it is statistically significantly different from the population by checking to see our p-value. P-values are very important, and we calculate them for nearly every statistical test we run. They represent whether or not we exceeded \(\alpha\) and are calculated using the test statistic. We look for p-values to be less than \(\alpha\) (.05) in order to reject the null. In this case, with a p-value of 1.644e-08 (\(p = .00000000164\)), we can see that it is much less than alpha. This means we reject the null hypothesis and accept the research (aka alternative) hypothesis that there was a difference between the population and sample. And we can be very confident in that answer.

It’s also worth visualizing the difference to see what it looks like!

In some ways, given those distributions, it’s not surprising to see that it’s a real difference!

Experiment 2

But hold on, let’s say we only had 10 people in our treatment group, what might happen? Let’s run an experiment to find out! We will keep everything the same as the above (incluing our hypotheses), but reduce the number of individuals in our sample to 10.

## 
##  One-sample z-Test
## 
## data:  treatment2
## z = -1.6142, p-value = 0.1065
## alternative hypothesis: true mean is not equal to 5
## 95 percent confidence interval:
##  3.869751 5.109341
## sample estimates:
## mean of x 
##  4.489546

It’s no longer significant! What happened? Well, we didn’t have enough information to know that the difference was real because we didn’t have enough cases in our sample. As our intuition suggests, when we have more information, we can be more certain of a particular outcome. This is true with statistics as well! When we have more cases in our sample, we increase our statistical power, which is our ability to detect real differences (or relationships) from our population.

Let’s see what it looks like when we chart it with fewer datapoints!

It’s easy to see why we can’t be certain about that difference anymore as there isn’t really a discernable sample distribution anymore!

Effects Size

But knowing that there is a real difference is not the same as knowing if it matters. In reality, maybe a small difference on something like this does matter. But often, we want to know exactly how large an effect we had. To do this, for one-sample z-tests, we can caculate Cohen’s d.

\[d = \frac{\bar{X} - \mu}{\sigma}\]

So, for the case above(using the sample with 100), we get:

\[d = \frac{4.5 - 5}{1} = .5\]

This is actually at the high-end of a moderate effects size! So, in this case, the relationship is real (which is what the p-value tell us) and moderate (which is what the effects size tells us).

So What?

Z-tests are basically the paradigm for all statistical tests of difference. The steps you followed here are the ones you generally follow for any test and form the basis of inference. Formulating a hypothesis and testing it appropriately is so much of what we do, it’s hard to overstate the importance of getting this part right.

But often we don’t know the population parameters we need to calculate a z-test. In fact, we rarely know the population parameters at all! What can we do in those cases? Well, we can use something that works basically the same way as a z-test, but with a slightly different distribution that takes into account the fact that we don’t have all the values in the population. It’s called a t-test.

Helpful Video