Video Transcript: Testing Hypotheses with Data - Part 1
Welcome. In this section of the course, we are again going to be building off of all the things we have talked about previously by now actually testing hypotheses with our data. A hypothesis test uses data to help evaluate some kind of initial claim that you have about a parameter from the population, the best way, probably, to go through a hypothesis test in idea is to work through sort of an example. Let's say I have a fair coin, or at least I have a coin that you believe is fair to start to test if this coin is fair. Though you ask me to flip the coin repeatedly, and you start recording the results. Well, flip number one, I flip heads, and you're like, "Oh, well, I mean the probability of heads on a fair coin is point five, so all seems good so far. Flip two, I also flip heads, and you think to yourself, hmm, okay, what's the probability of you flipping two heads in a row? Well, out of the four possible outcomes that you can have from flipping a coin twice, heads, heads, heads, tails, tails, heads, tails, tails, only one of those 4.25 would give you heads back to back. This we remember learning back in our probability section. So, if I flip two heads in a row, do you still think the coin is fair? You might think, well, you know, anyone can flip a coin and get heads two times in a row. Okay, cool. So, you still think the coin is fair? So, flip number three. I also flip a heads. Hmm, well, the probability of me flipping three heads in a row if the coin is fair would only be 12.5%, so do you still think the coin is fair? You might still think the coin is fair, but you're starting to look at me a little bit funny now that I keep seem to be flipping heads, so I flip the coin again, surprise, surprise, it's heads again, now again. What are the chances? What's the probability of me actually flipping a fair coin four times in a row, and they all land on heads. Well, now there's only like a 6.25% chance of that happening. The probability is 0.0625 Now you're probably starting to question whether or not this coin is fair. I flipped the coin a fifth time again, getting heads. The probability of this happening is 3.125% or just a little over 3% chance of its happening. Now, do you think the coin is fair? At this point, you no longer believe the coin is actually fair. Well, what we've done is we've actually run a hypothesis test. We had some initial hypothesis, you thought the coin was fair to start, but just to make sure, you collected data, and in the process of collecting that data, your data started to lead you down a path where you no longer believed what you originally thought. Now, again, could a person with a fair coin flip five heads in a row? Yes, that could happen. In fact, we calculated the probability. There is a just over 3% chance of that actually happening, but that's such a low chance that you're probably starting to question whether that coin is fair. Welcome to the steps of hypothesis testing. That coin you thought to be fair, more formally, that's called your null hypothesis, your initial hypothesis, you use data, and you gathered all of that data to calculate some notion of how likely something is to happen, that's a test statistic. When you are calculating the probability of getting the data that you saw, that is actually what we call a p value, and then last but not least, using that test statistic in that p value, you
make some kind of decision on that original null hypothesis that you had. You go, I no longer believe that null hypothesis. That's the idea of hypothesis testing. Let's do another example, but with our bike data. According to the central limit theorem, sample means follow a normal distribution as long as the sample size is big enough, right? Okay. Well, you believe the average daily number of total users is 4000, but you want to know if there is more than that. So, again, initially walking in, you thought the average number of daily users was 4000 However, you collect a sample of 731 days. The average from that sample is 4504 with a standard deviation of 1937 so what is the probability that you see a sample just like that one a sample where you get 4504 people on average in that sample when the true average is actually 4000 Well, again we can look at that based on a normal distribution. If I have the center of the normal distribution at 4000 and I get a sample all the way in the right hand side, 4504 I could get some idea of how far away those two numbers are. Now, again, I have to use the standard deviation of the normal distribution to help me scale this, so we would have the standard deviation, but as we also remember, for when we were talking about z scores, when looking at averages, we divided that standard deviation by the square root of n, so we're going to take our standard deviation here from a sample, since I don't have the population, 1937 divided by the square root of 731 Well, that gives us essentially a standardized standard deviation of x bar, or more formally, a standard error of x bar. Remember that from the confidence intervals of 71.64 Huh? Well, again, what's the probability you see this sample of 4504 a sample of 731 days that average 4504 users when the truth is actually 4000 Well, let's see here, there's a difference here of 504 in our two averages. Well, again, how many standard deviations is that though we could take that difference and divide it by our standard error of 71.64 Wow, that's over seven standard deviations away from the mean. So, what is that probability? Really, really small, less than 0.0001 there's a less than 0.01% chance of this actually happening, and this makes sense, right? Remember what we learned with the normal distribution, most of our data is within three standard deviations of the mean, here I'm seven standard deviations away from the mean. The chances of that happening are really, really small. Now, could they happen? Yes, of course, it could. But is it probable? Not really. Again, what have we done? We have done a hypothesis test. So, again, you believe the average daily number of total users is 4000 but you want to know if there is more than that. You collect a sample of 731 days with an average daily number of total users at 4504 standard deviation of 1937. So, what's the probability? What's the probability you see that sample, if your initial thought was actually right. Well, if your initial thought was actually right, the chances of you seeing that original sample are very small, probability of 0.0001 even less than that. So, do you still believe your initial original hypothesis. Do you still believe the average daily number of total users is 4000 when the chances of you seeing the sample you did under that
belief is extremely small? Probably not. But again, what do we have here? We have a null hypothesis, an original thought. We have some test statistic, we collected some data, we then figured out the probability that we saw that data, we call that a p value. And then, based on all those things, we make some decision on our null hypothesis. Welcome to hypothesis testing. In fact, let's summarize. A hypothesis test uses data to help evaluate an initial claim about a parameter from the population. There are four main steps to hypothesis testing. First step, state your hypotheses. Second step. Calculate a test statistic. Basically, gather data. Third step, p value. What's the probability you see that data underneath those original hypotheses? And last but not least, based on that p value, based on that probability, make some kind of decision on that null hypothesis. so let's go ahead and let's jump into the first step of hypothesis testing. Let's talk about your null and your alternative hypothesis. So, again, hypothesis testing uses data to help evaluate an initial claim about some parameter from the population that initial claim is what we refer to as the null hypothesis. We denote that by a capital H followed by a zero. It's basically our tentative assumption. It's what we believe going in now. The opposite of that, the opposite of that initial belief is what we refer to as the alternative hypothesis. We denote this by a capital H with a little a beside it. This is again the opposite of the null hypothesis. So if our initial thought is the null, then what is opposite of our initial thought would be essentially the alternative, so the hypothesis testing procedure uses data from a sample to basically compete these two different hypotheses against each other. Now, I'll be honest with you, it's not always obvious how the null and the alternative hypotheses should be formulated. Developing these things can be difficult. The context of the situation is very, very important in determining how these hypotheses should be stated. In all honesty, in some cases it's easier to identify the alternative hypothesis first before the null. Why is that the case? Well, typically the alternative hypothesis is what we're trying to test. It's what we're trying to collect evidence for. You can think about it kind of like a court case. The null hypothesis would be innocent until proven guilty. Well, what is the prosecution collecting data for? The prosecution is collecting data not for innocence. The prosecution is collecting data for guilt. So, whatever guilty is, that would be the alternative hypothesis. So, we have some initial thought, innocence, whereas the alternative hypothesis would be what the prosecution is trying to collect information on, and that would be guilt. So, a lot of times it's easier to figure out guilt, it's easier to figure out what are you trying to actually collect data for, and then the opposite of that would be your initial belief. So, from our example, the average daily number of total users being 4000 would be our null hypothesis, it's the status quo, our original thought, our initial claim about the data, and again, Where does this come from? Sometimes you can sort of just think about it, it's something that's always been there. Sometimes it's previous data, you know. Maybe historically the average
has been 4000 but we're wondering to see if that average has changed. Maybe it's now higher than what it used to be. So, in our case, that would be the null hypothesis, and this null hypothesis is on the population parameter, it's on some grander number that we don't actually know, so again, when looking at this, we have some idea that there's some grand average number of users on a daily basis for our bike data, we can't see all of it, all we can see is just a sample of days, and so that being the case, that sample is going to help us test this claim, but it's not going to necessarily be perfect. Our null hypothesis is always on some kind of population parameter, so because it's on some kind of population parameter of interest, it is not on a sample statistic, you see parameters, they're unknown, so we can actually formulate a hypothesis about them, because we don't know if it's true or not. Statistics, on the other hand, they are known. I know exactly what my samples average is. My samples average is 4504 There is no argument there. I have a sample, I calculated the average. There's no need for a null hypothesis about this sample. The sample has a strict number that we know, the parameter, the thing that summarizes the whole population. We don't know that, and so, because we don't know that, that's what we're actually testing now. Essentially, again, this is truth until you can prove otherwise. The whole point of hypothesis testing is, once you formulate this idea of the null hypothesis, that is what you will assume to be true until you can collect enough data to prove otherwise. Again, think about it like a court case: it is innocent until proven guilty. Now, you always contain one of the following inside of a null hypothesis: either an equal sign, a greater than or equal sign, or a less than or equal sign, so in our example we're saying that the average is 4000 Now, what do we do with this null hypothesis? Well, by the end, by the time we're done with hypothesis testing, we'll either reject that null hypothesis, basically say guilty, or we will fail to reject the null hypothesis. Basically, say not guilty. Now we never say we accept the null hypothesis. Wait, why? Well, we are never collecting data for the null hypothesis. The data we collect is to try and prove the alternative. The question just comes, did you have enough data to disprove the null, guilty, or did you just not have enough data to disprove the null, not guilty? That's why, in a court case, you never hear them say this person is innocent, they always say guilty or not guilty, because either you collected enough evidence to prove guilt, or you did not collect enough evidence to prove guilt. Innocence was always assumed. That's the same idea here. This null hypothesis is always assumed until you can prove otherwise. So, let's think about the opposite of this, the alternative hypothesis. So, in our case, for example, the average daily number of total users is greater than 4000 We have some initial thought. This initial thought is, hey, you know, we used to think that the average daily users was 4000 but now I really think the average daily users is over 4000. Ah, so this is what we're trying to prove. This is what we typically gather data for. So I'm running this hypothesis test because I really think our average is higher now
than what it used to be. So now I'm trying to prove this alternative, the alternative hypothesis typically has the opposite sign of the null, but again it still contains a population parameter. So the null hypothesis was, I think, that essentially the average is 4000 The alternative is, I think, the average, again still mu, is greater than 4000 Now, again, we can never say we prove this idea. We're trying to gather evidence for it, but that isn't the claim that's on trial. The claim that's on trial is guilty, or I'm sorry, the claim that's on trial is innocence. So, again, we assume innocence here with the alternative, the alternative is guilty. So we say we have enough data to reject that innocence idea. We no longer think they're innocent. We haven't proven guilt. We just have enough evidence to no longer think innocence again. I know it's a little bit confusing, but the idea being that you're never 100% sure that someone is guilty, you're never 100% sure here of your alternative hypothesis, you just have a lot of evidence to support the claim. Now, when you look at a null and you look at an alternative hypothesis, again, they typically are opposites of each other. So, for example, we have things like one-sided tests, where we think the null is equal to or below a number. Then the alternative would be that the null - I'm sorry - that the population parameter is above that number. So you can think of mu zero here on this screen, mu with a little zero beside it as the number itself. So, for example, here mu zero would be 4000 but essentially you can say something like, well, I think that the average is equal to or below a number. Well, then the alternative would be that the average is above that number. Of course, the opposite could be tested as well. Maybe you think the average is equal to or greater than a number. Well, then your alternative would be that the average is less than a number. Of course, you may not know whether it's greater than or less than. So, for example, it may be that I think the average is equal to a. Spec number, but the alternative could be that maybe it's bigger than that number, or maybe it's smaller than that number. I don't really know. So this is going to change a little bit on how we work out some of the later steps, but I don't want you to think that we always have to look in one direction, where we always say, well, it has to be greater than or it has to be less than, in all honesty, you may not know. You can say, well, I think the average was 4000 but maybe it's gone higher, maybe it's gone lower. I don't know. Let's see, that's a two-sided test. A one-sided test was, I think the average was 4000 before, now I think it's higher. I'm only looking one direction, all right. Let's summarize. So, the null hypothesis, again denoted by a capital H with a little zero beside it, is basically your initial claim, some kind of tentative assumption about the population, and some kind of parameter summarizing that population. The alternative hypothesis, on the other hand, denoted by capital H with a little a beside it, that's the opposite of what is stated in the null hypothesis. Now, a one-sided test happens when the alternative hypothesis only points in one direction, I think it's greater than, or I think it's less than. Well, a two-sided hypothesis test allows both directions to be
possible. I don't know if it's greater than or less than, but I just think it's different than what it was. That's a two-sided test. So, hopefully this lecture has laid the foundation a little bit on sort of what is hypothesis testing. Again, just go back to that coin flip example, but then we started talking about the idea of the very first and most important step of any hypothesis test, that's defining your hypotheses. What do you believe? What are you trying to test? You have to know that before you start looking at your data. But for now, that is the end of this lecture. I look forward to seeing you in the next one.