Video Transcript: Testing Hypotheses with Data - Part 3
Welcome. Let's finish up this section of the course on testing hypotheses with data by looking at an actual example of a hypothesis test for means. So, let's go with our bike data example. So, you believe the average daily number of total users is 4000 but you want to know if there's more than that, so you can decide on orders for future bikes to be added, so you collect a sample of 731 days with an average daily number of total users at 4504 and a standard deviation of 1937 So, with a significant level of 5% basically anything more rare than 5% you no longer believe it. Conduct a hypothesis test on this claim. Awesome. Well, let's go through our steps of hypothesis testing. Step one: null hypothesis and alternative hypothesis. So, null hypothesis 4000 and alternative hypothesis greater than 4000 Well, why greater than? Let's take a look. You want to know if there is more than 4000 so you can decide on orders for future bikes to be added. Essentially, you're collecting data for more than 4000 so that's your alternative. The null should be the opposite of that, so you could put equal to 4000 if you'd like, or in all honesty, less than or equal to 4000 Anything less than or equal to 4000 would make you have the same choice. It's not bigger than 4000 so we don't need to order any more bikes. So our null is less than or equal to 4000 The alternative, what we're trying to test is more than 4000 on average. So, let's collect our sample data. Awesome, our sample data is that our sample mean is 4504 a standard deviation of 1937 and a sample size of 731 so underneath this sample, how far away is this sample from our original thought? So we can take our sample mean x bar 4504 subtract off our original thought 4000 and divide by the standard error that would be 1937 over the square root of our sample size 731 this would give us sample data test statistic of 7.03 so now the question is What's the probability of getting a test statistic that big? Well, again, there are a variety of different ways you can do this. A lot of calculators have this built in, computers definitely have this built in, and we can use just a t table if we like as well. I'm going to show you the idea on a t table, because that's all we really need to look at, so when looking on this t table here, what we need to do is identify which distribution we have, how many degrees of freedom we have. This is how we use a t table for hypothesis testing, so that far left hand column is still degrees of freedom, all right. Well, degrees of freedom. Why degrees of freedom? We're using a t distribution here, we're looking at means. Okay, so for our degrees of freedom, it is always our sample size, 731 minus one. Oh, so our degrees of freedom is 730 Perfect. So we just have to find the 730 row in our table. Oh, oh, well, unfortunately, there's no 730 row in our table. However, what we do in that scenario is we just look at the lower number, so we'll just round down to 500 Okay, so by lowering down to 500 is our degrees of freedom, then we can see that the biggest number on the 500 row in our table is 3.31 but wait a minute, our number is 7.03. 0h boy, that means our data is so big, we have so much evidence, it doesn't even fit on this table, but let's take a look. What column is this 3.31 under? Well, if we go to the very, very
top, you can see there are two rows, a one tail and a two tail row. The one tail would be the p value for a one tail hypothesis test. The two tail row would be the p value for a two tail hypothesis test. So we're going to look at the one tail row, because we have a one sided hypothesis test. So this is a p value of 0.0005 And we know this p value isn't even small enough. Take a look at the very, very top row, as our evidence gets smaller, as the test statistic number gets smaller on the 500 row. The p value on the very top row, the probability gets bigger and bigger and bigger, which again should make sense. The closer your data is to your original thought. The probability of it actually happening is going to keep going up, and so the further we move to the right in the table, the more evidence we have, the smaller and smaller the probability is. So our probability in this table is so small it's not even on this table, so we could say something along the lines of the chances of this happening are less than 0.0005, or in other words, there's a less than 0.05% chance of you getting this sample under the assumption of the null hypothesis. Let me show you visually. So, again, here is our shaded area where we would say, hey, if our sample is this far away, we no longer believe the null hypothesis. Our sample is seven standard deviations away from the mean, which is so far away it has a probability of happening of less than 0.0005, so because our p value 0.0005 is so much smaller than our significance level alpha of 0.05 We no longer believe the null hypothesis. We reject the null hypothesis. That initial thought of 4000 bikes or less on average. Nope, not even close. We have more than 4000 average users in a day, which is great for our company. That just means we need to order some more bikes. So, hopefully this example helps you see the idea and the power of hypothesis testing. But that's a one-sided test. Let me show you the same example, but with a two-sided test, so you believe the average daily number of total users is 4000 but you want to know if there is more, so you can decide on orders for future bikes to be added, or if there are less than 4000 so you can actually pull stock from the streets, so the bikes don't sit unused, so here this is a two-sided test. I want to know if it's bigger than 4000 or if it's smaller than 4000 Either one of those possibilities is available. Again, you collect the same sample, 731 days, same average number of users, same standard deviation. In fact, we have the same significance level. The only difference here is our original step, the null hypothesis is that mu is equal to 4000 whereas the alternative is that mu is not equal to 4000 We're basically just saying I don't care if it's higher than 4000 or lower than 4000 I need to see if either of those are a possibility. So you collect your data, you go through that whole process. It's the same idea, it's the same calculation. The sample means the same, the standard deviation was the same, the sample size was the same. We still get the same test statistic, 7.03 Where this comes in is in the p value. Let's take a look again. We would look at the same row that we looked at before, ideally 730 We don't have that in our table, so I'm gonna look at the 500 row again. If you were to use a calculator or Excel
or any other kind of software, you could look up the exact number. We're just trying to get an approximation. So I still look at the 500 row. I still try and find 7.03 somewhere in that 500 row. I don't see 7.03 anywhere. My numbers aren't even big enough. If it was 2, I could say it was in between 1.965 and 2.334 I could say it's in that range, but it's 7, and so 7 is not even on this table. So, what do I do now? Well, instead of looking at a one tailed test, we can look at the p value for a two tail test. Notice the relationship between those top two rows. What do you see? The two-tailed test is twice the probability of the one-tailed test. That kind of makes sense, doesn't it? We're saying it could be higher or lower. Your sample can't be both higher and lower, so it's going to be one of them, but it could have been the other one as well. So we essentially multiply our probability by two. See, a two-sided test has a p value that's twice that of a one-sided test. Think about it this way. Well, we calculated the probability of u being a 7.03 but it could have been a negative 7.03 Why? Because we are allowing there to be a chance that it could have been smaller too. You don't remember when doing this null hypothesis ahead of time, we do this before we even collect our data, so beforehand we don't know if it's going to be higher or lower, which means when we test things, we have to give that possibility a chance as well. So, yes, we have our data that says it was higher, but we also thought it could be lower, so we're going to take the probability that it was higher, multiply it by two, and get the probability it's also lower, as well as higher. So we have that 0.005 in both tails, which sums up to a probability of 0.001 Now again, when compared to our significance level of 0.05 this is still very rare. I don't want to make it seem like, oh, well, now we aren't sure. No, it's still extremely rare, and so, because of that, we still reject our original null hypothesis. We no longer believe that we have 4000 average users in a day, which, for our company, since it seems to be going up, seems to be doing really good. So, hopefully, these two examples have shown you a little bit more about some of the power of hypothesis testing. We can now start testing claims about our data because of the fact that the sample statistics from our data follow predictable distributions called sampling distributions. In fact, the most common sampling distribution is the normal distribution, and from those normal distributions we can calculate probabilities of events. So that's why we talked about probabilities and normal distributions and sampling distributions. So now you can do something much like we did here. I think that my data has some kind of attribute. Let's test it. Let's collect some samples and see if that's actually true or not, and that's the beauty and the benefit of what we can do with hypothesis testing. Now that being the case, let's talk around, or let's talk about the idea of ethics around inference around hypothesis testing with data, because just because you're testing hypotheses does not mean you're always going to be right. There are errors in hypothesis tests. Hypothesis tests, remember, depend on samples, and samples, just like when we had confidence intervals, aren't always going to be
the best representation of the truth, so hypothesis tests may be wrong, right? I mean, again, go back to that fair coin example. Could a fair coin actually get five flips in a row? Yes, it could. It's just not probable that it happens, but let's say you accuse me of not having a fair coin, and then you actually do some fancy waiting, and you try and figure it out, and you put it on a scale, and my coin actually is fair. I just got lucky and got five heads in a row again, that could happen. So, hypothesis tests aren't 100% accurate. That's the whole idea of that notion of how rare is rare, is that I know it's not going to be 100% right, but I want there if it's a really small chance that I'm no longer going to believe the null again. Could the null be true? Sure, there is a chance it could be true. It's just a small chance, which means we again could make an error. In fact, there are two types of errors with hypothesis testing. So, let's take a look. Let's imagine you have truth. The null hypothesis is actually true, and let's imagine you actually do not reject the null hypothesis. Well, congratulations, you got it correct. However, if the null hypothesis was actually true, let's say my coin was fair, I just got lucky, but you rejected that null hypothesis. What we say is you've made a type one error. Let's flip it over to the other side. Let's imagine the null hypothesis is false. Now, if you correctly rejected the null hypothesis, then good. Up the null hypothesis wasn't true to begin with. You rejected the null hypothesis, so you made the right choice. But let's imagine the null hypothesis was actually false, but you did not reject the null hypothesis. This is what we call a type two error. So, a type one error is when we reject the null hypothesis when the null hypothesis was actually true. In other words, you have a false rejection. So, the probability of making this type one error in a hypothesis test is called, wait, what? The significance level. Yes, yes, it is. If you think about it, what is the significance level? It's your definition is of how rare is rare enough where I'm no longer going to believe it. Well, let's say that 5% is your cutoff. If you say, "Hey, if this has a less than 5% chance of happening, then I'm no longer going to believe the null hypothesis is true. Well, okay. Well, then you said, though, that there is a 5% chance that it actually happens. So, the probability of rejecting the null when the null was actually true is the same as your significance level. For example, in our example, 5% If I said 5% is too rare for me. Well, then 5% of the time you're going to be wrong. In fact, most hypothesis tests are referred to as significance tests, because the only thing they look for, the only thing they type, they try to control is the type one error. They look at significance levels, a type two error, on the other hand, is accepting the null hypothesis when the null hypothesis was actually false. In other words, you've falsely accepted something. Now, just to get a little bit more terminology under your belt, the probability of not making a type two error is what we call power, again. That's just something that people heavy in statistics refer to. You don't have to know all the details of that. Now, the thing you should know is that it's really difficult to control the type two error, and in fact, we can only control one of them at a time.
If I focus on the type one error, then the type two error has freedom to move. If I focus on the type two error, then the type one error has freedom to move. So we just have to be careful, and that's why we always say be very careful with inferences that you make. What if your sample of data happened to be drawn on data only from summer months with clear days. Well, remember summer months, as we've seen previously, and days that are clear typically have higher numbers of users. So, if we only focused on summer months with clear days in our sample, then maybe the days would be estimated to have too many users, especially if I wanted to know the average users across the whole year. Well, then that means that we could lead to incorrect actions, we could do the wrong thing. So you have to be mindful. This is why when we talk all the way about sampling at the very beginning of the course, we focus so much on the collection of data. Hypothesis tests completely depend on the data they're built from. If you give it bad data, it's going to have bad results, just like the example I showed you above, in that example above, if our population really was the average daily users of the whole year, but our sample only contained days from the summer, then we're not doing a good job sampling, and because we do a bad job sampling, hypothesis tests can't get around that, so they're going to give you bad results. Like I said, everything on this class builds on itself, so it's not just probabilities and normal distributions and sampling distributions. The root of everything goes back to that data and how much effort you put into the sampling of the data to make sure it was done right. Hypothesis test results reveal something, but they also don't reveal everything. So, just be mindful of that. You may not know why the results are the way they are, so you have to be careful with inference, and you also have to be careful about justification, you see, people sometimes forget the possibility of errors when making claims from a statistical test, for example, we ran a statistical test where we no longer believe that the average daily users is 4000 We have to be careful about saying we know that more than 4000 bikes per day are used on average. Do we? We have evidence to say that more than 4000 bikes per day are used on average, but we don't actually know unless we were to calculate the whole population number of days. So just be careful, be careful about how strong a wording you actually say. This notion of we know something may be not true. We have strong evidence of something that is true. So again, think about it like a court case. We incorrectly claim people are guilty. Sometimes that means we have to be careful about rushing to judgment. We have strong evidence to say this person is guilty, but do we know they're guilty? Maybe not. So, let's summarize. A type one error is rejecting the null hypothesis when the null hypothesis was actually true. A type two error is the opposite, it's accepting the null hypothesis when the null hypothesis was actually false. And remember, the reason why we make errors is because hypothesis tests completely depend on the data they're built from, and like we saw with confidence intervals and with sampling and with a variety of
sections here in this course, data is not always 100% accurate in describing something, so just be careful and always keep this in mind, your data could be wrong, which means that you might have a possibility of an error. Don't ever forget that people sometimes forget the possibility of making errors when they make claims from a statistical test. Never say you know something, you just have strong evidence for it. Well, hopefully that gives you some idea around hypothesis testing. That is the end of this section, as well as this lecture, and I look forward to seeing you in the next one.