1. Null Hypothesis• Imagine there's a virus and we have two drugs (A and B) we can use to treat it.• Let's say we give each drug to 3 people and measure the recovery time.• People may recover in different times and that might be due a lot of random reasons which have nothing to do with the drugs.• Let's say the average recovery time for Drug A is 15 hours less than that of Drug B.– We can form a hypothesis that Drug A's recovery time is 15 hours less than Drug B.– We can verify our hypothesis by repeating the experiment many times.– Based on the majority results of the experiments we can either reject or fail to reject our hypothesis.• The other thing about our hypothesis is the number 13. – It could've easily been 12.5 hours or 14 hours. There's a lot of reasonable hypotheses to test. How do we know which one to test?* Answer: We just test if there's no difference between Drug A and B.• The hypothesis that there is no difference between things is called the Null Hypothesis.– Here, we don't care about the amount of difference.– Note: Without the null hypothesis, we need preliminary data in order to make a statement that we can test in follow up experiments. The reason is null hypothesis doesn't care about the difference value and just test whether there's a difference or not. Therefore, the only value it represents is 0 (no difference).– This is a desirable feature as it narrows down the many possible hypotheses into one hypothesis (i.e. null hypothesis).
2. Alternative Hypothesis• In order to decide if we reject of fail to reject the null hypothesis, we run the data through something called a statistical test.• A statistical test needs three things:– Data– A Null (or primary) Hypothesis– An Alternative Hypothesis• Alternative Hypothesis is simply the opposite of the Null Hypothesis (i.e. there's a difference).2.1. Why is Alternative Hypothesis important?• One way to test the null hypothesis is to calculate a (overall) mean value (i.e. mean of all recovery times) for all of the data from both drugs and calculate the distances between each observation and the mean AND compare those to distances calculated from "individual" means for Drug A and Drug B (i.e. mean of Drug A recovery times, mean of Drug B recovery times)• The distances around the single mean represent the null hypothesis (there's no difference), and the distances around the two separate means represents the alternative hypothesis.– If the distances around the two means are much shorter than the distances around the single mean, then that suggests that using two means to summarize the data makes more sense than using one → We reject the null hypothesis (and visa versa).• Note: Failing to reject the null hypothesis is the same thing as realizing that using two averages means that you have overfit the data.• Note: When we have two groups of data, the alternative hypothesis is pretty obvious (simply the opposite of null hypothesis). – However, when we have 3 or more groups, the null hypothesis is there is no difference between any of the groups. But, there are a few possible alternative hypotheses:* There's a difference between all groups.* All the groups are the same except for the first one.* All the groups are the same except for the first two.* and so on ...– Therefore, it's important to state which alternative hypothesis we're using.