In Class Activity - Hypothesis testing¶

Note

This exercise authored by Brenden Lake and is released under the license.

The chapter discussed some basic issues in hypothesis testing. In this notebook, you will explore these ideas in Python.

Importing packages¶

import numpy as np
import numpy.random as npr
import seaborn as sns
import matplotlib.pyplot as plt
import scipy.stats as stats

Problem 0: Critical regions and critical values¶

Let’s recreate the example from Section 10.4.1, which talks about computing the critical region for the ESP study. Here are some of the details, but I suggest you review Section 10.4.1 first.

Our population parameter \(\theta\) is just the overall probability that people respond correctly when asked to guess which side of the card is displayed in the ESP study. Our test statistic \(X\) is the count of the number of people who did so, out of a sample size of \(N\). This is exactly what the binomial distribution describes! So, we would say that the null hypothesis predicts that \(X\) is binomially distributed, which is written \( X \sim \mbox{Binomial}(\theta,N). \) Since the null hypothesis states that \(\theta = 0.5\), and our experiment has \(N=100\) people, we can use this as a sampling distribution.

Your goal here is to compute the critical region for \(\alpha = .05\) (the type-I error rate) using sampling. Thus, you should randomly generate 10,000 experiments, each with \(N=100\) coin flips, and record the test statistic \(X\) for each simulated experiment. Your goal is to find the lower and upper “critical values” given your sampling distribution (See figure below for a visualization. Your simulations should roughly match this plot).

You may find the following functions helpful: np.random.binomial, np.sort, and sns.displot

# Your answer goes here

Problem 1: Critical values and sample size¶

Copy the code above and see how the location of the critical values changes as a function of the number of simulated participants in an experiment. Thus, rather than \(N=100\), what are the critical values when \(N=200\) or \(N=500\)? You will want to use \(X/N\) as your sample statistic instead (percent of participants that get it right) so you can compare more easily across different numbers of participants.

# Your answer goes here

Problem 2: Binomial test¶

Check your simulated critical values in Problem 0 to see how they correspond with an exact binomial test. To do so, check out the scipy function stats.binom_test

# Your answer here