Lecture 10 – Permutation Testing¶

DSC 80, Winter 2023¶

Announcements¶

• Discussion 3 is today at 5PM.
• Lab 3 scores and solutions (to non-discussion problems) have been released, or will be shortly.
• Project 2's checkpoint is due tomorrow at 11:59PM, and the full project is due on Thursday, February 9th at 11:59PM.
• Lab 4 (hypothesis and permutation testing) is due on Monday, February 6th at 11:59PM.
• Check out the Grade Report on Gradescope, which details your scores and slip days on most assignments so far.
• I'll be in the CSES "CS Jeopardy" event today from 5-7PM in CSE 1202.
• I know discussion is from 5-6PM 😢 – if you're interested, you can come after.

Agenda¶

• Hypothesis testing.
• Example: Penguin bill lengths 🐧.
• Permutation testing.

Example: Penguins (again!)¶

(source)

Consider the penguins dataset from a few lectures ago.

Average bill length by island¶

It appears that penguins on Torgersen Island have shorter bills on average than penguins on other islands.

Setup¶

• Null Hypothesis: Island and bill length are not related – the low average bill length of Torgersen Island penguins is due to chance alone.
• In other words, if we picked 47 penguins randomly from the population of 333 penguins, it is reasonable to see an average this low.
• Alternative Hypothesis: Island and bill length are related – the low average bill length of Torgersen Island penguins is not due to chance alone.

The plan¶

• The null hypothesis states that the 47 bill lengths of Torgersen Island penguins were drawn uniformly at random from the 333 bill lengths in the population.
• That is, if we repeatedly sampled groups of 47 penguins from the population and computed their mean bill length, it would not be uncommon to see an average bill length this low.
• Plan: Repeatedly sample (without replacement) 47 penguins from the population and compute their average bill length, and see where the Torgersen Island average bill length lies in this distribution.
• Average bill length is our test statistic.
• This is not a test statistic we've used in this lecture yet (and this is what separates this example from previous examples).

Simulation¶

Again, while you could do this with a for-loop (and you can use a for-loop for hypothesis tests in labs and projects), we'll use the faster size approach here.

Instead of using np.random.multinomial, which samples from a categorical distribution, we'll use np.random.choice, which samples from a known sequence of values.