import pandas as pd
import numpy as np
import os

import seaborn as sns
import plotly.express as px
pd.options.plotting.backend = 'plotly'

Lecture 11 – Permutation Testing, Missingness Mechanisms

DSC 80, Winter 2023

Announcements

• Lab 3's reflection form is due for extra credit tomorrow at 11:59PM.
• Lab 4 (hypothesis and permutation testing) is due on Monday, February 6th at 11:59PM.
  • See this post on Ed for clarifications.
• Project 2 is due on Thursday, February 9th at 11:59PM.
  • See this post on Ed for help with Question 7 if you'd like to finish the project before the weekend. (We'll cover the relevant lecture material on Monday.)
• Several assignment grades have been released; check Gradescope and Ed for details.
  • Many students' Project 1 grades increased last night 👀.

Agenda

• Using permutation testing to compare two categorical distributions.
• Missingness mechanisms.
  • In what ways can data be missing? Why do we care?
  • How do we identify missingness mechanisms using data?

Additional resources:

• Permutation testing:
  • Extra lecture notebook: Fast Permutation Tests.
  • Great visualization.
• Missingness mechanisms:
  • Course notes.
  • Wikipedia.
  • This textbook page.
  • This journal article.

Differences between categorical distributions

Hypothesis testing vs. permutation testing

"Standard" hypothesis testing helps us answer questions of the form:

I have a population distribution, and I have one sample. Does this sample look like it was drawn from the population?

Permutation testing helps us answer questions of the form:

I have two samples, but no information about any population distributions. Do these samples look like they were drawn from the same population?

Example: Married vs. unmarried couples

Let's load in a cleaned version of the couples dataset from the last lecture.

couples_fp = os.path.join('data', 'married_couples_cleaned.csv')
couples = pd.read_csv(couples_fp)
couples.head()

	mar_status	empl_status	gender	age
0	married	Working as paid employee	M	51
1	married	Working as paid employee	F	53
2	married	Working as paid employee	M	57
3	married	Working as paid employee	F	57
4	married	Working as paid employee	M	60

couples.sample(5)

	mar_status	empl_status	gender	age
774	married	Not working - looking for work	M	35
668	unmarried	Working as paid employee	M	50
490	married	Working as paid employee	M	52
367	married	Not working - disabled	F	55
208	married	Working as paid employee	M	49

Understanding employment status in households

• Do married households more often have a stay-at-home spouse?
• Do households with unmarried couples more often have someone looking for work?
• How much does the employment status of the different households vary?

To answer these questions, let's compute the distribution of employment status conditional on household type (married vs. unmarried).

# Note that this is a shortcut to picking a column for values and using aggfunc='count'.
empl_cnts = couples.pivot_table(index='empl_status', columns='mar_status', aggfunc='size')
cond_distr = empl_cnts / empl_cnts.sum()
cond_distr

mar_status	married	unmarried
empl_status
Not working - disabled	0.048518	0.077055
Not working - looking for work	0.047844	0.118151
Not working - on a temporary layoff from a job	0.014151	0.022260
Not working - other	0.122642	0.056507
Not working - retired	0.063342	0.018836
Working as paid employee	0.610512	0.594178
Working, self-employed	0.092992	0.113014

Differences in the distributions

Are the distributions of employment status for married people and for unmarried people who live with their partners different?

Is this difference just due to noise?

cond_distr.plot(kind='barh', title='Distribution of Employment Status, Conditional on Household Type', barmode='group')