# Lecture 1 – Introduction¶

## Welcome to DSC 80! 🎉

### Agenda¶

• Who are we?
• What does a data scientist do?
• What is this course about, and how will it run?
• The data science lifecycle.
• A fun example.

#### Suraj Rampure (call me Suraj, pronounced “soo-rudge”)¶

• Originally from Windsor, ON, Canada 🇨🇦.
• BS (’20) and MS (’21) in EECS from UC Berkeley 🐻.
• Designed and taught several data science courses as a student there.
• Fifth quarter teaching at UCSD 🌴, and second time teaching DSC 80.
• Also running the capstone program.
• Previously taught DSC 10 (x3), DSC 40A, DSC 90 (x2).
• Outside the classroom 👨‍🏫: watching basketball, finding new restaurants, traveling, FaceTiming my dog 🐶, etc.

### Course staff¶

In addition to the instructor, we have 11 tutors, who are here to help you in discussion, office hours, and on Ed:

Nicole Brye, John Driscoll, Doris Gao, Yuxin Guo, Daniel Li, Weiyue Li, Karthikeya Manchala, Yash Potdar, Ethan Shapiro, Costin Smilovici, and Tiffany Yu.

## What is data science? 🤔¶

### What is data science?¶

Everyone seems to have their own definition of what data science is.

### The DSC 10 approach¶

In DSC 10, we told you that data science is about drawing useful conclusions from data using computation.

In DSC 10, you:

• Used Python to explore and visualize data.
• Used simulation to make inferences about a population, given just a sample.

Let's look at a few more definitions of data science.

### What is data science?¶

In 2010, Drew Conway published his famous Data Science Venn Diagram.

### What is data science?¶

There isn't agreement on which "Venn Diagram" is correct!

### What does a data scientist do?¶

The chart below is taken from the 2016 Data Science Salary Survey, administered by O'Reilly. They asked respondents what they spend their time doing on a daily basis. What do you notice?

The chart below is taken from the followup 2021 Data/AI Salary Survey, also administered by O'Reilly. They asked respondents:

What technologies will have the biggest effect on compensation in the coming year?

### What does a data scientist do?¶

My take: in DSC 80, and in the DSC major more broadly, we are training you to ask and answer questions using data.

As you take more courses, we're training you to answer questions whose answers are ambiguous – this uncertainly is what makes data science challenging!

Let's look at some examples of data science in practice.

Moving average of the average number of guesses taken for each Wordle word, based on patterns shared on Twitter. (source)

### Is Wordle Dying? The Data Weighs In¶

Compared to the peak of 350,000 shares in mid-February, a typical day in September 2022 only saw about 32,000 Wordle shares on Twitter. That’s a 91 percent drop in a span of seven months. (source)

### ChatGPT¶

You've almost certainly heard about ChatGPT over the past few months. It's a powerful language model, but it's not perfect. Let's experiment (inspiration) – you can try it for free at chat.openai.com.

You cannot use it for any assignments in this class. However, in Project 4, you're going to build a language model of your own!

### Data science involves people 🧍¶

The decisions that we make as data scientists have the potential to impact the livelihoods of other people.

• COVID case forecasting.
• Criminal sentencing.

### Warning!¶

• Good data analysis is not:

• A simple application of a statistics formula.
• A simple application of statistical software.
• There are many tools out there for data science, but they are merely tools. They don’t do any of the important thinking – that's where you come in!

“The purpose of computing is insight, not numbers.” - R. Hamming. Numerical Methods for Scientists and Engineers (1962).

## Course content¶

### Course goals¶

DSC 80 is about the practice of dealing with messy, ambiguous, and complex data.

In this course, you will...

• Practice translating potentially vague questions into quantitative questions about measurable observations.
• Learn to reason about "black-box" processes (e.g. complicated models).
• Understand computational and statistical implications of working with data.
• Learn to use real data tools (and rely on documentation).
• Get a taste of the "life of a data scientist."

### Course outcomes¶

After this course, you will...

• Be prepared for internships and data science "take home" interviews!
• Have the background and maturity to succeed in the upper-division.

### Topics¶

This course was desgined by a former data scientist at Amazon (Aaron Fraenkel). As such, you'll be learning skills that you need to know as a data scientist.

• Week 1: From babypandas to pandas
• Week 2: More on DataFrames
• Week 3: Working with messy data
• Week 4: Hypothesis and permutation testing, missing values
• Week 5: Imputation, web scraping
• Week 6: Web scraping, Midterm Exam
• Week 7: Text features
• Week 8: Feature engineering, sklearn basics
• Week 9: sklearn pipelines and model evaluation
• Week 10: Classifier evaluation, fairness
• Week 11: Final Exam

## Course logistics¶

### Course website¶

The course website is your one-stop-shop for all things related to the course.

### dsc80.com

Make sure to read the syllabus!

### Getting set up¶

• Ed: Q&A forum. Must be active here, since this is where all announcements will be made.
• Canvas: no.

In addition, you must fill out our Welcome Survey.

### Accessing course content on GitHub¶

You will access all course content by pulling the course GitHub repository:

github.com/dsc-courses/dsc80-2023-wi

We will post HTML versions of lecture notebooks on the course website, but otherwise you must git pull from this repository to access all course materials (including blank copies of assignments).

### Environment setup¶

• You have two choices:
• Set up your own Python environment (strongly recommended).
• Use DataHub.
• Either way, follow the instructions on the Tech Support page of the course website.
• Once you set up your environment, you will pull the course repo every time a new assignment comes out.
• Note: You will submit your work to Gradescope directly, without using Git.
• Will post a demo video with Lab 1.

### Assignments¶

In this course, you will learn by doing!

• Labs (35%): 9 total. Due weekly on Mondays (or Tuesday, in Weeks 2 and 7).
• Projects (35% + 5% checkpoints): 5 total. Usually due on Thursdays, and usually have a "checkpoint."

In DSC 80, assignments will usually consist of both a Jupyter Notebook and a .py file. You will write your code in the .py file; the Jupyter Notebook will contain problem descriptions and test cases. Lab 1 will explain the workflow.

### Discussions and lab reflections¶

In order to have you reflect on your lab work, we will offer extra credit each week if you do all 3 of the following:

1. Submit the lab.
2. Attend discussion in-person (Wednesdays 5-5:50PM in Mosaic 114), where we'll take up solutions to the most recent lab.
3. Submit a lab reflection form to Gradescope by Saturday.

Each week you do all 3, you'll earn 0.3% of extra credit – this could total 2.7%.

This scheme starts next week. Discussion will be podcasted.

### Exams¶

• Midterm Exam (10%): Wednesday, February 15th, in-person during lecture.
• Final Exam (15%): Wednesday, March 22nd, 11:30AM-2:30PM, in-person (location TBD).
• Let us know on the Welcome Survey if you have a conflict.

### A typical week in DSC 80¶

Sunday Monday Tuesday Wednesday Thursday Friday Saturday
Lecture Lecture & Discussion Lecture
Lab due Project/checkpoint due Lab reflection due (extra credit)

### Support 🫂¶

It is no secret that this course requires a lot of work - becoming fluent with working with data is hard!

• You will learn how to solve problems independently – documentation and the internet will be your friends.
• Learning how to effectively check your work and debug is extremely useful.
• Learning to stick with a problem (tenacity) is a very valuable skill; but don't be afraid to ask for help.

Once you've tried to solve problems on your own, we're glad to help.

• Office hours are offered – most are in-person, but a few are remote. See the Calendar 📆 for details.
• Ed is your friend too. Make your conceptual questions public, and make your debugging questions private.
You'll have to work a lot, but we'll make the time spent worth it.

## The data science lifecycle 🚴¶

### The scientific method¶

You learned about the scientific method in elementary school.

However, it hides a lot of complexity.

• Where did the hypothesis come from?
• What data are you modeling? Is the data sufficient?
• Under which conditions are the conclusions valid?

### The data science lifecycle¶

All steps lead to more questions! We'll refer back to the data science lifecycle repeatedly throughout the quarter.

Goal: recreate this map.

myFlightradar24 is a free site that allows you to enter information about flights that you've been on and produces interesting visualizations using that information. The dataset we're working with is taken from my personal myFlightradar24 page, and contains every flight I've been on since 2014, plus some additional ones from years prior.

### Step 0: Keep just the relevant columns¶

flights currently contains a lot of information that we're not going to use.

### Step 1: Extract 3-letter airport codes¶

These three letter codes (e.g. 'SAN', 'JFK') are called "IATA codes." They're much more convenient to work with than say, 'San Diego / San Diego International (SAN/KSAN)'.

Later this quarter, we'll learn how to perform this task in a more robust fashion, but for now, we can use Python string methods.

### Step 2: Find the number of flights between each airport pair¶

The color of the line between a pair of airports is determined by how often one flew between that pair of airports. myFlightradar24 seems to use the following scheme:

• The most common few pairs are in black.
• The next most common few pairs are in purple.
• Otherwise, if a pair has only been flown once, it is colored in orange, and all other pairs are colored in red.

Issue: This DataFrame counts (A, B) as being different than (B, A), though on the map these appear to be the same. How can we come up with a single row for every pair of airports, without using a for-loop?

### Step 3: Create unique strings for each airport pair¶

Suppose two IATA codes are 'XYZ' and 'ABC'. We can combine them to create a single string, by:

• Sorting them alphabetically.
• Concatenating them with '-'.

In this case, that yields 'ABC-XYZ'. If we do this for all rows in counts, we can then group by these new strings and construct one row per airport pair!

### Step 5: Find the latitude and longitude of each airport¶

In order to draw a map, we need to know the position of each airport. Our dataset doesn't currently have that information, so we'll need to acquire it from somewhere else.

Let's keep only the rows where the 'iata_code' is not missing, and only the relevant columns.

There are two ways to combine the information in this DataFrame with both_counts:

• merge the two DataFrames.
• Perform a "hash join," in which we make a dictionary mapping 'iata_code's to locations, and then use the dictionary to find the location of each airport given its 'iata_code'.

Let's try the latter. To do so, we need to construct a dictionary with the required information.

We're getting closer...

### Step 6: Draw a line between a single pair of airports¶

We'll use the folium package to draw lines. It's not included in the dsc80 conda environment we're having you set up, so we must install it.

The core object in folium is the Map. To initialize one, it's a good idea to provide it a "center" – we can use the my hometown, on the eastern part of the continent.

To draw a line between two points, we create folium.PolyLine object, and call the add_to method on it.

Great! Now we need to do this for all pairs of airports, and make sure the color of each line matches the frequency of the route.

### Step 6: Draw a line for each pair of airports¶

We'll use the scheme mentioned earlier to determine the color of each line.

Now, we'll create a folium.PolyLine for each pair in both_counts, setting the color appropriately.

Let's see what we've produced!

It's not perfect:

• There are no labels for the individual airports.
• It doesn't draw the geodesic line between each pair of airports.

We don't have time to make these fixes now, but try making those changes yourself (and send me an email if you do)!

### Next time¶

• A deep dive into pandas.
• Lab 1 will be released tomorrow!