Lecture 1 – Introduction

DSC 80, Winter 2023

Welcome to DSC 80! 🎉


About the instructor

Suraj Rampure (call me Suraj, pronounced “soo-rudge”)

Course staff

In addition to the instructor, we have 11 tutors, who are here to help you in discussion, office hours, and on Ed:

Nicole Brye, John Driscoll, Doris Gao, Yuxin Guo, Daniel Li, Weiyue Li, Karthikeya Manchala, Yash Potdar, Ethan Shapiro, Costin Smilovici, and Tiffany Yu.

Learn more about them at dsc80.com/staff.

What is data science? 🤔

What is data science?

Everyone seems to have their own definition of what data science is.

The DSC 10 approach

In DSC 10, we told you that data science is about drawing useful conclusions from data using computation.

In DSC 10, you:

Let's look at a few more definitions of data science.

What is data science?

In 2010, Drew Conway published his famous Data Science Venn Diagram.

What is data science?

There isn't agreement on which "Venn Diagram" is correct!

What does a data scientist do?

The chart below is taken from the 2016 Data Science Salary Survey, administered by O'Reilly. They asked respondents what they spend their time doing on a daily basis. What do you notice?

The chart below is taken from the followup 2021 Data/AI Salary Survey, also administered by O'Reilly. They asked respondents:

What technologies will have the biggest effect on compensation in the coming year?

What does a data scientist do?

My take: in DSC 80, and in the DSC major more broadly, we are training you to ask and answer questions using data.

As you take more courses, we're training you to answer questions whose answers are ambiguous – this uncertainly is what makes data science challenging!

Let's look at some examples of data science in practice.

Moving average of the average number of guesses taken for each Wordle word, based on patterns shared on Twitter. (source)

Is Wordle Dying? The Data Weighs In

Compared to the peak of 350,000 shares in mid-February, a typical day in September 2022 only saw about 32,000 Wordle shares on Twitter. That’s a 91 percent drop in a span of seven months. (source)


You've almost certainly heard about ChatGPT over the past few months. It's a powerful language model, but it's not perfect. Let's experiment (inspiration) – you can try it for free at chat.openai.com.

You cannot use it for any assignments in this class. However, in Project 4, you're going to build a language model of your own!

Data science involves people 🧍

The decisions that we make as data scientists have the potential to impact the livelihoods of other people.


“The purpose of computing is insight, not numbers.” - R. Hamming. Numerical Methods for Scientists and Engineers (1962).

Course content

Course goals

DSC 80 is about the practice of dealing with messy, ambiguous, and complex data.

In this course, you will...

Course outcomes

After this course, you will...


This course was desgined by a former data scientist at Amazon (Aaron Fraenkel). As such, you'll be learning skills that you need to know as a data scientist.

Course logistics

Course website

The course website is your one-stop-shop for all things related to the course.


Make sure to read the syllabus!

Getting set up

In addition, you must fill out our Welcome Survey.

Accessing course content on GitHub

You will access all course content by pulling the course GitHub repository:


We will post HTML versions of lecture notebooks on the course website, but otherwise you must git pull from this repository to access all course materials (including blank copies of assignments).

Environment setup


In this course, you will learn by doing!

In DSC 80, assignments will usually consist of both a Jupyter Notebook and a .py file. You will write your code in the .py file; the Jupyter Notebook will contain problem descriptions and test cases. Lab 1 will explain the workflow.

Discussions and lab reflections

In order to have you reflect on your lab work, we will offer extra credit each week if you do all 3 of the following:

  1. Submit the lab.
  2. Attend discussion in-person (Wednesdays 5-5:50PM in Mosaic 114), where we'll take up solutions to the most recent lab.
  3. Submit a lab reflection form to Gradescope by Saturday.

Each week you do all 3, you'll earn 0.3% of extra credit – this could total 2.7%.

This scheme starts next week. Discussion will be podcasted.


A typical week in DSC 80

Sunday Monday Tuesday Wednesday Thursday Friday Saturday
Lecture Lecture & Discussion Lecture
Lab due Project/checkpoint due Lab reflection due (extra credit)


Support 🫂

It is no secret that this course requires a lot of work - becoming fluent with working with data is hard!

Once you've tried to solve problems on your own, we're glad to help.

You'll have to work a lot, but we'll make the time spent worth it.

The data science lifecycle 🚴

The scientific method

You learned about the scientific method in elementary school.

However, it hides a lot of complexity.

The data science lifecycle

All steps lead to more questions! We'll refer back to the data science lifecycle repeatedly throughout the quarter.

Example: myFlightradar24


Goal: recreate this map.


myFlightradar24 is a free site that allows you to enter information about flights that you've been on and produces interesting visualizations using that information. The dataset we're working with is taken from my personal myFlightradar24 page, and contains every flight I've been on since 2014, plus some additional ones from years prior.

Step 0: Keep just the relevant columns

flights currently contains a lot of information that we're not going to use.

Step 1: Extract 3-letter airport codes

These three letter codes (e.g. 'SAN', 'JFK') are called "IATA codes." They're much more convenient to work with than say, 'San Diego / San Diego International (SAN/KSAN)'.

Later this quarter, we'll learn how to perform this task in a more robust fashion, but for now, we can use Python string methods.

Step 2: Find the number of flights between each airport pair

The color of the line between a pair of airports is determined by how often one flew between that pair of airports. myFlightradar24 seems to use the following scheme:

Issue: This DataFrame counts (A, B) as being different than (B, A), though on the map these appear to be the same. How can we come up with a single row for every pair of airports, without using a for-loop?

Step 3: Create unique strings for each airport pair

Suppose two IATA codes are 'XYZ' and 'ABC'. We can combine them to create a single string, by:

In this case, that yields 'ABC-XYZ'. If we do this for all rows in counts, we can then group by these new strings and construct one row per airport pair!

Step 4: Actually find the number of flights between each airport pair

Step 5: Find the latitude and longitude of each airport

In order to draw a map, we need to know the position of each airport. Our dataset doesn't currently have that information, so we'll need to acquire it from somewhere else.

Fortunately, we have access to a CSV that contains this information!

Let's keep only the rows where the 'iata_code' is not missing, and only the relevant columns.

There are two ways to combine the information in this DataFrame with both_counts:

Let's try the latter. To do so, we need to construct a dictionary with the required information.

We're getting closer...

Step 6: Draw a line between a single pair of airports

We'll use the folium package to draw lines. It's not included in the dsc80 conda environment we're having you set up, so we must install it.

The core object in folium is the Map. To initialize one, it's a good idea to provide it a "center" – we can use the my hometown, on the eastern part of the continent.

To draw a line between two points, we create folium.PolyLine object, and call the add_to method on it.

Great! Now we need to do this for all pairs of airports, and make sure the color of each line matches the frequency of the route.

Step 6: Draw a line for each pair of airports

We'll use the scheme mentioned earlier to determine the color of each line.

Now, we'll create a folium.PolyLine for each pair in both_counts, setting the color appropriately.

Let's see what we've produced!

It's not perfect:

We don't have time to make these fixes now, but try making those changes yourself (and send me an email if you do)!

Next time