Final Project: The Data Science Lifecycle 🔁

Checkpoint 1 (1%) Due: Friday, February 27th at 11:59PM

Checkpoint 2 (1%) Due: Friday, March 6th at 11:59PM

Final Project (10%) Due: Tuesday, March 17th at 11:59PM

Welcome to the Final Project, the final assignment of the quarter! 👋

This project aims to be a culmination of everything you’ve learned this quarter. In the project, you will conduct an open-ended investigation into one of three datasets: Recipes and Ratings 🍽, League of Legends ⌨️, or Power Outages 🔋. Specifically, you’ll draw several visualizations to help understand the distributions of key variables, assess the missingness mechanisms of columns with missing values, test hypotheses about the data, and finally, build and improve a predictive model. This project will be entirely manually graded by us – that’s right, no autograders!

The Final Project is worth 10% of your overall grade, which means it’s worth double what previous projects were worth. There are also two checkpoints, each worth 1%. The checkpoints are structured differently than other project checkpoints; rather than having you submit any code, you’ll answer a few questions about your progress on the project. More details are in the Checkpoint Submission section towards the bottom of this page.

As your final deliverables, you’ll submit two things:

1. An analysis, submitted as a Jupyter Notebook. This will contain the details of your work. Focus on completing your analysis first, as the analysis is the bulk of the project.
2. A report, submitted as a website. This will contain a narrative “story” with visuals. Work on this after finishing most of your analysis.

We encourage you to build something you are proud of as this will give you something concrete to put on your resume and show to potential employers!

Table of Contents

1. Choosing a Dataset
2. Part 1: Analysis
   1. Step 1: Introduction
   2. Step 2: Data Cleaning and Exploratory Data Analysis
   3. Step 3: Assessment of Missingness
   4. Step 4: Hypothesis Testing
   5. Step 5: Framing a Prediction Problem
   6. Step 6: Baseline Model
   7. Step 7: Final Model
   8. Step 8: Fairness Analysis
   9. Style
3. Part 2: Report
   1. Step 1: Initializing a Jekyll GitHub Pages Site
   2. Step 2: Choosing a Theme
   3. Step 3: Embedding Content
   4. Optional: Local Setup
4. Submission and Rubric
   1. Checkpoint 1 Submission
   2. Checkpoint 2 Submission
   3. Final Submission
   4. Rubric

Choosing a Dataset

In this project, you will perform an open-ended investigation into a single dataset. You must choose one of the following three datasets.

Recipes and Ratings 🍽    League of Legends ⌨️    Power Outages 🔋

The dataset description pages linked above each have three sections:

• Getting the Data: Describes how to access the data and, in some cases, what various features mean. (In general, you’re going to have to understand what your data means on your own!)
• Example Questions and Prediction Problems: Example questions to explore in Part 1: Steps 1-4, and example prediction problems to build models for in Part 1: Steps 5-8. You may use the provided examples, but we recommend coming up with your own questions and prediction problems!
• Special Considerations: Things to be aware of when working with the given dataset, e.g. some additional requirements.

When selecting which dataset you are going to use for your project, try choosing the one whose topic appeals to you the most as that will make finishing the project a lot more enjoyable.

To help contextualize the kinds of analysis you can do in this project, it might help to look at these examples from past students. These examples offer insights into crafting effective research questions, but bear in mind that they have their unique strengths and weaknesses. Treat them as a foundation for inspiration, but don’t just repeat or copy their work – be original!

1. League of Legends First Blood Statistical Analysis: This project excelled in clarifying their research aims, making the study understandable to a broader audience. In your own project, ensure that you provide a lucid and detailed explanation of your research focus.
2. Analyzing Power Outages: This project presents a noval way to do the data visualization. In your project, please think about what is the best way to present your data.
3. Investigation on the Relationship Between Amount of Sugar and Rating of Recipes: In this project, the students have effectively highlighted the improvement they made based on the baseline model. In your own project, also ensure that you explain what you have done to improve your baseline model.

Before choosing a dataset, read the rest of this page to see what’s required of you!

Part 1: Analysis

Before beginning your analysis, you’ll need to set up a few things.

1. Pull the latest version of the course Github repository. Within the projects/project04 folder, there is a template.ipynb notebook that you will use as a template for the project. If you delete the file or want another copy of the template, you can re-download it from here. This is where your analysis will live; you will submit the full contents of this notebook to us.
2. Select one of the three datasets mentioned above, download it, and load it into your template notebook.

Once you have your dataset loaded in your notebook, it’s time for you to find meaning in the real-world data you’ve collected! Follow the steps below.

For each step, we specify what must be done in your notebook and what must go on your website, which we expand on in Part 2. We recommend you write everything in your notebook first, and then move things over to your website once you’ve completed your analysis.

In Steps 1-4, you’ll develop a deeper understanding of your dataset while trying to answer a single question.

Step 1: Introduction

Step 2: Data Cleaning and Exploratory Data Analysis

Step 3: Assessment of Missingness

Step 4: Hypothesis Testing

In Steps 5-8, you will build a predictive model, based on the knowledge of your dataset you developed in Steps 1-4.

Step 5: Framing a Prediction Problem

Step 6: Baseline Model

Step 7: Final Model

Step 8: Fairness Analysis

Style

To keep your notebook organized and easy to follow, please abide by the following formatting rules:

• Your work for each of the eight project steps described aboveshould be completed in code cells underneath the Markdown header of that section’s name (e.g. “Step 8: Fairness Analysis”).
• You should only include work that is relevant to posing, explaining, and answering the question(s) you state in your website. You should include data quality, cleaning, and missingness assessments, and intermediate models and features you tried, though these should broadly be relevant to the tasks at hand.
• Make sure to clearly explain what each component of your notebook means. Specifically:
  • All plots should have titles, labels, and a legend (if applicable), even if they don’t make it into your website. Plots should be self-contained – readers should be able to understand what they describe without having to read anything else.
  • All code cells should contain a comment describing how the code works (unless the code is self-explanatory – use your best judgment).

Part 2: Report

The purpose of your website is to provide the general public – your classmates, friends, family, recruiters, and random internet strangers – with an overview of your project and its findings, without forcing them to understand every last detail. We don’t expect the website creation process to take very much time, but it will certainly be rewarding. Once you’ve completed your analysis and know what you will put in your website, start reading this section.

Your website must clearly contain the following eight headings, corresponding to the eight steps mentioned in Part 1:

• Introduction
• Data Cleaning and Exploratory Data Analysis
• Assessment of Missingness
• Hypothesis Testing
• Framing a Prediction Problem
• Baseline Model
• Final Model
• Fairness Analysis

Don’t include “Step 1”, “Step 2”, etc. in your headings – the headings should be exactly as they are above.

The specific content your website should contain is described in the “Report on Website” columns above. Make sure to also give your website a creative title that relates to the question you’re trying to answer, and clearly include your name(s).

Your report will be in the form of a static website, hosted for free on GitHub Pages. More specifically, you’ll use Jekyll, a framework built into GitHub Pages that allows you to create professional-looking websites just by writing Markdown (dsc80.com is built using Jekyll!). GitHub Pages does the “hard” part of converting your Markdown to HTML.

If you’d like to follow the official GitHub Pages & Jekyll tutorial, you’re welcome to, though we will provide you with a perhaps simpler set of instructions here.

Step 1: Initializing a Jekyll GitHub Pages Site

1. Create a GitHub account, if you don’t already have one.
2. Create a new GitHub repository for your project.
   • GitHub Pages sites live at <username>.github.io/<reponame> (for instance, the site for github.com/dsc-courses/dsc80-2025-sp is dsc-courses.github.io/dsc80-2025-sp).
   • As such, don’t include “DSC 80” or “Final Project” in your repo’s name – this looks unprofessional to future employers, and gives you a generic-sounding URL. Instead, mention that this is a project for DSC 80 at UCSD in the repository description.
   • Make sure to make your repository public.
   • Select “ADD a README file.” This ensures that your repository starts off non-empty, which is necessary to continue.
3. Click “Settings” in the repository toolbar (next to “Insights”), then click “Pages” in the left menu.
4. Under “Branch”, click the “None” dropdown, change the branch to “main”, and then click “Save.” You should soon see “GitHub Pages source saved.” in a blue banner at the top of the page. This initiates GitHub Pages in your repository.
5. After 30 seconds, reload the page again. You should see “Your site is live at http://username.github.io/reponame/”. Click that link – you now have a site!
6. Click “Code” in the repo toolbar to return to the source code for your repo.

Note that the source code for your site lives in README.md. As you push changes to README.md, they will update on your site automatically within a few minutes! Before moving forward, make sure that you can make basic edits:

1. Clone your repo locally.
2. Make some edits to README.md.
3. Push those changes back to GitHub, using the following steps:
   • Add your changes to “staging” using git add README.md (repeat this for any other files you add).
   • Commit your changes using git commit -m '<some message here>', e.g. git commit -m 'changed title of website'.
   • Push your changes using git push.

Moving forward, we recommend making edits to your website source code locally, as described here, rather than directly editing README.md on GitHub. This is in part due to the fact that you’ll be creating folders and adding files to your source code.

Step 2: Choosing a Theme

The default “theme” of a Jekyll site is not all that interesting, so you will be required to change it.

To change the theme of your site:

1. Create a file in your repository called _config.yml.
2. Go here, and click the links for various themes until you find one you like.
3. Open the linked repository for the theme you’d like to use and scroll down to the “Usage” section of the README. Copy the necessary information from the README to your _config.yml and push it to your site.

For instance, if I wanted to use the Merlot theme, I’d put the following in my _config.yml:

remote_theme: pages-themes/merlot@v0.2.0
plugins:
  - jekyll-remote-theme # add this line to the plugins list if you already have one

Step 3: Embedding Content

Now comes the interesting part – actually including content in your site. This Markdown cheat sheet contains tips on how to format text and other page components in Markdown (and if you’d benefit by seeing an example, you could always look at the Markdown source of this very page – meta!).

What will be a bit trickier is embedding plotly plots in your site so that they are interactive. Note that you are required to do this, you cannot simply take screenshots of plots from your notebooks and embed them in your site. Here’s how to embed a plotly plot directly in your site.

1. First, you’ll need to convert your plot to HTML. If fig is a plotly Figure object (for instance, the result of calling px.express, go.Figure, or .plot when pd.options.plotting.backend = "plotly" has been run), then the method fig.write_html saves the plot as HTML to a file. Call it using fig.write_html('file-name.html', include_plotlyjs='cdn').
   • Change 'file-name.html' to the path where you’d like to initially save your plot.
   • include_plotlyjs='cdn' tells write_html to load the source code for the plotly library from a server, rather than including the entire source code in your HTML file. This drastically reduces the size of the resulting HTML file, keeping your repo size down.
2. Move the .html file(s) you’ve created into a folder in your website repo called assets.
   • Depending on where your template notebook is saved, you could combine this step with the step above by calling fig.write_html with the correct path (e.g. fig.write_html("../league-match-analysis/assets/matches-per-year.html")).
3. In README.md, embed your plot using the following syntax:

<iframe
  src="assets/file-name.html"
  width="800"
  height="600"
  frameborder="0"
></iframe>

• iframe stands for “inline frame”; it allows us to embed HTML files within other HTML files.
• You can change the width and height arguments, but don’t change the frameBorder argument.

Refer here for a working example.

Try your best to make your plots look professional and unique to your group – add axis labels, change the font and colors, add annotations, etc. Remember, potential employers will see this – you don’t want your plots to look like everyone else’s! If you’d like to match the styles of the plotly plots used in lecture (e.g. with the white backgrounds), you can import and use the dsc80_utils.py that’s in the proj04 folder of our public repo, alongside template.ipynb.

To convert a DataFrame in your notebook to Markdown source code (which you need to do in Step 2 of Part 1), use the .to_markdown() method on the DataFrame. For instance,

print(counts[['Quarter', 'Count']].head().to_markdown(index=False))

displays a string, containing the Markdown representation of the first 5 rows of counts, including only the 'Quarter' and 'Count' columns (and not including the index). You can see how this appears here; remember, no screenshots. You may need to play with this a little bit so that you don’t show DataFrames that are super, super wide and look ugly.

Optional: Local Setup

The above instructions give you all you need to create and make updates to your site. However, you may want to set up Jekyll locally, so that you can look at how changes to your site would look without having to push and wait for GitHub to re-build your site. To do so, follow the steps here and then here.

Submission and Rubric

As mentioned at the top of this page, this project has two checkpoints, each worth 1% of your overall grade. Each checkpoint is graded out of 20 points.

Checkpoint 1 Submission

The first checkpoint is due on Friday, February 27th at 11:59PM. You’ll need to complete some steps of Part 1 and answer the following questions directly on Gradescope:

1. (2 points) Which of the three datasets did you choose? Why?
2. (6 points) Upload a screenshot of a plotly visualization you created in Step 2.
3. (6 points) What is the pair of hypotheses you plan on testing in Step 4? What is the test statistic you plan on using?
4. (6 points) What is the column you plan on trying to predict in Steps 5-8? Is it a classification or regression problem?

Checkpoint 2 Submission

The second one is due on Friday, March 6th at 11:59PM. You’ll need to complete more of Part 1 and start on Part 2, then answer the following questions directly on Gradescope:

1. (7.5 points) For Step 4, what are your null and alternative hypotheses, and what was the result of your test?
2. (7.5 points) Briefly explain your baseline model and your plans for improving the model.
3. (5 points) Submit a working GitHub page webpage URL for the project. On the webpage, you need to at least include a project title.

Final Submission

You will ultimately submit your project in two ways. For both submissions, make sure to tag your partner.

1. Upload a PDF version of your notebook to the specific “Final Project Notebook PDF (Dataset)” assignment on Gradescope for your dataset.
   • To export your notebook as a PDF, first, restart your kernel and run all cells. Then, go to “File > Print Preview”. Then, save a print preview of the webpage as a PDF. There are other ways to save a notebook as a PDF but they may require that you have additional packages installed on your computer, so this is likely the most straightforward.
   • It’s fine if your plotly graphs don’t render in the PDF output of your notebook. However, make sure none of the code is cut off in your notebook’s PDF. You will lose 5% of the points available on this project if your code is cut off.
   • This notebook asks you to include a link to your website; make sure to do so.
2. Submat a link to your website to the “Final Project Website Link (All Datasets)” assignment on Gradescope.

You don’t need to submit your actual .ipynb file anywhere. While your website must be public and you should share it with others, you should not make your code for this project available publicly.

We will determine your submission time by the later of your two submissions. So, if you submit your PDF before the deadline and your website link one day late, that counts as a late Final Project submission.

There are a lot of moving parts to this assignment – don’t wait until the last minute to try and submit!

Rubric

Your project will be graded out of 200 points. The rough rubric is shown below. If you satisfy these requirements as described above and perform appropriate analyses, you will receive full credit.

Note that the rubric is intentionally vague when it comes to Steps 5-8. This is because an exact rubric would specify exactly what you need to do to build a model, but figuring out what to do is a large part of Steps 5-8.