In [53]:
from dsc80_utils import *

Lecture 9 – HTTP and HTML¶

DSC 80, Winter 2026¶

Midterm Exam 📝¶

One week from today during the usual lecture.

  • Pen and paper only. No calculators, phones, computers.
  • You are allowed to bring one double-sided 8.5" x 11" sheet of handwritten notes.
    • No writing on a tablet and printing it out.
  • Covers Lectures 1-8 and all related assignments.
  • To review problems from old exams, go to practice.dsc80.com.
    • Also look at the Resources tab on the course website.

Lab 5¶

  • Due Sunday.
  • No hidden tests.
  • No resubmissions.

Make sure to read the Campuswire announcement about Lab 5 and the Midterm Exam.

Agenda 📆¶

  • Introduction to HTTP.
  • Making HTTP requests.
  • Data formats.
  • APIs and web scraping.
  • Midterm review.

Introduction to HTTP¶

No description has been provided for this image

Data sources¶

  • Often, the data you need doesn't exist in "clean" .csv files.

  • Solution: Collect your own data!

    • Design and administer your own survey or run an experiment.
    • Find related data on the internet.
  • The internet contains massive amounts of historical record; for most questions you can think of, the answer exists somewhere on the internet.

Collecting data from the internet¶

  • There are two ways to programmatically access data on the internet:

    • through an API.
    • by scraping.

  • We will discuss the differences between both approaches, but for now, the important part is that they both use HTTP.

HTTP¶

  • HTTP stands for Hypertext Transfer Protocol.

    • It was developed in 1989 by Tim Berners-Lee (and friends).

  • It is a request-response protocol.

    • Protocol = set of rules.

  • HTTP allows...

    • computers to talk to each other over a network.
    • devices to fetch data from "web servers."

  • The "S" in HTTPS stands for "secure".

No description has been provided for this image

UCSD was a node in ARPANET, the predecessor to the modern internet (source).

The request-response model¶

HTTP follows the request-response model.

No description has been provided for this image

  • A request is made by the client.

  • A response is returned by the server.

  • Example: YouTube search 🎥.

    • Consider the following URL: https://www.youtube.com/results?search_query=apple+vision+pro.
    • Your web browser, a client, makes an HTTP request with a search query.
    • The server, YouTube, is a computer that is sitting somewhere else.
    • The server returns a response that contains the search results.
    • Note: ?search_query=apple+vision+pro is called a "query string."

Request methods¶

The request methods you will use most often are GET and POST; see Mozilla's web docs for a detailed list of request methods.

  • GET is used to request data from a specified resource.

  • POST is used to send data to the server.

    • For example, uploading a photo to Instagram or entering credit card information on Amazon.

Example GET request¶

Below is an example GET HTTP request made by a browser when accessing datascience.ucsd.edu.

GET / HTTP/1.1
Connection: keep-alive
Host: datascience.ucsd.edu
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36
sec-ch-ua: "Chromium";v="121", "Not A(Brand";v="99"
sec-ch-ua-platform: "macOS"

  • The first line (GET / HTTP/1.1) is called the "request line", and the lines afterwards are called "header fields". Header fields contain metadata.

  • We could also provide a "body" after the header fields.

  • To see HTTP requests in Google Chrome, follow these steps.

Example GET response¶

The response below was generated by executing the request on the previous slide.

HTTP/1.1 200 OK
Date: Sun, 04 Feb 2024 17:35:01 GMT
Server: Apache
X-Powered-By: PHP/7.4.33
Link: <https://datascience.ucsd.edu/wp-json/>; rel="https://api.w.org/"
Link: <https://datascience.ucsd.edu/wp-json/wp/v2/pages/113>; rel="alternate"; type="application/json"
...

<html lang="en-US">
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0"/>
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <link rel="profile" href="https://gmpg.org/xfn/11"/>
        <title>Halıcıoğlu Data Science Institute &#8211;UC San Diego</title>
        <script>
...

Consequences of the request-response model¶

  • When a request is sent to view content on a webpage, the server must:

    • process your request (i.e. prepare data for the response).
    • send content back to the client in its response.

  • Remember, servers are computers.

    • Someone has to pay to keep these computers running.
    • This means that every time you access a website, someone has to pay.

Making HTTP requests¶

Making HTTP requests¶

There are (at least) two ways to make HTTP requests outside of a browser:

  • From the command line, with curl.

  • From Python, with the requests package.

Making HTTP requests using requests¶

  • requests is a Python module that allows you to use Python to interact with the internet!
  • There are other packages that work similarly (e.g. urllib), but requests is arguably the easiest to use.
In [54]:
import requests

Example: GET requests via requests¶

For instance, let's access the source code of the UCSD homepage, https://ucsd.edu.

In [55]:
res = requests.get('https://ucsd.edu')

res is now a Response object.

In [56]:
res
Out[56]:
<Response [200]>

The text attribute of res is a string that containing the entire response.

In [57]:
type(res.text)
Out[57]:
str
In [58]:
len(res.text)
Out[58]:
59784
In [59]:
print(res.text[:1000])

<!DOCTYPE html>
<html lang="en">
  <head>
  
  

 





                                                        

    <meta charset="utf-8"/>
    <meta content="IE=edge" http-equiv="X-UA-Compatible"/>
    <meta content="width=device-width, initial-scale=1" name="viewport"/>
    <title>University of California San Diego</title>
    <meta content="University of California, San Diego" name="ORGANIZATION"/>
    <meta content="index,follow,noarchive" name="robots"/>
    <meta content="UCSD" name="SITE"/>
    <meta content="University of California San Diego" name="PAGETITLE"/>
    <meta content="The University California San Diego is one of the world's leading public research universities, located in beautiful La Jolla, California" name="DESCRIPTION"/>
    <!-- Facebook -->
    <meta content="University of California San Diego" property="og:title"/>
    <meta content="The University California San Diego is one of the world's leading public research universities, located in beautiful La Joll

Example: POST requests via requests¶

The following call to requests.post makes a post request to https://httpbin.org/post, with a 'name' parameter of 'King Triton'.

In [60]:
post_res = requests.post('https://httpbin.org/post',
                         data={'name': 'King Triton'})
post_res
Out[60]:
<Response [200]>
In [61]:
post_res.text
Out[61]:
'{\n  "args": {}, \n  "data": "", \n  "files": {}, \n  "form": {\n    "name": "King Triton"\n  }, \n  "headers": {\n    "Accept": "*/*", \n    "Accept-Encoding": "gzip, deflate, br, zstd", \n    "Content-Length": "16", \n    "Content-Type": "application/x-www-form-urlencoded", \n    "Host": "httpbin.org", \n    "User-Agent": "python-requests/2.32.3", \n    "X-Amzn-Trace-Id": "Root=1-69818930-1d4e5ff70ac76eb22e9a91d7"\n  }, \n  "json": null, \n  "origin": "128.54.158.137", \n  "url": "https://httpbin.org/post"\n}\n'
In [62]:
# More on this shortly!
post_res.json()
Out[62]:
{'args': {},
 'data': '',
 'files': {},
 'form': {'name': 'King Triton'},
 'headers': {'Accept': '*/*',
  'Accept-Encoding': 'gzip, deflate, br, zstd',
  'Content-Length': '16',
  'Content-Type': 'application/x-www-form-urlencoded',
  'Host': 'httpbin.org',
  'User-Agent': 'python-requests/2.32.3',
  'X-Amzn-Trace-Id': 'Root=1-69818930-1d4e5ff70ac76eb22e9a91d7'},
 'json': None,
 'origin': '128.54.158.137',
 'url': 'https://httpbin.org/post'}

What happens when we try and make a POST request somewhere where we're unable to?

In [63]:
yt_res = requests.post('https://youtube.com',
                       data={'name': 'King Triton'})
yt_res
Out[63]:
<Response [400]>
In [64]:
yt_res.text
Out[64]:
'<html lang="en" dir="ltr"><head><title>Oops</title><style nonce="LsULlnAmv0wUvxaLPUzdQg">html{font-family:Roboto,Arial,sans-serif;font-size:14px}body{background-color:#f9f9f9;margin:0}#content{max-width:440px;margin:128px auto}svg{display:block;pointer-events:none}#monkey{width:280px;margin:0 auto}h1,p{text-align:center;margin:0;color:#131313}h1{padding:24px 0 8px;font-size:24px;font-weight:400}p{line-height:21px}sentinel{}</style><link rel="shortcut icon" href="https://www.youtube.com/img/favicon.ico" type="image/x-icon"><link rel="icon" href="https://www.youtube.com/img/favicon_32.png" sizes="32x32"><link rel="icon" href="https://www.youtube.com/img/favicon_48.png" sizes="48x48"><link rel="icon" href="https://www.youtube.com/img/favicon_96.png" sizes="96x96"><link rel="icon" href="https://www.youtube.com/img/favicon_144.png" sizes="144x144"></head><body><div id="content"><h1>Something went wrong</h1><p><svg id="monkey" viewBox="0 0 490 525"><path fill="#6A1B9A" d="M325 85c1 12-1 25-5 38-8 29-31 52-60 61-26 8-54 14-81 18-37 6-26-37-38-72l-4-4c0-17-9-33 4-37l33-4c9-2 9-21 11-30 1-7 3-14 5-21 8-28 40-42 68-29 18 9 36 19 50 32 13 11 16 31 17 48z"/><path fill="none" stroke="#6A1B9A" stroke-width="24" stroke-linecap="round" stroke-miterlimit="10" d="M431 232c3 15 21 19 34 11 15-9 14-30 5-43-12-17-38-25-59-10-23 18-27 53-21 97s1 92-63 108"/><path fill="#6A1B9A" d="M284 158c35 40 63 85 86 133 24 52-6 113-62 123-2 0-4 1-6 1-53 9-101-33-101-87V188l83-30z"/><path fill="#F7CB4D" d="M95 152c-3-24 13-57 46-64l27-5c9-2 16-19 17-28l3-15 20-3c44 14 42 55 18 69 22 0 39 26 32 53-5 18-20 32-39 36-13 3-26 5-40 8-50 8-80-14-84-51z"/><path fill="#6A1B9A" d="M367 392c-21 18-77 70-25 119h-61c-27-29-32-69 1-111l85-8z"/><path fill="#6A1B9A" d="M289 399c-21 18-84 62-32 111h-61c-37-34-5-104 43-134l50 23z"/><path fill="#EDB526" d="M185 56l3-15 20-3c25 8 35 25 35 41-12-18-49-29-58-23z"/><path fill="#E62117" d="M190 34c8-28 40-42 68-29 18 9 36 19 50 32 10 9 14 23 16 37L187 46l3-12z"/><path fill="#8E24AA" d="M292 168c0 0 0 201 0 241s20 98 91 85l-16-54c-22 12-31-17-31-37 0-20 0-108 0-137S325 200 292 168z"/><path fill="#F7CB4D" d="M284 79c11-9 23-17 35-23 25-12 54 7 59 38v1c4 27-13 51-36 53-12 1-25 1-37 0-22-1-39-27-32-52v-1c2-6 6-12 11-16z"/><path fill="#6A1B9A" d="M201 203s0 84-95 140l22 42s67-25 89-86-16-96-16-96z"/><path fill="#BE2117" d="M224 54l-67-14c-10-2-13-15-5-21s18-6 26 0l46 35z"/><circle fill="#4A148C" cx="129" cy="161" r="12"/><circle fill="#4A148C" cx="212" cy="83" r="7"/><circle fill="#4A148C" cx="189" cy="79" r="7"/><path fill="#F7CB4D" d="M383 493c11-3 19-8 25-13 7-10 4-16-5-20 8-9 2-22-8-18 1-1 1-2 1-3 3-9-9-15-15-8-3 4-8 7-13 9l15 53z"/><path fill="#EDB526" d="M252 510c5 6 0 15-9 15h-87c-10 0-16-8-13-15 5-12 21-19 36-16l73 16z"/><ellipse transform="rotate(19.126 278.35 14.787)" fill="#E62117" cx="278" cy="15" rx="9" ry="7"/><path fill="#F7CB4D" d="M341 510c5 6 0 15-9 15h-87c-10 0-16-8-13-15 5-12 21-19 36-16l73 16z"/><path fill="#EDB526" d="M357 90c-12-19-35-23-55-11-19 12-25 32-13 52"/><path fill="#E62117" d="M110 427l21-9c5-2 7-8 5-13l-42-94c-3-6-9-9-15-6l-11 5c-6 2-9 9-7 15l36 97c2 5 8 7 13 5z"/><path fill="#B0BEC5" d="M37 278l41-17c11-4 22-5 33-1 5 2 10 4 14 6 6 3 4 11-3 11-9 0-18 1-26 3l2 12c1 6-2 11-8 13l-36 15c-5 2-10 1-14-2l-9-7-2 17c0 2-2 4-4 5l-3 1c-3 1-7 0-8-3L1 300c-1-3 0-7 4-9l4-2c2-1 5 0 7 1l12 10 1-11c0-5 3-9 8-11z"/><path fill="#F7CB4D" d="M103 373c10 2 14 10 8 19 6-1 10 4 10 9 0 3-3 6-6 7l-26 11c-2 1-5 1-8 0-6-3-7-9-2-16-7-1-13-9-6-17-8-1-12-8-8-15l3-3 23-11c9-4 19 8 12 16z"/><ellipse transform="rotate(173.3 233.455 334.51)" fill="#8E24AA" cx="234" cy="335" rx="32" ry="46"/></svg></p><style nonce="LsULlnAmv0wUvxaLPUzdQg">#yt-masthead{margin:15px auto;width:440px;margin-top:25px}#logo-container{margin-right:5px;float:left;cursor:pointer;text-decoration:none}.logo{background:center/contain no-repeat url(//www.gstatic.com/youtube/img/branding/youtubelogo/2x/youtubelogo_50.png);width:128px;height:30px;cursor:pointer;display:inline-block}#masthead-search{display:-webkit-box;display:-webkit-flex;display:flex;margin-top:3px;max-width:650px;overflow:hidden;padding:0;position:relative}.search-button{border-left:0;border-top-left-radius:0;border-bottom-left-radius:0;float:right;height:29px;padding:0;border:solid 1px transparent;border-color:#d3d3d3;background:#f8f8f8;color:#333;cursor:pointer}.search-button:hover{border-color:#c6c6c6;background:#f0f0f0;-webkit-box-shadow:0 1px 0 rgba(0,0,0,.1);box-shadow:0 1px 0 rgba(0,0,0,.1)}.search-button-content{border:none;display:block;opacity:.6;padding:0;text-indent:-10000px;background:no-repeat url(//www.gstatic.com/youtube/src/web/htdocs/img/search.png);-webkit-background-size:auto auto;background-size:auto;width:15px;height:15px;-webkit-box-shadow:none;box-shadow:none;margin:0 25px}#masthead-search-terms-border{-webkit-box-flex:1;-webkit-flex:1 1 auto;flex:1 1 auto;border:1px solid #ccc;-webkit-box-shadow:inset 0 1px 2px #eee;box-shadow:inset 0 1px 2px #eee;background-color:#fff;font-size:14px;height:29px;line-height:30px;margin:0 0 2px;overflow:hidden;position:relative;-webkit-box-sizing:border-box;box-sizing:border-box;-webkit-transition:border-color .2s ease;transition:border-color .2s ease}#masthead-search-terms{background:transparent;border:0;font-size:16px;height:100%;left:0;margin:0;outline:none;padding:2px 6px;position:absolute;width:100%;-webkit-box-sizing:border-box;box-sizing:border-box}sentinel{}</style><div id="yt-masthead"><a id="logo-container" href="https://www.youtube.com/" title="YouTube home"><span class="logo" title="YouTube home"></span></a><form id="masthead-search" class="search-form" action="https://www.youtube.com/results"><script nonce="59w79Sa6-bR4tg6OjM5PsA">document.addEventListener(\'DOMContentLoaded\', function () {document.getElementById(\'masthead-search\').addEventListener(\'submit\', function(e) {if (document.getElementById(\'masthead-search-terms\').value == \'\') {e.preventDefault();}});});</script><div id="masthead-search-terms-border" dir="ltr"><input id="masthead-search-terms" autocomplete="off" name="search_query" value="" type="text" placeholder="Search" title="Search" aria-label="Search"><script nonce="59w79Sa6-bR4tg6OjM5PsA">document.addEventListener(\'DOMContentLoaded\', function () {document.getElementById(\'masthead-search-terms\').addEventListener(\'keydown\', function() {if (!this.value && (event.keyCode == 40 || event.keyCode == 32 || event.keyCode == 34)) {this.onkeydown = null; this.blur();}});});</script></div><button id="masthead-search-button" class="search-button" type="submit" dir="ltr"><script nonce="59w79Sa6-bR4tg6OjM5PsA">document.addEventListener(\'DOMContentLoaded\', function () {document.getElementById(\'masthead-search-button\').addEventListener(\'click\', function(e) {if (document.getElementById(\'masthead-search-terms\').value == \'\') {e.preventDefault(); return;}e.preventDefault(); document.getElementById(\'masthead-search\').submit();});});</script><span class="search-button-content">Search</span></button></form></div></div></body></html>'

yt_res.text is a string containing HTML – we can render this in-line using IPython.display.HTML.

In [65]:
from IPython.display import HTML
In [66]:
HTML(yt_res.text)
Out[66]:
Oops

Something went wrong

HTTP status codes¶

  • When we request data from a website, the server includes an HTTP status code in the response.
  • The most common status code is 200, which means there were no issues.
  • Other times, you will see a different status code, describing some sort of event or error.
    • Common examples: 400 – bad request, 404 – page not found, 500 – internal server error.
    • The first digit of a status describes its general "category".
  • See https://httpstat.us for a list of all HTTP status codes.
    • It also has example sites for each status code; for example, https://httpstat.us/404 returns a 404.
In [67]:
yt_res.status_code
Out[67]:
400
In [68]:
# ok checks if the result was successful.
yt_res.ok
Out[68]:
False

Handling unsuccessful requests¶

  • Unsuccessful requests can be re-tried, depending on the issue.

    • A good first step is to wait a little, then try again.

  • A common issue is that you're making too many requests to a particular server at a time – if this is the case, increase the time between each request. You can even do this programatically, say, using time.sleep.

  • See the textbook for more examples.

Data formats¶

The data formats of the internet¶

Responses typically come in one of two formats: HTML or JSON.

  • The response body of a GET request is usually either JSON (when using an API) or HTML (when accessing a webpage).

  • The response body of a POST request is usually JSON.

  • XML is also a common format, but not as popular as it once was.

JSON¶

  • JSON stands for JavaScript Object Notation. It is a lightweight format for storing and transferring data.

  • It is:

    • very easy for computers to read and write.
    • moderately easy for programmers to read and write by hand.
    • meant to be generated and parsed.

  • Most modern languages have an interface for working with JSON objects.

    • JSON objects resemble Python dictionaries (but are not the same!).

JSON data types¶

Type Description
String Anything inside double quotes.
Number Any number (no difference between ints and floats).
Boolean true and false.
Null JSON's empty value, denoted by null.
Array A collection of comma-separated values, like Python lists.
Object A collection of key-value pairs, like Python dictionaries. Keys must be strings, values can be anything (even other objects).

See json-schema.org for more details.

Example JSON object¶

See data/family.json.

No description has been provided for this image
In [69]:
import json
from pathlib import Path

f = Path('data') / 'family.json'
family_tree = json.loads(f.read_text())
In [70]:
f.read_text()
Out[70]:
'{\n    "name": "Grandma",\n    "age": 94,\n    "children": [\n        {\n        "name": "Dad",\n        "age": 60,\n        "children": [{"name": "Me", "age": 24}, \n                     {"name": "Brother", "age": 22}]\n        },\n        {\n        "name": "Aunt",\n        "children": [{"name": "Cousin 1", "age": 34}, \n                     {"name": "Cousin 2", "age": 36, "children": \n                        [{"name": "Cousin 2 Jr.", "age": 2}]\n                     }\n                    ]\n        }\n    ]\n}'
In [71]:
family_tree
Out[71]:
{'name': 'Grandma',
 'age': 94,
 'children': [{'name': 'Dad',
   'age': 60,
   'children': [{'name': 'Me', 'age': 24}, {'name': 'Brother', 'age': 22}]},
  {'name': 'Aunt',
   'children': [{'name': 'Cousin 1', 'age': 34},
    {'name': 'Cousin 2',
     'age': 36,
     'children': [{'name': 'Cousin 2 Jr.', 'age': 2}]}]}]}
In [72]:
family_tree['children'][0]['children'][1]['age']
Out[72]:
22
In [73]:
family_tree['children'][0]['children'][1]['name']
Out[73]:
'Brother'
In [ ]:
In [74]:
family_tree['children'][1]['children'][0]['age']
Out[74]:
34

Aside: eval¶

  • eval, which stands for "evaluate", is a function built into Python.

  • It takes in a string containing a Python expression and evaluates it in the current context.

In [75]:
x = 4
eval('x + 5')
Out[75]:
9
  • It seems like eval can do the same thing that json.loads does...
In [76]:
eval(f.read_text())
Out[76]:
{'name': 'Grandma',
 'age': 94,
 'children': [{'name': 'Dad',
   'age': 60,
   'children': [{'name': 'Me', 'age': 24}, {'name': 'Brother', 'age': 22}]},
  {'name': 'Aunt',
   'children': [{'name': 'Cousin 1', 'age': 34},
    {'name': 'Cousin 2',
     'age': 36,
     'children': [{'name': 'Cousin 2 Jr.', 'age': 2}]}]}]}
  • But you should almost never use eval. The next slide demonstrates why.

eval gone wrong¶

Observe what happens when we use eval on a string representation of a JSON object:

In [77]:
f_other = Path('data') / 'evil_family.json'
eval(f_other.read_text())
#f_other.read_text()

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[77], line 2
      1 f_other = Path('data') / 'evil_family.json'
----> 2 eval(f_other.read_text())
      3 #f_other.read_text()

File <string>:6

File ~/Documents/DSC-80-TA/dsc80/lectures/wi26/lec09/dsc80_utils.py:81, in err()
     80 def err():
---> 81     raise ValueError('All of your files have been deleted.')

ValueError: All of your files have been deleted.

  • Oh no! Since `evil_family.json contained malicious code, we now lost all of our files! (not really)

  • This happened because eval evaluates all parts of the input string as if it were Python code.

  • You never need to do this – instead, use the .json() method of a response object, or use the json library.

Using the json module¶

Let's process the same file using the json module. Note:

  • json.load(f) loads a JSON file from a file object.
  • json.loads(f) loads a JSON file from a string.
In [78]:
f_other = Path('data') / 'evil_family.json'
s = f_other.read_text()
s
Out[78]:
'{\n    "name": "Grandma",\n    "age": 94,\n    "children": [\n        {\n        "name": err(),\n        "age": 60,\n        "children": [{"name": "Me", "age": 24}, \n                     {"name": "Brother", "age": 22}]\n        },\n        {\n        "name": "Aunt",\n        "children": [{"name": "Cousin 1", "age": 34}, \n                     {"name": "Cousin 2", "age": 36, "children": \n                        [{"name": "Cousin 2 Jr.", "age": 2}]\n                     }\n                    ]\n        }\n    ]\n}'
In [79]:
json.loads(s)

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
Cell In[79], line 1
----> 1 json.loads(s)

File ~/ENTER/envs/dsc80/lib/python3.12/json/__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    341     s = s.decode(detect_encoding(s), 'surrogatepass')
    343 if (cls is None and object_hook is None and
    344         parse_int is None and parse_float is None and
    345         parse_constant is None and object_pairs_hook is None and not kw):
--> 346     return _default_decoder.decode(s)
    347 if cls is None:
    348     cls = JSONDecoder

File ~/ENTER/envs/dsc80/lib/python3.12/json/decoder.py:337, in JSONDecoder.decode(self, s, _w)
    332 def decode(self, s, _w=WHITESPACE.match):
    333     """Return the Python representation of ``s`` (a ``str`` instance
    334     containing a JSON document).
    335 
    336     """
--> 337     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338     end = _w(s, end).end()
    339     if end != len(s):

File ~/ENTER/envs/dsc80/lib/python3.12/json/decoder.py:355, in JSONDecoder.raw_decode(self, s, idx)
    353     obj, end = self.scan_once(s, idx)
    354 except StopIteration as err:
--> 355     raise JSONDecodeError("Expecting value", s, err.value) from None
    356 return obj, end

JSONDecodeError: Expecting value: line 6 column 17 (char 84)

  • Since err() is not a string in JSON (there are no quotes around it), json.loads is not able to parse it as a JSON object.

  • This "safety check" is intentional.

Handling unfamiliar data¶

  • Never trust data from an unfamiliar site.

  • Never use eval on "raw" data that you didn't create!

  • The JSON data format needs to be parsed, not evaluated as a dictionary.

    • It was designed with safety in mind!

Aside: pd.read_json¶

pandas also has a built-in read_json function.

In [80]:
pd.read_json(f)
Out[80]:
name age children
0 Grandma 94 {'name': 'Dad', 'age': 60, 'children': [{'name...
1 Grandma 94 {'name': 'Aunt', 'children': [{'name': 'Cousin...

It only makes sense to use it, though, when you have a JSON file that has some sort of tabular structure. Our family tree example does not.

APIs and scraping¶

Programmatic requests¶

  • We learned how to use the Python requests package to exchange data via HTTP.

    • GET requests are used to request data from a server.
    • POST requests are used to send data to a server.

  • There are two ways of collecting data through a request:

    • By using a published API (application programming interface).
      • This is a convenient and well-defined way to get data.
    • By scraping a webpage to collect its HTML source code.
      • This is the DIY approach, for when an API does not exist.

APIs¶

An application programming interface (API) is a service that makes data directly available to the user in a convenient fashion.

Advantages:

  • The data are usually clean, up-to-date, and ready to use.

  • The presence of an API signals that the data provider is okay with you using their data.

  • The data provider can plan and regulate data usage.

    • Some APIs require you to create an API "key", which is like an account for using the API.
    • APIs can also give you access to data that isn't publicly available on a webpage.

The only problem is that APIs don't always exist for the data you want! In that case, we resort to web scraping.

API terminology¶

  • A URL, or uniform resource locator, describes the location of a website or resource.

  • An API endpoint is a URL of the data source that the user wants to make requests to.

  • For example, on the Reddit API:

    • the /comments endpoint retrieves information about comments.
    • the /hot endpoint retrieves data about posts labeled "hot" right now.
    • To access these endpoints, you add the endpoint name to the base URL of the API.

API requests¶

  • API requests are just GET/POST requests to a specially maintained URL.
  • Let's test out the Pokémon API.

First, let's make a GET request for 'squirtle'. To do this, we need to make a request to the correct URL.

In [81]:
def create_url(pokemon):
    return f'https://pokeapi.co/api/v2/pokemon/{pokemon}'

create_url('squirtle')
Out[81]:
'https://pokeapi.co/api/v2/pokemon/squirtle'
In [82]:
r = requests.get(create_url('squirtle'))
r
Out[82]:
<Response [200]>

Remember, the 200 status code is good! Let's take a look at the content:

In [83]:
r.content[:1000]
Out[83]:
b'{"abilities":[{"ability":{"name":"torrent","url":"https://pokeapi.co/api/v2/ability/67/"},"is_hidden":false,"slot":1},{"ability":{"name":"rain-dish","url":"https://pokeapi.co/api/v2/ability/44/"},"is_hidden":true,"slot":3}],"base_experience":63,"cries":{"latest":"https://raw.githubusercontent.com/PokeAPI/cries/main/cries/pokemon/latest/7.ogg","legacy":"https://raw.githubusercontent.com/PokeAPI/cries/main/cries/pokemon/legacy/7.ogg"},"forms":[{"name":"squirtle","url":"https://pokeapi.co/api/v2/pokemon-form/7/"}],"game_indices":[{"game_index":177,"version":{"name":"red","url":"https://pokeapi.co/api/v2/version/1/"}},{"game_index":177,"version":{"name":"blue","url":"https://pokeapi.co/api/v2/version/2/"}},{"game_index":177,"version":{"name":"yellow","url":"https://pokeapi.co/api/v2/version/3/"}},{"game_index":7,"version":{"name":"gold","url":"https://pokeapi.co/api/v2/version/4/"}},{"game_index":7,"version":{"name":"silver","url":"https://pokeapi.co/api/v2/version/5/"}},{"game_index":7,"v'

Looks like JSON. We can extract the JSON from this request with the json method (or by passing r.text to json.loads).

In [84]:
pokemon = r.json()
pokemon.keys()
Out[84]:
dict_keys(['abilities', 'base_experience', 'cries', 'forms', 'game_indices', 'height', 'held_items', 'id', 'is_default', 'location_area_encounters', 'moves', 'name', 'order', 'past_abilities', 'past_stats', 'past_types', 'species', 'sprites', 'stats', 'types', 'weight'])
In [85]:
pokemon['abilities'][1]['ability']['url']
Out[85]:
'https://pokeapi.co/api/v2/ability/44/'
In [86]:
pokemon['abilities'][1]['ability']['name']
Out[86]:
'rain-dish'

Let's try a GET request for 'janine'.

In [87]:
r = requests.get(create_url('janine'))
r
Out[87]:
<Response [404]>

We receive a 404 error, since there is no Pokemon named 'janine'!

Scraping¶

Scraping is the act of programmatically "browsing" the web, downloading the source code (HTML) of pages that you're interested in extracting data from.

Big advantage: You can always do it! For example, Google scrapes webpages in order to make them searchable.

Disadvantages:

  • It is often difficult to parse and clean scraped data.

    • Source code often includes a lot of content unrelated to the data you're trying to find (e.g. formatting, advertisements, other text).

  • Websites can change often, so scraping code can get outdated quickly.

  • Websites may not want you to scrape their data!

  • In general, we prefer APIs, but scraping is a useful skill to learn.

Example: Scraping the HDSI faculty page¶

To fully understand how to scrape, we need to understand how HTML documents are structured and how to extract information out of them.

But as a preview of what's to come next week, let's start by making a request to the HDSI Faculty page, https://datascience.ucsd.edu/faculty.

In [88]:
# Sometimes, the requests library gets weirdly strict about the HDSI webpage,
# so we'll skip its security checks using verify=False.
fac_response = requests.get('https://datascience.ucsd.edu/faculty/', verify=False)
fac_response

/Users/msgol/ENTER/envs/dsc80/lib/python3.12/site-packages/urllib3/connectionpool.py:1099: InsecureRequestWarning: Unverified HTTPS request is being made to host 'datascience.ucsd.edu'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Out[88]:
<Response [200]>

The response is a long HTML document.

In [89]:
len(fac_response.text)
Out[89]:
256864
In [90]:
print(fac_response.text[:1000])

<!DOCTYPE html><html lang="en-US"><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta http-equiv="X-UA-Compatible" content="IE=edge"><link rel="profile" href="https://gmpg.org/xfn/11" /><title>Faculty &#8211; Halıcıoğlu Data Science Institute &#8211; UC San Diego</title> <script defer src="data:text/javascript;base64,CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAvKiBZb3UgY2FuIGFkZCBtb3JlIGNvbmZpZ3VyYXRpb24gb3B0aW9ucyB0byB3ZWJmb250bG9hZGVyIGJ5IHByZXZpb3VzbHkgZGVmaW5pbmcgdGhlIFdlYkZvbnRDb25maWcgd2l0aCB5b3VyIG9wdGlvbnMgKi8KICAgICAgICAgICAgICAgICAgICAgICAgICAgIGlmICggdHlwZW9mIFdlYkZvbnRDb25maWcgPT09ICJ1bmRlZmluZWQiICkgewogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFdlYkZvbnRDb25maWcgPSBuZXcgT2JqZWN0KCk7CiAgICAgICAgICAgICAgICAgICAgICAgICAgICB9CiAgICAgICAgICAgICAgICAgICAgICAgICAgICBXZWJGb250Q29uZmlnWydnb29nbGUnXSA9IHtmYW1pbGllczogWydSb2JvdG86NDAwLDUwMCcsICdKb3N0OjQwMCw3MDAnLCAnVGVrbzo3MDAnXX07CgogICAgICAgICAgICAgICAgICAgICAgICAgICAgKGZ1bmN0aW9uKCk

To parse HTML, we'll use the BeautifulSoup library.

In [91]:
from bs4 import BeautifulSoup
soup = BeautifulSoup(fac_response.text)

Now, soup is a representation of the faculty page's HTML code that Python knows how to extract information from.

In [92]:
# Magic for now. We'll learn what this means next class.
divs = soup.find_all('div', class_='vc_grid-item')
names = [div.find('h4').text for div in divs]
titles = [div.find(class_='pendari_people_title').text for div in divs]

faculty = pd.DataFrame({
    'name': names, 
    'title': titles, 
})
faculty.head()
Out[92]:
name title
0 Ilkay Altintas SDSC Chief Data Science Officer & HDSI Foundin...
1 Tiffany Amariuta Assistant Professor
2 Mikio Aoi Assistant Professor
3 Ery Arias-Castro Professor
4 Vineet Bafna Professor

Now we have a DataFrame!

In [93]:
faculty[faculty['title'].str.contains('Lecturer') | faculty['title'].str.contains('Teaching')]
Out[93]:
name title
7 Peter Chi Associate Teaching Professor
14 Justin Eldridge Associate Teaching Professor
15 Shannon Ellis Associate Teaching Professor
... ... ...
40 Giorgio Quer Lecturer
48 Jack Silberman Lecturer
52 Janine Tiefenbruck Lecturer

9 rows × 2 columns

What if we want to get faculty members' pictures?

In [94]:
from IPython.display import Image, display

def show_picture(name):
    idx = faculty[faculty['name'].str.lower().str.contains(name.lower())].index[0]
    display(Image(url=divs[idx].find('img')['src'], width=200, height=200))
In [95]:
show_picture('justin')
No description has been provided for this image

Best practices for scraping¶

  1. Send requests slowly and be upfront about what you are doing!
  2. Respect the policy published in the page's robots.txt file.
    • Many sites have a robots.txt file in their root directory, which contains a policy that allows or disallows automatic access to their site.
    • If there isn't one, like in Project 3, use a 0.5 second delay between requests.
  3. Don't spoof your user-agent (i.e. don't try to trick the server into thinking your automated requests are coming from different sources).
  4. Read the Terms of Service for the site and follow it.

Consequences of irresponsible scraping¶

If you make too many requests:

  • The server may block your IP Address.
  • You may take down the website.
    • A journalist scraped and accidentally took down the Cook County Inmate Locater.
    • As a result, inmate's families weren't able to contact them while the site was down.

The anatomy of HTML documents¶

What is HTML?¶

  • HTML (HyperText Markup Language) is the basic building block of the internet.
  • It defines the content and layout of a webpage, and as such, it is what you get back when you scrape a webpage.
  • See this tutorial for more details.

For instance, here's the content of a very basic webpage.

In [96]:
# Mac or Linux:
!cat data/example1.html

# Windows:
# type data\example1.html

<html>
  <head>
    <title>Page title</title>
  </head>

  <body>
    <h1>This is a heading</h1>
    <p>This is a paragraph.</p>
    <p>This is <b>another</b> paragraph.</p>
  </body>
</html>

Using IPython.display.HTML, we can render it directly in our notebook.

In [97]:
from IPython.display import HTML
HTML(filename=Path('data') / 'example1.html')
Out[97]:
Page title

This is a heading

This is a paragraph.

This is another paragraph.

The anatomy of HTML documents¶

  • HTML document: The totality of markup that makes up a webpage.

  • Document Object Model (DOM): The internal representation of an HTML document as a hierarchical tree structure.

  • HTML element: An object in the DOM, such as a paragraph, header, or title.

  • HTML tags: Markers that denote the start and end of an element, such as <p> and </p>.

No description has been provided for this image
(source)

Useful tags to know¶

Element Description
<html> the document
<head> the header
<body> the body
<div> a logical division of the document
<span> an inline logical division
<p> a paragraph
<a> an anchor (hyperlink)
<h1>, <h2>, ... header(s)
<img> an image

There are many, many more, but these are by far the most common. See this article for examples.

Example: Images and hyperlinks¶

Tags can have attributes, which further specify how to display information on a webpage.

For instance, <img> tags have src and alt attributes (among others):

<img src="king-selfie.png" alt="A photograph of King Triton." width=500>

Hyperlinks have href attributes:

Click <a href="https://practice.dsc80.com">this link</a> to access past exams.

What do you think this webpage looks like?

In [98]:
# Mac or Linux:
!cat data/example2.html

# Windows:
# !type data\example2.html

<html>
  <head>
    <title>Example</title>
  </head>

  <body>
    <h1>Foodie Faces</h1>
    <img src="../imgs/lunch.jpg" width="200" alt="avocado toast face" />
    <img src="../imgs/breakfast.jpg" width="200" alt="Trump oatmeal" />
    <p>
      These meals were inspired by the book Foodie Faces by Bill Wurtzel and Claire Wurtzel, available on
      <a href="https://www.amazon.com/Foodie-Faces-Bill-Wurtzel/dp/0316423548"
        >Amazon</a
      >.
    </p>
    <center>
      <h3>
        Hope this made you smile!
      </h3>
    </center>
  </body>
</html>

The <div> tag¶

<div style="background-color:lightblue">
  <h3>This is a heading</h3>
  <p>This is a paragraph.</p>
</div>

  • The <div> tag defines a division or a "section" of an HTML document.

    • Think of a <div> as a "cell" in a Jupyter Notebook.

  • The <div> element is often used as a container for other HTML elements to style them with CSS or to perform operations involving them using JavaScript.

  • <div> elements often have attributes, which are important when scraping!

Document trees¶

Under the document object model (DOM), HTML documents are trees. In DOM trees, child nodes are ordered.

No description has been provided for this image

What does the DOM tree look like for this document?

No description has been provided for this image

Summary, next time¶

Summary: APIs vs. scraping¶

  • APIs are made by organizations that host data.
    • For example, X (formally known as Twitter) has an API.
    • APIs provide a code-friendly way to access data.
    • Usually, APIs provide data as JSON objects.
  • Scraping is the act of emulating a web browser to access its source code.
    • As you'll see in Lab 5, it's not technically supported by most organizations.
    • When scraping, you get back data as HTML and have to parse that HTML to extract the information you want.

Next time¶

We'll learn the details of web scraping using BeautifulSoup.