In [1]:

import pandas as pd
import numpy as np
import re

pd.options.plotting.backend = 'plotly'

import util

Lecture 18 – Regular Expressions, Bag of Words¶

DSC 80, Spring 2023¶

Agenda¶

More regular expressions, including how to use them in Python.
- Example: Log parsing.
- Limitations.
- Remember to look here](dsc80.com/resources/#regular-expressions for resources.
Text features.
Bag of words 💰.

More regular expressions¶

Even more regex syntax¶

operation	example	matches ✅	does not match ❌
escape character	`ucsd\.edu`	`'ucsd.edu'`	`'ucsd!edu'`
beginning of line	`^ark`	`'ark two'` `'ark o ark'`	`'dark'`
end of line	`ark$`	`'dark'` `'ark o ark'`	`'ark two'`
zero or one	`cat?`	`'ca'` `'cat'`	`'cart'` (matches `'ca'` only)
built-in character classes*	`\w+` `\d+`	`'billy'` `'231231'`	`'this person'` `'858 people'`
character class negation	`[^a-z]+`	`'KINGTRITON551'` `'1721$$'`	`'porch'` `'billy.edu'`

Example (built-in character classes)¶

*Note: in Python's implementation of regex,

\d refers to digits.
\w refers to alphanumeric characters ([A-Z][a-z][0-9]_).
\s refers to whitespace.
\b is a word boundary.

What does \d{3} \d{3}-\d{4} match?
What does \bcat\b match? Does it find a match in 'my cat is hungry'? What about 'concatenate'?

Exercise¶

Write a regular expression that matches any string that:

is between 5 and 10 characters long, and
is made up of only vowels (either uppercase or lowercase, including 'Y' and 'y'), periods, and spaces.

Examples include 'yoo.ee.IOU' and 'AI.I oey'.

✅ Click here to see the answer after you've tried it yourself at regex101.com.

One answer: ^[aeiouyAEIOUY. ]{5,10}$
Key idea: Within a character class (i.e. [...]), special characters do not generally need to be escaped.

Regex in Python¶

`re` in Python¶

The re package is built into Python. It allows us to use regular expressions to find, extract, and replace strings.

In [2]:

import re

re.search takes in a string regex and a string text and returns the location and substring corresponding to the first match of regex in text.

In [3]:

re.search('AB*A', 
          'here is a string for you: ABBBA. here is another: ABBBBBBBA')

Out[3]:

<re.Match object; span=(26, 31), match='ABBBA'>

re.findall takes in a string regex and a string text and returns a list of all matches of regex in text. You'll use this most often.

In [4]:

re.findall('AB*A', 
           'here is a string for you: ABBBA. here is another: ABBBBBBBA')

Out[4]:

['ABBBA', 'ABBBBBBBA']

re.sub takes in a string regex, a string repl, and a string text, and replaces all matches of regex in text with repl.

In [5]:

re.sub('AB*A', 
       'billy', 
       'here is a string for you: ABBBA. here is another: ABBBBBBBA')

Out[5]:

'here is a string for you: billy. here is another: billy'

Raw strings¶

When using regular expressions in Python, it's a good idea to use raw strings, denoted by an r before the quotes, e.g. r'exp'.

In [6]:

re.findall('\bcat\b', 'my cat is hungry')

Out[6]:

[]

In [7]:

re.findall(r'\bcat\b', 'my cat is hungry')

Out[7]:

['cat']

In [8]:

# Huh?
print('\bcat\b')

cat

Capture groups¶

Surround a regex with ( and ) to define a capture group within a pattern.
Capture groups are useful for extracting relevant parts of a string.

In [9]:

re.findall(r'\w+@(\w+)\.edu', 
           'my old email was billy@notucsd.edu, my new email is notbilly@ucsd.edu')

Out[9]:

['notucsd', 'ucsd']

Notice what happens if we remove the ( and )!

In [10]:

re.findall(r'\w+@\w+\.edu', 
           'my old email was billy@notucsd.edu, my new email is notbilly@ucsd.edu')

Out[10]:

['billy@notucsd.edu', 'notbilly@ucsd.edu']

Earlier, we also saw that parentheses can be used to group parts of a regex together. When using re.findall, all groups are treated as capturing groups.

In [11]:

# A regex that matches strings with two of the same vowel followed by 3 digits
# We only want to capture the digits, but...
re.findall(r'(aa|ee|ii|oo|uu)(\d{3})', 'eeoo124')

Out[11]:

[('oo', '124')]

Example: Log parsing¶

Web servers typically record every request made of them in the "logs".

In [12]:

s = '''132.249.20.188 - - [24/Feb/2023:12:26:15 -0800] "GET /my/home/ HTTP/1.1" 200 2585'''

Let's use our new regex syntax (including capturing groups) to extract the day, month, year, and time from the log string s.

In [13]:

exp = '\[(.+)\/(.+)\/(.+):(.+):(.+):(.+) .+\]'
re.findall(exp, s)

Out[13]:

[('24', 'Feb', '2023', '12', '26', '15')]

While above regex works, it is not very specific. It works on incorrectly formatted log strings.

In [14]:

other_s = '[adr/jduy/wffsdffs:r4s4:4wsgdfd:asdf 7]'
re.findall(exp, other_s)

Out[14]:

[('adr', 'jduy', 'wffsdffs', 'r4s4', '4wsgdfd', 'asdf')]

The more specific, the better!¶

Be as specific in your pattern matching as possible – you don't want to match and extract strings that don't fit the pattern you care about.
- .* matches every possible string, but we don't use it very often.

A better date extraction regex:
```
\[(\d{2})\/([A-Z]{1}[a-z]{2})\/(\d{4}):(\d{2}):(\d{2}):(\d{2}) -\d{4}\]
```
- \d{2} matches any 2-digit number.
- [A-Z]{1} matches any single occurrence of any uppercase letter.
- [a-z]{2} matches any 2 consecutive occurrences of lowercase letters.
- Remember, special characters ([, ], /) need to be escaped with \.

In [15]:

Out[15]:

'132.249.20.188 - - [24/Feb/2023:12:26:15 -0800] "GET /my/home/ HTTP/1.1" 200 2585'

In [16]:

new_exp = '\[(\d{2})\/([A-Z]{1}[a-z]{2})\/(\d{4}):(\d{2}):(\d{2}):(\d{2}) -\d{4}\]'
re.findall(new_exp, s)

Out[16]:

[('24', 'Feb', '2023', '12', '26', '15')]

A benefit of new_exp over exp is that it doesn't capture anything when the string doesn't follow the format we specified.

In [17]:

other_s

Out[17]:

'[adr/jduy/wffsdffs:r4s4:4wsgdfd:asdf 7]'

In [18]:

re.findall(new_exp, other_s)

Out[18]:

[]

Limitations¶

Limitations of regexes¶

Writing a regular expression is like writing a program.

You need to know the syntax well.
They can be easier to write than to read.
They can be difficult to debug.

Regular expressions are terrible at certain types of problems. Examples:

Anything involving counting (same number of instances of a and b).
Anything involving complex structure (palindromes).
Parsing highly complex text structure (HTML, for instance).

Below is a regular expression that validates email addresses in Perl. See this article for more details.

StackOverflow crashed due to regex! See this article for the details.

Text features¶

Review: Regression and features¶

In DSC 40A, our running example was to use regression to predict a data scientist's salary, given their GPA, years of experience, and years of education.

After minimizing empirical risk to determine optimal parameters, $w_0^*, \dots, w_3^*$, we made predictions using:

$$\text{predicted salary} = w_0^* + w_1^* \cdot \text{GPA} + w_2^* \cdot \text{experience} + w_3^* \cdot \text{education}$$

GPA, years of experience, and years of education are features – they represent a data scientist as a vector of numbers.
- e.g. Your feature vector may be [3.5, 1, 7].

This approach requires features to be numerical.

Moving forward¶

Suppose we'd like to predict the sentiment of a piece of text from 1 to 10.

10: Very positive (happy).
1: Very negative (sad, angry).

Example:

Input: "DSC 80 is a pretty good class."
Output: 7.

We can frame this as a regression problem, but we can't directly use what we learned in 40A, because here our inputs are text, not numbers.

Text features¶

Big question: How do we represent a text document as a feature vector of numbers?

If we can do this, we can:
- use a text document as input in a regression or classification model (in a few lectures).
- quantify the similarity of two text documents (today).

Example: San Diego employee salaries¶

Transparent California publishes the salaries of all City of San Diego employees.
The latest available data is from 2021.

In [19]:

salaries = pd.read_csv('https://transcal.s3.amazonaws.com/public/export/san-diego-2021.csv')
util.anonymize_names(salaries)

In [20]:

salaries.head()

Out[20]:

	Employee Name	Job Title	Base Pay	Other Pay	Benefits	Total Pay	Pension Debt	Total Pay & Benefits	Year	Notes	Agency	Status
0	Mara Xxxx	City Attorney	218759.0	-2560.00	108652.0	216199.0	427749.18	752600.18	2021	NaN	San Diego	FT
1	Todd Xxxx	Mayor	218759.0	-81.00	95549.0	218678.0	427749.18	741976.18	2021	NaN	San Diego	FT
2	Elizabeth Xxxx	Investment Officer	259732.0	-870.00	71438.0	258862.0	221041.09	551341.09	2021	NaN	San Diego	FT
3	Terence Xxxx	Police Officer	212837.0	39683.00	56569.0	252520.0	222375.06	531464.06	2021	NaN	San Diego	FT
4	Andrea Xxxx	Independent Budget Analyst	224312.0	59819.00	54213.0	284131.0	192126.79	530470.79	2021	NaN	San Diego	FT

Aside on privacy and ethics¶

Even though the data we downloaded is publicly available, employee names still correspond to real people.

Be careful when dealing with PII (personably identifiable information).
- Only work with the data that is needed for your analysis.
- Even when data is public, people have a reasonable right to privacy.

Remember to think about the impacts of your work outside of your Jupyter Notebook.

Goal: Quantifying similarity¶

Our goal is to describe, numerically, how similar two job titles are.

For instance, our similarity metric should tell us that 'Deputy Fire Chief' and 'Fire Battalion Chief' are more similar than 'Deputy Fire Chief' and 'City Attorney'.

Idea: Two job titles are similar if they contain shared words, regardless of order. So, to measure the similarity between two job titles, let's count the number of words they share in common.

Before we do this, we need to be confident that the job titles are clean and consistent – let's explore.

Exploring job titles¶

In [21]:

jobtitles = salaries['Job Title']
jobtitles.head()

Out[21]:

0                 City Attorney
1                         Mayor
2            Investment Officer
3                Police Officer
4    Independent Budget Analyst
Name: Job Title, dtype: object

How many employees are in the dataset? How many unique job titles are there?

In [22]:

jobtitles.shape[0], jobtitles.nunique()

Out[22]:

(12305, 588)

What are the most common job titles?

In [23]:

jobtitles.value_counts().iloc[:100]

Out[23]:

Police Officer                   2123
Fire Fighter Ii                   331
Assistant Engineer - Civil        284
Grounds Maintenance Worker Ii     250
Fire Captain                      248
                                 ... 
Grounds Maintenance Manager        27
Electrician                        27
Executive Assistant                26
Paralegal                          26
Librarian Iv                       25
Name: Job Title, Length: 100, dtype: int64

In [24]:

jobtitles.value_counts().iloc[:25].sort_values().plot(kind='barh')

Lecture 18 – Regular Expressions, Bag of Words¶

DSC 80, Spring 2023¶

Agenda¶

More regular expressions¶

Even more regex syntax¶

Example (built-in character classes)¶

Exercise¶

Regex in Python¶

re in Python¶

Raw strings¶

Capture groups¶

Example: Log parsing¶

The more specific, the better!¶

Limitations¶

Limitations of regexes¶

Text features¶

Review: Regression and features¶

Moving forward¶

Text features¶

Example: San Diego employee salaries¶

Aside on privacy and ethics¶

Goal: Quantifying similarity¶

Exploring job titles¶

`re` in Python¶