# Lecture 17 – Regular Expressions¶

## DSC 80, Winter 2023¶

### 📣 Announcements¶

• Lab 6 (web scraping and APIs) is due today at 4:00PM (no slip days!).
• No slip days are allowed because we will take up the solutions in Discussion 6 at 5:00PM today.
• Project 3 is due tomorrow at 11:59PM.
• Midterm Exam scores are released, and regrades are due tonight at 11:59PM.
• Remember that it's only worth 10%, and that we have a redemption policy.
• Lab 7 (regular expressions and text features) is due on Monday, February 27th at 11:59PM.
• Followup from last class: BeautifulSoup objects are mutable! See this post on Ed by Trey for more details.

### Agenda¶

Lots and lots of regular expressions! Good resources:

## Motivation¶

### Who called? 📞¶

• Goal: Extract all phone numbers from a piece of text, assuming they are of the form '(###) ###-####'.
• We can do this using the same string methods we've come to know and love.
• Strategy:
• Split by spaces.
• Check if there are any consecutive "words" where:
• the first "word" looks like an area code, like '(678)'.
• the second "word" looks like the last 7 digits of a phone number, like '999-8212'.

Let's first write a function that takes in a string and returns whether it looks like an area code.

Let's also write a function that takes in a string and returns whether it looks like the last 7 digits of a phone number.

Finally, let's split the entire text by spaces, and check whether there are any instances where pieces[i] looks like an area code and pieces[i+1] looks like the last 7 digits of a phone number.

### Is there a better way?¶

• This was an example of pattern matching.
• It can be done with string methods, but there is often a better approach: regular expressions.

## Basic regular expressions¶

### Regular expressions¶

• A regular expression, or regex for short, is a sequence of characters used to match patterns in strings.
• For example, $\d{3}$ \d{3}-\d{4} describes a pattern that matches US phone numbers of the form '(XXX) XXX-XXXX'.
• Think of regex as a "mini-language" (formally: they are a grammar for describing a language).
• Pros: They are very powerful and are widely used (virtually every programming language has a module for working with them).
• Cons: They can be hard to read and have many different "dialects."

### Writing regular expressions¶

• You will ultimately write most of your regular expressions in Python, using the re module. We will see how to do so shortly.
• However, a useful tool for designing regular expressions is regex101.com.
• We will use it heavily during lecture; you should have it open as we work through examples. If you're trying to revisit this lecture in the future, you'll likely want to watch the podcast.

### Literals¶

• A literal is a character that has no special meaning.
• Letters, numbers, and some symbols are all literals.
• Some symbols, like ., *, (, and ), are special characters.
• Example: The regex hey matches the string 'hey'. The regex he. also matches the string 'hey'.

### Regex building blocks 🧱¶

The four main building blocks for all regexes are shown below (table source, inspiration).

operation order of op. example matches ✅ does not match ❌
concatenation 3 AABAAB 'AABAAB' every other string
or 4 AA|BAAB 'AA', 'BAAB' every other string
closure
(zero or more)
2 AB*A 'AA', 'ABBBBBBA' 'AB', 'ABABA'
parentheses 1 A(A|B)AAB
(AB)*A
'AAAAB', 'ABAAB'
'A', 'ABABABABA'
every other string
'AA', 'ABBA'

Note that |, (, ), and * are special characters, not literals. They manipulate the characters around them.

Example (or, parentheses):

• What does DSC 30|80 match?
• What does DSC (30|80) match?

Example (closure, parentheses):

• What does blah* match?
• What does (blah)* match?

### Exercise¶

Write a regular expression that matches 'billy', 'billlly', 'billlllly', etc.

• First, think about how to match strings with any even number of 'l's, including zero 'l's (i.e. 'biy').
• Then, think about how to match only strings with a positive even number of 'l's.

✅ Click here to see the answer after you've tried it yourself at regex101.com. bi(ll)*y will match any even number of 'l's, including 0. To match only a positive even number of 'l's, we'd need to first "fix into place" two 'l's, and then follow that up with zero or more pairs of 'l's. This specifies the regular expression bill(ll)*y.

### Exercise¶

Write a regular expression that matches 'billy', 'billlly', 'biggy', 'biggggy', etc.

Specifically, it should match any string with a positive even number of 'l's in the middle, or a positive even number of 'g's in the middle.

✅ Click here to see the answer after you've tried it yourself at regex101.com. Possible answers: bi(ll(ll)\*|gg(gg)\*)y or bill(ll)\*y|bigg(gg)\*y.
Note, bill(ll)\*|gg(gg)\*y is not a valid answer! This is because "concatenation" comes before "or" in the order of operations. This regular expression would match strings that match bill(ll)\*, like 'billll', OR strings that match gg(gg)\*y, like 'ggy'.

## Intermediate regex¶

### More regex syntax¶

operation example matches ✅ does not match ❌
wildcard .U.U.U. 'CUMULUS'
'JUGULUM'
'SUCCUBUS'
'TUMULTUOUS'
character class [A-Za-z][a-z]* 'word'
'Capitalized'
'camelCase'
'4illegal'
at least one bi(ll)+y 'billy'
'billlllly'
'biy'
'bily'
between $i$ and $j$ occurrences m[aeiou]{1,2}m 'mem'
'maam'
'miem'
'mm'
'mooom'
'meme'

., [, ], +, {, and } are also special characters, in addition to |, (, ), and *.

Example (character classes, at least one): [A-E]+ is just shortform for (A|B|C|D|E)(A|B|C|D|E)*.

Example (wildcard):

• What does . match?
• What does he. match?
• What does ... match?

Example (at least one, closure):

• What does 123+ match?
• What does 123* match?

Example (number of occurrences): What does tri{3, 5} match? Does it match 'triiiii'?

Example (character classes, number of occurrences): What does [1-6a-f]{3}-[7-9E-S]{2} match?

### Exercise¶

Write a regular expression that matches any lowercase string has a repeated vowel, such as 'noon', 'peel', 'festoon', or 'zeebraa'.

✅ Click here to see the answer after you've tried it yourself at regex101.com. One answer: [a-z]\*(aa|ee|ii|oo|uu)[a-z]\*
This regular expression matches strings of lowercase characters that have 'aa', 'ee', 'ii', 'oo', or 'uu' in them anywhere. [a-z]\* means "zero or more of any lowercase characters"; essentially we are saying it doesn't matter what letters come before or after the double vowels, as long as the double vowels exist somewhere.

### Exercise¶

Write a regular expression that matches any string that contains both a lowercase letter and a number, in any order. Examples include 'billy80', '80!!billy', and 'bil8ly0'.

✅ Click here to see the answer after you've tried it yourself at regex101.com. One answer: (.\*[a-z].\*[0-9].\*)|(.\*[0-9].\*[a-z].\*)
We can break the above regex into two parts – everything before the |, and everything after the |. The first part, .\*[a-z].\*[0-9].\*, matches strings in which there is at least one lowercase character and at least one digit, with the lowercase character coming first. The second part, .\*[0-9].\*[a-z].\*, matches strings in which there is at least one lowercase character and at least one digit, with the digit coming first. Note, the .\* between the digit and letter classes is needed in the event the string has non-digit and non-letter characters. This is the kind of task that would be easier to accomplish with regular Python string methods.

### Even more regex syntax¶

operation example matches ✅ does not match ❌
escape character ucsd\.edu 'ucsd.edu' 'ucsd!edu'
beginning of line ^ark 'ark two'
'ark o ark'
'dark'
end of line ark$ 'dark' 'ark o ark' 'ark two' zero or one cat? 'ca' 'cat' 'cart' (matches 'ca' only) built-in character classes* \w+ \d+ 'billy' '231231' 'this person' '858 people' character class negation [^a-z]+ 'KINGTRITON551' '1721$$' 'porch' 'billy.edu' *Note: in Python's implementation of regex, • \d refers to digits. • \w refers to alphanumeric characters ([A-Z][a-z][0-9]_). • \s refers to whitespace. • \b is a word boundary. Example (escaping): • What does he. match? • What does he\. match? • What does (858) match? • What does $858$ match? Example (anchors): • What does 858-534 match? • What does ^858-534 match? • What does 858-534$ match?

### Example (built-in character classes)¶

*Note: in Python's implementation of regex,

• \d refers to digits.
• \w refers to alphanumeric characters ([A-Z][a-z][0-9]_).
• \s refers to whitespace.
• \b is a word boundary.
• What does \d{3} \d{3}-\d{4} match?
• What does \bcat\b match? Does it find a match in 'my cat is hungry'? What about 'concatenate'?

### Exercise¶

Write a regular expression that matches any string that:

• is between 5 and 10 characters long, and
• is made up of only vowels (either uppercase or lowercase, including 'Y' and 'y'), periods, and spaces.

Examples include 'yoo.ee.IOU' and 'AI.I oey'.

✅ Click here to see the answer after you've tried it yourself at regex101.com. One answer: ^[aeiouyAEIOUY. ]{5,10}\$
Key idea: Within a character class (i.e. [...]), special characters do not generally need to be escaped.

## Regex in Python¶

### re in Python¶

The re package is built into Python. It allows us to use regular expressions to find, extract, and replace strings.

re.search takes in a string regex and a string text and returns the location and substring corresponding to the first match of regex in text.

re.findall takes in a string regex and a string text and returns a list of all matches of regex in text. You'll use this most often.

re.sub takes in a string regex, a string repl, and a string text, and replaces all matches of regex in text with repl.

### Raw strings¶

When using regular expressions in Python, it's a good idea to use raw strings, denoted by an r before the quotes, e.g. r'exp'.

### Capture groups¶

• Surround a regex with ( and ) to define a capture group within a pattern.
• Capture groups are useful for extracting relevant parts of a string.
• Notice what happens if we remove the ( and )!
• Earlier, we also saw that parentheses can be used to group parts of a regex together. When using re.findall, all groups are treated as capturing groups.

## Summary, next time¶

### Summary¶

• Regular expressions are used to match and extract patterns from text.
• You don't need to force yourself to "memorize" regex syntax – refer to the resources in the Agenda section of the lecture and on the Resources tab of the course website.
• Also refer to the three tables of syntax in the lecture:
• Note: You don't always have to use regular expressions! If Python/pandas string methods work for your task, you can still use those.
• Play Regex Golf to practice! 🏌️

### Next time¶

• A few more examples of regular expressions.
• Using regular expressions in pandas (through .str).
• Describing text data quantitatively.