from dsc80_utils import *
Announcements 📣¶
- Lab 5 due Monday (not tomorrow, sorry about the mixup!)
- Project 3 checkpoint due Tues.
- Don't wait until the last minute to start Part 3! It takes a while to finish.
Agenda 📆¶
- Most of today's lecture will be about regular expressions. Good resources:
- regex101.com, a helpful site to have open while writing regular expressions.
- Python
re
library documentation and how-to.- The "how-to" is great, read it!
- regex "cheat sheet" (taken from here).
- These are all on the resources tab of the course website as well.
Motivation¶
contact = '''
Thank you for buying our expensive product!
If you have a complaint, please send it to complaints@compuserve.com or call (800) 867-5309.
If you are happy with your purchase, please call us at (800) 123-4567; we'd love to hear from you!
Due to high demand, please allow one-hundred (100) business days for a response.
'''
Who called? 📞¶
- Goal: Extract all phone numbers from a piece of text, assuming they are of the form
'(###) ###-####'
.
print(contact)
Thank you for buying our expensive product! If you have a complaint, please send it to complaints@compuserve.com or call (800) 867-5309. If you are happy with your purchase, please call us at (800) 123-4567; we'd love to hear from you! Due to high demand, please allow one-hundred (100) business days for a response.
We can do this using the same string methods we've come to know and love.
Strategy:
- Split by spaces.
- Check if there are any consecutive "words" where:
- the first "word" looks like an area code, like
'(678)'
. - the second "word" looks like the last 7 digits of a phone number, like
'999-8212'
.
- the first "word" looks like an area code, like
Let's first write a function that takes in a string and returns whether it looks like an area code.
def is_possibly_area_code(s):
'''Does `s` look like (678)?'''
return (len(s) == 5 and
s.startswith('(') and
s.endswith(')') and
s[1:4].isnumeric())
is_possibly_area_code('(123)')
True
is_possibly_area_code('(99)')
False
Let's also write a function that takes in a string and returns whether it looks like the last 7 digits of a phone number.
def is_last_7_phone_number(s):
'''Does `s` look like 999-8212?'''
return len(s) == 8 and s[0:3].isnumeric() and s[3] == '-' and s[4:].isnumeric()
is_last_7_phone_number('999-8212')
True
is_last_7_phone_number('534 1100')
False
Finally, let's split the entire text by spaces, and check whether there are any instances where pieces[i]
looks like an area code and pieces[i+1]
looks like the last 7 digits of a phone number.
# Removes punctuation from the end of each string.
pieces = [s.rstrip('.,?;"\'') for s in contact.split()]
for i in range(len(pieces) - 1):
if is_possibly_area_code(pieces[i]):
if is_last_7_phone_number(pieces[i+1]):
print(pieces[i], pieces[i+1])
(800) 867-5309 (800) 123-4567
Is there a better way?¶
- This was an example of pattern matching.
- It can be done with string methods, but there is often a better approach: regular expressions.
print(contact)
Thank you for buying our expensive product! If you have a complaint, please send it to complaints@compuserve.com or call (800) 867-5309. If you are happy with your purchase, please call us at (800) 123-4567; we'd love to hear from you! Due to high demand, please allow one-hundred (100) business days for a response.
import re
re.findall(r'\(\d{3}\) \d{3}-\d{4}', contact)
['(800) 867-5309', '(800) 123-4567']
🤯
Basic regular expressions¶
Regular expressions¶
- A regular expression, or regex for short, is a sequence of characters used to match patterns in strings.
- For example,
\(\d{3}\) \d{3}-\d{4}
describes a pattern that matches US phone numbers of the form'(XXX) XXX-XXXX'
. - Think of regex as a "mini-language" (formally: they are a grammar for describing a language).
- For example,
- Pros: They are very powerful and are widely used (virtually every programming language has a module for working with them).
- Cons: They can be hard to read and have many different "dialects."
Writing regular expressions¶
You will ultimately write most of your regular expressions in Python, using the
re
module. We will see how to do so shortly.However, a useful tool for designing regular expressions is regex101.com.
We will use it heavily during lecture; you should have it open as we work through examples. If you're trying to revisit this lecture in the future, you'll likely want to watch the podcast.
Literals¶
A literal is a character that has no special meaning.
Letters, numbers, and some symbols are all literals.
Some symbols, like
.
,*
,(
, and)
, are special characters.Example: The regex
hey
matches the string'hey'
. The regexhe.
also matches the string'hey'
.
Regex building blocks 🧱¶
The four main building blocks for all regexes are shown below (table source, inspiration).
operation | order of op. | example | matches ✅ | does not match ❌ |
---|---|---|---|---|
concatenation | 3 | AABAAB |
'AABAAB' |
every other string |
or | 4 | AA\|BAAB |
'AA' , 'BAAB' |
every other string |
closure (zero or more) |
2 | AB*A |
'AA' , 'ABBBBBBA' |
'AB' , 'ABABA' |
parentheses | 1 | A(A\|B)AAB (AB)*A |
'AAAAB' , 'ABAAB' 'A' , 'ABABABABA' |
every other string'AA' , 'ABBA' |
Note that |
, (
, )
, and *
are special characters, not literals. They manipulate the characters around them.
Example (or, parentheses):
- What does
DSC 30|80
match? - What does
DSC (30|80)
match?
Example (closure, parentheses):
- What does
blah*
match? - What does
(blah)*
match?
Question 🤔 (Answer at dsc80.com/q)
Code: billy
Write a regular expression that matches 'billy'
, 'billlly'
, 'billlllly'
, etc.
- First, think about how to match strings with any even number of
'l'
s, including zero'l'
s (i.e.'biy'
). - Then, think about how to match only strings with a positive even number of
'l'
s.
Question 🤔 (Answer at dsc80.com/q)
Code: billy2
Write a regular expression that matches 'billy'
, 'billlly'
, 'biggy'
, 'biggggy'
, etc.
Specifically, it should match any string with a positive even number of 'l'
s in the middle, or a positive even number of 'g'
s in the middle.
Intermediate regex¶
More regex syntax¶
operation | example | matches ✅ | does not match ❌ |
---|---|---|---|
wildcard | .U.U.U. |
'CUMULUS' 'JUGULUM' |
'SUCCUBUS' 'TUMULTUOUS' |
character class | [A-Za-z][a-z]* |
'word' 'Capitalized' |
'camelCase' '4illegal' |
at least one | bi(ll)+y |
'billy' 'billlllly' |
'biy' 'bily' |
between $i$ and $j$ occurrences | m[aeiou]{1,2}m |
'mem' 'maam' 'miem' |
'mm' 'mooom' 'meme' |
.
, [
, ]
, +
, {
, and }
are also special characters, in addition to |
, (
, )
, and *
.
Example (character classes, at least one): [A-E]+
is just shortform for (A|B|C|D|E)(A|B|C|D|E)*
.
Example (wildcard):
- What does
.
match? - What does
he.
match? - What does
...
match?
Example (at least one, closure):
- What does
123+
match? - What does
123*
match?
Example (number of occurrences): What does tri{3, 5}
match? Does it match 'triiiii'
?
Example (character classes, number of occurrences):
What does [1-6a-f]{3}-[7-9E-S]{2}
match?
Question 🤔 (Answer at dsc80.com/q)
Code: vowels
Write a regular expression that matches any lowercase string has a repeated vowel, such as 'noon'
, 'peel'
, 'festoon'
, or 'zeebraa'
.
Question 🤔 (Answer at dsc80.com/q)
Code: letnum
Write a regular expression that matches any string that contains both a lowercase letter and a number, in any order. Examples include 'billy80'
, '80!!billy'
, and 'bil8ly0'
.
Even more regex syntax¶
operation | example | matches ✅ | does not match ❌ |
---|---|---|---|
escape character | ucsd\.edu |
'ucsd.edu' |
'ucsd!edu' |
beginning of line | ^ark |
'ark two' 'ark o ark' |
'dark' |
end of line | ark$ |
'dark' 'ark o ark' |
'ark two' |
zero or one | cat? |
'ca' 'cat' |
'cart' (matches 'ca' only) |
built-in character classes* | \w+ \d+ |
'billy' '231231' |
'this person' '858 people' |
character class negation | [^a-z]+ |
'KINGTRITON551' '1721$$' |
'porch' 'billy.edu' |
*Note: in Python's implementation of regex,
\d
refers to digits.\w
refers to alphanumeric characters ([A-Z][a-z][0-9]_
). Whenever we say "alphanumeric" in an assignment, we're referring to\w
!\s
refers to whitespace.\b
is a word boundary.
Example (escaping):
- What does
he.
match? - What does
he\.
match? - What does
(858)
match? - What does
\(858\)
match?
Example (anchors):
- What does
858-534
match? - What does
^858-534
match? - What does
858-534$
match?
Example (built-in character classes):
- What does
\d{3} \d{3}-\d{4}
match? - What does
\bcat\b
match? Does it find a match in'my cat is hungry'
? What about'concatenate'
or'kitty cat'
?
Remember, in Python's implementation of regex,
\d
refers to digits.\w
refers to alphanumeric characters ([A-Z][a-z][0-9]_
). Whenever we say "alphanumeric" in an assignment, we're referring to\w
!\s
refers to whitespace.\b
is a word boundary.
Question 🤔 (Answer at dsc80.com/q)
Code: vowels2
Write a regular expression that matches any string that:
- is between 5 and 10 characters long, and
- is made up of only vowels (either uppercase or lowercase, including
'Y'
and'y'
), periods, and spaces.
Examples include 'yoo.ee.IOU'
and 'AI.I oey'
.
Regex in Python¶
re
in Python¶
The re
package is built into Python. It allows us to use regular expressions to find, extract, and replace strings.
import re
re.search
takes in a string regex
and a string text
and returns the location and substring corresponding to the first match of regex
in text
.
re.search('AB*A',
'here is a string for you: ABBBA. here is another: ABBBBBBBA')
<re.Match object; span=(26, 31), match='ABBBA'>
re.findall
takes in a string regex
and a string text
and returns a list of all matches of regex
in text
. You'll use this most often.
re.findall('AB*A',
'here is a string for you: ABBBA. here is another: ABBBBBBBA')
['ABBBA', 'ABBBBBBBA']
re.sub
takes in a string regex
, a string repl
, and a string text
, and replaces all matches of regex
in text
with repl
.
re.sub('AB*A',
'billy',
'here is a string for you: ABBBA. here is another: ABBBBBBBA')
'here is a string for you: billy. here is another: billy'
Raw strings¶
When using regular expressions in Python, it's a good idea to use raw strings, denoted by an r
before the quotes, e.g. r'exp'
.
re.findall('\bcat\b', 'my cat is hungry')
[]
re.findall(r'\bcat\b', 'my cat is hungry')
['cat']
# Huh?
print('\bcat\b')
ca
Capture groups¶
- Surround a regex with
(
and)
to define a capture group within a pattern.
- Capture groups are useful for extracting relevant parts of a string.
re.findall(r'\w+@(\w+)\.edu',
'my old email was billy@notucsd.edu, my new email is notbilly@ucsd.edu')
['notucsd', 'ucsd']
- Notice what happens if we remove the
(
and)
!
re.findall(r'\w+@\w+\.edu',
'my old email was billy@notucsd.edu, my new email is notbilly@ucsd.edu')
['billy@notucsd.edu', 'notbilly@ucsd.edu']
- Earlier, we also saw that parentheses can be used to group parts of a regex together. When using
re.findall
, all groups are treated as capturing groups.
# A regex that matches strings with two of the same vowel followed by 3 digits
# We only want to capture the digits, but...
re.findall(r'(aa|ee|ii|oo|uu)(\d{3})', 'eeoo124')
[('oo', '124')]
Example: Log parsing¶
Web servers typically record every request made of them in the "logs".
s = '''132.249.20.188 - - [24/Feb/2023:12:26:15 -0800] "GET /my/home/ HTTP/1.1" 200 2585'''
Let's use our new regex syntax (including capturing groups) to extract the day, month, year, and time from the log string s
.
exp = r'\[(.+)\/(.+)\/(.+):(.+):(.+):(.+) .+\]'
re.findall(exp, s)
[('24', 'Feb', '2023', '12', '26', '15')]
While above regex works, it is not very specific. It works on incorrectly formatted log strings.
other_s = '[adr/jduy/wffsdffs:r4s4:4wsgdfd:asdf 7]'
re.findall(exp, other_s)
[('adr', 'jduy', 'wffsdffs', 'r4s4', '4wsgdfd', 'asdf')]
The more specific, the better!¶
- Be as specific in your pattern matching as possible – you don't want to match and extract strings that don't fit the pattern you care about.
.*
matches every possible string, but we don't use it very often.
- A better date extraction regex:
\[(\d{2})\/([A-Z]{1}[a-z]{2})\/(\d{4}):(\d{2}):(\d{2}):(\d{2}) -\d{4}\]
- `\d{2}` matches any 2-digit number.
- `[A-Z]{1}` matches any single occurrence of any uppercase letter.
- `[a-z]{2}` matches any 2 consecutive occurrences of lowercase letters.
- Remember, special characters (`[`, `]`, `/`) need to be escaped with `\`.
s
'132.249.20.188 - - [24/Feb/2023:12:26:15 -0800] "GET /my/home/ HTTP/1.1" 200 2585'
new_exp = r'\[(\d{2})\/([A-Z]{1}[a-z]{2})\/(\d{4}):(\d{2}):(\d{2}):(\d{2}) -\d{4}\]'
re.findall(new_exp, s)
[('24', 'Feb', '2023', '12', '26', '15')]
A benefit of new_exp
over exp
is that it doesn't capture anything when the string doesn't follow the format we specified.
other_s
'[adr/jduy/wffsdffs:r4s4:4wsgdfd:asdf 7]'
re.findall(new_exp, other_s)
[]
Question 🤔 (Answer at dsc80.com/q)
Code: weird_re
^\w{2,5}.\d*\/[^A-Z5]{1,}
Select all strings below that contain any match with the regular expression above.
"billy4/Za"
"billy4/za"
"DAI_s2154/pacific"
"daisy/ZZZZZ"
"bi_/_lly98"
"!@__!14/atlantic"
Limitations of regular expressions¶
Writing a regular expression is like writing a program.
- You need to know the syntax well.
- They can be easier to write than to read.
- They can be difficult to debug.
Regular expressions are terrible at certain types of problems. Examples:
- Anything involving counting (same number of instances of a and b).
- Anything involving complex structure (palindromes).
- Parsing highly complex text structure (HTML, for instance).
Summary, next time¶
Summary¶
- Regular expressions are used to match and extract patterns from text.
- You don't need to force yourself to "memorize" regex syntax – refer to the resources in the Agenda section of the lecture and on the Resources tab of the course website.
- Also refer to the three tables of syntax in the lecture:
- Note: You don't always have to use regular expressions! If Python/pandas string methods work for your task, you can still use those.
- Play Regex Golf to practice! 🏌️
pandas
.str
methods can use regular expressions; just setregex=True
.
Next time¶
- Text features: Bag of words, TF-IDF