# Chapter 6 String manipulation with stringr and regular expressions

When analyzing data, a significant number of variables will be in some sort of text format. When you want to manipulate those variables, an easy approach would be exporting the data to MS Excel and then just perform those manipulations by hand. This is very time-consuming, though, and, hence, I rather recommend the R way, which scales well and works fast for data sets of varying sizes.

Quick reminder: a string is an element of a character vector and can be created by simply wrapping some text in back ticks:

string <- "Hi, how are you doing?"
vector_of_strings <- c("Hi, how are you doing?", "I'm doing well, HBY?", "Me too, thanks for asking.")

The stringr package contains a multitude of commands (49 in total) which can be used to achieve a couple of things: manipulating character vectors; operations which are sensitive to different locales; matching patterns. Basically, those goals can also be achieved with base R functions, but stringr’s advantage is its consistency. The makers of stringr describe it as

A consistent, simple and easy to use set of wrappers around the fantastic ‘stringi’ package. All function and argument names (and positions) are consistent, all functions deal with “NA”’s and zero length vectors in the same way, and the output from one function is easy to feed into the input of another.

Every stringr function starts with str_ – which facilitates finding the proper command: just type str_ and RStudio’s auto-suggest function should take care of the rest (if it doesn’t pop up by itself, you can trigger it by hitting the tab-key). Also, they take a vector of strings as their first argument, which facilitates using them in a %>%-pipeline and adding them to a mutate()-call.

One important component of stringr functions is regular expressions which will be introduced later as well.

## 6.1 Basic manipulations

In the following, I will introduce you to a number of different operations that can be performed on strings.

### 6.1.1 Changing the case of the words

A basic operation is changing words’ case.

library(tidyverse) #stringr is part of the core tidyverse
str_to_lower(vector_of_strings)
## [1] "hi, how are you doing?"     "i'm doing well, hby?"
## [3] "me too, thanks for asking."
str_to_upper(vector_of_strings)
## [1] "HI, HOW ARE YOU DOING?"     "I'M DOING WELL, HBY?"
## [3] "ME TOO, THANKS FOR ASKING."
str_to_title(vector_of_strings)
## [1] "Hi, How Are You Doing?"     "I'm Doing Well, Hby?"
## [3] "Me Too, Thanks For Asking."
str_to_sentence(vector_of_strings)
## [1] "Hi, how are you doing?"     "I'm doing well, hby?"
## [3] "Me too, thanks for asking."

### 6.1.2 Determining a string’s length

Determining the string’s number of characters goes as follows:

str_length(vector_of_strings)
## [1] 22 20 26

### 6.1.3 Extracting particular characters

Characters can be extracted (by position) using str_sub

str_sub(vector_of_strings, start = 1, end = 5) # extracting first to fifth character
## [1] "Hi, h" "I'm d" "Me to"
str_sub(vector_of_strings, start = -5, end = -1) # extracting fifth-to-last to last character
## [1] "oing?" " HBY?" "king."

You can also use str_sub() to replace strings. E.g., to replace the last character by a full stop, you can do the following:

str_sub(vector_of_strings, start = -1) <- "."
vector_of_strings
## [1] "Hi, how are you doing."     "I'm doing well, HBY."
## [3] "Me too, thanks for asking."

However, in everyday use you would probably go with str_replace() and regular expressions.

### 6.1.4 Concatenating strings

Similar to how c() puts together different elements (or vectors of length 1) into a single vector, str_c() can be used to concatenate several strings into a single string. This can, for instance, be used to write some birthday invitations.

names <- c("Inger", "Peter", "Kalle", "Ingrid")

str_c("Hi", names, "I hope you're doing well. As per this letter, I invite you to my birthday party.")
## [1] "HiIngerI hope you're doing well. As per this letter, I invite you to my birthday party."
## [2] "HiPeterI hope you're doing well. As per this letter, I invite you to my birthday party."
## [3] "HiKalleI hope you're doing well. As per this letter, I invite you to my birthday party."
## [4] "HiIngridI hope you're doing well. As per this letter, I invite you to my birthday party."

Well, this looks kind of ugly, as there are no spaces and commas are lacking as well. You can fix that by determining a separator using the sep argument.

str_c("Hi", names, "I hope you're doing well. As per this letter, I invite you to my birthday party.", sep = ", ")
## [1] "Hi, Inger, I hope you're doing well. As per this letter, I invite you to my birthday party."
## [2] "Hi, Peter, I hope you're doing well. As per this letter, I invite you to my birthday party."
## [3] "Hi, Kalle, I hope you're doing well. As per this letter, I invite you to my birthday party."
## [4] "Hi, Ingrid, I hope you're doing well. As per this letter, I invite you to my birthday party."

You could also collapse the strings contained in a vector together into one single string using the collapse argument.

str_c(names, collapse = ", ")
## [1] "Inger, Peter, Kalle, Ingrid"

This can also be achieved using the str_flatten() function.

str_flatten(names, collapse = ", ")
## [1] "Inger, Peter, Kalle, Ingrid"

### 6.1.5 Repetition

Repeating (or duplicating) strings is performed using str_dup(). The function takes two arguments: the string to be duplicated and the number of times.

str_dup("felix", 2)
## [1] "felixfelix"
str_dup("felix", 1:3)
## [1] "felix"           "felixfelix"      "felixfelixfelix"
str_dup(names, 2)
## [1] "IngerInger"   "PeterPeter"   "KalleKalle"   "IngridIngrid"
str_dup(names, 1:4)
## [1] "Inger"                    "PeterPeter"
## [3] "KalleKalleKalle"          "IngridIngridIngridIngrid"

### 6.1.6 Removing unnecessary whitespaces

Often text contains unnecessary whitespaces.

unnecessary_whitespaces <- c("    on the left", "on the right    ", "    on both sides   ", "   literally    everywhere  ")

Removing the ones at the beginning or the end of a string can be accomplished using str_trim().

str_trim(unnecessary_whitespaces, side = "left")
## [1] "on the left"               "on the right    "
## [3] "on both sides   "          "literally    everywhere  "
str_trim(unnecessary_whitespaces, side = "right")
## [1] "    on the left"            "on the right"
## [3] "    on both sides"          "   literally    everywhere"
str_trim(unnecessary_whitespaces, side = "both") # the default option
## [1] "on the left"             "on the right"
## [3] "on both sides"           "literally    everywhere"

str_trim() could not fix the last string though, where unnecessary whitespaces were also present in between words. Here, str_squish is more appropriate. It removes leading or trailing whitespaces as well as duplicated ones in between words.

str_squish(unnecessary_whitespaces)
## [1] "on the left"          "on the right"         "on both sides"
## [4] "literally everywhere"

## 6.2 Regular expressions

Up to now, you have been introduced to the more basic functions of the stringr package. Those are useful, for sure, yet limited. However, to make use of the full potential of stringr, you will first have to get acquainted to regular expressions (also often abbreviated as “regex” with plural “regexes”).

Those regular expressions are patterns that can be used to describe certain strings. Hence, if you want to replace certain words with another one, you can write the proper regex and it will identify the strings you want to replace and the stringr function (i.e., str_replace()) will take care of the rest. Exemplary use cases of regexes are the identification of phone numbers, email addresses, or whether a password you choose on a web page consists of enough characters, an upper-case character, and at least one special character.

Before you dive into regexes, beware that they are quite complicated in the beginning (honestly, I was quite overwhelmed when I encountered them first). Yet, mastering them is very rewarding and will definitely pay off in the future.

### 6.2.1 Literal characters

The most basic regex patterns consist of literal characters only. str_view() tells you which parts of a string match a pattern is present in the element.

five_largest_cities <- c("Stockholm", "Göteborg", "Malmö", "Uppsala", "Västerås")

#str_c(five_largest_cities, collapse = "\n") %>% cat()

Note that regexes are case-sensitive.

str_view(five_largest_cities, "stockholm")
str_view(five_largest_cities, "Stockholm")

They also match parts of words:

str_view(five_largest_cities, "borg")

Moreover, they are “greedy,” they only match the first occurrence (in “Stockholm”):

str_view(five_largest_cities, "o")

This can be addressed in the stringr package by using str_._all() function – but more on that later.

If you want to match multiple literal characters (or words, for that sake), you can connect them using the | meta character (more on meta characters later).

str_view(five_largest_cities, "Stockholm|Göteborg")

Every letter of the English alphabet (or number/or combination of those) can serve as a literal character. Those literal characters match themselves. This is, however, not the case with the other sort of characters, so-called meta characters.

### 6.2.2 Metacharacters

When using regexes, the following characters are considered meta characters and have a special meaning:

. \ | ( ) { } [ ] ^ $- * + ? #### 6.2.2.1 The wildcard Did you notice how I used the dot to refer to the entirety of the str_._all() functions? This is basically what the . meta-character does: it matches every character except for a new line. The first call extracts all function names from the stringr package, the second one shows the matches (i.e., the elements of the vector where it can find the pattern). stringr_functions <- ls("package:stringr") #str_c(stringr_functions, collapse = "\n") %>% cat() str_detect(stringr_functions, "str_._all") ## [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE ## [13] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE ## [25] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE ## [37] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE ## [49] FALSE FALSE FALSE FALSE Well, as you can see, there are none. This is due to the fact that the . can only replace one character. We need some sort of multiplier to find them. The ones available are: • ? – zero or one • * – zero or more • + – one or more • {n} – exactly n • {n,} – n or more • {n,m} – between n and m In our case, the appropriate one is +: str_detect(stringr_functions, "str_.+_all") ## [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE ## [13] FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE ## [25] TRUE FALSE FALSE FALSE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE ## [37] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE ## [49] FALSE FALSE FALSE FALSE However, if you want to match the character dot? This problem may arise when searching for clock time. A naive regex might look like this: vectors_with_time <- c("13500", "13M00", "13.00") #str_c(vectors_with_time, collapse = "\n") %>% cat() str_detect(vectors_with_time, "13.00") ## [1] TRUE TRUE TRUE Yet, it matches everything. We need some sort of literal dot. Here, the meta character \ comes in handy. By putting it in front of the meta character, it does no longer has its special meaning, and is interpreted as a literal character. This procedure is referred to as “escaping.” Hence, \ is also referred to as the “escape character.” Note that you will need to escape \ as well, and therefore it will look like this: \\.. str_detect(vectors_with_time, "13\\.00") ## [1] FALSE FALSE TRUE #### 6.2.2.2 Sets of characters You can also define sets of multiple characters using the [ ] meta characters. This can be used to define multiple possible characters that can appear in the same place. sp_ce <- c("spice", "space") #str_c(sp_ce, collapse = "\n") %>% cat() str_view(sp_ce, "sp[ai]ce") You can also define certain ranges of characters using the - meta character: Same holds for numbers: american_phone_number <- "(555) 555-1234" str_view(american_phone_number, "\$$[:digit:]{3}\$$ [0-9]{3}-[0-9]{4}") There are also predefined sets of characters, for instance digits or letters, which are called character classes. You can find them on the stringr cheatsheet. Furthermore, you can put almost every meta character inside the square brackets without escaping them. This does not apply to the the caret (^) in first position, the dash -, the closing square bracket ], and the backslash \. str_view(vector_of_strings, "[.]") #str_c(vector_of_strings, collapse = "\n") %>% cat() #### 6.2.2.3 Negating sets of characters Sometimes you will also want to exclude certain sets of characters or words. In order to achieve this, you can use the ^ meta character at the beginning of the range or set you are defining. str_view(sp_ce, "sp[^i]ce") #### 6.2.2.4 Anchors There is also a way to define whether you want the pattern to be present in the beginning ^ or at the end $ of a string. sentences are a couple of (i.e., 720) predefined example sentences. If I were now interested in the number of sentences that begin with a “the,” I could write the following regex:

shortened_sentences <- sentences[1:10]

#str_c(shortened_sentences, collapse = "\n") %>% cat()

str_view(shortened_sentences, "^The") 

If I wanted to know how many start with a “The” and end with a full stop, I could do this one:

str_view(shortened_sentences, "^The\\B.+\\.$")  #### 6.2.2.6 Lookarounds A final common task is to extract certain words or values based on what comes before or after them. Look at the following example: heights <- c("1m30cm", "2m01cm", "3m10cm") #str_c(heights, collapse = "\n") %>% cat() Here, in order to identify the height in meters, the first task is to identify all the numbers that are followed by an “m.” The regex syntax for this looks like this: A(?=pattern) with A being the entity that is supposed to be found (hence, in this case, [0-9]+). str_view(heights, "[0-9]+(?=m)") The second step now is to identify the centimeters. This could of course be achieved using the same regex and replacing m by cm. However, we can also harness a so-called negative look ahead A(?!pattern), a so-called look behind (?<=pattern)A. The negative counterpart, the negative look behind (?<!pattern)A could be used to extract the meters. The negative look ahead basically returns everything that is not followed by the defined pattern. The look behind returns everything that is preceded by the pattern, the negative look behind returns everything that is not preceded by the pattern. In the following, I demonstrate how you could extract the centimeters using negative look ahead and look behind. str_view(heights, "[0-9]+(?!m)") # negative look ahead str_view(heights, "(?<=m)[0-9]+") # look behind ## 6.3 More advanced string manipulation Now that you have learned about regexes, you can unleash the full power of stringr. The basic syntax of a stringr function looks as follows: str_.*(string, regex("")). Some stringr functions also have the suffix _all which implies that they perform the operation not only on the first match (“greedy”) but on every match. In order to demonstrate the different functions, I will again rely on the subset of example sentences. ### 6.3.1 Detect matches str_detect can be used to determine whether a certain pattern is present in the string. str_detect(shortened_sentences, "The\\b") ## [1] TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE This also works very well in a dplyr::filter() call. Finding all action movies in the IMDB data set can be solved like this: imdb_raw <- read_csv("data/imdb2006-2016.csv") ## Rows: 1000 Columns: 12 ## ── Column specification ──────────────────────────────────────────────────────── ## Delimiter: "," ## chr (5): Title, Genre, Description, Director, Actors ## dbl (7): Rank, Year, Runtime (Minutes), Rating, Votes, Revenue (Millions), M... ## ## ℹ Use spec() to retrieve the full column specification for this data. ## ℹ Specify the column types or set show_col_types = FALSE to quiet this message. imdb_raw %>% filter(str_detect(Genre, "Action")) ## # A tibble: 303 × 12 ## Rank Title Genre Description Director Actors Year Runtime (Minut… Rating ## <dbl> <chr> <chr> <chr> <chr> <chr> <dbl> <dbl> <dbl> ## 1 1 Guard… Actio… A group of… James G… Chris… 2014 121 8.1 ## 2 5 Suici… Actio… A secret g… David A… Will … 2016 123 6.2 ## 3 6 The G… Actio… European m… Yimou Z… Matt … 2016 103 6.1 ## 4 9 The L… Actio… A true-lif… James G… Charl… 2016 141 7.1 ## 5 13 Rogue… Actio… The Rebel … Gareth … Felic… 2016 133 7.9 ## 6 15 Colos… Actio… Gloria is … Nacho V… Anne … 2016 109 6.4 ## 7 18 Jason… Actio… The CIA's … Paul Gr… Matt … 2016 123 6.7 ## 8 25 Indep… Actio… Two decade… Roland … Liam … 2016 120 5.3 ## 9 27 Bahub… Actio… In ancient… S.S. Ra… Prabh… 2015 159 8.3 ## 10 30 Assas… Actio… When Callu… Justin … Micha… 2016 115 5.9 ## # … with 293 more rows, and 3 more variables: Votes <dbl>, ## # Revenue (Millions) <dbl>, Metascore <dbl> If you want to know whether there are multiple matches present in each string, you can use str_count. Here, it might by advisable to set the ignore_case option to TRUE: str_count(shortened_sentences, regex("the\\b", ignore_case = TRUE)) ## [1] 2 2 1 0 0 1 2 1 0 0 If you want to locate the match in the string, use str_locate. This returns a matrix, which is basically a vector of multiple dimensions. str_locate(shortened_sentences, regex("The\\b", ignore_case = TRUE)) ## start end ## [1,] 1 3 ## [2,] 6 8 ## [3,] 19 21 ## [4,] NA NA ## [5,] NA NA ## [6,] 1 3 ## [7,] 1 3 ## [8,] 1 3 ## [9,] NA NA ## [10,] NA NA Moreover, this is a good example for the greediness of stringr functions. Hence, it is advisable to use str_locate_all which returns a list with one matrix for each element of the original vector: str_locate_all(shortened_sentences, regex("The\\b", ignore_case = TRUE)) ## [[1]] ## start end ## [1,] 1 3 ## [2,] 25 27 ## ## [[2]] ## start end ## [1,] 6 8 ## [2,] 19 21 ## ## [[3]] ## start end ## [1,] 19 21 ## ## [[4]] ## start end ## ## [[5]] ## start end ## ## [[6]] ## start end ## [1,] 1 3 ## ## [[7]] ## start end ## [1,] 1 3 ## [2,] 27 29 ## ## [[8]] ## start end ## [1,] 1 3 ## ## [[9]] ## start end ## ## [[10]] ## start end ### 6.3.2 Mutating strings Mutating strings usually implies the replacement of certain elements (e.g., words) with other elements (or removing them, which is basically a special case of replacing them). In stringr this is performed using str_replace(string, pattern, replacement) and str_replace_all(string, pattern, replacement). If I wanted, for instance, replace the first occurrence of “m” letters by “meters,” I would go about this the following way: str_replace(heights, "m", "meters") ## [1] "1meters30cm" "2meters01cm" "3meters10cm" Note that str_replace_all would have lead to the following outcome: str_replace_all(heights, "m", "meters") ## [1] "1meters30cmeters" "2meters01cmeters" "3meters10cmeters" However, I also want to replace the “cm” with “centimeters,” hence, I can harness another feature of str_replace_all(): str_replace_all(heights, c("m" = "meters", "cm" = "centimeters")) ## [1] "1meters30centimeterseters" "2meters01centimeterseters" ## [3] "3meters10centimeterseters" What becomes obvious is that a “simple” regex containing just literal characters more often than not does not suffice. It will be your task to fix this. And while on it, you can also address the meter/meters problem – a “1” needs meter instead of meters. Another feature is that the replacements are performed in order. You can harness this for solving the problem. Solution. Click to expand! Solution: str_replace_all(heights, c("(?<=[2-9]{1})m" = "meters", "(?<=[0-9]{2})m" = "meters", "(?<=1)m" = "meter", "(?<=01)cm$" = "centimeter", "cm\$" = "centimeters"))
## [1] "1meter30centimeters"  "2meters01centimeter"  "3meters10centimeters"

### 6.3.3 Extracting text

str_extract(_all)() can be used to extract matching strings. In the mtcars data set, the first word describes the car brand. Here, I harness another regexp, the \\w which stands for any word character. Its opponent is \\W for any non-word character.

mtcars %>%
rownames_to_column(var = "car_model") %>%
transmute(manufacturer = str_extract(car_model, "^\\w+\\b"))
##    manufacturer
## 1         Mazda
## 2         Mazda
## 3        Datsun
## 4        Hornet
## 5        Hornet
## 6       Valiant
## 7        Duster
## 8          Merc
## 9          Merc
## 10         Merc
## 11         Merc
## 12         Merc
## 13         Merc
## 14         Merc
## 16      Lincoln
## 17     Chrysler
## 18         Fiat
## 19        Honda
## 20       Toyota
## 21       Toyota
## 22        Dodge
## 23          AMC
## 24       Camaro
## 25      Pontiac
## 26         Fiat
## 27      Porsche
## 28        Lotus
## 29         Ford
## 30      Ferrari
## 31     Maserati
## 32        Volvo

### 6.3.4 Split vectors

Another use case here would have been to split it into two columns: manufacturer and model. One approach would be to use str_split(). This function splits the string at every occurrence of the predefined pattern. In this example, I use a word boundary as the pattern:

manufacturer_model <- rownames(mtcars)
str_split(manufacturer_model, "\\b") %>%
head()
## [[1]]
## [1] ""      "Mazda" " "     "RX4"   ""
##
## [[2]]
## [1] ""      "Mazda" " "     "RX4"   " "     "Wag"   ""
##
## [[3]]
## [1] ""       "Datsun" " "      "710"    ""
##
## [[4]]
## [1] ""       "Hornet" " "      "4"      " "      "Drive"  ""
##
## [[5]]
## [1] ""           "Hornet"     " "          "Sportabout" ""
##
## [[6]]
## [1] ""        "Valiant" ""

This outputs a list containing the different singular words/special characters. This doesn’t make sense in this case. Here, however, the structure of the string is always roughly the same: “$manufacturer$$model description$.” Moreover, the manufacturer is only one word. Hence, the task can be fixed by splitting the string after the first word, which should indicate the manufacturer. This can be accomplished using str_split_fixed(). Fixed means that the number of splits is predefined. This returns a matrix that can easily become a tibble.

str_split_fixed(manufacturer_model, "(?<=\\w)\\b", n = 2) %>%
as_tibble() %>%
rename(manufacturer = V1,
model = V2) %>%
mutate(model = str_squish(model))
## Warning: The x argument of as_tibble.matrix() must have unique column names if .name_repair is omitted as of tibble 2.0.0.
## Using compatibility .name_repair.
## This warning is displayed once every 8 hours.
## Call lifecycle::last_lifecycle_warnings() to see where this warning was generated.
## # A tibble: 32 × 2
##    manufacturer model
##    <chr>        <chr>
##  1 Mazda        "RX4"
##  2 Mazda        "RX4 Wag"
##  3 Datsun       "710"
##  4 Hornet       "4 Drive"
## # … with 22 more rows`