29 Programming with strings
29.0.1 Encoding
You will not generally find the base R Encoding() to be useful because it only supports three different encodings (and interpreting what they mean is non-trivial) and it only tells you the encoding that R thinks it is, not what it really is.
And typically the problem is that the declaring encoding is wrong.
The tidyverse follows best practices15 of using UTF-8 everywhere, so any string you create with the tidyverse will use UTF-8.
It’s still possible to have problems, but they’ll typically arise during data import.
Once you’ve diagnosed you have an encoding problem, you should fix it in data import (i.e. by using the encoding argument to readr::locale()).
29.0.2 Length and subsetting
This seems like a straightforward computation if you’re only familiar with English, but things get complex quick when working with other languages.
Four most common are Latin, Chinese, Arabic, and Devangari, which represent three different systems of writing systems:
Latin uses an alphabet, where each consonant and vowel gets its own letter.
Chinese. Logograms. Half width vs full width. English letters are roughly twice as high as they are wide. Chinese characters are roughly square.
Arabic is an abjad, only consonants are written and vowels are optionally as diacritics. Additionally, it’s written from right-to-left, so the first letter is the letter on the far right.
Devangari is an abugida where each symbol represents a consonant-vowel pair, , vowel notation secondary.
For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. — http://utf8everywhere.org
# But
str_split("check", boundary("character", locale = "cs_CZ"))
#> [[1]]
#> [1] "c" "h" "e" "c" "k"This is a problem even with Latin alphabets because many languages use diacritics, glyphs added to the basic alphabet. This is a problem because Unicode provides two ways of representing characters with accents: many common characters have a special codepoint, but others can be built up from individual components.
x <- c("á", "x́")
str_length(x)
#> [1] 1 2
# str_width(x)
str_sub(x, 1, 1)
#> [1] "á" "x"
# stri_width(c("全形", "ab"))
# 0, 1, or 2
# but this assumes no font substitution
cyrillic_a <- "А"
latin_a <- "A"
cyrillic_a == latin_a
#> [1] FALSE
stringi::stri_escape_unicode(cyrillic_a)
#> [1] "\\u0410"
stringi::stri_escape_unicode(latin_a)
#> [1] "A"29.0.3 str_c
NULLs are silently dropped.
This is particularly useful in conjunction with if:
name <- "Hadley"
time_of_day <- "morning"
birthday <- FALSE
str_c(
"Good ", time_of_day, " ", name,
if (birthday) " and HAPPY BIRTHDAY",
"."
)
#> [1] "Good morning Hadley."
29.0.4 str_dup()
Closely related to str_c() is str_dup().
str_c(a, a, a) is like a + a + a, what’s the equivalent of 3 * a?
That’s str_dup():
29.1 Performance
fixed(): matches exactly the specified sequence of bytes.
It ignores all special regular expressions and operates at a very low level.
This allows you to avoid complex escaping and can be much faster than regular expressions.
The following microbenchmark shows that it’s about 3x faster for a simple example.
microbenchmark::microbenchmark(
fixed = str_detect(sentences, fixed("the")),
regex = str_detect(sentences, "the"),
times = 20
)
#> Unit: microseconds
#> expr min lq mean median uq max neval
#> fixed 61.5 67.1 166 71.4 83 1714 20
#> regex 177.8 189.3 209 193.0 203 370 20As you saw with str_split() you can use boundary() to match boundaries.
You can also use it with the other functions:
x <- "This is a sentence."
str_view_all(x, boundary("word"))29.1.1 Extract
colours <- c("red", "orange", "yellow", "green", "blue", "purple")
colour_match <- str_c(colours, collapse = "|")
colour_match
#> [1] "red|orange|yellow|green|blue|purple"
more <- sentences[str_count(sentences, colour_match) > 1]
str_extract_all(more, colour_match)
#> [[1]]
#> [1] "blue" "red"
#>
#> [[2]]
#> [1] "green" "red"
#>
#> [[3]]
#> [1] "orange" "red"If you use simplify = TRUE, str_extract_all() will return a matrix with short matches expanded to the same length as the longest:
str_extract_all(more, colour_match, simplify = TRUE)
#> [,1] [,2]
#> [1,] "blue" "red"
#> [2,] "green" "red"
#> [3,] "orange" "red"
x <- c("a", "a b", "a b c")
str_extract_all(x, "[a-z]", simplify = TRUE)
#> [,1] [,2] [,3]
#> [1,] "a" "" ""
#> [2,] "a" "b" ""
#> [3,] "a" "b" "c"We don’t talk about matrices here, but they are useful elsewhere.
29.2 Grouped matches
Earlier in this chapter we talked about the use of parentheses for clarifying precedence and for backreferences when matching. You can also use parentheses to extract parts of a complex match. For example, imagine we want to extract nouns from the sentences. As a heuristic, we’ll look for any word that comes after “a” or “the”. Defining a “word” in a regular expression is a little tricky, so here I use a simple approximation: a sequence of at least one character that isn’t a space.
noun <- "(a|the) ([^ ]+)"
has_noun <- sentences %>%
str_subset(noun) %>%
head(10)
has_noun %>%
str_extract(noun)
#> [1] "the smooth" "the sheet" "the depth" "a chicken" "the parked"
#> [6] "the sun" "the huge" "the ball" "the woman" "a helps"str_extract() gives us the complete match; str_match() gives each individual component.
Instead of a character vector, it returns a matrix, with one column for the complete match followed by one column for each group:
has_noun %>%
str_match(noun)
#> [,1] [,2] [,3]
#> [1,] "the smooth" "the" "smooth"
#> [2,] "the sheet" "the" "sheet"
#> [3,] "the depth" "the" "depth"
#> [4,] "a chicken" "a" "chicken"
#> [5,] "the parked" "the" "parked"
#> [6,] "the sun" "the" "sun"
#> [7,] "the huge" "the" "huge"
#> [8,] "the ball" "the" "ball"
#> [9,] "the woman" "the" "woman"
#> [10,] "a helps" "a" "helps"(Unsurprisingly, our heuristic for detecting nouns is poor, and also picks up adjectives like smooth and parked.)
29.3 Spitting
Use str_split() to split a string up into pieces.
For example, we could split sentences into words:
sentences %>%
head(5) %>%
str_split(" ")
#> [[1]]
#> [1] "The" "birch" "canoe" "slid" "on" "the" "smooth"
#> [8] "planks."
#>
#> [[2]]
#> [1] "Glue" "the" "sheet" "to" "the"
#> [6] "dark" "blue" "background."
#>
#> [[3]]
#> [1] "It's" "easy" "to" "tell" "the" "depth" "of" "a" "well."
#>
#> [[4]]
#> [1] "These" "days" "a" "chicken" "leg" "is" "a"
#> [8] "rare" "dish."
#>
#> [[5]]
#> [1] "Rice" "is" "often" "served" "in" "round" "bowls."Because each component might contain a different number of pieces, this returns a list. If you’re working with a length-1 vector, the easiest thing is to just extract the first element of the list:
Otherwise, like the other stringr functions that return a list, you can use simplify = TRUE to return a matrix:
sentences %>%
head(5) %>%
str_split(" ", simplify = TRUE)
#> [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,] "The" "birch" "canoe" "slid" "on" "the" "smooth" "planks."
#> [2,] "Glue" "the" "sheet" "to" "the" "dark" "blue" "background."
#> [3,] "It's" "easy" "to" "tell" "the" "depth" "of" "a"
#> [4,] "These" "days" "a" "chicken" "leg" "is" "a" "rare"
#> [5,] "Rice" "is" "often" "served" "in" "round" "bowls." ""
#> [,9]
#> [1,] ""
#> [2,] ""
#> [3,] "well."
#> [4,] "dish."
#> [5,] ""You can also request a maximum number of pieces:
fields <- c("Name: Hadley", "Country: NZ", "Age: 35")
fields %>% str_split(": ", n = 2, simplify = TRUE)
#> [,1] [,2]
#> [1,] "Name" "Hadley"
#> [2,] "Country" "NZ"
#> [3,] "Age" "35"Instead of splitting up strings by patterns, you can also split up by character, line, sentence and word boundary()s:
x <- "This is a sentence. This is another sentence."
str_view_all(x, boundary("word"))Show how separate_rows() is a special case of str_split() + summarise().
29.5 Locations
str_locate() and str_locate_all() give you the starting and ending positions of each match.
These are particularly useful when none of the other functions does exactly what you want.
You can use str_locate() to find the matching pattern, str_sub() to extract and/or modify them.
29.6 stringi
stringr is built on top of the stringi package. stringr is useful when you’re learning because it exposes a minimal set of functions, which have been carefully picked to handle the most common string manipulation functions. stringi, on the other hand, is designed to be comprehensive. It contains almost every function you might ever need: stringi has 256 functions to stringr’s 53.
If you find yourself struggling to do something in stringr, it’s worth taking a look at stringi.
The packages work very similarly, so you should be able to translate your stringr knowledge in a natural way.
The main difference is the prefix: str_ vs. stri_.
29.6.1 Exercises
-
Find the stringi functions that:
- Count the number of words.
- Find duplicated strings.
- Generate random text.
How do you control the language that
stri_sort()uses for sorting?
29.6.2 Exercises
-
What do the
extraandfillarguments do inseparate()? Experiment with the various options for the following two toy datasets. Both
unite()andseparate()have aremoveargument. What does it do? Why would you set it toFALSE?Compare and contrast
separate()andextract(). Why are there three variations of separation (by position, by separator, and with groups), but only one unite?-
In the following example we’re using
unite()to create adatecolumn frommonthanddaycolumns. How would you achieve the same outcome usingmutate()andpaste()instead of unite? Write a function that turns (e.g.) a vector
c("a", "b", "c")into the stringa, b, and c. Think carefully about what it should do if given a vector of length 0, 1, or 2.