3.2 Essential dplyr commands

The dplyr package (Wickham et al., 2020) provides many useful commands, but the following 6 verbs are essential for transforming data and computing simple summary statistics:

  1. arrange sorts cases (rows);
  2. filter selects cases (rows) by logical conditions;
  3. select selects and reorders variables (columns);
  4. mutate computes variables (columns) and adds them to existing ones;
  5. summarise collapses multiple values of a variable (rows of a column) to a single one;
  6. group_by changes the unit of aggregation (in combination with mutate and summarise).

The following sections illustrate each of these commands in the context of examples. To keep things simple and entertaining, we use the toy dataset of sw <- dplyr::starwars to introduce the commands, but will proceed to more realistic datasets in the exercises (in Section 3.5).

3.2.1 arrange sorts rows

Using arrange sorts cases (rows) by putting specific variables (columns) in specific orders (e.g., ascending or descending). For instance, we could want to arrange cases (rows) by the name of individuals (in alphabetical order). The dplyr function arrange() let’s us do this by calling:

# (a) Sort rows alphabetically (by name):
arrange(.data = sw, ... = name)

Before we proceed, 2 simple observations will facilitate our future life a lot:

  1. In R, we can generally omit argument names (as long as the order of arguments makes it clear what is meant). Thus, we can write the same command more easily as:
# (b) Sort rows alphabetically (by name):
arrange(sw, name)
  1. In dplyr and other tidyverse packages, we can rewrite commands by using the so called pipe (written by the symbols %>%) of the magrittr package (Bache & Wickham, 2014):
# (c) Sort rows alphabetically (by name):
sw %>% arrange(name)

Think of the pipe as passing whatever is on its left (here: sw) to the first argument of the function on its right (here: .data). (More details about using the pipe operator are provided below in Section 3.3.)

In other words, the last 3 commands (a), (b), and (c) are identical and yield the same output:

#> # A tibble: 87 x 13
#>    name  height  mass hair_color skin_color eye_color birth_year gender
#>    <chr>  <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr> 
#>  1 Ackb…    180    83 none       brown mot… orange          41   male  
#>  2 Adi …    184    50 none       dark       blue            NA   female
#>  3 Anak…    188    84 blond      fair       blue            41.9 male  
#>  4 Arve…     NA    NA brown      fair       brown           NA   male  
#>  5 Ayla…    178    55 none       blue       hazel           48   female
#>  6 Bail…    191    NA black      tan        brown           67   male  
#>  7 Barr…    166    50 black      yellow     blue            40   female
#>  8 BB8       NA    NA none       none       black           NA   none  
#>  9 Ben …    163    65 none       grey, gre… orange          NA   male  
#> 10 Beru…    165    75 brown      light      blue            47   female
#> # … with 77 more rows, and 5 more variables: homeworld <chr>, species <chr>,
#> #   films <list>, vehicles <list>, starships <list>

This output contains the tibble sw, but arranged the rows alphabetically by the variable name, which is exactly what we wanted. Although this is neat, 2 immediate questions are:

  • How can we arrange rows in different (e.g., descending, rather than ascending) orders?

  • How can we arrange rows by more than 1 variable?

Both of these tasks are solved rather intuitively by adjusting our calls to arrange:

# Sort rows in descending order:
sw %>% 
  arrange(desc(name)) 
#> # A tibble: 87 x 13
#>    name  height  mass hair_color skin_color eye_color birth_year gender
#>    <chr>  <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr> 
#>  1 Zam …    168    55 blonde     fair, gre… yellow            NA female
#>  2 Yoda      66    17 white      green      brown            896 male  
#>  3 Yara…    264    NA none       white      yellow            NA male  
#>  4 Wilh…    180    NA auburn, g… fair       blue              64 male  
#>  5 Wick…     88    20 brown      brown      brown              8 male  
#>  6 Wedg…    170    77 brown      fair       hazel             21 male  
#>  7 Watto    137    NA black      blue, grey yellow            NA male  
#>  8 Wat …    193    48 none       green, gr… unknown           NA male  
#>  9 Tion…    206    80 none       grey       black             NA male  
#> 10 Taun…    213    NA none       grey       black             NA female
#> # … with 77 more rows, and 5 more variables: homeworld <chr>, species <chr>,
#> #   films <list>, vehicles <list>, starships <list>
# Sort by multiple variables:
sw %>% 
  arrange(eye_color, gender, desc(height))
#> # A tibble: 87 x 13
#>    name  height  mass hair_color skin_color eye_color birth_year gender
#>    <chr>  <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr> 
#>  1 Taun…    213    NA none       grey       black             NA female
#>  2 Shaa…    178    57 none       red, blue… black             NA female
#>  3 Lama…    229    88 none       grey       black             NA male  
#>  4 Tion…    206    80 none       grey       black             NA male  
#>  5 Kit …    196    87 none       green      black             NA male  
#>  6 Plo …    188    80 none       orange     black             22 male  
#>  7 Gree…    173    74 <NA>       green      black             44 male  
#>  8 Nien…    160    68 none       grey       black             NA male  
#>  9 Gasg…    122    NA none       white, bl… black             NA male  
#> 10 BB8       NA    NA none       none       black             NA none  
#> # … with 77 more rows, and 5 more variables: homeworld <chr>, species <chr>,
#> #   films <list>, vehicles <list>, starships <list>

See ?dplyr::arrange for more help and additional examples.

Details

Note some details on using arrange in the above examples:

  • All basic dplyr commands can be called as verb(.data, ...) or — by using the pipe operator from magrittr — as .data %>% verb(...) (see vignette("magrittr") for details). Importantly, the pipe operator %>% is different from the + operator used in ggplot calls.

  • In contrast to base R commands, sequences of multiple variables in tidyverse commands can be written as comma-separated variables, rather than as vectors of variable names (e.g., c("gender", "height")) and are unquoted.

  • When specifying multiple variables in arrange, their order (x, y, ...) specifies the order or priority of operations (first by x, then by y, etc.).

Practice

  • Arrange the sw data in different ways, combining multiple variables and (ascending and descending) orders.

  • Where are the cases containing missing (NA) values in sorted variables placed?

3.2.2 filter selects rows

Using filter selects cases (rows) by logical conditions or a criterion. It keeps all rows for which the criterion is TRUE and drops all rows for which the criterion is FALSE or NA.

For instance, 2 identical ways to extract all humans from sw are:

# Filter to keep all humans:
filter(sw, species == "Human")

# The same command using the pipe:
sw %>%           # Note: %>% is NOT + (used in ggplot) 
  filter(species == "Human")

and result in:

#> # A tibble: 35 x 13
#>    name  height  mass hair_color skin_color eye_color birth_year gender
#>    <chr>  <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr> 
#>  1 Luke…    172    77 blond      fair       blue            19   male  
#>  2 Dart…    202   136 none       white      yellow          41.9 male  
#>  3 Leia…    150    49 brown      light      brown           19   female
#>  4 Owen…    178   120 brown, gr… light      blue            52   male  
#>  5 Beru…    165    75 brown      light      blue            47   female
#>  6 Bigg…    183    84 black      light      brown           24   male  
#>  7 Obi-…    182    77 auburn, w… fair       blue-gray       57   male  
#>  8 Anak…    188    84 blond      fair       blue            41.9 male  
#>  9 Wilh…    180    NA auburn, g… fair       blue            64   male  
#> 10 Han …    180    80 brown      fair       brown           29   male  
#> # … with 25 more rows, and 5 more variables: homeworld <chr>, species <chr>,
#> #   films <list>, vehicles <list>, starships <list>

To filter by some criterion (here: a test that determines whether species == "Human" is TRUE or FALSE), we needed to know both the variable by which we wanted to filter (here: species) and its value of interest (here: "Human"). Note that the output of applying filter to sw is a new tibble, but this tibble only contains 35 cases (i.e., the humans from sw).

Filtering by more than one condition can be very effective, but requires some knowledge about logical operators:

# Filter by multiple (additive) conditions: 
sw %>%
  filter(height > 180, mass <= 75)  # tall and light individuals
#> # A tibble: 3 x 13
#>   name  height  mass hair_color skin_color eye_color birth_year gender homeworld
#>   <chr>  <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr>  <chr>    
#> 1 Jar …    196    66 none       orange     orange            52 male   Naboo    
#> 2 Adi …    184    50 none       dark       blue              NA female Coruscant
#> 3 Wat …    193    48 none       green, gr… unknown           NA male   Skako    
#> # … with 4 more variables: species <chr>, films <list>, vehicles <list>,
#> #   starships <list>
# The same command using the logical operator (&): 
sw %>%
  filter(height > 180 & mass <= 75)  # tall and light individuals
#> # A tibble: 3 x 13
#>   name  height  mass hair_color skin_color eye_color birth_year gender homeworld
#>   <chr>  <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr>  <chr>    
#> 1 Jar …    196    66 none       orange     orange            52 male   Naboo    
#> 2 Adi …    184    50 none       dark       blue              NA female Coruscant
#> 3 Wat …    193    48 none       green, gr… unknown           NA male   Skako    
#> # … with 4 more variables: species <chr>, films <list>, vehicles <list>,
#> #   starships <list>
# Filter for a range of a specific variable:
sw %>%
  filter(height >= 150, height <= 165)  # (a) using height twice
#> # A tibble: 9 x 13
#>   name  height  mass hair_color skin_color eye_color birth_year gender homeworld
#>   <chr>  <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr>  <chr>    
#> 1 Leia…    150    49 brown      light      brown             19 female Alderaan 
#> 2 Beru…    165    75 brown      light      blue              47 female Tatooine 
#> 3 Mon …    150    NA auburn     fair       blue              48 female Chandrila
#> 4 Nien…    160    68 none       grey       black             NA male   Sullust  
#> 5 Shmi…    163    NA black      fair       brown             72 female Tatooine 
#> 6 Ben …    163    65 none       grey, gre… orange            NA male   Tund     
#> 7 Cordé    157    NA brown      light      brown             NA female Naboo    
#> 8 Dormé    165    NA brown      light      brown             NA female Naboo    
#> 9 Padm…    165    45 brown      light      brown             46 female Naboo    
#> # … with 4 more variables: species <chr>, films <list>, vehicles <list>,
#> #   starships <list>
sw %>%
  filter(between(height, 150, 165))     # (b) using between(...)
#> # A tibble: 9 x 13
#>   name  height  mass hair_color skin_color eye_color birth_year gender homeworld
#>   <chr>  <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr>  <chr>    
#> 1 Leia…    150    49 brown      light      brown             19 female Alderaan 
#> 2 Beru…    165    75 brown      light      blue              47 female Tatooine 
#> 3 Mon …    150    NA auburn     fair       blue              48 female Chandrila
#> 4 Nien…    160    68 none       grey       black             NA male   Sullust  
#> 5 Shmi…    163    NA black      fair       brown             72 female Tatooine 
#> 6 Ben …    163    65 none       grey, gre… orange            NA male   Tund     
#> 7 Cordé    157    NA brown      light      brown             NA female Naboo    
#> 8 Dormé    165    NA brown      light      brown             NA female Naboo    
#> 9 Padm…    165    45 brown      light      brown             46 female Naboo    
#> # … with 4 more variables: species <chr>, films <list>, vehicles <list>,
#> #   starships <list>
# Filter by multiple (alternative) conditions: 
sw %>%
  filter(homeworld == "Kashyyyk" | skin_color == "green")
#> # A tibble: 8 x 13
#>   name  height  mass hair_color skin_color eye_color birth_year gender homeworld
#>   <chr>  <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr>  <chr>    
#> 1 Chew…    228   112 brown      unknown    blue             200 male   Kashyyyk 
#> 2 Gree…    173    74 <NA>       green      black             44 male   Rodia    
#> 3 Yoda      66    17 white      green      brown            896 male   <NA>     
#> 4 Bossk    190   113 none       green      red               53 male   Trandosha
#> 5 Rugo…    206    NA none       green      orange            NA male   Naboo    
#> 6 Kit …    196    87 none       green      black             NA male   Glee Ans…
#> 7 Pogg…    183    80 none       green      yellow            NA male   Geonosis 
#> 8 Tarf…    234   136 brown      brown      blue              NA male   Kashyyyk 
#> # … with 4 more variables: species <chr>, films <list>, vehicles <list>,
#> #   starships <list>

A common criterion for filtering is that we want to (a) only obtain cases with missing values on some variable(s), or (b) only keep cases without missing values on some variable(s):

# (a) Filter cases with missing (NA) values on specific variables:
sw %>%
  filter(is.na(gender))
#> # A tibble: 3 x 13
#>   name  height  mass hair_color skin_color eye_color birth_year gender homeworld
#>   <chr>  <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr>  <chr>    
#> 1 C-3PO    167    75 <NA>       gold       yellow           112 <NA>   Tatooine 
#> 2 R2-D2     96    32 <NA>       white, bl… red               33 <NA>   Naboo    
#> 3 R5-D4     97    32 <NA>       white, red red               NA <NA>   Tatooine 
#> # … with 4 more variables: species <chr>, films <list>, vehicles <list>,
#> #   starships <list>
# (b) Filter cases with existing (non-NA) values on specific variables:
sw %>%
  filter(!is.na(mass), !is.na(birth_year)) 
#> # A tibble: 36 x 13
#>    name  height  mass hair_color skin_color eye_color birth_year gender
#>    <chr>  <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr> 
#>  1 Luke…    172    77 blond      fair       blue            19   male  
#>  2 C-3PO    167    75 <NA>       gold       yellow         112   <NA>  
#>  3 R2-D2     96    32 <NA>       white, bl… red             33   <NA>  
#>  4 Dart…    202   136 none       white      yellow          41.9 male  
#>  5 Leia…    150    49 brown      light      brown           19   female
#>  6 Owen…    178   120 brown, gr… light      blue            52   male  
#>  7 Beru…    165    75 brown      light      blue            47   female
#>  8 Bigg…    183    84 black      light      brown           24   male  
#>  9 Obi-…    182    77 auburn, w… fair       blue-gray       57   male  
#> 10 Anak…    188    84 blond      fair       blue            41.9 male  
#> # … with 26 more rows, and 5 more variables: homeworld <chr>, species <chr>,
#> #   films <list>, vehicles <list>, starships <list>

As filter selects cases, its result should typically be a table with the same number of columns as the original one, but fewer rows. See ?dplyr::filter for more help and additional examples.

Details

Note some details on using filter:

  • Separating multiple conditions by commas is the same as using the logical AND (&).

  • As seen with arrange, variable names are unquoted.

  • A comma between conditions or tests (x, y, ...) means the same as & (logical AND), as each test results in a vector of Boolean values.

  • Unlike in base R, rows for which the condition evaluates to NA are dropped.

  • Additional filter functions include near() for testing numerical (near-)identity.

Practice

  1. Verify for an example that filtering by 2 criteria yields the same result as filtering twice (once for each criterion).
# (a) Filtering by 2 criteria: 
tb1 <- sw %>% 
  filter(height >= 150, height <= 165) %>%
  arrange(name)

# (b) Filtering first by criterion 1, then by criterion 2: 
tb2 <- sw %>%
  filter(height >= 150) %>% 
  filter(height <= 165) %>%
  arrange(name) 

# (c) Filtering first by criterion 2, then by criterion 1: 
tb3 <- sw %>%
  filter(height <= 165) %>%
  filter(height >= 150) %>%
  arrange(name)

# Verify equality (of name variable):
all.equal(tb1$name, tb2$name)
#> [1] TRUE
all.equal(tb1$name, tb3$name)
#> [1] TRUE

Can you explain why we added arrange(name) to the end of each filter pipe?

  1. Use filter on the sw data to select some either diverse or narrow subset of individuals. For instance,
  • which individual with blond hair and blue eyes has an unknown mass?
  • of which species are individuals that are over 2m tall and have brown hair?
  • which individuals from Tatooine are not male (but may be NA)?
  • which individuals are neither male nor female OR heavier than 130kg?
sw %>%
  filter(hair_color == "blond", eye_color == "blue")

sw %>%
  filter(height > 200, hair_color == "brown")

sw %>%
  filter(homeworld == "Tatooine", (gender != "male" | is.na(gender)))

sw %>%
  filter((gender != "male" & gender != "female") | mass > 130)

slice selects rows by number

If we want to select specific rows of a data table and already know their row number, we can use the slice command of dplyr:

slice(sw, 2)    # get the 2nd row
slice(sw, 1:3)  # get the first 3 rows
slice(sw, c(1, nrow(sw)))  # get the 1st and last rows

Strictly speaking, we would not need slice, as we could always create a column that contains the number of the corresponding row and then filter for the numeric values of this column:

tb <- sw # copy data (to keep sw)

# Add a column that contains the row number:
tb$row_nr <- 1:nrow(sw)

# Apply filters to this column:
filter(tb, row_nr == 2)  # get the 2nd row
filter(tb, row_nr == 1:3)  # get the first 3 rows
filter(tb, row_nr == 1 | row_nr == nrow(sw))  # get the 1st and last rows

However, the last example shows that the tests for specific numeric values of row_nr can get cumbersome. Hence, slice is a welcome addition to our dplyr vocabulary.

Practice

Predict the outcome of slice(sw, 1:nrow(sw)) and then evaluate the expression and your prediction.

slice(sw, 1:nrow(sw))

3.2.3 select selects columns

Using select selects variables (columns) by their names or numbers. As it works exactly like filter (but selects columns, rather than rows), we can immediately select not just one, but multiple variables. Actually, there are many ways of achieving the same result:

# Select 4 specific variables (columns) of sw:
select(sw, name, species, birth_year, gender)
#> # A tibble: 87 x 4
#>    name               species birth_year gender
#>    <chr>              <chr>        <dbl> <chr> 
#>  1 Luke Skywalker     Human         19   male  
#>  2 C-3PO              Droid        112   <NA>  
#>  3 R2-D2              Droid         33   <NA>  
#>  4 Darth Vader        Human         41.9 male  
#>  5 Leia Organa        Human         19   female
#>  6 Owen Lars          Human         52   male  
#>  7 Beru Whitesun lars Human         47   female
#>  8 R5-D4              Droid         NA   <NA>  
#>  9 Biggs Darklighter  Human         24   male  
#> 10 Obi-Wan Kenobi     Human         57   male  
#> # … with 77 more rows
# The same when using the pipe:
sw %>%           # Note: %>% is NOT + (used in ggplot) 
  select(name, species, birth_year, gender)
#> # A tibble: 87 x 4
#>    name               species birth_year gender
#>    <chr>              <chr>        <dbl> <chr> 
#>  1 Luke Skywalker     Human         19   male  
#>  2 C-3PO              Droid        112   <NA>  
#>  3 R2-D2              Droid         33   <NA>  
#>  4 Darth Vader        Human         41.9 male  
#>  5 Leia Organa        Human         19   female
#>  6 Owen Lars          Human         52   male  
#>  7 Beru Whitesun lars Human         47   female
#>  8 R5-D4              Droid         NA   <NA>  
#>  9 Biggs Darklighter  Human         24   male  
#> 10 Obi-Wan Kenobi     Human         57   male  
#> # … with 77 more rows
# The same when providing a vector of variable names: 
sw %>%
  select(c(name, species, birth_year, gender)) 
#> # A tibble: 87 x 4
#>    name               species birth_year gender
#>    <chr>              <chr>        <dbl> <chr> 
#>  1 Luke Skywalker     Human         19   male  
#>  2 C-3PO              Droid        112   <NA>  
#>  3 R2-D2              Droid         33   <NA>  
#>  4 Darth Vader        Human         41.9 male  
#>  5 Leia Organa        Human         19   female
#>  6 Owen Lars          Human         52   male  
#>  7 Beru Whitesun lars Human         47   female
#>  8 R5-D4              Droid         NA   <NA>  
#>  9 Biggs Darklighter  Human         24   male  
#> 10 Obi-Wan Kenobi     Human         57   male  
#> # … with 77 more rows
# The same when providing column numbers:
sw %>%
  select(1, 10, 7, 8) 
#> # A tibble: 87 x 4
#>    name               species birth_year gender
#>    <chr>              <chr>        <dbl> <chr> 
#>  1 Luke Skywalker     Human         19   male  
#>  2 C-3PO              Droid        112   <NA>  
#>  3 R2-D2              Droid         33   <NA>  
#>  4 Darth Vader        Human         41.9 male  
#>  5 Leia Organa        Human         19   female
#>  6 Owen Lars          Human         52   male  
#>  7 Beru Whitesun lars Human         47   female
#>  8 R5-D4              Droid         NA   <NA>  
#>  9 Biggs Darklighter  Human         24   male  
#> 10 Obi-Wan Kenobi     Human         57   male  
#> # … with 77 more rows
# The same when providing a vector of column numbers: 
sw %>%
  select(c(1, 10, 7, 8)) 
#> # A tibble: 87 x 4
#>    name               species birth_year gender
#>    <chr>              <chr>        <dbl> <chr> 
#>  1 Luke Skywalker     Human         19   male  
#>  2 C-3PO              Droid        112   <NA>  
#>  3 R2-D2              Droid         33   <NA>  
#>  4 Darth Vader        Human         41.9 male  
#>  5 Leia Organa        Human         19   female
#>  6 Owen Lars          Human         52   male  
#>  7 Beru Whitesun lars Human         47   female
#>  8 R5-D4              Droid         NA   <NA>  
#>  9 Biggs Darklighter  Human         24   male  
#> 10 Obi-Wan Kenobi     Human         57   male  
#> # … with 77 more rows

When selecting ranges of variables, the : operator allows selecting ranges of variables:

# Select ranges of variables with ":":
sw %>%
  select(name:mass, gender:species)
#> # A tibble: 87 x 6
#>    name               height  mass gender homeworld species
#>    <chr>               <int> <dbl> <chr>  <chr>     <chr>  
#>  1 Luke Skywalker        172    77 male   Tatooine  Human  
#>  2 C-3PO                 167    75 <NA>   Tatooine  Droid  
#>  3 R2-D2                  96    32 <NA>   Naboo     Droid  
#>  4 Darth Vader           202   136 male   Tatooine  Human  
#>  5 Leia Organa           150    49 female Alderaan  Human  
#>  6 Owen Lars             178   120 male   Tatooine  Human  
#>  7 Beru Whitesun lars    165    75 female Tatooine  Human  
#>  8 R5-D4                  97    32 <NA>   Tatooine  Droid  
#>  9 Biggs Darklighter     183    84 male   Tatooine  Human  
#> 10 Obi-Wan Kenobi        182    77 male   Stewjon   Human  
#> # … with 77 more rows

Selecting can also be used to re-arrange variables. In this case, the function everything() is useful, but refers to every variable not already specified:

# Select to re-order variables (columns) with everything():
sw %>%
  select(species, name, gender, everything())
#> # A tibble: 87 x 13
#>    species name  gender height  mass hair_color skin_color eye_color birth_year
#>    <chr>   <chr> <chr>   <int> <dbl> <chr>      <chr>      <chr>          <dbl>
#>  1 Human   Luke… male      172    77 blond      fair       blue            19  
#>  2 Droid   C-3PO <NA>      167    75 <NA>       gold       yellow         112  
#>  3 Droid   R2-D2 <NA>       96    32 <NA>       white, bl… red             33  
#>  4 Human   Dart… male      202   136 none       white      yellow          41.9
#>  5 Human   Leia… female    150    49 brown      light      brown           19  
#>  6 Human   Owen… male      178   120 brown, gr… light      blue            52  
#>  7 Human   Beru… female    165    75 brown      light      blue            47  
#>  8 Droid   R5-D4 <NA>       97    32 <NA>       white, red red             NA  
#>  9 Human   Bigg… male      183    84 black      light      brown           24  
#> 10 Human   Obi-… male      182    77 auburn, w… fair       blue-gray       57  
#> # … with 77 more rows, and 4 more variables: homeworld <chr>, films <list>,
#> #   vehicles <list>, starships <list>

A number of additional helper functions allow more sophisticated selections by testing variable names:

# Select variables with helper functions:
sw %>%
  select(starts_with("s"))
#> # A tibble: 87 x 3
#>    skin_color  species starships
#>    <chr>       <chr>   <list>   
#>  1 fair        Human   <chr [2]>
#>  2 gold        Droid   <chr [0]>
#>  3 white, blue Droid   <chr [0]>
#>  4 white       Human   <chr [1]>
#>  5 light       Human   <chr [0]>
#>  6 light       Human   <chr [0]>
#>  7 light       Human   <chr [0]>
#>  8 white, red  Droid   <chr [0]>
#>  9 light       Human   <chr [1]>
#> 10 fair        Human   <chr [5]>
#> # … with 77 more rows
sw %>%
  select(ends_with("s"))
#> # A tibble: 87 x 5
#>     mass species films     vehicles  starships
#>    <dbl> <chr>   <list>    <list>    <list>   
#>  1    77 Human   <chr [5]> <chr [2]> <chr [2]>
#>  2    75 Droid   <chr [6]> <chr [0]> <chr [0]>
#>  3    32 Droid   <chr [7]> <chr [0]> <chr [0]>
#>  4   136 Human   <chr [4]> <chr [0]> <chr [1]>
#>  5    49 Human   <chr [5]> <chr [1]> <chr [0]>
#>  6   120 Human   <chr [3]> <chr [0]> <chr [0]>
#>  7    75 Human   <chr [3]> <chr [0]> <chr [0]>
#>  8    32 Droid   <chr [1]> <chr [0]> <chr [0]>
#>  9    84 Human   <chr [1]> <chr [0]> <chr [1]>
#> 10    77 Human   <chr [6]> <chr [1]> <chr [5]>
#> # … with 77 more rows
sw %>%
  select(contains("_"))
#> # A tibble: 87 x 4
#>    hair_color    skin_color  eye_color birth_year
#>    <chr>         <chr>       <chr>          <dbl>
#>  1 blond         fair        blue            19  
#>  2 <NA>          gold        yellow         112  
#>  3 <NA>          white, blue red             33  
#>  4 none          white       yellow          41.9
#>  5 brown         light       brown           19  
#>  6 brown, grey   light       blue            52  
#>  7 brown         light       blue            47  
#>  8 <NA>          white, red  red             NA  
#>  9 black         light       brown           24  
#> 10 auburn, white fair        blue-gray       57  
#> # … with 77 more rows
sw %>%
  select(matches("or"))
#> # A tibble: 87 x 4
#>    hair_color    skin_color  eye_color homeworld
#>    <chr>         <chr>       <chr>     <chr>    
#>  1 blond         fair        blue      Tatooine 
#>  2 <NA>          gold        yellow    Tatooine 
#>  3 <NA>          white, blue red       Naboo    
#>  4 none          white       yellow    Tatooine 
#>  5 brown         light       brown     Alderaan 
#>  6 brown, grey   light       blue      Tatooine 
#>  7 brown         light       blue      Tatooine 
#>  8 <NA>          white, red  red       Tatooine 
#>  9 black         light       brown     Tatooine 
#> 10 auburn, white fair        blue-gray Stewjon  
#> # … with 77 more rows

As select selects variables, its result should typically be a table with the same number of cases as the original one, but fewer columns. See ?dplyr::select for more help and additional examples, as well as ?dplyr::select_if for conditional variants.

A dplyr function closely related to select is rename, which does exactly what it says:

# Renaming variables:
sw %>%
  rename(creature = name, from_planet = homeworld) 
#> # A tibble: 87 x 13
#>    creature height  mass hair_color skin_color eye_color birth_year gender
#>    <chr>     <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr> 
#>  1 Luke Sk…    172    77 blond      fair       blue            19   male  
#>  2 C-3PO       167    75 <NA>       gold       yellow         112   <NA>  
#>  3 R2-D2        96    32 <NA>       white, bl… red             33   <NA>  
#>  4 Darth V…    202   136 none       white      yellow          41.9 male  
#>  5 Leia Or…    150    49 brown      light      brown           19   female
#>  6 Owen La…    178   120 brown, gr… light      blue            52   male  
#>  7 Beru Wh…    165    75 brown      light      blue            47   female
#>  8 R5-D4        97    32 <NA>       white, red red             NA   <NA>  
#>  9 Biggs D…    183    84 black      light      brown           24   male  
#> 10 Obi-Wan…    182    77 auburn, w… fair       blue-gray       57   male  
#> # … with 77 more rows, and 5 more variables: from_planet <chr>, species <chr>,
#> #   films <list>, vehicles <list>, starships <list>

Details

Note some details on using select:

  • select works both by specifying variable (column) names and by specifying column numbers.

  • Again, variable names are unquoted.

  • The sequence of variable names (separated by commas) specifies the order of columns in the resulting tibble.

  • Selecting and adding everything() allows re-ordering variables.

  • Various helper functions (e.g., starts_with, ends_with, contains, matches, num_range) refer to (parts of) variable names.

  • rename renames specified variables (without quotes) and keeps all other variables.

Practice

  1. What is the result of sw %>% select(height)? More specifically, how does it differ from the vector sw$height?
x <- sw %>% select(height)
y <- sw$height 

# difference: 
is.vector(x)
#> [1] FALSE
is.vector(y)
#> [1] TRUE
  1. Use select on the dplyr::starwars data (sw) to select and re-order specific subsets of variables (e.g., all variables starting with “h”, all even columns, all character variables, etc.).

3.2.4 mutate computes new variables

Using mutate computes new variables (columns) from scratch or existing ones:

# Preparation: Save only a subset variables of sw as sws:   
sws <- select(sw, name:mass, birth_year:species) 
sws    # => 87 cases (rows), but only 7 variables (columns)
#> # A tibble: 87 x 7
#>    name               height  mass birth_year gender homeworld species
#>    <chr>               <int> <dbl>      <dbl> <chr>  <chr>     <chr>  
#>  1 Luke Skywalker        172    77       19   male   Tatooine  Human  
#>  2 C-3PO                 167    75      112   <NA>   Tatooine  Droid  
#>  3 R2-D2                  96    32       33   <NA>   Naboo     Droid  
#>  4 Darth Vader           202   136       41.9 male   Tatooine  Human  
#>  5 Leia Organa           150    49       19   female Alderaan  Human  
#>  6 Owen Lars             178   120       52   male   Tatooine  Human  
#>  7 Beru Whitesun lars    165    75       47   female Tatooine  Human  
#>  8 R5-D4                  97    32       NA   <NA>   Tatooine  Droid  
#>  9 Biggs Darklighter     183    84       24   male   Tatooine  Human  
#> 10 Obi-Wan Kenobi        182    77       57   male   Stewjon   Human  
#> # … with 77 more rows
# Compute 2 new variables and add them to existing ones:
mutate(sws, id = 1:nrow(sw), height_feet = .032808399 * height)
#> # A tibble: 87 x 9
#>    name       height  mass birth_year gender homeworld species    id height_feet
#>    <chr>       <int> <dbl>      <dbl> <chr>  <chr>     <chr>   <int>       <dbl>
#>  1 Luke Skyw…    172    77       19   male   Tatooine  Human       1        5.64
#>  2 C-3PO         167    75      112   <NA>   Tatooine  Droid       2        5.48
#>  3 R2-D2          96    32       33   <NA>   Naboo     Droid       3        3.15
#>  4 Darth Vad…    202   136       41.9 male   Tatooine  Human       4        6.63
#>  5 Leia Orga…    150    49       19   female Alderaan  Human       5        4.92
#>  6 Owen Lars     178   120       52   male   Tatooine  Human       6        5.84
#>  7 Beru Whit…    165    75       47   female Tatooine  Human       7        5.41
#>  8 R5-D4          97    32       NA   <NA>   Tatooine  Droid       8        3.18
#>  9 Biggs Dar…    183    84       24   male   Tatooine  Human       9        6.00
#> 10 Obi-Wan K…    182    77       57   male   Stewjon   Human      10        5.97
#> # … with 77 more rows
# The same using the pipe:
sws %>%
  mutate(id = 1:nrow(sw), height_feet = .032808399 * height)
#> # A tibble: 87 x 9
#>    name       height  mass birth_year gender homeworld species    id height_feet
#>    <chr>       <int> <dbl>      <dbl> <chr>  <chr>     <chr>   <int>       <dbl>
#>  1 Luke Skyw…    172    77       19   male   Tatooine  Human       1        5.64
#>  2 C-3PO         167    75      112   <NA>   Tatooine  Droid       2        5.48
#>  3 R2-D2          96    32       33   <NA>   Naboo     Droid       3        3.15
#>  4 Darth Vad…    202   136       41.9 male   Tatooine  Human       4        6.63
#>  5 Leia Orga…    150    49       19   female Alderaan  Human       5        4.92
#>  6 Owen Lars     178   120       52   male   Tatooine  Human       6        5.84
#>  7 Beru Whit…    165    75       47   female Tatooine  Human       7        5.41
#>  8 R5-D4          97    32       NA   <NA>   Tatooine  Droid       8        3.18
#>  9 Biggs Dar…    183    84       24   male   Tatooine  Human       9        6.00
#> 10 Obi-Wan K…    182    77       57   male   Stewjon   Human      10        5.97
#> # … with 77 more rows

A closely related dplyr verb is transmute, which only keeps computed variables and drops all other ones:

# Transmute computes and only keeps new variables:
sws %>%
  transmute(id = 1:nrow(sw), 
            height_feet = .032808399 * height)
#> # A tibble: 87 x 2
#>       id height_feet
#>    <int>       <dbl>
#>  1     1        5.64
#>  2     2        5.48
#>  3     3        3.15
#>  4     4        6.63
#>  5     5        4.92
#>  6     6        5.84
#>  7     7        5.41
#>  8     8        3.18
#>  9     9        6.00
#> 10    10        5.97
#> # … with 77 more rows

Although mutate and transmute compute the same variables, mutate is of an incremental nature (by adding new variables to the existing table), whereas transmute drastically changes the table (by only keeping new variables). For most purposes, adding new columns to the existing table is perfectly fine.

Interestingly, a variable computed by transmute can immediately be used for computing another variable:

# Compute variables based on multiple others (including computed ones):
sws %>%
  mutate(BMI = mass / ((height / 100)  ^ 2),  # compute body mass index (kg/m^2)
         BMI_low  = BMI < 18.5,               # classify low BMI values
         BMI_high = BMI > 30,                 # classify high BMI values
         BMI_norm = !BMI_low & !BMI_high      # classify normal BMI values 
         )
#> # A tibble: 87 x 11
#>    name  height  mass birth_year gender homeworld species   BMI BMI_low BMI_high
#>    <chr>  <int> <dbl>      <dbl> <chr>  <chr>     <chr>   <dbl> <lgl>   <lgl>   
#>  1 Luke…    172    77       19   male   Tatooine  Human    26.0 FALSE   FALSE   
#>  2 C-3PO    167    75      112   <NA>   Tatooine  Droid    26.9 FALSE   FALSE   
#>  3 R2-D2     96    32       33   <NA>   Naboo     Droid    34.7 FALSE   TRUE    
#>  4 Dart…    202   136       41.9 male   Tatooine  Human    33.3 FALSE   TRUE    
#>  5 Leia…    150    49       19   female Alderaan  Human    21.8 FALSE   FALSE   
#>  6 Owen…    178   120       52   male   Tatooine  Human    37.9 FALSE   TRUE    
#>  7 Beru…    165    75       47   female Tatooine  Human    27.5 FALSE   FALSE   
#>  8 R5-D4     97    32       NA   <NA>   Tatooine  Droid    34.0 FALSE   TRUE    
#>  9 Bigg…    183    84       24   male   Tatooine  Human    25.1 FALSE   FALSE   
#> 10 Obi-…    182    77       57   male   Stewjon   Human    23.2 FALSE   FALSE   
#> # … with 77 more rows, and 1 more variable: BMI_norm <lgl>

As mutate typically changes variables (by computing new ones), it seems appropriately named. However, note that mutate does typically not change the identity of the cases (rows) of a data table. See ?dplyr::mutate for more help and additional examples.

Details

Note some details on mutate and transmute:

  • mutate computes new variables (columns) and adds them to existing ones, while transmute drops existing ones.

  • Each mutate command specifies a new variable name (without quotes), followed by = and a rule for computing the new variable from existing ones.

  • Again, variable names are unquoted.

  • Multiple mutate steps are separated by commas, each of which creates a new variable.

  • See http://r4ds.had.co.nz/transform.html#mutate-funs for useful functions for creating new variables.

Practice

Compute a new variable mass_pound from mass (in kg) and the age of each individual in sw relative to Yoda’s age. (Note that the variable birth_year is provided in years BBY, i.e., Before Battle of Yavin.)

# Determine Yoda's age: 
age_yoda <- sws %>%
  filter(name == "Yoda") %>%
  .$birth_year
age_yoda

sws %>% 
  mutate(mass_pound = mass/.45,
         age_since_yoda = age_yoda - birth_year) %>%
  select(name, mass, mass_pound, birth_year, age_since_yoda)

3.2.5 summarise computes summaries

summarise computes a function for a specified variable and collapses the values of the specified variable (i.e., the rows of a specified columns) to a single value:

# Summarise allows computing a function for a variable (column): 
summarise(sw, mn_mass = mean(mass, na.rm = TRUE))  # => 97.31 kg 
#> # A tibble: 1 x 1
#>   mn_mass
#>     <dbl>
#> 1    97.3
# The same using the pipe: 
sw %>%
  summarise(mn_mass = mean(mass, na.rm = TRUE))  # => 97.31 kg 
#> # A tibble: 1 x 1
#>   mn_mass
#>     <dbl>
#> 1    97.3

In most cases, we want to compute not just one summary statistic of a variable (e.g., mass), but several ones:

# Multiple summarise steps allow applying 
# different functions for 1 dependent variable: 
sw %>%
  summarise(n_mass = sum(!is.na(mass)), 
            mn_mass = mean(mass, na.rm = TRUE),
            md_mass = median(mass, na.rm = TRUE),
            sd_mass = sd(mass, na.rm = TRUE),
            max_mass = max(mass, na.rm = TRUE),
            big_mass = any(mass > 1000)
            )
#> # A tibble: 1 x 6
#>   n_mass mn_mass md_mass sd_mass max_mass big_mass
#>    <int>   <dbl>   <dbl>   <dbl>    <dbl> <lgl>   
#> 1     59    97.3      79    169.     1358 TRUE

Similarly, we often want to obtain summary information about more than one variable. For instance, we may want to know basic statistics about the height and weight variables in our sw data, and count some characteristics of character variables:

# Multiple summarise steps also allow applying 
# different functions to different dependent variables: 
sw %>%
  summarise(# Descriptives of height:  
            n_height = sum(!is.na(height)), 
            mn_height = mean(height, na.rm = TRUE),
            sd_height = sd(height, na.rm = TRUE), 
            # Descriptives of mass:
            n_mass = sum(!is.na(mass)), 
            mn_mass = mean(mass, na.rm = TRUE),
            sd_mass = sd(mass, na.rm = TRUE),
            # Counts of character variables:
            n_names = n(), 
            n_species = n_distinct(species),
            n_worlds = n_distinct(homeworld)
            )
#> # A tibble: 1 x 9
#>   n_height mn_height sd_height n_mass mn_mass sd_mass n_names n_species n_worlds
#>      <int>     <dbl>     <dbl>  <int>   <dbl>   <dbl>   <int>     <int>    <int>
#> 1       81      174.      34.8     59    97.3    169.      87        38       49

While summarise provides many different summary statistics by itself, it is even more useful in combination with group_by (discussed next). See ?dplyr::summarise for more help and additional examples.

Details

Note some details on summarise:

  • summarise collapses multiple values into one value and returns a new tibble with as many rows as values computed.

  • Each summarise step specifies a new variable name (without quotes), followed by =, and a function for computing the new variable from existing ones.

  • Multiple summarise steps are separated by commas.

  • Again, variable names are unquoted.

  • See https://dplyr.tidyverse.org/reference/summarise.html for examples and useful functions in combination with summarise.

Practice

  1. Someone speculates that — on average — humans have longer names than droids, but droids are heavier than humans. Can you compute some summaries (e.g., by combining filter with summarise commands) to check this?
    Hint: The length of a character string s can be computed with nchar(s).
# Average name length and mass of humans: 
sw %>%
  filter(species == "Human") %>%
  summarise(n_humans = n(), 
            # Name length:
            mn_name_len = mean(nchar(name)), 
            sd_name_len = sd(nchar(name)),
            # Descriptives of mass (from above):
            n_mass = sum(!is.na(mass)), 
            mn_mass = mean(mass, na.rm = TRUE),
            sd_mass = sd(mass, na.rm = TRUE))
#> # A tibble: 1 x 6
#>   n_humans mn_name_len sd_name_len n_mass mn_mass sd_mass
#>      <int>       <dbl>       <dbl>  <int>   <dbl>   <dbl>
#> 1       35        11.3        4.11     22    82.8    19.4
# Average name length and mass of droids: 
sw %>%
  filter(species == "Droid") %>%
  summarise(n_droids = n(),
            # Name length: 
            mn_name_len = mean(nchar(name)), 
            sd_name_len = sd(nchar(name)),
            # Descriptives of mass (from above):
            n_mass = sum(!is.na(mass)), 
            mn_mass = mean(mass, na.rm = TRUE),
            sd_mass = sd(mass, na.rm = TRUE))
#> # A tibble: 1 x 6
#>   n_droids mn_name_len sd_name_len n_mass mn_mass sd_mass
#>      <int>       <dbl>       <dbl>  <int>   <dbl>   <dbl>
#> 1        5         4.6       0.894      4    69.8    51.0
  • It looks like the average name length is about twice as high for humans (but note that there are only 5 droids in the dataset).

  • The hypothesis about droids being heavier on average is wrong, as the mean differences point in the opposite direction (but both distributions contain missing values and show large variations).

  1. Apply all summary functions mentioned in ?dplyr::summarise to the sw dataset.

3.2.6 group_by aggregates variables

Using group_by does not change the data, but the unit of aggregation for other commands, which is particularly useful in combination with mutate and summarise.

When used by itself, group_by returns the same tibble in a grouped form. For instance, the following commands will group sws by species:

# Grouping does not change the data, but lists its groups: 
group_by(sws, species)  # => 38 groups of species
#> # A tibble: 87 x 7
#> # Groups:   species [38]
#>    name               height  mass birth_year gender homeworld species
#>    <chr>               <int> <dbl>      <dbl> <chr>  <chr>     <chr>  
#>  1 Luke Skywalker        172    77       19   male   Tatooine  Human  
#>  2 C-3PO                 167    75      112   <NA>   Tatooine  Droid  
#>  3 R2-D2                  96    32       33   <NA>   Naboo     Droid  
#>  4 Darth Vader           202   136       41.9 male   Tatooine  Human  
#>  5 Leia Organa           150    49       19   female Alderaan  Human  
#>  6 Owen Lars             178   120       52   male   Tatooine  Human  
#>  7 Beru Whitesun lars    165    75       47   female Tatooine  Human  
#>  8 R5-D4                  97    32       NA   <NA>   Tatooine  Droid  
#>  9 Biggs Darklighter     183    84       24   male   Tatooine  Human  
#> 10 Obi-Wan Kenobi        182    77       57   male   Stewjon   Human  
#> # … with 77 more rows
# The same using the pipe: 
sws %>%
  group_by(species)  # => 38 groups of species
#> # A tibble: 87 x 7
#> # Groups:   species [38]
#>    name               height  mass birth_year gender homeworld species
#>    <chr>               <int> <dbl>      <dbl> <chr>  <chr>     <chr>  
#>  1 Luke Skywalker        172    77       19   male   Tatooine  Human  
#>  2 C-3PO                 167    75      112   <NA>   Tatooine  Droid  
#>  3 R2-D2                  96    32       33   <NA>   Naboo     Droid  
#>  4 Darth Vader           202   136       41.9 male   Tatooine  Human  
#>  5 Leia Organa           150    49       19   female Alderaan  Human  
#>  6 Owen Lars             178   120       52   male   Tatooine  Human  
#>  7 Beru Whitesun lars    165    75       47   female Tatooine  Human  
#>  8 R5-D4                  97    32       NA   <NA>   Tatooine  Droid  
#>  9 Biggs Darklighter     183    84       24   male   Tatooine  Human  
#> 10 Obi-Wan Kenobi        182    77       57   male   Stewjon   Human  
#> # … with 77 more rows

This seems rather mundane, but becomes very powerful when combining the group_by statement with a subsequent mutate or summarise command.

3.2.6.1 Grouped mutates

When combining group_by with a subsequent mutate, the scope of the variables computed by mutate is the group defined by group_by. For instance, the following pipe counts the number of individuals of each species and computes their mean height within each species:

sws %>%
  group_by(species) %>%
  mutate(n_individuals = n(),
         mn_height = mean(height, na.rm = TRUE))
#> # A tibble: 87 x 9
#> # Groups:   species [38]
#>    name  height  mass birth_year gender homeworld species n_individuals
#>    <chr>  <int> <dbl>      <dbl> <chr>  <chr>     <chr>           <int>
#>  1 Luke…    172    77       19   male   Tatooine  Human              35
#>  2 C-3PO    167    75      112   <NA>   Tatooine  Droid               5
#>  3 R2-D2     96    32       33   <NA>   Naboo     Droid               5
#>  4 Dart…    202   136       41.9 male   Tatooine  Human              35
#>  5 Leia…    150    49       19   female Alderaan  Human              35
#>  6 Owen…    178   120       52   male   Tatooine  Human              35
#>  7 Beru…    165    75       47   female Tatooine  Human              35
#>  8 R5-D4     97    32       NA   <NA>   Tatooine  Droid               5
#>  9 Bigg…    183    84       24   male   Tatooine  Human              35
#> 10 Obi-…    182    77       57   male   Stewjon   Human              35
#> # … with 77 more rows, and 1 more variable: mn_height <dbl>

As before, the new variables (here: n_individuals and mn_height) are added to the tibble, but now their values are computed relative to the group_by variable (here: species) as the unit of aggregation. Interestingly, this implies that there exists no such thing as “the mean” of a variable, as any mean is always relative to some unit of aggregation. By changing the unit of aggregation, we can compute many different means for the same variable. For instance, we can compute the mean height of individuals overall, by species, by gender, etc.:

sws %>%
  mutate(mn_height_1 = mean(height, na.rm = TRUE)) %>%  # aggregates over ALL cases
  group_by(species) %>%
  mutate(mn_height_2 = mean(height, na.rm = TRUE)) %>%  # aggregates over current group (species)
  group_by(gender) %>%
  mutate(mn_height_3 = mean(height, na.rm = TRUE)) %>%  # aggregates over current group (gender)
  group_by(name) %>%
  mutate(mn_height_4 = mean(height, na.rm = TRUE)) %>%  # aggregates over current group (name)
  select(name, height, mn_height_1:mn_height_4)
#> # A tibble: 87 x 6
#> # Groups:   name [87]
#>    name               height mn_height_1 mn_height_2 mn_height_3 mn_height_4
#>    <chr>               <int>       <dbl>       <dbl>       <dbl>       <dbl>
#>  1 Luke Skywalker        172        174.        177.        179.         172
#>  2 C-3PO                 167        174.        140         120          167
#>  3 R2-D2                  96        174.        140         120           96
#>  4 Darth Vader           202        174.        177.        179.         202
#>  5 Leia Organa           150        174.        177.        165.         150
#>  6 Owen Lars             178        174.        177.        179.         178
#>  7 Beru Whitesun lars    165        174.        177.        165.         165
#>  8 R5-D4                  97        174.        140         120           97
#>  9 Biggs Darklighter     183        174.        177.        179.         183
#> 10 Obi-Wan Kenobi        182        174.        177.        179.         182
#> # … with 77 more rows

3.2.6.2 Grouped summaries

Our summarise commands above yielded some summary of one or several variables as 1 line of output. When combining group_by with a subsequent summarise, we obtain the corresponding summary for each group:

sws %>%
  group_by(species) %>%
  summarise(n_individuals = n(),
            mn_height = mean(height, na.rm = TRUE),
            mn_mass = mean(mass, na.rm = TRUE)
            ) %>%
  arrange(desc(mn_height))
#> # A tibble: 38 x 4
#>    species  n_individuals mn_height mn_mass
#>    <chr>            <int>     <dbl>   <dbl>
#>  1 Quermian             1      264      NaN
#>  2 Wookiee              2      231      124
#>  3 Kaminoan             2      221       88
#>  4 Kaleesh              1      216      159
#>  5 Gungan               3      209.      74
#>  6 Pau'an               1      206       80
#>  7 Besalisk             1      198      102
#>  8 Cerean               1      198       82
#>  9 Chagrian             1      196      NaN
#> 10 Nautolan             1      196       87
#> # … with 28 more rows

Note that the group_by followed by summarise returns a new tibble, with 38 rows (= groups of species) and

  • 1 column of the group variable (here species) and
  • 3 columns of the 3 newly summarised variables.

Here, we also arranged this output tibble by descending means of height.

3.2.6.3 Grouping by multiple variables

Using group_by with multiple variables yields a tibble containing the combination of all variable levels. For instance, how many combinations of hair_color and eye_color exist, we could count them as follows:

sw %>%
  group_by(hair_color, eye_color)  # => 35 groups (combinations)
#> # A tibble: 87 x 13
#> # Groups:   hair_color, eye_color [35]
#>    name  height  mass hair_color skin_color eye_color birth_year gender
#>    <chr>  <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr> 
#>  1 Luke…    172    77 blond      fair       blue            19   male  
#>  2 C-3PO    167    75 <NA>       gold       yellow         112   <NA>  
#>  3 R2-D2     96    32 <NA>       white, bl… red             33   <NA>  
#>  4 Dart…    202   136 none       white      yellow          41.9 male  
#>  5 Leia…    150    49 brown      light      brown           19   female
#>  6 Owen…    178   120 brown, gr… light      blue            52   male  
#>  7 Beru…    165    75 brown      light      blue            47   female
#>  8 R5-D4     97    32 <NA>       white, red red             NA   <NA>  
#>  9 Bigg…    183    84 black      light      brown           24   male  
#> 10 Obi-…    182    77 auburn, w… fair       blue-gray       57   male  
#> # … with 77 more rows, and 5 more variables: homeworld <chr>, species <chr>,
#> #   films <list>, vehicles <list>, starships <list>

A common application of using group_by with multiple varialbes is to count the number of cases (here: individuals) in each sub-group:

# Counting the frequency of cases in groups:
sw %>%
  group_by(hair_color, eye_color) %>%
  count() %>%
  arrange(desc(n))  
#> # A tibble: 35 x 3
#> # Groups:   hair_color, eye_color [35]
#>    hair_color eye_color     n
#>    <chr>      <chr>     <int>
#>  1 black      brown         9
#>  2 brown      brown         9
#>  3 none       black         9
#>  4 brown      blue          7
#>  5 none       orange        7
#>  6 none       yellow        6
#>  7 blond      blue          3
#>  8 none       blue          3
#>  9 none       red           3
#> 10 black      blue          2
#> # … with 25 more rows
# The same using summarise and n():
sw %>%
  group_by(hair_color, eye_color) %>%
  summarise(n = n()) %>%
  arrange(desc(n))  
#> # A tibble: 35 x 3
#> # Groups:   hair_color [13]
#>    hair_color eye_color     n
#>    <chr>      <chr>     <int>
#>  1 black      brown         9
#>  2 brown      brown         9
#>  3 none       black         9
#>  4 brown      blue          7
#>  5 none       orange        7
#>  6 none       yellow        6
#>  7 blond      blue          3
#>  8 none       blue          3
#>  9 none       red           3
#> 10 black      blue          2
#> # … with 25 more rows

See ?dplyr::group_by for more help and additional examples.

Details

Note some details on group_by:

  • group_by changes the unit of aggregation for other commands (especially mutate and summarise).

  • Again, variable names are unquoted.

  • When using group_by with multiple variables, they are separated by commas.

  • Using group_by with mutate results in a tibble that has the same number of cases (rows) as the original tibble. By contrast, using group_by with summarise results in a new tibble with all combinations of variable levels as its cases (rows).

Practice

  1. In the last practice section above, we used 2 combinations of filter and summarise to check the hypotheses that — on average — humans have longer names than droids, but droids are heavier than humans. Now that we learned about group_by, try to perform this check in 1 pipe.
# Average name length and mass of humans vs. droids: 
sw %>%
  filter(species == "Human" | species == "Droid") %>%
  group_by(species) %>%
  summarise(n_cases = n(), 
            # Name length:
            mn_name_len = mean(nchar(name)), 
            sd_name_len = sd(nchar(name)),
            # Descriptives of mass (from above):
            n_mass = sum(!is.na(mass)), 
            mn_mass = mean(mass, na.rm = TRUE),
            sd_mass = sd(mass, na.rm = TRUE))
#> # A tibble: 2 x 7
#>   species n_cases mn_name_len sd_name_len n_mass mn_mass sd_mass
#>   <chr>     <int>       <dbl>       <dbl>  <int>   <dbl>   <dbl>
#> 1 Droid         5         4.6       0.894      4    69.8    51.0
#> 2 Human        35        11.3       4.11      22    82.8    19.4
  1. Yoda says: “Taller creatures heavier are than smaller ones.”
    Test his hypothesis for the sw dataset in 1 pipe by
  • selecting only the relevant variables name, height, and mass,
  • computing variables for the median height and a logical variable is_tall that is TRUE if and only if an individual is taller than the median height,
  • grouping the data by is_tall;
  • counting the cases and computing the mean mass for tall vs. non-tall individuals.
sw %>%
  select(name, height, mass) %>%
  mutate(md_height = median(height, na.rm = TRUE),
         is_tall = height > md_height) %>%
  group_by(is_tall) %>%
  summarise(n = n(),
            mn_mass = mean(mass, na.rm = TRUE),
            sd_mass = sd(mass, na.rm = TRUE))
#> # A tibble: 3 x 4
#>   is_tall     n mn_mass sd_mass
#>   <lgl>   <int>   <dbl>   <dbl>
#> 1 FALSE      43   103.    234. 
#> 2 TRUE       38    91.1    25.7
#> 3 NA          6   NaN     NaN
  • Yoda seems wrong: The 38 tall creatures (with a height above the median) have an average mass value of 91.1 kg, whereas the 43 smaller creatures (with a height below or equal to the median) have an average mass value of 103 kg.

You just smoked a median-split analysis in a pipe — congratulations! However, before getting too excited, we should try to understand why our results came out in this way. To explain Yoda’s mistake, let’s look at a scatterplot that plots mass as a function of height (and colors the points by the value of our is_tall variable):

sw_tall <- sw %>%
  select(name, height, mass) %>%
  mutate(md_height = median(height, na.rm = TRUE),
         is_tall = height > md_height) 

# All individuals:
ggplot(sw_tall, aes(x = height, y = mass)) + 
  geom_point(aes(color = is_tall), size = 2) + 
  geom_text(aes(label = name), hjust = -.2, angle = 45, size = 2, alpha = 2/3) + 
  coord_cartesian(ylim = c(0, 1700)) +
  scale_color_manual(values = c("firebrick", "steelblue", "gold")) +
  labs(title = "Individual's mass by height",
       x = "Height (in cm)", y = "Mass (in kg)") + 
  theme_classic()
Scatterplot of `mass` by `height` in the full `sw` dataset.

Figure 3.2: Scatterplot of mass by height in the full sw dataset.

Note that we used coord_cartesian to restrict the range of y values shown to ylim = c(0, 1700).

The scatterplot shows that the sw data contains a blatant outlier: Jabba Desilijic Tiure, the crime lord aka. ‘Jabba the Hutt’, has a mass of 1358 kg despite his below-average height of 175 cm. Only considering creatures with a mass up to 170 kg suggests that Yoda’s hypothesis is perfectly valid when this outlier is excluded:

# Only showing mass values from 0 to 180: 
ggplot(sw_tall, aes(x = height, y = mass)) + 
  geom_abline(aes(intercept = lm(mass ~ height)$coefficients[1], slope = lm(mass ~ height)$coefficients[2]), 
              linetype = 2, col = "orange") + 
  # stat_ellipse(aes(color = is_tall), alpha = .5) + 
  geom_point(aes(color = is_tall), size = 2) + 
  geom_text(aes(label = name), hjust = -.2, angle = 45, size = 2, alpha = 2/3) + 
  coord_cartesian(ylim = c(0, 170)) +
  scale_color_manual(values = c("firebrick", "steelblue", "gold")) +
  labs(title = "Individual's mass by height without outlier",
       x = "Height (in cm)", y = "Mass (in kg)") + 
  theme_classic()
Scatterplot of `mass` by `height` without Jabba the Hutt.

Figure 3.3: Scatterplot of mass by height without Jabba the Hutt.

This is yet another instance of the lesson taught by Anscombe’s quartet (in Section 2.1): We should never interpret the results of some statistical calculation without properly inspecting the underlying data.

References

Bache, S. M., & Wickham, H. (2014). magrittr: A forward-pipe operator for R. Retrieved from https://CRAN.R-project.org/package=magrittr

Wickham, H., François, R., Henry, L., & Müller, K. (2020). dplyr: A grammar of data manipulation. Retrieved from https://CRAN.R-project.org/package=dplyr