8.6 Lab 2: Scraping (many) tables
When there are many tables on a website, scraping them becomes a bit more complicated. Let’s work through a common case scenario: scraping a table from Wikipedia with a list of the most populated cities in the United States.
url <- 'https://en.wikipedia.org/wiki/List_of_United_States_cities_by_population'
html <- read_html(url)
tables <- html_table(html, fill=TRUE) # What does "fill" do? ?html_table
length(tables)## [1] 12The function now returns 12 different tables. We had to use the option fill=TRUE because some of the tables appear to have incomplete rows.
In this case, identifying the part of the html code that contains the table is a better approach. To do so, let’s take a look at the source code of the website. In Google Chrome, go to View > Developer > View Source. All browsers should have similar options to view the source code of a website.
In the source code, search for the text of the page (e.g. 2017 rank). Right above it you will see: <table class="wikitable sortable"...">. This is the CSS selector that contains the table. (You can also find this information by right-clicking anywhere on the table, and choosing Inspect).
Now that we now what we’re looking for, let’s use html_nodes() to identify all the elements of the page that have that CSS class. (Note that we need to use a dot before the name of the class to indicate it’s CSS.)
wiki <- html_nodes(html, '.wikitable')
length(wiki)## [1] 7There are 7 tables in total, and we will extract the first one.
data <- html_table(wiki[[2]])
str(data)## 'data.frame': 311 obs. of 11 variables:
## $ 2017rank : int 1 2 3 4 5 6 7 8 9 10 ...
## $ City : chr "New York[6]" "Los Angeles" "Chicago" "Houston[7]" ...
## $ State[5] : chr "New York" "California" "Illinois" "Texas" ...
## $ 2017estimate : chr "8,622,698" "3,999,759" "2,716,450" "2,312,717" ...
## $ 2010Census : chr "8,175,133" "3,792,621" "2,695,598" "2,100,263" ...
## $ Change : chr "+5.47%" "+5.46%" "+0.77%" "+10.12%" ...
## $ 2016 land area : chr "301.5 sq mi" "468.7 sq mi" "227.3 sq mi" "637.5 sq mi" ...
## $ 2016 land area : chr "780.9 km2" "1,213.9 km2" "588.7 km2" "1,651.1 km2" ...
## $ 2016 population density: chr "28,317/sq mi" "8,484/sq mi" "11,900/sq mi" "3,613/sq mi" ...
## $ 2016 population density: chr "10,933/km2" "3,276/km2" "4,600/km2" "1,395/km2" ...
## $ Location : chr "40°39'49<U+2033>N 73°56'19<U+2033>W<U+FEFF> / <U+FEFF>40.6635°N 73.9387°W<U+FEFF> / 40.6635; -73.9387<U+FEFF> ("| __truncated__ "34°01'10<U+2033>N 118°24'39<U+2033>W<U+FEFF> / <U+FEFF>34.0194°N 118.4108°W<U+FEFF> / 34.0194; -118.4108<U+FEFF"| __truncated__ "41°50'15<U+2033>N 87°40'54<U+2033>W<U+FEFF> / <U+FEFF>41.8376°N 87.6818°W<U+FEFF> / 41.8376; -87.6818<U+FEFF> (3 Chicago)" "29°47'12<U+2033>N 95°23'27<U+2033>W<U+FEFF> / <U+FEFF>29.7866°N 95.3909°W<U+FEFF> / 29.7866; -95.3909<U+FEFF> (4 Houston)" ...As in the previous case, we still need to clean the data before we can use it.
We’ll use regular expressions to remove endnotes and commas in the population numbers, and clean the variable names. (We’ll come back to this later in the course.)
data$city_name <- gsub('\\[.*\\]', '', data$City) # What does gsub do?
data$population <- data[,"2017estimate"]
data$population <- as.numeric(gsub(",", "", data$population))
data$rank <- data[,"2017rank"]
data <- data[,c("rank", "population", "city_name")]
# data <- data %>% select(rank, population, city_name)
#data[,c(14,13,12)]Now we’re ready to generate the figure:
library(ggplot2)
library(plotly)
plot_ly(data = data,
x = ~rank,
y = ~population,
text = ~city_name,
type = 'scatter',
mode = 'text+markers',
textposition = 'middle right')Let’s take a subset of the larger cities (population>1000000).
library(plotly)
data.big <- data %>% filter(population>1000000)
plot_ly(data = data.big,
x = ~rank,
y = ~population,
text = ~city_name,
type = 'scatter',
mode = 'text+markers',
textposition = 'middle right',
markers = list(color = "black"))