12.2 Scrapping from Web

We will use the rvest package to scrap directly from the web. However, it is sometimes convenient to know what to extract using some minor tools. We will use SelectorGadget from Chrome browser.

With the keyword SelectorGadget, use internet search engine to download and install the file. The program is easy to use. The first click will select area and then subsequent click will include or exclude elements.

To install and load the rvest package, we use the following code:

install.packages("rvest")
library(rvest)

12.2.1 Wikipedia Table

We will do two scrapping exercises:

1. scarp from Wikipedia table, and
2. scrap from an unfriendly website.

The following code extracts the student t’s distribution table from Wikipedia. Using the SelectorGadget, we can see that the table is called .wikitable. Then we will extract that using html_nodes() and then we parse the html data into a dataframe using html_table().

link <-paste0("https://en.wikipedia.org/wiki/",
              "Student%27s_t-distribution")
webpage <- read_html(link)
data <- html_nodes(webpage,".wikitable")
table<- html_table(data[[1]],header = FALSE)

12.2.2 Other Websites

To scarp from unstructural data, then we need to find what is the selector using the SelectorGadget. Then we can read the data as text.

link<-paste0("http://www.fas.nus.edu.sg/ecs/",
             "people/staff.html")
webpage <- read_html(link)
data <- html_nodes(webpage,"br+ table td")
content <-html_text(data)

Then we can transform dataset into dataframe.

df = data.frame(matrix(content,ncol=5,byrow=T),
                stringsAsFactors=FALSE)
colnames(df)<-df[1,]
df[-1,]

## [1] <NA> <NA> <NA> <NA> <NA>
## <0 rows> (or 0-length row.names)

row.names(df) <- NULL
head(df[2:3], n=3)

## [1] <NA> NA.1
## <0 rows> (or 0-length row.names)