12.4 Lab 13: Sentiment analysis
First we’ll need some data that contains text of any sort. We’ll work with the large set of tweets that you know from before. You can find a description here.
library(DBI) db <- dbConnect(RSQLite::SQLite(), "./www/tweets-sentiment-db.sqlite") dbListTables(db)
##  "table_tweets"
##  "target" "ids" "date" "flag" "user" "text"
data <- dbGetQuery(db, "SELECT target, ids, date, user, text FROM table_tweets ORDER BY RANDOM() LIMIT 10000") dbDisconnect(db)
Then we’ll need to load a couple of packages necessary to do the corresponding analyses.
# Load Packages library("tm") library("SnowballC") library("wordcloud") library("RColorBrewer") library("tidytext") library("kableExtra")
And we convert the text into the tidy text format as described in the book Text Mining with R by Julia Silge and David Robinson.
# Put text into tidytext format text_df <- data_frame(line = 1:length(data$text), text = data$text, user = data$user, date = data$date) # text_df # Tokenization and tranform into tidy structure text_df <- text_df %>% unnest_tokens(word, text)
Below we produce a table the contains the most frequent words in the tweets we analyze (with stopwords).
Q: What are stopwords?
kable(text_df %>% # # Count words (frequency) anti_join(stop_words) %>% dplyr::count(word, sort = TRUE) %>% top_n(20), caption = 'Most frequent words in responses without stopwords') %>% kable_styling(bootstrap_options = c("striped", "hover", "condensed")) %>% scroll_box(height = "400px")
And naturally we can also visualize that data in a barplot containing the most frequent words.
# library(ggplot2) text_df %>% # generate barplot dplyr::count(word, sort = TRUE) %>% anti_join(stop_words) %>% top_n(20) %>% mutate(word = reorder(word, n)) %>% ggplot(aes(word, n)) + geom_col() + xlab(NULL) + coord_flip()
Below we produce a table the contains the most frequent words in the tweets we analyze without stopwords.
A word cloud is another way to visualized the most frequent words in documents, e.g. tweets.
library(wordcloud) text_df %>% # generate a wordcloud filter(word!="crime") %>% anti_join(stop_words) %>% dplyr::count(word) %>% with(wordcloud(word, n, max.words = 100))
Finally, let’s try to assess the sentiment of those tweets. Wikipedia describes sentiment analysis as follows:
“Opinion mining (sometimes known as sentiment analysis or emotion AI) refers to the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information. Sentiment analysis is widely applied to voice of the customer materials such as reviews and survey responses, online and social media, and healthcare materials for applications that range from marketing to customer service to clinical medicine.
Generally speaking, sentiment analysis aims to determine the attitude of a speaker, writer, or other subject with respect to some topic or the overall contextual polarity or emotional reaction to a document, interaction, or event. The attitude may be a judgment or evaluation (see appraisal theory), affective state (that is to say, the emotional state of the author or speaker), or the intended emotional communication (that is to say, the emotional effect intended by the author or interlocutor)."
Dictionary-based methods like the one we use below find the total sentiment of a document (e.g. a tweet) by adding up the individual sentiment scores for each word in the text.
Below we produce a table of the frequency of words reflecting joy that appear in the tweets. Beforehand we have to define which words reflect joy. There are different dictionaries that provide us with respective words.
# First we need to define a set of words reflecting "joy" for instance nrcjoy <- get_sentiments("nrc") %>% filter(sentiment == "joy") x <- text_df %>% inner_join(nrcjoy) %>% dplyr::count(word, sort = TRUE) knitr::kable(x[1:10,], caption = 'Frequency of happy words') %>% kable_styling(bootstrap_options = c("striped", "hover", "condensed"))
Now we’ll add sentiment scores to the original table that contains our tweets.
library(tidyr) x <- text_df %>% inner_join(get_sentiments("bing")) %>% # add sentiment per word dplyr::count(line, index = line %/% 80, sentiment) %>% # count up positive and negative words per line/tweet spread(sentiment, n, fill = 0) %>% # spread out frequency of postivie/negative words in two columns mutate(sentiment = positive - negative) # calculate difference between number of positive and negative words knitr::kable(x[1:10,], caption = 'Sentiment of individual statements/lines') %>% kable_styling(bootstrap_options = c("striped", "hover", "condensed"))
# Add to sentiment estimates to dataset data$line <- 1:nrow(data) # generate line id data <- left_join(data, x[, c(1, 3:5)])
Then we can produce a table with the most frequent positive and negative words.
bing_word_counts <- text_df %>% inner_join(get_sentiments("bing")) %>% dplyr::count(word, sentiment, sort = TRUE) %>% ungroup() knitr::kable(bing_word_counts[1:10,], caption = 'Most frequent positive or negative words.') %>% kable_styling(bootstrap_options = c("striped", "hover", "condensed"))
…and a barplot of the same…
Finally, a wordcloud with the postive and negative words.
And as a final plot a histogram of the sentiment scores across all tweets in our dataset.
library(plotly) plot_ly(x = data$sentiment, type = "histogram") %>% layout(yaxis = list(title = "N"), xaxis = list(title = "Distribution of Sentiment Scores (Histogram)"))
Finally, let’s aggregate the data by user to get a ranking as to be able to compare users with regard to their overall sentiment.
library(tidyr) data %>% group_by(user) %>% dplyr::summarise(n_tweets = n(), avg_sentiment_score = mean(sentiment, na.rm=TRUE)) %>% arrange(desc(n_tweets), avg_sentiment_score)
- To what kind of data could we apply sentiment analysis?
- Actors? Groups? Text?
- What phenomena could we study?
- To what kind of data could we apply sentiment analysis?