6 Surveys and Survey Experiments with Qualtrics
In this chapter, we will go from using data that someone else collected to collecting data of our own using Qualtrics. Qualtrics makes collecting survey data online easy. If you have ever used something like Google Forms, it is, in many respects, quite similar, except that it is much more powerful. As you will see, we can quickly and easily build a survey experiment, administer the experiment, download the results, and import them into R. Let’s get started!
6.1 Creating a Qualtrics account
Stanford is kind enough to provide all of us with access to Qualtrics. Navigate to https://uit.stanford.edu/service/survey to get started.
Click on the big, green “Set up a Qualtrics account” button and it will tell you to go to https://stanforduniversity.qualtrics.com. Go to that link!
There, it will ask if you have a preexisting Qualtrics account or not. Click the button that pertains to you. If you don’t have a preexisting account, it will either prompt you to log-in to your Stanford account via WebAuth, or else it will automatically log-in for you and set up your account if you are already logged in.
Now you should be into Qualtrics! Since you haven’t made a project previously, you only have one option: Create new project. Everytime you want to make a new survey you will click this button. We will make an example survey to show you the ropes.
It will bring you to a page with a bunch of options for creating a new survey. You can start from these pre-designed surveys if they fall in line with the kind of survey you want to run. There are even pre-built surveys for academics. For learning purposes though, we will start from scratch by clicking “Create your own”.
Now it asks us to name our project. I called it Trial Survey.
6.2 Building a survey
With a project created, we have been ported to Qualtric’s main user interface, where we can build a survey. Have a look around at all of the different options. Qualtrics is full of tools for creating your ideal survey, but at times that functionality can be a bit overwhelming.
We will start simple. Let’s start by learning how to add a question to our survey. Click the button that says “Text Entry” on the far right side of the interface. It should pull up the following menu.
There are a bunch of options and I recommend experimenting/reading up on all of them. The most essential are Text Entry, Multiple Choice, and Matrix Table. Our first question will use Text Entry. It allows us to ask a question and then have the user put in any value that they like. Multiple Choice on the other hand would let us specify the answers they can provide. For a question like – How old are you? – there are too many potential responses for multiple choice to work well, so text entry will do.
We can change the text at the top of the question box to specify the question the respondent should answer. For example:
Next let’s make a more complicated multiple choice question using the matrix table. It basically allows you to ask multiple multiple choice questions at once.
For example, we can copy a question from the GSS about trust in institutions to see the extent to which our respondents trust different groups, like their family or the government.
Cool! Maybe trust of this sort varies by income - we might expect rich people to support institutions more than the poor, for example, because they have historically benefitted from them.
But what if they ignore the question and put in something other than their income into the question response? How do we make sure they answer in the way that we want them to? We can use validation for that. On the right hand side, below the question type, we can see a bunch of options for each question. Look for the area that says Validation Type. Click Content Validation. It will produce subsidiary options under Content Type. We can click Number to make sure that the respondents put a number in the text box!
If we want to put the income question after age, we can simply drag it upwards so that it comes before the question on trust.
We don’t have to come up with all of the questions ourselves - Qualtrics has a question bank that you can use to find pre-written questions. Just click on the “Import Questions from..” button at the bottom of the page.
I found one in the Survey Library -> Qualtrics Library -> Higher Education section about the quality of your professors… how would you rate us?
Cool, we have created a completely nonsensical survey, but that’s okay.
6.3 Survey options
Let’s say you want to edit some meta-options about the survey and how it behaves, for example, whether respondents can go backwards to previous questions or not. Click the “Survey Options” button near the top of the page and it will pull up a bunch of selections you can make and attributes you can change about the survey.
“Look and Feel” will let you change how your survey looks, though probably not how it feels.
6.4 A quick survey experiment
To set up a survey experiment, we have to set up a block of questions from which respondents will only get one of the N questions in the block. This selection is randomized, so that we can see how the randomization affects their responses. Here is a quick example.
First add a new block:
Now we have two blocks:
Now add the two different versions of the survey prompt to the block. Here I took a question from a survey about how Americans respond to hostage crises.
The last step is to make sure that the questions from this block are sampled randomly for each respondent, and that only one of these questions will be selected. We can achieve this by editing the block’s options. Click block options like so and select the Question Randomization option:
This should pull up the Question Randomization menu. We can choose: no randomization, to randomize the order of questions, or to randomly select N questions from the block to present to the respondent. Select the last optin and set the number to 1 and we will have our first survey experiment.
6.5 Publish and Distribute
You can try out your survey by pressing the blue “Preview” button at the top. Or if you are ready, press the green “Publish” button. Once it is published, we can distribute it to respondents. We could do this on Mechanical Turk, as we will learn in a later class, or we can just send it to our friends, as I want you to do for this week’s lab. To do that click on the “Distributions” tab at the top of the survey. It will ask how you want to distribute your survey.
You can do it by email, or by my preferred way, which is to “Get a single reusable link” which you can send to anyone who you want to take your survey. You can also generate a trackable link for each respondent if you want to keep track of who responded.
Let’s follow the link and fill it out ourselves.
Once you complete both pages, you will be sent to the completion page.
Now inside the Data and Analysis pane in Qualtrics, you should see a response!
We can export it using the “Export & Import” data button and then clicking Export Data. Doing so will pull up this pane.
Export it as a .csv because those files are super easy to load into R. It will download a .zip file to your computer. Once you unzip it, you will be able to load the resulting file into R. First, I would rename it, so that it is easy to input into R. I called the file trial_survey.csv. Drag it to your R project and then you can load it into R using the read.csv() function. Simple!
survey <- read.csv("Data/trial_survey.csv")Cool - what does it look like?
head(survey)## StartDate
## 1 Start Date
## 2 {"ImportId":"startDate","timeZone":"America/Denver"}
## 3 2020-01-15 13:41:21
## EndDate Status
## 1 End Date Response Type
## 2 {"ImportId":"endDate","timeZone":"America/Denver"} {"ImportId":"status"}
## 3 2020-01-15 13:42:45 IP Address
## IPAddress Progress Duration..in.seconds.
## 1 IP Address Progress Duration (in seconds)
## 2 {"ImportId":"ipAddress"} {"ImportId":"progress"} {"ImportId":"duration"}
## 3 68.65.165.140 100 84
## Finished
## 1 Finished
## 2 {"ImportId":"finished"}
## 3 True
## RecordedDate
## 1 Recorded Date
## 2 {"ImportId":"recordedDate","timeZone":"America/Denver"}
## 3 2020-01-15 13:42:46
## ResponseId RecipientLastName
## 1 Response ID Recipient Last Name
## 2 {"ImportId":"_recordId"} {"ImportId":"recipientLastName"}
## 3 R_79wZUWUMY18MclX
## RecipientFirstName RecipientEmail
## 1 Recipient First Name Recipient Email
## 2 {"ImportId":"recipientFirstName"} {"ImportId":"recipientEmail"}
## 3
## ExternalReference LocationLatitude
## 1 External Data Reference Location Latitude
## 2 {"ImportId":"externalDataReference"} {"ImportId":"locationLatitude"}
## 3 37.423004150390625
## LocationLongitude DistributionChannel
## 1 Location Longitude Distribution Channel
## 2 {"ImportId":"locationLongitude"} {"ImportId":"distributionChannel"}
## 3 -122.1638946533203125 anonymous
## UserLanguage Q1
## 1 User Language How old are you?
## 2 {"ImportId":"userLanguage"} {"ImportId":"QID1_TEXT"}
## 3 EN 12
## Q3
## 1 How many dollars did you make this past year, after taxes?
## 2 {"ImportId":"QID3_TEXT"}
## 3 20000
## Q2_1
## 1 I am going to name some groups in this country. As far as the people in these groups are concerned, how much trust do you have for them? - Banks
## 2 {"ImportId":"QID2_1"}
## 3 Hardly any
## Q2_2
## 1 I am going to name some groups in this country. As far as the people in these groups are concerned, how much trust do you have for them? - The Government
## 2 {"ImportId":"QID2_2"}
## 3 Only some
## Q2_3
## 1 I am going to name some groups in this country. As far as the people in these groups are concerned, how much trust do you have for them? - Corporations
## 2 {"ImportId":"QID2_3"}
## 3 Hardly any
## Q2_4
## 1 I am going to name some groups in this country. As far as the people in these groups are concerned, how much trust do you have for them? - Your neighbors
## 2 {"ImportId":"QID2_4"}
## 3 A great deal
## Q2_5
## 1 I am going to name some groups in this country. As far as the people in these groups are concerned, how much trust do you have for them? - Your family members
## 2 {"ImportId":"QID2_5"}
## 3 A great deal
## Q2_6
## 1 I am going to name some groups in this country. As far as the people in these groups are concerned, how much trust do you have for them? - The Media
## 2 {"ImportId":"QID2_6"}
## 3 Only some
## Q5
## 1 Overall, how well do the professors at this university teach?
## 2 {"ImportId":"QID4"}
## 3 Not well at all
## Q5.1
## 1 A terrorist group has taken 100 US citizens hostage and threatens to kill them if the US does not release several of their members currently being held at Guantanamo bay.\nWhat should the President do?
## 2 {"ImportId":"QID5"}
## 3 Agree to the deal
## Q6
## 1 A terrorist group has taken 2 US citizens hostage and threatens to kill them if the US does not release several of their members currently being held at Guantanamo bay.\nWhat should the President do?
## 2 {"ImportId":"QID6"}
## 3Looks great except for the second row! Let’s drop that and then you will be ready to analyze it using the tools from last class.
survey <- survey[-2,]
head(survey)## StartDate EndDate Status IPAddress Progress
## 1 Start Date End Date Response Type IP Address Progress
## 3 2020-01-15 13:41:21 2020-01-15 13:42:45 IP Address 68.65.165.140 100
## Duration..in.seconds. Finished RecordedDate ResponseId
## 1 Duration (in seconds) Finished Recorded Date Response ID
## 3 84 True 2020-01-15 13:42:46 R_79wZUWUMY18MclX
## RecipientLastName RecipientFirstName RecipientEmail
## 1 Recipient Last Name Recipient First Name Recipient Email
## 3
## ExternalReference LocationLatitude LocationLongitude
## 1 External Data Reference Location Latitude Location Longitude
## 3 37.423004150390625 -122.1638946533203125
## DistributionChannel UserLanguage Q1
## 1 Distribution Channel User Language How old are you?
## 3 anonymous EN 12
## Q3
## 1 How many dollars did you make this past year, after taxes?
## 3 20000
## Q2_1
## 1 I am going to name some groups in this country. As far as the people in these groups are concerned, how much trust do you have for them? - Banks
## 3 Hardly any
## Q2_2
## 1 I am going to name some groups in this country. As far as the people in these groups are concerned, how much trust do you have for them? - The Government
## 3 Only some
## Q2_3
## 1 I am going to name some groups in this country. As far as the people in these groups are concerned, how much trust do you have for them? - Corporations
## 3 Hardly any
## Q2_4
## 1 I am going to name some groups in this country. As far as the people in these groups are concerned, how much trust do you have for them? - Your neighbors
## 3 A great deal
## Q2_5
## 1 I am going to name some groups in this country. As far as the people in these groups are concerned, how much trust do you have for them? - Your family members
## 3 A great deal
## Q2_6
## 1 I am going to name some groups in this country. As far as the people in these groups are concerned, how much trust do you have for them? - The Media
## 3 Only some
## Q5
## 1 Overall, how well do the professors at this university teach?
## 3 Not well at all
## Q5.1
## 1 A terrorist group has taken 100 US citizens hostage and threatens to kill them if the US does not release several of their members currently being held at Guantanamo bay.\nWhat should the President do?
## 3 Agree to the deal
## Q6
## 1 A terrorist group has taken 2 US citizens hostage and threatens to kill them if the US does not release several of their members currently being held at Guantanamo bay.\nWhat should the President do?
## 3