3.2 Collecting data using surveys
Data may be collected in many ways (laboratory experiments, taking measurements, field observations, etc.). For both observational and experimental studies, though, collecting data using surveys is common. Surveys are very difficult to do well: question wording is crucial, and surprisingly difficult to get right (Fink 1995).
Questions in a survey may be open-ended (respondents can write their own answers) or closed (respondents select from a small number of possible answers, as in multiple-choice questions). Both open and closed questions have advantages and disadvantages. Answers to open questions usually lend themselves to qualitative analysis.
This section only briefly examines surveys:
3.2.1 Asking survey questions
Many issues must be kept in mind when framing survey questions; here are some.
- Avoid leading questions which may indicate how respondents are expected to answer. Question wording is the usual reason for leading questions.
- Avoid ambiguity: Avoid terms that may be unfamiliar, and questions that are unclear.
- Avoid asking the uninformed, and avoid asking respondents about issues they don't know about (for example, people will even give directions to places that do not even exist (Collett and O’Shea 1976)). Many people will tend to give a response even if they do not understand, but such responses are worthless.
- Avoid complex and double-barrelled questions; these are often hard to understand.
- Avoid problems with confidentiality, which would be considered unethical. Ethics committees usually look very carefully for questions that are unethical. In special cases and with justification, ethics committees may allow such questions.
- Ensure that questions are clearly and precisely worded.
- Ensure that options for multiple-choice questions are mutually exclusive (all answers fit into only one category) and exhaustive (the categories cover all possible options).
Example 3.3 (Leading question) This survey question is a leading question, because the expected response is obvious:
Because bottles from bottled water create enormous amounts of non-biodegradable landfill and hence pose a threat to sensitive native wildlife, do you support a ban on bottled water in Australia?
Example 3.4 (Question wording) Question wording can be important. These two questions would produce different percentages of respondents agreeing:
- Which is easier to buy: cigarettes, beer or marijuana?
- Which is easier to obtain: cigarettes, beer or marijuana?
Example 3.5 (Leading question) Consider this survey question:
Do you like this new orthotic?
Although not obvious, this question may incite respondents to please, since liking is the only option presented. Better would be to ask:
Even better (but more difficult to implement) is to ask the second question above, but randomly chose the order of the 'like' and 'dislike'; that is, ask some respondents if they 'like or dislike' the new orthotic, and others if they 'dislike or like' the new orthotic.Do you like or dislike this new orthotic?
Example 3.6 (Ambiguous question) Consider this survey question:
This question is ambiguous: Faster now compared to what or when?Do children run faster now?
Example 3.7 (Asking the uninformed) Consider this survey question:
Only people involved in the industry are likely to be able to properly answer this question. Nonethless, many people will still give an opinion, even if they are uninformed. This data will be effectively useless (response bias), but the researcher may not realise this.Is the use of fibre composites for waterside recreational purposes likely to cause the material to swell?
Example 3.8 (Unclear wording) Consider this survey question:
It is not clear what a 'yes' answer means.I don't go out of my way to purchase low-fat food unless they are also low in calories but not necessarily salt. Do you agree?
Example 3.9 (Double-barrelled question) Consider this survey question:
This question would be better asked as two separate questions: one asking about jogging, and one about swimming.Do you jog and swim for exercise?
Example 3.10 (Confidentiality) Consider this survey question:
Respondents are unlikely to admit to breaking rules.Do you have a water tank that has been installed illegally, without council permission?
Lemma 3.1 (Survey questions) Consider this survey question:
What is the biggest problem with this survey question?Consider this book that you are currently reading. How useful do you think this book would be for students and young professionals in the field?
Example 3.11 (Mutually exclusive options) Consider this survey question (from Chan et al. (2008)):
Approximately how much time do you spend on attending to patient's medications at the event of a non-critical case? (Includes writing down a medication list, searching for medications)
The options are:
- 0--5 minutes
- 5--10 minutes
- More than 10 minutes
The following (humourous) video shows how survey questions can be manipulated by those not wanting to be ethical:
3.2.2 Online and paper surveys
Surveys may be conducted using paper-based surveys, or online surveys; both have advantages and disadvantages (Porter 2004).
Paper-based surveys require the survey information to be manually entered into the computer for later analysis, which is time consuming and expensive, and prone to data-entry errors. Paper-based surveys can also be costly to prepare, especially if physical mailing and photocopying is necessary. However, people may be more likely to complete paper-based surveys if they are presented with a survey face-to-face and someone waits to collect the completed survey.
Online surveys make data collection easier and data entry easier: data are entered directly onto a computer. This means less manual handling and less chance of data entry errors. Online surveys are also easier to share with a geographically-diverse group of people (for example, through email or social media), but only if the relevant contact details are available. However, online surveys may have a lower response rate, as respondents may be reluctant to click on links in emails (especially from unknown sources), may ignore emails, or the emails may be flagged a spam.