4.2 Collecting data using surveys

Data may be collected in many ways (laboratory experiments, taking measurements, field observations, etc.). For both observational and experimental studies, though, collecting data using surveys is common. Surveys are very difficult to do well: question wording is crucial, and surprisingly difficult to get right (Fink 1995).

Questions in a survey may be open-ended (respondents can write their own answers) or closed (respondents select from a small number of possible answers, as in multiple-choice questions). Both open and closed questions have advantages and disadvantages. Answers to open questions usually lend themselves to qualitative analysis.

This section only briefly examines surveys:

4.2.1 Asking survey questions

Many issues must be kept in mind when framing survey questions; here are some.

  • Avoid leading questions which may indicate how respondents are expected to answer. Question wording is the usual reason for leading questions.
  • Avoid ambiguity: Avoid terms that may be unfamiliar, and questions that are unclear.
  • Avoid asking the uninformed, and avoid asking respondents about issues they don't know about (for example, people will even give directions to places that do not even exist (Collett and O’Shea 1976)). Many people will tend to give a response even if they do not understand, but such responses are worthless.
  • Avoid complex and double-barrelled questions; these are often hard to understand.
  • Avoid problems with confidentiality, which would be considered unethical. Ethics committees usually look very carefully for questions that are unethical. In special cases and with justification, ethics committees may allow such questions.
  • Ensure that questions and clearly and precisely worded.
  • Ensure that options for multiple-choice questions are mutually exclusive (all answers fit into only one category) and exhaustive (the categories cover all possible options).

Example 4.3 (Leading question) This survey question is a leading question, because the expected response is obvious:

Because bottles from bottled water create enormous amounts of non-biodegradable landfill and hence pose a threat to sensitive native wildlife, do you support a ban on bottled water in Australia?

Example 4.4 (Question wording) Question wording can be important. These two questions would produce different percentages of respondents agreeing:

  • Which is easier to buy: cigarettes, beer or marijuana?
  • Which is easier to obtain: cigarettes, beer or marijuana?

Example 4.5 (Leading question) Consider this survey question:

Do you like this new orthotic?

Although not obvious, this question may incite respondents to please, since liking is the only option presented. Better would be to ask:

Do you like or dislike this new orthotic?

Even better (but more difficult to implement) is to ask the second question above, but randomly chose the order of the 'like' and 'dislike'; that is, ask some respondents if they 'like or dislike' the new orthotic, and others if they 'dislike or like' the new orthotic.

Example 4.6 (Ambiguous question) Consider this survey question:

Do children run faster now?

This question is ambiguous: Faster now compared to what or when?

Example 4.7 (Asking the uninformed) Consider this survey question:

Is the use of fibre composites for waterside recreational purposes likely to cause the material to swell?

Only people involved in the industry are likely to be able to properly answer this question. Nonethless, many people will still give an opinion, even if they are uninformed. This data will be effectively useless (response bias), but the researcher may not realise this.

Example 4.8 (Unclear wording) Consider this survey question:

I don't go out of my way to purchase low-fat food unless they are also low in calories but not necessarily salt. Do you agree?

It is not clear what a 'yes' answer means.

Example 4.9 (Double-barrelled question) Consider this survey question:

Do you jog and swim for exercise?

This question would be better asked as two separate questions: one asking about jogging, and one about swimming.

Example 4.10 (Confidentiality) Consider this survey question:

Do you have a water tank that has been installed illegally, without council permission?

Respondents are unlikely to admit to breaking rules.

Lemma 4.1 (Survey questions) Consider this survey question:

Consider this book that you are currently reading. How useful do you think this book would be for students and young professionals in the field?

What is the biggest problem with this survey question?
There are two questions; it is double-barrelled. Better is to ask the two questions separately, one about students and one about young professionals. This allows us to separate the two components of the original survey question.

Example 4.11 (Mutually exclusive options) Consider this survey question (from Chan et al. (2008)):

Approximately how much time do you spend on attending to patient's medications at the event of a non-critical case? (Includes writing down a medication list, searching for medications)

The options are:

  • 0--5 minutes
  • 5--10 minutes
  • More than 10 minutes
This is a poor question, because a respondent does not know which option to select for an answer of "5 minutes".

The following (humourous) video shows how survey questions can be manipulated by those not wanting to be ethical:

4.2.2 Online and paper surveys

Surveys may be conducted using paper-based surveys, or online surveys; both have advantages and disadvantages (Porter 2004).

Paper-based surveys require the survey information to be manually entered into the computer for later analysis, which is time consuming and expensive, and prone to data-entry errors. Paper-based surveys can also be costly to prepare, especially if physical mailing and photocopying is necessary. However, people may be more likely to complete paper-based surveys if they are presented with a survey face-to-face and someone waits to collect the completed survey.

Online surveys make data collection easier and data entry easier: data are entered directly onto a computer. This means less manual handling and less chance of data entry errors. Online surveys are also easier to share with a geographically-diverse group of people (for example, through email or social media), but only if the relevant contact details are available. However, online surveys may have a lower response rate, as respondents may be reluctant to click on links in emails (especially from unknown sources), may ignore emails, or the emails may be flagged a spam.


Chan, Esther, Simone Taylor, Jennifer Marriott, and Bill Barger. 2008. “Exploration of Attitudes and Barriers to Bringing Patient’s Own Medications to the Emergency Department: A Survey of Paramedics.” Australasian Journal of Paramedicine 6 (4).
Collett, Peter, and Gregory O’Shea. 1976. “Pointing the Way to a Fictional Place: A Study of Direction Giving in Iran and England.” European Journal of Social Psychology 6 (4): 447–58.
Fink, Arlene. 1995. The Survey Handbook. The Survey Kit. SAGE Publications, Incorporated.
Porter, Stephen R. 2004. “Pros and Cons of Paper and Electronic Surveys.” New Directions for Institutional Research 2004 (121): 91–97.