10 Procedures for collecting data

So far, you have learnt to ask a RQ, identify different ways of obtaining data, and design the study.

In this chapter, you will learn how to collect the data needed to answer the RQ. You will learn to:

  • record the important steps in data collection.
  • describe study protocols.
  • ask survey questions.
  • describe the basic differences between online and paper surveys.

10.1 Protocols

If the RQ is well-constructed, all terms are clearly defined, and the research design is clear and well explained, then collecting the data should be reasonably easy to implement. However, data collection may be time-consuming, tedious and expensive so getting the data collection correct first time is important.

Before collecting the data, a plan should be established and documented that explains exactly how the data will be obtained.

Unforeseen complications are not unusual, so often a pilot study (or a practice run) is conducted before the real data collection takes place, to see if the planned procedure is practical and optimal. This plan is a draft protocol.

Definition 10.1 (Protocol) A protocol is a procedure documenting the details of the design and implementation of studies, and for data collection.

A pilot study allows the researcher to:

  • Determine the feasibility of the data collection protocol.
  • Identify unforeseen challenges.
  • Obtain data that might help with sample size calculations.
  • Potentially save time and money.

Definition 10.2 (Pilot study) A pilot study is a small test run of the study protocol, used to check that the protocol seems appropriate and practical, and to identify possible problems with the design or protocol.

After the pilot study, the planned protocol may need to be refined. Once the protocol has been finalised, then the data can be collected.

Protocols ensure repeatability of the study (to ensure successful replication of results by others) to enable others to confirm or compare results, and others can understand exactly what was done, and how:

...scientific reports must describe experiments in sufficient detail to allow other researchers to reproduce them.

Pedro Mendes254 (p. 3081)

Protocols should clearly indicate how design aspects (such as blinding the individuals, random allocation of treatments, etc.) will happen. This final record is the final protocol. Someone else should be able to read the protocol and approximately repeat the study (this is part of ethical research practice: Sect. 4). Diagrams can be useful to aid explanations. All studies should have a well-established protocol for describing how the study was done.

Example 10.1 (Protocol) A study255 examined the forward-leaning angle from which people could recover and not fall, to determine if this angle was different (on average) for younger and older people.

The paper goes into great detail to explain the protocol (almost 1.5 pages, plus a diagram).

Example 10.2 (Protocol) Consider this partial protocol, which shows ethics and honesty in describing a protocol:

Fresh cow dung was obtained from free-ranging, grass fed, and antibiotic-free Milking Shorthorn cows (Bos taurus) in the Tilden Regional Park in Berkeley, CA. Resting cows were approached with caution and startled by loud shouting, whereupon the cows rapidly stood up, defecated, and moved away from the source of the annoyance. Dung was collected in ZipLoc bags (1 gallon), snap-frozen and stored at \(-80\) C.

--- Emily E. Hare et al.256, p. 10

A study257 examined three different types of male catheters, to compare 'withdrawal friction force'. The paper goes into great detail to explain the protocols (almost a whole page, plus a (painful-looking) diagram).

One approach to documenting the data collection process is to use the STAR method:

  • S (Structured): The protocol is organised logically, with attention to detail.
  • T (Transparent): All the necessary information is provided: the protocol is transparent, comprehensive and accurate.
  • A (Accessible): The protocol is easy to access, easy to follow, and easy to comprehend.
  • R (Reporting): The protocol is reported in whole and in detail, so it could be replicated.

10.2 Collecting data using surveys

Data may be collected in many ways (laboratory experiments, taking measurements, field observations, etc.). For both observational and experimental studies, though, collecting data using surveys is common. Surveys are very difficult to do well: question wording is crucial, and surprisingly difficult to get right.258

Questions in a survey may be open-ended (respondents can write their own answers) or closed (respondents select from a small number of possible answers, as in multiple-choice questions). Both open and closed questions have advantages and disadvantages. Answers to open questions usually lend themselves to qualitative analysis.

This section only briefly examines surveys:

10.2.1 Asking survey questions

Many issues must be kept in mind when framing survey questions; here are some.

  • Avoid leading questions which may indicate how respondents are expected to answer. Question wording is the usual reason for leading questions.
  • Avoid ambiguity: Avoid terms that may be unfamiliar, and questions that are unclear.
  • Avoid asking the uninformed, and avoid asking respondents about issues they don't know about (for example, people will even give directions to places that do not even exist).259 Many people will tend to give a response even if they do not understand, but such responses are worthless.
  • Avoid complex and double-barrelled questions; these are often hard to understand.
  • Avoid problems with confidentiality, which would be considered unethical. Ethics committees usually look very carefully for questions that are unethical. In special cases and with justification, ethics committees may allow such questions.
  • Ensure that questions are clearly and precisely worded.
  • Ensure that options for multiple-choice questions are mutually exclusive (all answers fit into only one category) and exhaustive (the categories cover all possible options).

Example 10.3 (Leading question) This survey question is a leading question, because the expected response is obvious:

Because bottles from bottled water create enormous amounts of non-biodegradable landfill and hence pose a threat to sensitive native wildlife, do you support a ban on bottled water in Australia?

Example 10.4 (Question wording) Question wording can be important. These two questions would produce different percentages of respondents agreeing:

  • Which is easier to buy: cigarettes, beer or marijuana?
  • Which is easier to obtain: cigarettes, beer or marijuana?

Example 10.5 (Leading question) Consider this survey question:

Do you like this new orthotic?

Although not obvious, this question may incite respondents to please, since liking is the only option presented. Better would be to ask:

Do you like or dislike this new orthotic?

Even better (but more difficult to implement) is to ask the second question above, but randomly chose the order of the 'like' and 'dislike'; that is, ask some respondents if they 'like or dislike' the new orthotic, and others if they 'dislike or like' the new orthotic.

Example 10.6 (Ambiguous question) Consider this survey question:

Do children run faster now?

This question is ambiguous: Faster now compared to what or when?

Example 10.7 (Asking the uninformed) Consider this survey question:

Is the use of fibre composites for waterside recreational purposes likely to cause the material to swell?

Only people involved in the industry are likely to be able to properly answer this question. Nonetheless, many people will still give an opinion, even if they are uninformed. This data will be effectively useless (response bias), but the researcher may not realise this.

Example 10.8 (Unclear wording) Consider this survey question:

I don't go out of my way to purchase low-fat food unless they are also low in calories but not necessarily salt.
Do you agree or disagree?

It is not clear what a 'yes' answer means.

Example 10.9 (Double-barrelled question) Consider this survey question:

Do you jog and swim for exercise?

This question would be better asked as two separate questions: one asking about jogging, and one about swimming.

Example 10.10 (Confidentiality) Consider this survey question:

Do you have a water tank that has been installed illegally, without council permission?

Respondents are unlikely to admit to breaking rules.

Consider this survey question:

Consider this book that you are currently reading. How useful do you think this book would be for students and young professionals in the field?

What is the biggest problem with this survey question?

There are two questions; it is double-barrelled.

It is better is to ask the two questions separately: one about students, and one about young professionals.

This allows us to separate the two components of the original survey question.

Example 10.11 (Mutually exclusive options) Consider this survey question (from Esther Chan et al.260):

Approximately how much time do you spend on attending to patient's medications at the event of a non-critical case? (Includes writing down a medication list, searching for medications)

The options are:

  • 0--5 minutes
  • 5--10 minutes
  • More than 10 minutes

This is a poor question, because a respondent does not know which option to select for an answer of "5 minutes".

The following (humourous) video shows how survey questions can be manipulated by those not wanting to be ethical:

10.2.2 Online and paper surveys

Surveys may be conducted using paper-based surveys, or online surveys; both have advantages and disadvantages.261

Paper-based surveys require the survey information to be manually entered into the computer for later analysis, which is time consuming and expensive, and prone to data-entry errors.

Paper-based surveys can also be costly to prepare, especially if physical mailing and photocopying is necessary. However, people may be more likely to complete paper-based surveys if they are presented with a survey face-to-face and someone waits to collect the completed survey.

Online surveys make data collection easier and data entry easier: data are entered directly onto a computer. This means less manual handling and less chance of data entry errors. Online surveys are also easier to share with a geographically-diverse group of people (for example, through email or social media), but only if the relevant contact details are available.

However, online surveys may have a lower response rate, as respondents may be reluctant to click on links in emails (especially from unknown sources), may ignore emails, or the emails may be flagged a spam.

10.3 Summary

Having a detailed procedure for collecting the data (the protocol) is important.

Using a pilot study to trial the protocol an often reveal unexpected changes necessary for a good protocol.

Sometimes, data can be collected using surveys, either on paper or online. However, creating good survey questions is far more difficult than it looks...

10.4 Quick review questions

  1. What is the biggest problem with this survey question: 'Do you have bromodosis?' (Possible answers are: Yes/No)

  2. What is the biggest problem with this survey question: Do you spend too much time connected to the internet? (Possible answers are: Yes/No)

  3. What is the biggest problem with this survey question: 'Do you eat fruits and vegetables?' (Possible answers are: Yes/No)

  4. Which of these is a purpose of producing a well-defined protocol?

    • It allows the researchers to make the study externally valid.
    • It ensures that others know exactly what was done.
    • It ensures that the study is repeatable for others.
  5. Are the following survey questions likely to be leading questions?

    • Do you, or do you not, believe that permeable pavements are a viable alternative to traditional pavements?
    • Do you support a ban on bottled water?
    • Do you believe that double-gloving by paramedics reduces the risk of infection, increases the risk of infection, or makes no difference to the risk of infection?
    • Do you think that Australia should ban breakfast cereals with unhealthy levels of sugar?

Progress:

  1. Language: Do you know what 'bromodosis'? Most people do not. So how can they answer the question truthfully?
  2. Ambiguous: 'Too much', compared to what?
  3. Double-barrelled: Some people may eat fruits but not vegetables for example.

10.5 Exercises

Selected answers are available in Sect. D.10.

Exercise 10.1 What is the problem with this survey question:

What is your age? (Select one option)

  • Under 18
  • Over 18

Exercise 10.2 Which of these survey questions is better, and why?

  1. Should concerned dog owners vaccinate their pets?
  2. Should dogs be required to be vaccinated or not?

Exercise 10.3 Before the 2019 State of the Union address, American president Donald Trump distributed an online survey to gather information. Some of the questions are given below. Critique these questions:

  1. Do you agree that President Trump is taking our country in the RIGHT DIRECTION?

    • Yes
    • No
    • No Opinion
    • Other, please specify:
  2. Do you agree with President Trump's unwavering commitment to, and respect for, our incredible veterans and TROOPS?

    • Yes
    • No
    • No Opinion
    • Other, please specify:
  3. Are you satisfied with President Trump's efforts to revitalize American manufacturing?

    • Yes
    • No
    • No Opinion
    • Other, please specify: