So far, you have learnt to ask a RQ, identify different ways of obtaining data, and design the study.
In this chapter, you will learn how to obtain the sample to study. You will learn to:
- distinguish, and explain, precision and accuracy.
- distinguish between random and non-random sampling.
- select random samples.
- identify and describe why random samples are preferred over non-random samples.
- identify, describe and use simple random sampling, systematic sampling, stratified sampling, cluster sampling, and multistage sampling.
- identify ways of obtaining samples that are more likely to be representative samples.
A RQ implies that every member of the population should be studied (the P in POCI stand for 'population'). However, being able to do so is very rare because of cost, time, ethics, logistics or practicality. Hence, a subset of the population (a sample) is almost always studied.
A sample consists of some individuals (or cases, or (if the individuals are people) subjects) from the population. The purpose of a sample is to approximate the population.
A study is externally valid if the results can be generalised to other groups in the population, apart from the sample studied. This is only possible if the sample is chosen to well-represent the whole population.
However, since a sample doesn't include every member of the population, conclusions made from samples cannot be certain to apply to the whole population.
In research, the goal is to learn about the population, but only a sample can be studied.
This book is essentially about how to learn about a population based on information from just one of the many possible samples.
Example 5.1 (Samples) A study (based on Richard B. Lipton et al.127) of the effect of aspirin in treating headaches cannot possibly use every single human alive who might one day wish to take aspirin.
Not only would this be prohibitively expensive, time-consuming, and impractical, but such a study would not even use those humans who had not been born yet who might use aspirin. (That is, using the whole target population is impossible.)
A sample must be used.
Having seen that using a sample is necessary, other issues are raised:
- How can we learn something useful about the whole population if only some of that population is studied?
- Which individuals should be included in the sample?
- How many individuals should be included in the sample be?
The last issue must be left until later, after learning more about the implications of studying a sample rather than a population.
Using a sample instead of the entire population presents challenges. Every sample is likely to be a bit different, so what is learnt from a sample depends on which individuals happen to be present in the sample being used. This is called sampling variation. That is, each sample produced different data, and so may lead to different answers to the RQ.
This is the challenge of research: How to make decisions about populations, using an imperfect sample information. Perhaps surprisingly, lots can be learnt about the population if we approach the task of selecting a sample correctly.
Almost always, samples are studied, not populations.
Every sample is likely to be different, and hence the results from every sample are likely to be different. This is called sampling variation.
While we can never be certain about the conclusions from the sample, special tools can be used to help make decisions about the population from a sample.
The animation below shows how the estimates calculated from a sample vary from sample to sample. We know that 50% of cards in a fair pack are red, but each sample of 10 cards can produce a different percentage of red cards (and does not always produce an estimate of 50%).
Which do you think predicted correctly the winner of the election? Why?
|Study||Number in sample||Population||Method|
|A||10 000 000||Specific groups||Voluntary survey|
|B||50 000||All Americans||Random survey|
Two issues concerning sampling, raised in Sect. 5.1, were:
- which individuals should be in the sample, and
- how many individuals should be in the sample be.
These two issues address two different aspects of sampling: precision and accuracy (Fig. 5.1).
Accuracy refers to how close a sample estimate is to the population value (on average). Accuracy is related to the statistical concept of bias. Precision refers to how close all the possible sample estimates are likely to be (that is, how much variation is likely in the sample estimates).
Definition 5.1 (Accuracy) Accuracy refers to how close a sample estimate is to the population value, on average.
Definition 5.2 (Precision) Precision refers to how close the sample estimates from different samples are likely to be to each other.
Using this language:
- The type of sampling (i.e., the way in which the samples is selected) impacts the accuracy of the sample estimate. In other words, the type of sampling impacts the external validity of the study.
- The size of the sample impacts the precision of the sample estimate.
For example, large samples are more likely to be precise estimates because each possible sample value will produce similar estimates, but they may or may not be accurate estimates.
Similarly, random samples are likely to produce accurate estimates (and hence the study is more likely to be externally valid), but they may not be precise unless the sample is also large.
Example 5.2 (Precision and accuracy) To estimate the average age of all Queenslanders, we could ask 9000 Queensland school children (a large sample indeed!).
This will give a precise answer because the sample is large, but inaccurate answer because the sample is not representative of all Queenslanders. In fact, the sample may give a precise and accurate answer to a different question: 'What is the average age of Queensland school children?'
One key to obtaining accurate estimates about the population is to ensure that the sample studied is representative of the population of interest (that is, to ensure the study is externally valid).
So, how can a representative sample of the population be found? Whenever a sample is taken, only some of the population is selected. The selected individuals can be chosen using either random sampling or non-random sampling.
The word random here has a specific meaning that is different than how it is often used in everyday use.
Definition 5.3 (Random) In research and statistics, random means "determined completely by chance".
In a random sample, each individual in the population can be selected on the basis of impersonal chance. (Remember that random means that the sample is determined completely by chance!) Some examples of random sampling methods appear in the following sections (Table 5.2).
The results obtained from a random sample probably generalise to the population from which the sample is drawn; that is, random samples are likely to produce externally valid studies.
|Type||Stage 1||Stage 2||Reference|
|Systematic||Start at a random location||Take every \(n\)th element thereafter||Sect. 5.5|
|Stratified||Split into a few large groups ('strata')||Select simple random sample from every stratum||Sect. 5.6|
|Cluster||Split into many small groups ('clusters'); select simple random sample of clusters||Select all in the chosen clusters||Sect. 5.7|
|Multistage||Select simple random sample from the larger stage||Select simple random sample from those chosen in Stage 1; etc.||Sect. 5.8|
Consider testing a pot of soup by 'sampling'. If the soup is stirred, we don't need to taste the whole pot of soup to see how the soup tastes.
The same principle applies in research: If we use a random sample (analogous to the stirring the soup), we don't need to study every member of the population. If we don't use a random sample (that is, we don't stir the soup), we do not get an overall impression of the population (or the soup).
A non-random sample requires some kind of personal input. Examples of non-random samples include:
- Judgement sample: Individuals are selected, based on the researchers' judgement, depending on whether the researcher thinks they are likely to be agreeable or helpful. For example, researchers may decided to survey people who are not in a hurry.
- Convenience sample: Individuals are selected because they are convenient for the researcher. For example, researchers may gather data from their family and friends.
- Voluntary response (self-selecting) sample: Individuals participate if they wish to. For example, a voluntary response survey, or a TV station call-in survey.
In non-random sampling, those who are in the study may be different than those who are not in the study. That is, non-random samples are not likely to be externally valid.
Using a non-random sample means that the results may not generalise to the intended population: they probably do not produce externally valid studies.
Example 5.3 (Different ways to sample) During the COVID-19 (coronavirus) pandemic in 2020, a Facebook poll asked the question:
Do you think a Coronavirus vaccine should be compulsory?
The result was reported as '79 per cent of Australians oppose a compulsory vaccination', from a sample of over 53,000 responses.
However, this sample was a voluntary response sample, not a random sample, so the results may not be accurate. For example, many anti-vaccination groups instructed their members to flood the poll with 'No' responses (including celebrity chef Pete Evans), and the poll could have been completed by non-Australians as well as Australians.
A different study129 asked Australians:
The Federal Government's 'No Jab, No Pay' policy withholds certain benefits and payments from families who don't fully vaccinate their children. Do you agree with this policy?
In the sample of 1809 respondents, 83.7% either agreed or strongly agreed with this statement.
While this study did not use a random sample, the researchers made efforts to sample a representative cross-section of Australians:
Researchers recruited Australian adults aged 18-years and older to participate in the study through a large, well-established online panel provider. While not a random sample of the Australian population, researchers made efforts to ensure the sample included individuals representing a wide range of demographics (e.g., age, gender, location, income, political preferences, religiosity).
--- Smith, Attwell, and Evers130, p. 194
Furthermore, 'respondents were paid small token sum for their participation in the study' to encourage all selected respondents to provide an answer.
In Sect. 5.11, random and non-random samples are compared using an example.
Definition 5.4 In a simple random sample, every possible sample of the same size has the same chance of being selected.
A simple random sample is chosen from a list of all members of the population (the sampling frame) using tables of random numbers (Appendix B.1) or websites like https://www.random.org. A smaller version of this webpage, which only generates one number at a time, is shown below; just press Generate. The numbers generated by this widget come from the true random number generator at RANDOM.ORG. (The webpage generates as many numbers as you want all at the same time.)
Definition 5.5 (Sampling frame) The sampling frame is a list of all the members of the population (the individuals, or cases, or subjects).
Often, establishing the sampling frame is difficult or impossible, and so finding a random sample is also difficult.
For example, to study a simple random sample of wombats131 would require having a list of all wombats, so some could be selected using random number tables.
Other good (random) sampling methods use a system to select randomly, rather than by human choice (discussed in the following sections).
This book always assumes simple random samples, for simplicity, unless otherwise noted.
Example 5.4 (Simple random sampling) Suppose we are interested in this RQ:
For students at a large course at a particular university, is the average number of letters typed on a keyboard in 10 seconds the same for females and males?
Suppose a sample of 40 students is needed. The sampling frame is the list of all students enrolled. Obtaining the sampling frame is feasible here (lecturers have access to this information for grading).
Suppose budget and time constraints mean only 40 students can be selected for the study above.
Describe how to use the course enrolment list to find a simple random sample of 40 students to study.
We would need a list of all students enrolled in the course.
We could place this list into a spreadsheet, and then use random numbers to select 40 at random (without repeating numbers).
For example, if the number of students enrolled was 204, we would obtain 40 random numbers between 1 and 204 inclusive.
Every student that was chosen using this method would be part of the study.
If a student could not be contacted, more students could be chosen at random to ensure 40 students participated.
In systematic sampling, the first case is randomly selected; then, every (say) fifth element is selected thereafter.
In general, we say that every \(n\)th individual is selected.
There is little advantage in using a systematic sample to take every \(n\)th individual from a convenience sample if it is just as easy and efficient to take every individual with a convenience sample.
Example 5.5 (Systematic sampling) Suppose for a particular study, a sample of 40 students in a particular course is needed.
If 441 students are enrolled, 40 students could be randomly selected, by choosing a number at random between 1 and \(441/40\) (approximately 11) as a starting point; suppose the random number selected is 9. The first student selected is the 9th person in the list (which may be ordered alphabetically, by student ID, or any other means).
Thereafter, every \(441/40\)th person, or 11th person, in the list is selected: people numbered 9, 20, 31, 42...
The animation below shows how a sample of 40 students could be selected from a class of 441 students using a systematic random sample of size 40, by starting at student number 9 and then talking every 11th person.
Care needs to be taken when using systematic samples to ensure a pattern is not hidden.
For example, consider a study where residents in a large, eight-level residential accommodation complex are to be visited and a survey administered. Each floor of the building has a similar layout (Fig. 5.2), with nine apartments per level.
If the researchers decide to systematically sample every tenth apartment, the very same apartment on each floor woud be chosen.
For example, suppose Apartments 1-10, 2-10, 3-10,... 8-10 were chosen. These apartments are all larger than all the other apartments. The residents of these apartments may be wealthier than the other residents, so the systematic sample will not be a representative sample of residents.
In stratified sampling, the population is split into a small number of large (usually homogeneous) groups called strata, then cases are selected using a simple random sample from each stratum.
The strata must be unrelated to the variables.
For example, if the RQ is about comparing the percentage of females and males who wear hats at midday, a stratified sample of size 100 is not obtained by selecting 50 females and 50 males, for example. This is merely selecting people from each level of the explanatory variable.
The sex of the person is the explanatory variable; it does not define the strata.
Example 5.6 (Stratified sampling) To select students in a large course at a particular university, 20 of the females and 20 of the males could be selected. The sample is stratified by sex of the person.
At the university where I work, about 67% of the students are females. So, I could ensure that two-thirds of the sample was females (around 26.7, say 27) and about one-third males (about 13.3, say 13).
The animation below shows how a stratified random sample of size 40 might be selected, by randomly selecting 20 female and 20 male students.
Similarly, the second animation below shows how a stratified random sample of size 40 might be selected, by randomly selecting 27 female and 13 male students.
In cluster sampling, the population is split into a large number of small groups called clusters, then a simple random sample of clusters is selected and every member of the chosen small groups is part of the sample.
Example 5.7 (Cluster sampling) To select students in a large course at a particular university again, a simple random sample of (say) three of the many tutorials could be selected, and every student enrolled in those selected tutorials constitute the sample.
The animation below shows how a sample of approximately 40 students could be obtained using cluster sampling, using the tutorials as clusters.
In multistage sampling, large groups are selected using a simple random sample, then smaller groups within those large groups are selected using a simple random sample. The simple random sampling can continue for as many levels as necessary.
Example 5.8 (Multistage sampling) To select students in a large course at a particular university again, a simple random sample of (say) ten of the many tutorials could be selected (Stage 1), and then 4 people randomly selected from each of these 10 selected tutorials (Stage 2).
The animation below shows how a sample of approximately 40 students could be obtained using multistage sampling, by randomly selecting 10 classes at random in Stage 1, then randomly selecting students from each class in Stage 2.
Example 5.9 (Multistage sampling) Multistage sampling is often used by the Australian Bureau of Statistics (ABS). For example, to obtain a random sample of Queenslanders, one procedure is:
- Stage 1: Randomly select some cities in Queensland;
- Stage 2: Randomly select some suburbs in these chosen cities;
- Stage 3: Randomly select some streets in these chosen suburbs;
- Stage 4: Randomly select some houses in these chosen streets.
This is cheaper than simple random sampling, as data collectors can be employed in a smaller number of Queensland cities (only those chosen in Stage 1).
Obtaining a truly random sample is usually hard or impossible. Often the best we can do is to select a sample that is diverse enough to be somewhat representative of the diversity in the population. Even so, the results from any non-random sample may not generalise to the intended population. The results will generalise to the population which the sample does represent.
Ideally, even if obtaining a random sample is impossible, we would prefer a sample where those in the sample are not likely to be different than those not in the sample, at least for the variables of interest.
Example 5.10 (Representative sample) A randomly-chosen group of Queensland and Northern Territory residents is asked to evaluate two types of hand prosthetics.
It is probable (but not certain) that their views would be similar to those of all Australians. There is no obvious reason why residents of Queensland and the Northern Territory would be very different from residents in the rest of Australia, regarding their view of hand prosthetics.
Even though the sample is not a random sample of all Australians, the results may generalise to all Australians (though we cannot be sure).
Example 5.11 (Non-representative samples) Suppose we wish to determine the average time per day that Australian households use their air-conditioners for cooling in summer.
If a group of Queensland and Northern Territory residents is asked, this sample would not be expected to represent all Australians: it would over-represent the average number of hours air-conditioners are used for cooling in summer.
In this case, those in the sample are likely to be very different to those not in the sample, regarding their air-conditioners usage for cooling in winter.
In contrast, suppose a group of Tasmanians was asked the same question. This second sample would not be expected to represent all Australians either (it would under-represent).
Again, those in the sample are likely to be very different to those not in the sample, regarding their air-conditioners usage for cooling in winter.
Sometimes, a combination of different sampling methods is used.
Example 5.12 (A combination of sampling methods) In a study of pathogens present on magazines in doctors' surgeries in Melbourne, some suburbs can be selected at random, and then (within each suburb) surgeries are used which volunteer to be part of the study.
In a study of diets of children at child-care centres, researchers used samples in 2010 and 2016, described as follows:
In 2010, a stratified random sampling procedure was used to select representative cross-sections of providers working in licensed center-based programs and licensed providers of family home-based care from publically available lists. [...] Additional participants were also recruited in 2016 using a combination of stratified random and open, convenience-based sampling.
--- Nicole Larson, Katie A. Loth, and Marilyn S. Nanney133, p. 336
Sometimes, practicalities override how the sample can be obtained, which may not result in a random sample. Even so, the impact of this on the conclusions should be noted (that is, in discussing the limitations of the study). Sometimes, ways exist to obtain a sample that is more likely to be representative.
Random samples are often difficult to obtain, and sometimes representative samples are the best that can be done.
In a good representative sample, those in the sample are not obviously different than those not in the sample. Try to ensure a broad cross-section of the target population appears in the sample.
Example 5.13 (Attempts to increase representativeness) To find a sample of university students, students at Cafe A could be approached every Monday morning at 8am, for four consecutive weeks.
This is a convenience sample, and not a random sample. However, the sample would be more likely to be representative if a broader cross-section of students was approached:
- Students at Cafe A on Monday at 8am;
- Students at the Cafe B on Tuesday at 11:30am; and
- Students entering the Library on Thursdays at 2pm.
This is still not a random sample, but the sample now comprises more than just students who attend university on Mondays at 8am, at Cafe A.
Ideally, students would not be included more than once in our sample, though this is difficult to ensure.
The researchers takes a random sample from each of the large groups (cases).
This is a stratified sample.
Sometimes, information may be recorded from those in the sample, and this information used to make some comment about whether our sample seems reasonably representative.
For example, the sex and age of a sample of university students may be recorded; if the proportion of females in the sample, and the average age of students in the sample, are similar to those of the whole university population, then the sample may be somewhat representative of the population (though we cannot be sure).
Example 5.14 (Comparing samples and populations) A study of the adoption of electric vehicles (EVs) by Americans134 used a sample of \(n = 121\) found through social media (such as Facebook) and professional engineering channels. This is not a random sample.
The authors compared some characteristics of the sample with the American population from the 2010 census (Table 5.3), stating:
The sample has a higher representation of males and individuals in the 18--44 age group [...] compared to the US population. In addition, the sample has a higher representation of [...] wealthier individuals.
--- Egbue, Long, and Samaranayake135, p. 1931
In interpreting the results of this study, the authors say:
...the results of this study are more applicable to people with an engineering or technical background...
--- Egbue, Long, and Samaranayake136, p. 1931
|65 and older||13.39||13.00|
|$75,000 and over||51.78||22.51|
|Prefer not to say||19.64||0.00|
The sample may not be representative of the population for many reasons, all of which compromise how well the sample represents the population (i.e., compromises external validity). This is called sampling bias. Biased samples are less likely to produce externally valid studies.
Definition 5.6 (Selection bias) Selection bias is the tendency of a sample to over- or under-estimate a population quantity.
In selection bias, the wrong sampling frame may be used, or non-random sampling is used. The sample is biased because those in the sample may be different than those not in the sample (but this may not be obvious).
Example 5.15 (Selection bias) Consider Example 5.11, about estimating the average time per day that air conditioners are used for cooling in summer.
Using people only from Queensland and the Northern Territory in the sample is using the wrong sampling frame: the sampling frame does not represent the target population ('Australians').
This is selection bias.
Non-response bias occurs when chosen participants do not respond for some reason. The problem is that the responses from those who do not respond may be different than the responses of those who do respond. Non-response bias can occur because of a poorly-designed survey, using voluntary-response sampling, chosen participants refusing to participate, participants forgetting to return completed surveys, etc.
Example 5.16 (Non-response bias) Consider a study to determine the average number of hours of overtime worked by various professions. People who work a large amount of overtime are likely to be too busy to answer the survey.
Those who answer the survey may be likely to work less overtime than those who do not answer the survey.
This is an example of non-response bias.
Response bias occurs when participants provide incorrect information: the answers provided by the participants may not reflect the truth. This may be intentional (for example, if the survey questions are very personal or controversial in nature) or non-intentional (for example, if the question is poorly written or is misunderstood).
All but 2% of the home buyers have at least one computer at home, and 62% have two or more. Of those with a computer, 99% are connected to the internet.
The article later reveals the survey was conducted on-line (and recall the survey was done in 2001...).
What type of bias is apparent?
Non-response bias: Those who respond are different than those who do (less likely to own a computer).
Consider using these samples:
- Obtaining data using a telephone survey.
- Obtaining data using a TV stations call-in at about 6:15pm.
- Asking students at your university because it is easier than finding a random sample.
For each of the above samples, give an example of an outcome for which the sample would likely give over-estimate of the population value.
There are many correct answers; here are some:
- The percentage of people that own a telephone.
- The percentage of people that are shift-workers
- The percentage of Australians that are studying at university.
As a demonstration of sampling schemes,139 consider taking a non-random sample of 10% of the pixels of an image (Fig. 5.3). What is the image? Seeing the big picture is hard using these non-random samples.
Indeed, any type of random sample makes seeing the big picture easier.
For example, for a cluster sample we treat each column as a cluster, and select some columns at random. Then, the entire chosen columns are selected.
For a systematic sample, we take:
- every 20th pixel for a 5% sample;
- every 10th pixel for a 10% sample;
- every 4th pixel for a 25% sample; and
- every second pixel for a 50% sample.
For a multi-stage sample we select some columns at random, then select some pixels in those columns at random.
For a stratified sample, we select :
- a simple random sample from the background greenery, and then
- a simple random sample from the person.
These two are then combined to get an overall random sample.
Almost always, the entire population of interest cannot be studied, so a sample (a subset of the population) must be studied. Samples can be
- random samples or
- non-random samples.
Conclusions made from random samples can usually be generalized to the population (that is, they are externally valid).
Random sampling methods include:
- simple random samples;
- systematic samples;
- stratified samples;
- cluster samples; and
- multi-stage samples.
Random samples are likely to be externally valid.
Non-random sampling methods include:
- convenience samples;
- judgement samples; and
- self-selecting samples.
Random samples are often very difficult to obtain, so the best we can do is to aim for reasonably representative samples, where those who are in the sample are unlikely to be different than those who are not in the sample. Non-random samples may not be externally valid.
The following video may be helpful.
Suppose we randomly select a student and send them a postal survey, but the student has moved address and so never receives the survey. What type of bias will this result in?
What is the main advantage of using a random sample?
What is the main advantage of using a large sample?
A large sample is always better than a random sample: True or false?
Suppose I classify a natural forest region into two zones, which are quite different: Region A is mostly dunes and lightly vegetated, and is on the coastal side of a ridge; Region B is more densely vegetated and on the inland side of the ridge.
I then take a random sample of sugar ants (Camponotus app) from Region A, and another random sample of sugar ants from Region B, to study the average size of the ants.
What is the best description of the type of sampling method being used?
Select all the sampling methods that are random sampling methods.
- Judgement sampling
- Stratified sampling
- Simple random sampling
- Voluntary sampling
- Cluster sampling
- Multi-stage sampling
- Self-selected sampling
A study comparing two new prototype vaccines for malaria recruited a sample of 200 Americans who had contracted malaria.
These recruits were obtained by approaching all patients with a confirmed malaria diagnosis who were admitted to hospitals.
Patients were free to volunteer for the study or not.
The study was then conducted to a high standard. Which of the following statements are true?
- This is a voluntary response sample.
- The study is likely to have high external validity.
- The sample size is too small for the study results to be useful for the general population.
Selected answers are available in Sect. D.5.
Exercise 5.1 Suppose we needed to estimate the average number of pages in a book in a university library (including all five campuses), using a sample of 200 books.
- Describe how you might select a simple random sample of books.
- Describe how you might select a stratified sample of books.
- Describe how you might select a cluster sample of books.
- Describe how you might select a convenience sample of books.
- Describe how you might select a multi-stage sample of books.
- Which would be most practical?
Exercise 5.2 Suppose we need a sample of 20 residents from apartments in a large residential apartment complex, comprising 20 floors with 30 apartments in each floor. We plan to interview the residents of these apartments.
- One approach to obtaining a sample is to randomly select five floors, then randomly select four apartments from each of those five floors, and interview the oldest resident of that apartment. What type of sampling scheme is this?
- Another approach is to select one floor at random, and select the first 20 apartments on that floor then interview the oldest resident of that apartment. What type of sampling scheme is this?
- Another approach is to wait at the ground-floor elevator, and ask people who emerge to participate in our interview. What type of sampling scheme is this?
- Another approach is to select five floors at random, then wait by the elevator and interview residents as they arrive at the elevator. What type of sampling scheme is this?
- Which of the above sampling methods are good, and which are poor? Explain your answers.
Exercise 5.3 Suppose a researcher needs a sample of customers who shop at a large, local shopping centre to complete a survey.
- The researcher stations themselves outside the supermarket at the shopping centre one morning, and approaches every 10th person who walks past. What is the sampling method?
- The researcher waits at the main entrance for 30 minutes at 8am every morning for a week, and approaches every 5th person. What is the sampling method?
- The researcher leaves a pile of survey forms at an unattended booth in the shopping centre, and a locked barrel in which to place completed surveys. What is the sampling method?
- The researcher goes to the shopping centre every day for two weeks, at a different time and location each day, and approaches someone every 15 minutes. What is the sampling method?
- Which would the best sampling method?
- Which (if any) of the methods produce a random sample?
Exercise 5.4 A study140 investigated how children in Brisbane travel to state schools. Suppose researchers randomly sampled four schools from a list of Brisbane state schools, and invited every family at each of those four schools to complete a survey.
What type of sampling method is this?