5 External validity: Sampling
So far, you have learnt to ask a RQ, identify different ways of obtaining data, and design the study.
In this chapter, you will learn how to obtain the sample to study. You will learn to:
 distinguish, and explain, precision and accuracy.
 distinguish between random and nonrandom sampling.
 select random samples.
 identify and describe why random samples are preferred over nonrandom samples.
 identify, describe and use simple random sampling, systematic sampling, stratified sampling, cluster sampling, and multistage sampling.
 identify ways of obtaining samples that are more likely to be representative samples.
5.1 The idea of sampling
A RQ implies that every member of the population should be studied (the P in POCI stand for 'population'). However, being able to do so is very rare because of cost, time, ethics, logistics or practicality. Hence, a subset of the population (a sample) is almost always studied.
A sample consists of some individuals (or cases, or (if the individuals are people) subjects) from the population. The purpose of a sample is to approximate the population.
A study is externally valid if the results can be generalised to other groups in the population, apart from the sample studied. This is only possible if the sample is chosen to wellrepresent the whole population.
However, since a sample doesn't include every member of the population, conclusions made from samples cannot be certain to apply to the whole population.
In research, the goal is to learn about the population, but only a sample can be studied.
This book is essentially about how to learn about a population based on an imperfect sample.
Example 5.1 (Samples) A study (based on Richard B. Lipton et al.^{132}) of the effect of aspirin in treating headaches cannot possibly use every single human alive who might one day wish to take aspirin.
Not only would this be prohibitively expensive, timeconsuming, and impractical, but such a study would not even use those humans who had not been born yet who might use aspirin. (That is, using the whole target population is impossible.)
A sample must be used.
Having seen that using a sample is necessary, other issues are raised:
 How can we learn something useful about the whole population if only some of that population is studied?
 Which individuals should be included in the sample?
 How many individuals should be included in the sample be?
The last issue must be left until later, after learning more about the implications of studying a sample rather than a population.
Using a sample instead of the entire population presents challenges. Every sample is likely to be a bit different, so what is learnt from a sample depends on which individuals happen to be present in the sample being used. This is called sampling variation. That is, each sample produced different data, and so may lead to different answers to the RQ.
This is the challenge of research: How to make decisions about populations, using an imperfect sample information. Perhaps surprisingly, lots can be learnt about the population if we approach the task of selecting a sample correctly.
Almost always, samples are studied, not populations.
Every sample is likely to be different, and hence the results from every sample are likely to be different. This is called sampling variation.
While we can never be certain about the conclusions from the sample, special tools can be used to help make decisions about the population from a sample.
The animation below shows how the estimates calculated from a sample vary from sample to sample. We know that 50% of cards in a fair pack are red, but each sample of 10 cards can produce a different percentage of red cards (and does not always produce an estimate of 50%).
Two surveys were conducted before the 1936 presidential election in the USA to predict the winner,^{133} summarised in Table 5.1.
Which do you think predicted correctly the winner of the election? Why?
Study  Number in sample  Population  Method 

A  10 000 000  Specific groups  Voluntary survey 
B  50 000  All Americans  Random survey 
5.2 Precision and accuracy
Two issues concerning sampling, raised in Sect. 5.1, were: which individuals should be in the sample, and how many individuals should be in the sample be. These two issues address two different aspects of sampling: precision and accuracy (Fig. 5.1).
Accuracy refers to how close a sample estimate is to the population value (on average). Accuracy is related to the statistical concept of bias. Precision refers to how close all the possible sample estimates are likely to be (that is, how much variation is likely in the sample estimates).
Definition 5.1 (Accuracy) Accuracy refers to how close a sample estimate is to the population value, on average.
Definition 5.2 (Precision) Precision refers to how close the sample estimates from different samples are likely to be to each other.
Using this language:
 The type of sampling (i.e., the way in which the samples is selected) impacts the accuracy of the sample estimate. In other words, the type of sampling impacts the external validity of the study.
 The size of the sample impacts the precision of the sample estimate.
For example, large samples are more likely to be precise estimates because each possible sample value will produce similar estimates, but they may or may not be accurate estimates. Similarly, random samples are likely to produce accurate estimates (and hence the study is more likely to be externally valid), but they may not be precise unless the sample is also large.
Example 5.2 (Precision and accuracy) To estimate the average age of all Queenslanders, we could ask 9000 Queensland school children (a large sample indeed!).
This will give a precise answer because the sample is large, but inaccurate answer because the sample is not representative of all Queenslanders. In fact, the sample may give a precise and accurate answer to a different question: 'What is the average age of Queensland school children?'
5.3 Types of sampling
One key to obtaining accurate estimates about the population is to ensure that the sample studied is representative of the population of interest (that is, to ensure the study is externally valid).
So, how can a representative sample of the population be found? Whenever a sample is taken, only some of the population is selected. The selected individuals can be chosen using either random sampling or nonrandom sampling.
The word random here has a specific meaning that is different than how it is often used in everyday use.
Definition 5.3 (Random) In research and statistics, random means "determined completely by chance".
5.3.1 Random sampling methods
In a random sample, each individual in the population can be selected on the basis of impersonal chance. (Remember that random means that the sample is determined completly by chance!) Some examples of random sampling methods appear in the following sections (Table 5.2).
The results obtained from a random sample probably generalise to the population from which the sample is drawn; that is, random samples are likely to produce externally valid studies.
Type  Stage 1  Stage 2  Reference  

Systematic  Start at a random location  Take every \(n\)th element thereafter  Sect. 5.5  
Stratified  Split into a few large groups ('strata')  Select simple random sample from every stratum  Sect. 5.6  
Cluster  Split into many small groups ('clusters'); select simple random sample of clusters  Select all in the chosen clusters  Sect. 5.7  
Multistage  Select simple random sample from the larger stage  Select simple random sample from those chosen in Stage 1; etc.  Sect. 5.8 
Consider testing a pot of soup by 'sampling'. If the soup is stirred, we don't need to taste the whole pot of soup to see how the soup tastes.
The same principle applies in research: If we use a random sample (analagous to the stirring the soup), we don't need to study every member of the population. If we don't use a random sample (that is, we don't stir the soup), we do not get an overall impression of the population (or the soup).
5.3.2 Nonrandom sampling methods
A nonrandom sample requires some kind personal input. Examples of nonrandom samples include:
 Judgement sample: Individuals are selected, based on the researchers' judgement, depending on whether the researcher thinks they are likely to be agreeable or helpful. For example, researchers may decided to survey people who are not in a hurry.
 Convenience sample: Individuals are selected because they are convenient for the researcher. For example, researchers may gather data from their family and friends.
 Voluntary response (selfselecting) sample: Individuals participate if they wish to. For example, a voluntary response survey, or a TV station callin survey.
In nonrandom sampling, those who are in the study may be different than those who are not in the study. That is, nonrandom samples are not likely to be externally valid.
Using a nonrandom sample means that the results may not generalise to the intended population: they probably do not produce externally valid studies.
Example 5.3 (Different ways to sample) During the COVID19 (coronavirus) pandemic in 2020, a Facebook poll asked the question:
Do you think a Coronavirus vaccine should be compulsory?
The result was reported as '79 per cent of Australians oppose a compulsory vaccination', from a sample of over 53,000 responses.
However, this sample was a voluntary response sample, not a random sample, so the results may not be accurate. For example, many antivaccination groups instructed their members to flood the poll with 'No' responses (including celebrity chef Pete Evans), and the poll could have been completed by nonAustralians as well as Australians.
A different study^{134} asked Australians:
The Federal Government's 'No Jab, No Pay' policy withholds certain benefits and payments from families who don't fully vaccinate their children. Do you agree with this policy?
In the sample of 1809 respondents, 83.7% either agreed or strongly agreed with this statement.
While this study did not use a random sample, the researchers made efforts to sample a representative crosssection of Australians:
Researchers recruited Australian adults aged 18years and older to participate in the study through a large, wellestablished online panel provider. While not a random sample of the Australian population, researchers made efforts to ensure the sample included individuals representing a wide range of demographics (e.g., age, gender, location, income, political preferences, religiosity).
 Smith, Attwell, and Evers,^{135} p. 194
Furthermore, 'respondents were paid small token sum for their participation in the study' to encourage all selected respondents to provide an answer.
In Sect. 5.11, random and nonrandom samples are compared using an example.
5.4 Simple random sampling
Definition 5.4 In a simple random sample, every possible sample of the same size has same chance of being selected.
A simple random sample is chosen from a list of all members of the population (the sampling frame) using tables of random numbers (Appendix B.1) or websites like https://www.random.org. A smaller version of this webpage, which only generates one number at a time, is shown below; just press Generate. The numbers generated by this widget come from the true random number generator at RANDOM.ORG. (The webpage generates as many numbers as you want all at the same time.)
Definition 5.5 (Sampling frame) The sampling frame is a list of all the members of the population (the individuals, or cases, or subjects).
Often, establishing the sampling frame is difficult or impossible, and so finding a random sample is also difficult.
For example, to study a simple random sample of wombats^{136} would require having a list of all wombats, so some could be selected using random number tables.
This is clearly absurd, and other random sampling methods, such as special ecological sampling methods, would be used instead.^{137}
Other good (random) sampling methods use a system to select randomly, rather than by human choice (discussed in the following sections).
This book always assumes simple random samples, for simplicity, unless otherwise noted.
Example 5.4 (Simple random sampling) Suppose we are interested in this RQ:
For students at a large course at a particular university, is the average number of letters typed on a keyboard in 10 seconds the same for females and males?
Suppose a sample of 40 students is needed. The sampling frame is the list of all students enrolled. Obtaining the sampling frame is feasible here (lecturers have access to this information for grading).
Suppose budget and time constraints mean only 40 students can be selected for the study above.
Describe how to use the course enrolment list to find a simple random sample of 40 students to study.
5.5 Systematic sampling
In systematic sampling, the first case is randomly selected; then, every (say) fifth element is selected thereafter.
In general, we say that every \(n\)th individual is selected.
There is no advantage in using a systematic sample to take every \(n\)th individual if it is just as easy and efficient to take every individual.
Example 5.5 (Systematic sampling) Suppose for a particular study, a sample of 40 students in a particular course is needed.
If 441 students are enrolled, 40 students could be randomly selected, by choosing a number at random between 1 and \(441/40\) (approximately 11) as a starting point; suppose the random number selected is 9. The first student selected is the 9th person in the list (which may be ordered alphabetically, by student ID, or any other means).
Thereafter, every \(441/40\)th person, or 11th person, in the list is selected: people numbered 9, 20, 31, 42...
The animation below shows how a sample of 40 students could be selected from a class of 441 students using a systematic random sample of size 40, by starting at student number 9 and then talking every 11th person.
Care needs to be taken when using systematic samples to ensure a pattern is not hidden.
For example, consider a study where residents in a large, eightlevel residential accommodation complex are to be visited and a survey administered. Each floor of the building has a similar layout (Fig. 5.2), with nine apartments per level.
If the researchers decide to systematically sample every tenth apartment, the very same apartment on each floor woud be chosen.
For example, suppose Apartments 110, 210, 310,... 810 were chosen. These apartments are all larger than all the other apartments. The residents of these apartments may be wealthier than the other residents, so the systematic sample will not be a representative sample of residents.
5.6 Stratified sampling
In stratified sampling, the population is split into a small number of large (usually homogeneous) groups called strata, then cases are selected using a simple random sample from each stratum.
The strata must be unrelated to the variables.
For example, if the RQ is about comparing the percentage of females and males who wear hats at midday, a stratified sample of size 100 is not obtained by selecting 50 females and 50 males, for example. This is merely selecting people from each level of the explanatory variable.
The sex of the person is the explanatory variable; it does not define the strata.
Example 5.6 (Stratified sampling) To select students in a large course at a particular university, 20 of the females and 20 of the males could be selected. The sample is stratified by sex of the person.
At the university where I work, about 67% of the students are females. So, I could ensure that twothirds of the sample was females (around 26.7, say 27) and about onethird males (about 13.3, say 13).
The animation below shows how a stratified random sample of size 40 might be selected, by randomly selecting 20 female and 20 male students.
Similarly, the second animation below shows how a stratified random sample of size 40 might be selected, by randomly selecting 27 female and 13 male students.
5.7 Cluster sampling
In cluster sampling, the population is split into a large number of small groups called clusters, then a simple random sample of clusters is selected and every member of the chosen small groups is part of the sample.
Example 5.7 (Cluster sampling) To select students in a large course at a particular university again, a simple random sample of (say) three of the many tutorials could be selected, and every student enrolled in those selected tutorials constitute the sample.
The animation below shows how a sample of approximately 40 students could be obtained using cluster sampling, using the tutorials as clusters.
5.8 Multistage sampling
In multistage sampling, large groups are selected using a simple random sample, then smaller groups within those large groups are selected using a simple random sample. The simple random sampling can continue for as many levels as necessary.
Example 5.8 (Multistage sampling) To select students in a large course at a particular university again, a simple random sample of (say) ten of the many tutorials could be selected (Stage 1), and then 4 people randomly selected from each of these 10 selected tutorials (Stage 2).
The animation below shows how a sample of approximately 40 students could be obtained using multistage sampling, by randomly selecting 10 classes at random in Stage 1, then randomly selecting students from each class in Stage 2.
Example 5.9 (Multistage sampling) Multistage sampling is often used by the Australian Bureau of Statistics (ABS). For example, to obtain a random sample of Queenslanders, one procedure is:
 Stage 1: Randomly select some cities in Queensland;
 Stage 2: Randomly select some suburbs in these chosen cities;
 Stage 3: Randomly select some streets in these chosen suburbs;
 Stage 4: Randomly select some houses in these chosen streets.
This is cheaper than simple random sampling, as data collectors can be employed in a smaller number of Queensland cities (only those chosen in Stage 1).
5.9 Representative sampling
Obtaining a truly random sample is usually hard or impossible. Often the best we can do is to select a sample that is diverse enough to be somewhat representative of the diversity in the population. Even so, the results from any nonrandom sample may not generalise to the intended population. The results will generalise to the population which the sample does represent.
Ideally, even if obtaining a random sample is impossible, prefer a sample where those in the sample are not likely to be different than those not in the sample, at least for the variables of interest.
Example 5.10 (Representative sample) A randomlychosen group of Queensland and Northern Territory residents is asked to evaluate two types of hand prosthetics.
It is probable (but not certain) that their views would be similar to those all of Australians. There is no obvious reason why residents of Queensland and the Northern Territory would be very different from residents in the rest of Australia, regarding their view of hand prosthetics.
Even though the sample is not a random sample of all Australians, the results may generalise to all Australians (though we cannot be sure).
Example 5.11 (Nonrepresentative samples) Suppose we wish to determine the average time per day that Australia households use their airconditioners for cooling in summer.
If a group of Queensland and Northern Territory residents is asked, this sample would not be expected to represent all Australians: it would overrepresent the average number of hours airconditioners are used for cooling in summer.
In this case, those in the sample are likely to be very different to those not in the sample, regarding their airconditioners usage for cooling in winter.
In contrast, suppose a group of Tasmanians was asked the same question. This second sample would not be expected to represent all Australians either (it would underrepresent).
Again, those in the sample are likely to be very different to those not in the sample, regarding their airconditioners usage for cooling in winter.
Sometimes, a combination of different sampling methods is used.
Example 5.12 (A combination of sampling methods) In a study of pathogens present on magazines in doctors' surgeries in Melbourne, some suburbs can be selected at random, and then (within each suburb) surgeries are used which volunteer to be part of the study.
Sometimes, practicalities override how the sample can be obtained, which may not result in a random sample. Even so, the impact of this on the conclusions shoud be noted (that is, in discussing the limitations of the study). Sometimes, ways exist to obtain a sample that is more likely to be representative.
Random samples are often difficult to obtain, and sometimes representative samples are the best that can be done.
In a good representative sample, those in the sample are not obviously different than those not in the sample. Try to ensure a broad crosssection of the target population appears in the sample.
Example 5.13 (Attempts to increase representativeness) To find a sample of university students, students at Cafe A could be approached every Monday morning at 8am, for four consecutive weeks.
This is a convenience sample, and not a random sample. However, the sample would be more likely to be representative if a broader crosssection of students was approached:
 Students at Cafe A on Monday at 8am;
 Students at the Cafe B on Tuesday at 11:30am; and
 Students entering the Library on Thursdays at 2pm.
This is still not a random sample, but the sample now comprises more than just students who attend university on Mondays at 8am, at Cafe A.
Ideally, students would not be included more than once in our sample, though this is difficult to ensure.
To assess the quality of bearings from a manufacturer, a researcher takes a random sample of 25 bearings from each of the three cases delivered.
What type of sampling scheme is being used?
(Answer is here^{138}.)
Sometimes, information may be recorded from those in the sample, and this information used this to make some comment about whether our sample seems reasonably representative.
For example, the sex and age of a sample of university students may be recorded; if the proportion of females in the sample, and the average age of students in the sample, are similar to those of the whole university population, then the sample may be somewhat representative of the population (though we cannot be sure).
Example 5.14 (Comparing samples and populations) A study of the adoption of electric vehicles (EVs) by Americans^{139} used a sample of \(n=121\) found through social media (such as Facebook) and professional engineering channels. This is not a random sample.
The authors compared some characteristics of the sample with the American population from the 2010 census (Table 5.3), stating:
The sample has a higher representation of males and individuals in the 1844 age group [...] compared to the US population. In addition, the sample has a higher representation of [...] wealthier individuals.
 Egbue, Long, and Samaranayake,^{140} p. 1931
In interpreting the results of this study, the authors say:
...the results of this study are more applicable to people with an engineering or technical background...
 Egbue, Long, and Samaranayake,^{141} p. 1931
Sample  Population  

Gender  
Male  77.68  49.20 
Female  22.32  50.80 
Age  
Under 18  0.00  24.00 
1844  55.36  36.50 
4564  31.25  26.40 
65 and older  13.39  13.00 
Annual income  
Under $75,000  28.56  67.49 
$75,000 and over  51.78  22.51 
Prefer not to say  19.64  0.00 
5.10 Bias in sampling
The sample may not be representative of the population for many reasons, all of which compromise how well the sample represents the population (i.e., compromises external validity). This is called sampling bias. Biased samples are less likely to produce externally valid studies.
Definition 5.6 (Selection bias) Selection bias is the tendency of a sample to over or underestimate a population quantity.
In selection bias, the wrong sampling frame may be used, or nonrandom sampling is used. The sample is biased because those in the sample may be different than those not in the sample.
Example 5.15 (Selection bias) Consider Example 5.11, about estimating the average time per day that air conditioners are used for cooling in summer.
Using people only from Queensland and the Northern Territory in the sample is using the wrong sampling frame: the sampling frame does not represent the target population ('Australians'). This is selection bias.
Nonresponse bias occurs when chosen participants do not respond for some reason. The problem is that the responses from those who do not respond may be different than the responses who do respond. Nonresponse bias can occur because of a poorlydesigned survey, using voluntaryresponse sampling, chosen participants refusing to participate, participants forgetting to return completed surveys, etc.
Example 5.16 (Nonresponse bias) Consider a study to determine the average number of hours of overtime worked by various professions. People who work a large amount of overtime are likely to be too busy to answer the survey.
Those who answer the survey may be likely to work less overtime than those who do not answer the survey. This is an example of nonresponse bias.
Response bias occurs when participants provide incorrect information: the answers provided by the participants may not reflect the truth. This may be intentional (for example, if the survey questions are very personal or controversial in nature) or nonintentional (for example, if the question is poorly written or is misunderstood).
One (true) survey concluded (Jennifer Hieger,^{142} cited in D. E. Bock, P. F. Velleman, and R. D. De Veaux,^{143} p. 283):
All but 2% of the home buyers have at least one computer at home, and 62% have two or more. Of those with a computer, 99% are connected to the internet.
The article later reveals the survey was conducted online (and recall the survey was done in 2001...). What type of bias is apparent?
(Answer is here^{144}.)
For these samples, to what populations will results generalise?
 Obtaining data using a telephone survey.
 Obtaining data using a TV stations callin.
 Asking your friends to participate because it is easier than finding a random sample.
For each of the above samples, give an example of an outcome which would be likely to overestimate the true (population) value.
5.11 Final example
As a demonstration sampling schemes,^{145} consider taking a nonrandom sample of 10% of the pixels of an image (Fig. 5.3). What is the image? Seeing the big picture is hard using these nonrandom samples.
In contrast, taking simple random sample makes the big picture much clearer (Fig. 5.4).
Indeed, any type of random sample makes seeing the big picture easier.
For example, for a cluster sample we treat each column as a cluster, and select some columns at random. Then, the entire chosen columns are selected.
For a systematic sample, we take:
 every 20th pixel for a 5% sample;
 every 10th pixel for a 10% sample;
 every 4th pixel for a 25% sample; and
 every second pixel for a 50% sample.
For a multistage sample we select some columns at random, then select some pixels in those columns at random.
For a stratified sample, we select :
 a simple random sample from the background greenery, and then
 a simple random sample from the person.
These two are then combined to get an overall random sample.
5.12 Summary
Almost always, the entire population of interest cannot be studied, so a sample (a subset of the population) must be studied. Samples can be random samples or nonrandom samples. Conclusions made from random samples can usually be generalized to the population (that is, they are externally valid).
Random sampling methods include simple random samples, systematic samples, stratified samples, cluster samples and multistage samples. Random samples are likely to be externally valid. Nonrandom sampling methods include convenience samples, judgement samples and selfselecting samples.
Random samples are often very difficult to obtain, so the best we can do is to aim for reasonably representative samples, where those who are in the sample are unlikely to be different than those who are not in the sample. Nonrandom samples may not be externally valid.
The following video may be helpful.
5.13 Quick review questions

Suppose we randomly select a student and send them a postal survey, but the student has moved address and so never receives the survey. What type of bias will this result in?

What is the main advantage of using a random sample?

What is the main advantage of using a large sample?
A large sample is always better than a random sample: True or false?

Suppose I classify a natural forest region into two zones, which are quite different: Region A is mostly dunes and lightly vegetated, and is on the coastal side of a ridge; Region B is more densely vegetated and on the inland side of the ridge.
I then take a random sample of sugar ants (Camponotus app) from Region A, and another random sample of sugar ants from Region B, to study the average size of the ants.
What is the best description of the type of sampling method being used?
Progress:
5.14 Exercises
Selected answers are available in Sect. D.5.
Exercise 5.1 Suppose we needed to estimate the average number of pages in a book in a university library (including all five campuses), using a sample of 200 books.
 Describe how you might select a simple random sample of books.
 Describe how you might select a stratified sample of books.
 Describe how you might select a cluster sample of books.
 Describe how you might select a convenience sample of books.
 Describe how you might select a multistage sample of books.
 Which would be most practical?
Exercise 5.2 Suppose we need a sample of 20 residents from apartments in a large residential apartment complex, comprising 20 floors with 30 apartments in each floor. We plan to interview the residents of these apartments.
 One approach to obtaining a sample is to randomly select five floors, then randomly select four apartments from each of those five floors, and interview the oldest resident of that apartment. What type of sampling scheme is this?
 Another approach is to select one floor at random, and select the first 20 apartments on that floor then interview the oldest resident of that apartment. What type of sampling scheme is this?
 Another approach is to wait at the groundfloor elevator, and ask people who emerge to participate in our interview. What type of sampling scheme is this?
 Another approach is to select five floors at random, then wait by the elevator and interview residents as they arrive at the elevator. What type of sampling scheme is this?
 Which of the above sampling methods are good, and which are poor? Explain your answers.
Exercise 5.3 Suppose a researcher needs a sample of customers who shop at a large, local shopping centre to complete a survey.
 The researcher stations themselves outside the supermarket at the shopping centre one morning, and approaches every 10th person who walks past. What is the sampling method?
 The researcher waits at the main entrance for 30 minutes at 8am every morning for a week, and approaches every 5th person. What is the sampling method?
 The researcher leaves a pile of survey forms at an unattended booth in the shopping centre, and a locked barrell in which to place completed surveys. What is the sampling method?
 The researcher goes to the shopping centre every day for two weeks, at a different time and location each day, and approaches someone every 15 minutes. What is the sampling method?
 Which would the best sampling method?
 Which (if any) of the methods produce a random sample?
Exercise 5.4 A study^{146} investigated how children in Brisbane travel to state schools. Suppose researchers randomly sampled four schools from a list of Brisbane state schools, and invited every family at each of those four schools to complete a survey.
What type of sampling method is this?