4 Asking Questions

The chapter emphasizes the value of directly asking questions in social research, contrasting it with observing behavior:

Advantage of Human Subjects: Unlike studying dolphins, where researchers can only observe behavior, studying humans allows for direct questioning. Historically, talking to people has been a crucial part of social research, and it remains so.

Surveys and In-depth Interviews: Social research often involves surveys (systematic recruitment of many participants with structured questionnaires) and in-depth interviews (semi-structured conversations with fewer participants). The digital age has a more profound impact on surveys, which is the primary focus of this chapter.

Historical Evolution of Surveys:

First Era (1930s): Researchers sampled geographic areas and conducted face-to-face interviews.
Second Era (around 1970): With the rise of telephones, researchers shifted to random-digit dialing and telephone interviews. This change made surveys faster, cheaper, and more flexible.
Third Era (Digital Age): The digital age is ushering in a new era characterized by non-probability sampling, computer-administered interviews, and the linkage of surveys to big data sources.

Challenges and Opportunities: The transition to the third era is driven by the decline of second-era approaches and the potential of new digital opportunities. For instance, nonresponse rates in telephone surveys can now exceed 90%. However, the digital age offers new ways to conduct surveys and link them to big data.

Key Insight: The initial methods of a new era might be imperfect, but with time and effort, researchers can refine them. For instance, random-digit dialing took time to perfect.

Value of Surveys in the Big Data Age: The chapter argues that big data sources won’t replace surveys. Instead, the availability of big data increases the value of surveys. The chapter will delve into the total survey error framework, new approaches to representation and measurement, and research templates for linking survey data to big data sources.

4.1 Asking versus Observing

The chapter emphasizes the continued importance of asking questions in research, even in the era of big data:

Need for Direct Questions: Despite the vast amount of behavioral data available from big data sources, there will always be a need to ask people questions. This is because:

Big data sources can have issues with accuracy, completeness, and accessibility.
Some critical social outcomes and predictors, such as emotions, knowledge, expectations, and opinions, are internal states. These exist inside people’s heads, and often the best way to understand them is by asking directly.

Illustrative Study - Facebook and Friendships: Moira Burke and Robert Kraut’s research on the impact of Facebook interactions on the strength of friendships serves as a case in point. Despite having complete access to Facebook’s vast data, they had to use surveys to answer their research question. Their primary outcome of interest, the subjective feeling of closeness between friends, is an internal state. Additionally, they used surveys to gather data on communication through channels other than Facebook, like email, phone, and face-to-face interactions. Their study concluded that communication via Facebook did lead to increased feelings of closeness.

Complementary Nature of Asking and Observing: Big data sources won’t replace the need for surveys. In fact, the availability of big data can increase the value of surveys. The relationship between observing behavior (through big data) and asking questions (through surveys) is complementary. They enhance each other’s value, much like peanut butter complements jelly.

In summary, while observing behavior through big data offers rich insights, there’s an irreplaceable value in directly asking questions to understand internal states and nuances not captured by behavioral data.

4.2 Total Survey Error Framework

The chapter delves into understanding the errors that can arise in sample surveys:

Total Survey Error: Estimates from sample surveys can be imperfect. The difference between the estimate from a sample survey (e.g., estimated average height of students) and the true value in the population (e.g., actual average height) is termed as the total survey error.

Bias vs. Variance: The total survey error framework divides errors into two types:

Bias: Systematic error. It’s the difference between the mean of replicate estimates from multiple survey replications and the true value.
Variance: Random error. It’s the variability of these estimates. Ideally, researchers aim for no bias and low variance. However, in many cases, they have to balance between the two. Sometimes, a small-bias, small-variance procedure can be more accurate than an unbiased procedure with high variance.

Representation vs. Measurement: The two main sources of errors in surveys are:

Representation: Problems related to who you talk to. It’s about inferring the attitudes or characteristics of the entire population based on the sample.
Measurement: Problems related to what you learn from those you talk to. It’s about inferring the actual attitudes or characteristics of respondents based on their answers.

For instance, if you’re estimating attitudes about online privacy among adults in France, you need to infer the actual attitudes of respondents from their answers (measurement) and then infer the attitudes of the entire French adult population from the sample (representation).

In essence, the total survey error framework helps researchers understand and categorize the potential errors in surveys, emphasizing the importance of both representation and measurement.

Representation

Representation in surveys is about making accurate inferences from the respondents to the broader target population.

Literary Digest’s Straw Poll (1936): The magazine’s poll aimed to predict the 1936 US presidential election outcome. They sent out ballots to 10 million people, predominantly sourced from telephone directories and automobile registration records. Despite a massive response (2.4 million returned ballots), the poll incorrectly predicted Alf Landon would defeat Franklin Roosevelt. In reality, Roosevelt won by a landslide.

Sampling Framework:

Target Population: The group defined by the researcher as the population of interest. For the Literary Digest, it was the voters in the 1936 presidential election.
Frame Population: The list of people used for sampling. For the Literary Digest, it was the 10 million people from telephone directories and automobile registration records. Differences between the target and frame populations can lead to coverage error and, potentially, coverage bias.
Sample Population: The people the researcher will attempt to interview. Differences between the sample and frame populations introduce sampling error.
Respondents: Those successfully interviewed. If respondents differ from the sample population, there can be nonresponse bias.

Key Lessons:

Large Data Doesn’t Guarantee Accuracy: A large amount of data can provide a precise estimate, but it might be of the wrong thing. This is termed as being “precisely inaccurate.”
Account for Sampling Process: Researchers need to consider how their sample was collected when making estimates. A skewed sampling process requires a more complex estimation process.

In essence, while large datasets can offer insights, it’s crucial to understand the sampling process and potential biases to make accurate inferences.

Measurement

Measurement in surveys is about inferring respondents’ thoughts and actions from their responses.

Measurement Challenges: The answers researchers receive, and the inferences they make, can critically depend on how questions are asked. For instance, the way a question is framed or the specific words used can influence responses.

Question Form Effects: The structure of a question can lead to different answers. For instance, two similar questions about blame for crime in society produced opposite results when phrased slightly differently. One question framed individuals as more to blame, while the other framed social conditions as more responsible.

Question Wording Effects: The specific words used in a question can also influence responses. For example, when asked about governmental priorities, respondents showed more support for “aid to the poor” than “welfare,” even though they might seem synonymous.

Constructing Questions Carefully

Given the potential for different responses based on question form and wording:

Researchers should construct questions with care and not accept responses uncritically.
Analyzing survey data requires reading the actual questionnaire to understand the context.
It’s beneficial to copy questions from high-quality surveys to ensure they’ve been tested.
If potential question wording or form effects are suspected, researchers can run a survey experiment to compare responses.
Pre-testing or pilot-testing questions with a sample from the target population can be invaluable.

In essence, while surveys are a powerful tool for gathering information, the way questions are framed and worded can significantly influence the responses. Thus, careful construction and testing of questions are crucial.

Cost

Surveys come with associated costs, both in terms of time and money.

Cost Considerations: While the total survey error framework is comprehensive, it often causes researchers to overlook a crucial factor: cost. Cost, measurable by time or money, is a real constraint in survey research. Focusing solely on minimizing error without considering cost might not always be beneficial.

Keeter and Colleagues’ Study: Scott Keeter and his team conducted a study to understand the effects of rigorous field operations on reducing non-response in telephone surveys. They ran two simultaneous studies: one with “standard” recruitment procedures and another with “rigorous” recruitment procedures. The rigorous approach involved more frequent calls, longer contact periods, and extra callbacks for initial refusals. This rigorous method did reduce non-response but was twice as expensive and eight times slower. Surprisingly, both studies yielded nearly identical estimates.

Quality vs. Quantity: The study raises a question: Is it better to have multiple reasonable surveys or one perfect survey? At some point, the cost advantages of multiple reasonable surveys might outweigh the benefits of a single pristine survey with minimal error.

Opportunities in the Digital Age: The digital age offers opportunities not just to make estimates with lower error but also to estimate different quantities and make faster, cheaper estimates, even if they come with higher errors. Researchers who focus solely on minimizing error might miss out on these opportunities.

In essence, while ensuring quality in surveys is essential, it’s crucial to balance this with cost considerations, especially in the digital age where speed and adaptability are vital.

4.3 Who to Ask

The digital age is reshaping the landscape of sampling in surveys.

Probability vs. Non-Probability Sampling: Historically, probability sampling, where all members of the target population have a known, nonzero chance of being sampled, has dominated. However, the digital age is making probability sampling harder in practice and is opening new avenues for non-probability sampling.

Challenges with Probability Sampling: Even though probability sampling offers strong theoretical guarantees, real-world challenges like coverage errors and nonresponse have been increasing. For instance, nonresponse rates in some commercial telephone surveys can be as high as 90%, making data collection more expensive and estimates less reliable.

Rise of Non-Probability Sampling: Non-probability sampling methods, where not everyone has a known and nonzero probability of inclusion, have evolved significantly. One example is the use of online panels, where researchers can access a sample of respondents with desired characteristics. These panels are often constructed using ad hoc methods, like online banner ads.

Case Study - Xbox Users & 2012 US Election: A study by Wei Wang and colleagues used a non-probability sample of American Xbox users to predict the outcome of the 2012 US election. Despite the Xbox sample being skewed (e.g., 93% male), post-stratification adjustments enabled them to produce accurate estimates.

Post-Stratification: This technique uses auxiliary information about the target population to improve estimates from a sample. It involves dividing the population into groups, estimating the desired metric for each group, and then taking a weighted average to produce an overall estimate. The key is to form the right groups, ensuring that each group is homogeneous in terms of the response propensity.

Multilevel Regression with Post-stratification (Mr. P): Wang and colleagues used a combination of multilevel regression and post-stratification, termed “Mr. P.” Multilevel regression pools information from closely related groups to estimate support within a specific group. For instance, if there’s a lack of data about a very specific demographic group, the method uses statistical models to combine estimates from similar groups.

*Results from the Xbox Study**: When Wang and colleagues applied Mr. P. to the Xbox non-probability sample, they produced estimates that closely matched the actual support Obama received in the 2012 election. In fact, their estimates were more accurate than an aggregate of traditional public opinion polls. This shows that statistical adjustments, like Mr. P., can effectively correct biases in non-probability data.

Key Lessons:

Unadjusted Non-Probability Samples Can Be Misleading: Raw data from non-probability samples can lead to inaccurate estimates.
Proper Analysis Can Yield Accurate Results: When analyzed correctly, non-probability samples can produce reliable estimates. This means that non-probability samples, when adjusted properly, don’t necessarily lead to flawed results like the Literary Digest’s 1936 poll.

Choosing Between Sampling Approaches: Researchers face a challenging decision between using traditional probability sampling methods, which are becoming more expensive and less ideal in practice, and non-probability sampling methods, which are cost-effective and faster but less familiar. However, if working with non-probability samples or nonrepresentative big data sources, techniques like post-stratification can offer more accurate results than unadjusted estimates.

In essence, while probability sampling has been the gold standard, the digital age is pushing researchers to reconsider non-probability sampling methods, especially given the challenges with the former and the advancements in the latter.

4.4 New Ways of Asking Questions

The digital age offers innovative ways to ask questions in surveys:

Traditional vs. Digital Age Surveys: Traditional surveys can be seen as closed, boring, and detached from real life. The digital age offers opportunities to make surveys more open, engaging, and integrated into daily life.

Survey Modes: The environment in which questions are asked, known as the survey mode, can significantly impact responses. Historically, face-to-face and telephone modes dominated. The digital age introduces computer-administered modes, which differ from interviewer-administered ones.

Benefits of Computer-Administered Surveys:

Reduced Social Desirability Bias: Respondents might be more honest without the presence of a human interviewer, leading to more accurate data on sensitive topics.
Elimination of Interviewer Effects: Responses won’t be influenced by the characteristics or biases of a human interviewer.
Cost-Effective: Removing human interviewers can reduce costs.
Flexibility: Respondents can participate at their convenience.

Challenges of Computer-Administered Surveys: Human interviewers can build rapport, clarify questions, and maintain engagement, which can be challenging in computer-administered surveys.

Innovative Approaches in the Digital Age:

Ecological Momentary Assessment: Measures internal states at relevant times and places.
Wiki Surveys: Combines the strengths of open-ended and closed-ended questions.
Gamification: Making surveys more enjoyable for participants.

In essence, while the digital age presents challenges to traditional survey methods, it also offers innovative techniques and approaches that can make surveys more engaging, relevant, and insightful.

Ecological Momentary Assessments

Ecological momentary assessment (EMA) is a novel approach to surveying that integrates questions into participants’ daily lives.

What is EMA?: EMA involves breaking traditional surveys into smaller parts and integrating them into participants’ lives. This allows for questions to be posed at relevant times and places, rather than during a lengthy interview that might occur weeks after the events in question.

Characteristics of EMA:

Data collection in real-world environments.
Focus on individuals’ current or very recent states or behaviors.
Assessments can be event-based, time-based, or randomly prompted.
Multiple assessments are completed over time.

Smartphones and EMA: The ubiquity of smartphones, which people interact with frequently throughout the day, greatly facilitates EMA. Smartphones’ built-in sensors, such as GPS and accelerometers, can even trigger measurements based on activity.

Case Study - Re-entry after Prison: Naomi Sugie’s research illustrates the potential of EMA. She studied the re-entry process of individuals leaving prison in Newark, New Jersey. Given a smartphone, participants received two types of surveys: an “experience sampling survey” during the day and a “daily survey” in the evening. The phones also recorded geographic locations and kept encrypted records of call and text meta-data. This combined approach of asking and observing provided a detailed, high-frequency set of measurements about the lives of these individuals as they reintegrated into society.

Findings: Sugie’s research revealed that finding stable employment is challenging for ex-offenders. Their work experiences were often informal, temporary, and sporadic. However, the emotional state of these individuals was surprising. Those who continued to search for work reported more feelings of emotional distress than those who stopped searching.

Ethical Considerations: Sugie’s research, especially passive data collection, raised ethical concerns. However, she addressed these by obtaining meaningful informed consent, allowing participants to turn off geographic tracking temporarily, and ensuring data protection.

Key Takeaways:

EMA is compatible with traditional sampling methods.
High-frequency, longitudinal measurements can be valuable for studying dynamic social experiences.
Combining survey data collection with big data sources introduces additional ethical considerations, but these can be addressed with careful planning and respect for participants.

WikiSurveys

Wiki surveys introduce a blend of open and closed questions, leveraging technology to evolve based on participant input:

Closed vs. Open Questions: Traditional surveys often use closed questions, where respondents choose from a set list of answers. Open questions, on the other hand, allow respondents to provide their own answers. While closed questions are easier to analyze, open questions can yield more diverse and potentially valuable insights.

Wiki Surveys: Inspired by Wikipedia’s user-generated content model, wiki surveys are designed to be both open and closed. They evolve over time based on participant input. These surveys aim to be greedy (capturing as much information as possible), collaborative (allowing participants to build on each other’s ideas), and adaptive (changing in response to participant input).

How Wiki Surveys Work:

A seed list of ideas or questions is created.
Participants are presented with pairs of ideas and choose between them.
Participants can contribute their own ideas, which, after approval, are added to the pool of ideas presented to others.

Case Study - New York City’s PlaNYC 2030: The NYC Mayor’s Office used a wiki survey to gather resident feedback for their citywide sustainability plan. They started with 25 seed ideas and asked residents to choose between pairs of ideas. Over four months, 1,436 respondents contributed 31,893 responses and 464 new ideas. Notably, 8 of the top 10 scoring ideas were uploaded by participants, highlighting the value of open input.

Benefits of Wiki Surveys:

They capture diverse insights that might be missed in traditional closed surveys.
They allow for large-scale participation and engagement.
They provide a platform for future methodological research.

In essence, wiki surveys represent a new approach to survey design in the digital age, combining the structure of closed questions with the flexibility and inclusiveness of open questions.

Gamification

Gamification aims to make surveys more engaging by incorporating game-like elements:

The Need for Gamification: Traditional surveys can be tedious for participants. Without a human interviewer in computer-administered surveys, maintaining participation can be challenging. To address this, surveys must become more enjoyable and game-like.

Friendsense on Facebook: Sharad Goel, Winter Mason, and Duncan Watts developed “Friendsense”, a survey packaged as a Facebook game. The goal was to estimate how similar people think they are to their friends in terms of attitudes and how similar they actually are. This study aimed to understand people’s ability to accurately perceive their social environment, which has implications for political polarization and social change dynamics.

How Friendsense Worked:

After consenting to participate, the app selected a friend from the respondent’s Facebook account and posed a question about that friend’s attitude.
Alongside questions about randomly chosen friends, the respondent also answered questions about themselves.
Post answering a question about a friend, the respondent was informed if her answer was correct. If the friend hadn’t answered, the respondent could encourage her friend to participate, promoting viral recruitment.
The attitude questions were adapted from the General Social Survey and ranged from serious political questions to lighthearted ones about preferences and hypothetical superpowers.

Key Findings:

Friends were more likely to have similar answers than strangers, but even close friends disagreed on about 30% of questions.
Respondents overestimated their agreement with friends, implying that a significant diversity of opinions between friends goes unnoticed.
Participants were equally aware of disagreements with friends on both serious political matters and lighthearted topics.

Gamification’s Impact on Data Quality: While there might be concerns that gamification could compromise data quality, the argument is that bored participants pose a more significant risk. Engaging participants can lead to more genuine and insightful responses.

In summary, gamification offers a fresh approach to survey design, making the process more engaging for participants and potentially yielding richer data.

4.5 Surveys Linked to Big Data

Linking surveys to big data sources can produce insights that wouldn’t be possible with either data source on its own:

Traditional Surveys vs. Linked Surveys: Most surveys are standalone efforts that don’t leverage other existing data. However, there’s significant potential in linking survey data with big data sources. By merging these two types of data, researchers can often achieve results that were previously unattainable.

Two Approaches to Linking:

Enriched Asking: Here, the big data source provides a core measure of interest, and the survey data builds the necessary context around it.
Amplified Asking: In this approach, the big data source doesn’t have a core measure of interest but is used to amplify the survey data.

Complementary Nature: These examples emphasize that surveys and big data sources are complementary rather than substitutes. Depending on the research question, one can view these studies as either “custom-made” survey data enhancing “ready-made” big data or vice versa.

In summary, the integration of surveys with big data sources can offer richer insights and more comprehensive results, showcasing the synergistic potential of combining traditional research methods with modern data sources.

Enriched Asking

Enriched asking is a method where survey data provides context to a big data source that contains key measurements but lacks others:

Concept of Enriched Asking: In this approach, a big data source has some crucial measurements, but other essential data is missing. Researchers then gather this missing data through a survey and link the two data sources.

Example - Facebook Interaction Study: Burke and Kraut’s study on whether Facebook interactions increase friendship strength is an example of enriched asking. They combined survey data with Facebook log data. However, they didn’t face the typical challenges of enriched asking, such as the difficulty of linking individual-level datasets (record linkage) and assessing the quality of the big data source.

Challenges:

Record Linkage: Linking datasets can be tricky without a unique identifier in both sources.
Data Source Quality: The process of creating big data might be proprietary, making it hard to assess its quality.

Case Study - Voting Patterns in the US: Ansolabehere and Hersh used enriched asking to study voting patterns. They combined survey data with a master voting file from Catalist LCC, which had digitized and harmonized governmental voting records. This allowed them to compare reported voting behavior in surveys with actual voting behavior.

Key Findings:

Over-reporting of voting is widespread.
Over-reporting isn’t random; those more likely to vote are also more likely to over-report.
The actual differences between voters and non-voters are smaller than they appear in surveys.

General Lessons:

Combining big data sources with survey data can yield insights not possible with either source alone.
Commercial data sources, like Catalist’s, aren’t “ground truth” but can be more accurate than other available data.
Researchers can benefit from private companies’ investments in collecting and harmonizing complex datasets.

Amplified Asking

Amplified asking is a technique that combines a small amount of survey data with a large big data source to produce estimates at a scale or granularity that wouldn’t be possible with either data source individually:

Amplified Asking Defined: This approach uses a predictive model to merge a limited amount of survey data with a vast big data source. The goal is to generate estimates at a scale or granularity that would be unattainable using only one of the data sources.

Blumenstock’s Research: Joshua Blumenstock aimed to gather data to guide development in impoverished countries. Traditional methods like sample surveys or censuses had their limitations. Blumenstock’s idea was to combine the strengths of both: the flexibility of sample surveys and the comprehensive nature of censuses.

Data from Mobile Phone Provider: Blumenstock collaborated with Rwanda’s largest mobile phone provider, obtaining anonymized transaction records from about 1.5 million customers from 2005 to 2009. These records had details about calls and text messages, including start time, duration, and approximate geographic location.

Goal: The objective was to measure wealth and well-being, which weren’t directly present in the call records. The challenge was to predict survey responses based on these records.

Procedure:

Feature Engineering: Convert call records into a set of characteristics or “features” for each person, such as total days with activity, number of distinct contacts, money spent on airtime, etc.
Supervised Learning: Use these features to build a model predicting survey responses. Blumenstock employed logistic regression for this.

Results: Predictions varied in accuracy. For instance, predicting radio ownership had an accuracy of 97.6%, but this was only slightly better than a simple prediction based on the most common answer (97.3%). However, for questions like “Do you own a bicycle?”, predictions improved significantly from 54.4% to 67.6%.

Further Improvements: A year later, Blumenstock and colleagues achieved better results by using more sophisticated methods and focusing on a composite wealth index. They used call records to predict wealth for the entire dataset, and with geospatial information, they estimated the geographic distribution of subscriber wealth in Rwanda. When compared to traditional survey methods, their results were quite similar but were achieved faster and at a fraction of the cost.

Recipe for Amplified Asking: This method requires two ingredients:

A big data source that is wide but thin (covers many people but lacks detailed information).
A survey that is narrow but thick (covers fewer people but provides detailed information). By combining these two, a machine learning model can be built using the big data source to predict survey answers. This model can then be used to impute survey answers for everyone in the big data source.

Indirect Interest in Big Data: The primary interest isn’t always in the big data source itself. For instance, Blumenstock and colleagues were primarily interested in call records because they could predict survey answers, not because of the call records per se.

Results Visualization: The study’s results were visualized in figures, showing the process of converting call records into a matrix and then using a supervised learning model to predict survey responses. This model was then used to estimate wealth and place of residence for all 1.5 million customers. The results were comparable to the Demographic and Health Survey, a traditional survey standard.

Trade-offs and Challenges:

Amplified asking can produce timely, cost-effective, and granular estimates.
However, the theoretical basis for this approach isn’t robust. It’s unclear when this method will work and when it won’t.
There are concerns about potential biases due to the inclusion or exclusion of certain groups in the big data source.
Quantifying uncertainty around estimates remains a challenge.

Future Potential: Amplified asking is connected to various areas in statistics, such as small-area estimation, imputation, and model-based post-stratification. As researchers delve deeper into these areas, the methodological foundations of amplified asking are expected to improve.

Iterative Improvement: Comparing Blumenstock’s first and second attempts at this approach highlights the importance of iterative research. Initial attempts might not be perfect, but with persistence, results can improve. When evaluating new digital-age research methods, it’s essential to consider both their current effectiveness and their potential future impact.

In essence, amplified asking showcases the potential of combining traditional survey methods with big data to produce more comprehensive and cost-effective results.

4.6 Conclusion

The shift from the analog to the digital age presents fresh opportunities for survey researchers:

Value of Surveys in the Digital Age: Despite the rise of big data sources, they won’t replace surveys. In fact, the presence of big data sources enhances the importance of surveys.

Total Survey Error Framework: This framework, developed during the first two eras of survey research, can guide researchers in the digital age. It aids in the development and assessment of modern survey methods.

Key Areas of Opportunity:

Non-Probability Sampling: This approach is becoming increasingly relevant in the digital age (covered in section 3.4).
Computer-Administered Interviews: The digital age offers innovative ways to conduct surveys, making them more engaging and relevant (discussed in section 3.5).
Linking Surveys and Big Data: Combining traditional survey data with big data sources can yield richer insights (elaborated in section 3.6).

Embracing Evolution: Survey research has always evolved due to technological and societal changes. It’s crucial to welcome this evolution while also valuing the insights from earlier research eras.

MATHY NOTES

This section provides a more mathematical perspective on the ideas discussed in the chapter, aiming to bridge the gap between the main content and more technical material on the topics. Here’s a summary:

Probability Sampling

The goal is to estimate the unemployment rate in the U.S.
The target population is represented as \(U = \{1, \ldots, k, \ldots, N\}\).
Simple random sampling without replacement is introduced, where each person has an equal chance of being included in the sample.
The sample mean is used to estimate the population unemployment rate.
In cases of unequal probabilities of inclusion, the Horvitz-Thompson estimator is used, which is a weighted sample mean.
Stratified sampling is introduced, where the population is divided into mutually exclusive groups called strata.
Post-stratification is a technique that uses auxiliary information to improve estimates.

Probability Sampling with Nonresponse

Real surveys often have nonresponse, where not everyone in the sample answers every question.
Two main types of nonresponse: item nonresponse (some respondents don’t answer some items) and unit nonresponse (some people selected don’t respond at all).
Surveys with unit non-response are seen as a two-stage sampling process: selection and response.
The bias introduced by nonresponse can be calculated, and it’s shown that nonresponse won’t introduce bias if there’s no variation in the outcome or response propensities, or if there’s no correlation between response propensity and the outcome.
Post-stratification can reduce the bias caused by nonresponse.

Non-probability Sampling

This includes a wide variety of designs.
The sampling design focuses not on the researcher-driven probability of inclusion but on the respondent-driven probability of inclusion.

Opt-in Samples: These samples can be problematic due to their respondent-driven nature. However, if researchers possess good auxiliary information and a robust statistical model, they can account for the issues inherent in opt-in samples, even when the sampling frame has significant coverage error.

Bethlehem’s Extension: Bethlehem (2010) extended many of the derivations about post-stratification to include both nonresponse and coverage errors.

Techniques for Non-probability Samples:

Sample Matching: This method matches respondents in a non-probability sample to respondents in a probability sample based on auxiliary information. An example of this approach is provided by Ansolabehere and Rivers (2013).
Propensity Score Weighting: This technique involves weighting respondents in a non-probability sample based on the estimated probability that they would be included in the sample. Lee (2006) and Schonlau et al. (2009) provide insights into this method.
Calibration: Calibration adjusts the weights of respondents in a non-probability sample so that the sample matches known population totals on auxiliary variables. Lee and Valliant (2009) discuss this technique in detail.

Common Theme: A recurring theme among these techniques is the critical role of auxiliary information. This information, which is known for both the sample and the population, can be used to adjust the non-probability sample to make it more representative of the population.

In summary, while non-probability samples present challenges, various techniques, often relying on auxiliary information, can be employed to address these challenges and produce more reliable estimates.