Nonresponse error occurs when sampling units selected for a sample are not interviewed. Sampled units typically do not respond because they are unable, unavailable, or unwilling to do so. In a web-based survey, for example, a survey link may be incompatible with a respondent’s browser, leaving the respondent unable to complete the survey (the chance that a respondent will try another browser, enable cookies, or download an update necessary to view the survey is realistically pretty low). A respondent may be on vacation for the duration of the data collection period, meaning he or she is unavailable. Respondents may also be unwilling to take the survey because they simply don’t want to take a survey, don’t trust the researcher, are uninterested in the subject matter, or fear embarrassment or violations of privacy, among myriad other reasons.
Why is nonresponse problematic?
Nonresponse is a problem for survey quality because it almost always introduces systematic bias into the data. This results in poorer data quality and can significantly bias any estimates derived from the data. We typically refer to nonresponse as being either missing completely at random (MCAR) or being systematic. If nonresponse is MCAR, then there is no underlying reason why certain sampling units failed to complete the survey. In this case, nonresponse would introduce no bias into the results because there is no systematic skew in the data. In practice, nonresponse is virtually guaranteed never to be MCAR. Assuming that it is would be a very poor assumption. For example, certain segments of the population are less likely to respond to a random survey of the American public: the poor, racial minorities, youth, the extremely old, and the less educated. These are all systematic biases.
Systematic bias, as discussed above, occurs when there is some underlying reason why sampling units do not participate in the survey. This biases any results based upon the data to the extent to which respondents differ from non-respondents on variables of importance to the analysis. For example, an election survey that undersamples African-Americans will underestimate the vote for a Democratic candidate because black voters so heavily support the Democratic Party. This bias is exacerbated by the extent of the nonresponse. Using the election example, if the response rate among black voters is only marginally lower than among white voters, the nonresponse bias, though still present, will likely be small. The greater that response differential based on race, the larger the bias in the vote estimates, though.
There are several techniques researchers can use to minimize nonresponse and to offset the bias it introduces into data. During the data collection period, researchers can use:
- Call backs/Reminders – Researchers should contact sampling units multiple times during the data collection period with reminders to complete the survey. Many people receive a survey invitation, decide to fill the survey out later, but then need an additional impetus to come back to the survey. Reminders are very effective at boosting response rates, especially when they try different appeals to bring in respondents. The precise window between reminders should vary depending upon the length of the collection period. A survey open for a week or two should use relatively frequent reminders, whereas a survey open for months can use them less often.
- Refusal conversions – If an individual has explicitly refused to complete the survey but has not asked the researcher to cease additional contact, a common tactic is to employ staff skilled at response conversion techniques to convince that respondent to participate. This can be a tough job, but it can be effective at turning a no into a yes.
- Incentives – Respondents usually feel no obligation to complete a survey. They usually do not know a researcher and do not care much about the survey itself, if at all. Frankly, respondents are doing the researcher a favor by participating. Offering an incentive to participants can be an extra boost that convinces many to participate. Common incentives might include a small guaranteed cash prize or being entered into a lottery with other participants for the chance to win a large prize.
- Oversampling – If there are certain subgroups that a researcher suspects will show lower response rates, a common technique is to oversample that group. For example, if a researcher suspects that respondents without a high school degree will be much more likely than normal to refuse to participate, the sampling plan might include oversampling those individuals in comparison to their proportion of the population. By oversampling some group with lower than normal response rates, the chance that an adequate number of group members will be in the final sample is higher.
After primary data collection is complete, researchers might consider:
- Weighting – Weighting in essence counts respondents for more or less depending upon their characteristics. For example, if the final dataset includes a disproportionately small number of African-Americans, sampling weights might increase the effect of blacks in the sample on any estimates derived from the data and diminish the effect of non-blacks who are overrepresented. Weighting can be dangerous, though, because it assumes that sampling units that responded are similar to those that did not respond. Using the race example, weighting would assume that those African-Americans who took the survey are similar to those who did not. This would be an erroneous assumption if blacks who did not respond tend to be poorer and less well educated than those who did.
- Proxy respondents – Researchers might consider interviewing a person who was not selected in the sample as a proxy respondent to provide responses for the individual who did not complete the survey. For example, in a survey of teenagers, researchers might interview parents of nonrespondents who can provide some data on behalf of their children. If an establishment survey is targeting its surveys at the chief communications officers of organizations, researchers might contact another qualified person at those institutions where the communications officers did not respond. The quality of proxy responses should be considered suspect, especially if the proxy respondent is providing reports of attitudes or behaviors of some other individual.
Author: Patrick R. Miller, DISM Survey Research Associate