Chapter 5: Item Nonresponse

This section examines and quantifies the extent of missing data, formally called item nonresponse, in the NLSY79.  To provide readers with a detailed view of this problem, six surveys are analyzed.  Nonresponse rates are examined first in the 1979 survey and then in the surveys that occur at roughly five-year intervals (1984, 1989, 1994, 1998, and 2004).  These years were chosen to capture the major changes in the NLSY79.  Examining the 1979 survey shows the initial levels of nonresponse.  Examining the 1984 survey shows the amount of nonresponse in the survey just before one part of the respondent pool was dropped.  The 1989 data show nonresponse after the first set of NLSY79 respondents was dropped.  The 1994 data show what occurred after users and interviewers were switched from paper-and-pencil interviewing (PAPI) to computer-assisted personal interviewing (CAPI).  While no major survey changes occurred during the 1998 and 2004 surveys, these surveys show nonresponse rates after many respondents had participated around 20 times.

This section focuses on the three types of missing data:  refusals, invalid skips, and don’t knows.  Overall, the section shows that in these six rounds of the NLSY79, 20 million questions were asked.  Out of all the questions asked to respondents, about 1.5 percent do not have valid answers and are missing data.  Of the three missing data categories, about half the missing data are don’t knows and about half are invalid skips.  Given the vast majority of invalid skips occur in paper-and-pencil years, the percentage of problems attributed to this category has been steadily falling as more computer survey rounds are fielded.

• 5.1 Introduction
• 5.2 Item Nonresponse by Section
• 5.3 Item Nonresponse by Respondents
• 5.4 Item Nonresponse within Problem Sections

Return to Table of Contents

5.1 Introduction

Missing data, or nonresponse, happens in a number of ways in the NLSY79. First, a number of respondents do not participate at all, causing all information in that particular survey to be missing.  Participation rates and reasons for noninterview in each survey round are discussed in  Chapter 2.

A second reason missing data occurs is that respondents do not provide a valid answer to a question.  When this happens, interviewers make a determination about whether to mark the answer as a  refusal or don’t know value.  Users should be cautioned that the assignment of refusals and don’t knows is likely to vary across interviewers.  Moreover, some respondents may believe it is impolite to refuse a question and decline to answer by saying they do not know.  Hence, whether a question is marked either a refusal or a don’t know is somewhat arbitrary.

The last major way missing data occurs is when the interviewer incorrectly follows the survey instrument’s flow.  Incorrect flows result in some respondents being skipped over a set of questions that should be answered while others answer questions that they should not have been asked.  Data archivists have removed from the data most of the extraneous question responses.  While extra information can be removed, missing data is not imputed in the NLSY79. Missing data caused by this reason is flagged with a special “invalid skip” code.   Readers should note that the number of invalid skips drops precipitously beginning in 1993 with the introduction of CAPI.  Nevertheless, invalid skips are still possible in CAPI data.  If the CAPI survey contains a programming mistake, the instrument could incorrectly sequence a respondent.  When these errors are found, the CAPI survey is patched in the field to prevent further invalid skips but the incorrect cases are not asked the questions again.

All missing data are clearly flagged in the NLSY79 data set.  Five negative numbers are used to indicate to users that the variable does not contain useful information.  The five values are (-1) refusal, (-2) don’t know, (-3) invalid skip, (-4) valid skip, and (-5) noninterview.  These five numbers are reserved as missing value flags and, with a few exceptions (see appendix 5 in the NLSY79 Codebook Supplement), are rarely used in the NLSY79 for valid data values.

In the tables that follow, every attempt has been made to look at only variables in a given survey year that were filled in by either a respondent or an interviewer.  The goal was to eliminate all created, machine check, date and time stamp, and variables generated in data post-processing from the analysis.  Given there is no automatic way to check every question to see if it meets these criteria, the number of questions analyzed by the below tables overstates the number of questions actually filled in by the respondent or interviewer.  The overstatement occurs because some questions with meaningful titles are actually hidden machine checks.  While every effort was made to eliminate these questions it is impossible to eliminate all of them.

This section is not the only research on the extent of missing data in the NLS.  Olsen (1992) investigated the effect of switching from PAPI to CAPI interviewing.  His research shows fewer interviewer errors occur from navigating the instrument as well as fewer don’t knows in the CAPI survey.  More importantly, CAPI respondents appeared more willing to reveal sensitive material in the alcohol use section.  Mott (1985, 1984, and 1983) examines the NLSY79’s fertility data.  In these reports, he examines the 1982 and 1983 surveys and finds very low refusal rates for the data in general.  However, by shifting to a confidential abortion reporting method, the willingness to respond greatly increases.  Mott (1998) examines the amount of missing data about the children of NLSY79 females.  He finds that Hispanics or Latinos and, to a smaller extent blacks, have a much higher probability of not finishing the child assessments after starting the interview.

The rest of this chapter contains three parts.  The next part examines which sections of the NLSY79 have high nonresponse rates.  Then, responses are examined to see how many times individuals do not respond to questions.  The last section examines which particular questions in sections with high nonresponse rates are causing problems.

Return to top

5.2 Item Nonresponse by Section

Which parts of the NLSY79 have the highest rates of nonresponse?  The extent of nonresponse is shown for each year in tables 5.2.1 to 5.2.6 and examines every survey section.  The first column of the tables contains the section names within the survey.  The second column shows the total number of questions that all respondents and all interviewers should have answered in that section.  This number is determined by first calculating within each section the number of questions each respondent should answer.  A question is considered answerable if it does not have a valid skip (-4) or noninterview (‑5) as its answer.  A total for the section is obtained by summing up the answers for all NLSY79 respondents.

The third (don’t know), fourth (refusal), and fifth (invalid skip) columns show the total number of nonresponses found in each section.  Columns six, seven, and eight show the same information except in percentage form.  The ninth column shows the total percentage of questions missed and is the sum of the previous three percentages.  The last column, labeled rank, shows which sections have the most (closer to 1) and least (further from 1) amount of nonresponse.

The bottom row of each table combines the information and shows totals.  For example, the bottom of the “Number Questions Asked” column in the 1979 survey shows that almost four million questions (3,975,146) were expected to be filled in by respondents or interviewers.  While the 1979 survey contains many questions, other years are not far behind.  In 1984, there were 3 million questions, 1989 had 1.8 million, 1994 had 3.7 million questions, 1998 had had 4.1 million questions and 2004 had 3.7 million.  Readers are cautioned that each year of NLSY79 data contains far more data points since the tables exclude questions obviously labeled as machine checks, date and time stamps, and questions with valid skip or noninterview data flags.

The six tables show that the overall rate of missing data for many years dropped steadily over time.  In 1979, 2.7 percent of the questions in the survey were not answered.  This number drops to 1.9 percent in 1984 and then falls to 0.9 percent in 1989 and reaches a low point of 0.7 percent in 1994.  After 1994 the number rises again with 0.92 percent in 1998 and 1.42 percent in 2004.  Hence, nonresponse problems are of slightly less concern after the initial round of surveying.

Combining the data from all sections in all the tables shows the majority of nonresponse is caused by don’t knows and invalid skips.  The surveys examined asked a total of 20 million questions.  Of these questions more than 140,000 or 0.7 percent were don’t knows and slightly more than 127,000, or 0.6 percent were invalid skips.  The last category, refusal, contains about 26,000 questions which is roughly 0.1 percent of all questions asked.

Examining the tables over time shows a steady decrease in the amount of data missing due to invalid skips.  In 1979, invalid skips accounted for 2.1 percent of the questions asked.  This number dropped sharply to 1.2 percent by 1984 and then down to 0.25 percent by 1989.  Analysis indicated that CAPI dramatically lowered the problem of invalid skips with only 57 questions out of almost 3.7 million incorrectly skipped in 1994 and 75 questions out of 4 million in 1998.

While invalid skips fall over time, the percentage of refusals has increased slightly.  Refusals accounted for 0.01 percent in 1979, 0.07 percent in 1984, 0.10 percent in 1989, 0.16 percent in 1994, 0.19 percent in 1998, and 0.20 percent in 2004.  Nevertheless, while refusals steadily increase over time in absolute terms the numbers are still quite small.

While invalid skips fall and refusals are rising over time, the trend in don’t knows is more complex.  Don’t knows accounted for 0.6 percent in 1979, 0.6 percent in 1984, 0.5 percent in 1989, 0.5 percent in 1994, 0.7 percent in 1998, and 1.1 percent in 2004.  These figures suggest that don’t knows are making a U-shaped pattern over time.

The last column, labeled rank, shows that missing data are not confined to a single section or area of the survey.  Table 5.2.1 shows that in 1979 the work experience section, with 14.5 percent of the questions missing valid data, had the most problems.  Fourteen percent of all questions asked in this section are labeled as invalid skips and only 0.5 percent of the questions were either refusals or don’t knows.  Military experience, the second most problematic section had almost half the rate of missing data (7.8 percent) as work experience.  The table shows the problem of invalid skips is not related to subject matter since the section (rank 21 out of 21) with the least problems, titled “On Jobs,” also focuses on labor market issues, like work experience.

While the “On Jobs” section of the survey consistently has the least problems in these surveys, the section with the most problems changes.  Table 5.2.2, which examines the 1984 survey, shows the most problems in the “Fertility” section.  Of the almost half-million questions asked in the fertility section, 5.6 percent contain missing data.  While the majority of problems (3.4 percent) were due to invalid skips, a surprisingly large 2 percent of the missing responses are don’t knows.  The second most problematic section in the 1984 survey was “Drug Use”, where 2.7 percent of the questions have missing data.  Like “Fertility,” the major portion of the problem is invalid skips (1.8 percent), but don’t knows (0.8 percent) also account for a significant share.  Interestingly, refusals account for only 0.1 percent, a relatively small proportion for a sensitive topic, suggesting that some of the don’t knows were hidden refusals.

Table Extent of Refusals, Don’t Knows & Invalid Skips, 1979

Table 5.2.2 Extent of Refusals, Don't Knows & Invalid Skips in 1984

Table 5.2.3 shows the amount of nonresponse in the 1989 survey.  The most problematic section is “Income”, missing data in 1.3 percent of its questions, with the CPS section a close second with 1.2 percent.  Unlike earlier years, the major missing data problem in both the “Income” (1 percent) and CPS (0.8 percent) sections are don’t knows, not invalid skips (0.1 percent income and 0.4 percent CPS).

Table 5.2.3 Extent of Refusals, Don't Knows & Invalid Skips in 1989

Table 5.2.4 Extent of Refusals, Don't Knows & Invalid Skips in 1994 NLSY79 Survey

Return to top

5.3 Item Nonresponse by Respondents

This section provides details on the amount of missing data associated with each respondent.  Each table in this section shows the number of respondents who are missing data in one of the surveys.  The tables are split into two parts.  The left hand part, columns one to four, shows the total number of questions that have missing data for each group of respondents.  The right hand part, columns five to nine, shows the percentage of questions that have missing data.

The right hand side of Table 5.3.1, which examines the percentage of questions missing data, shows a similar picture.  Refusal rates are relatively low.  There are 12,620 respondents who refused less than one percent of their questions, which means only 66 respondents refused one percent or more of the questions they were expected to answer.  Thirty-five percent, or 8,185 respondents, answered don’t know to less than one percent of their questions.  Again, the largest group was respondents who were incorrectly skipped over questions.  Only 4,313 respondents were incorrectly skipped over less than one percent of the questions, but 8,373 of the respondents were illegally skipped over one percent or more of their questions and 227 were skipped over more than 10 percent.

Refusal rates have increased steadily over time even though the more difficult respondents have presumably left the survey. Table 5.3.2, which examines the 1984 survey, shows an increase over the 1979 refusal rates. which examines the 1984 survey, shows an increase over the 1979 refusal rates.  While the number of respondents answering the survey is shrinking, the number refusing to answer questions is increasing.  For example, while in 1979 only 10 respondents refused to answer more than 10 questions, in 1984 there were 41 respondents.  This pattern of increase is evident in Table 5.3.3, which examines 1989, through to Table 5.3.6, which examines 2004. By 2004, there were 185 respondents who refused to answer more than 10 questions.

Increasing refusal rates are also seen in the percentage side of the table.  In 1979, only 66 respondents refused to answer one percent or more of the questions they were asked.  This increased in subsequent surveys to 320 respondents in 1984, 355 respondents in 1989, 480 respondents in 1994, 549 respondents in 1998, and 655 respondents in 2004.

Don’t know rates have also risen over time.  In the 1979 survey, 8,185 respondents had less than one percent of their questions labeled as don’t knows.  This number drops in 1984 to 7,003 respondents and further drops to 6,423 in 1989 and 5,942 in 1994, 4,741 in 1998 and 3,185 in 2004.  While rates have risen, relatively few individuals have high levels of don’t knows.  In 1979, only 68 respondents didn’t know the answer to more than five percent of the questions they were asked.  This number falls to 19 respondents in 1984 and then rises to 66 in 1989 before falling back to 46 respondents in 1994 and then jumps back to 66 in 1998, and ends with 149 in 2004.

While don’t know and refusal rates have risen, incorrect skip problems have clearly shrunk over time.  In 1979, there were only 2,347 respondents who were correctly sequenced through the entire survey.  In 1984, this number rises to 7,802 respondents, followed by a rise to 9,334 respondents in 1989.  In 1994 and 1998 almost every respondent was correctly sequenced.  Only 57 and 46 respondents were incorrectly skipped through part of the survey in each year respectively.  Moreover, most of the respondents were only incorrectly skipped in a single question.  In 2004 there were 349 respondents who were incorrectly skipped through 1 percent of their questions and 22 who were incorrectly skipped through 2 percent or more.

Table 5.3.1 Number of Respondents with Missing Data in 1979 Survey

Table 5.3.2 Number of Respondents with Missing Data in 1984 Survey

Table 5.3.3 Number of Respondents with Missing Data in 1989 Survey

Table 5.3.4 Number of Respondents with Missing Data in 1994 Survey

Return to top

5.4 Item Nonresponse within Problem Sections

How much missing data are associated with particular questions?  This part of the chapter provides readers with an in-depth view of the questions within survey sections having a high amount of missing data.  Like the previous parts, this section provides tables for each of the selected survey years.  The first table (Table 5.4.1) examines questions from the 1979 survey’s “Work Experience” section.  This section has more missing data (14.5 percent) than any other 1979 survey section.  The second set of tables 5.4.2 through 5.4.6) examines the most problematic section of the 1984 survey, “Fertility and Abortion.”  The third set of tables (Tables 5.4.7 and 5.4.8) examines the most problematic 1989 survey section, “Income and Assets.”  Since the 1994 “Income and Asset” section again ranked first in missing data, the next set of tables (Tables 5.4.9 and 5.4.10) substitutes the “Drug and Alcohol Use Supplements” given the high degree of research interest in understanding nonresponse in these sections. Table 5.4.11 highlights nonresponse in 1998 in the Marital History section.  Table (5.4.12) tracks nonresponse problems in the over-40 health section.

To ensure the sets of tables are not overwhelming, all sections, like fertility, that could be naturally divided are split.  Additionally, only the most important question or questions with high rates of nonresponse are shown.  Table 5.4.1, which examines the amount of missing data in the 1979 survey, shows the highest amount of missing data are associated with a pair of retrospective questions that asked respondents to remember what happened two years earlier.  Interviewers incorrectly skipped slightly less than 1,750 respondents over R01150., weeks worked in 1977, and R01153., hours worked per week in 1977.  Examining the 1979 questionnaire shows that these questions appear at the bottom of a page.  Prior to these questions is a fairly complicated half page of instructions and questions that the interviewer must read, understand, and partially speak.  It seems likely that many interviewers did not understand the instructions and skipped to the next page.

Table 5.4.1 Amount of Missing Data Per Question in the Work Experience Section, 1979 Survey

Tables 5.4.2–5.4.6, which examine the “Fertility” section, show a much lower number of invalid skips in all parts except in the abortion questions.  While invalid skips do not reach the level seen in Table 5.4.1, on average 190 female respondents were not asked each abortion question (190 is an average from all abortion questions, not just those shown in the tables).  The table also shows a number of other trends.  First, respondents have higher levels of don’t know answers the more precise the question being asked.  For example, in Table 5.4.2, when males were asked the date of birth of their first child, only one did not know the year, three did not know the month and 10 did not know the day.  This phenomena is most clearly seen in Table 5.4.5, which shows the year and month of the respondent’s first sexual encounter.  Only 43 respondents did not know the year, but 1,410 respondents did not know the month.  This problem with dates is also seen in the abortion data where only four respondents did not know the year when they had their first abortion, but 13 did not know the month.

Refusal rates in the “Fertility” section are quite low except for a number of key questions.  Asking the number of times they had sex in the last month elicited high rates of refusal for males and females.  This question elicited 167 male and 135 female refusals.  Interestingly, most individuals were willing to answer if they ever had sex since only 45 males and 54 females refused to answer these questions.  Birth control questions did not have exceptionally high rates of refusal.  Seventeen female respondents and no males refused to answer the birth control questions. Table 5.4.6 shows that 28 females refused to answer if they ever had an abortion and 28 more refused to state if they dropped out of school before they terminated the pregnancy.

Table 5.4.2 Amount of Missing Data Per Question in Male Fertility Section, 1984 Survey

Table 5.4.3 Amount of Missing Data Per Question in Female Fertility Section, 1984 Survey

Table 5.4.4 Amount of Missing Data Per Question in Feeding Part of Fertility Section, 1984 Survey

Table 5.4.5 Amount of Missing Data Per Question in Child Part of Fertility Section, 1984 Survey

Table 5.4.6 Amount of Missing Data Per Question in Abortion Questions of Fertility Section, 1984 Survey

Other questions with high numbers of don’t knows are R29813., which asked about the amount of money received from other sources like interest and dividends, R29825., which asks about a partner’s income, and R29827., which asks the number of exemptions used when filing a Federal tax return.

The asset table (Table 5.4.8) also shows invalid skips are rare but don’t know and refusal rates are not.  Surprisingly, one of the questions with the highest amount of missing data (315 missing answers) asks, “how much is your car worth (R29852.)?”  Another question missing many observations asks the amount of the respondent’s savings (R29835.).  While the car worth question primarily elicits don’t knows, the savings question resulted in 160 refusals.  Three other questions elicited high numbers of don’t knows:  value of stocks and bonds (R29837.) – 219 don’t knows; amount taken out of savings last year (R29842.) – 222 don’t knows; and the market value of other items such as jewelry (R29854.) – 151 don’t knows.

Table 5.4.7 Amount of Missing Data Per Question in Income Section, 1989 Survey

Table 5.4.8 Amount of Missing Data Per Question in Asset Section, 1989 Survey

Table 5.4.9 and 5.4.10 examine the drug and alcohol use supplements in the 1994 survey.  In these CAPI modules, there are no invalid skips.  Interestingly, there are extremely low refusal and don’t know rates within the “Alcohol” section (Table 5.4.9). The question with the highest refusals (nine respondents) asks if the individual had a drink since the 1989 interview.  The typical question in the “Alcohol” section received only two refusals.  Don’t know rates are also low.  The maximum number of don’t knows at nine occurs in R49803., which asks if the respondent needs to drink more alcohol now in order to get drunk.  On average, the “Alcohol” section records only 1.5 don’t knows per question.

Table 5.4.9 Amount of Missing Data Per Question in Alcohol Use Section, 1994 Survey

These low numbers of refusals and don’t knows are not seen in Table 5.4.10, which examines the “Drug Use” section.  On average, the typical question in this supplement elicited 23 don’t knows and 48 refusals.  Readers should understand that this supplement was generally filled in directly by the respondent, not by the interviewer.  To provide respondents with practice using a computer, the questionnaire asked them two practice questions not related to drug use.  Refusal rates are even high for these two test questions, which ask how many more children the respondent expects to have and what type of entertainment, such as movies, concerts, or plays, the respondent went to last year.

The highest number of refusals (119) occurs in R50532., which asks the age the respondent first used marijuana. The second largest number of refusals occurs in a similar question, R50536., which asks the age of first cocaine use. These same questions have very high don’t know responses (113 marijuana and 48 cocaine). One other question with a very high don’t know rate is R50525., which asks if the respondent ever smoked cigarettes daily. Almost 80 individuals did not know the answer to this question. Given that the question wording is straightforward, it is likely a number of respondents are using don’t know as a polite way of refusing to answer the question.

Table 5.4.10 Amount of Missing Data Per Question in Drug Use Section, 1994 Survey

The top ten questions show that a large number of respondents (ranging from 119 to 181 respondents, depending on the question) have difficulty with questions asking them about their spouse’s rate and amount of pay, hours worked and weeks worked.  In addition questions which ask details about a spouse’s previous marriage are also quite difficult for many respondents to answer.

Table 5.4.11 Amount of Missing Data Per Question in Marital History Section, 1998 Survey

Table 5.4.12 examines the top questions with missing data problems from the health section in 2004. In this table, reference numbers starting with “R” are for questions asked of all respondents in the survey, while reference numbers starting with “H” represent questions in the “over 40 health module.”  This module was designed to provide researchers with more information about the health of the respondent when they turned 40 years old and is asked of respondents in the first interview after they turn 40.

While other data from the survey show that many people know if they are covered by health insurance, table 5.4.12 reveals that many do not know details about this coverage.  For example, one question with a large number of don’t knows is R83036., which asks if the respondent’s health insurance plan is an HMO, a preferred provider plan (PPO) or a network of affiliated doctors.  This question had 428 missing responses out of 6,175 total responses (a 7% missing response rate).  Other questions with high don’t know rates ask if the respondent’s children are covered by health insurance.  The health question with the highest refusal rate asks the respondent how much they weigh, with 114 people refusing to divulge the number.  Finally, in the 40+ health module a number of NLSY79 respondents have difficulty answering questions about the health and life status of their biological father.  This is not surprising given a small but significant number of respondents stated in the past that they have never met their biological father.

Table 5.4.12 Amount of Missing Data Per Question in Health Section, 2004 Survey

References