Chapter 2: Sample Design and Attrition (con't)

Table of Contents

2.7 Sample Representativeness and Attrition

In Table 2.7.1, the percentage of sampled respondents of each race is presented for the base survey year (1967) and the last interview year for which data is available. This table also provides information on numbers of deceased respondents by race. Figure 2.7.1 characterizes the percentage by race of the original sample that has been interviewed at each survey point.

Table 2.7.1 Sample Characteristics by Race: 1967 and 2003

Figure 2.7.1 Interview Completion Rates among Living Respondents by Race and Survey Year

Finally, Table 2.7.2 presents the number of interviews completed by respondents, broken down by race. In this table, the "number who completed" columns show how many respondents completed exactly that number of surveys. The "cumulative %" columns show a cumulative total percent of those completing at least a given number of surveys rather than a percentage of those completing an exact number of surveys.

Table 2.7.2 Number of Interviews Respondents Completed out of 21 Surveys, by Race: 1967-2003

Return to top

2.8 Sample Weights

This section is divided into a description of the procedures used to develop sample weights and a discussion of the practical application of these weights. Before using NLS data in an analysis, the user should consult the practical usage discussion below to determine when weighting of data is appropriate. Sample-based weights are designed to reflect the underlying population in the year in which the cohort was initially surveyed. Individual weights are assigned after each interview; these weights produce group estimates that are demographically representative of each cohort's base-year population when used in tabulations. Sampling weights for each respondent can be found on the corresponding public data release.

Base-Year Sampling Weights

Population data derived from the NLS are based on multi-stage ratio estimates. The first step was to assign each sample case a basic weight consisting of the reciprocal of the final probability of selection. This probability reflects the differential sampling by race within each stratum. The base-year weights for all those interviewed were adjusted to account for the overrepresentation of blacks in the sample as well as for persons selected after screening who were not interviewed in the initial survey. This adjustment was made separately for each of 16 groupings for the Mature Women, based on the four Census regions (Northeast, North Central, South, and West), urban/rural residence, and race (non-black/black).

In the first stage of ratio weight adjustment, differences at the time of the 1960 Census between the distribution by race and residence of the population as estimated from the sample PSUs and that of total population in each of the four major regions of the country were taken into account. Using 1960 Census data, estimated population totals by race and residence for each region were computed by appropriately weighting the Census counts for PSUs in the sample. Ratios were then computed between these estimates (based on sample PSUs) and the actual population totals for the region as shown by the 1960 Census.

In the second stage ratio adjustment, sample proportions were adjusted to independent current estimates of the civilian noninstitutionalized population by age, sex, and race. These estimates were prepared by carrying forward the most recent Census data (1960) to take account of subsequent aging of the population, mortality, and migration between the United States and other countries (Census Bureau 1966). The adjustment was made by race within three age groups.

Sampling Weight Nonresponse Adjustment

Since the initial interview, reductions in sample size have occurred due to noninterviews. To compensate for these losses, the sampling weights of the individuals who were interviewed are revised. The Mature Women cohort is a panel of individuals into which no new individuals were added after the base year. As a result, all reweighting after the initial survey was calibrated to base-year population parameters. This revision was done in two stages. First, out-of-scope noninterviews in each year were identified by the Census Bureau and eliminated from the sample of noninterviews. This group consisted of individuals who were institutionalized, had died, were members of the armed services, or had moved outside the United States--that is, individuals who were no longer members of the U.S. noninstitutionalized civilian population (Note: in 2003, an attempt was made to interview some of the institutionalized respondents.)

The second stage in the adjustment acknowledges the possible nonrepresentative characteristics of the in-scope interviews. For each survey year, those who are eligible but not interviewed, as well as those who are interviewed, were distributed into 24 nonresponse adjustment cells based on race (black and non-black), length of residence in the United States at first interview (nine or fewer years, ten or more years, N/A), and education (N/A, eight or fewer years, nine to eleven years, twelve or more years) reported in 1967. Within each of the cells, the base-year sampling weights of those interviewed were increased by a factor equal to the reciprocal of the reinterview rate (using base-year weights) in that year.

In 1991, CHRR began investigating the effects of differential nonresponse on sampling weights as then calculated. The original weighting routine was designed to minimize an increase in variance caused by large weights for individuals with certain characteristics. One effect of this original procedure was that certain subsegments of the sample were assigned identical sampling weights. CHRR adjusted the weights to avoid this problem.

Practical Usage

The Mature Women sample was based upon stratified, multi-stage random samples with an oversample of blacks. Each case in each interview year was assigned a weight specific to that year. This weight can be interpreted as an estimate of the number of people in the corresponding population that the individual in the sample represents. This section discusses some ramifications of the weights when used for data analysis.

To tabulate characteristics of the sample (i.e., sample means, totals, or proportions) for a single interview year in order to describe the population being represented, it is necessary to weight the observations using the weights provided. For example, to estimate the average hours worked in 1987 by women age 30-44 as of March 31, 1967, researchers would simply use the weighted average of hours worked, where weight is the 1987 sample weight. These weights are approximately correct when used in this way, with item nonresponse possibly generating small errors. Other applications for which users may wish to apply weighting, but for which the application of weights may not produce the intended result, include:

Samples Generated by Dropping Observations with Item Nonresponses: Users often confine their analysis to subsamples of respondents who provided valid answers to certain questions. In this case, a weighted mean will not represent the entire population, but rather those persons in the population who would have given a valid response to the specified questions. Item nonresponse because of refusals, don't knows, or invalid skips is usually quite small, so the degree to which the weights are incorrect is probably quite small. In the event that item nonresponse constitutes a small proportion of the variables under analysis, population estimates (i.e., weighted sample means, medians, and proportions) would be reasonably accurate. However, population estimates based on data items that have relatively high nonresponse rates, such as family income, may not necessarily be representative of the underlying population of the cohort.

Data from Multiple Waves: Because the weights are specific to a single wave of the study, and because respondents occasionally missed an interview but were contacted in a subsequent wave, a problem similar to item nonresponse arises when the data are used longitudinally. In addition, the weights for a respondent in different years may occasionally be quite dissimilar, leaving the user uncertain about which weight is appropriate. In principle, if a user wished to apply weights to multiple wave data, weights would have to be recomputed based upon the persons for whom complete data are available. If the sample is limited to respondents interviewed in a terminal or end point year, the weight for that year can be used. Users with a more complex sample selection often can obtain reasonably accurate results by using the base-year weights.

Regression Analysis: A common question is whether one should use the provided weights to perform weighted least squares when doing regression analysis. Such a course of action may lead to incorrect estimates. If particular groups follow significantly different regression specifications, the preferred method of analysis is to estimate a separate regression for each group or to use dummy (or indicator) variables to specify group membership. If one wishes to compute the population average effect of, for example, education upon earnings, one may simply compute the weighted average of the regression coefficients obtained for each group, using the sum of the weights for the persons in each group as the weights to be applied to the coefficients. While least squares is an estimator that is linear in the dependent variable, it is nonlinear in explanatory variables, so weighting the observations will generate different results than taking the weighted average of the regression coefficients for the groups. The process of stratifying the sample into groups thought to have different regression coefficients and then testing for equality of coefficients across groups using an F-test is described in most statistics texts.

Researchers unsure of the appropriate grouping may wish to consult a statistician or other person knowledgeable about the data set before specifying the regression model. Note that if subgroups have different regression coefficients, a regression on a random sample of the population would be misspecified.

Custom Weighting Program

Currently, every Mature Women survey contains a created variable that is the respondent's cross-sectional weight.  Using these weights provides a simple method for users to correct the raw data for the effects of over-sampling of blacks and the initial clustering of respondents at the survey's beginning.  Unfortunately, while each set of weights provides an accurate adjustment for any single year, none of the weights provide an accurate method of adjusting multiple years' worth of data.  Users analyzing more than one year of Mature Women data should use longitudinal weights, which improve a researcher's ability to accurately calculate summary statistics from multiple years of data.

Users can create longitudinal weights for the Mature Women through two avenues: by going to the Bureau of Labor Statistics Web site at www.bls.gov/nls or through the NLS Investigator at www.nlsinfo.org/investigator. For the BLS Web site, users first select the cohort-specific link they need (Mature Women, in this case) then choose the "custom weights" option to bring up the Custom Weighting program.

To create a set of custom weights, users provide an email address, then select the survey years corresponding to their research and pick the "Create Custom Weights" button.  The custom weighting program will generate a set of longitudinal weights and email users a compressed file in Winzip format.  The resulting file contains two columns of data, with the columns separated by a blank space.  The first column is the public identification (ID) number of each Mature Women respondent.  The second column is the weight.  If the respondent did not participate in every survey checked off, then the respondent is given a weight of zero.  If the respondent did participate, she is given a positive longitudinal weight.

The Web Investigator method of generating custom weights works in a similar fashion. The Investigator has a Custom Weights option button on the main menu page that prompts users to choose a cohort and the desired years, then generates the weights. Results are sent to the user's Investigator account in a compressed file in Winzip format.

The custom weighting program is a version of the program used to create the cross-sectional weights for the original cohorts since the 1990s.  The primary difference between the cross-sectional and longitudinal weighting programs is in how the list of respondents is created.  In the cross-sectional case the weighting program is given a list of all people who participated in a particular survey round.  In the longitudinal case the weighting program creates a "dummy" survey round where the user specifies who participated and who did not.  This "dummy" round is based on the set of surveys selected.  It then calculates which respondents participated in every survey round chosen by the researcher and uses that list to generate weights. 

The original cohorts weighting is derived from the base year weights via a two-step process.  First, all out-of-scope noninterviews (respondents who have died, been institutionalized, or moved outside the U.S.) are eliminated from the pool of respondents who are classified as noninterviews.  Second, those who are in-scope, whether or not they do an interview, are distributed into 24 cells based on race (black/non-black), length of residence at the time of the first interview (nine or less years, ten or more years, or unknown) and education (eight or less years, nine to eleven years, twelve or more years, or unknown).

These cells are then examined to see if the cells have too few respondents.  If a cell has too few respondents, it is collapsed with an adjoining cell.  Once the optimal number of cells is created, all of the weights associated with respondents in a particular cell are totaled.  These totals are then divided to create an adjustment factor.  This adjustment factors is then multiplied by each respondent's base year weight, which results in the custom longitudinal weight for a respondent.

References

Census Bureau. Current Population Reports. Series P-25, No. 352, November 18, 1966.

Parnes, Herbert S.; Shea, John R.; Spitz, Ruth S.; Zeller, Frederick A. Dual Careers, Volume 1: A Longitudinal Study of Labor Market Experience of Women. Manpower Research Monograph 21, vol. 1. Washington, DC: U.S. Government Printing Office, 1970.

Rhoton, Patricia. "Attrition and the National Longitudinal Surveys of Labor Market Experience: Avoidance, Control and Correction." Columbus, OH: CHRR, The Ohio State University, 1984.

Rhoton, Patricia and Nagi, Karima. "Attrition by Wealth in the Original NLS Cohorts." Columbus, OH: CHRR, The Ohio State University, 1991.

Shea, John R.; Roderick, Roger D.; Zeller, Frederick A.; and Kohen, Andrew I. Years for Decision, Volume 1: A Longitudinal Study of the Educational and Labor Market Experience of Young Women. Manpower Research Monograph 24, vol. 1. Washington, DC: U.S. Government Printing Office, 1971.

Zagorsky, Jay and Rhoton, Pat. "Attrition and the National Longitudinal Surveys' Mature Women Cohort." Columbus, OH: CHRR, The Ohio State University, 1998.