Final Report on the Effects of Sample Attrition on Estimates of Channeling's Impacts. A. A Model of Attrition

01/13/1986

Models of the response to survey interviews are recent in origin and few attempts to estimate models of the response mechanism are present in the literature (Madow et al., 1983). Fortunately, knowledge of some aspects of the channeling demonstration evaluation assist in the specification of a model of response.

As discussed in Chapter II, a major finding about attrition was the fact that a substantially higher proportion of controls than treatment group members refused the baseline (often giving their assignment to the control group as the reason for their refusal). That difference in baseline nonresponse led to treatment/control differences in rates of followup interview nonresponse, simply because a followup interview was not attempted if the baseline was not completed. This difference also carried over to the other (Medicare, nursing home, in-community) analysis samples. Thus, experimental status is an important determinant of whether observations are included in the analysis samples. Two binary variables for treatment status were included in the model, one for treatments in the basic model and one for treatments in the financial control model, to account for the known difference between models in data availability for treatment group members. Site binary variables were also included in the model to capture differences in response rates by site that could arise due to site differences in interviewer quality or supervision, or in the types of persons referred to channeling.

Another major cause of followup interview nonresponse is death. This suggests that variables related to health status and other factors that may affect mortality should be included in the general response model. Such factors include the following:

  • Impairments on activities of daily living
  • Age
  • Whether referred to channeling by a hospital or nursing home
  • Unmet needs
  • Whether received help with various household tasks or personal care
  • Whether on a waiting list for a nursing home

Reasons given by those refusing the baseline suggest another reason why such health status variables are potentially important predictors of response: those who are severely impaired may simply be unable to complete the interviews. Even if proxy respondents are present, they often are too busy caring for the impaired sample member to be interviewed.

Besides these indicators of impairment, more indirect measures of the willingness and ability to complete an interview include:

  • Cognitive impairment
  • Whether a proxy assisted the sample member with the screen questions
  • Whether living alone (if someone lives alone, a proxy is less likely to he available to answer questions if the sample member is unable to do so)
  • The number of contacts required to complete the screen
  • Whether the screen interviewer felt the sample member would require help with the baseline
  • The number of missing items on the screen.

The last three variables use the experience with the screen interview as predictors of the sample member's willingness or ability to cooperate with followups.

Finally, there are socioeconomic variables that may have little direct bearing on attrition, but may affect outcome measures which in turn affect the probability of response. Since outcome measures cannot be used to predict response (because they are not observed for nonrespondents), we include these screen determinants of outcomes in the response equation. These factors include sex, ethnicity, Medicaid coverage, and income.

Probit models for the likelihood of being in the 6-, 12-, and 18-month followup samples as a function of the characteristics discussed above were estimated, and the results are presented in Table V.1. Unfortunately, probit coefficients do not have the same interpretation as regression coefficients, which indicate, for a given predictor variable, the effect of a unit change in the predictor variable on the dependent variable. A rough approximation of the effect of a given predictor on the probability of being in the sample is obtained by multiplying the probit coefficient by 0.4.24 Thus, sample members in the treatment group of the basic model are (.134 * 0.4) * 100 = 5 percentage points more likely than otherwise identical control group members in the same site to be in the 6- month followup sample.

TABLE V.1: Probit Coefficients for a Model of Being in the 6, 12, and 18 Month Followup Samples
Screen Variable 6-Month 12-Month 18-Month
  Coefficient     t-value     Coefficient     t-value     Coefficient     t-value  
TREATMENT STATUS
Basic model 0.134** (2.81) 0.200** (4.28) 0.126 (1.94)
Financial control 0.419** (8.67) 0.331** (7.03) 0.319** (4.69)
SITE
Basic Model
   Baltimore -0.118 (-1.23) 0.012 (0.13) -0.154 (-1.13)
   E. Kentucky 0.097 (0.96) 0.210* (2.17) 0.136 (0.94)
   Houston -0.089 (-0.89) 0.038 (0.38) -0.194 (-1.33)
   Middlesex County -0.396** (-4.28) -0.217* (-2.40) -0.214 (-1.59)
   S. Maine -0.188 (-1.94) -0.44 (-0.47) 0.118 (0.86)
Financial Control
   Cleveland -0.206* (-2.22) 0.052 (0.58) -0.012 (-0.09)
   Greater Lynn -0.129 (-1.43) 0.135 (1.56) 0.110 (0.87)
   Miami -0.419** (-4.77) -0.170* (-2.01) -0.260* (-2.05)
   Philadelphia -0.147 (-1.70) 0.041 (0.49) -0.049 (-0.40)
   (Rensselaer)            
IMPAIRMENT OF ABILITY TO PERFORM ACTIVITY OF DAILY LIVING (ADL)a
Extremely severe -0.171** (-2.76) -0.241** (-4.00) -0.162 (-1.92)
Highly severe -0.056 (-1.04) -0.066 (-1.28) -0.036 (-0.49)
Moderately severe -0.019 (-0.34) -0.040 (-0.76) -0.004 (-0.06)
(Mild or none)            
CONTINENCEa
Colostomy bag, device, need help -0.305** (-5.05) -0.304** (-5.04) -0.229** (-2.65)
Incontinent -0.065 (-1.77) -0.131** (-3.68) -0.137** (-2.75)
(Continent)            
REFERRAL SOURCE
Hospital or nursing home -0.190** (-4.10) -0.163** (-3.58) -0.095 (-1.44)
Home health agency -0.072 (-1.43) -0.145** (-2.97) -0.079 (-1.12)
(Other)            
ETHNICITY
Black 0.060 (1.24) 0.003 (0.06) 0.065 (0.97)
Hispanic 0.526** (4.86) 0.497** (4.83) 0.486** (3.71)
(White)            
MALE -0.218** (-5.50) -0.275** (-7.06) -0.324** (-5.84)
AGE (in years) -0.004 (-1.82) -0.008** (-3.62) -0.009** (-2.92)
COGNITIVE IMPAIRMENTb
Severe -0.064 (-1.18) -0.021 (-0.39) -0.062 (-0.85)
Moderate -0.035 (-0.79) -0.056 (-1.31) -0.085 (-1.41)
(Mild, none)            
INTERVIEWER ASSESSED UNMET NEEDSb
High -0.024 (-0.53) -0.005 (-0.11) 0.039 (0.61)
Medium -0.015 (-0.34) 0.024 (0.57) -0.003 (-0.05)
(Low)            
MEDICAID INSURANCE 0.055 (1.20) 0.041 (0.94) 0.089 (1.49)
PROXY USE AT SCREEN -0.075 (-1.47) -0.042 (-0.86) -0.092 (-1.37)
REGULAR HELP RECEIVED WITH
Meal preparation -0.063 (-1.03) -0.055 (-0.94) -0.036 (-0.46)
Housework, shopping 0.149* (2.36) 0.065 (1.08) 0.169* (2.07)
Taking medicine -0.045 (-0.89) -0.037 (-0.75) -0.046 (-0.67)
Medical treatments at home -0.069 (-1.56) -0.059 (-1.39) -0.088 (-1.45)
Personal care -0.049 (-0.86) -0.042 (-0.75) -0.129 (-1.70)
INCOME
<$500/mo. 0.040 (0.54) 0.041 (0.57) -0.012 (-0.12)
$500-$999/mo. -0.007 (-0.11) 0.003 (0.05) -0.052 (-0.54)
(>$1,000/mo.)            
ON WAITING LIST (or applied for) NURSING HOME -0.007 (-0.12) 0.002 (0.03) -0.104 (-1.34)
NUMBER OF CONTACTS TO OBTAIN SCREEN INTERVIEW -0.048** (-2.98) -0.044** (-2.82) -0.041 (-1.93)
NUMBER OF MISSING ITEMS ON SCREEN 0.016 (1.82) 0.027** (3.16) 0.024 (1.76)
EXPECTED TO NEED HELP TO COMPLETE BASELINE 0.016 (0.34) -0.014 (-0.31) 0.040 (0.64)
LIVING ARRANGEMENTb
With child -0.046 (-0.85) -0.053 (-0.99) -0.129 (-1.70)
With other (no spouse or child) -0.116 (-1.65) -0.075 (-1.09) -0.114 (-1.19)
Alone -0.111* (-2.15) -0.032 (-0.64) -0.136 (-1.88)
(With spouse, not with child)            
CONSTANT 1.23** (5.91) 1.20** (5.87) 1.06** (3.62)
PERCENT IN SAMPLE 66   57   44  
SAMPLE SIZE 6,326   6,326   3,165  
-2 LOG LIKELIHOOD RATIO 415.05   468.96   233.34  
DEGREES OF FREEDOM 45   45   45  
NOTE: For categorical variables the names of omitted categories are enclosed in parentheses, except where it is obvious (e.g., male).
  1. Missing values for this variable were replaced by the mean.
  2. A missing value indicator was included for this variable in the model (coefficient not reported).

* Statistically significant at the 5 percent level for a two-tailed test.
** Statistically significant at the 1 percent level for a two-tailed test.

The likelihood ratio statistics reported at the bottom of Table V.1 are for tests of whether all probit coefficients (except the intercept) are simultaneously zero. The large values of these test statistics indicate that this hypothesis is strongly rejected and suggest that the screen variables as a group do lead to significantly improved predictions of whether specific sample members respond. Furthermore, we can determine from the t-values which of the factors are important determinants of being in the samples. Consistent with the response rates discussed in Chapter II, treatment group members are significantly more likely to respond than are controls, except in the basic model at 18 months. There are significant between-site differences in the probability of response , but only those in the Miami site are consistently less likely (relative to Rensselaer) to respond at all three interviews. As expected, extremely severe ADL impairments at screen reduces the likelihood of response (less so at 18 months), and so do continence problems. Another indicator of poor health status, whether referred to channeling by a hospital or nursing home, also substantially reduces the likelihood of being in the 6- and 12-month followup samples. On the other hand, no explanation is apparent for why hispanics are substantially more likely to he included in all three followup samples, as compared to blacks and whites (note, however, that only about 2 percent of the sample members are hispanic in the basic sites and 5 percent in the financial control sites). Furthermore, males are consistently less likely than females to respond; those who receive regular help with housework and shopping are more likely to respond at the 6 and 18 month interviews. Those living alone are less likely to respond at 6 months, although living arrangement does not seem to affect the likelihood of responding to the later interviews. Finally, the more contacts it took to obtain a screen interview from a given sample member, the less likely it was that a followup interview was obtained. This variable is apparently a good proxy for the tendency to cooperate with interviews.

We also estimated models of the probability that sample members were included in the other analysis samples. Probit models analogous to the ones presented in Table V.1 were estimated for the probability that the sample member was included in the nursing home and in-community samples at 6 and 12 months.25 The results are generally quite similar to those obtained for the followup samples in terms of what factors are related to attrition. This is not surprising, given the relatively small number of cases that are included in the other samples but excluded from the followup samples. The primary differences between the other analysis samples and the followup sample in the factors affecting whether observations are available for analysis are:

  • Medicaid eligibility is a highly significant predictor of inclusion in the nursing home samples, but not in the followup or in-community samples.

  • ADL is not a significant predictor of inclusion in the nursing home samples, but the severely impaired are significantly less likely to be included in the other analysis samples.

  • Older individuals are significantly less likely to be in the community samples but no less likely to be in the followup or nursing home samples.

The fact that Medicaid eligibles are much more likely to be in the nursing home samples than noneligibles, but no more likely to be in the other samples, is not surprising, given that the nursing home samples are defined to include all individuals known to be on Medicaid throughout a period, even if no followup interviews were completed. The significance of age in predicting inclusion in the in-community sample reflects the fact that the oldest individuals are more likely to be in hospitals or nursing homes, even though they are no less likely to complete (or have a proxy complete) the interview. It is unclear why severe ADL impairments does not significantly decrease the likelihood of being in the nursing home analysis samples, unless the severely impaired tend to he Medicaid-eligible and are therefore automatically included in the nursing home samples, despite the fact that they were less likely to complete the interviews necessary to be included in the other samples.

The finding that about half of the screen variables appear to have significant effects on the probability that observations are available for analysis suggests that, as expected, attrition is not entirely random. Nevertheless, it does not appear to be strongly related to the set of screen factors at our disposal. This is clear from Table V.2, which displays the distribution of predicted probabilities obtained from the model for both responders and nonresponders. Although there is some difference between these two distributions, as evidenced by the Chi-square test showing that they differ significantly more than would be expected by chance, it is clear that the model does not discriminate well between responders and nonresponders. Responders tend to have only slightly higher predicted probabilities of response than nonresponclers. A goodness of fit measure, analogous to the R2 statistic produced for regressions, was quite low for all of the models.

It is important that this lack of predictive power be properly interpreted, however. What it shows is that attrition is not closely tied to the fairly extensive set of screen characteristics, but rather occurs for a wide variety of unknown reasons. This should be viewed as evidence that those who drop out of the sample are not strikingly different from those that remain in, i.e., that attrition bias is relatively unlikely. This is especially so since much of the attrition occurs at baseline, which is only a short time after the screen interview was conducted. Any relationship between personal characteristics and attrition at baseline, therefore, should not be masked by drastic changes in the characteristics between the time of measurement (screen) and the time the response decision was made.

It is true that if the attrition correction term (M) is actually affected by personal characteristics, but none of these characteristics appear in the attrition equation, that the attrition model will produce poor estimates of M and the Heckman procedure described in Chapter III will erroneously indicate that there is no bias. However, this typically occurs because very few characteristics of nonresponders are available for inclusion in the response model in most applications of this procedure. In this analysis, however, the screen provides a great deal of information on sample members, and these data are used in the attrition model. The relevant criteria in assessing the ability of the model to control for attrition is not how well it fits (since attrition may be totally or largely random) but rather that important variables that might affect attrition and whose coefficients in the outcome equation we are most interested in appear in the model of response. In this study, treatment status clearly affects attrition. That relationship is reflected in the estimated response model; hence, the model, despite low predictive power, produces a very adequate instrument for M. Attrition bias, if it exists, should be identified by a significant coefficient on M.

TABLE V.2: Measures of the Predictive Accuracy of the Response Models and Distribution of Responders and Nonresponders at the Followup Interviews by Predicted Probability of Response
(percent)
Sample & Sample Member Response Status Predicted Probability of Response   Total   Number of
  Observations  
  R2a   Chi-
  squareb  
(df)
0-   0.10   0.11-   0.20   0.21-   0.30   0.31-   0.40   0.41-   0.50   0.51-   0.60   0.61-   0.70   0.71-   0.80   0.81-   0.90     0.91-   1.0
FOLLOWUP SAMPLE (%)
6-Months
   Nonresponders    0 0 1 5 13 25 30 22 5 0 100.0 (2,149) 0.065 399.9**
(7)
   Responders 0 0 0 1 6 17 28 33 14 0 100.0 (4,177)
12-Months
   Nonresponders 0 1 4 13 24 27 21 10 1 0 100.0 (2,703) 0.072 435.8**
(8)
   Responders 0 0 1 6 15 25 31 19 3 0 100.0 (3,623)
18-Months
   Nonresponders 0 5 17 27 26 18 7 1 0 0 100.0 (1,760) 0.071 213.4**
   Responders 0 1 8 19 29 25 15 3 0 0 100.0 (1405)
NOTE: Percent ages do not always sum to 100 due to rounding. Predicted probabilities were obtained from the estimated probit models presented in Table V.1.
  1. The R2 measure is Efrons R2 for qualitative response models (see Amemlya, 1981), the analogue to the R2 statistic for linear regression.
  2. These Chi-square statistics test whether the distribution of the predicted probabilities of response for respondents differ from the distribution for nonrespondents by more than might be expected by chance. All these statistics strongly reject the hypothesis that the distributions are equal for respondents and nonrespondents, which implies that the model does discriminate to some degree between respondents and nonrespondents.

** Statistically significant at the 5 percent level.

Another very important feature of the attrition model used here is that it includes several factors that are unlikely to affect outcomes. Chief among these factors is the number of contacts required to complete the screen, which has a statistically significant effect on the probability of response. Even some of the variables that do appear in both the attrition and outcome equations are not exactly the same because they come from different sources and take different forms in the two equations. Having some nonoverlapping variables in the attrition model and outcome model greatly increases the validity of the attrition bias correction procedure. Thus, we proceed in the next section to use the attrition model estimated here to control for attrition bias in estimates of program impacts.

View full report

Preview
Download

"atritn.pdf" (pdf, 4.74Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®