In the channeling evaluation, as in other longitudinal studies, we are faced with the fact that some members of the research sample are lost to the analysis due to attrition occurring during the demonstration evaluation.1 In an earlier report (Brown and Harrigan, 1983) we showed that the treatment and control groups at the time of randomization consisted of similar types of individuals; hence, post-randomization differences between those two groups could be attributed to the effects of channeling. However, sample attrition may distort the treatment/control group comparison, depending on the type of attrition that takes place. Attrition that does not depend in any systematic way on factors relevant to the outcome being measured leads to less precise estimates of program impacts (due to the reduction of the sample size), but does not lead to biased estimates. However, if the pattern of attrition is different for the treatment and control groups, the sample of treatment and control group members available for analysis will no longer be similar in their characteristics. In this case, differences in outcomes between the groups cannot be attributed to the effects of channeling alone, and impact estimates that do not adjust for the differences induced by different attrition patterns will he biased.
The purpose of this report is to investigate whether there is evidence of bias due to attrition in the estimates of channeling's impacts, which are based on interviews administered 6, 12, and 18 months after randomization, and on other data collected on sample members. The conclusions presented here were based on a variety of analyses that were conducted over the course of the evaluation and were used to guide the decision about the proper methodology to use in estimating channeling impacts.
In this technical report we assume that the reader is familiar with the channeling demonstration and research methodology, which is described in other project reports (see Carcagno et al., forthcoming). We limit our discussion in this report to how impact estimates for a subset of the key outcome variables examined in the channeling evaluation are affected by sample attrition. The effects of incomplete data on estimates of channeling impacts on mortality are not examined here, but are addressed in Wooldridge and Schore (forthcoming, Appendix F). That analysis revealed no evidence of bias due to missing data. The Wooldridge and Schore report also includes an analysis of the effects of attrition on estimated channeling impacts on hospital and nursing home outcomes (see Appendix E of that report). The current report summarizes and extends the analysis presented there, and examines evidence on whether attrition affects impact estimates for other outcomes.
The remainder of the report is organized as follows. Chapter II defines the various analysis samples used in the evaluation and describes the extent of attrition and the profiles of those remaining in the 6, 12, and 18 month analysis samples. Chapter III discusses how bias due to attrition might arise in the impact estimates and describes a procedure that will be used to correct statistically for the effects of attrition. Chapter IV presents a heuristic analysis of attrition bias, using Medicare claims data, which are available for respondents and nonrespondents, to determine whether treatment/control differences in Medicare-covered services computed on just the analysis sample differ from those obtained for the full research sample. Chapter V contains the estimates of statistical models to predict whether a sample member will remain in the analysis sample at 6, 12, and 18 months, based on his or her characteristics as measured at the screen interview. The results of these models are then used to construct variables that control for the potential effects of attrition on estimates of program impacts. Estimates of channeling impacts with and without this accounting for possible attrition bias are then compared. Results from sensitivity tests, reported on in detail elsewhere, are also summarized in this chapter. Finally, Chapter VI summarizes the results of this analysis and draws inferences about attrition bias in other channeling impact estimates.