Final Report on the Effects of Sample Attrition on Estimates of Channeling's Impacts

Randall S. Brown, Peter A. Mossel, Jennifer Schore, Nancy Holden and Judy Roberts

Mathematica Policy Research, Inc.

The paper was written as part of contract #HHS-100-80-0157 between the U.S. Department of Health and Human Services (HHS), Office of Social Services Policy (now the Office of Disability, Aging and Long-Term Care Policy (DALTCP)) and Mathematica Policy Research, Inc., and contract #HHS-100-80-0133 between DALTCP and Temple University. Additional funding was provided by the Administration on Aging and Health Care Financing Administration (now the Centers for Medicare and Medicaid Services). For additional information about this subject, you can visit the DALTCP home page at http://aspe.hhs.gov/_/office_specific/daltcp.cfm or contact the office at HHS/ASPE/DALTCP, Room 424E, H.H. Humphrey Building, 200 Independence Avenue, S.W., Washington, D.C. 20201. The e-mail address is: webmaster.DALTCP@hhs.gov. The Project Officer was Robert Clark.

This report was prepared for the Department of Health and Human Services under Contract Number HHS-100-80-0157. The DHHS project officer is Ms. Mary Harahan, Office of the Secretary, Department of Health and Human Services, Room 447F, Hubert H. Humphrey Building, Washington, D.C. 20201. The opinions and views expressed in this report are those of the authors. They do not necessarily reflect the views of the Department of Health and Human Services, the contractor or any other funding organization.

ACKNOWLEDGMENTS

As is clear from the long list of authors, a fairly large number of individuals contributed to this report. The authors wish to acknowledge and thank Dan Buckley, Joan Mattei, and Nancy Holden for conducting the computer programming necessary to generate the many tables in this report, and Annette Protonentis for expert word processing, table formatting advice, and extraordinary patience in the production of the finished document. Peter Kemper provided extremely valuable advice throughout and comments on early drafts of this paper. Judith Wooldridge conducted some of the sensitivity tests that are reported in detail elsewhere but are summarized here. Finally, the internal and external reviewers selected by the Department of Health and Human Services provided useful comments on an early draft of this report. To all contributors, thank you for your assistance.

EXECUTIVE SUMMARY

The National Long Term Care Demonstration was established by the U.S. Department of Health and Human Services to evaluate community-based approaches to long term care for the elderly. The channeling demonstration was designed to determine the impact of providing community-based services on costs, utilization of services, informal caregivers, and client wellbeing.

In designing the evaluation of the demonstration, great care was taken to ensure that the results of that evaluation would not be called into serious doubt because of methodological shortcomings. Thus, an experimental design was used, under which eligible channeling applicants in each of the 10 sites were randomly assigned to the treatment group which was offered channeling services, or to the control group which was not. Because of the random assignment, the control group should be very similar to the treatment group on both observable and unobservable characteristics, and therefore, their experience should provide the best possible estimate of what would have happened to the treatment group had the demonstration not existed.

One aspect of the evaluation which could, however, raise questions about the accuracy of the estimates of channeling impacts is the fact that impacts can be estimated only on those sample members for whom followup data on outcomes is available. The loss of sample members from the analysis samples entails--in addition to reduction in sample sizes--the risk that sample members remaining in the treatment and control groups might differ on observed and unobserved characteristics, leading to biased estimates of channeling impacts.

In order to eliminate effects that attrition might have on the comparability of the treatment and control groups, regression models were used throughout the channeling evaluation to estimate program impacts. This statistical procedure controls for any observed initial differences between the two groups of observations remaining after attrition. However, use of regression does not ensure that the estimates are not biased by attrition, because it controls only for observed differences between the two groups. Two conditions are required for regression estimates of channeling impacts on a particular outcome variable to be biased as a result of attrition: (1) the presence of unobserved factors that affect .both the likelihood of response at followup and the value of the outcome variable at followup, and (2) a different rate or pattern of attrition for treatment and control groups.

Because of the differing data needs and sources of data for the various outcomes of interest in the evaluation, many different analysis samples were used. All of the analyses, however, relied to some degree on those with completed interviews at baseline, and/or at the followup interview covering a given six-month interval (ending 6, 12, and 18 months after randomization). The proportion of the full sample included in the various analysis samples was nearly always substantially lower for the control group than for the treatment group in all three time periods, especially in the financial control model. These differences arose primarily because of the large treatment/control difference in response rates at the baseline. Thus, one of the conditions required for attrition bias was present. However, despite this difference in rates of attrition, the analysis samples exhibited only minor treatment/control differences on initial screen characteristics.

To investigate whether the primary source of bias in impact estimates--unobserved factors affecting both response and the outcome being examined--was present, two types of approaches were taken: a heuristic approach and a statistical modeling approach. The heuristic approach was to make use of the Medicare claims data available for virtually the entire sample, on Medicare-covered use of and reimbursements for hospitals, nursing homes, and formal community-based services. To learn something about the likelihood that there were large differences on unobserved characteristics between those sample observations available for analysis and those that were not, channeling impacts on these Medicare-covered services were estimated for the full sample and then again for the various analysis samples. Estimates of channeling impacts on this partial set of service use measures were generally very similar for the analysis and full samples, which led to the following conclusions: (1) estimated impacts on hospital outcomes were definitely not biased by attrition, (2) estimated impacts on total nursing home and total formal service use (not just that paid for by Medicare) were not likely to be biased, and (3) estimated impacts on other (well-being and informal care) outcomes probably were not biased.

The statistical modeling approach was then used to provide additional evidence on the existence and magnitude of attrition bias. The procedure that was used required the estimation of a model to predict whether the sample member was in the analysis sample (using all of the observations), and then the use of the estimated model to construct a new variable for each member of the analysis sample. This new variable, when included as an additional control variable in the statistical (regression) model used to estimate channeling impacts, accounts for the effects of attrition on these estimates.

Comparison of the estimates of channeling impacts obtained with and without inclusion of the term to control for attrition showed no major differences in the estimates, for any of the key outcomes examined. A somewhat more general model also yielded results that implied that attrition bias was small or nonexistent.

Finally, the statistical modeling approach and exploitation of the Medicare data were supplemented by additional specialized analyses of the effects of attrition on estimates of channeling impacts on nursing home use and mortality. Using a variety of imputation procedures for cases without nursing home use data showed that estimates of nursing home impacts did not appear to be biased by sample attrition. Similar sensitivity tests for mortality estimates led to the same conclusion. This was further supported by the finding that the vast majority of individuals for whom no definite information on death was available (from death records or interview attempts) were in fact alive, because they either were found to have Medicare claims for services after the dates on which mortality was measured, or were not found to be deceased in an examination of updated Medicare status files.

The results from these various approaches lead us to conclude that, in spite of the observed treatment/control differences in attrition rates, there is very little evidence that attrition resulted in biased estimates of channeling impacts. The occasional bits of evidence to the contrary were scattered and inconsistent across time, model, and outcome variables. Although each of the approaches employed has its flaws, the (rare) availability of substantial information on attriters both before and during the followup period and the fact that all of the approaches point to the same basic conclusion provides a high degree of confidence in the inference that attrition has not led to distorted estimates of channeling impacts.

I. INTRODUCTION

In the channeling evaluation, as in other longitudinal studies, we are faced with the fact that some members of the research sample are lost to the analysis due to attrition occurring during the demonstration evaluation.¹ In an earlier report (Brown and Harrigan, 1983) we showed that the treatment and control groups at the time of randomization consisted of similar types of individuals; hence, post-randomization differences between those two groups could be attributed to the effects of channeling. However, sample attrition may distort the treatment/control group comparison, depending on the type of attrition that takes place. Attrition that does not depend in any systematic way on factors relevant to the outcome being measured leads to less precise estimates of program impacts (due to the reduction of the sample size), but does not lead to biased estimates. However, if the pattern of attrition is different for the treatment and control groups, the sample of treatment and control group members available for analysis will no longer be similar in their characteristics. In this case, differences in outcomes between the groups cannot be attributed to the effects of channeling alone, and impact estimates that do not adjust for the differences induced by different attrition patterns will he biased.

The purpose of this report is to investigate whether there is evidence of bias due to attrition in the estimates of channeling's impacts, which are based on interviews administered 6, 12, and 18 months after randomization, and on other data collected on sample members. The conclusions presented here were based on a variety of analyses that were conducted over the course of the evaluation and were used to guide the decision about the proper methodology to use in estimating channeling impacts.

In this technical report we assume that the reader is familiar with the channeling demonstration and research methodology, which is described in other project reports (see Carcagno et al., forthcoming). We limit our discussion in this report to how impact estimates for a subset of the key outcome variables examined in the channeling evaluation are affected by sample attrition. The effects of incomplete data on estimates of channeling impacts on mortality are not examined here, but are addressed in Wooldridge and Schore (forthcoming, Appendix F). That analysis revealed no evidence of bias due to missing data. The Wooldridge and Schore report also includes an analysis of the effects of attrition on estimated channeling impacts on hospital and nursing home outcomes (see Appendix E of that report). The current report summarizes and extends the analysis presented there, and examines evidence on whether attrition affects impact estimates for other outcomes.

The remainder of the report is organized as follows. Chapter II defines the various analysis samples used in the evaluation and describes the extent of attrition and the profiles of those remaining in the 6, 12, and 18 month analysis samples. Chapter III discusses how bias due to attrition might arise in the impact estimates and describes a procedure that will be used to correct statistically for the effects of attrition. Chapter IV presents a heuristic analysis of attrition bias, using Medicare claims data, which are available for respondents and nonrespondents, to determine whether treatment/control differences in Medicare-covered services computed on just the analysis sample differ from those obtained for the full research sample. Chapter V contains the estimates of statistical models to predict whether a sample member will remain in the analysis sample at 6, 12, and 18 months, based on his or her characteristics as measured at the screen interview. The results of these models are then used to construct variables that control for the potential effects of attrition on estimates of program impacts. Estimates of channeling impacts with and without this accounting for possible attrition bias are then compared. Results from sensitivity tests, reported on in detail elsewhere, are also summarized in this chapter. Finally, Chapter VI summarizes the results of this analysis and draws inferences about attrition bias in other channeling impact estimates.

II. THE NATURE AND EXTENT OF ATTRITION IN THE ANALYSIS SAMPLES

The outcome measures for which channeling impacts are estimated are obtained from a variety of services. Thus, for each of the major areas of analysis in the evaluation, "analysis samples" have been defined. The analysis sample for any area is composed of that subset of the full research sample for which the necessary individual data on independent and dependent (outcome) variables are available. However, most of these analysis samples are tied closely to whether or not the sample member completed the baseline survey and the followup surveys administered 6, 12, and (for a subset of the sample) 18 months after randomization. The data sources and analysis samples are described below, followed by a comparison across treatment groups and models in the proportions of the research sample that have the data necessary for the various analyses. Total attrition rates, and reasons for attrition, are given for the samples used in the analysis of impacts at 6, 12, and 18 months after randomization. The chapter concludes with an analysis of whether the treatment and control groups available for analysis are composed of different types of individuals as a consequence of attrition.

A. The Demonstration and the Evaluation

Channeling² consists of a set of seven core functions--outreach, screening, comprehensive needs assessment, care planning, service arrangement, monitoring, and reassessment--deemed necessary to rationalize service use and ultimately reduce costs and improve client well-being. To this end two models of channeling are being tested in the demonstration in 10 sites. The basic case management model adds limited funds to the core functions, giving case managers somewhat greater flexibility in designing and implementing care plans. The financial control model adds to the core functions by substantially expanding the service coverage of public programs, pooling funds from separate government programs, and allowing case managers to authorize services to be paid for from the funds pool. These are combined with a cap on average annual service expenditures per client (60 percent of the state's average reimbursement rate for intermediate and skilled nursing home care), and a limit on the annual cost of individual care plans (85 percent of the state's average nursing home reimbursement rate) that can be exceeded only with state approval. Channeling operations began in a phased startup between February and June 1982 at 10 sites, 5 implementing the basic case management model, and 5 the financial control model.

Impacts of channeling are estimated in this evaluation by statistically comparing the experiences of two groups of individuals: the treatment group, members of which were entitled to participate in channeling, and the control group, whose members were not allowed to participate. Individuals who applied or were referred to channeling and were found to be eligible were randomly assigned to one of these two groups.

To support this analysis, various data were collected on these sample members from a variety of sources, described below, on the outcomes which channeling was expected to influence. The specific outcomes examined fall into the following broad categories:

• Nursing home use and costs
• Hospital use and costs
• Use and cost of formal community-based services
• Receipt of care from informal caregivers
• Sample members' well being
• Mortality

Many specific variables in each of these categories were examined in the course of the evaluation for evidence of channeling impacts. Below we describe the sources of data for variables in each of these areas and the samples available for analysis.

1. The Data

The analyses of channeling impacts relies on many sources of data on sample members. The data may be classified broadly as “interview” data or “records” data. Interview data³ sources include the screen interview, which was administered to all persons referred or applying to channeling to assess their eligibility for the program; the baseline interview, administered to eligible sample members as soon as possible after they were assigned to the treatment or control group (interviews were usually completed within 2 weeks after randomization); and the followup interviews, administered 6, 12, and 18 months after randomization in order to obtain data on outcomes which channeling was hypothesized to influence.⁴ Records data include Medicare and Medicaid claims data, records data from providers of services (e.g., nursing homes) that sample members claimed in the interviews to have used, financial control system data (for channeling clients in financial control sites) from the channeling agencies, and death records. These data sources are described below.

The Screen. The screen questionnaire, administered primarily by telephone by channeling intake workers, was designed primarily to assess eligibility for channeling and contained data on sample members' ability to perform various activities of daily living, their unmet needs for assistance of several types, and some sociodemographic characteristics. Applicants determined to be eligible for channeling were then randomly assigned to treatment or control status by research staff. Screen interviews were completed with 6,340 eligible sample members. Unfortunately, 14 screen interviews were lost in the mail,⁵ so that the remaining screen sample consists of 6,326 observations. The screen sample is thus the full research sample, and we refer to it as such throughout this report.

The Baseline. The screen interview does not, however, contain the comprehensive data that were necessary for either the evaluation or the development of a care plan for channeling clients. A thorough, in-person baseline assessment of treatment group members was required in order for program case managers to develop an appropriate care plan for participants. A single instrument was developed that would serve both the purpose of care planning and research. It was considered important that channeling staff members collect the data necessary for developing an appropriate care plan; therefore, the baseline interview (but not the followup interviews) was administered by channeling staff for the treatment group and by research interviewers for the control group.⁶ Treatment group members who refused the baseline assessment interview could not participate in channeling, since no care plan could be developed for them. However, since these individuals could differ substantially from other treatment group members, nonresponding members of the treatment group were interviewed by research interviewers whenever possible. This enabled us to retain them in the analysis sample. Overall, 108 (3 percent) of the baseline interviews for the treatment group were administered by research interviewers.

The Followup Interviews. For sample members who completed the baseline, followup interviews at 6, 12, and 18 months after randomization were attempted by research interviewers to gather the data on sample members' outcomes that were necessary to assess the impact of channeling. Although a completed baseline was a condition for being contacted for a followup interview, a noncompleted 6-month interview did not make the sample member ineligible for a 12-month interview. Thus, some sample members who did not complete a 6-month interview did complete a 12-month interview.

The situation was different at the 18-month interview. First, because of budgetary reasons, only half of the sample members randomized were eligible for an 18-month interview.⁷ Second, an 18-month followup was attempted only if the sample member belonged to this 18-month cohort, and had a completed baseline, 6-month and 12-month followup interview.

Medicare Claims Data. Medicare claims data were collected for all sample members who said that they were eligible for Medicare and for whom a valid. Medicare identification number could be verified by HCFA. Nearly the entire sample (97 percent) was eligible for Medicare. Claims provided data on sample members' hospital use, some nursing home use, and use of other medical services and community-based services paid for by Medicare. See Wooldridge and Schore (forthcoming) for a detailed discussion of Medicare data.

Medicaid Claims Data. Medicaid claims were collected for all sample members who said they were eligible for Medicaid at any interview and signed a consent form, if this information and a valid Medicaid ID number could be verified by the state Medicaid program. Medicaid claims were a key source of data on nursing home outcomes and use of formal community services.

Provider Records Data. Data on the nursing home use of specific sample members were collected from nursing homes for sample members stating in an interview that they had spent time in that institution during the reference period or were living there at the time of the interview. Records data were also collected from area hospitals on those few sample members who were not on Medicare.⁸

Financial Control System Data. Because of the pooling of Medicare and Medicaid funds in the financial control model, data on use of formal community services by treatment group members in that model were obtained from the channeling agencies' records.

Death Records. Data on mortality were obtained from a search of state death records for all sample memers who failed to complete their last scheduled interview. These data were supplemented by data on mortality obtained in the attempt to field the followup interviews and from client-tracking data (for treatment group members).

2. The Analysis Samples

For 5 of the 6 categories of outcomes identified above, the sources and therefore the completeness of the necessary data differ. The analysis samples for each of these areas are:

• Mortality--full research sample
• Hospital outcomes--6, 12, and 18 month Medicare samples
• Nursing home outcomes--6, 12, and 18 month nursing home samples
• Well-being outcomes--6, 12, and 18 month followup samples
• Receipt of formal community based services and informal care--6, 12, and 18 month in-community samples

These samples and the relationship between them are described below.

Full Sample. This sample includes all of the 6,326 initially randomized individuals, and was used to estimate the impacts of channeling on mortality, as measured by whether sample members were alive at 6 and 12 months after randomization. The full 18 month sample, used to estimate impacts on mortality at 18 months, includes the 3,165 members of the full sample who were in the 18 month cohort. A search of state death records was conducted for all sample members not known to be alive from the interviews, and these records data were supplemented with information on deaths obtained from attempts to field followup interviews and from channeling programs' client tracking systems. Sample members identified as dead from either source were assumed to be alive; hence, there was no missing data on mortality. An analysis of the validity of this assumption, presented in Wooldridge and Schore (forthcoming, Appendix F), makes use of Medicare claims data and updated status files to verify that the assumption is correct for virtually all sample members. Hence, there is no bias in estimates of channeling impacts on mortality.

The Medicare Sample. The Medicare sample was employed to examine channeling's impacts on the use of hospital and other medical services, and on home health expenditures reimbursed by Medicare. The Medicare sample is the subset of the 6,326 initially randomized individuals (the full sample) who completed baseline interviews and who are either known to be Medicare entitled or known not to be Medicare entitled. This sample was used for analyzing, outcomes in the first 12 months following randomization. To be consistent with the analyses of channeling's impacts on outcome measures obtained from follow-up interviews, the 18-month Medicare sample is restricted to those members of the Medicare sample who were also in the 18-month cohort.

The Nursing Home Samples. Because Medicare claims do not provide a complete history of nursing home use, the samples used for the nursing home analysis differed from those used for the hospital analysis. Most nursing home expenses are paid by Medicaid, for Medicaid-covered individuals, or by private payment, for those not covered by Medicaid. Therefore, the nursing home analysis employed a two-pronged data collection strategy, relying on Medicaid (and Medicare) records to provide complete nursing home information for sample members who were covered by Medicaid, and on provider (and Medicare) records for those who were not Medicaid-covered. However, in order to identify the relevant providers for this latter group, either a followup interview or caregiver interview had to have been completed.

These data requirements resulted in three nursing home samples, one for each six-month period. These are subsamples of the Medicare samples, and include individuals who either completed a followup interview, were Medicaid covered throughout the six-month period, or died in the period but had a caregiver who provided followup information. In addition, Medicare sample members who were dead throughout a six-month period, or who died during the period and were Medicaid-covered at the start of the period and at death were also included in the nursing home sample for that period.

The Followup Samples. The followup samples were used to analyze outcomes obtained from the followup surveys administered at 6, 12, and 18 months after randomization. The two major categories of impact analyses which relied on these samples are those dealing with sample members' well being and functional ability and those dealing with case management services. The followup sample at 6 months includes the subset of the screen sample with both a complete baseline and a complete 6-month followup interview. In like manner, the sample at 12 months is composed of screen sample members who completed a baseline and a 12 month followup (but not necessarily a 6 month interview). The 18 month sample includes only those in the early cohort who completed a baseline and followup interviews at all three time periods.

The In-Community Samples. Estimation of channeling impacts on receipt of formal and informal care required data on these outcome variables from the followup surveys. The interview data on receipt of such services pertained to the reference week--the week 6, 12, or 18 months after randomization, for sample members residing in the community during this reference week. Thus, for those who were in a hospital, in a nursing home, or deceased at this time (but alive at some time during the 6 month period), use during weeks when they were in the community is unknown.⁹ (Of course, sample members who had died prior to the start of the period spent no time in the community and therefore are excluded from the relevant set of observations.) Therefore, the 6, 12, and 18 month in-community samples were composed of those sample members who completed the relevant interview and were living in the community during the reference week.

These five different sample types form a hierarchy, with each being nested, or nearly so, within the one above it.¹⁰ Figure II.1 shows the relationship between them.

Before proceeding to the examination of response rates, there are two points that should be noted. First, each of the analysis samples described above are used to examine channeling impacts on many outcome measures falling under the general area for which the samples were defined. Clearly, we cannot examine all of these estimates for evidence of attrition, so we have identified the key outcome measures within each substantive area and confined our analysis of attrition bias to these variables. The variables are listed in Chapter IV.

FIGURE II.1: Flowchart of Inclusion In and Attrition From Analysis Samples

The other point to be made is that an observation can be omitted from the analysis of impacts on a given outcome measure due to item nonresponse on that variable, even if the sample member is in the appropriate analysis sample. In general, there are relatively few such cases of missing data on individual outcome variables when the observation is in the appropriate analysis sample, and, of course, which observations are missing depends on the specific variable examined. Thus, the possible effect of item nonresponse on impact estimates is ignored in this report.

B. Response Rates and Reasons for Attrition

In Table II.1, we present the full sample size and percent of the full sample included in each of the analysis samples. Figures are presented separately by model and experimental status.

The results indicate that at 6 months, 88 percent of the full sample was in the Medicare sample. Loss of observations was due almost entirely to sample members' failure to complete the baseline (as we shall see later); nonverifiability of Medicare eligibility was responsible for only about one-tenth of the cases excluded from this analysis sample.

Only 73 percent of the full sample was included in the nursing home sample, with the additional loss of observation arising because of sample member death and nonresponse to the 6 month interview. The followup sample included two thirds of the full sample, while the in-community sample at six months was comprised of the 55 percent of sample members who completed the 6 month followup and were residing in the community.

The proportions of the full sample included in the analysis samples at 12 months were comparable to those at 6 months. The proportion in the followup sample was lower, as expected, given the substantial fraction of sample members who died within the 7 to 12 month period. The proportion of the full sample included in the nursing home sample actually increased between the first and second six month periods, however, because some sample members who died within the first period were excluded from the 6 month analysis (since their utilization of-nursing homes was unknown) but included in the 12-month analysis (because their utilization was known to be zero).

Two figures are presented for the in-community sample at 12 months. The first is the proportion of the full sample that was included in this sample; the second is the proportion of those alive at the beginning of the period. The latter is the relevant measure of sample inclusion in this sample, since the in-community sample is used to estimate impacts on use of formal and informal care for sample members during the time they spent in the community. Since sample members who were deceased at the beginning of the period could never be in the community, this outcome is undefined rather than just missing for this sample. The proportion of those alive at the beginning of the 7 to 12 month period who are included in this sample is by chance the same (55 percent) as the proportion of those alive at the beginning of the 1 to 6 month period (the full sample) who are included in the 6 month in-community sample.

For the 18 month sample the sample retention rates are similar to the rates in other periods for the Medicare and nursing home samples but considerably lower for the followup and in-community samples because of additional deaths and the requirement that sample members complete both of the earlier interviews.

Comparing treatment and control groups we see that the differences in the proportion of observations available for analysis are substantial and fairly constant (about 10 percentage points) across all periods and analysis samples. Thus, the treatment/control difference for all samples appears to be due to the differential response rates at baseline. The difference is especially pronounced for the financial control model (about 13 percentage points, compared to 6 to 8 points for most samples in the basic model).

1. Reasons for Attrition

To obtain a somewhat more detailed picture of the reasons for sample loss and how it differs across experimental groups and models, we present in Table II.2a and Table II.2b a breakdown of the total attrition from the full sample by the reason data were unavailable, for each analysis sample. The results are discussed below.

The Medicare Sample. Most of the attrition from the Medicare sample occurred at baseline--11.4 percent of the screen sample overall were excluded from the Medicare sample because they did not complete a baseline interview, while an additional 0.8 percent were dropped because Medicare entitlement could not be verified or Medicare records were not obtained. Of those who attrited at baseline, 20 percent died before the interview could be conducted, and the rest either refused to complete the interview or could not be reached. Similar attrition rates occurred in the 18-month Medicare sample.

As noted above, the overall attrition rates were substantially higher in the control group than in the treatment group, especially in the financial contol model. These higher attrition rates among control group members are attributable to their higher baseline nonresponse rates. Breakdowns of the reasons for baseline nonresponse, presented later, show that controls were much more likely than treatments to refuse to participate in the baseline interview. This higher rate of refusal for controls is due to the incentives of the treatment group to respond (completion of the baseline was a requirement to receive channeling services) and to the fact that some control members were angry about being excluded from receiving channeling services.¹¹

The Nursing Home Sample. Attrition rates from these samples are relatively high, at 27 percent, 25 percent, and 29 percent for the 6-, 12-, and 18-month samples, respectively. As shown in Table II.2a and Table II.2b, the set of cases omitted from a particular nursing home sample includes all of those omitted from the Medicare sample (for the reasons given above), as well as two other groups of approximately equal size. Between 6 and 8 percent of the screen sample members were omitted from the analysis samples because they died during a period, but were not Medicaid-covered throughout the period and did not have a caregiver interview. Another 6 to 11 percent, approximately, were alive throughout a given six month period, but were excluded from the analysis sample because they did not complete the followup and did not have Medicaid coverage throughout the period.

Differences in rates of attrition from the nursing home samples by treatment status and by model exhibit patterns similar to those observed in the Medicare samples. In particular: (1) attrition rates are much higher among control group members than among treatment group members in both models; (2) control group attrition rates are similar across models; and (3) treatment group attrition rates are higher in the basic sites than in the financial control sites. When categories of attrition are considered, it becomes obvious that most of the differences between treatments and controls occurred as attrition from the Medicare samples for the reasons described above. Treatment/control differences in the remaining two categories are small, and differ little across models.

The Followup Samples. As seen in Table II.2a and Table II.2b, attrition from these samples is quite high, at about 34 percent, 43 percent, and 56 percent overall for the 6-, 12-, and 18-month samples, respectively. Sample members who failed to complete followup interviews include those who were nonrespondents at baseline (as described above and therefore ineligible for followup interviews), those who died after completing the baseline interview but before the followup interview could be attempted, and others who failed to respond to the interview because they refused to complete the interview or could not be reached. For the 6-month interview, about 11 percent of the screen sample were eliminated because they did not complete a baseline, an additional 14 percent had died by the 6-month anniversary, and about 9 percent did not respond for other reasons. Death accounts for the increasing attrition rates over time--about 23 percent died before the 12-month followup and about 31 percent of the 18-month cohort were deceased before the 18 month interview could be conducted. The proportion of the sample that dropped out for reasons other than death was fairly similar over time, although a greater proportion was excluded from the 18-month sample because of failure to complete one or both of the earlier followups.

Again, the overall attrition rates are higher for control group members than the treatment group members, and these differences are greater in the financial control model than in the basic model. Differential baseline nonresponse again accounts almost entirely for the treatment/control differences in the proportion of the samples with incomplete followups.

The In-Community Samples. Finally, rates of attrition from the in-community samples are broken down by reason, and displayed in Table II.2c. In addition to attrition due to lack of a completed followup, an additional 10 to 12 percent of the full sample were lost to analysis because the sample member was in a hospital or nursing home. This proportion of the sample who responded but were not in the community on their anniversary date was very similar across models, experimental groups, and time periods, especially for the 6 and 12 month periods.

2. Reasons for Interview Nonresponse

The previous discussion indicated that nonresponse to the interviews, especially the baseline, was the primary reason for treatment/control differences in the proportion of the full sample that was available for analysis in any area. To understand these differences in nonresponse, Table II.3 disaggregates the nonresponse category by the reasons for nonresponse, again by model and by treatment status. From this table we learn that control group members are considerably more likely than treatment group members to refuse the baseline interview in both models.¹² In none of the followup interviews do we find such large treatment/control differences in reasons for nonresponse as in the baseline. As expected, death accounts for most of the nonresponse at each of the followup interviews for all groups, ranging from 12.3 percent of the sample at 6 months to 34.8 percent at 18 months.¹³

C. Treatment/Control Group Differences in Characteristics in the Analysis Samples

As explained in Section A, the experimental design of the evaluation ensured that, subject to chance variation, the treatment and control groups were initially made up of individuals with similar (measured and unmeasured) characteristics. In order to be able to interpret treatment/control differences in outcomes as valid estimates of channeling's impact, it is important to ensure that the initial similarity is not undone by the effects of attrition. In an earlier report (Brown and Harrigan, 1983) the similarity of the treatment and control groups at randomization was confirmed by using data collected at the screen to compare the characteristics of the two groups. That analysis is extended in this chapter by estimating treatment/control differences on screen characteristics for the samples available for analysis at 6, 12, and 18 months after randomization. Because there are so many analysis samples and because the treatment/control differences in the proportion of the sample with available data for any analysis appear to be driven by the treatment/control differences in interview nonresponse, we confine our investigation of initial differences to the followup samples.

Treatment/control differences in the following screen characteristics are presented in Table II.4a (basic sites) and Table II.4b (financial control sites):

• Impairments on activities of daily living; incontinence
• How referred to channeling (by hospital or nursing home, home health agency, other)
• Ethnicity (black, hispanic, white)
• Sex
• Age
• Cognitive impairment (severe, moderate, mild/none)
• Interviewer-assessed unmet needs (low, medium, high)
• Whether sample member has Medicaid coverage
• Whether a proxy completed or helped complete the'screen
• Whether regular help received with
  • meal preparation
  • housework or shopping
  • taking medicine
  • medical treatments at home
  • personal care
• Income (0-500, 501-1000, over 1000 dollars per month)
• Whether on waiting list or applied for nursing home
• Number of contacts required to obtain screen interview
• Number of missing items on the screen
• Whether expect help will be needed to complete the baseline and followup interviews
• Living arrangement (with child, with spouse but not with child, with other, or alone)

Some item nonresponse occurred for the screen variables. To some extent, item nonresponse was minimized, where appropriate, by imputing the baseline value of the measurement of the same characteristic. Remaining item nonresponse was dealt with by imputing the mean of the nonmissing values if only little nonresponse existed, or including a separate nonresponse category if item nonresponse exceeded 5 percent.

The numbers of primary interest in each of these tables are the treatment/control differences, estimated by regression to control for the different distribution of treatment and control groups across sites.¹⁴ The (unadjusted) treatment group means are also given as a reference point, and can be used to obtain a profile of those who remain in the samples over time.

Some differences between treatment and control groups will occur by chance; hence, we concentrate on those for which the probability that a difference of the size observed would occur by chance is less than 5 percent. In each of the tables the first column gives treatment/control differences for the full research sample, which are comparable--except for a slightly different sample definition--to those presented in Brown and Harrigan (1983).

The overall picture presented by Table II.4a and Table II.4b is that the differences between treatment and control groups are quite small, either before or after attrition, and very few are statistically significant. In the basic sites, we note significant differences in three of the 17 variables examined--ethnicity (at 6 and 12 months), number of missing items on the screen (at 6 and 12 months), and living arrangement (at 18 months only). However, the difference in missing items on the screen existed at randomization and thus is not attributable to treatment/control differences in attrition. Continence, on the other hand, was significantly different for the two groups at randomization but not at followup. In the financial control model, however, treatment/control differences in referral source (at 18 months) and income (at 6 and 12 months) widened and became statistically significant. Thus, of the 51 comparisons presented for each model (17 variables at three points in time), only 3 in each model--about what would be expected to occur by chance--were statistically significant that were not explainable by significant differences at the time of randomization. This is about the number of such differences that might be expected to occur by chance in this number of tests.

In addition to the fact that there were few instances of statistically significant differences, the types of variables for which they were found and the lack of pattern in these results increase the belief that attrition did not lead to serious differences between the two groups. The lack of difference between the two groups on ADL, unmet needs, and other indicators of impairment increases our confidence that the two groups do not differ on unobserved dimensions of health status or other .factors that are related to outcomes of interest. The two significant differences which appear only at 18 months seem to be more happenstance than systematic differences in attrition patterns.

Overall, the pattern and magnitude of treatment/control differences on screen characteristics for the followup samples at 6, 12, and 18 months lead us to believe that attrition did not result in treatment and control groups that differ substantially on observed screen characteristics. However, this does not ensure that estimates of channeling impacts are not biased by attrition. The conditions which lead to bias is the topic of the next chapter.

III. HOW ATTRITION CAN LEAD TO BIAS AND A STATISTICAL PROCEDURE FOR ELIMINATING THE BIAS

In the previous chapter we found that attrition produced no systematic pattern of treatment/control differences in the observed, initial characteristics of the samples available for analysis. This finding does not rule out the possibility that at the time of follow up the two groups differ on unmeasured characteristics that also affect the level of a particular outcome measure. If, as a consequence of attrition, the treatment and control groups differ on average on these unmeasured characteristics, then impact estimates that do not control for these differences will be biased.

In this chapter we outline, first informally and later using statistical notation, the conditions under which attrition bias may arise. Next, we outline a statistical procedure, due to Heckman (1976, 1979) that corrects for possible bias. Finally, we show how the direction of the bias can be determined when some prior knowledge exists concerning the mechanism causing attrition.

We focus here on giving a heuristic explanation of the procedures. More complete coverage of statistical details and derivations of these methods and their justification is given in the references listed at the end of this report.

A. How Attrition Bias Occurs

As indicated in the previous chapter, differences between treatment and control groups on screen characteristics that arise because of different patterns of attrition do not necessarily imply that estimates of channeling impacts are biased. If nonresponse was affected only by screen characteristics (e.g., ADL), then inclusion of these screen characteristics as explanatory (auxiliary control) variables in the outcome regression would control for the effects of attrition. This fact is not widely understood. Two conditions are necessary for impact estimates to be biased: (1) attrition is affected by the outcome measure being examined (e.g., whether in a nursing home), or by some unobserved factor (e.g., health status at the time of the attempted followup) that affects both attrition and the outcome measure, and (2) the pattern of attrition differs for treatment and control groups. Thus, finding attrition-induced differences between treatment and control groups on observed characteristics would have implied two things. First, impacts would have to be estimated by regression controlling for all initial characteristics on which treatments and controls differ as a consequence of attrition. Second, attrition-induced differences on observed characteristics would raise the suspicion that treatments and controls differ on unmeasured characteristics as well.

To see how different attrition patterns affect impact estimates, consider the following example. Suppose that in the full sample, 20 percent of the treatment group and 30 percent of the control group are in a nursing home at the 6-month followup. Thus, channeling reduced the probability of being in a nursing home by 10 percentage points. However, contrast the effects on these results under two different assumptions about the mechanism governing attrition. In the first case, assume that attrition was random within each experimental group--that is, attrition was affected only by experimental status: controls have a 70 percent probability of response and treatments have an 80 percent probability. Clearly, restricting the analysis to just the responders would have no effect on the impact estimate: since all treatment group members have the same probability of response, the proportion of responders in nursing homes is the same as the proportion for the full sample (20 percent), and similarly 30 percent of the controls who respond to the interview will be in nursing homes. Hence, the impact estimate of 10 percentage points is unaffected by attrition, even though the attrition rates are different for treatments and controls. Furthermore, although we do not show this here, the same results hold if attrition is affected by any screen characteristic that is controlled for.

Consider how our conclusions about the effects of attrition change, however, if the probability of response is also affected by the value of the outcome being examined. For example, suppose that the probability of response among those who are in a nursing home at followup (Y=I) is only 50 percent for treatment group members and 40 percent for control group members. Suppose that the probabilities of response for those not in nursing homes (Y=0) are 95 percent for the treatment group and 85 percent for the control group (i.e., for each value of Y, treatment group members are 10 percentage points more likely to respond than controls). In this case, the treatment and control group means for the responders only (R=1) are:¹⁵

Note that the proportion of responders in nursing homes is much smaller than the values that would have been obtained if no attrition occurred (about half as large). More important, however, we see that subtracting the control group mean from the treatment group mean for respondents gives a predicted impact of channeling on nursing home placement of -5.2 percentage points, about half the true impact in this example.

These overly simplified examples demonstrate how different mechanisms of attrition may or may not cause bias in impact estimates. Overall attrition rates for treatment and control groups in the second example are 14 and 28.5 percent, respectively--quite close to the rates actually observed for some of our analysis samples. An attrition mechanism of this type could result in estimated impacts that are too small to be statistically significant, for cases in which the estimate for the full sample would have been large and highly significant.

The statistical correction procedure described in this chapter controls for attrition by determining whether responding sample members who have a relatively low predicted probability of remaining in the sample (given their screen characteristics) are more likely to have a larger (or smaller) than expected value of the outcome (Y), given the values of auxiliary control variables. This is determined by obtaining for each observation a predicted "attrition-correction" term and including it in the regression equation used to estimate channeling impacts. If this type of correlation between unobserved factors in the attrition equation and unobserved factors in the outcome equation does exist: the coefficient on the correction term will have a significant coefficient. If, in addition, attrition is affected by experimental status, the estimated channeling impact will change substantially. The statistical procedure is described below.

B. A Joint Model of Impacts and Attrition

Regression analysis was used throughout the evaluation of channeling impacts to eliminate potential bias in impact estimates that could arise due to treatment/control differences on observed characteristics at baseline, to control for the different distribution of treatment and control groups across sites, and to provide more efficient impact estimates than a simple treatment/control group comparison of means would yield. The regression model used to estimate channeling's impacts can be described as follows. Let Y₁ be the outcome of interest, such as number of hospital days or number of nursing home days. Define T_B = 1 if the sample member belongs to the treatment group in the basic case management model, and T_B = 0 if he or she does not. Similarly, define T_F = 1 if the sample member belongs to the treatment group in the financial control model, and T_F = 0 otherwise. Finally, define a set of auxiliary control variables (X₁) such as site, sex, race, income, and impairment in functioning. These variables are included in the outcome equation to control for preexisting differences between sample members on characteristics that affect the value of the outcome. The model is then

where a_B and a_F are the estimates of the impact of channeling on the .outcome, Y₁, for the basic and financial control models, respectively; b₁ is a vector of coefficients on the auxiliary variables; and u₁ is the disturbance term capturing all of the unobserved factors which influence Y₁. To facilitate the exposition below, this equation is rewritten in terms of Z and b, where Z is a vector that contains variables T_B, T_F, and X₁, and b is the true, unobserved value of the regression parameters (a_B, a_F, and b₁) in equation (1). In the absence of sample attrition, if the random assignment to treatment or control groups was performed correctly and the usual assumptions of least-squares regression are satisfied,¹⁶ then regression estimates of a_B and a_F are unbiased estimates of the impacts of channeling on outcome Y₁ for the basic case management and the financial control models, respectively.

As noted above, however, we are not able to estimate this model on the full sample because of attrition. To the extent that the included auxiliary control variables (X₁) account fully for the effect of any differences between responders and nonresponders on the outcome variable (Y₁), the estimated coefficients in equation (1), including a_B and a_F, remain unbiased. However, if there are unmeasured characteristics that affect both the probability of attrition and the outcome of interest, the estimated coefficients in equation (1) will in general be biased.

The following exposition describes the mechanism by which this bias occurs. Suppose that the attrition process can be described by the equations:

The dependent variable in equation (2) is an unobserved continuous variable, Y₂*, representing the sample member's propensity to respond to the interviews (or otherwise be included in a particular analysis sample). Each sample member has his or her own tendency to cooperate with the research and refuses when the perceived effort to respond exceeds a “tolerance.” This tendency to respond is not observed directly, but individuals with values exceeding a constant--without loss of generality assumed to be zero--are observed to respond (Y₂ = 1), while those with values less than or equal to zero are nonresponders (Y₂ = 0). Propensity to respond is assumed to be a function of observable characteristics, X₂ (which includes treatment status and may include other variables also included in X₁), as well as unobservable characteristics and circumstances, represented by the disturbance term u₂, assumed to follow a standard normal distribution.¹⁷

Bias arises in the estimates of a_B and a_F if the unobserved factors affecting attrition (u₂) are correlated with the unobserved factors (u₁) that affect the outcome measure (Y₁). This can be seen by examining the general expression for the vector of regression coefficients for equation (1), which we will refer to as b~:

Without sample attrition, the expected value of the estimated regression coefficients is the true value of the parameters (b), because the last term in the expression above has an expected value of zero. With attrition, however, the expected value of b~, given that it is estimated on only observations in the analysis sample is:

If u₁ and u₂ are correlated (i.e., if there are unobserved factors that affect both Y₁, and the probability of attrition), the expected value of the final expression in square brackets will not be zero, and therefore the expected value of the regression estimates of the parameters of equation (1), including the expected value of the estimates of a_B and a_F, will not be equal to the true values of these parameters. Thus, the estimates are biased by sample attrition, and the size and direction of the bias are unknown.¹⁸

The nature of this bias and a procedure for correcting it were expounded by Heckman (1976, 1979). Heckman showed that the bias due to sample attrition is analogous to the bias due to omitting an important explanatory variable. That is, we have

As noted above, one of the assumptions of least squares regression is that the expected value of u₁ is zero, so estimates of b will be unbiased. However, when sample attrition exists, the regression can be estimated only on those sample members with complete data, so unbiasedness of the resulting estimates requires that the expected value of u₁, conditional upon the sample members' availability for analysis, be equal to zero. If u₁ and u₂ are correlated, however, this conditional expectation of u₁ is not zero but is a function of u₂ and X₂. In this case, if Y₁ is regressed on Z, and there is correlation between the variables in Z and those in X₂, regression estimates of b will be biased because an "omitted" term (the nonzero conditional expected value of u₁) is correlated with the regressors Z. The estimated coefficients on the variables in Z, including those on treatment status, will reflect not only the effect of Z on Y₁, but also the relationship between Z and the conditional expectation of u₁.

In this evaluation attrition could lead to bias in estimates of channeling impacts because those conditions that lead to bias may well be present. For example, suppose that the sample members who are the most impaired at followup are least likely to respond and also likely to have systematically higher (or lower) values of Y₁ (e.g., hospital days). Since the auxiliary control variables measured at screen or baseline do not fully reflect impairment levels at the time of followup, u₁ and u₂ will be correlated. Furthermore, many of the variables Z and X₂ that affect the outcome and the likelihood of attrition, respectively, are likely to be the same or to be highly correlated (e.g., both the outcome and likelihood of attrition may be affected by treatment/control status). Thus, there is a strong possibility that the two conditions that together produce biased estimates of regression parameters may be present and, therefore, that estimates of channeling impacts will be biased by attrition.

Fortunately, with an additional assumption, a statistical correction for attrition bias is possible. Heckman showed that although the second term on the right-hand side of equation (6) is unobserved, the n term has a relatively simple form if u₁ and u₂ are assumed to have a bivariate normal distribution, and this term can be estimated. Heckman shows that

(7)	E(u1 | u2 > -X2b2	=	12		f(X2b2 / 2)
			2		F(X2b2 / 2)
		=	(12 / 2)M

where ₁₂ is the covariance of u₁ and u₂, ₂ is the standard deviation of u₂, b₂ is the vector of the estimated coefficients from the attrition equation, f(X₂b₂ / ₂) is the standard normal density function evaluated at X₂b₂ / ₂, and F(X₂b₂ / ₂) is the standard normal distribution function evaluated at the same point. If the parameters b₂ of the attrition equation were known, the term M could be constructed for each sample member and used as an additional variable in the regression model. Inclusion of this variable in this regression eliminates it from the error term and therefore eliminates the correlation between Z and the error term in equation (6), thereby eliminating the (asymptotic) attrition bias in estimates of b. The regression coefficient obtained on this M term is an estimate of ₁₂ / ₂, the (normalized) covariance between u₁ and u₂.

The parameters b₂ are not known, but can be readily estimated. Thus, the procedure developed by Heckman and used in this report to eliminate attrition bias can be described as follows:

1. Using all observations (both responders and nonresponders), estimate the parameters of the attrition model given in equations (2) and (3) using maximum likelihood probit.¹⁹

2. From the estimated probit coefficients (b₂) and the data on X₂, form the correction term (M) for the observations which have valid data for the outcome regression--this excludes those lost due to attrition--and estimate equation (8) by least squares:²⁰

where this equation is simply equation (1) with the nonzero conditional expectation (cM) of the old disturbance term plus a new disturbance term (u₁*) substituted for the old disturbance term (u₁). The statistical significance of c, the coefficient on M, is an indication of whether there are unobserved factors affecting both attrition and Y₁, a necessary condition for the estimates of a_B and a_F to be biased.

In the discussion of results in the next section, we assess the extent of attrition bias in estimates of channeling impacts in two ways: first, by examining the estimate of c, to determine whether the condition necessary for bias is met, and if so, the size and sign of the correlation between the two disturbance terms; and second, by comparing the regression estimates of a_B and a_F obtained when potential attrition bias is not controlled for (i.e., from estimating equation (1)) to the impact estimates obtained when this potential bias is controlled for (by estimating equation (8)). In interpreting these results, it is useful to bear in mind the determinants of the bias in a particular coefficient. Inserting the expression in equation (7) into equation (5), the bias in the uncorrected estimates of a_B, a_F, and b₁ is shown to be

(9)	bias ≡ E(b~)-b	=	(12 / 2) (Z Z) -1Z M
		=	(12 / 2) PZ,M,

where the term P_Z,M is a vector of auxiliary regression coefficients obtained from regressing the constructed M term on the other variables (Z's) in equation (8).²¹ Thus, the bias in the regression coefficient on any particular explanatory variable (e.g., a_B, the coefficient on T_B in the outcome equation) is equal to the covariance between u₁ and u₂ (normalized by ₂), multiplied by the coefficient on this same variable (e.g., T_B) from a second, auxiliary regression of the constructed M variable on all of the Z variables.

The usefulness of this expression is best demonstrated by elaborating on our previous example. Suppose that we are interested in estimating the impacts of channeling on the number of hospital days (using a followup sample). Also, suppose that those who are most impaired at the time of the followup are less likely to be available for analysis than are less impaired sample members and that the effects of this impairment on hospital days is imperfectly controlled for with the baseline control variables. Since the most impaired individuals are most likely to be in a hospital and least likely to be in the analysis sample, the covariance between u₁ and u₂ (₁₂) will be negative. Furthermore, since treatment group members are more likely to be available for analysis than control group members, it can be shown that the auxiliary regression coefficient of treatment status contained in P_Z,M is expected to be negative.²² Thus, we would expect the attrition bias in the estimate of a_B to be positive. That is, the estimated impact will be a larger number than it should be. Thus, we could find an estimated impact of zero when in fact the impact was negative, implying a reduction in hospital days due to channeling. This analytic assessment of the direction of bias is consistent with the heuristic argument that the sample members most likely to be lost to analysis are control group members with relatively large numbers of hospital days, and if these cases were appropriately represented in the analysis sample, the treatment/control difference in expected hospital days would have been a larger negative number. Based on this reasoning, the following reference table can be used to draw inferences about the expected direction of the bias (if any) due to attrition in estimates of channeling impacts:

Expected Relationship (12) Between Outcome (Y1) and the Likelihood that Sample Member is Available For Analysis	Expected Bias1 in Estimated Impacts	Interpretation
0	0	Impact estimates unbiased
+	-	Impacts understated if channeling is predicted to increase Y (aB, aF positive); impact overstated if channeling is predicted to decrease Y1 (aB, aF negative)2
-	+	impacts overstated if channeling is predicted to increase Y1 (aB, aF positive); impacts understated if channeling is predicted to decrease Y1 (aB, aF negative)2
Using equation (9), knowledge of the sign of 12 is sufficient to determine the expected direction of the bias. This results from the fact that we know that response rates are higher for the treatment group, and therefore, that the coefficients on TB and TF in the auxiliary regression of M on TB, TF, and X2 are negative. This interpretation may be confusing for impacts that are expected to be negative in sign, such as impacts on hospital or nursing home use. For example, if the bias is expected to be positive, this means the estimate is too large, i.e., it should be a larger negative number (e.g., -6) rather than a smaller negative number (e.g., -2). Thus, the reduction due to channeling is understated. On the other hand, if the estimated impact is expected to be positive in sign, such as impacts on case management, and the bias is positive, the impact of channeling is overstated because of attrition bias. Thus, the expected direction of channeling impacts as well as the expected direction of the bias are required in order to draw the desired inferences about the effects of attrition on channeling impacts.

Using this table for our example, we expect ₁₂ to be negative because those who are most impaired are likely to have more hospital days, but are less likely to respond. Thus, the expected bias in the impact estimate is positive, and since channeling is predicted to reduce hospital days, the estimated reduction in hospital days will be understated if attrition bias is not corrected for.

IV. USE OF MEDICARE DATA TO ESTIMATE THE EFFECTS OF ATTRITION ON IMPACT ESTIMATES

The previous chapter described the types of attrition that could lead to bias in impact estimates, and a statistical method for measuring and eliminating this bias. Before implementing these statistical procedures, however, we take advantage of the fact that Medicare data are available for virtually the entire sample to determine directly how restriction of the analysis to a subset of the full research sample affects some estimates of channeling impacts.

A. How Medicare Data Were Used to Examine Attrition Bias

As discussed in the last section, the analysis samples used to estimate channeling's impacts on various outcomes were restricted to those sample members for whom the necessary data were available. These restrictions in data availability were due to nonresponse at the baseline or followup interviews, lack of information about Medicare or Medicaid eligibility, and in the cases of the nursing home samples and followup samples, lack of data due to death. Although various sources of data are available for only certain subsamples of the full screen sample for these reasons, Medicare data were obtained for about 97.5 percent of the full sample. (Medicare eligibility was indeterminate or inadvertent errors were made in claims requests for 2.5 percent of the screen sample members.) Thus, we are able to use these Medicare data to compare estimates of channeling's impacts on Medicare-covered services obtained on the full sample with impacts on these outcomes estimated on the more restrictived analysis samples. Major differences in impact estimates between the full sample and the analysis samples on these variables would suggest that impact estimates for other outcome variables that are obtained on these analysis samples also are likely to be biased because of attrition. On the other hand, if estimates of impacts on these Medicare-covered outcomes were similar for the full and analysis samples, we would be more confident (though not certain) that there is no bias due to attrition in estimates of impacts on other outcomes (specifically, those that can be estimated only on the various analysis samples).

In this analysis we compare impact estimates between the screen sample and the analysis samples on 11 outcome variables. The following 6 variables were constructed solely from Medicare claims data:

• Number of days spent in hospital under Medicare
• Number of days spent in nursing home under Medicare
• Hospital expenditures reimbursed by Medicare
• Nursing home expenditures reimbursed by Medicare
• Physician Medicare reimbursements
• Medicare reimbursements for other medical services (e.g., outpatient services, lab tests, x-rays)

In addition, 5 other variables were examined that required adding data from the Financial Control System (FCS) on use of certain services to that reflected in the Medicare data. The FCS data contain claims for services used by treatment group members in the financial control model, and are necessary for this analysis because, due to the pooling of Medicare and channeling funds for the provision of certain services covered by Medicare, Medicare claims were not filed for these services when arranged for by channeling in financial control sites. These additional variables are:

• Number of skilled nursing visits under Medicare or channeling
• Number of home health aide visits under Medicare or channeling
• Skilled nursing reimbursements under Medicare or channeling
• Home health aide reimbursements under Medicare or channeling
• Total reimbursements by Medicare or channeling for skilled nursing, therapy, home health aides, special equipment and supplies

Because of the need to use FCS data, these variables are defined slightly differently for treatments in financial control sites than for other sample members. However, there is no reason to expect that inclusion of the FCS data should affect estimated impacts for the analysis samples differently from the way it affects estimates for the full sample.²³ All outcomes are defined for months 1 to 6, 7 to 12, and, for the 18-month cohort, months 13 to 18.

Because they are restricted to what is available from Medicare claims, these 11 variables each capture only a portion of the total use of that service, which is the primary focus of the evaluation. Nonetheless, since the variables are available for attriters as well as nonattriters, they can provide valuable clues as to whether attrition is likely to be a problem for the analysis of total use of these services, and for other outcomes as well. This is especially true for hospital use, nearly all of which is covered by Medicare for the sample members in the channeling evaluation. We estimate that over 98 percent of total hospital days used by the sample are reflected in the Medicare claims. Thus, the comparisons presented below between impacts on Medicare-covered hospital days and expenditures estimated on the full sample to those estimated on the Medicare samples provide very good evidence of whether impacts on total hospital use are biased by restricting the analysis to the Medicare sample.

For other outcomes, the Medicare/FCS data offer less complete coverage of total use. The nursing home use and expenditures covered by Medicare is only a small fraction of total nursing home use and the variables on use of skilled nursing and home health aides do not cover all formal services used by sample members. Furthermore, there are no variables obtainable from Medicare records (and therefore available for the full sample) that pertain to sample member well being, receipt of case management, and receipt of informal services. For these outcomes, as well as for others obtained from the followup interviews, we will also rely on the statistical procedures identified in Chapter III to determine whether attrition bias exists. Similar impacts for the full and analysis samples on the Medicare outcomes presented here, however, would provide some evidence that attrition bias is not a problem in these samples for any outcomes.

Since not all members of the full screen sample completed a baseline interview, the standard control variables used in all the final impact analyses have been replaced by a parallel set of variables based on the screen interview only. These variables, essentially the ones examined in Table II.4a and Table II.4b are described in more detail in Chapter V of this report.

B. Comparison of Estimated Impacts for the Full Sample to Those for the Analysis Samples

Table IV.1 and Table IV.2 present the estimated impacts and control group means on six month Medicare outcomes for the full and analysis samples for the basic and financial control models, respectively. Table IV.3, Table IV.4, Table IV.5 and Table IV.6 contain similar results for the 7 to 12 and 13 to 18 month periods. The evidence of bias due to attrition is reviewed separately for each of the analysis samples.

1. The Medicare Sample

The Medicare sample, used to estimate impacts on hospital outcomes, included'88 percent of the full sample, as shown in Table II.1 at 6 and 12 months and the same fraction of the 18 month cohort at 18 months. Comparison of the impact estimates on hospital days and expenditures for the full and Medicare samples shows little difference, either between control group means or impacts, in any time period. In no case is the impact estimate statistically significant at the .05 level, in either sample. The full sample estimate of impacts on hospital days at 6 months in the financial control model (-1.54 days) is nearly significant, whereas the corresponding sample estimate is somewhat smaller (-1.15) and statistically insignificant. However, the relatively small size of this treatment/control difference (less than 10 percent of the control group mean) and the lack of significant impacts on expenditures or on days in other time periods or for the other (basic) model suggests that the estimates of hospital impacts are not biased by attrition from the Medicare sample.