Final Report on the Effects of Sample Attrition on Estimates of Channeling's Impacts. A. The Demonstration and the Evaluation


Channeling2 consists of a set of seven core functions--outreach, screening, comprehensive needs assessment, care planning, service arrangement, monitoring, and reassessment--deemed necessary to rationalize service use and ultimately reduce costs and improve client well-being. To this end two models of channeling are being tested in the demonstration in 10 sites. The basic case management model adds limited funds to the core functions, giving case managers somewhat greater flexibility in designing and implementing care plans. The financial control model adds to the core functions by substantially expanding the service coverage of public programs, pooling funds from separate government programs, and allowing case managers to authorize services to be paid for from the funds pool. These are combined with a cap on average annual service expenditures per client (60 percent of the state's average reimbursement rate for intermediate and skilled nursing home care), and a limit on the annual cost of individual care plans (85 percent of the state's average nursing home reimbursement rate) that can be exceeded only with state approval. Channeling operations began in a phased startup between February and June 1982 at 10 sites, 5 implementing the basic case management model, and 5 the financial control model.

Impacts of channeling are estimated in this evaluation by statistically comparing the experiences of two groups of individuals: the treatment group, members of which were entitled to participate in channeling, and the control group, whose members were not allowed to participate. Individuals who applied or were referred to channeling and were found to be eligible were randomly assigned to one of these two groups.

To support this analysis, various data were collected on these sample members from a variety of sources, described below, on the outcomes which channeling was expected to influence. The specific outcomes examined fall into the following broad categories:

  • Nursing home use and costs
  • Hospital use and costs
  • Use and cost of formal community-based services
  • Receipt of care from informal caregivers
  • Sample members' well being
  • Mortality

Many specific variables in each of these categories were examined in the course of the evaluation for evidence of channeling impacts. Below we describe the sources of data for variables in each of these areas and the samples available for analysis.

1. The Data

The analyses of channeling impacts relies on many sources of data on sample members. The data may be classified broadly as “interview” data or “records” data. Interview data3 sources include the screen interview, which was administered to all persons referred or applying to channeling to assess their eligibility for the program; the baseline interview, administered to eligible sample members as soon as possible after they were assigned to the treatment or control group (interviews were usually completed within 2 weeks after randomization); and the followup interviews, administered 6, 12, and 18 months after randomization in order to obtain data on outcomes which channeling was hypothesized to influence.4 Records data include Medicare and Medicaid claims data, records data from providers of services (e.g., nursing homes) that sample members claimed in the interviews to have used, financial control system data (for channeling clients in financial control sites) from the channeling agencies, and death records. These data sources are described below.

The Screen. The screen questionnaire, administered primarily by telephone by channeling intake workers, was designed primarily to assess eligibility for channeling and contained data on sample members' ability to perform various activities of daily living, their unmet needs for assistance of several types, and some sociodemographic characteristics. Applicants determined to be eligible for channeling were then randomly assigned to treatment or control status by research staff. Screen interviews were completed with 6,340 eligible sample members. Unfortunately, 14 screen interviews were lost in the mail,5 so that the remaining screen sample consists of 6,326 observations. The screen sample is thus the full research sample, and we refer to it as such throughout this report.

The Baseline. The screen interview does not, however, contain the comprehensive data that were necessary for either the evaluation or the development of a care plan for channeling clients. A thorough, in-person baseline assessment of treatment group members was required in order for program case managers to develop an appropriate care plan for participants. A single instrument was developed that would serve both the purpose of care planning and research. It was considered important that channeling staff members collect the data necessary for developing an appropriate care plan; therefore, the baseline interview (but not the followup interviews) was administered by channeling staff for the treatment group and by research interviewers for the control group.6 Treatment group members who refused the baseline assessment interview could not participate in channeling, since no care plan could be developed for them. However, since these individuals could differ substantially from other treatment group members, nonresponding members of the treatment group were interviewed by research interviewers whenever possible. This enabled us to retain them in the analysis sample. Overall, 108 (3 percent) of the baseline interviews for the treatment group were administered by research interviewers.

The Followup Interviews. For sample members who completed the baseline, followup interviews at 6, 12, and 18 months after randomization were attempted by research interviewers to gather the data on sample members' outcomes that were necessary to assess the impact of channeling. Although a completed baseline was a condition for being contacted for a followup interview, a noncompleted 6-month interview did not make the sample member ineligible for a 12-month interview. Thus, some sample members who did not complete a 6-month interview did complete a 12-month interview.

The situation was different at the 18-month interview. First, because of budgetary reasons, only half of the sample members randomized were eligible for an 18-month interview.7 Second, an 18-month followup was attempted only if the sample member belonged to this 18-month cohort, and had a completed baseline, 6-month and 12-month followup interview.

Medicare Claims Data. Medicare claims data were collected for all sample members who said that they were eligible for Medicare and for whom a valid. Medicare identification number could be verified by HCFA. Nearly the entire sample (97 percent) was eligible for Medicare. Claims provided data on sample members' hospital use, some nursing home use, and use of other medical services and community-based services paid for by Medicare. See Wooldridge and Schore (forthcoming) for a detailed discussion of Medicare data.

Medicaid Claims Data. Medicaid claims were collected for all sample members who said they were eligible for Medicaid at any interview and signed a consent form, if this information and a valid Medicaid ID number could be verified by the state Medicaid program. Medicaid claims were a key source of data on nursing home outcomes and use of formal community services.

Provider Records Data. Data on the nursing home use of specific sample members were collected from nursing homes for sample members stating in an interview that they had spent time in that institution during the reference period or were living there at the time of the interview. Records data were also collected from area hospitals on those few sample members who were not on Medicare.8

Financial Control System Data. Because of the pooling of Medicare and Medicaid funds in the financial control model, data on use of formal community services by treatment group members in that model were obtained from the channeling agencies' records.

Death Records. Data on mortality were obtained from a search of state death records for all sample memers who failed to complete their last scheduled interview. These data were supplemented by data on mortality obtained in the attempt to field the followup interviews and from client-tracking data (for treatment group members).

2. The Analysis Samples

For 5 of the 6 categories of outcomes identified above, the sources and therefore the completeness of the necessary data differ. The analysis samples for each of these areas are:

  • Mortality--full research sample
  • Hospital outcomes--6, 12, and 18 month Medicare samples
  • Nursing home outcomes--6, 12, and 18 month nursing home samples
  • Well-being outcomes--6, 12, and 18 month followup samples
  • Receipt of formal community based services and informal care--6, 12, and 18 month in-community samples

These samples and the relationship between them are described below.

Full Sample. This sample includes all of the 6,326 initially randomized individuals, and was used to estimate the impacts of channeling on mortality, as measured by whether sample members were alive at 6 and 12 months after randomization. The full 18 month sample, used to estimate impacts on mortality at 18 months, includes the 3,165 members of the full sample who were in the 18 month cohort. A search of state death records was conducted for all sample members not known to be alive from the interviews, and these records data were supplemented with information on deaths obtained from attempts to field followup interviews and from channeling programs' client tracking systems. Sample members identified as dead from either source were assumed to be alive; hence, there was no missing data on mortality. An analysis of the validity of this assumption, presented in Wooldridge and Schore (forthcoming, Appendix F), makes use of Medicare claims data and updated status files to verify that the assumption is correct for virtually all sample members. Hence, there is no bias in estimates of channeling impacts on mortality.

The Medicare Sample. The Medicare sample was employed to examine channeling's impacts on the use of hospital and other medical services, and on home health expenditures reimbursed by Medicare. The Medicare sample is the subset of the 6,326 initially randomized individuals (the full sample) who completed baseline interviews and who are either known to be Medicare entitled or known not to be Medicare entitled. This sample was used for analyzing, outcomes in the first 12 months following randomization. To be consistent with the analyses of channeling's impacts on outcome measures obtained from follow-up interviews, the 18-month Medicare sample is restricted to those members of the Medicare sample who were also in the 18-month cohort.

The Nursing Home Samples. Because Medicare claims do not provide a complete history of nursing home use, the samples used for the nursing home analysis differed from those used for the hospital analysis. Most nursing home expenses are paid by Medicaid, for Medicaid-covered individuals, or by private payment, for those not covered by Medicaid. Therefore, the nursing home analysis employed a two-pronged data collection strategy, relying on Medicaid (and Medicare) records to provide complete nursing home information for sample members who were covered by Medicaid, and on provider (and Medicare) records for those who were not Medicaid-covered. However, in order to identify the relevant providers for this latter group, either a followup interview or caregiver interview had to have been completed.

These data requirements resulted in three nursing home samples, one for each six-month period. These are subsamples of the Medicare samples, and include individuals who either completed a followup interview, were Medicaid covered throughout the six-month period, or died in the period but had a caregiver who provided followup information. In addition, Medicare sample members who were dead throughout a six-month period, or who died during the period and were Medicaid-covered at the start of the period and at death were also included in the nursing home sample for that period.

The Followup Samples. The followup samples were used to analyze outcomes obtained from the followup surveys administered at 6, 12, and 18 months after randomization. The two major categories of impact analyses which relied on these samples are those dealing with sample members' well being and functional ability and those dealing with case management services. The followup sample at 6 months includes the subset of the screen sample with both a complete baseline and a complete 6-month followup interview. In like manner, the sample at 12 months is composed of screen sample members who completed a baseline and a 12 month followup (but not necessarily a 6 month interview). The 18 month sample includes only those in the early cohort who completed a baseline and followup interviews at all three time periods.

The In-Community Samples. Estimation of channeling impacts on receipt of formal and informal care required data on these outcome variables from the followup surveys. The interview data on receipt of such services pertained to the reference week--the week 6, 12, or 18 months after randomization, for sample members residing in the community during this reference week. Thus, for those who were in a hospital, in a nursing home, or deceased at this time (but alive at some time during the 6 month period), use during weeks when they were in the community is unknown.9 (Of course, sample members who had died prior to the start of the period spent no time in the community and therefore are excluded from the relevant set of observations.) Therefore, the 6, 12, and 18 month in-community samples were composed of those sample members who completed the relevant interview and were living in the community during the reference week.

These five different sample types form a hierarchy, with each being nested, or nearly so, within the one above it.10Figure II.1 shows the relationship between them.

Before proceeding to the examination of response rates, there are two points that should be noted. First, each of the analysis samples described above are used to examine channeling impacts on many outcome measures falling under the general area for which the samples were defined. Clearly, we cannot examine all of these estimates for evidence of attrition, so we have identified the key outcome measures within each substantive area and confined our analysis of attrition bias to these variables. The variables are listed in Chapter IV.

FIGURE II.1: Flowchart of Inclusion In and Attrition From Analysis Samples
Flowchart: Page 1 of 3.
Flowchart: Page 2 of 3.
Flowchart: Page 3 of 3.

The other point to be made is that an observation can be omitted from the analysis of impacts on a given outcome measure due to item nonresponse on that variable, even if the sample member is in the appropriate analysis sample. In general, there are relatively few such cases of missing data on individual outcome variables when the observation is in the appropriate analysis sample, and, of course, which observations are missing depends on the specific variable examined. Thus, the possible effect of item nonresponse on impact estimates is ignored in this report.

View full report


"atritn.pdf" (pdf, 4.74Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®