Methodological Issues in the Evaluation of the National Long Term Care Demonstration. B. Data Sources Used in the Analysis


The second most important feature of the evaluation design was the collection of comprehensive longitudinal data from a variety of sources on a large sample of individuals. These features increase the precision (i.e., decrease the variance) of the impact estimates-in two ways: (1) by gathering various data items from the best sources (e.g., nursing home data from Medicare, Medicaid, and provider records) the analysis variables contain less "noise" due to measurement error, and (2) the large sample sizes decrease the likelihood that observed treatment/control differences are due to chance rather than to the effects of channeling.

To conduct the evaluation, data were required on both the initial (preprogram) characteristics of the sample and on outcome variables which measure the post-randomization experience of the sample. Outcomes which channeling was expected to influence were grouped into 6 substantive areas:

  • Hospital use
  • Nursing home use
  • Use of formal community care
  • Receipt of informal care from family and friends
  • Mortality
  • Well-being of clients and their caregivers

In order to obtain the best data to address these issues, various sources were required, including both interviews with sample members and records from specific programs and agencies. These sources are described below.

1. Interview Data

Interview data4 sources include: (1) the screen interview, which was administered to all persons referred or applying to channeling to assess their eligibility for the program; (2) the baseline interview, administered to eligible sample members as soon as possible after they were assigned to the treatment or control group (average length of time between screen and baseline was about one week for treatment group members and almost two weeks for controls); and (3) the followup interviews, administered 6, 12, and 18 months after randomization in order to obtain data on outcomes which channeling was hypothesized to influence.5

The Screen. The screen questionnaire, administered primarily by telephone by channeling intake workers, was designed primarily to assess eligibility for channeling and contained data on sample members' ability to perform various activities of daily living, their unmet needs for assistance of several types, and some sociodemographic characteristics. Applicants determined to be eligible for channeling were then randomly assigned to treatment or control status by research staff. Screen interviews were completed with 6,341 eligible sample members. Unfortunately, 14 screen interviews were lost in the mail and one case assigned to the control group was erroneously allowed into the channeling program, so that there are actually 6,326 individuals who could be included in the analysis. These observations thus constitute the full research sample, and we refer to it as such throughout this report.

The Baseline. The screen interview does not, however, contain the comprehensive data that were necessary for either the evaluation or the development of a care plan for channeling clients. A thorough, in-person baseline assessment of treatment group members was required in order for program case managers to develop an appropriate care plan for participants. A single instrument was developed that would serve both the purpose of care planning and research. It was considered important that channeling staff members collect the data necessary for developing an appropriate care plan; therefore, the baseline interview (but not the followup interviews) was administered by channeling staff for the treatment group and by research interviewers for the control group.6 Treatment group members who refused the baseline assessment interview could not participate in channeling, since no care plan could be developed for them. However, since these individuals could differ substantially from other treatment group members, nonresponding members of the treatment group were interviewed by research interviewers whenever possible. This enabled us to retain them in the analysis sample and thereby helped to preserve the equivalence of the treatment and control groups. Overall, 108 (3 percent) of the baseline interviews for the treatment group were administered by research interviewers. Sample members who failed to complete baselines were not followed up and were excluded from most of the channeling analyses.

The Followup Interviews. For sample members who completed the baseline, followup interviews at 6, 12, and 18 months after randomization were attempted by research interviewers to gather the data on sample members' outcomes that were necessary to assess the impacts of channeling. Although a completed baseline was a condition for being contacted for a followup interview, a noncompleted 6-month interview did not make the sample member ineligible for a 12-month interview. Thus, some sample members who did not complete a 6-month interview did complete a 12-month interview.

The situation was different at the 18-month interview. First, to reduce the length of the data collection period and costs, only the first half of the sample members randomized were eligible for an 18-month interview.7 Second, an 18-month followup was attempted only if the sample member belonged to this first half of the sample (referred to as the 18-month cohort), and had a completed baseline, 6-month and 12-month followup interview.

2. Records Data

Records data used in the channeling evaluation included Medicare and Medicaid claims data, records data from providers of services (e.g., nursing homes) that sample members claimed in the interview to have used, financial control system data from the channeling projects (for channeling clients in financial control sites), and death records.

Medicare Claims Data. Medicare claims data were collected for all sample members who said that they were eligible for Medicare and for whom a valid Medicare identification number could be verified by HCFA. Nearly the entire sample (97 percent) was eligible for Medicare. Claims provided data on sample members' hospital use, some nursing home use, and use of other medical services and community-based services paid for by Medicare. See Wooldridge and Schore (1986) for a detailed discussion of Medicare data.

Medicaid Claims Data. Medicaid claims were collected for all sample members who said they were eligible for Medicaid at any interview and signed a consent form authorizing use of the data, if a valid Medicaid identification number could be verified by the state Medicaid program. Medicaid claims were a key source of data on nursing home outcomes and use of formal community services.

Provider Records Data. Data on the nursing home use of specific sample members were collected from nursing homes for sample members stating in an interview that they had spent time in that institution during the reference period or were living there at the time of the interview. Records data were also collected from area hospitals on those few sample members who were not on Medicare. For a random 20 percent subsample of the research sample, records were also collected from other types of service providers (e.g., home health agencies) that were named in followup interviews by sample members.8

Financial Control System Data. Because of the pooling of Medicare, Medicaid, and in some cases other government funds in the financial control model, data on use of formal community services by treatment group members in that model were obtained from the channeling project's records.

Death Records. Data on mortality were obtained from a search of state death records for all sample members who failed to complete their last scheduled interview. These data were supplemented by data on mortality obtained in the attempt to field the followup interviews and from client-tracking data (for treatment group members).

View full report


"methodes.pdf" (pdf, 2.16Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®