Methodological Issues in the Evaluation of the National Long Term Care Demonstration. A. The Equivalence of Treatment and Control Groups at Randomization


Due to the random assignment of eligible channeling applicants, the control and treatment groups should be composed of individuals that on average were very similar at the time of application on any observed or unobserved characteristic. Hence, the control group should yield reliable estimates of what would have happened to clients in the absence of channeling, and comparison of outcomes for the treatment and control groups therefore should yield reliable estimates of channeling impacts.

Only two factors (other than measurement error) could cause the mean values of the pre-application characteristics of the full treatment and control groups to differ: deviation from the randomization procedures and normal sampling variability. Deviations from the carefully developed randomization procedures could be either deliberate (e.g., intake workers purposely misrecording as treatments some applicants who were randomly assigned to the control group, but who had especially pressing needs for assistance) or accidental. The dedication and professionalism of the channeling program staff at each site and the safeguards built into the assignment procedure made either occurrence very unlikely. Site staff were extremely cooperative in faithfully executing the procedures. (See Phillips et al., 1986, for details of the randomization procedures.)

Sampling variability, on the other band, is the difference between the two groups that occurs simply by chance. For the sample sizes available at the model level, such differences between the two groups should be very small, and are expected to be statistically insignificant.

Despite the expected small, chance differences between the two groups, the implications of large chance differences for estimates of program impacts was so great that it was necessary to verify that in fact the two groups were comparable. This assessment was carried out by comparing mean values of screen characteristics for the treatment and control groups in each model, adjusting for the unequal distribution of the two groups across sites. The following screen characteristics were examined:

  • Demographic: age, sex, ethnic background.

  • Financial Resources: monthly income, types of insurance coverage.

  • Living Arrangement: proportion in long-term care institution; proportion living alone, with spouse, with others, or with spouse and others.

  • Health and Functioning (see below): activities of daily living (ADL) index, cognitive impairments affecting functioning, unmet needs for service.

  • Help Received: whether help was received in the areas of meal preparation, housework or shopping, taking medicine, medical treatments at home, and personal care; expected lack of sufficient support from family and friends in coming months (fragile informal supports).

  • Referral Source: whether referred to channeling by family, by a hospital, by a home health agency, etc.

  • Nursing Home Application: whether had applied for admission to nursing home or were on a nursing home waiting list at screen.

Estimates of the differences between the treatment and control groups were obtained by regressing the screen characteristics on two binary variables representing treatment status (one each for basic and financial control models) and 10 binary site variables. The coefficients on the treatment variables provided the estimates of the treatment/control differences in means, controlling for the different distribution of the two groups across sites. Estimates of the treatment/control differences in means of these variables at each site were also examined. Both the model and site level differences were tested to determine whether they were larger than could reasonably be expected to occur because of chance sample variation.

This analysis, presented in detail in Brown and Harrigan (1983), showed that there were very few variables for which treatment/control differences were statistically significant. Of the 53 screen variables examined for each model, there was only one characteristic for which differences were statistically significant in the basic model and four in the financial control model. Furthermore, even the significant differences were small in magnitude (three percentage points or less for binary variables) and (with one exception) occurred for characteristics possessed by less than seven percent of the sample. Treatment/control comparisons at the site level yielded similar conclusions: the number of statistically significant differences was no larger than would be expected by chance and no patterns of differences were found to indicate that noncomparable groups were obtained in any site.

Thus, although there may be unobserved differences between the two groups, the comparisons an observed characteristics provided no evidence of either systematic deviations from the random assignment procedures or important treatment/control differences arising by chance. We concluded that the control group provided a reliable measure of what would have happened to the treatment group in the absence of channeling, and therefore, simple comparisons of outcomes for treatment and control groups (controlling for differences in distribution across sites) should yield unbiased estimates of channeling impacts.

View full report


"methodes.pdf" (pdf, 2.16Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®