Methodological Issues in the Evaluation of the National Long Term Care Demonstration. V. CONCLUSION

04/01/1986

A great deal of care went into the design of the sample and estimation procedures used in the channeling evaluation to ensure that the validity of the results would not be subject to doubt. There were a few uncertainties inherent in the design of the demonstration that lead to questions about the generalizability of the results, including:

  • Whether the sites in which the demonstration was conducted were representative (they were not randomly selected, were heavily concentrated in the eastern part of the country, and appeared to have somewhat fewer nursing home beds and more community services available than most areas)
  • Whether the sample members were representative of the population at high risk of institutionalization (sample members may have been more strongly opposed to living in nursing homes or may have had better informal support systems than comparably impaired community residents who did not apply to channeling)
  • Whether the limited duration of the demonstration affected either the types of individuals who applied or were referred to channeling or the effectiveness of the program
  • Whether following sample members for only 12 or 18 months was too short a period to observe the effects of channeling (channeling impacts on outcomes such as institutionalization may only become apparent only over a period of years)

We do not believe that the limited duration of the demonstration or the shortness of the followup period led to misleading inferences about channeling impacts. At the time they were referred or chose to apply to channeling, many of the sample members were at a critical point where they needed assistance, either because they had just left or were about to leave a hospital or nursing home, or because they had an informal support system that was inadequate for their current needs or in danger of collapse. Furthermore, a large fraction of sample members (about 40 percent) were admitted to a hospital during the first 6 months after randomization. These are precisely the circumstances in which the case management and services provided by channeling were expected to benefit clients. Thus, the argument that the effects of the program were unlikely to be observed during the 12 or 18 months immediately after entering the program has little credence. Nor is it likely that individuals who potentially would have benefitted greatly from participation in channeling neither applied nor were referred to channeling because it was only guaranteed to last a few years.

The basic design questions of the representativeness of the sites and applicants, on the other hand, are potentially more serious. However, they cannot be answered with the available data. Whether program impacts would be different in different environments or for different types of individuals than those analyzed are questions that can rarely be answered in any evaluation.

The potential effect of the local environment on channeling impacts raises another important limitation of the design: comparison of impacts for the basic and financial control models confound such environmental effects with effects of differences between the two models. Since one of the goals of the evaluation was to determine whether the additional resources and control over funds available to case managers in the financial model led to greater impacts, this was a potentially serious shortcoming. However, because of the general lack of significant impacts for either model, the importance of this limitation is muted.

In addition to these design features, another potential limitation of the evaluation was the fact that the estimates of the effects of channeling were attenuated in two ways: first not all treatment group members actually participated in channeling, and second, some controls received case management from other agencies or sources, and many controls received community-based services. While the case management may not have been as comprehensive as that offered by channeling and the services may not have been as extensive, control group receipt of services means that the evaluation did not provide estimates of the absolute effects of the case management and services provided by channeling but rather of how the effects of channeling compare to the effects of services that were already in existence. Furthermore, estimated impacts will understate the actual program effects on participants, because the outcomes for the treatment group are averaged over all treatment group members, including the 20 percent who never had a care plan completed.

These dampening effects have two implications for the interpretation of impact estimates. First, the impacts of channeling on those who actually received the treatment were 25 percent larger than the estimated treatment/control differences (as are the average costs of providing these services). However, since relatively few of the impacts were significantly different from zero, and costs were also measured on a per treatment group member basis, this has little importance for the evaluation. Second, and more important, the general lack of statistically significant impact estimates does not mean that case management and formal services in general or the channeling program is particular have no effects on impaired elderly people. A separate report (Brown and Phillips, 1986) addresses the broader issue of the effects of case management and services per se on nursing home use.

What we have focused on in this report are issues that were known about at the beginning or that arose during the course of the evaluation and which could be mitigated by analytic methods. We have shown that given the basic design, the estimated treatment/control differences provided valid, robust estimates of channeling impacts. The random assignment of eligible sample members to treatment and control groups ensured that the comparison of the two groups provides an unbiased estimate of the difference between treatment group members' actual outcomes and what would have happened to them in the absence of the program. The large sample sizes, statistical testing procedures, and methods of piecing together evidence across time periods, models, and outcome measures provide a high degree of confidence that the evaluation neither concluded that channeling influenced some outcome when no impact actually occurred nor failed to detect important impacts that did occur.

A variety of potential threats to the validity of the results were identified and assessed including sample composition issues (whether the treatment and control groups had similar initial characteristics and whether the expected equivalence of the two groups was distorted by sample attrition), data issues (whether the differences between the two groups in who collected the baseline data led to differential measurement of those data, and whether the use of outcome data collected from proxy respondents distorted estimates of channeling impacts), and estimation issues (whether observations from all sites and from both treatment and control groups should be combined to obtain a single regression estimate of channeling impacts for each model, whether impacts of channeling were the same for the early and late cohorts of sample members, and whether regression provided robust estimates of impacts for the outcome variables that were not distributed normally). Of all these potential problems, only the noncomparability of the baseline data was determined to be likely to distort estimates of program impacts. To avoid this distortion, baseline variables judged to be noncomparably measured were excluded from use as control variables in the regression equation. (Where they existed, screen counterparts to these noncomparable baseline variables were used as substitutes.) All of the other potential problems with the data or regression estimation approach were found to have little or no actual effect on impact estimates or their interpretation, so it-was not necessary to implement special procedures broadly. The isolated cases in which there was some evidence of a potential problem for specific outcome variables were identified and examined in detail in technical reports dealing with those outcomes, and where appropriate, alternative estimates were presented.


View full report

Preview
Download

"methodes.pdf" (pdf, 2.16Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®