Methodological Issues in the Evaluation of the National Long Term Care Demonstration. C. Interpretation of Estimates

04/01/1986

Performing statistical tests at the .05 significance level ensures a low probability of erroneously concluding that channeling affected a given outcome when the observed treatment/control difference is actually due to chance ("type I" errors). The use of multivariate tests further decreases the probability of such errors. The discussion of the power of the statistical tests presented earlier suggested that the sample sizes were sufficiently large that with t-tests performed at the .05 significance level we could be quite confident that large channeling impacts (i.e., those of policy-relevant magnitude) would not be misclassified as due to chance. However, strict adherence to the more stringent multivariate test to reduce further the probability of type I errors means that it is more likely that we will make the opposite error--concluding that channeling had no impact when the program was truly effective (type II errors).23

Because of the desire to avoid both types of error, we do not rely solely on the hierarchical testing structure raised in the previous section, nor on any single statistic to ascertain whether channeling affected outcomes of interest. The sheer number of outcomes examined and test performed means that strict reliance on test statistics would result in a number of both type I and type II errors.

To decrease the number of such errors, throughout the analysis we looked not only at the statistical significance of the estimates but also their magnitudes and patterns. Specifically, to assess whether channeling affected a given outcome, we looked for consistency in the direction, size, and statistical significance of estimated impacts on: (1) related outcome measures, (2) the same outcome in other time periods, and (3) the same outcome in the other channeling model.

We also examined the estimated impact at the site level, to see if the model level estimate was essentially due to one or two particular sites rather than being widespread. Dependence on patterns across model, time period, and site to verify whether impacts exist cannot be rigid, since there are reasons why effects may differ across these dimensions. Nevertheless, if patterns exist they provide evidence that the observed differences were due to channeling rather than to chance.

Finally, we also drew on theory and results from the process analysis (Carcagno et al., 1986) to assess the likelihood that the estimates obtained represented real impacts rather than chance differences. It is clearly inappropriate to conclude that only those estimates with the expected sign were due to real effects of channeling and all others were due to chance. However, awareness of how outcomes interrelate and the process by which channeling was likely to bring about effects on individuals, coupled with knowledge about how the local programs operated, were useful inputs to the assessment of whether impacts existed when the statistical evidence was mixed.

The following example demonstrates this strategy of evaluating the impact estimates. Suppose that estimated channeling impacts on nursing home days in the basic model at 6 months were statistically significant, and admissions were not significant and that the multivariate test was not significant. However, suppose that the t-statistics on these other impact estimates and the multivariate test statistics were quite near the critical values necessary for the estimates to be considered significant, and the estimates were all of the expected sign. Suppose further that the estimates for the 7 to 12 month period were also of the same sign and roughly comparable in magnitude, but not significant at the .05 level. In such cases, it would seem likely that channeling did have an impact on nursing home use, although perhaps not a strong effect or perhaps an effect that was concentrated in a few sites or in clients of a certain type. The magnitude of the estimates and effects on subgroups of sample members were examined to address these possibilities. Finally, since channeling-Induced reductions in nursing home use were expected to be obtained by increases in formal community-based services, we would examine service impact estimates for confirmation. Thus, we used the collective evidence of several estimates and test statistics to determine whether channeling influenced outcomes in a given area.

In addition to ascertaining whether channeling affected outcomes, we also examined the size of the estimated impact. In general, it makes little difference whether channeling impacts in a given area were zero or just very modest in size. To obtain some indication of the proportionate magnitude of impacts, mean values of the outcome variables for the control group were presented alongside the estimated impacts on these outcomes in tables displaying the results. Impacts exceeding 20 percent of the control group mean were generally felt to be large, although this could vary with the absolute magnitude of the control group mean (20 percent of a very small mean is still a very small impact). The dollar value of the impact also provided a useful way of assessing the importance of a given estimate for some outcome measures.

View full report

Preview
Download

"methodes.pdf" (pdf, 2.16Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®