Methodological Issues in the Evaluation of the National Long Term Care Demonstration. E. Differences Between Early and Late Cohorts of Sample Members


From the outset of the demonstration it was recognized that the impacts of channeling might vary with the length of time since the client entered the program, as clients' needs and health status change and as case managers and clients become more familiar with each other. However, comparing estimates of channeling impacts at 18 months to those obtained at 12 months could result in misleading inferences about such changes because, as pointed out in Chapter II, only half of the sample was followed up at 18 months, and time constraints led to defining this group as the half who entered the sample earliest. Erroneous inferences would occur if channeling's effectiveness changed with calendar time (because of specific changes in the environment in which channeling operates or in the program itself) rather than with the length of time the sample member was in the program. Alternatively, program effectiveness could change if the type of clients served by channeling changed over time. Since the 18-month cohort consists of those enrolling earliest, we must ensure that any differences between 12- and 18-month results are not due to differences in the calendar period covered by the early and late cohorts or to differences between the cohorts rather than to the length of time spent in channeling.

To distinguish changes in impacts due to length of time in the program from those due to cohort effects such as those just described, estimated impacts on a set of 14 key outcomes (those used in the attrition and pooling analyses) at 6 and 12 months for the early cohort were compared to the corresponding estimates for the late cohort. Equivalence of the impacts at these earlier points would suggest that comparison of 18-month estimates obtained on only the early cohort to estimated impacts at 12 months based on the full sample should be interpreted as effects of the length of time in channeling. A finding of statistically significant differences between cohorts in impacts during the 1-6 and 7-12 mouth periods would indicate that 18-month results should be compared to 6- and 12-month results estimated on only the early cohort.31 While such cohort differences for the early periods would not necessarily imply that any differences in estimated impacts between 12 and 18 months would be due to cohort effects rather than to the length of time in channeling, it would suggest that possibility.

To investigate this issue, the standard regression model shown in equation 1 was modified in order to estimate separate impacts of channeling for each cohort on the key outcome variables listed in Section D above. The modification was to replace each of the binary treatment status variables in equation 1 with two new binary variables, the first equal to 1 only for treatment group members in that model in the early cohort and the second equal to 1 for treatments in the late cohort for that model. Two additional binary variables were also added to the regression equation, one for each channeling model, indicating whether the sample member was in the late cohort. The coefficients on the four new treatment variables provided estimates of channeling impacts for the two cohorts for each channeling model. The coefficients on the cohort indicator variables provided estimates of the differences in mean outcomes between cohorts for the control group in each model, controlling for possible differences between the cohorts on other explanatory variables.

For each key outcome measure, the revised regression equation was estimated and an F-test was performed (separately for basic and financial control models) to test for significant differences between the impact of channeling for the early cohort and the impact for the late cohort. In addition, multivariate tests were conducted on groups of related outcome measures to determine whether jointly, across the set of outcomes, impacts for the early cohort differed from those estimated for the late cohort.

The tests indicated that channeling impacts differed very little between cohorts at 6 and 12 months after randomization. Of the five instances of significantly different estimates (out of 72 tests), two were for receipt of case management at 6 months, for which the impact estimates were large, positive, and highly significant for both cohorts. Thus, even though the estimates were statistically different, the inferences to be drawn from the case management results were the same for both cohorts. The fact that it was changes in the control group which were responsible for the observed differences between cohorts in impacts on case management suggests that channeling may have changed relatively little, but the availability of non-channeling case management may have changed over time.

The remaining three instances of significant differences by cohort were isolated, and two of these occurred at 6 months. This is important, since it is the comparison of impacts at 12 and 18 months that we were most concerned about being distorted by cohort effects. 'Whether formal care was received" was the only 12-month outcome for which a statistically significant difference across cohorts was found, and only for the basic model (although the cohort differential in the financial control model was nearly as large and had a test statistic only slightly smaller than the critical value for significance at the .05 level). The difference in impacts was due entirely to the significant difference (decline) between the early and late cohorts in the proportion of the control group receiving formal care. Whether this drop was due to different attrition of controls for the two cohorts, to changes in the types of clients attracted, or to changes in the local availability of formal services is not clear. However, the two former explanations do not seem likely given that the proportion of controls receiving formal care at baseline was very similar for the two cohorts in the basic model--57 and 55 percent for early and late cohorts, respectively. The fact that estimated impacts on hours of formal care did not differ significantly across cohorts further increased our confidence that cohort differences did not distort the comparison of 12- and 18-month impacts in general.

We concluded that estimates at 18 months on the early cohort could be compared to those at 12 months for the full sample with little concern that the comparison would be distorted by differences between the cohorts. The exception to this conclusion was that if such comparisons for formal care outcomes suggested sizeable changes in impacts between 12 and 18 months, it would be important to interpret these changes in light of the cohort difference identified here. In the final analysis of channeling impacts on use of formal community services, Corson et al. (1986) did in fact find a marked decline in impacts between 12 and 18 months, which was attributed to this cohort effect.

View full report


"methodes.pdf" (pdf, 2.16Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®