Final Report on the Effects of Sample Attrition on Estimates of Channeling's Impacts. B. Comparison of Estimated Impacts for the Full Sample to Those for the Analysis Samples

01/13/1986

Table IV.1 and Table IV.2 present the estimated impacts and control group means on six month Medicare outcomes for the full and analysis samples for the basic and financial control models, respectively. Table IV.3, Table IV.4, Table IV.5 and Table IV.6 contain similar results for the 7 to 12 and 13 to 18 month periods. The evidence of bias due to attrition is reviewed separately for each of the analysis samples.

1. The Medicare Sample

The Medicare sample, used to estimate impacts on hospital outcomes, included'88 percent of the full sample, as shown in Table II.1 at 6 and 12 months and the same fraction of the 18 month cohort at 18 months. Comparison of the impact estimates on hospital days and expenditures for the full and Medicare samples shows little difference, either between control group means or impacts, in any time period. In no case is the impact estimate statistically significant at the .05 level, in either sample. The full sample estimate of impacts on hospital days at 6 months in the financial control model (-1.54 days) is nearly significant, whereas the corresponding sample estimate is somewhat smaller (-1.15) and statistically insignificant. However, the relatively small size of this treatment/control difference (less than 10 percent of the control group mean) and the lack of significant impacts on expenditures or on days in other time periods or for the other (basic) model suggests that the estimates of hospital impacts are not biased by attrition from the Medicare sample.

TABLE IV.1: Estimated Impacts and Control Group Means for Medicare-Covered Outcomes Using Full and Analysis Samples: Basic Case Management Model, Months 1-6
Medicare-Covered Outcomes Full Sample Medicare Sample Nursing Home Sample Followup Sample In Community Sample
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
Hospital Days 11.4 -0.3
(-0.43)
11.2 -0.2
(-0.29)
10.3 -0.3
(-0.28)
10.0 -0.5
(-0.58)
6.4 -0.3
(-0.39)
Nursing Home Days 1.8 -0.3
(-0.73)
1.4 -0.0
(-0.03)
1.3 0.1
(0.12)
1.4 0.1
(0.11)
0.5 0.2
(0.67)
Hospital Medicare Expenditures 3097 -126
(-0.56)
3047 -66
(-0.28)
2727 -34
(-0.13)
2626 -210
(-0.80)
1875 -122
(-0.54)
Nursing Home Medicare Expenditures 79 -13
(-0.62)
77 -10
(-0.46)
72 -21
(-0.93)
81 -21
(-0.88)
35 5
(0.25)
Skilled Nursing Visits 5.8 0.7
(1.59)
5.9 0.9
(1.89)
5.7 1.1
(1.90)
5.6 1.2*
(2.01)
5.8 1.4*
(2.19)
Home Health Aide Visits 7.0 0.6
(0.73)
7.0 0.8
(0.95)
6.9 0.9
(0.92)
7.0 0.9
(0.95)
7.3 1.0
(0.90)
Skilled Nursing Reimbursements 233 31
(1.76)
239 38*
(2.00)
233 44*
(2.04)
232 49*
(2.16)
241 54*
(2.14)
Home Health Aide Reimbursements 219 21
(0.93)
219 30
(1.26)
217 30
(1.12)
223 31
(1.12)
237 29
(0.93)
Total Reimbursements for Nursing, Therapy, Home Health Aides, Special Equipment and Supplies 734 80
(1.69)
744 101*
(2.02)
741 110*
(1.96)
753 122*
(2.07)
790 114
(1.72)
Medicare Reimbursements for Other Medical Services
(lab tests, x-rays, outpatient services)
188 8
(1.26)
192 9
(0.28)
185 5
(0.14)
170 2
(0.05)
144 17
(0.46)
Physician Medicare Reimbursements 552 -2
(-0.05)
546 6
(0.13)
505 19
(0.39)
487 -10
(-0.20)
365 -9
(-0.21)
Sample Sizea 1286 3018 1104 2712 903 2184 826 1989 684 1641
NOTE: Data on the dependent (outcome) variables were obtained exclusively from Medicare claims and (for treatment group members in the financial control model) FCS data. Control group means are simple arithmetic means; treatment/control differences were estimated by regression of outcome variables on screen control variables and treatment status. See text for explanation and definition of samples and outcome variables.
  1. Sample sizes are slightly smaller than those given in Table II.2 because it was necessary to limit this analysis to observations with known Medicare eligibility.


TABLE IV.2: Estimated Impacts and Control Group Means for Medicare-Covered Outcomes Using Full and Analysis Samples: Financial Control Model, Months 1-6
Medicare-Covered Outcomes Full Sample Medicare Sample Nursing Home Sample Followup Sample In Community Sample
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
Hospital Days 16.4 -1.5
(-1.93)
16.1 -1.2
(-1.35)
15.2 -0.9
(-0.99)
14.0 -1.2
(-1.27)
8.5 0.3
(0.42)
Nursing Home Days 2.8 -0.3
(-0.73)
2.6 -0.0
(-0.08)
1.7 0.3
(0.78)
1.4 0.5
(1.06)
0.8 -0.1
(-0.3)
Hospital Medicare Expenditures 4493 -220
(-0.98)
4313 -45
(-0.19)
3914 95
(0.37)
3460 -13
(-0.05)
2557 164
(0.72)
Nursing Home Medicare Expenditures 124 -4(-0.17) 116 4
(0.18)
94 15
(0.68)
79 26
(1.11)
59 5
(0.24)
Skilled Nursing Visits 8.2 0.6
(1.36)
8.6 0.5
(0.89)
8.9 0.3
(0.50)
8.9 0.3
(0.45)
9.2 0.1
(0.10)
Home Health Aide Visits 14.6 -2.9**
(-3.86)
15.3 -0.35**
(-4.27)
15.8 -3.8**
(-4.17)
15.8 -3.5**
(-3.57)
16.6 -3.3**
(-2.93)
Skilled Nursing Reimbursements 298 -8
(-0.46)
313 -16
(-0.85)
324 -22
(-1.05)
322 -24
(-1.08)
333 -26
(-1.04)
Home Health Aide Reimbursements 450 -158**
(-7.23)
471 -178**
(-7.55)
486 -191**
(-7.37)
483 -184**
(-6.66)
503 -183**
(-5.86)
Total Reimbursements for Nursing, Therapy, Home Health Aides, Special Equipment and Supplies 1230 -196**
(-3.40)
1271 -179**
(-3.58)
1312 -187**
(-3.39)
1381 -194**
(-3.31)
1365 -192**
(-2.91)
Medicare Reimbursements for Other Medical Services
(lab tests, x-rays, outpatient services)
251 -4
(-0.13)
240 15
(0.48)
231 29
(0.88)
206 51
(1.50)
174 66
(1.83)
Physician Medicare Reimbursements 819 -14
(-0.33)
787 26
(0.61)
755 23
(0.47)
682 15
(0.30)
504 50
(1.11)
Sample Sizea 1256 3150 1047 2842 861 2409 755 2147 623 1807
NOTE: See Table IV.1 for notes.


TABLE IV.3: Estimated Impacts and Control Group Means for Medicare-Covered Outcomes Using Full and Analysis Samples: Basic Case Management Model, Months 7-12
  Full Sample Medicare Sample Nursing Home Sample Followup Sample Alive at 6 Months
Full Sample In Community Sample
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
Hospital Days 6.7 0.0
(0.04)
6.8 0.0
(0.04)
5.1 0.05
(0.07)
6.3 -0.01
(-0.01)
8.2 -0.1
(-0.09)
4.3 0.5
(0.82)
Nursing Home Days 1.2 -0.3
(-0.70)
1.2 -0.5
(-1.14)
1.1 -0.5
(-1.33)
1.2 -0.5
(-1.08)
1.4 -0.4
(-0.79)
0.5 -0.1
(-0.23)
Hospital Medicare Expenditures 1742 120
(0.70)
1809 62
(0.34)
1491 73
(0.40)
1661 27
(0.13)
2127 112
(0.55)
1322 88
(0.43)
Nursing Home Medicare Expenditures 70 -25
(-1.45)
76 -30
(1.58)
61 -14
(-0.71)
72 -16
(-0.61)
86 -32
(-1.50)
34 5
(0.22)
Skilled Nursing Visits 2.6 0.6
(1.71)
2.7 0.6
(1.47)
2.6 0.8
(1.74)
3.2 0.9
(1.61)
3.1 0.7
(1.64)
3.7 0.9
(1.41)
Home Health Aide Visits 3.1 0.5
(0.70)
3.2 0.5
(0.63)
2.9 0.7
(0.77)
3.4 0.8
(0.67)
3.8 0.5
(0.61)
3.9 1.1
(0.78)
Skilled Nursing Reimbursements 105 29*
(2.02)
114 27
(1.70)
105 36*
(2.10)
129 42*
(1.97)
129 33
(1.94)
150 43
(1.74)
Home Health Aide Reimbursements 106 14
(0.76)
110 14
(0.68)
98 22
(0.98)
116 23
(0.83)
130 14
(0.64)
134 32
(0.98)
Total Medicare Home Health and Part B Reimbursements 376 66
(1.61)
399 59
(1.34)
380 72
(1.47)
464 74
(1.23)
460 74
(1.56)
507 103
(1.48)
Nonphysician Medicare Reimbursements 120 45
(1.59)
129 39
(1.32)
133 32
(1.01)
161 26
(0.63)
140 49
(1.46)
149 12
(0.26)
Physician Medicare Reimbursements 342 44
(1.30)
357 35
(0.96)
312 31
(0.88)
360 32
(0.72)
399 46
(1.14)
276 42
(1.05)
Sample Size 1296 3018 1104 2712 935 2294 695 1732 1053 2489 547 1371
NOTE: See Table IV.1 for notes.


TABLE IV.4: Estimated Impacts and Control Group Means for Medicare-Covered Outcomes Using Full and Analysis Samples: Financial Control, Months 7-12
  Full Sample Medicare Sample Nursing Home Sample Followup Sample Alive at 6 Months
Full Sample In Community Sample
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
Hospital Days 8.5 -0.4
(-0.66)
8.9 -0.7
(-0.94)
7.9 -0.3
(-0.38)
9.2 -0.7
(-0.83)
10.1 -0.5
(-0.60)
5.7 -0.9
(-1.41)
Nursing Home Days 1.8 -0.0
(-0.04)
1.9 -0.1
(-0.13)
1.3 0.2
(0.40)
1.6 0.0
(0.02)
2.2 -0.0
(-0.08)
0.5 -0.1
(-0.19)
Hospital Medicare Expenditures 2328 -77
(-0.45)
2385 -105
(-0.57)
1964 110
(0.60)
2233 -5
(-0.02)
784 -86
(-0.43)
1777 -172
(-0.85)
Nursing Home Medicare Expenditures 61 -5
(-0.29)
65 1
(0.07)
60 1
(0.04)
78 -4
(-0.15)
72 6
(0.30)
16 23
(1.13)
Skilled Nursing Visits 3.8 1.6**
(4.30)
4.2 1.4**
(3.46)
4.3 1.3**
(3.01)
5.2 1.7**
(3.08)
4.6 1.9**
(4.46)
5.5 2.1**
(3.18)
Home Health Aide Visits 6.7 1.5*
(1.99)
7.0 1.3
(1.54)
7.3 1.5
(1.60)
9.0 1.8
(1.50)
8.0 1.7
(1.88)
9.1 2.9*
(2.12)
Skilled Nursing Reimbursements 140 40**
(2.86)
153 32*
(2.06)
158 27
(1.62)
190 37
(1.73)
168 50**
(2.98)
200 48
(1.94)
Home Health Aide Reimbursements 198 -21
(-1.14)
210 -33
(-1.63)
218 -33
(-1.49)
266 -41
(-1.43)
237 -28
(-1.29)
267 -16
(-0.48)
Total Medicare Home Health and Part B Reimbursements 638 73
(1.79)
682 41
(0.93)
696 47
(0.96)
851 54
(0.89)
764 87
(1.85)
847 136*
(1.96)
Nonphysician Medicare Reimbursements 191 0
(0.01)
183 9
(0.31)
179 5
(0.16)
202 29
(0.73)
224 1
(0.03)
172 24
(0.53)
Physician Medicare Reimbursements 589 -10
(-0.30)
519 -18
(-0.48)
461 3
(0.07)
528 12
(0.28)
597 -7
(-0.17)
413 12
(0.29)
Sample Size 1256 3150 1047 2842 881 2458 657 1860 1050 2630 519 1481
NOTE: See Table IV.1 for notes.


TABLE IV.5: Estimated Impacts and Control Group Means for Medicare-Covered Outcomes Using Full and Analysis Samples: Basic Case Management Model, Months 13-18
  Full Sample Medicare Sample Nursing Home Sample Followup Sample Alive at 6 Months
Full Sample In Community Sample
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
Hospital Days 4.0 0.9
(1.10)
4.2 1.0
(1.13)
3.0 0.6
(0.71)
3.8 1.0
(0.82)
5.8 0.9
(0.83)
2.9 0.5
(0.53)
Nursing Home Days 1.2 0.0
(0.00)
1.2 -0.5
(-0.74)
0.9 -0.5
(-0.95)
1.4 -0.8
(-0.92)
1.7 -0.1
(-0.17)
0.5 -0.4
(-0.85)
Hospital Medicare Expenditures 1127 316
(1.34)
1234 283
(1.13)
842 154
(0.66)
1023 341
(1.11)
1610 324
(1.02)
870 167
(0.56)
Nursing Home Medicare Expenditures 25 4
(0.23)
21 5
(0.27)
22 -3
(-0.18)
30 -5
(-0.21)
36 1
(0.06)
0 8
(0.71)
Skilled Nursing Visits 2.3 0.3
(0.58)
2.4 0.3
(0.55)
2.2 0.4
(0.63)
3.0 0.8
(0.84)
3.2 0.2
(0.34)
2.9 1.3
(1.11)
Home Health Aide Visits 2.7 0.7
(0.67)
2.9 0.6
(0.55)
2.7 0.5
(0.42)
3.8 0.7
(0.35)
3.9 0.6
(0.47)
4.0 1.7
(0.65)
Skilled Nursing Reimbursements 92 18
(0.93)
97 20
(0.94)
89 24
(1.05)
122 44
(1.24)
131 18
(0.70)
120 69
(1.58)
Home Health Aide Reimbursements 94 19
(0.77)
101 17
(0.63)
94 19
(0.62)
134 26
(0.53)
135 17
(0.49)
138 55
(0.90)
Total Medicare Home Health and Part B Reimbursements 322 78
(1.30)
342 76
(1.16)
326 63
(0.82)
444 118
(1.01)
460 87
(1.09)
441 170
(1.19)
Nonphysician Medicare Reimbursements 125 -20
(-0.56)
128 -15
(-0.40)
103 -13
(-0.33)
141 -14
(-0.22)
172 -32
(-0.66)
117 -14
(-0.19)
Physician Medicare Reimbursements 239 51
(1.21)
252 51
(1.14)
213 0
(0.00)
245 71
(1.17)
329 62
(1.10)
199 70
(1.20)
Sample Size 673 1571 592 1415 475 1119 279 678 471 1123 216 519
NOTE: See Table IV.1 for notes.


TABLE IV.6: Estimated Impacts and Control Group Means for Medicare-Covered Outcomes Using Full and Analysis Samples: Financial Control Model, Months 13-18
  Full Sample Medicare Sample Nursing Home Sample Followup Sample Alive at 6 Months
Full Sample In Community Sample
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
  Control  
Mean
T/C
  Differences  
Hospital Days 6.7 0.1
(0.07)
6.6 0.4
(0.43)
4.4 1.4
(1.60)
6.3 1.3
(1.04)
9.1 0.4
(0.37)
4.8 -0.9
(-1.00)
Nursing Home Days 2.5 -1.1
(-1.58)
2.8 -1.6*
(-2.36)
1.8 -1.1
(-1.92)
2.5 -1.5
(1.71)
3.3 -1.5
(-1.59)
0.9 -1.1*
(-2.12)
Hospital Medicare Expenditures 1890 60
(0.25)
1799 187
(0.71)
1285 393
(1.63)
1809 146
(0.47)
2553 91
(0.28)
1548 -180
(-0.60)
Nursing Home Medicare Expenditures 68 -35*
(-2.11)
77 -46*
(-2.54)
58 -36*
(2.16)
84 -52*
(-1.99)
92 -49*
(-2.10)
15 -13
(-1.15)
Skilled Nursing Visits 3.4 0.9
(1.75)
3.7 0.8
(1.26)
3.7 0.5
(0.73)
5.7 0.4
(0.41)
4.6 1.4*
(2.01)
6.9 0.4
(0.38)
Home Health Aide Visits 4.7 2.5*
(2.42)
5.0 2.1
(1.82)
4.9 2.6
(1.92)
7.6 3.8
(1.83)
6.4 3.5*
(2.52)
9.4 4.9
(1.89)
Skilled Nursing Reimbursements 125 21
(1.03)
132 15
(0.69)
133 4
(0.16)
205 -6
(-0.18)
169 33
(1.24)
249 -7
(-0.15)
Home Health Aide Reimbursements 140 16
(0.63)
149 -1
(-0.02)
143 11
(0.34)
222 13
(0.27)
190 25
(0.71)
273 19
(0.31)
Total Medicare Home Health and Part B Reimbursements 575 115
(1.84)
535 96
(1.40)
527 116
(1.46)
803 181
(1.54)
695 177*
(2.18)
945 252
(1.76)
Nonphysician Medicare Reimbursements 176 11
(0.29)
167 21
(0.54)
129 57
(1.38)
165 98
(1.52)
235 10
(0.21)
86 185*
(2.49)
Physician Medicare Reimbursements 459 -40
(-0.91)
461 -37
(-0.79)
370 -14
(-0.32)
530 -78
(-1.27)
613 -42
(-0.73)
454 -107
(-1.83)
Sample Size 608 1522 501 1372 399 1129 249 719 450 1110 195 553
NOTE: See Table IV.1 for notes.

The differences between the impact estimates for the full and Medicare samples on other outcome measures are less important, since the Medicare sample is used to examine only hospital outcomes. However, since attrition from the Medicare sample is due almost entirely to baseline nonresponse, which is responsible for much of the total attrition in other samples and for all of the treatment/control differences in attrition for all analysis samples, it is useful to see the effect of baseline attrition on estimated impacts on other variables.

For 2 of the 9 other variables examined in the basic model at 6 months, the estimated impacts were somewhat larger when estimated on the Medicare sample than when estimated on the full sample, and went from being statistically insignificant to significant. The variables were skilled nursing reimbursements and total reimbursements for the Medicare-covered community-based services (nursing, therapy, home health aides, etc.). Differences in control group means on these variables increased slightly (by 1 to 3 percent) when restricted to the Medicare sample, compared to 4 to 5 percent for the treatment group. Thus, despite the change in significance level, the estimated impact was the about the same proportion of the control group mean. For other time periods and for the financial control model we observe even smaller differences between the two samples on these or other variables. Thus, it seems likely that the observed differences at 6 months for the basic model are due to chance rather than to a systematic difference between treatments and controls in the types of individuals who fail to respond at baseline. We return to this issue in the discussion of the in-community analysis samples, which were the samples used to estimate the effects of channeling on formal community care.

2. The Nursing Home Sample

The nursing home samples, used to estimate impacts on nursing home outcomes in the evaluation, includes between 71 and 75 percent of the full sample, depending upon which time period is examined (see Table II.1). Comparison of the estimates of impacts on Medicare-covered nursing home days and expenditures for the full and nursing home sample reveals no substantive differences: in all instances, estimates are small and statistically insignificant. However, since only a small proportion of total nursing home use is covered by Medicare, we also compare impact estimates on other outcome measures estimated on these two samples. We find no noteworthy differences in estimated impacts on hospital use or other formal service measures for either model or any time period, with the exception of the basic model results at 6 months. Here, as was found for the Medicare sample, the estimated impacts on skilled nursing and total community-based service reimbursements were statistically significant for the analysis sample but not for the full sample, and somewhat larger. This occurred despite the very small change in the control group means of these variables. This suggests that treatment group members who were not in the nursing home sample used less Medicare-covered community services than treatment group members who were not included in the analysis. However, this pattern does not occur in other time periods nor for the financial control model, nor does a difference occur for other variables even for this period. Thus, it is highly unlikely that total nursing home impacts estimated on the nursing home sample are seriously distorted.

3. The Followup Sample

The followup sample was used in the evaluation to estimate impacts on well being, and included anywhere from 40 to 70 percent of the full sample, depending on the time period, model, and experimental group examined. Since no well-being measures are available in the Medicare data, the comparisons of the full and followup samples provides only indirect evidence of bias that is even less direct than that for the other analysis samples examined.

The results are very similar to those obtained for the nursing home sample: only for the basic model at 6 months are the estimates noticeably different for the full and followup samples, and then only for 3 of the 11 variables examined. The impact estimates for skilled nursing visits and reimbursements, and for total reimbursements for Medicare-covered community-based services are 16 to 21 percent of the control group mean for the analysis sample, and statistically significant, compared to 11 to 13 percent of the control group mean and statistically insignificant for the full sample. Again, the treatment group means for the analysis sample are larger than for the full sample, by 4.5 to 7.5 percent, compared to little or no change for the control group. At 12 months, this pattern recurs only for skilled nursing reimbursements. In the financial control model, there are no changes in statistical significance, but the significant differences in home health visits and reimbursements are slightly larger for the analysis sample than for the full sample at 6 months. Again, we conclude that bias is not expected to be a major problem for impact estimates obtained on the followup sample.

4. The In-Community Sample

Finally, we examine the in-community sample for evidence of attrition bias. At 6 and 12 months, this sample is comprised of 50 to 60 percent of those alive at the beginning of the analysis period; at 18 months, only 40 to 50 percent of those alive at 12 months are in the community. This sample is used to estimate the impacts of channeling on the receipt of formal and informal care; hence, the full versus in-community sample comparisons of most direct relevance are those for skilled nursing and home health aide services, although the comparison for other outcomes also provides important information on the effects of attrition.

In the basic model at 6 months, we find no differences between the full and analysis sample estimates of impacts on hospital or nursing home outcomes. There also is no evidence of attrition bias in analysis sample estimates of impacts on home health aide visits or reimbursements, or in reimbursements for physician and other medical services. However, estimated impacts on skilled nursing visits and reimbursements are statistically significant when estimated on the in-community sample, but not when estimated on the full sample. This finding is in marked constrast to the full versus in-community comparisons for the 7 to 12 month period for this model, where no such discrepancy is observed. Furthermore, no such discrepancy is observed for the financial control model in either period, despite the fact that treatment/control differences in response rates are so much larger in this model.

If attrition bias were a serious problem for estimates obtained on the in-community sample, we would expect to see some consistent discrepancies between full and analysis sample estimates either across outcomes or time periods or models. The lack of consistency observed in these comparisons makes it unlikely that estimated impacts on formal and informal care outcomes, based an the in-community sample, are seriously distorted by attrition.

View full report

Preview
Download

"atritn.pdf" (pdf, 4.74Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®