Methodological Issues in the Evaluation of the National Long Term Care Demonstration. G. The Effects on Impact Estimates of Using Proxy Respondents


Because of the frailty of the sample, many sample members required the help of others (family, friend, nurse, caregiver) to complete the interview. However, proxies' responses to questions may differ considerably from those that the sample members would have given, especially to questions about attitudes or feelings. This issue raised concerns from the beginning of the evaluation about whether use of proxies at followup would distort our estimates of channeling impacts.

In order for proxy use at followup to bias impact estimates, it must be true that proxies for either the treatment group, the control group, or both respond differently than sample members would. There are three ways in which proxy use at followup could affect impact estimates:

  1. If proxies over- or underreported (relative to sample members) to the same extent for treatment and control groups, but rates of proxy use differed for treatment and controls.

  2. If proxies for the treatment group over- or underreported more or less than did proxies for control group members (whether rates of proxy use differed or not).

  3. If proxies over- or underreported to the same extent for both groups and rates of proxy use were the same. (In this case, the bias will be proportional because if the dependent variable mean is, say, overstated by a certain proportion for both treatments and controls, then the treatment/control difference is overstated by the same proportion.)

Of these, the first was considered to be the most likely to occur, and the second the least likely. The third situation would be clearly less serious than the other two, since proportional misreporting for both treatments and controls implies that impacts expressed as a percent of the control group mean will be unaffected. Therefore, we looked first at rates of proxy use and compared them for treatment and control groups, and then we compared impact estimates for self-respondents and proxy respondents.

Rates of proxy use for treatment and control groups were be remarkably similar for the two groups at all 3 followup interviews, both in answering specific questions and in overall response to the interview. Overall, about 40 to 45 percent of the interviews were completed without any assistance from proxies, while another 40 to 45 percent were completed entirely by proxies. For 45 to 50 percent of the sample members, a proxy answered the specific interview questions about the sample member's attitudes about satisfaction and contentment with life and with service arrangement.

The similarity of rates for the two groups made it, less likely that proxy use distorted estimates of channeling impacts. However, it was still possible, unless proxies responded no differently from sample members on average. To examine this question, the mean responses of proxies and self-respondents to several key questions at followup were compared. These comparisons showed that sample members with proxy respondents were recorded as being more impaired (on ADL and IADL tasks), less satisfied with life, and lonelier than sample members who responded themselves. However, examination of records data showed that sample members requiring proxies also had many more hospital and nursing home days, which suggests that the reported differences on interview items between those with and those without proxies may be real differences rather than the result of differential reporting by proxies and sample members. However, this conclusion could not be drawn without direct investigation of the effect on impact estimates of using proxy respondents.

To provide an indication of whether impact estimates were affected by proxy use, we estimated impacts on key outcomes separately for sample members with proxy respondents and those who responded themselves. We did this by modifying the standard regression model, replacing the binary treatment variables (T) with interaction terms (T* respondent type), then testing to see if impacts (the coefficients on T* respondent type) were equal.

We found relatively few significant differences in impacts (16 out of 90) between these two groups, but more than would be expected by chance. For impairment/health status outcomes (ADL, IADL, hospital days, nursing home days) we found a few significant differences but no systematic pattern. Among the formal and informal care measures, we found statistically significant differences in impacts across types of respondents only for the outcome variable indicating whether any informal care was received. The treatment group had a significantly lower proportion receiving informal care (from visiting caregivers or from anyone) than the control group among self respondents, but not among proxy respondents. However, it was unclear whether this difference was due to differences in physical or cognitive impairment between the types of clients who required proxy respondents and those who did not or to responses by proxy members that were not accurate reflections of what the sample members would have given themselves.

Six variables measuring sample members' attitudes were also examined, including their loneliness, overall satisfaction with life, confidence about receipt of care, contentment, self rating of health, and degree of concern about receiving needed care. Again we found relatively little difference in impacts across respondent types, except for the global life satisfaction variable. Among sample members with proxy respondents, the proportion reporting low satisfaction at 6 and 12 months was significantly smaller for the treatment group than for controls in both models, but no such pattern occurred for self respondents. Again, the relevant question was whether these results were due to differential reporting by proxies, or whether they, perhaps reflected the fact that proxy users were the most impaired (and presumably, least satisfied initially) and channeling may have had the biggest impact on the morale of those who were originally the most impaired/least satisfied (perhaps because they were not receiving needed services).

To distinguish between these two alternative explanations for the differences in impacts between self and proxy respondent cases, the regression model used to estimate impacts for the two groups was modified by including additional interaction terms involving treatment status and baseline measures of other factors that could affect channeling impacts. These factors were ones that were used in the analysis of channeling impacts on particular subgroups (see Chapter III): ADL, continence, unmet needs, referral source, Medicaid eligibility, living arrangement, whether on a nursing home waiting list, cognitive impairment, and site. Respondent type was added to this model as an additional set of subgroups. If the apparent differences in impacts across proxy use categories observed for informal care and global life satisfaction were in fact due to differences in impacts across impairment levels, impacts estimated from the revised subgroup regression for these two outcomes should no longer differ significantly across proxy use category, because the differences in channeling's effects across impairment subgroups would now be controlled for.

Once these other interactions were entered, impacts on informal care were no longer significantly different across types of respondents. Thus, it appeared that for informal care, proxy use did not affect impact estimates.

For the outcome variable representing sample members' satisfaction with life, however, the difference in impacts by respondent type remained statistically significant. Differences without controlling for subgroup effects were statistically significant for both models at 12 months and for the financial control model at 6 months. After controlling for other subgroup effects, only the 6-month basic model results indicated- significantly different impacts by respondent type. However, it was clear that in all three cases, the overall significant improvement in life satisfaction was driven by the treatment/control difference for those with proxy respondents. Thus, for this outcome the difference in impacts across types of respondent were not merely reflecting impact differentials across baseline impairment or unmet need categories.

From the set of analyses conducted we concluded that with one possible exception the use of proxy respondents did not result in distorted estimates of channeling impacts. The potential exception to this was the result for life satisfaction, for which it was difficult to distinguish between two plausible alternatives. It is possible that, as caregivers, proxies for treatment group members were so pleased with the additional help channeling provided that their response reflected the proxy's own satisfaction more than that of the sample member. On the other hand, it may have been the case that sample members requiring proxies at followup were those most dissatisfied with life at baseline and it was this dissatisfied group for which channeling had the biggest effect on reported life satisfaction. Yet another possible explanation is that those who required proxies at followup but were not highly impaired at baseline may be the group whose health or ability to function deteriorated, the most over the six months. Channeling impacts on satisfaction could be greatest for this group. In any case, however, channeling appears to have had an impact on satisfaction. Whether these impacts were for a certain set of sample members or for the caregivers of those sample members is unclear.

View full report


"methodes.pdf" (pdf, 2.16Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®