Effectiveness of Alternative Ways of Implementing Care Management Components in Medicare D-SNPs: The Brand New Day Study - Executive Summary

Jelena Zurovac, Randy Brown, Bob Schmitz and Richard Chapman

Mathematica Policy Research

January 2014

Introduction

Evidence on best practices in care management for chronically ill Medicare beneficiaries offers few clear guidelines about what works best. Given the wide variation both within and across plans in how special needs plan (SNP) services are provided, it becomes important to identify how best to implement or improve intervention. In this study, we sought to understand which of two alternative ways of implementing each of several components of care management lead to better health outcomes in Brand New Day SNP. We used an efficient orthogonal design that allowed us to simultaneously compare effectiveness of alternative approaches to implementing 11 components of care management services. Efficient orthogonal designs have been used extensively in manufacturing, and in some health care organizations, but not in published health care evaluations. Such designs enable the testing of multi-component interventions and various ways of deploying each component, offering great potential as a tool for continuous improvement in health care quality.

This study compares key patient outcomes at Brand New Day, a Medicare SNP for dual eligibles with severe mental illness, under two alternatives--routine care (services routinely provided at the plan before the study) and enhanced care (more frequent or more intensive services)--for each of 11 care components. The tested components included frequency-of-routine contacts; depression screening frequency, use of depression screening instruments, and mode of referral to specialists; member education and coaching strategies; medication and chronic disease management (DM); and management of care transitions, including frequency of follow-up and use of protocols and tools.

Study Design and Analysis Methods

Randomization, Outcomes, and Data

The study participants were: (1) care managers (referred to as "life coaches") in Brand New Day, who implemented the interventions; and (2) the 1,422 dually eligible noninstitutionalized members with severe and persistent mental illness who comprised these life coaches' caseloads. We randomly assigned each of the 28 life coaches to implement a different, pre-selected combination of alternatives (routine care or enhanced care) for each of 11 components, over a one-year period. The life coaches implemented the same intervention components for all of their members. Although we randomly assigned the life coaches, several other types of care management staff assigned to care for a given life coach's members also provided the components, including clinical directors, field intervention nurses (field nurses), and primary care physicians (PCPs).

For each component we analyzed whether members assigned to the enhanced care variant experienced different outcomes than those assigned to the routine care variant. Outcomes examined included: (1) the number of inpatient admissions for any reason; (2) the number of psychiatric inpatient admissions; (3) the incidence of readmission within 30 days of discharge (for those with a hospital admission); and (4) the number of emergency room (ER) visits. The program period spanned from July 1, 2011, through June 30, 2012. We received approval for the study from the New England Institutional Review Board. U.S. Office of Management and Budget approval was not required because Mathematica did no primary data collection.

To analyze the effectiveness of enhanced versus routine care, we used two sources of secondary data obtained from the plan: (1) de-identified claims data on members' service use and chronic conditions; and (2) de-identified data on members' demographic characteristics and risk level, as assessed by the plan. For the implementation analysis, we used data collected by the plan via tracking tools to assess the care managers' fidelity to their assigned component options. Care management staff were instructed to use the tracking tool form after each contact with the members to record which components were provided. We also conducted discussions with care management staff to understand how faithfully the components were implemented and any barriers they encountered.

Impact and Implementation Analysis Methods

We used regression analysis to compare the outcomes for members receiving routine care to the outcomes for members receiving enhanced care, controlling for any pre-intervention differences between the two groups in members' and care managers' characteristics. All four outcomes were analyzed over these follow-up periods after program start up on July 1, 2011 for all members: 1-6 months, 7-12 months; and the full 12 month period. Analyses of effects of components on readmissions were done for hospitalized members only over the full 12 month follow-up period. Regression analyses controlled for member characteristics observed over the two-year baseline period (July 1, 2009, to June 30, 2011).

Implementation analysis is particularly important because a finding from regression analyses that routine and enhanced care options for a given component are equally effective in terms of observed health outcomes might be incorrect if such care was not fully implemented. We used the tracking tool data to assess the fidelity to assignments by examining: (1) the proportion of members receiving the assigned option at least once; (2) the annualized number of times each component or option was provided per member; and (3) the proportion of members receiving the option at least as often as assigned. We supplemented these data with telephone discussions with care management staff in July through November 2012 (between a few weeks and four months after the intervention period ended, but before the analysis results were produced. These discussions provided information on their views on why enhanced care may have been more effective than routine care for some components but not for others, and to identify implementation facilitators and barriers.

Study Findings and Discussion

Descriptive Results

The population of members in the study was composed mostly of middle-age adults, included more men than women, and was largely Caucasian. Members primarily lived in urban areas. During both the baseline and follow-up years, approximately a quarter of members were hospitalized; also, members experienced slightly over one ER visit per member per year.

Findings

Outcomes for patients whose care managers were assigned to the enhanced version were not significantly different from those for patients with care managers assigned to the routine version for most of the 11 care components. However, there were a few exceptions:

• Requiring more frequent depression screening using a particular instrument was associated with approximately 41 percent fewer ER visits over the full year of follow-up. Although the number of screenings per member per year was low, staff screened over three times as many members assigned to enhanced care (25.5 percent versus 7.9 percent) and conducted nearly four times as many screenings per member per year (0.59 versus 0.15). Further, the difference on this outcome for the year was driven by the difference in the first six months of the study, a finding that is not surprising because fidelity analysis showed that most screenings were performed in the first six months of the study.

• Members assigned to frequent depression screenings with an instrument also had 53 percent more short-term readmissions after a hospital discharge than members assigned to less frequent screening. This finding is difficult to explain; we did not expect this component to affect readmissions in either direction.

• Assigning life coaches to reinforce DM education during routine contacts was associated with approximately 50 percent lower likelihood of readmissions over the full year of follow-up (for hospitalized members). Even though care managers liked this component, some staff were uncomfortable addressing medical issues, indicating a need for additional training or use of medical staff.

• Assigning field nurses to conduct follow-up visits after a discharge from a hospital for medical (nonpsychiatric) discharge was associated with a significantly higher short-term hospital readmission rate than routine care. However, very few field nurses actually provided such care.

• Results for outcomes measured over the periods of 1-6 months and 7-12 months were generally similar to those for the full period, suggesting that most of the enhanced options neither influenced outcomes early on but then dissipated, nor that they took several months to take effect. One exception is that full medication review was associated with fewer ER visits over the 1-6 month (but not the 7-12 month) periods.

Some findings of no difference in outcomes may be attributable to a failure to implement the enhanced care option in a manner that sufficiently distinguished it from the routine care option. For example, although the teachback method was qualitatively more intensive, the fidelity analysis showed that life coaches used it less often than routine practices, indicating that members assigned to teachback might have received less coaching (a similar finding was observed in our companion study of two other SNPs; see Zurovac et al. 2013). Also, care managers performed many medication reviews to compensate for PCPs performing very few. And, even though the brownbag medication review performed by life coaches did not improve outcomes, plan staff saw it as highly effective. A clinical director noted that brownbag review prompted some members to engage their PCPs.

Care management staff reported several important lessons learned from the study implementation. Life coaches and clinical directors said that efforts to better integrate medical and psychiatric care were very helpful to members; they want to see more such efforts in the future. This feedback is consistent with the estimated outcome differences: two of the three enhanced care components that had better outcomes addressed members' medical needs: reinforcement of DM education and medication review (conducted by life coaches as part of routine contacts). Two components aimed to increase the engagement of field nurses, but due to very large caseloads, these nurses' involvement in the study was very limited. Life coaches and clinical directors reported that they want the plan to hire more field nurses and that they need easier access to them, indicating a need for additional integration of medical services.

Limitations

Several limitations in the study should be noted. Because only 28 life coaches participated in the study, only large differences in outcomes between routine and enhanced care options (38-64 percent of the mean outcome) were likely to be detected.

Given that we performed many comparisons between enhanced and routine care, it is possible that some findings resulted from chance. The number of significant differences was about what would be expected by chance for the 110 comparisons (three outcomes were analyzed for 11 components for all members for three periods and one outcome for 11 components for hospitalized members). Thus, it is unclear whether these represent true effects or chance differences. A joint test of whether all enhanced versus routine care differences were zero could not be rejected, indicating that even the few statistically significant observed differences may have been due to chance rather than to the interventions. This also indicates that as a group, enhanced components did not have a different effect on measured outcomes than routine practices.

The findings from the implementation analysis of the tracking data may be flawed by incomplete reporting by the care management staff on their activities. The tracking tool data showed provision of few components in the second half of the study; it is unclear whether that was due to: (1) the plan not providing complete tracking sheets for that period; and/or (2) plan staff providing fewer study components during the second half of the study. In the last few months of the study, the plan incorporated the tracking tool as part of the electronic care management system, which made it easier for staff; however, not all staff had access to the system and multiple organizational changes caused the study (and the recording of provided services) to be somewhat neglected during that period, which most likely explains gaps in the tracking tool data.

For several components, the enhanced care option was not implemented in a manner that distinguished it sufficiently from the routine care option because it was not implemented consistently or fully or because routine care was more intensive when delivered than specified by the participating plans. However, this is not a limitation, but rather, an important finding that can inform plans of the need to identify barriers that care managers face to implementing planned intervention components and seek ways to overcome them. The analyses in this report took an "intent-to-treat" approach in which component effects are computed by comparing outcomes of those assigned to the two options, regardless of whether or how thoroughly the options were actually delivered. Standard supervisory measures continued to test the components in a "real-world" environment with the currently available resources, rather than in a strictly controlled setting.

Implications for Policy and Practice

The study illustrates the potential of orthogonal design for improving the effectiveness and efficiency of care management programs, if enough observational units such as care managers are available. Orthogonal design combines the rigor of experimental design with the ability to produce rapid results on the effectiveness of multiple enhancements to routine practices in a single experiment. It accommodates planned testing of alternative approaches to multi-component interventions and permits practitioners and researchers to tailor interventions to the target population and test enhancements to routine care. Given that orthogonal design tests combinations of routine and enhanced care, there is no traditional control group; all members receive each component of care, but delivered in a different way or intensity level. Further, the designs can be created to ensure that every member receives the enhanced version of one or more of the components. In addition, orthogonal studies are attractive because the care management staff who implement the interventions all are engaged in testing new variations, because each care manager implements some enhanced care and some routine care options.

An important benefit of an orthogonal design study, as we have seen from the reaction of the participating plan, is increased clarity for care managers on how the plan expects them to deliver the various components of the intervention. Rather than implementing a broad model of care, care managers are told precisely how they are expected to implement each of the components of care management being tested. When routine care is not well defined or the way routine care is implemented differs across care managers, this structure itself can help standardize the care management intervention, leading to less variation in implementation across care managers. Further, fidelity analysis allows the plan to assess the degree to which components were carried out as specified, which can help the plan identify the areas of care management to focus on in their quality improvement efforts. The orthogonal design approach also encourages organizations to create a culture of learning, by providing participants with a rigorous approach for testing out their new ideas.

However, the study also identifies some important difficulties with conducting orthogonal design studies in health care organizations. The types of variations in how care coordination is delivered studied here are likely to generate only moderate size effects on hospitalizations or ER use. To have adequate statistical power to detect such effects, a sizeable number of care manager units are needed, because the variance of these outcomes across care managers is large. Without adequate power, statistically insignificant differences in outcomes between enhanced and routine versions of a care component cannot be taken as valid evidence that the routine (and typically less expensive) version of the intervention is just as effective as the enhanced version. Although the number of care managers (28) participating in this study exceeds the number used in some studies in other fields, it was not sufficient for this study, due to the large random variation in hospitalization rates across care managers.

The study also identifies how hard it can be to change the behavior of even dedicated health professionals. For each of the components, both the enhanced and routine care groups received the assigned component less often than specified in the study. Very few members received at least the minimum number of services (for example, contacts, post-discharge visits, screenings) as specified in the study protocol, and an even smaller percentage of members assigned to enhanced care received services at least as often as assigned. Even though these findings may have been due in part to under-reporting of services provided (evident in care managers submitting few tracking sheets in the second part of the study), it suggests there are various barriers to implementing the interventions that should be addressed. Some of the barriers reported by care management staff include high caseloads, difficulty keeping track of whether and when each member of their caseload had received each of the assigned intervention components, and multiple organizational changes occurring concurrently with the study that divert care managers' attention. This qualitative investigation of these barriers to implementation is just as important for learning as estimation of the effects of the various enhancements.