Health Information Exchange in Long-Term and Post-Acute Care Settings: Final Report. Appendix C: Quantitative Evaluation Plan



In this report, we outline a plan for quantitative analyses to assess how the growing, but still relatively rare, participation of LTPAC providers in electronic exchange of health information affects utilization, cost, and quality of care outcomes. Specifically, we consider methods and data to better understand how new care delivery and payment systems are resulting in greater integration of LTPAC providers into HIE arrangements and how outcomes differ in areas where exchange is occurring relative to those where it is not. Our plan is informed by the literature review and conceptual framework model developed as part of this project, stakeholder interviews, and what we learned in site visits to two states.


Awareness is growing that LTPAC providers play a critical role in care coordination and related payment and delivery system reforms intended to improve quality and reduce costs. These reforms include ACOs, hospital and post-acute care bundling, various integrated care delivery models, and Medicare's hospital readmission policy.82 eHIE between LTPAC providers and other providers is a promising and important strategy for achieving the goals of improving care coordination and quality, and reducing the cost of care.

Despite the increased focus on the importance of LTPAC in the care continuum, results from this project indicate that integration of these providers into electronic data exchange is still in its infancy. Exchange that is occurring generally is not the robust, bidirectional exchange typically envisioned in earlier studies regarding the potential for improvements in care delivery and outcomes.83

The basic findings are the following:

  • Lack of funding and the business case for LTPAC providers to participate in robust exchange has led to use of ad hoc and often very local solutions.

  • Even in markets where relatively robust exchange is occurring for acute care providers, including hospitals, laboratories, and outpatient care in clinics and physician offices, LTPAC providers most often are limited to view-only access to clinical documents and partial solutions that may reduce incentives for adopting more functional, interoperable EHR systems.

The ARRA of 2009 established HITECH, to accelerate the digitization of the American health care system through greater adoption and the MU of EHRs and the electronic exchange of health information.84 HITECH created three successive stages of MU regulations designed to ensure that providers use EHRs in ways most likely to improve the quality and efficiency of care and, through the EHR Incentive Programs, provided incentives and financial resources for a subset of largely acute care providers to purchase or upgrade EHRs to achieve MU.85 Certain Medicaid providers may also be eligible for EHR incentives under that program.86

SNFs, HHAs, and other post-acute care and long-term care providers, including NFs, have to date been ineligible for financial assistance under either Medicare or Medicaid EHR incentive program and technical assistance under HITECH. These providers are a critical component of the care continuum, especially for key outcomes of current policy aims such as reducing avoidable hospital readmissions, but lag behind in their adoption of certified EHRs and ability to engage in HIE. At the same time, HITECH's Stage 2 MU requirements and other factors such as readmission penalties have increased interest among eligible hospitals and other providers in electronic data exchange with LTPAC partners and other HITECH ineligible providers. Final rules have not been issued for Stage 3 MU. Specific public integrated care programs such as Integrated Care for Dual Eligible Individuals demonstrations and the Medicaid Health Home option are designed to manage and coordinate care across all providers, including LTPAC providers, facilitated where feasible by HIT. In Medicaid Health Homes, however, even such basic tools as timely notifications of hospital admissions and discharges have proved to be an obstacle, even between eligible providers.87 Accelerating interoperable HIE and data use across the care continuum will require special attention to LTPAC providers and other non-eligible providers and strategies to help them catch up and address their unique barriers.

The IMPACT Act of 2014 is one policy development that has the potential to accelerate LTPAC provider involvement in HIE through its new requirements for LTPAC reporting. The act further specifies that specific data elements within each patient assessment instrument be standardized and interoperable to allow for exchange and use of data among LTPAC providers and with other providers. The IMPACT Act requires that CMS standardize post-acute care patient assessment data, including data with regard to specified patient assessment instrument categories and quality measures. In addition, the IMPACT Act intends for data comparability to allow for cross-setting quality comparison in settings including SNFs, HHAs, IRFs, and LTCHs, and, importantly, it conveys the inclusion of patient-centeredness in its references and requirements related to capturing patient preferences and goals.88 The IMPACT Act also requires that standardized post-acute care assessment data elements be made interoperable so as to support the exchange of such data among post-acute care and other providers in order to support access to longitudinal information and coordinated care. The provisions in the IMPACT Act will drive data standardization in post-acute care settings and will support the use of interoperable HIT systems within the LTPAC and interoperable HIE with and by this sector. Because most NFs and HHAs provide both post-acute and long-term care, these requirements have the further potential to enhance eHIE for long-term services and supports as well as post-acute care.

Analytic Plan Roadmap

The purpose of this quantitative analysis plan is to outline and discuss methods for assessing the impact of eHIE participation by LTPAC providers on key outcomes. Considerations fundamental to any such assessment are understanding the degree to which LTPAC providers are engaged in HIE and the types of data they are able to transmit and access.

The plan is informed by key questions that ASPE seeks to address, such as what is the measurable impact of eHIE on the quality, continuity, and cost of care for LTPAC providers and their trading partners. We also draw on the conceptual framework developed in this project and the information and insights about data quality, availability, and related issues (e.g., data use agreements) gathered through stakeholder interviews and site visits in Pennsylvania and Minnesota.

To provide context, we first briefly describe a generic comprehensive approach to a quantitative impact analysis of LTPAC involvement in electronic heath information exchange. We then summarize the key lessons learned and limitations revealed based on insights gathered from the prior project work in locations in Pennsylvania and Minnesota, particularly with respect to the necessity of taking into account the unique characteristics of localities. Given this context, we provide examples of feasible evaluation analyses that could be conducted in Pennsylvania, Minnesota, or other potential markets where an identifiable "intervention" relating to enhanced LTPAC provider participation in electronic data sharing has occurred. We also summarize the pros and cons of each as an illustration of the type of information that should feed into selection of evaluation sites and discuss parameters that would affect evaluation costs.

Comprehensive Approach to Quantitative Impact Analysis

In the abstract, a comprehensive plan for evaluation of eHIE among LTPAC providers would require the following information to identify evaluation targets and factors that may affect outcomes:

  • In what geographic areas, markets, or types of care delivery organizations is electronic exchange of health information occurring?

    • What technologies are being used, and what types of information are LTPAC providers and their trading partners creating and transmitting? Different types of eHIE could range from view-only portals and ADT alerts to true interoperable eHIE--defined as the sharing information seamlessly, moving discrete data that can be inserted into another system's database and pulled into its EHR fields.89
    • What is the volume and regularity of exchanges per LTPAC admission and discharge?
    • What percent of total admissions and discharges do the electronic transmissions represent?
    • To what extent is exchanged information timely and comprehensive with respect to patient care across the care continuum?


Assessment of the impact of the areas, markets, or systems identified would then aim to answer the following questions:

  • What is the measureable impact of eHIE on the quality, continuity, and cost of care for: (1) the LTPAC providers; and (2) their trading partners?

    • How has eHIE affected specific utilization outcomes of interest, such as 30-day post-hospital discharge readmission rates and hospital admission rates from the ER?
    • How has exchange affected costs for each party of interest (e.g., insurer, organized delivery system, hospital, and LTPAC provider)?
    • Has exchange affected total Medicare expenditures and resource utilization?
    • Do effects differ when assessed from the episode of care (e.g., hospitalization to post-acute to discharge destination) perspective rather than the event (e.g., LTPAC admission) perspective?


To address the first group of questions, we would ideally analyze the types of information LTPAC providers and their trading partners are exchanging. There are two potential points of collection for these data: (1) network servers for public HIOs; and (2) servers supporting private HIEs, such as those of IDSNs and ACOs. Most public HIO participation agreements do not allow the HIO to capture specific information about the nature and type of most clinical data transmitted, however. Thus, network logs would only provide metadata, including the volume of information exchanged and the source and receiver of the information. Such metadata nevertheless could be useful in providing descriptive statistics indicating the geographic range and volume of data exchanged, as well as information about important network nodes and areas where exchange happens less than might be optimal. For private HIEs, more robust data could be available, and these organizations may be willing to share information. For example, during the Pennsylvania site visit, Geisinger was willing to share the number of DSMs that they sent from their EHR system to LTPAC providers during the most recent 30 days, although this would not provide insights on the type of information being transmitted. Data from such a private network could provide richer information, particularly if examined in conjunction with EHR logs.

To address the second group of questions related to the effects of eHIE on patient and provider outcomes, we would examine claims experience of individuals and data on provider characteristics, such as data from the Certification and Survey Provider Enhanced Reports (CASPER)90 and Medicare Cost Reports.91 The strongest evaluation design would be examination of outcomes before and after an identifiable policy or system change (pre/post analysis) for a group of affected participants (or providers) relative to a comparison group not affected by the change (difference-in-difference analysis). In the absence of an actual controlled experiment, we would estimate multivariate regression models comparing patient outcomes, costs, and other outcomes in situations where electronic exchange is occurring to the same outcomes in areas or for providers where electronic exchange is not occurring. Similarly, we could compare regions with robust HIE with similar regions where LTPAC providers lag behind. We would use a propensity score reweighting method, or other method if appropriate, to match the treatment group (i.e., patients in LTPAC facilities that electronically exchange information) with a comparison group based on observable provider, area, and patient characteristics (e.g., socio-demographics and health status) that are available in the given data source. Under this approach, impact assessments would be made based on adjusted comparisons between the experiences of the treatment group and comparison group during the years when exchange occurred.

Lessons Learned from This Project

Unfortunately, quantitative analysis of eHIE in LTPAC is fraught with problems because of the immature state of systems in even the more advanced areas, the fragmentation of systems and technology used across types of providers, and the very local character of exchange solutions. Much of the evaluation methodology for HIE that has been discussed in the literature is hypothetical and applicable only when more robust systems are in place for public exchange--for example, envisioning use of data from HIOs for public and disease monitoring purposes.92 In this section, we discuss major limitations to a comprehensive quantitative evaluation of eHIE based on project findings.

The Urban Institute team conducted case studies and site visits to learn about the following initiatives to enable eHIE with LTPAC providers: KeyHIE in the Northcentral/Northeast region of Pennsylvania, the Fairview Health Services/Ebenezer senior services initiative in Minneapolis, and the Allina Health/BHS initiate, also in Minneapolis. Previous memoranda summarized the eHIE in these markets, particularly as those activities pertain to the involvement of LTPAC providers.93 Specifically, these memoranda summarized key findings from the site visits, efforts to prepare for and implement HIE between LTPAC providers and their partners, and any relevant evaluations underway or completed in these markets. This section summarizes findings from these analyses relating to the feasibility of conducting the comprehensive quantitative evaluation described in the prior section.

Lack of Bidirectional HIE

From a real world perspective with systems as they now stand, a focus on specific use cases of particular policy interest, such as transmission of and access to real-time notifications of hospital admissions, discharges, or transfers, represents an evaluation opportunity: how often and how reliably is transmission to relevant providers occurring and how often do those providers access and act on the information? In turn, the volume and reliability of timely notifications would be hypothesized to affect patient outcomes of interest, such as readmissions. The results of our site visit and qualitative analyses indicate, however, that even among systems further along the planning and early implementation process we selected for examination, obtaining the data underlying such evaluations may be feasible in practice only within integrated systems of affiliated partners, if at all, because of privacy issues and unwillingness of competing organizations participating in an HIO to publish data to the exchange. Even in places where there is a reasonable infrastructure for exchange and a reasonable level of participation, the bulk of exchange occurs among acute care providers--largely hospitals, clinics, and physician offices.

For the country as a whole, the best available information suggests that LTPAC providers lag behind other key providers both in the adoption of EHRs capable of exchange and in the process of the bidirectional electronic exchange of health information.94 Results of our case studies indicate that also is true in the two markets we examined. In both markets, LTPAC providers are generally not involved in robust interoperable HIE with their partners:

  • One of the most commonly found exchange technologies among LTPAC providers in both markets were view-only hospital portals, which are uni-directional and do not capture encounters outside of the hospital or system.

  • Historically, staff at LTPAC facilities in both markets have used a variety of mechanisms, including telephone calls, and faxes to obtain critical patient information. However, using eHIE to gather information, particularly in home health settings, may require use of new technologies such as portable devices, tablets or notebook computers, to capture data during the patient encounter. Moreover, new data formats such as the Transform tool developed by KeyHIE and transport methods, such as Direct, which operate using the new technologies, have been introduced in LTPAC facilities.

  • KeyHIE, a relatively mature HIO covering 53 counties in Northcentral and Northeast Pennsylvania, reports that in April 2015, 82% of its user transactions were through a portal, with only 18% of access being query-based through an EHR. None of the query-based exchange occurred among LTPAC providers, and LTPAC EHRs do not support query-based exchange.

  • Sixty-one long-term care facilities have participation agreements with KeyHIE, but only 18 (11%) are publishing to the exchange. Among the 28 HHAs with participation agreements, only nine (32%) are publishing data. In contrast, 90% of hospitals and 64% of physician practices with participation agreements with KeyHIE are publishing or sharing electronic health data.

  • In 2007, the Minnesota state legislature passed a law stating that "January 1, 2015, all hospitals and health care providers must have in place an interoperable EHR within their hospital system or clinical practice setting." This legislation has contributed to the state's high EHR adoption rate, even among NHs, which are excluded from the mandate.95 However, the definition of EHR in the mandate does not mean a certified EHR that supports data exchange or meets requirements under MU. According to a recent report of Minnesota clinics on the adoption and use of EHRs and HIE, only 10% of NFs and 4% of HHAs in Minnesota currently exchange electronic data using their EHRs.96

  • The Minnesota site visit focused on two initiatives to promote exchange with LTPAC providers in the Minneapolis region: the Fairview-Ebenezer project and the Benedictine-Allina project. One major factor shaping this market is the dominance of Epic EHR systems; the Fairview and Allina health systems both use Epic with the EpicCare Everywhere, an exchange tool used to share patient records with other providers.

Data Access Barriers

There are a number of data challenges in terms of quality, comparability, timeliness, reporting for dual eligibles, and use of data from managed care providers. These challenges depend on the type of data and region under study and also the maturity of the structures and processes in the selected sites.

For the Medicare population, who are dominant users of post-acute care, CMS data are an obvious source for analysis of utilization and cost outcomes and can be used to construct some quality of care measures for both participating providers and any comparison providers. For Medicare FFS beneficiaries, Medicare claims and drug data include most acute and post-acute utilization and spending for beneficiaries, including dual eligibles, and MDS files provide NH utilization for all payers, including Medicaid. These data are available in a uniform format, with a lag of about two years. Medicare claims and beneficiary data and MDS files currently are available through 2013, and data files for 2014 are expected by the end of 2015. Medicare files include a reasonably accurate indicator of Medicaid enrollment. They do not include information on Medicaid financed NF or home health use or cost, but Medicare is the primary payer for most post-acute care for beneficiaries using these providers. Medicare files also are not a reliable source of comprehensive data for beneficiaries enrolled in managed care plans for whom encounter data would be required.97

Medicaid data are effectively not available unless obtained directly from states or providers because of lags in availability. Historically, reporting of eligibility, managed care plan encounters, and claims data to CMS though the Medicaid Statistical Information System (MSIS) has been neither complete nor of uniform quality across states and years. The more analytically friendly, cleaned and processed versions of MSIS data--Medicaid Analytic Extract (MAX) files--produced by a CMS contractor at present are available through 2012 for only 35 states, and their future is uncertain. States are in the process of switching from MSIS to a new reporting system called Transformed MSIS (T-MSIS). At least three states, Colorado, Rhode Island, and North Carolina have stopped submitting MSIS data. The T-MSIS, which ultimately is intended to provide timelier, research-friendly files as part of the CMS Integrated Data Repository. At this point, however, it is still in the development stage, so that both availability and comparability with MAX data for earlier years to examine change over time is in question.

Findings from the case studies provide additional insight into data access in Pennsylvania and Minnesota, and particularly to barriers encountered by earlier evaluation efforts.


In 2010, Abt Associates joined Geisinger as an internal evaluator on two projects, both of which are now finished: Geisinger's Keystone Beacon award from ONC and a HIE grant from the Agency for Healthcare Research and Quality (AHRQ). Both projects aimed to address key questions such as who uses KeyHIE and to what extent can KeyHIE's impact on quality and outcomes be measured using the available data.

Abt's initial evaluation design included both quantitative and qualitative methods. Similar to our ideal quantitative evaluation framework, the Abt evaluation team planned to conduct a difference-in-difference analysis, where the data sources they sought for included:

  • KeyHIE's transactional data, including information on admissions, discharges, and transfers; searches used by clinicians to find a specific patient's health information; documents and data downloaded by clinicians; and documents viewed by clinicians.

  • Claims or billing data to study the utilization of services (e.g., ER use) and readmissions.

However, the Abt evaluation team faced significant challenges in collecting the quantitative data needed to conduct their evaluation. Most of the providers participating in KeyHIE declined to give Abt access to their data for research purposes, perhaps out of concern that they would be handing over their data to a major competitor (Geisinger) who would be able to study patient flows and readmission patterns. The Abt evaluation team, like the other Beacon grant evaluators, was also unsuccessful in obtaining Medicare claims data from CMS because the evaluation would not directly benefit the Medicare program.

Without Medicare claims data, the Abt evaluation team was unable to conduct the difference-in-difference analysis originally planned. And with limited permission to access KeyHIE transaction data and billing data from only a handful of participating providers, the Abt evaluation team was unable to complete a descriptive quantitative analysis.


Data access barriers appear less severe in in Minnesota. SHADAC is the state evaluator for Minnesota's CMS SIM grant. As part of its evaluation, SHADAC is charged with evaluating the SIM-funded development and implementation grants awarded through the e-Health Grants Program, including the Fairview-Ebenezer initiative's 12-month development grant. Of the 12 collaboratives awarded a first round e-Health grant, seven include LTPAC providers.

The goals of the SHADAC quantitative and qualitative evaluation include documenting what is going on, what did not go as planned, and implementation barriers and facilitators. The evaluation also focuses on coordination and transitions outcomes (i.e., improved care transitions, quality of care, and costs) and process measures related to being able to exchange health information and use of technology.

Quantitative data sources that the SHADAC evaluation team plans to use for its evaluation of the e-Health grants include a state-fielded EHR survey and a continuum of accountability matrix. Grantees completed the matrix as part of their grant application so the SHADAC evaluation team will be able to assess change over time. Some grantees completed the matrix as a collaborative, and others as an organization.98

SHADAC is going to try to obtain data for its evaluation from the state's APCD, but those data likely will be used to assess the Medicaid ACOs, not the e-Health grants. At present, access to the APCD data are by law limited to staff at the Minnesota Department of Health (MDH) or organizations working under contract with MDH to conduct research on its behalf.99 The theoretical benefits of APCDs, which are in place in about 14 states, are that the data are likely to be available in a more timely fashion than through Medicare, but their value for external evaluation research relies on the availability of data with identifiers for providers and patients that allow tracking over time. In February 2015, a workgroup recommended that the legislature authorize MDH to release public use files, but provider and patient identifiers would not be included.

Revised Quantitative Evaluation Approaches

In light of what we have learned about the reality of LTPAC providers and HIE, the following describes a feasible approach to address the research questions of interest, relying on the incentives and opportunities presented by IDSNs and ACOs, and the use of interoperable HIT tools among LTPAC providers (e.g., the Transform tool).

An overarching lesson was that having a relatively advanced HIE infrastructure is a necessary, but not sufficient condition for integrating exchange with LTPAC providers into the system. At a minimum, evaluating outcomes of exchange including LTPAC requires the ability to identify locations or organizations where LTPAC providers have been integrated into exchange beginning at some identifiable event defining an "intervention" period for a pre/post design. In practice, it is relatively simple to define the event, such as initiation of a program. However, given that implementation is a process that can take substantial time to complete, the intervention period during which change may reasonably be expected can be more difficult to clearly delineate. Thus, it may important consider stages of implementation, early operation, and maturity in evaluation design. In order to implement the stronger difference-in-difference design, the challenge is finding a credible comparison group not exposed to the intervention to examine over the same time periods.

The move toward ACOs under the ACA comes from the need to contain costs in Medicare, but interest and implementation of the model extends to Medicaid programs and predates the ACA. ACOs are networks of physicians and other providers that are held accountable for the cost and quality of the full continuum of care delivered to a group of patients. The ACA authorized Medicare to contract with ACOs with the aim of achieving the "triple aim" of improving quality of care, improving population health, and reducing costs. Similar to the IDSNs of the 1990s, the premise is that ACOs will accomplish these aims by coordinating care, managing chronic disease, and aligning financial incentives for hospitals and physicians. In theory, ACOs can improve quality and lower costs using several methods, including disease management programs, improved care coordination, alignment of incentives for physicians and hospitals via shared savings, use of non-physician providers, and the formation of PCMHs.100 Over the past five years, both the number of participating ACOs and the number of participation options for them have grown dramatically, while potentially generating $400 million in savings for Medicare.101

ACOs are increasingly turning their attention to post-acute providers to better manage cost and quality across the care continuum. A recent descriptive analysis of the structural and functional provider relationships finds that ACOs are expanding their partnerships and developing relationships with LTPAC providers. For example, more than half of Pioneer ACOs have core or structural partnerships102 with HHAs, more than 40 with hospice facilities, and more than 20% with NFs.103 ACOs are also using functional relationships104 to extend the care continuum beyond what can be achieved with care partners alone, particularly for urgent care and post-acute care providers.105

An evaluation of eHIE among ACOS/IDSNs and partnership/acquired LTPAC providers would aim to address the following research questions:

  • Prior to forming a partnership with an ACO/IDSN, what type of EHR systems were LTPAC providers using? Were they electronically exchanging health information with other providers? What type(s) of information were they exchanging and how?

  • How did LTPAC provider's EHR system and technology change after forming a partnership?

  • Once the partnership started, what type of information was exchanged within the system? Outside the system? What technology is being used to exchange this information?

  • How did patient outcomes, utilization, and costs change after the partnership was formed?

There are a number of advantages to evaluating HIE between LTPAC providers and their exchange partners within an ACO, or similar model of care (e.g., IDSN) setting. First, as previously mentioned, this is priority policy area in the Medicare program and findings from this evaluation would complement prior and ongoing evaluations of the ACO model. Second, it would likely be easier to obtain data by partnering with a single ACO or IDSN as opposed to partnering with an HIE that represents multiple organizations. As shown with the prior Abt evaluation, providers participating in KeyHIE declined to provide access to their data for research purposes out of concern that they would be handing over key information to a major competitor. It might also be easier to access CMS claims data because an ACO-focused evaluation would directly benefit the Medicare program. Finally, our case studies and prior research106 indicate that a key advantage of private HIEs within IDSNs and ACOs is that eHIE, particularly with LTPAC providers, is more robust within these private organizations than in state-sponsored HIEs.

The major drawbacks of this approach, however, are that this type of evaluation would be limited and not provide an overall assessment of HIE within a region or market. The organizations that ACOs or IDSNs connect are sometimes restricted based on strategic and proprietary interests. For example, hospitals may choose to connect with the ambulatory care and post-acute care providers with whom they would like to more closely affiliate, regardless of existing referral patterns in the market. This complicates overall participation in HIE, data re-use, and ultimately care coordination.


Given what was learned in Pennsylvania and Minnesota, a first step in an evaluation would be an evaluability analysis of candidate sites, using interviews with relevant informants within proposed ACOs/IDSNs and focusing on such critical issues as willingness to participate in an evaluation, data availability and access, existence of comparisons, and volume of exchange with LTPAC providers occurring. Having identified the most promising site or sites, we would use a mixed methods approach to address the research questions listed above. We would conduct a survey of LTPAC providers within the selected "treatment" ACO/IDSN, and ideally, comparison group providers. This survey would assess the technology used, the regularity and frequency of use, the primary objectives of use, the motivations to engage in exchange, implementation challenges, and the benefits realized. To complement the survey and fill in any potential gaps in understanding of the exchange environment in which the LTPAC providers are operating, we would conduct additional targeted case study interviews with key decision makers within the ACO/IDSN across the care continuum. Finally, we would conduct quantitative data analysis with the best available data, which would depend on the location and organizations selected.

Analyses could draw on claims data, EHR and other clinical data, and measures developed from the survey data. Claims data could provide direct measures of patient encounters (e.g., readmission rates) and some treatments and medications. Claims data are accessible from government entities (states, CMS) and from private payers, and increasingly, states and other stakeholders are working to to establish APCDs. Based on the experience of a recent ACO evaluation, as a federal contractor, we anticipate that CMS would be willing to approve a data use request for research identifiable Medicare claims and enrollment data from the CCW and ACO-specific data that contains identifying information for participating providers and aligned beneficiaries and their corresponding ACOs.107 A critical issue for the value of the latter information is the ability to identify ACOs that have integrated LTPAC facilities into their networks.

While claims data is currently the main data source used to calculate outcome measures, it might be feasible to use clinical data from EHRs. Much of the information in claims data is now being captured by EHRs and is available at the system level. A notable limitation of EHR data, in contrast to claims, is that comparable data may not be available for potential comparison groups. In addition, the possibility of data sharing arrangements would need to be explored early on.

We would attempt to find a comparison group that consists of similar FFS Medicare beneficiaries in markets not served by an ACO and who do not receive care from an ACO/IDSN. Alternatively, comparisons might be feasible between IDSNs or ACOs in locations where there is a distinct difference in LTPAC participation across networks.

Examples of Potential Settings


The Geisinger Health System is one example of an ACO-like model that has incorporated LTPAC providers and continues to do so. In contrast to Abt's evaluation of KeyHIE as a whole, we would only assess eHIE among LTPAC providers and their partners within the Geisinger system.

Results from our site visit suggest that in a departure from its traditional business strategy, Geisinger is increasingly becoming interested in purchasing LTPAC providers. Initially, Geisinger focused mostly on acquiring HHAs. For example, in 2014, Geisinger acquired Sun Home Health and Hospice.108 Several respondents indicated that after completing the acquisition of these HHA sites, Geisinger has focused on the NF sector. One interviewee indicated that Geisinger is trying to develop a "SNFist model" where providers can make decisions at the NF site instead of taking the patients back to the hospital.

From an evaluation and policy perspective, a unique aspect of Geisinger is their development of the Transform tool. Geisinger's 2010 Beacon Community grant provided funding for LTPAC provider outreach and the development of the Transform tool. The KeyHIE Transform tool takes MDS and OASIS data and converts the clinically meaningful information to a CCD. This CCD can be exchanged using KeyHIE so that the all participating providers could access the CCD. The Transform tool is inexpensive relative to the cost of interfacing with an exchange, which appeals to LTPAC providers who may otherwise not be willing to participate in information exchange. The Transform tool was launched in 2013 and provides a unique opportunity for a quasi-experimental design evaluation, with the "pre" period being before 2013 and the "post" period including 2013 and later years. A key question to address in an evaluation would be whether LTPAC providers acquired by Geisinger use the Transform tool and/or whether their EHRs were integrated into Geisinger's system. Another key issue to address is the extent to which Geisinger is working with LTPAC providers that they did not acquire, and the extent to which these providers use Transform.


The Benedictine-Allina project also represents an example of an ACO-like model that has incorporated LTPAC providers. Allina Health is a non-profit health care system based in Minneapolis that owns or operates 14 hospitals and more than 90 clinics throughout Minnesota and Western Wisconsin. Allina Health is participating in the Medicare ACO program. The BHS is one of the largest senior care organizations in the United States, with 36 NFs, 25 ALFs, and one HHA.

We would propose to evaluate the March 2013 e-Health Connectivity Grant as a policy intervention. In 2013, Benedictine received $375,000 from the state of Minnesota to develop MatrixCare software so that it can exchange CCDs with Allina's Epic system peer-to-peer. This new software was launched in December 2013, creating a "post-intervention" period of 2014 and later.


Colorado represents a number of potential evaluation opportunities, from the perspective of delivery system reforms involving both Medicare and Medicaid, HIE infrastructure, and data. Colorado also still is largely a FFS state, although its SIM plan includes transitioning to capitation over the next several years.

Colorado's Medicaid ACC, launched in 2011, draws on seven RCCOs state-wide that develop networks of providers. The RCCOs are responsible for connecting beneficiaries with needed clinical and other services and fostering communications between providers to improve care coordination. The ACC did not initially enroll dually eligible beneficiaries, but it expanded membership to include them in 2014 under the state's Financial Alignment Initiative demonstration.109 The focus will be on improving chronic disease management and transitions between hospitals, rehabilitation hospitals, NFs and community residence.

Physician Health Partners, a medical management company based in Denver, became a Medicare Pioneer ACO in 2012 in partnership with the Primary Physician Partners and South Metro Primary Care. The ACO serves about 30,000 Medicare beneficiaries in the seven-county metro area, and in 2014 began participating in the Medicare Shared Savings Program.

The state has a large and well-established regional HIO, the CORHIO, which in 2011 received a challenge grant from the ONC to increase connections with LTPAC facilities including post-acute rehabilitation hospitals, NHs, assisted living centers, home health care agencies and hospice. As of June, the CORHIO network included 48 hospitals, more than 2,600 providers, 131 long-term care facilities, 39 behavioral health centers, four large medical laboratories, EMS providers, the Colorado Springs Military Health System, and the state health department.

CORHIO provides bidirectional exchange with provider EHRs, but most LTPAC providers are using secure, web-based query access to a community health record system from which they can have real-time access to patient information and the ability to generate CCDs, regardless of whether they have an interoperable EHR. In 2015, CORHIO received a new ONC grant to support implementation of the Transform tool, which would allow LTPAC providers with or without EHRs to translate information from MDS and OASIS assessments and share them through the HIO. Thus, two possible evaluation points are defined by the initial 2011 challenge grant to increase connections with LTPAC providers and the 2015 grant to implement Transform.

The state also has an APCD, administered by the non-profit CIVHC. The APCD was established by the legislature in 2010, and as of January 2015, its data warehouse reported health insurance claims from Medicare, Medicaid, and the 20 largest health plans for individual, large group fully-insured, small group and some self-insured lives, as well as Medicaid and Medicare. The claims represent more than 3.5 million unique covered lives and 65% of the insured population in Colorado. Medicare claims for 2009-2011 and 2013 data for commercial payers and Medicaid is currently available through the Data Release Review Process and will be available on the data website in 2015.110 Unlike Minnesota, Colorado allows release of APCD data at varying levels of detail and specificity for research under a CMS-like review process requiring "that the intended use supports reaching the Colorado Triple Aim of better health, better care, and lower costs."

Setting Strengths and Limitations

It is important to consider several factors while conducting an evaluability assessment of the proposed sites. Table C-1 uses the three settings described above to illustrate the type of questions to be addressed in selecting an evaluation site or sites. This table provides cross-setting information on several factors, including existing contacts, the availability specific settings and interventions, and the relative ease of access to quantitative data.

Each site has a specific setting and intervention to evaluate. We would evaluate the launch of the KeyHIE Transform Tool in 2013 in Pennsylvania, the 2013 e-Health Connectivity grant in Minnesota, and the 2011 and/or 2015 HITECH grants in Colorado. However, the magnitude of these interventions is likely to vary across settings. For example, the Transform Tool has a more global focus, with the ability to be used by more providers, relative to the smaller e-Health Connectivity grant intervention. Similarly, Colorado's new grant to implement Transform has a broader application than the earlier grant. In contrast to Pennsylvania, however, it might be easier to find a valid comparison group in Minnesota and Colorado, where there is a relatively high prevalence of similar health care systems in the region, compared with Pennsylvania, where Geisinger is one of the most unique and advanced IDSNs in the nation. Colorado is likely the best site in terms of claims data access due to the availability of APCD data to researchers.

TABLE C-1. Cross-Setting Comparisons
  Pennsylvania Minnesota Colorado
Contacts from site visits? Yes Yes No
Specific setting to evaluate? Geisinger BHS-Allina Health Physician Health Partners and other potential options
Intervention for evaluation? 2010 Beacon Community grant to develop its Transform tool to convert MDS and OASIS data into a CCD (launched in 2013) March 2013 e-Health Connectivity grant for exchange and use of CCDs between BHS (long-term care system) and Allina (hospital system), via MatrixCare and EpicCare software (launched December 2013). 2011 HITECH grant to expand LTPAC access

2015 HITECH grant to implement Transform tool

Comparison group feasibility? Challenging, due to Geisinger's uniqueness Relatively easy, due to high prevalence of IDSNs in Minneapolis region Relatively easy, due to multiple regional networks in operation.
Medicare ACO program? [could improve likelihood of CMS approval for claims data] Medicare Shared Savings Plan ACO (Keystone ACO) Pioneer ACO (Allina) Pioneer ACO (Physician Health Partners)
APCD? No Yes, but not accessible to evaluation except for state contractors Yes

General Limitations

There are some general limitations that apply to all settings as well. First, small sample sizes could hinder evaluation efforts at each of the potential sites, especially given the limited post-implementation period of the interventions considered and the relatively low prevalence of NH residence, hospitalizations and post-acute care. About 20% of all Medicare enrollees use hospitals in a year; about 5-6% use SNFs, Part A Home Health, and Part B Home Health, respectively, not adjusting for enrollees using more than one type of post-acute care; and 3% of those age 65 or older reside in NHs.111 Similarly, the interventions to be evaluated are not discrete, that is, implementation was likely phased-in over a relatively long period of time. Second, sample selection could bias any potential estimates since these interventions were not randomly assigned, and each site could also suffer from omitted variable biases as multiple policy interventions and changes to the health care were occurring during the same analysis period. Third, research organizations in any of these settings will likely need to obtain multiple IRB and data use agreement approvals, thus creating substantial time costs in obtaining data. Finally, across all settings, it will likely be very challenging, if not impossible, to directly obtain data from providers (e.g., EHR data) due to privacy and security concerns. However, researchers could potentially obtain aggregated EHR data for sites that cooperate.

Cost Considerations

The costs associated with any given evaluation would depend on the size (e.g., number of providers, number of patients), and the type of data to be used. Any cost estimate would need to include the cost of data acquisition, such as the cost of an LTPAC provider survey; the cost of identifying, requesting, and, if applicable, negotiating for claims and other data. If a comparison group design is selected, additional survey and data collection costs would need to be factored in. Costs of developing analytic files and conducting analyses could vary significantly depending on the source and type of data. For example, standard format Medicare and assessment files, which are widely used and familiar to researchers, likely would be less expensive to process than data from other sources, which might require considerable interaction with the organization providing the data to specify the data needed and understand its format or to oversee work done within the organization.

The Urban Institute has experience working with contractors with the capability of doing provider surveys. Overall costs depend on various factors, including the number of providers (sample size), the length of the instrument, financial incentives or other methods to increase response rates, etc.

The cost of CMS data depends on the method of access. The Virtual Research Data Center is priced based on an access fee of $40,000 per year for a single user ($25,000 for federal contractors), and a one-time project fee that is data specific and depends on the cohort extracted. No additional charges are incurred for adding years of data to a cohort, but changes in the cohort that require additional extracting generate charges. Projects requiring more than 500 GB of space also have to pay $2,000 for each additional 500 GB block of space. The costs of data obtained from the CCW depend on whether the CCW is asked to extract data for a cohort (versus having the requester provide a "finder" file of beneficiary identifiers, in which case there is no extract charge) and the complexity of the algorithm for extraction. Charges for Medicare data are based on the size of the cohort, the number of services for which files are requested and the number of years of data. For example, for a cohort of 1 million or fewer beneficiaries, inpatient, outpatient, SNF, and durable medical equipment files are $2,000 per year, Carrier claims are $4,000 per year, and Part D Event data are $5,000 per year. There is no charge for beneficiary files if service files are requested.

Summary and Conclusions

This report outlined a feasible structure for evaluations to assess the impact of LTPAC involvement in eHIE on outcomes of particular interest to federal and local policymakers. For context, we first recapped the relatively undeveloped state of HIE among LTPAC providers; reasons for lagging implementation and use of exchange tools, such as DSM, rather than widespread use of certified interoperable EHRs and bidirectional exchange; and current policy changes and initiatives that may accelerate HIE. Chief among those are the IMPACT Act--which requires new levels of reporting by LTPAC providers and specifies that data be standardized and interoperable to support HIE--and a new ONC focus on grants to reach providers such as LTPAC providers who were ineligible for previous incentive programs.

Fundamental needs for such an evaluation are the following:

  • The ability to examine outcomes before and after an identifiable policy or system change (pre/post analysis).

  • Ideally, the ability to identify a comparison group not affected by the change and analyze outcomes over the same time period (difference-in-difference analysis).

  • Access to claims data (pre and post) to assess outcomes.

  • Information about provider characteristics and the volume and types of eHIE happening (e.g., the volume and regularity of exchanges per admission or discharge).

Based on what we learned from earlier project activities, particularly with regard to the state of LTPAC eHIE involvement and data access in our two site visit states, we concluded by outlining a high-level approach for evaluations based on identifying appropriate ACOs or IDSNs and comparing outcomes for their members using LTPAC services. We provided three examples of ACO/IDSNs in Pennsylvania, Minnesota, and Colorado where such evaluations would be possible. There are, of course, other possibilities among provider partners in Accelerating Change and Transformation in Organizations and Networks (ACTION III),112 states originally considered in the early stages of this project, as well as in additional potential in other states that, like Colorado, received new ONC grants to implement the Transform tool (Delaware and Illinois), or where Transform is already being implemented, including Pennsylvania's KeyHIE.

Identifying appropriate comparisons remains a challenge in any of the settings, but focusing on a discrete set of LTPAC partners in selected ACOs/IDSNs provides a more manageable structure than, for example, starting with LTPAC providers and identifying the multiple hospitals and practices with whom they interact. A simplifying approach to data acquisition would be to focus on Medicare FFS beneficiaries, for whom CMS claims and beneficiary information would represent all spending except Medicaid financed long-term care for beneficiaries who are eligible for both programs. However, partnerships with ACOs/IDSNs may hold the potential for both claims data and analysis of the volume and nature of data exchanges if data access and privacy issues can be addressed.

View full report


"LTPACsetting.pdf" (pdf, 1.22Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®