An Environmental Scan of Pay for Performance in the Hospital Setting: Final Report. Data Collection and Validation


  • Data Collection.
    • Data Sources. As with measurement selection, a key driver of data sources used was the goal of minimizing hospital burden. As such, there was heavy reliance on the use of data already collected by other entities (e.g., CMS, JCAHO) (14/23) or administrative data (either their own or from state reporting efforts). However, the clinical information used to populate measures for national measurement efforts is largely still being gathered from medical records, as opposed to claims data or EHRs, so this still represents a significant burden to hospitals. Although EHRs in particular are often touted as a panacea for the burden of data collection, many organizations do not yet have EHRs. And even if they do, the data captured by EHR are in text versus data fields, which makes the tool difficult to use for measure construction. Even with an EHR, manual review is still required to extract relevant information. Other data sources used by program sponsors included (1) hospital self-reports, such as formal attestations (e.g., Leapfrog) or informal, in-depth conversations (e.g., with small programs) (16/23); (2) plan administrative/claims data (13/23); (3) patient experience survey data (10/23); and (4) national databases (3/23).6
  • “Small-numbers problem.” Lack of an adequate number of cases was mainly an issue for hospitals that were small and/or CAHs, according to the sponsors with whom we talked. However, even for larger hospitals, a small number of events could occur; and if the data were based solely on a single payer’s data, the numbers would be insufficient for producing a stable score. Sponsors reported addressing the small-numbers problem primarily by using all payer data (versus only sponsor data) to score hospitals. Additionally, some sponsors allowed the data to drive which measures were tracked—by looking to see which measures had substantial patient volume. Another approach was to use participation in quality improvement activities or implementation of health information technology. Few sponsors reported using composite measures7 or multiple years of data, which borrow strength across the data to address the small-numbers problem.
  • Timeliness. Timeliness of data was a concern, especially for quality improvement purposes. According to many sponsors, the typical lags of several months to half a year or more—for data collection, cleaning and processing, validation, and reporting—rendered the information useless to hospitals for improving performance in real time. These lags also affected the length of time between actual performance and when incentive payments were made, leading to a disconnect between these two events. Sponsors expressed a desire to obtain data as close to real time as possible in order to strengthen the impact of feedback to providers and other hospital staff.
  • Accuracy. Sponsors expressed concern about the accuracy of coding administrative data, noting that hospitals potentially face the conflicting goals of coding to increase reimbursement versus coding to reflect care that was actually provided.  
  • Data Validation. Almost no sponsors were engaged in their own validation of the data used to score hospitals. Instead, they relied heavily on the audit functions of the organizations that originally collected the data (e.g., CMS, Joint Commission). When measures are generated from all-payer claims data, any validation that occurs typically consists of a review of the final performance scores by hospitals prior to payout and/or public reporting of results. Sponsors indicated that it was too labor intensive and expensive to validate data. While sponsors recognized that CMS and the Joint Commission may not have foolproof validation methods in place, many reasoned that “if it is good enough for the government or Joint Commission, it’s good enough for us.”

View full report


"PayPerform07.pdf" (pdf, 1.22Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®