Toward a National Health Information Infrastructure: A Key Strategy for Improving Quality in Long-Term Care. C. Quality Indicators and Quality Measures


The NF quality is recognized as affecting a myriad of clinical outcomes that encompass functional, psychosocial and other aspects of resident health and well-being. In 2001 the IOM again published a report on the quality of long term care, emphasizing that "defining or evaluating quality of long-term care is fraught with problems, made more difficult by the unevenness of the available empirical evidence.… Opinions about what constitutes excellent, good, or poor quality also are changing and sometimes conflicting. Some of the available information is open to interpretation, and conclusions are sometimes based on personal and clinical experience rather than on empirical evidence."18

Beginning in 1999, CMS began to use MDS data elements for the creation of "quality indicators" (QIs). These QIs are used in the nursing home survey process to indicate areas of potential quality problems which trigger more intensive review during a survey. A subset of these QIs are published on the CMS Nursing Home Compare Website as a source of public information about nursing home quality ( CMS believes that the quality indicators have been sufficiently validated to qualify as true measures of quality, although others question their validity.19 There has never been any published research on the relationship (validity) of these quality indicators to actual nursing home quality. A study of the validity of the more recently developed quality measures was completed (see discussion below). Although conceptually similar, the numerators and denominators within the ratios and the risk adjustments are constructed differently across the quality indicators and measures.

More recently, CMS embarked on an effort to develop "quality measures" (QMs) that could be used nationally to provide consumers with information about nursing home quality that would assist in their decision making process about nursing home placement. In an April 24, 2002 press release, Health and Human Services Secretary Tommy Thompson said, "A key step towards improving quality is getting consumers the information they need to make informed health care choices.… By generating and publishing quality data, we are both helping consumers to make decisions that best meet their needs and creating market incentives for nursing homes to further improve quality."20

CMS hired the National Quality Forum (NQF) to select, through a consensus process, measures of nursing home quality using MDS data elements. However, the NQF was unable to reach a consensus about which quality measures could be used in a national reporting effort and therefore delayed endorsing a nursing home measurement set. NQF members on various NQF councils were unable to approve all nursing home quality measures, particularly noting the need to review evidence of the validity of the proposed risk adjustment methods.21

Based on results from a study on the validity of nursing home quality measures, CMS determined that there was in fact sufficient information to select the new quality measures (derived from MDS data) and that those QMs that would be used in national reporting beginning in November 2002. Under contract to CMS, Abt Associates produced a report that, among other things, examined the inter-assessor agreement (a measure of reliability).22 The study found that all but one of the quality measures was reliable (i.e., a weighted kappa statistic value of .40 or higher).

Controversy surrounds even the new quality measures; in particular there continues to be considerable criticism about the adequacy of the risk-adjustment of these new measures and whether a more rigorous validation study would find relationships between these measures and actual nursing home quality. For example, one of three risk-adjustment experts consulted by the NQF to provide recommendations to CMS about the use of the quality measures as a source of consumer information concluded that " would be irresponsible to report any of these quality indicators to the public" because there is insufficient information "that these risk-adjusted quality indicators accurately identify facilities with quality of care problems."23 Similarly, the GAO recently concluded "although public reporting of quality indicators is a worthwhile goal, it is important for CMS to await and consider input … to identify and evaluate appropriate quality indicators."24

CMS responded, noting their intent to develop more refined measures: "As the process of choosing nursing home measures evolved, it became clear that the NQF needs adequate time to fully consider and discuss the Validation Report and to take into account any lessons learned from the Six-State Pilot Study. Once the NQF reaches consensus and delivers its recommendations, we expect to move to a timely implementation of the updated measures on Nursing Home Compare."25

Clearly, significant challenges exist in providing useful and accurate information about quality in long-term care. Failure to reach agreement on a set of quality measures in nursing homes highlights one of the challenges in measuring nursing home quality. Nursing home quality is a multi-faceted concept. The continued use of the "QIs" in the survey process and as a source of consumer information on the Nursing Home Compare website and the "QMs" as another type of consumer information and the mechanism to identify nursing homes in need of quality improvement underscores the uncertainty about how to measure nursing home quality. The number of QIs and QMs reflect a number of care domains that are of interest to different stakeholders. In addition, many believe that quality of life is at least equally important in understanding the quality of nursing home care,26 but are included in either the QIs or QMs.

View full report


"toward.pdf" (pdf, 237.05Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®