Analysis of Crosscutting Medicare Functional Status Quality Metrics Using the Continuity and Assessment Record and Evaluation (CARE) Item Set

1. Introduction and Background

1.1 Purpose of the Contract

The purpose of this contract was to provide the Assistant Secretary for Planning and Evaluation/Health Policy (ASPE/HP) and the Centers for Medicare & Medicaid Services (CMS) with recommendations for functional status quality metrics to be used across acute and post-acute care (PAC) providers, including inpatient rehabilitation facilities (IRFs), long-term care hospitals (LTCHs), skilled nursing facilities (SNFs), and home health agencies (HHAs). RTI International used data from the uniform assessment data set—the Continuity Assessment Record and Evaluation (CARE) item set, which was collected during the CMS PAC Payment Reform Demonstration (PAC-PRD)—for this work. The CARE data provide standardized patient assessment data, including admission and discharge functional status data and other clinical data for factors affecting functional outcomes, in these four types of PAC providers.

The PAC-PRD was mandated in Section 5008 of the Deficit Reduction Act of 2005 with the goal of collecting data to inform payment reform recommendations. The demonstration examined the relative costliness and outcomes of patients admitted to different types of PAC providers. The work in the PAC-PRD differs from past attempts in this area because it uses a single, standardized assessment data set—the CARE item set—for measuring the clinical complexity of patients in acute care hospitals and all four types of PAC providers. Data generated from use of the CARE item set during the 3-year demonstration, which concluded in December 2010, are a valuable source of uniform patient assessment data to examine the development of aligned, crosscutting quality metrics. Specifically, analysis of functional status data generated from the standardized CARE item set is of particular interest to CMS to develop crosscutting functional status quality metrics appropriate for use across a wide range of diagnoses and across PAC providers.

In this project, RTI used CARE data to explore the development of risk-adjusted facility-level functional status outcome measures that are applicable across the different types of providers. The beneficiary sample included any beneficiary in the PAC-PRD initial and supplemental phases (Phases 1 and 2) of data collection efforts. The work also included a systematic review of prior work in this area, including existing functional measures used by other payers or provider groups.

Current quality reporting, payment, and value-based purchasing programs have been largely developed within each type of provider. Historically, the setting-based framework of Medicare does not encourage collaboration or shared accountability among providers in either care coordination placement decisions or quality improvement efforts. In addition, each PAC provider collects patient assessment data that are unique to that type of provider. The current patient assessment data sets include the Minimum Data Set (MDS 3.0) used in the SNFs, the Inpatient Rehabilitation Facility–Patient Assessment Instrument (IRF-PAI) used in the IRFs, and the Outcome Assessment and Instrument Set (OASIS-C) used in the HHAs. LTCHs began collecting selected quality-related clinical data using the LTCH CARE data set on October 1, 2012. Although similar clinical and functional status data are collected on the MDS, IRF-PAI, and OASIS, the item definitions, measurement scales, data collection procedures, and time frames differ such that data are not directly comparable.

The CARE item set is a logical source of patient assessment information to be analyzed for development of aligned, crosscutting quality metrics. Examining the potential development of crosscutting functional status quality metrics using the CARE tool is an important next step.

1.2 Project Activities

In considering the development of crosscutting functional status metrics, RTI conducted an environmental scan, identified key analytic issues pertinent to functional status quality metric development (discussed in detail in Section 4 of this report), conducted data analyses to address these analytic issues, and convened a technical expert panel (TEP).

The TEP consisted of 11 clinical experts along with participants from ASPE, CMS, and RTI. At the meeting, held on August 15, 2012, RTI presented details on the analytical issues to consider in constructing the functional status quality metrics. The purpose of the meeting was to gather feedback from the TEP on various methodological approaches.

This report summarizes the work, including a summary of the environmental scan and a discussion about each of the key analytic issues, and details the comments and perspectives of members of the TEP members.

2. Summary of Environmental Scan

2.1 Functional Status

The World Health Organization's International Classification of Functioning, Disability and Health (ICF) describes the term "function" as an umbrella term that encompasses all body structures and functions, activities, and participation in daily life (World Health Organization, 2001). Examples of functional status within the components of body structures and functions include swallowing and bladder and bowel continence. Examples of functional status within the area of activities would include eating, bathing, dressing; in the area of participation, examples would include working and participating in recreational activities.

In describing the importance of functional status, the National Committee on Vital and Health Statistics, Subcommittee on Health (2001), noted,

Information on functional status is becoming increasingly essential for fostering healthy people and a healthy population. Achieving optimal health and well-being for Americans requires an understanding across the life span of the effects of people's health conditions on their ability to do basic activities and participate in life situations in other words, their functional status.

As noted above, functional status is a broad term that covers various components and several levels (e.g., body, person, society). For this work, we focused on the functional status within the area of activities.

The concept of functional status in the area of activities has not been measured directly. Unlike height or weight, which are measured in a uniform manner using established standardized units of measure (i.e., inches, centimeters, pounds, and kilograms), functional status has no standard unit of measure or scale. Instead, functional status has traditionally been documented using subsets of specific daily activities, such as eating, bathing, walking, and stair climbing, and a person's performance while completing these daily activities. A patient's level of performance for each daily activity is reported using a rating or response scale, which is a multiple-point response option set that ranks the patient's performance level.

One important consideration about measuring functional assessment status across all types of providers is that certain functional status activities may not be relevant, or feasible to collect, for all patients in all types of settings. For example, dressing with street clothes may not occur in some long-term care hospitals, because patients are medically acute and they wear hospital gowns because of their use of medical equipment and devices. Assessment of upper body and lower body dressing on a functional assessment instrument typically requires the use of street clothes. For patients admitted to SNFs, walking 150 feet may be too challenging at the time of admission, but walking this distance may be a goal for discharge. For patients admitted to IRFs, stepping up and down a full flight of stairs may not be completed at admission, but it could be a discharge goal. Therefore, patients may have missing data for functional status activities depending on their clinical acuity, because it is not safe or feasible for them to perform the activity.

A great deal of the research focused on measuring functional status of PAC rehabilitation patients has used data from the FIM^® instrument, formerly known as the Functional Independence Measure.¹ The FIM^® instrument is the functional assessment instrument that is included in the IRF assessment tool known as the IRF-PAI. Early analyses of the FIM^® data showed that functional status consists of two concepts or constructs: motor function and cognitive function (Linacre, Heinemann, Wright, Granger, & Hamilton, 1994). For the FIM^® instrument, motor function—sometimes referred to as physical function—combines mobility (the ability to move around), self-care (the ability to attend to personal activities), and bowel and bladder management. The second construct identified within the FIM^® instrument was cognitive function, which includes two communication activities (comprehension and expression), as well as social interaction, problem solving, and memory. More recent research using the Activity Measures for Post Acute Care (AM-PAC), a self-report functional assessment scale, has suggested that motor activities should be separated into two separate constructs, mobility and self-care, when analyses include patients with different diagnoses (Haley et al., 2004; Haley et al., 2011; Jette, Haley, & Ni, 2003; Jette et al., 2007). These constructs are distinct because patients with different primary diagnoses (e.g., stroke, hip replacement, spinal cord injury) will have different patterns of motor activity limitations. Self-care limitations are more likely associated with medical conditions affecting the use of the arms and hands, whereas mobility activities are associated with balance and use of the lower extremities. Therefore, using subscales of mobility or self-care may provide more precise measures of patient functional status outcomes than a combined motor measure that includes mobility and self-care as well as bowel and bladder management.

We report the environmental scan on quality metrics in two sections: motor functional status and communication, problem solving, and memory.

2.2 Motor Functional Status (Self-Care and Mobility) Quality Metrics

Although many functional assessment instruments exist, only a few instruments or items have been used to construct functional status quality metrics for performance measurement at the facility level. Some functional assessment data have been aggregated at a facility level and have been used for internal quality reporting at the unit or facility level or as a quality metric as part of accreditation requirements, such as the Joint Commission's ORYX initiative (The Joint Commission, 2012).

IRF-PAI/FIM^® motor function data have been aggregated at the facility level and have been used by the Joint Commission as ORYX quality metrics since approximately 1997. The Joint Commission recently announced that it would suspend reporting requirements for its ORYX non-core measures, such as the IRF-PAI/FIM^® metrics, for accredited LTCHs and IRFs effective January 2013. This decision reflects the Joint Commission Board of Commissions' recognition of the evolving national measurement priorities and movement toward the use of standardized federally mandated performance measures for LTCHs and IRFs and the field's concerns about disparate data collection and reporting requirements(The Joint Commission, 2012).

No National Quality Forum (NQF)-endorsed functional status metrics are routinely used in IRFs, SNFs, or LTCHs, even though IRFs and SNFs collect functional status data (i.e., IRF-PAI, MDS).

A review of the NQF's list of endorsed performance measures found few functional status quality metrics, and several were process measures that involved the collection of functional status data. The current NQF-endorsed functional status quality metrics include four metrics that use the OASIS data set used by HHAs, seven that use the outpatient Focus on Therapeutic Outcomes (FOTO) functional assessments items, two that use the AM-PAC, one for patients with chronic obstructive pulmonary disease (COPD), and one focused on functional decline among long-term care residents, as well as two process measures.

The 17 NQF-endorsed functional status quality metrics are shown in Table 2-1.

2.3 Memory, Problem Solving, and Communication

Cognitive function may be considered one domain encompassing many subdomains, such as communication, problem solving, and memory. Recent research has questioned whether these cognitive abilities should be split into two factors, with communication representing a single construct and problem solving and memory representing another construct (Hula, Doyle, & Austermann Hula, 2010).

The area of communication itself can be challenging when considering development of a quality metric. Communication disorders are important to address, because these disorders can lead to isolation, depression, and loss of independence. A patient may have difficulty communicating because of an impairment of voice, speech, language, hearing, cognition, or some combination of these. The rate of recovery and the expected discharge outcomes for patients with communication problems will vary based on the etiology. For example, one patient may have difficulty with expression because of a motor speech disorder manifested as weakness, slowness, or incoordination of speech. Another patient may have aphasia, a language disorder that may involve spoken and written language expression, auditory comprehension, and reading and writing abilities. Still another may have communication difficulties due to a cognitive impairment.

Many of the current assessment instruments include items that focus on the ability to communicate; typically, the body structure or function that leads to the communication problem is not reported. Yet, knowledge of the etiology of a communication limitation is important to consider when measuring the outcomes of care with a quality metric.

Cognitive and communication status have been more challenging to measure for clinicians who are looking for a brief assessment instrument. Several brief cognitive status instruments are used clinically or in research. For example, the Brief Interview for Mental Status (BIMS) is included on the MDS 3.0 and was used in the PAC-PRD CARE Item Set. The BIMS documents problems with short-term memory and recall by asking patients to remember and repeat three common words ("sock," "blue," and "bed"). However, measures of cognition or communication have not been used extensively for provider-level analyses. The FIM^® cognitive items (comprehension, expression, social interaction, problem solving, and memory) have been used as an ORYX measure for IRFs.

The American Speech-Language-Hearing Association developed functional communication measures (FCMs), of which seven (shown in Table 2-2) received time-limited endorsement² from the NQF in 2008. Among these NQF-endorsed measures, one is focused on memory, four are focused on communication, one is focused on attention, and one addresses reading. However, these measures are under review and may be withdrawn from NQF consideration.

Several challenges to developing cognition functional status metrics include the need for detailed data on the patient's functional status at admission and discharge and the reason for limitation (e.g., motor speech disorder, language disorder). The reason for the limitation is important to know, because it may affect the expected recovery trajectory (i.e., rate of improvement) and thus may be an important risk adjuster.

3. Analytic Approach

3.1 Description of Data

As previously mentioned, RTI's primary data source for this project was the CARE Item Set data, which were collected as part of the PAC-PRD. The PAC-PRD began collecting data in April 2008 and continued through December 2010. Data were collected in geographically diverse markets from 206 providers across two phases of data collection. The markets were chosen to represent both rural and urban populations, as well as areas with different supplies of PAC providers. For example, Dallas, Texas, was chosen to represent an area of high PAC supply because LTCHs, freestanding IRFs, a large number of HHAs, and SNFs all operate in that market. In contrast, Wilmington, North Carolina, was chosen to represent an area of low PAC supply because of the absence of LTCHs and freestanding IRFs and the relatively smaller volume of providers per beneficiary population. Each participating provider collected patient assessment data on 200–250 Medicare beneficiaries over a 6- to 9-month data collection period. The market areas in the first phase included Boston, Chicago, Rochester (New York), Louisville (Kentucky), Lakeland/Tampa (Florida), Lincoln and Omaha (Nebraska)/Sioux Falls (South Dakota), Dallas, Sacramento/San Francisco, Wilmington (North Carolina), Portland (Oregon)/Seattle (Washington), and Columbia (Missouri). The second phase of data collection targeted more complex patient populations and added providers in other parts of the country, including California, New Hampshire/Maine, New York/New Jersey/Pennsylvania, Virginia/ Maryland/District of Columbia, North Carolina, and Ohio/Michigan.

Participating providers included short-stay acute hospitals, which provided standardized information on patient severity (at admission [in Phase 2] and discharge), and PAC facilities (LTCHs, IRFs, HHAs, and SNFs), which submitted patient severity information at admission and discharge. The PAC-PRD work focused on predicting variations in resource intensity and outcomes associated with each setting. This work expands on the initial examination of functional change to develop facility-level functional status quality metrics. Use of the standardized CARE tool will enable the Department of Health and Human Services to provide critical analyses on cross-setting functional status outcomes.

The CARE tool was designed to measure differences in patient complexity, resource needs, and outcomes for patients in acute and PAC settings. The major types of items collected by the CARE item set include

• administrative information, such as provider, beneficiary, and payer data;
• admission information, including prior use and premorbid status;
• current medical items such as diagnoses, procedures, major treatments, skin integrity, and physiologic factors related to the current admission;
• cognitive status and other interview-based items to measure orientation, risk of delirium, depression, and pain;
• functional status, including self-care, mobility, and instrumental activities of daily living (IADL) at admission and discharge;
• other factors affecting outcomes, such as frailty; and
• discharge items, including discharge destination and caregiver information.

As part of the PAC-PRD, interrater reliability testing was conducted in a subsample of participating PAC-PRD providers (n = 34), with each site collecting data on 10–15 patients. A total of 455 pairs of assessments were collected. A second reliability activity was conducted using video-based patient vignettes to examine clinician agreement across provider types. A full description of the reliability methods and results from traditional interrater reliability and video reliability testing is given in a separate report; the results suggest that the CARE items are reliable and performed consistently across provider types.

As discussed previously, functional status has traditionally been documented using subsets of specific activities, such as eating, bathing, and transferring. For the CARE item set, the functional status data include the following items (activities):

• eating
• oral hygiene
• toileting hygiene
• dressing upper body
• dressing lower body
• wash upper body
• shower/bathe self
• put on/take off footwear
• roll left to right
• sit to lying
• lying to sitting on side of bed
• sit to stand
• bed to chair/wheelchair transfer
• toilet transfer
• walking or wheeling (in room, 50 feet, 100 feet, 150 feet)
• pick up object
• one step curb
• walk 50 feet with 2 turns
• four steps
• twelve steps
• walk 10 feet on uneven surfaces
• car transfers
• wheel short ramp (if using wheelchair)
• wheel long ramp (if using wheelchair)

For each of these items, each patient is assigned a code indicating the level of assistance needed to complete the activity. The numeric codes range from 6 to 1, with level 6 indicating complete independence, level 5 designating the need for setup or cleanup assistance, level 4 indicating that the person needs supervision or touching assistance, level 3 showing the need for partial or moderate assistance, level 2 denoting a need for substantial or maximal assistance, and level 1 indicating complete dependence.

For patients who do not complete an activity, a letter code is reported. The code of "M" or "S" indicates that the patient was unable to perform the activity because of medical or safety issues, respectively. The letter code of "N" denotes that the activity was not applicable, and "P" records that the patients refused. The letter "A" is used if the activity was attempted but not completed.

Several analyses required specific diagnostic data; therefore, RTI used IRF-PAI/FIM^® data to examine these analytic issues. We used the national Medicare IRF-PAI/FIM^® data for the years 2008–2010.

3.2 Data Analysis

3.2.1 Overview of the Analytic Issues

For this work, we focused on the following eight key analytic issues that need to be addressed in the development of functional status metrics:

1. using single items or multiple items combined into a scale
2. using a combined motor scale or separating self-care and mobility into two quality metrics
3. calculating a change value or a discharge status
4. calculating a mean or median value or a percentage meeting a benchmark
5. using summed raw scores or Rasch measures
6. defining the target population (denominator)
7. identifying exclusion criteria
8. specifying the case-mix adjustment process and variables

The questions we posed for the eight issues are as follows:

1. Should the functional status quality metrics be based on an individual item or multiple items that create a scale?
2. Should the functional status quality metrics be a single motor scale or should the motor items be separated into self-care and mobility scales?
3. Should the performance score be calculated as a change value (discharge score minus admission status) or a discharge status, with both adjusted for admission functional status?
4. Should the functional status scores be reported as a mean (average) or median value or as a percentage of patients who meet or exceed a benchmark?
5. If multiple items are used, should the functional status quality metric use the summed raw scores (unweighted or weighted) or the Rasch-based measure?
6. How should the population (i.e., denominator) for the functional status quality metric be defined for IRFs, SNFs, LTCHs, and HHAs?
7. What exclusion criteria need to be considered?
8. What approach should be used for risk adjustment—strata, regression modeling, or both?

3.2.2 Descriptive Analyses

Analyses of the functional status data began with a review of the distribution of numeric and letter codes by provider type and by admission and discharge. Analyses also included scatter plots to show the relationship between functional assessment change scores and functional status discharge scores. RTI also ran several regression analyses as part of the case-mix adjustment work to test for statistical interactions.

3.2.3 Rasch Analysis

Because we are measuring a latent trait, a concept that is constructed from several variables rather than directly observed, we used the one-parameter Rasch model to gain a better understanding the functional status items (activities). More specifically, we examined the order of functional status items (from easiest to hardest) that characterize the construct of the functional assessment items on the CARE item set. We provide a brief overview of Rasch analysis next.

Rasch analysis uses scored responses from the functional assessment items to create the equivalent of a functional status scale or ruler. Rasch analysis uses only available data to estimate a person's location along the dimension of functional status scale, and therefore analyses can be conducted if some data are missing. Rasch analysis can also inform the optimal selection of key items in order to construct functional status scales that sufficiently span an entire range of patient functioning, so that both the least able and most able (lowest- and highest-functioning) patients are adequately measured. In addition, Rasch analysis can indicate where items overlap or are redundant in terms of the level of function they capture.

Rasch analysis has been used to examine the FIM^® instrument (Granger, Hamilton, Linacre, Heinemann, & Wright, 1993; Heinemann, Linacre, Wright, Hamilton, & Granger, 1993; Linacre et al., 1994; Wright, Linacre, Smith, Heinemann, & Granger, 1997), the MDS (Wang, Byers, & Velozo, 2008), and the OASIS (Fortinsky, Garcia, Joseph Sheehan, Madigan, & Tullai-McGuinness, 2003). Rasch analysis has also been used to examine the extent to which existing functional assessment instruments (e.g., the FIM^® instrument, MDS 2.0) capture the same construct (Velozo, Byers, Wang, & Joseph, 2007).

Rasch modeling can be used not only to examine items of an instrument during both the construction and evaluation and refinement phases of an instrument development, but also to calculate person (patient) measures, which provide an alternative approach to summing the item scores of an instrument. An advantage of Rasch measures is that they convert ordinal-level data (levels are ordered, but the distances between the levels are not equal) into interval-level data (levels are ordered and the levels are equidistant). If the data fit the Rasch model, using interval-level data therefore may provide more accurate and precise estimation of differences in functional status between patients and across providers than do differences in raw total scores.

We used the Winsteps 3.75 software program to run the Rasch rating scale analysis (Linacre, 2012).

4. Results

4.1 Descriptive Analyses

We began our analyses of the CARE data by examining the distribution of the numeric scores, letter codes, and missing data for each item by provider type and at admission and discharge. These analyses show the percentage of patients who were independent; who required setup, supervision or touching assistance, partial assistance, or substantial assistance; or who were dependent. If the patient did not perform the activity or the data were not collected or provided, that information was also reported. Because these data are reported by provider type, the percentage of patients at each level by provider type can be compared at admission and at discharge. The data patterns are best visualized as 100% bar graphs; given the large number of graphics, they are presented in Appendix A. The actual percent values are reported in tables in Appendix B.

In reviewing the data, across all items at admission, we noted that LTCH patients had the most frequent use of letter codes, indicating that activities were not completed by LTCH patients. In addition, LTCH patients had a higher percentage of patients who were dependent in self-care and mobility activities on admission. The higher use of letter codes and higher percentage of dependent patients is consistent with the acuity of LTCH patients. As expected, HHA patients tended to have the least frequent use of dependence codes, but they did have functional limitations.

For the self-care items, level 5 (setup or cleanup assistance) was frequently used at admission and discharge, but this level was not often used for the mobility activities. Setup or cleanup assistance means that another person provides equipment, items, or assistance to the patient before or after the patient performs the activity, and it tends to be more relevant for self-care items. The letter codes, which indicate that an activity did not occur, were most often used on admission and for the more challenging items. Missing data are reported in the distribution and may include patients who had an unexpected discharge; thus functional status data were not collected because of the need to attend to the patient's urgent medical needs.

The more challenging mobility items (steps, uneven surfaces, car transfers) were often not assessed or were missing on admission. "Picking up object from floor" was not assessed or missing at discharge more than 40% of the time for all types of providers except SNFs (36% were missing in the SNF setting).

Overall, the distribution of scores for the CARE function items was consistent with the expected complexity of the patients in the different PAC settings.

4.2 Use Single Items or Multiple Items Combined Into a Scale?

4.2.1 Description of the Issue and Research Question

The first question we addressed was this: Should the functional status quality metrics be based on an individual item or multiple items that create a scale?

Measures of functional status need to balance measurement precision with data collection requirements that are practical and feasible. The most precise way to measure a construct like motor function might be to collect data on many items (activities), but this method may not be feasible in a busy clinical environment. On the other hand, a single item may not measure the concept as precisely as multiple items or may not be feasible to collect for some patients.

Single items are faster, easier to administer, and require fewer resources for analysis. Some outcomes can be measured nearly as accurately using a single item as a multiple-item scale. For example, self-reported general health status, measured with a single question, can be almost as accurate and reliable as the full Medical Outcomes Study 36-Item Short-Form Health Survey (SF-36) in predicting mortality and hospitalization (DeSalvo, Fan, McDonell, & Fihn, 2005). The current Home Health Compare program reports three functional status quality metrics (ambulation, transferring, and bathing), each based on a single item. In contrast, ORYX measures (the Joint Commission), based on FIM^® data, use multiple items.

According to Kane and Radosevich (2011), most health outcomes should be measured using multiple items for several important reasons:

• Multiple items are more reliable than single-item scales. Multiple items average out measurement error when their scores are used to create a total score, and this averaging helps avoid issues such as regression to the mean that can be created by relying on a single item.
• Multiple items are better at measuring complex concepts than are single items.
• Multiple-item scales allow for more precise discrimination along the spectrum of a construct.
• Multiple items have better calibration than single-item measures. Increased calibration is associated with a better chance of detecting differences in outcomes.
• Multiple-item measures are less prone to floor and ceiling effects. For example, a 3-item measure of physical functioning in the Health Status Questionnaire found about 64% of scores at the ceiling (the top of the scale), whereas only 42% of scores were at the ceiling of the 10-item Physical Functioning Scale on the SF-36 (Kane & Radosevich, 2011).

4.2.2 Analysis Description and Results

We examined the CARE functional status data in terms of the relative relationship of each item or level on a functional status scale. We were also interested in seeing whether the data appeared to be distributed on an interval level, such that the distance between level 2 and level 3 is the same as the distance between level 5 and level 6. Motor items were analyzed together, but IADL items were not included. In addition, the tube feeding item (assistance with tube feeding administration) was not included because previous analyses showed that this item did not fit well into the functional status motor scale, as it is more of a nursing procedure than a daily activity. Wheelchair items were not included; instead, a single walking distance item (walking 150 feet) was selected.

We wanted to create a functional status scale that would be stable across time (admission and discharge) and across settings. Therefore, we used CARE data from acute and all PAC settings. In this sample, each patient had only one assessment—either at admission or at discharge, but not both. We conducted a Rasch analysis using the rating scale model, which assumes that the rating scale was the same for each item. As mentioned previously, when the activity was not attempted or did not occur, clinicians reported letter codes. We recoded all letter codes to 1 (dependent). The Rasch scale was set to a range of 0–100, with a high value indicating more independence.

The Rasch rating scale analysis was applied to the CARE function data to examine whether the "distance," or amount of ability, needed to move from one rating scale step to the next was consistent across the rating scale. As shown by Figure 4-1, the Rasch transformation allowed us to examine the distances between the rating scale steps for each item. For each item, we examined whether moving from a rating of dependent (level 1) to a rating of substantial assistance (level 2) for the item "eating" required the same distance or amount of ability as that to move from a rating of setup (level 5) to a rating of independent (level 6) for eating. Adding ordinal-level data and treating it like interval-level data is sometimes ignored in the literature, although a recent editorial addressed this issue (Grimby, Tennant, & Tesio, 2012).

Figure 4-1, from the Rasch Winsteps output, shows the ruler for the motor scale at the top and bottom of the box. The motor ruler ranges from 0 to 100, but only the range 10 to 90 is visible in the graphic. The figure shows the relative placement of each item's rating scale levels on the ruler. For each item, the expected scores along the ruler are shown on each row. The score between the two colons shows the range of function associated with that score level. For example, for the eating item, the second response option, substantial assistance, begins at the colon at approximately 15 on the motor scale and ends at the colon at about 23, where the lower bound for the third response option, partial assistance, picks up. The items are listed with the hardest item (12 steps) at the top of the right side of the figure and the easiest item (eating) at the bottom. In addition to showing how the individual items fit onto the motor ruler, this analysis also allowed us to examine groups of items rather than individual items so that the measurement would be more precise and individual item scores would be treated appropriately.

Figure 4-1. Rasch Analysis of Motor Items (Self-Care and Mobility) in the Continuity and Assessment Record and Evaluation Tool

     D

On the basis of Figure 4-1, a patient with a score of 50 on the motor ruler (marked by the vertical red line) is likely to be independent (level 6) on eating, require supervision/touching assistance (level 4) on lower body dressing, require substantial assistance (level 2) on climbing 4 exterior steps and walking 10 feet on uneven surfaces, and be dependent (level 1) with 12 interior steps.

Figure 4-1 shows that not all rating scale steps are equal. For example, on the eating item, the range on the ruler for level 3 is smaller than the range for level 5. Improving one unit from level 3 to level 4 may not be equal to moving from level 5 to level 6.

Another issue related to use of a single item is floor and ceiling effects. In Figure 4-1, a patient with a score of 6 on eating would be estimated to have a measure of about 47 or higher on the motor ruler. If eating was the only item reported, we could not see improvement above ~47 on our ruler that ranges from 0 to 100.

4.2.3 Technical Expert Panel Discussion and Conclusions

During the TEP meeting, RTI discussed the using single- and multiple-item scales for measuring motor functional status, including the advantages gained in reliability and avoiding floor and ceiling effects when using multiple items. All TEP members agreed that multiple-item scales are preferred for measuring functional status.

In supporting the multiple-item approach, TEP members cited the ability to provide a more global perspective on the patient. One member also suggested that it would increase the face validity of the measure. Another member noted that single items may be useful for limited clinical care and quality improvement, but multiple items provide a better perspective for assessing overall quality and may also give the measure stronger psychometric properties. One member noted that multiple items have the potential to eliminate the floor and ceiling effect of measurement.

One concern expressed by the TEP members about multiple items was the potential to increase the burden on and expense to the providers, but another member on the TEP noted that this information should be routinely collected. A concern was that a single item may be more easily understood by the clinician and consumer.

4.3 Use a Combined Motor Scale or Separate Self-Care and Mobility Into Two Quality Metrics?

4.3.1 Description of the Issue and Research Question

The second question we addressed with analyses was this: Should the functional status quality metrics be a single motor scale or should the motor items be separated into self-care and mobility scales?

As previously noted, research using Rasch analysis and traditional factor analyses showed that motor function and communication and cognition items measured different constructs (Linacre et al., 1994). Therefore, RTI believes that motor and communication items should not be summed, because patients may vary in status on these two different constructs. This variation is concealed when a single total raw score is calculated.

More recent research has examined the motor items of functional assessment instruments and noted that when patients' primary diagnoses differed, the patterns of functional abilities within the motor scale were different (Haley et al., 2004; Haley et al., 2011; Jette et al., 2003; Jette et al., 2007). This finding has led some researchers to split the motor items of functional assessment measures into separate self-care and mobility scales.

As noted in Section 2.2, the Joint Commission's ORYX initiative includes several functional status quality metrics, including the change in motor FIM^® score and discharge motor FIM^® score. The ORYX metrics were developed and have been used since about 1997 and 2001. A more recently developed set of quality metrics includes "Self-care as measured by the AM-PAC" and "Mobility ability measured by the AM-PAC." The AM-PAC metrics, which received time-limited endorsement from the NQF in 2009, reflect the more recent research that splits motor items into the self-care and mobility constructs.

4.3.2 Analysis Description and Results

For the second analytic issue, we analyzed the IRF-PAI/FIM^® motor functional status data to examine whether patients with different clinical diagnoses had different motor function patterns. The purpose of this analysis was to determine whether we would recommend a single motor functional status quality metric or two separate quality metrics—a self-care measure and a mobility measure. The three diagnosis groups used in this analysis were stroke, central cord syndrome/spinal cord injury, and unilateral hip replacement. These diagnoses were chosen so that we would have a sample of patients who would be expected to have various levels of impairment in motor and self-care function. RTI used the IRF-PAI/FIM^® data for this analysis to have an adequate number of patient records for all three diagnosis groups.

Patterns of activity limitations should be the same for all patients on a functional assessment scale. If patterns of activity limitations on the motor scale vary by diagnosis, then the separate mobility and self-care subscales should be used.

The IRF-PAI/FIM^® data set contains more data for patients with a stroke, so a random selection of stroke patient records was used to maintain equal sample sizes. We calculated the mean score for each item by diagnosis group to compare the patterns of scores by diagnosis. Rasch analysis was then run using the rating scale model, which assumes that the rating scale is the same for each item. We set the Rasch motor measure to a range of 0–100, with a high value indicating more independence. Analyses were conducted by diagnosis group to examine whether a functional assessment ruler of motor items can fit the pattern of functional abilities of all patients. For example, for most patients, mobility activities tend to be more challenging than self-care activities. However, as an illustrative example, patients with central cord syndrome/spinal cord injury may be able to walk and would likely perform well on the mobility activities, but they may have limited use of their upper extremities and would have marked difficulty with the self-care activities. This specific diagnosis-related pattern of activity limitations may not be evident if a simple motor scale was used for a functional status measure.

Figure 4-2 shows the mean admission FIM^® scores for motor items for patients with the three diagnoses: hip replacement (blue diamond), central cord syndrome (red square), and stroke (green triangles). If the pattern of motor function across the three diagnosis groups were the same, we would expect to see parallel lines for the mean scores by diagnosis; the lines would not cross. However, the three lines are not parallel. Item scores for bathing, lower body dressing, stairs and walk/wheelchair are lower than expected for patients with a unilateral hip replacement relative to the scores for the other items. This figure shows us that the patterns of motor scores for patients in these three diagnostic groups are not the same. Patients with hip replacements show a very different pattern of functional abilities than do patients with a stroke or central cord syndrome.

Figure 4-2. Mean Admission FIM^® Scores for Items by Diagnosis

     D

Figures 4-3, 4-4, and 4-5 show that the hierarchy order of motor items, from easiest to hardest to perform, is different for patients with these three diagnoses when data are analyzed using Rasch analysis. If the motor scale were similar for patients with these three diagnoses, the list of items on the right sides of the figures would be in the same order. However, we see that the order of the items changes across the groups, suggesting that the operational definition of motor function scale is different for each group. For patients with a stroke, toileting is the third-most challenging item on the list; it is harder than dressing lower body. Toilet transfer is in the middle of the item list. For the patients with central cord syndrome, dressing lower body is more challenging than toileting, and toilet transfer seems to be easier for these patients than for stroke patients. For patients with a hip replacement, toileting and toilet transfers have similar difficulty levels.

Figure 4-3. Map of Motor FIM^® Motor Items for Patients With a Stroke

     D

Figure 4-4. Map of Motor FIM^® Motor Items for Patients With Central Cord Syndrome

     D

Figure 4-5. Map of FIM^® Motor Items for Patients With a Hip Replacement

     D

Figure 4-6 shows a scatter plot of the self-care measures compared with motor scores. Each small box represents the location of an individual patient on both the self-care and the motor scale. Both the self-care scale and the motor scale were set to range from 0 to 100. If the self-care items were simply a subset of the broader set of motor items and operated the same way for patients with a stroke (green boxes), patients with central cord syndrome (red boxes), and patients with a hip replacement (blue diamonds), then the green, red, and blue markers should vary equally above and below the blue diagonal line, indicating equal scores on the two scales. However, we see that the scores for patients with a stroke (green boxes) tend to fall below the blue line more often than above the line, and the scores for patients with central cord syndrome tend to be above the blue line. This pattern suggests that the self-care items are not simply a subset of the broader concept of motor function for all these patients.

Figure 4-6. Scatter Plot of Self-Care and Motor Measures for Patients With a Stroke, Central Cord Syndrome, or Hip Replacement

     D

Similar to the self-care graphic above (Figure 4-6), Figure 4-7 shows a scatter plot of the mobility measures compared with motor scores. Each small box represents the location of an individual patient on the mobility and the motor scale, and both scales were set to range from 0 to 100. If the two measures were similar for patients with a stroke (green boxes), patients with central cord syndrome (red boxes), and patients with a hip replacement (blue diamond), then the green, red, and blue markers should vary equally above and below the blue diagonal line. However, we see that the scores for patients with a stroke (green boxes) tend to be above the blue line more often than below the line, and the scores for patients with central cord syndrome tend to below the blue line. This suggests that the mobility items are not simply a subset of the broader concept of motor function for all these patients.

Figure 4-7. Scatter Plot of Mobility and Motor Measures for Patients With a Stroke, Central Cord Syndrome, or Hip Replacement

     D

Taken together, Figures 4-2 through 4-7 show that the hierarchy of items on a motor scale score differs when patients have different diagnoses. Furthermore, a motor score of 50 for a patient with a stroke may not mean the same thing as a score of 50 for a patient with central cord syndrome. By splitting the motor items into self-care and mobility measures, we can expect more precise, accurate, and meaningful measurement.

4.3.3 Technical Expert Panel Discussion and Conclusions

TEP members agreed that separate quality metrics for self-care and mobility would be appropriate within the area of motor function, given that the measure would apply to patients with various diagnoses. Separate metrics would result in more precise, accurate measurements.

One TEP member stated that motor function may be a better predictor of resource use, but that separating the two concepts is better for outcomes. Several TEP members mentioned that separating motor into self-care and mobility scales provides better feedback for clinicians and enhances their ability to improve outcomes. Another member noted that separate measures would improve face validity.

4.4 Calculate a Change Value or a Discharge Status?

4.4.1 Description of the Issue and Questions

The third question we addressed with analyses was this: Should the performance score be calculated as a change value (discharge score minus admission status) or a discharge status, with both adjusted for admission functional status?

Functional status can be estimated in two different ways: change in functional status, which is calculated as the difference between the discharge score and the admission score, and the absolute value of functional status at discharge or follow-up. Both estimates are adjusted for functional status on admission (Food and Drug Administration, 2009).

Currently, the performance measure "Change in basic mobility as measured by the AM-PAC" uses a change score to document improvement in functional status. The Joint Commission ORYX measures, based on FIM^® motor data, have two different metrics: one uses a change score and the other uses a discharge score (The Joint Commission, 2012).

The question of which approach is best to use has been discussed in the literature with no clear consensus. Change in functional status is often reported and may initially seem like the best choice for measuring the effectiveness of care, but calculating change scores is subject to several methodological limitations (Streiner & Norman, 2008):

• Change scores can vary in the magnitude and the direction of the change, and change scores for individuals can be obscured when positive and negative changes cancel each other out.
• There are issues with change scores related to regression to the mean.
• Change scores are subject to measurement error from combining the baseline scores (with errors) and the discharge or follow-up scores (with errors), so change scores may be less reliable than individual baseline and follow-up scores.
• Change in functional status can be used only if the instrument is sensitive or responsive to change (Donabedian, 2005; Food and Drug Administration, 2009).
• Item or instrument floor effects (for patients who start at the low end of a scale) and ceiling effects (for patients who start at the high end of the scale) may obscure an individual's change in functioning, because the instrument cannot detect the full range of change.
• The clinical meaning of a change score is generally unknown, making the change score hard to interpret. Ideally, health status measures would have clinically meaningful thresholds that could define stages or complexity levels, such that moving from one stage to another stage would have clinical meaning (Collins & Johnston, 1995; Stineman et al., 2012; Stineman, Maislin, Fiedler, & Granger, 1997; Stineman, Ross, Fiedler, Granger, & Maislin, 2003a, 2003b; Stineman et al., 2011).

4.4.2 Analysis Description and Results

To explore the issue of using change or discharge scores, we began by examining the relationship between change and discharge scores for a group of clinically homogeneous patients.

The scatter plots in Figures 4-8 and 4-9 show the relationship between a change score and a discharge score. Data are IRF-PAI/FIM^® data for Medicare fee-for-service patients with a stroke in case-mix group 0110 who were discharged from IRFs in 2010. IRF-PAI/FIM^® data were used for these analyses, because IRF patients are assigned into case-mix groups for the IRF prospective payment system and thus a group of patients with similar clinical complexity could be selected for analyses.

Figure 4-8. Scatter Plot of Change and Discharge Scores: Self-Care

     D

Figure 4-9. Scatter Plot of Change and Discharge Scores: Mobility

     D

Figure 4-8 shows the relationship between the self-care change scores (vertical axis) and the discharge scores (horizontal axis). Each circle represents the location of an individual patient on the change and discharge scale. The graphic shows that as change scores increase the discharge scores increase, and the correlation between the self-care discharge score and the self-care change score for patients within this one case-mix group was high at 0.876.

Figure 4-9 shows the relationship between the mobility change scores (vertical axis) and the mobility discharge scores (horizontal axis). Each circle represents the location of an individual patient on the change and discharge scale. The graphic shows that as change scores increase the discharge scores increase, and the correlation between the mobility discharge score and the mobility change score for patients within this one case-mix group was very high at 0.976.

These scatter plots show that change and discharge scores for self-care and mobility are highly correlated and that using either the change or discharge scale for functional status quality metrics may lead to similar results. Additional facility-level analyses are needed.

4.4.3 Technical Expert Panel Discussion and Conclusions

TEP members suggested that RTI continue to examine both the change and discharge scores options. Some TEP members preferred the discharge option and felt it would work for stay-level and episode-level quality metrics. TEP members expressed concerns about regression to the mean and floor and ceiling effects for change scores, but they noted that the change score seems the best from a face validity perspective. It was also noted that the use of change or discharge score may vary by subgroup. One TEP member thought that change scores and discharge scores might be comparable after case-mix adjustment.

The importance of the discharge status was emphasized by several members, as it is a predictor of returning the patient to the community. Another TEP member commented that the discharge score allows you to see function at a point in time. For example, if the goal were for the patient to return home, a discharge score would be more important than a change score.

Another important point discussed was that a change score and the actual discharge score are comparable as long as the data are case-mix adjusted with admission status. Several felt that the importance of the measure differed by the patients and their goals. Several TEP members believe that two measures should be developed: change and discharge.

4.5 Calculate a Mean or a Median Value or a Percentage Meeting a Benchmark?

4.5.1 Description of the Issue and Question

The fourth analytical question for this work was this: Should the functional status scores be reported as a mean (average) or median value or as a percentage of patients who meet or exceed a benchmark?

Functional status scores are often reported as mean (average) values. On the other hand, many quality metrics report the percentage of patients who meet or exceed a benchmark. Benchmarks could include

• the mean or median scores of similar patients from the national database,
• a clinically important value, or
• more than "x" number of units.

The Joint Commission ORYX measures for the IRF-PAI/FIM^® instrument calculate a mean change score and a mean discharge score for IRFs. In contrast, the quality metric "Change in basic mobility as measured by the AM-PAC" uses the number (or proportion) of a clinician's patients in a particular risk-adjusted diagnostic category who meet a target of improvement in basic mobility functioning (National Quality Forum, 2012). The measure's developers recommend that the threshold be based on the percentage of patients who exceed one or more minimal detectable change (MDC) thresholds (the minimum change that is not likely to be measurement error).

Generally, because patient populations are heterogeneous, using the mean value may obscure important information about the functional status of individual patients. However, the right approach is not yet apparent, and more analyses need to be conducted for us to better understand this issue.

4.5.2 Technical Expert Panel Discussion and Conclusions

Some TEP members favored using percentage values—for example, the percentage of patients reaching a benchmark score. TEP members discussed that the staging approach might be a good approach, but no research has yet been done with the CARE data to examine this issue.

TEP members expressed concern that the mean or median value would indicate that 50% of patients are below the midpoint. Concern was raised that the population is too heterogeneous to use a mean value and important patient information will be lost in a mean-only measure. One member supported a national benchmark approach, with an emphasis that the measure should look for patients to achieve a clinically meaningful goal. One TEP member felt that confidence levels should be reported to reflect that the scores are not absolute.

TEP members agreed that additional analyses were needed to compare these different approaches.

4.6 Use Summed Raw Scores or Rasch Measures?

4.6.1 Description of the Issue and Question

The fifth analytic question was this: If multiple items are used, should the functional status quality metric use the summed raw scores (unweighted or weighted) or the Rasch-based measure?

Functional status data are often reported as summed raw scores, either unweighted or weighted. However, the raw scores are ordinal-level data, and the use of parametric statistics to analyze such data is not recommended, even though it is frequently done. (Ordinal-level data are ordered, but the distances between the levels are not equal.)

Rasch analysis may be used to analyze ordinal-level data and create interval-level data, so that the distances between the levels are equidistant. Another benefit of using Rasch analysis is that person measures can still be estimated, even when there is missing data. However, the process of calculating the Rasch-derived measure is not very transparent.

4.6.2 Analysis Description and Results

Figure 4-10 shows the relationship between the raw, unweighted, summed self-care scores and Rasch self-care measures using the CARE item set data. For illustrative purposes within this analysis, we included only records of patients with complete data. In the middle of the scale, the relationship between the raw summed score and the Rasch measure is linear. At the ends of the scale, the relationship is not linear. For example, the difference between a Rasch measure of 80 to 100 is 20 units (on the vertical axis); this difference is 4 units (38 to 42 units) for the summed raw score on the vertical axis. Therefore, for patients who have functional status levels in the middle of the scale, analysis of the raw summed score or Rasch measure would not likely alter conclusions. However, for patients with very low and very high levels of function, the use of a raw summed score or a Rasch measure might affect change score values relative to other patients. For example, the raw summed score may show minimal change, but the Rasch measure may show a large change in function.

Figure 4-10. Scatter Plot of Raw Self-Care Scores and Rasch Self-Care Measures

     D

4.6.3 Technical Expert Panel Discussion and Conclusions

In general, TEP members wanted work on both approaches—summed raw scores and Rasch measures—to continue to be examined. TEP members felt that the analysis needs to be meaningful for the target audience. Some TEP members are comfortable using a Rasch measure with a range from 0 to 100, which they felt was understandable to the public.

4.7 How Do We Define the Target Population?

4.7.1 Description of the Issue and Question

The sixth analytic question was this: How should the population (i.e., denominator) for the functional status quality metric be defined for IRFs, SNFs, LTCHs, and HHAs?

For each quality metric, the target population needs to be clearly defined. Functional status applies to all individuals, but in PAC, some patients receive treatment designed to meet rehabilitative or restorative goals, whereas other patients receive treatments that are focused primarily on medical issues with rehabilitation goals secondary. For patients with medical issues, function may still be an important aspect of care, because the goal may be to build up the patient's endurance for him or her to benefit from more intensive rehabilitation therapy.

As we define the target population for each type of provider, it is important to consider the types of patients admitted to each provider type:

• Patients admitted to an IRF receive intensive rehabilitation services, and the goals of care are restorative. Patients may have goals of improving self-care, mobility, communication skills, or cognitive function.
• Patients admitted to SNFs often have restorative or rehabilitative goals, but they may not need the same level of intensity provided in IRFs. Patients may have goals of improving self-care, mobility, communication skills, or cognitive function.
• HHA patients often have functional goals so that they can stay in their homes and communities. These goals may focus on improvement in daily living skills such as dressing and bathing. Some patients may receive home care because of wound care needs and receive limited therapy services.
• Patients admitted to LTCHs are often very clinically complex and may have limited mobility due to medical equipment or significant reductions in endurance and strength. LTCH patients will often have modest functional or cognitive improvement goals, but medical issues will be primary.

Many patients in PAC settings have varying degrees of restorative or rehabilitative goals. These functional goals may address self-care, mobility, communication, problem-solving, or memory limitations.

4.7.2 Technical Expert Panel Discussion and Conclusions

In general, the consensus of TEP members was to include all patients within each provider. TEP members felt that all patients in a PAC setting should be included in the target population, because function is important for all patients. TEP members did mention the need to account for different patient treatment goals. Patients with medical issues, such as wounds and pressure ulcers, should not decline in function while under the care of a PAC provider.

4.8 What Exclusion Criteria Need to Be Considered?

4.8.1 Description of the Issue and Question

The seventh question was this: What exclusion criteria need to be considered for functional status quality metrics?

One exclusion criterion may be patients who do not complete their full course of treatment or rehabilitative therapy because of death or an unexpected discharge. Patients who do not have complete stays because of a medical emergency or a discharge against medical advice should also be excluded, because an assessment of the patient's status at the time of discharge is not feasible. Reporting the patient's status before the medical emergency may not reflect the patient's status at the time of discharge. Included in the definition of incomplete stays would be patients discharged to acute care hospitals or to an equal or higher level of PAC.

A second exclusion criterion, for a given functional status area (e.g., self-care, mobility, communication), may be a patient's lack of functional limitations in that area at the initial assessment; such a patient would not be expected to show changes in that area of functional status. In this situation, a patient may be included in some functional status quality metrics but not others. For example, a patient who had a hip replacement may have mobility limitations but may not have communication limitations. The patient may not need therapy services for communication, and would not be expected to have improvement in communication skills. In fact, the patient may have an independent (level 6) score on the expression and comprehension items and cannot achieve a higher (better) score.

4.8.2 Technical Expert Panel Discussion and Conclusions

In general, TEP members agreed that it was appropriate to exclude patients who had incomplete stays (e.g., discharge to acute care, discharge against medical advice, patient died), patients who had unpredictable functional outcomes (e.g., persistent vegetative state, multiple sclerosis), and patients expected to have a decline in functional status (e.g., amyotrophic lateral sclerosis [ALS or Lou Gehrig's disease]). TEP members felt that further discussion is needed about including or excluding patients who do not need or receive therapy treatment.

4.9 What Case-Mix Adjustment Process and Variables Shall We Specify?

4.9.1 Description of the Issue and Question

The eighth and last research question was this: What approach should be used for risk adjustment—strata, regression modeling, or both?

In the context of quality metrics, case-mix adjustment (also known as risk adjustment) refers to statistical techniques that are used to adjust for differences in patient populations across facilities so that data across facilities may be compared. Patients' improvement in functional status is expected to vary on the basis of the treatment provided, but also on the basis of the person's age, clinical complexity, diagnosis, admission functional status, and other factors. To examine the effectiveness of care, we will need to adjust for these non-treatment-related factors that may affect the functional status outcome. Case-mix adjustment is an important issue for outcome measures, because the most clinically complex patients are less likely to have good outcomes.

Differences in patient case-mix occur across providers because patients do not select their health care providers or provider types randomly. Therefore, when patient outcomes of care are compared across time (i.e., admission to discharge) or across providers, these outcomes often need to be adjusted to control for patient-level factors so that the effect of the quality of a provider's care can be isolated. The purpose of case-mix adjustment is to allow for a fair comparison of health outcomes, so that observed differences can be attributed appropriately to the provider interventions and not population differences (Iezonni, 2003; McGlynn & Asch, 1998). Domains to consider for case-mix adjustment of quality metrics, and different approaches to risk adjustment, are described next.

There are several approaches to risk adjusting outcomes data. One option is to identify high- and low-risk groups and report the data stratified by these risk groups (i.e., strata). Outcomes for patients across providers could be compared within the same stratum. This approach is appropriate when adjustment for one key factor is needed, the factor is either dichotomous or has clinically meaningful cut points, and the number of patients is large enough to split them across two or more groups.

A second risk-adjustment approach uses regression modeling with demographic and clinical factors included in the model as covariates (i.e., risk factors). With this approach, multiple factors, including continuous and dichotomous factors, can be controlled for, and facility-specific predicted and expected values can be calculated to compare risk-adjusted data across providers.

A third case-mix adjustment approach would involve identifying risk groups (i.e., strata), and using regression models within each stratum. These data are then aggregated into a single estimate based on the national distribution of patients by strata. This combined approach would be needed if the effect of key covariates on the outcome varied by stratum (risk group). For example, if the effect of baseline functional status on discharge functional status varied by primary diagnosis, then data should be stratified by diagnosis, and regression models for each diagnosis group would be used. The regression results could be aggregated into a summary score based on weighting of the diagnosis groups (strata) using a national distribution or other standard. When the target population for the quality metric is heterogeneous, then the combined approach of strata and regression modeling may be the best option.

A significant area of controversy in the area of case-mix adjustment is the choice of the type of regression model used in the adjustment process. Concerns about clustering and small sample sizes within providers have led some measure developers to use hierarchical generalized linear models (HGLMs) rather than fixed-effects regression models. The HGLM approach has been criticized because it decreases the variation in the performance score, particularly for small providers. In the paper "Statistical Issues in Assessing Hospital Performance," commissioned by the Committee of Presidents of Statistical Societies, Ash and colleagues critically reviewed this issue and indicated that HGLMs are appropriate for use given the structure of the data and the purpose of the analyses (Ash et al., 2012).

4.9.2 Analysis Description and Results

Using IRF-PAI/FIM^® data, RTI ran simple regression analyses with discharge self-care scores as the dependent variable and admission self-care scores and age as independent variables. Separate models were run for three diagnoses—stroke, hip fracture, and lower extremity joint replacement—as well as a model with all patients.

Figure 4-11 shows the relationship between admission scores and predicted discharge scores for patients with each of the three diagnoses and all patients. The slope of the line (regression coefficient) is calculated from a regression model; the regression coefficients on admission functional status scores were 0.88 for all patients, 0.93 for patients with a stroke, 0.81 for patients with hip fracture, and 0.46 for patients with lower extremity joint replacement. The large difference in the slope between patients with a lower extremity joint replacement and patients with a stroke suggests that patients in the lower extremity group should be in a different stratum than patients with a stroke and patients with a hip fracture.

These data suggest that the combined approach of strata and regression should be considered for the case-mix adjustment approach.

Figure 4-11. Graphic of Regression Model Predicting Discharge Self-Care Scores by Diagnosis

     D

4.9.3 Technical Expert Panel Discussion and Conclusions

In general, TEP members agreed that additional analyses are needed to examine different case-mix adjustment approaches. TEP members agreed with the idea of defining strata on the basis of primary diagnosis. In addition, TEP members suggested factors that may be important to include as risk adjusters, including case-mix index, surgical or nonsurgical treatment, baseline functional status, cognitive function at admission, levels of depression at admission, geographic region, urbanicity, length of stay at the acute care hospital, prior level of function, age, and gender.

5. Conclusions

On the basis of the work conducted during this project, including the environmental scan, the analyses of CARE and other functional status data, and the input from the TEP, we are recommending further development of two motor functional status quality metrics: self-care and mobility.

On the basis of input from the TEP and initial analyses conducted under this project, RTI recommends the use of multiple items in the construction of crosscutting functional status metrics. RTI recommends additional analyses comparing metrics based on a discharge score and metrics based on a change score. RTI also recommends additional analyses to compare metrics that calculate the percentage of patients meeting a benchmark and metrics that use a mean or median value. The quality metrics will need to be case-mix adjusted, and RTI recommends using both strata and regression modeling. The TEP members expressed their interest in having all patients included as the target population, and RTI recommends excluding patients who do not complete their PAC stays.

RTI examined the literature to identify a third quality metric in the area of cognition. The number of assessment instruments that assess cognitive function is limited, and few have been developed into quality metrics. In the area of communication, challenges include understanding and documenting function and the expected recovery trajectory. For example, a patient with a communication limitation may have reduced muscle tone around the mouth, may have aphasia or a language disorder, or may be confused. The recovery trajectory for these patients needs to be taken into account as part of the quality metric.

A possible future area for functional status quality metrics development is screening and outcomes for swallowing disorders. The CARE data set and the MDS 3.0 include several items that might be used to create an outcome quality metric.

References

Ash, A. S., Fienberg, S. E., Louis, T. A., Normand, S.-L. T., Stukel, T. A., & Utts, J. (2012). Statistical issues in assessing hospital performance. Retrieved from http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/Downloads/Statistical-Issues-in-Assessing-Hospital-Performance.pdf

Collins, L. M., & Johnston, M. V. (1995). Analysis of stage-sequential change in rehabilitation research. American Journal of Physical Medicine & Rehabilitation, 74(2), 163-170.

DeSalvo, K. B., Fan, V. S., McDonell, M. B., & Fihn, S. D. (2005). Predicting mortality and healthcare utilization with a single question. Health Serv Res, 40(4), 1234-1246. doi:10.1111/j.1475-6773.2005.00404.x

Donabedian, A. (2005). Evaluating the quality of medical care. 1966. Milbank Q, 83(4), 691-729. doi:10.1111/j.1468-0009.2005.00397.x

Food and Drug Administration. (2009). Guidance for industry: Patient-reported outcome measures: Use in medical product development to support labeling claims. Retrieved from http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf

Fortinsky, R. H., Garcia, R. I., Joseph Sheehan, T., Madigan, E. A., & Tullai-McGuinness, S. (2003). Measuring disability in Medicare home care patients: Application of Rasch modeling to the outcome and assessment information set. Med Care, 41(5), 601-615. doi:10.1097/01.MLR.0000062553.63745.7A

Granger, C. V., Hamilton, B. B., Linacre, J. M., Heinemann, A. W., & Wright, B. D. (1993). Performance profiles of the functional independence measure. Am J Phys Med Rehabil, 72(2), 84-89.

Grimby, G., Tennant, A., & Tesio, L. (2012). The use of raw scores from ordinal scales: Time to end malpractice? J Rehabil Med, 44(2), 97-98. doi:10.2340/16501977-0938

Haley, S. M., Andres, P. L., Coster, W. J., Kosinski, M., Ni, P., & Jette, A. M. (2004). Short-form activity measure for post-acute care. Arch Phys Med Rehabil, 85(4), 649-660.

Haley, S. M., Ni, P., Lai, J. S., Tian, F., Coster, W. J., Jette, A. M., Straub, D., & Cella, D. (2011). Linking the activity measure for post acute care and the quality of life outcomes in neurological disorders. Arch Phys Med Rehabil, 92(10 Suppl), S37-43. doi:10.1016/j.apmr.2011.01.026

Heinemann, A. W., Linacre, J. M., Wright, B. D., Hamilton, B. B., & Granger, C. (1993). Relationships between impairment and physical disability as measured by the functional independence measure. Arch Phys Med Rehabil, 74(6), 566-573.

Hula, W. D., Doyle, P. J., & Austermann Hula, S. N. (2010). Patient-reported cognitive and communicative functioning: 1 construct or 2? Arch Phys Med Rehabil, 91(3), 400-406. doi:10.1016/j.apmr.2009.11.013

Iezonni, L. (2003). Risk adjustment for measuring healthcare outcomes (3rd ed.). Chicago, IL: Health Administration Press.

Jette, A. M., Haley, S. M., & Ni, P. (2003). Comparison of functional status tools used in post-acute care. Health Care Financ Rev, 24(3), 13-24.

Jette, A. M., Haley, S. M., Tao, W., Ni, P., Moed, R., Meyers, D., & Zurek, M. (2007). Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings. Phys Ther, 87(4), 385-398. doi:10.2522/ptj.20060121

Kane, R. L., & Radosevich, D. M. (2011). Conducting health outcomes research. Sudbury, MA: Jones & Bartlett.

Linacre, J. M. (2012). Winsteps® Rasch Measurement computer program (Version 3.75). Beaverton, OR: Winsteps.com.

Linacre, J. M., Heinemann, A. W., Wright, B. D., Granger, C. V., & Hamilton, B. B. (1994). The structure and stability of the Functional Independence Measure. Archives of Physical Medicine & Rehabilitation, 75(2), 127-132.

McGlynn, E. A., & Asch, S. M. (1998). Developing a clinical performance measure. Am J Prev Med, 14(3 Suppl), 14-21.

National Committee on Vital and Health Statistics Subcommittee on Health. (2001). Classifying and Reporting Functional Status (Vol. 2012, pp. 1).

National Quality Forum. (2012). Quality Positioning System Retrieved Sept. 21, 2012, from http://www.qualityforum.org

Stineman, M. G., Henry-Sanchez, J. T., Kurichi, J. E., Pan, Q., Xie, D., Saliba, D., Zhang, Z., & Streim, J. E. (2012). Staging activity limitation and participation restriction in elderly community-dwelling persons according to difficulties in self-care and domestic life functioning. American Journal of Physical Medicine & Rehabilitation, 91(2), 126-140.

Stineman, M. G., Maislin, G., Fiedler, R. C., & Granger, C. V. (1997). A prediction model for functional recovery in stroke. Stroke, 28(3), 550-556.

Stineman, M. G., Ross, R. N., Fiedler, R., Granger, C. V., & Maislin, G. (2003a). Functional independence staging: conceptual foundation, face validity, and empirical derivation. Archives of Physical Medicine & Rehabilitation, 84(1), 29-37.

Stineman, M. G., Ross, R. N., Fiedler, R., Granger, C. V., & Maislin, G. (2003b). Staging functional independence validity and applications. Archives of Physical Medicine & Rehabilitation, 84(1), 38-45.

Stineman, M. G., Xie, D., Pan, Q., Kurichi, J. E., Saliba, D., & Streim, J. (2011). Activity of daily living staging, chronic health conditions, and perceived lack of home accessibility features for elderly people living in the community. Journal of the American Geriatrics Society, 59(3), 454-462.

Streiner, D. L., & Norman, G. R. (2008). Health measurement scales: A practical guide to their development and use (4th ed.). New York: Oxford University Press.

The Joint Commission. (2012). Performance measurement. JointCommisionOnline, (August 29, 2012). Retrieved from

Velozo, C. A., Byers, K. L., Wang, Y. C., & Joseph, B. R. (2007). Translating measures across the continuum of care: Using Rasch analysis to create a crosswalk between the Functional Independence Measure and the Minimum Data Set. J Rehabil Res Dev, 44(3), 467-478.

Wang, Y. C., Byers, K. L., & Velozo, C. A. (2008). Rasch analysis of Minimum Data Set mandated in skilled nursing facilities. J Rehabil Res Dev, 45(9), 1385-1399.

World Health Organization. (2001). International Classification of Functioning, Disability and Health: Retrieved from http://apps.who.int/classifications/icfbrowser/

Wright, B. D., Linacre, J. M., Smith, R. M., Heinemann, A. W., & Granger, C. V. (1997). FIM measurement properties and Rasch model details. Scand J Rehabil Med, 29(4), 267-272.

End Notes

¹ FIM^® is a trademark of Uniform Data System for Medical Rehabilitation, a division of UB Foundation Activities, Inc.

² Time-limited endorsements are 2-year endorsements for those measures that meet all evaluation criteria except for adequate field testing.

Appendix A: Distribution of CARE Functional Status Item Levels by Provider Type at Admission and Discharge

RTI began analyses of the CARE Item Set functional status data by examining the distribution of the numeric and letter codes for each item for each provider type at admission and discharge. For each functional status item (activity), clinicians assign each patient a level that reflects the assistance the patient needs to complete the activity. There are six levels that range from Independent to Dependent:

Independent (level 6): Patient completes the activity by him/herself with no assistance from a helper.

Setup or clean up assistance (level 5): Helper SETS UP or CLEANS UP; patient completes activity. Helper assists only prior to or following the activity.

Supervision or touching assistance (level 4): Helper provides VERBAL CUES or TOUCHING/STEADYING assistance as patient completes activity. Assistance may be provided throughout the activity or intermittently.

Partial/moderate assistance (level 3): Helper does LESS THAN HALF the effort. Helper lifts, holds or supports trunk or limbs, but provides less than half the effort.

Substantial/maximal assistance (level 2): Helper does MORE THAN HALF the effort. Helper lifts or holds trunk or limbs and provides more than half the effort.

Dependent (level 1): Helper does ALL of the effort. Patient does none of the effort to complete the task.

If a patient did not perform an activity, a letter code is reported. Reasons that a patient may not perform an activity include that the patient had medical or safety issues, the activity was not applicable, the patient refused, or the activity was attempted but not completed.

The bar charts reported in this appendix show the distribution of assistance levels as well as times that the activity did not occur and the amount of missing data. A written summary of the results is provided in the report text.

Eating Codes at Admission

Oral Hygiene Codes at Admission

Toileting Hygiene Codes at Admission

Dressing Upper Body Codes at Admission

Dressing Lower Body Codes at Admission

Wash Upper Body Codes at Admission

Shower/Bathe Self at Admission

Put on/Take off Footwear Codes at Admission

Roll Left to Right at Admission

Sit to Lying at Admission

Lying to Sitting on Side of Bed Codes at Admission

Sit to Stand at Admission

Bed to Chair/Wheelchair Transfer at Admission

Toilet Transfer at Admission

Pick up Object From Floor at Admission

One-Step Curb at Admission

Walk 50 Feet With Two Turns at Admission

Four Steps at Admission

Twelve Steps at Admission

Walk 10 Feet on Uneven Surface at Admission

Car Transfer at Admission

Appendix B: Data Tables for Distribution of CARE Functional Status Item Levels

RTI began analyses of the CARE Item Set functional status data by examining the distribution of the numeric and letter codes for each item for each provider type at admission and discharge. For each functional status item (activity), clinicians assign each patient a level that reflects the assistance the patient needs to complete the activity. There are six levels that range from Independent to Dependent:

Independent (level 6): Patient completes the activity by him/herself with no assistance from a helper.

Setup or clean up assistance (level 5): Helper SETS UP or CLEANS UP; patient completes activity. Helper assists only prior to or following the activity.

Supervision or touching assistance (level 4): Helper provides VERBAL CUES or TOUCHING/ STEADYING assistance as patient completes activity. Assistance may be provided throughout the activity or intermittently.

Partial/moderate assistance (level 3): Helper does LESS THAN HALF the effort. Helper lifts, holds or supports trunk or limbs, but provides less than half the effort.

Substantial/maximal assistance (level 2): Helper does MORE THAN HALF the effort. Helper lifts or holds trunk or limbs and provides more than half the effort.

Dependent (level 1): Helper does ALL of the effort. Patient does none of the effort to complete the task.

If a patient did not perform an activity, a letter code is reported. Reasons that a patient may not perform an activity include that the patient had medical or safety issues, the activity was not applicable, the patient refused, or the activity was attempted but not completed.

The bar charts reported in Appendix A, which were generated from the tables that follow, show the distribution of assistance levels as well as times that the activity did not occur and the amount of missing data. A written summary of the results is provided in the report text.