Performance Improvement 2003. Office of Research, Development, and Information


Development and Psychometric Evaluation of Beneficiary Knowledge Indices from the Medicare Current Beneficiary Survey

This Phase 3 project evaluated the impact of the National Medicare Education Program (NMEP) and assessed changes in beneficiary knowledge from immediately before to immediately after national distribution of the Medicare & You (2000) handbook. This report expanded on previous work by developing knowledge measures from questions included in the Beneficiary Knowledge (BK) supplemental rounds 23 & 26 and Beneficiary Needs (BN) supplemental rounds 24 and 27 of the 1998 and 1999 Access To Care files, which were administered immediately before and after national distribution of the first Medicare & You (2000) handbook. In addition, this report evaluated the psychometric properties of each knowledge measure, including internal consistency reliability and construct validity. One of the most consistent findings across all years and interview types was a strong relationship between higher educational achievement and higher knowledge scores. Another finding was that, as hypothesized, the relationship between knowledge scores and enrollment in managed care was strongest for the seven- item quiz. This quiz included four questions concerning managed care plans, while the three-item quiz and the perceived knowledge index each contained only one question on this topic. The perceived knowledge index seemed to have the best psychometric properties. This index performed well in the reliability and validity analyses. On the basis of content considerations and the psychometric analysis results, the seven-item quiz appeared to be the most useful measure of beneficiary knowledge.

FEDERAL CONTACT: Tricia L. Rodgers 410-786-1833 PIC ID: 7802

PERFORMER: Research Triangle Institute, Research Triangle Park, NC


Early Experience Under Medicare+Choice: Final Summary Report

This report summarized the results that have been produced for the monitoring system of the Medicare+ Choice (M+C) program. The analysis focused on the experience of managed care organizations (MCOs) and beneficiaries across 69 study markets. Of all Metropolitan Statistical Areas (MSAs) with Medicare managed care, the study chose those with a population of at least 1.5 million or a Medicare managed care penetration rate of at least 30 percent. Sixty-nine MSAs met the criteria, and together they accounted for 74 percent of all Medicare managed care enrollees in 1998. The study found that while the changes implemented by the Balanced Budget Act in 1997 (increasing the payment rates in some mainly rural counties and expanding the types of organizations eligible to participate) brought problems, M+C MCO benefits still compared favorably to traditional Medicare supplemented with Medigap coverage and performance on quality indicators was generally good. Across the majority of study markets, M+C MCOs continued to deliver health care services of solid quality and to offer prescription drug coverage at a reasonable monthly premium. The study concluded that Medicare managed care, remains an important source of supplemental coverage, particularly for Medicare beneficiaries who lack employer-based coverage and who do not have access to Medicaid.

FEDERAL CONTACT: Dave Skellan, 410-786-0699 PIC ID: 7169

PERFORMER: Mathematica Policy Research, Inc., Washington, DC


Evaluation of Medicare’s Competitive Bidding Demonstration for DMEPOS: Second-Year Annual Evaluation Report

The Balanced Budget Act of 1997 (BBA 97) authorizes the Department of Health and Human Services to implement up to five demonstration projects of competitive bidding for Medicare Part B items and services, except physician services. On the basis of this authority, to test the use of competitive bidding to set prices for durable medical equipment and prosthetics, orthotics, and supplies (DMEPOS). the Centers for Medicare & Medicaid Services planned and implemented the DMEPOS Competitive Bidding Demonstration. BBA 97 required that the demonstration be evaluated for its impact on Medicare program payments, access, diversity of product selection, and quality. The purpose of this report was to describe the results to date of the evaluation of the DMEPOS Competitive Bidding Demonstration. The impact of the demonstration was evaluated on (1) Medicare expenditures, (2) beneficiary access to care,

(3) quality of care (including diversity of product selection), (4) competitiveness of the market, and (5) the reimbursement system. Based on approximately 2 years of operation, CMS’s Competitive Bidding Demonstration for DMEPOS showed the potential to decrease Medicare expenditures. Competitive bidding has lowered the prices paid by Medicare for the large majority of DMEPOS products and services. Because there is not yet data on utilization, it cannot be definitively concluded that total DMEPOS allowed    charges (the product of price times utilization) fell. It is estimated that Medicare-allowed charges for demonstration products would fall by nearly $8.5 million over the course of the demonstration, a reduction of 20 percent.

FEDERAL CONTACT: Ann Meadow, 410-786-6602

PIC ID: 7173

PERFORMER: University of Wisconsin, Madison, WI


Evaluation of the EverCare Demonstration Program

The EverCare demonstration attempted to reduce medical complications and dislocation trauma resulting from hospitalization, and to save the expense of hospital care when patients can be managed safely in nursing homes with expanded services. The EverCare evaluation combined data from site case studies, a network analysis of nurse practitioners (NPs), participant and caregiver surveys and participant utilization data to examine: (1) a comparison of enrollees and non-enrollees, (2) the process of implementation and operation of EverCare changes in the care process, as well as quality of care, (3) effects of the demonstration on enrollees’ health and health care utilization, (4) satisfaction of enrollees and their families, and (5) effects of the demonstration on the costs of care, as well as payment sources. The report provides a description of the methods used in data collection, descriptive analysis of the study population, and analyses from the resident surveys of health status, function and satisfaction, the NP time study and the analysis of hospital utilization. The main findings from the nurse practitioner time study were NPs spent about 35 percent of their working day on direct patient care and another 26 percent in indirect care activities. The mean time spent on a given patient per day was 42 minutes; of this     time, 20 minutes was direct care. Finally, it was found that NPs’ activities are varied. Much of their time was spent in communicating with vital parties, an important function that supports the physicians’ primary care role and should enhance families’ satisfaction with care. The nurse practitioners in this study were actively engaged in clinical work. By simply being present in the facility on such a frequent basis,  the NP may develop relationships with nursing home staff that ease the identification of early changes in nursing home residents’ status and monitoring of on-going treatments.

FEDERAL CONTACT: John Robst, 410-786-1217 PIC ID: 7185

PERFORMER: University of Minnesota, Minneapolis, MN


Questionnaire Development and Cognitive Testing Using Item Response Theory

The purpose of this project was to design a pool of survey questions to measure beneficiary knowledge of the Medicare program in future rounds of the Medicare Current Beneficiary Survey (MCBS). The intent of the project was to assist the Centers for Medicare and Medicaid Services in assessing how well the National Medicare Education Program is meeting its consumer information goals. The researchers developed questionnaire items using a comprehensive multi-step process that included: (1) background research, (2) review of Medicare informational materials and Medicare knowledge surveys, including the Medicare Current Beneficiary Survey (MCBS), and (3) multiple meetings and discussions with CMS and  the project’s seven-member Technical Advisory Panel (TAP). Knowledge scores calculated after the first round of interviews showed that respondents answered between 15 and 43 (out of 51) knowledge items correctly, or 29 to 84 percent. Based on feedback from the TAP and CMS, the survey questions were revised for a second round of testing. The language was simplified and more consistent terminology was used throughout. Fewer changes were suggested by the second round testing than by the first, suggesting that the instrument worked better. The report determined that using Item Response Theory (IRT) would produce a more precise measure of beneficiary knowledge, allowing CMS to more accurately determine areas in which beneficiaries lack knowledge and could use more information. In addition, an IRT-based knowledge scale would allow CMS to track knowledge, thereby allowing for the evaluation of the effectiveness of interventions and education programs.

FEDERAL CONTACT: Tricia L. Rodgers 410-786-1833 PIC ID: 7786

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

View full report


"pi_2003.pdf" (pdf, 346.19Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®