Evaluation of Selected Aspects of the National Medicare Education Program: Final Design Report

January 10, 2008

Prepared by:
RTI International
Lauren McCormack, PhD, MSPH; Nancy Berkman, PhD; Karen Isenberg, MPH; Douglas Rupert, MPH; Jennifer Uhrig, PhD; Derek Brown, PhD; Leslie Greenwald, PhD, MPA

Prepared for:
Office of the Assistant Secretary for Planning and Evaluation (ASPE)
U.S. Department of Health and Human Services (HHS)

 

Acknowledgment

The authors would like to acknowledge input provided on the report by our expert consultant, Shoshanna Sofaer, DrPH.

Introduction

The purpose of this project is to assist the Assistant Secretary for Planning and Evaluation (ASPE) in designing a systematic evaluation of the National Medicare Education Program (NMEP), a program sponsored by the Centers for Medicare & Medicaid Services (CMS) to educate and inform Medicare beneficiaries. A major substantive activity under this contract was an environmental scan or inventory of the existing data related to the NMEP. This inventory focused on the question “What evidence is currently available to indicate how well the NMEP program is working?” Based on this inventory, we then designed several options for a comprehensive evaluation of the NMEP that would assess the extent to which this program is meeting its programmatic goals as Medicare continues to expand and evolve. 

This report contains five chapters. In this introductory chapter, we provide background on the Medicare program as well as the NMEP. In Chapter 2, we summarize our review of prior NMEP evaluations and identify gaps in the research. In Chapter 3, we discuss measurement issues related to designing an evaluation, such as selecting and defining key outcome measures. In Chapter 4, we present four different study design options and walk through their rationales, research questions, data sources, and analytic methods. In the final chapter, we discuss advantages and disadvantages of the design options and identify future directions.

1.1 Background on the Medicare Program

The Medicare program provides health insurance to more than 44 million beneficiaries nationally and provides more than $374 billion in health care benefits. Participation in the Medicare program depends on whether individuals are aged 65 or older, have selected disabilities, or have end-stage renal disease (ESRD). The majority (about 85%) of the Medicare population is aged, and approximately 14% of beneficiaries are disabled. The remaining Medicare population (less than 1%) is eligible for Medicare benefits because of ESRD. The Medicare program is the largest health insurer in the United States and is crucial in ensuring health care for older adults.

The Medicare program has its roots in the Social Security program. The original Social Security Act, enacted in 1935, provided minimal income security through social insurance and included no provision for coverage of health care expenses. Despite the goals of providing a basic social income safety net, policy makers became aware that health care costs, particularly for the aged and disabled populations with multiple health care needs, would quickly undermine the income protections provided under the Social Security program. Therefore, the Medicare program was signed into law in 1965 as a mechanism to add health insurance to the already existing national Social Security program.

Medicare is partially financed through a combination of payroll taxes, beneficiary premiums, and beneficiary cost sharing. For most of its history, the Medicare program (now commonly referred to as “Original Medicare”) consisted of hospital insurance (covered under Medicare Part A) and general medical insurance, including physician services (Medicare Part B). Coverage under the Original Medicare program is offered under a fee-for-service (FFS) system in which Medicare pays medical providers predetermined rates for specific health care services.

Medicare, however, has evolved over time to include new programs. During the late 1980s, the program began to experiment with providing services to beneficiaries through capitated health maintenance organization (HMO) programs. Managed care programs, such as HMOs, offered an alternative way for Medicare beneficiaries to receive their Medicare benefits. In exchange for enrolling in a Medicare managed care plan, beneficiaries often received additional, nonstatutory benefits, including preventive care. Popularity of Medicare managed care (known first as the TEFRA HMO program) grew steadily , and the program was significantly expanded through passage of the Balanced Budget Act (BBA) of 1997. Under the BBA, which created the Medicare+Choice (M+C) program, Medicare expanded the types of managed care organizations eligible to contract with Medicare. Through the M+C program, also known as Medicare Part C, managed care organizations added new benefit options, including preferred provider organizations (PPOs) and private fee-for-service (PFFS) plans to the original HMO options. Although all beneficiaries have access to managed care options, the Medicare managed care program (currently renamed Medicare Advantage) provides coverage to a minority of Medicare beneficiaries (about 18.5% in 2007).

Possibly the most significant change to the Medicare program came with the enactment of the Medicare Modernization Act (MMA) of 2003. The MMA added to the Medicare program a benefit long absent from original Medicare voluntary prescription drug coverage. The Medicare Part D program, which became effective in January 2006, provides basic and catastrophic prescription drug coverage to beneficiaries through private insurance plans. Under Part D in 2006, beneficiaries received basic coverage of prescription drugs up to (depending on their specific plan) about $2,400. Catastrophic coverage continued after beneficiaries have paid out-of-pocket costs of $3,850. The private plans that provide Part D coverage include a wide range of stand-alone prescription drug plans (PDPs) and Medicare Advantage prescription drug plans (MA-PDs). Medicare beneficiaries choose annually which type of mechanism (PDPs or MA-PDs) they want for their Part D coverage. Beneficiaries also have the option of continuing to receive “creditable” drug coverage (e.g., coverage that is as good as the Part D benefit) through another source, such as a former employer or the Veterans Administration.

The Medicare Part D program is credited with providing critical prescripti on drug coverage to almost 24 million Medicare beneficiaries, some of whom had no reliable coverage before the new program. However, Medicare Part D has faced some criticism for its complex, hard-to-understand benefit design; the large number of coverage plan options; and the coverage gap between basic and catastrophic coverage.

1.2 National Medicare Education Program (NMEP)

CMS initiated the NMEP in 1998 to announce programmatic changes and to help Medicare beneficiaries make more informed health care decisions. The NMEP employs numerous channels to educate beneficiaries about their Medicare benefits; health plan choices; supplemental health insurance; beneficiary rights, responsibilities, and protections; and health behaviors. These channels include NMEP print materials like the annual Medicare & You handbook; toll-free telephone services like 1-800-MEDICARE; a CMS-sponsored Web site (http://www.Medicare.gov) that provides basic and comparative information on health insurance options and quality-of-care measures; a broad regional office education initiative called Regional Education About Choices in Health (REACH); the national alliance network; national training and support for information givers; and enhanced beneficiary counseling and other services from the State Health Insurance Assistance Programs (SHIPs) (CMS, 2007CMS, 2007). Following is a brief synopsis of each major NMEP activity directed by CMS.

Print Materials CMS provides a variety of print materialsincluding its flagship publication, the Medicare & You handbook to educate the beneficiary population about the Medicare program and health plan choices. CMS mailed the first Medicare & You handbook in November 1998 to more than 5 million Medicare beneficiaries in five pilot statesArizona, Florida, Ohio, Oregon, and Washington. A condensed Medicare & You bulletin was mailed to beneficiaries in the remaining 45 states and territories. CMS revised the handbook based on findings from the pilot test and, in 1999, mailed out the handbook nationally. Since that time, the handbook has been revised annually, incorporating changes based on beneficiary feedback and testing, consultation with low literacy experts, and feedback from lessons learned. Recent additions to the handbook include descriptions of patient rights and protections, preventive benefits, availability of health care options including Original Medicare and Medicare Advantage Organizations, and Prescription Drug Benefit (Part D) coverage. Medicare & You is available in English, Spanish, Braille, audiotape, and large print versions. It is mailed to new beneficiaries at the time of enrollment and, as mandated by the Social Security Act, to all beneficiaries annually. In addition to the national version, 26 area-specific versions also exist for designated regions.

In addition to the Medicare & You handbook, CMS publishes close to 40 topic-specific publications under the broad headings of (1)medical decision-making, (2)coverage and benefits, and (3)payment and rights. These materials discuss a variety of topics, including covered services, health care choices, supplemental health insurance coverage, Medicare rights and protections, and costs for specific services. As with the handbook, most publications are available in English, Spanish, Braille, audiotape, and large print versions.

Internet Activities In June 1998, CMS launched a Web site designed for beneficiaries ( http://www.medicare.gov/). The BBA of 1997 mandated that an Internet site be developed to provide accurate and reliable information to beneficiaries on Medicare basics, M+C plan comparison, and quality information to promote informed choices. Information on the site includes the Medicare & You handbook in printable format; general information about Original Medicare, Medicare Advantage plans, and Medicare Part D; a directory of Medicare participating physicians; lists of resources and telephone numbers for obtaining information about Medicare; and procedures to follow when filing Medicare claims or appeals. In addition, Web site users can order publications, search the site for specific content, and subscribe to a listserv that provides routine e-mail updates. The Web site also hosts several comparative databases:

• Medicare Prescription Drug Plan Finder
• Medicare Options Compare
• Hospital Compare
• Home Health Compare
• Nursing Home Compare
• Dialysis Facility Compare

Each of these databases allows individuals to compare their health plan options in a given area (by city, state, county, and zip code). The Medicare Prescription Drug Plan Finder allows Medicare beneficiaries and their caregivers to generate comparisons of Part D plans based on the beneficiary’s prescribed medications. Medicare Options Compare provides detailed comparisons of Medicare Advantage and other supplemental plans. Each of the other Compare sites contains publicly reported information about the services and quality of care provided at individual hospitals, home health agencies, nursing homes, and dialysis facilities.

Toll-Free Medicare Helpline 1-800-MEDICARE is a toll-free service available across the United States. The BBA of 1997 directed that a toll-free helpline be maintained to handle inquiries about beneficiaries’ benefits and available options under Medicare. CMS phased in the helpline between November 1998 and March 1999. The helpline operates 24 hours a day, 7 days a week, and is staffed by customer service representatives (CSRs) from 8:00 am to 4:30 pm. CSRs help answer general Medicare questions, provide information regarding health plan choices in a given area, process requests for plan comparison information and educational materials, and make referrals to other information sources when appropriate. Based on the complexity of questions, callers are directed to CSRs who can address increasingly difficult questions. Callers also have the option of using an automated response line to find answers to commonly asked questions or to request educational materials. Both CSR assistance and the automated response line are provided to callers in either English or Spanish. Also, a TTY line is available for the deaf and hearing impaired.

REACH Each of CMS’ 10 regional offices partners with community-based organizations to conduct educational outreach efforts at the regional, state, and local levels. Known as REACH, many of these activities are targeted to specific minority groups and beneficiaries with disabilities who are eligible for both Medicare and Medicaid. The BBA of 1997 required that each November, in conjunction with the annual health plan election period, a nationally coordinated educational and publicity campaign provide M+C-eligible individuals with information about health care plans and the election process. This outreach includes public presentations, exhibits at local health fairs, advertisements, radio talk shows, newspaper editorials, and other health-related events.

Enhanced Beneficiary Counseling from SHIPs SHIPs provide free, personalized health insurance counseling and assistance to Medicare beneficiaries through in-person meetings, telephone helplines, and group outreach and education. A SHIP Web site (www.shiptalk.org) also provides assistance to beneficiaries. These organizations are part of each state’s Office of Aging or Office of Insurance. Extensive training has been conducted with SHIP staff and volunteers to prepare them to help Medicare beneficiaries make informed decisions about their health care.

National Train-the-Trainer Program This program trains educators who work with CMS staff, partner organizations, and others who provide information, assistance, and education to Medicare beneficiaries. The focus of the program is to provide updated Medicare information to these intermediaries while enhancing their teaching and training skills. The training is provided in multiple formats, including videos, audio conference calls, Web casts, classroom presentations, resource kits, and quick-start guides. A Spanish-language training also is available for those educators and partner agencies that work with Spanish-speaking populations.

National Alliance Network CMS has enlisted national and local partners to support and participate in the NMEP. More than 200 national and local organizations that work on behalf of aged and disabled Americans are involved in this publicprivate partnership, including advocacy groups, health plan organizations, providers, employers, and unions. Together with CMS, these national partners reach out to other organizations at the state and local levels. They, in turn, educate Medicare beneficiaries and other interested organizations and individuals to help them better understand changes to the Medicare program. These education efforts include a Web site, print publications, forums, training sessions, public presentations, health fair exhibits, and other health-related events.

Employer Forum The Employer Forum is composed of employer members of the Washington Business Group on Health, Midwest Business Group Health Care Association, other employers, other interested organizations, and CMS staff. Participants in this Forum represent over 400 employers and more than 50 million employees. The Forum functions overall as a vehicle for an ongoing dialogue between the employer community and CMS, and an Employer Forum Conference Call is held monthly for 1 hour on the second Thursday of the month. This conference call was an outgrowth of CMS’ Employer-Union Conference conducted in March 2000, where the primary outcome was a strongly expressed desire by the employer community to continue an ongoing dialogue with CMS.

Health Outreach Initiative Zeroing In On Needs (HORIZONS) To ensure that information would be easily understood by various cultural groups, CMS launched the HORIZONS project in 1999 to improve communication with Medicare beneficiaries from diverse populations. These populations often have barriers to accessing information, such as language, location, or low literacy. Team members developed communication strategies targeted at four specific groups of beneficiaries: African Americans, Hispanics, Asian Americans and Pacific Islanders, and American Indians and Alaska Natives.

Part D and Other Educational Activities Since the MMA of 2003, fewer Medicare educational activities and materials have been formally labeled as being part of the NMEP campaign. Nevertheless, these activities continue to promote understanding of Medicare coverage options and often incorporate information about Part D and prescription drug coverage. At the same time, new educational activities have emergedsuch as the Welcome to Medicare brochure and the Part D prescription drug discount cardsthat were not part of the original NMEP program.

1.3 NMEP Evaluation Activities

Since 1998, NMEP activities have been monitored using various methods, including surveys, special research projects, and focus groups and interviews with local officials in communities across the country. The program also includes extensive and continued evaluation of beneficiary education activities. As a result of this testing, beneficiaries and CMS partners are routinely asked for the type and formats of information they want as well as for their assessment of the available materials and resources. CMS is also focusing on developing materials and educational opportunities targeted at specific topics and beneficiaries with specific interests.

Despite these evaluations, many NMEP activities remain largely unassessed, and several critical questions about the program are still unanswered. To date, few evaluations have examined what knowledge beneficiaries need to make informed enrollment decisions, to what extent NMEP activities promote informed decision making (IDM), and what the return on investment is for such activities. These are crucial questions because, as the Medicare program continues to grow in complexity, beneficiaries will become increasingly reliant on educational tools and materials in making enrollment decisions. If these tools are ineffective, beneficiaries may make uninformed decisions that lead to excessive program costs and, ultimately, poor health.

Evaluation of Selected Aspects of the National Medicare Education Program

Previous NMEP Evaluations

There are many important reasons to invest in future comprehensive evaluations of the NMEP, particularly as this program continues to evolve (and in some cases expand) to meet the changing information needs of Medicare beneficiaries. Since 2003, the focus of NMEP has appeared to shift with the MMA and the introduction of the Medicare Part D program. More recent studies have focused less on what has traditionally been defined as NMEP and more generally on beneficiary education or Part D education, specifically. Many of the Part D -focused activities, such as reliance on a 1-800-MEDICARE hotline to disseminate information on prescription drug options, are essentially significant expansions of original NMEP modelsmodels that often received only minimal evaluation. In addition, federal appropriations for NMEP include monies for traditional NMEP activities as well as MMA beneficiary education. This evolution toward more prescription drug issues may make it difficult (particularly with declining agency resources) to focus on the broader set of Medicare education topics that had historically been the focus of the NMEP, such as awareness of insurance option choices and managed care.

Given these issues, and the fact that the Medicare program has changed significantly in just the last few years, it is even more critical that beneficiaries are knowledgeable about the Medicare program and have the resources and support they need to make informed decisions in this changing environment. The NMEP is a critical part of an overall effort to educate Medicare beneficiaries and other stakeholders about the program so that health and health care decisions can be informed. This is consistent with other changes in the health care system that now require greater participation by consumers and patients in the decision-making process. It is likely that this expectation for involvement will continue to grow, placing greater responsibility on beneficiaries and their families. Poorly informed decisions could not only have a negative health impact on beneficiaries, but also end up costing the Medicare program more money in the long run.

Significant financial resources have been, and continue to be, invested in the NMEP. Activities that build on original NMEP activities, aimed at meeting the expanding information needs of beneficiaries for use of programs such as Part D, should ensure that only effective education initiatives are expanded and that ineffective and/or inefficient initiatives are modified (rather than perpetuated). Information is sorely needed about the return on these NMEP investments. Cost assessments can examine where there are efficiencies and inefficiencies in the program and where resources can be directed to affect the larger number of individuals. For these reasons, continuing to monitor and evaluate NMEP has increasing policy importance.

Evaluations of the NMEP should assess the effect of the progr am from multiple perspectivesthat of beneficiaries and their families; Medicare providers; and stakeholders such as employers, CMS partners, and policy makers. In doing so, the evaluation must be comprehensive in nature and will mostly likely require a mixed-method approach. To adequately measure the impact of the evaluation over time on key variables of interest, an evaluation should have a longitudinal component and rigorous analysis of the resulting data. This type of evaluation can help us learn whether the program is meeting the informational needs of its constituents and if it is successful in facilitating access to high-quality health care for those who are eligible.

We conducted an environmental scan to inventory, review, and summarize materials describing and evaluating all of CMS’ current and past NMEP activities.

2.1 Methodology for the Environmental Scan of Prior Evaluations

We conducted a systematic review of both the peer-reviewed published literature and unpublished materials (e.g ., official CMS and other agency reports and documents; trade publications; publications by advocacy, watchdog, and other stakeholder organizations; and congressional testimony). The key benefits of using a systematic evidence-based approach over a standard literature review are (1) the systematic approach can withstand scrutiny because the literature selection and abstraction process is not biased and (2) because it is designed to answer specific questions, this approach clearly focuses on the information needed for the review.

We focused our search of the peer-reviewed literature on manuscripts published from January 1998 (the first year the Medicare & You handbook was distributed) through December 2006. Unpublished materials were limited to those available through the Internet, directly from CMS, or from the project team’s libraries from September through December 2006. Included studies were limited to those evaluating NMEP activities; we excluded other CMS evaluations that may be considered to be related to beneficiary education activities if they were not identified as part of NMEP. Given this criterion, we erred on the side of inclusion; the abstractor and the task leader reviewed the studies and determined whether they met our inclusion criteria. Similarly, the task leader reviewed abstractions of all included studies to ensure they were accurate and complete. We reviewed a total of 53 studies.

2.2 Results of the Environmental Scan

A key determinant of a successful campaign is how well it achieved its goals. Therefore, an evaluation of the NMEP should assess how well it is meeting its goals. In turn, the measures used in the evaluation of the NMEP should be linked to these goals and be able to adequately assess whether they are being met. The previous evaluations of the NMEP that we reviewed have assessed each of the three goals of the NMEP to some degree (CMS, undatedCMS, undated). The goals are the following:

1. Create awareness among the beneficiary community of the new choices that are available.
2. Develop an understanding of those new choices and their ramifications.
3. Help beneficiaries use new information to make informed health care choices.

Create awareness among the beneficiary community of the new choices that are available	Develop an understanding of those new choices and their ramifications	Help beneficiaries use new information to make informed health care choices
General
Awareness of Medicare information sources (2)	Perceived knowledge of Medicare (5) General knowledge of health insurance (3) Demonstrated knowledge of Medicare (16) Perceived knowledge of Part D (1) Demonstrated knowledge of Part D (2) Confidence in ability to select best plan (2)	Information availability (5) Information needs (3) Information seeking (7) Information usefulness (2) Information satisfaction (5) CMS as a source of managed care information (1) Trust in information from CMS (2) Use of Medicare information sources (2)
Medicare & You handbook
Receipt of handbook (10) Awareness of (1) Recognize source (3) Awareness of Spanish language version (1)	Understandability of the handbook (5) Ease of use of handbook (5) Perceived knowledge (1) Demonstrated knowledge (7) Accuracy of content (1)	Use of/read handbook (12) Satisfaction/usefulness (8) Perceived usefulness (1) Trust in information (1) Frequency of use (1) Use look up phone number (3) Use find plan information (5) Use find services covered (1) Use cost comparison information (4) Use plan decision (3)
1-800-MEDICARE
Awareness of (5)	Ease of use (2) Accuracy of information provided (4) Operator knowledge (2) Appropriate referrals from operator (1)	Satisfaction (4) Use called helpline (12) Use questions answered (1) Use plan decision (1)
Medicare Web site
Awareness of (8)	Ease of use (3) Content appropriateness (2)	Use of Web site (11) Use of low vision group (1) Use of plan decision (1) Usefulness (12) Interest in/likelihood to use (2)
REACH Activities
Awareness of (1)		Use of plan decision (1) Reach/attendance (2)
State Health Insurance Assistance Programs (SHIPs)
Awareness of (7)	Type of information provided (1)	Beneficiary use of (5) Beneficiary likelihood to use (1) Use of plan decision (1)
Train-the-trainer program
		Participant satisfaction (1)
Note: Numbers in parentheses indicate the number of studies in which these measures have been used corresponding to the three goals.

Overall, we found a fairly substantial amount of information that described and/or evaluated the NMEP, presented primarily in technical reports by government contractors with highlights repeated in the peer-reviewed literature and, to a lesser extent, in reports produced by nongovernmental organiza tions and distributed via the Web. These studies employed a variety of research methods, including both qualitative and quantitative techniques. Qualitative methods included focus groups and individual in-depth interviews for cognitive and usability testing of NMEP information products and case studies. Among the studies that used quantitative methods, descriptive statistics were more commonly used than multivariate methods. Across studies, quantitative methods were more common than qualitative. Three studies used randomized study designs. A handful of literature reviews and papers described the NMEP program in terms of its goals and approaches. Overall, a small number of larger-scale evaluations took place over the years. They were conducted particularly early in the NMEP program’s history, included more detailed study designs and larger sample sizes, and used more rigorous statistical methods. Numerous other studies presented polling results and were geographically limited; these findings may not be generalizable to the larger population of beneficiaries, and analyses relied only on descriptive statistics.

2.2.1 Outcomes Addressed

Previous studies have addressed each of the three goals of the NMEP awareness, understanding, and access/use/satisfactionwith the greatest number focusing on Medicare-related knowledge. Most studies devoted some attention to some of the common vulnerable subgroups. Overall, the findings of these studies suggest that, although the percentages of beneficiaries who have access to or are aware of NMEP resources and Medicare information appear to have increased over time, access and awareness remain lower than might be considered necessary for the majority of the beneficiary population to successfully navigate such a complex program. Several studies showed low to moderate levels of understanding of the Medicare program and related health insurance options, particularly in the beginning of the NMEP implementation period. In multiple studies (including those that used the MCBS), knowledge was higher among those beneficiaries who were exposed to the Medicare &You handbook. However, when the Medicare Current Beneficiary Survey (MCBS) was the data source used, the questions frequently had “true/false” response options, and those choices give respondents a high probability of guessing the correct answer. Although the MCBS was fielded on an ongoing basis over this period, we did not find external evidence that it was used to consistently monitor knowledge or the other NMEP goals.

An assessment of knowledge is frequently used in the communication literature as an intermediate outcome, but it has recognizable limitations because of the potential weak link between knowledge and actual behavior. In the studies we reviewed, we found little information on what people did with the information they had access to or were aware of , or the new knowledge they gained; for example, limited data were available on how the information was used to make health plan decisions. We view this as a research gap that remains to be filled. Similarly, we also did not see evidence of rigorous intervention and experimental design research, especially with respect to potential ways to educate beneficiaries about NMEP issues and whether changes are made as a result of such interventions. Longitudinal studies were also generally lacking, and they are important for examining impact over time, including decision quality. The MCBS may be one vehicle for this activity.

We found a fair amount of data assessing beneficiary satisfaction with the information available to them. In these studies, beneficiaries generally reported that they were satisfied with information available to them. Lower satisfaction levels were evident among some vulnerable subgroups: these subgroups also tended to use the NMEP information less frequently. Given the ceiling effects (i.e., high scores) that often occur with measuring satisfaction, it may be prudent to de-emphasize this variable.

Little emphasis has been placed on evaluating the relative impact and cost-effectiveness of various NMEP activities in relation to one another. The research has also not emphasized the effect of changing, enhancing, or otherwise modifying interventions to determine if these activities could improve beneficiary access or understanding of the information. Particularly given limited financial resources, more activities in this area may be warranted.

2.2.2 Evaluations of Specific NMEP Activities

We also examined how previous evaluations have studied the full range of specific NMEP activities. Table 2-2 highlights the number of studies that addressed each NMEP activity out of the 53 total studies reviewed and congressional monies allocated to them between 1999 and 2005. The Medicare & You handbook was the most studied specific activity, followed by the 1-800-MEDICARE helpline.

Table 2-2. Extent of Previous Research and Funding for NMEP Activities

NMEP Activity	Number of Studies Reviewed (N = 53)		Total Budget Allocation in Millions (1999 2005)a
Medicare & You handbook	29		$271.04b
1-800-MEDICARE helpline	18		$650.30
http://www.medicare.gov/	19		$70.00
SHIPs	13		$206.26c
REACH activities	4		$206.26c
Train-the-trainer program	1		Unknown
a In 2005, funding for the NMEP components also included funds designated as part of MMA beneficiary education. b Amount includes all print materials for the NMEP and is not exclusive to the handbook. c Amount is for community outreach activities including both SHIPs and REACH.

2.3 Summary Assessment

We offer the following observations about the studies previously conducted to evaluate the NMEP.

CMS has sponsored numerous large and rigorously conducted evaluations of the NMEP. Studies include both quantitative and qualitative methods. Many of the individual studies appear to be of reasonable quality in terms of weighting responses to generalize the data to the larger beneficiary population and having adequate precision to assess access and use of NMEP components. Studies were predominantly of beneficiaries and placed little emphasis on proxy audiences.

P revious NMEP evaluations predominantly assessed (a) beneficiaries’ knowledge of Medicare and Part D; (b) beneficiaries’ awareness of the various NMEP components; (c) the appropriateness of information provided through various components to help beneficiaries understand their choices; and (d) beneficiaries’ use of and satisfaction with NMEP components.

P revious NMEP evaluations have lacked measures of whether NMEP and its components have led to change in the beneficiary population. These include whether beneficiaries (a) are more aware of their Medicare choices; (b) better understand and have improved knowledge of their choices; and (c) make informed health care choices at all and/or over time. There is little information on what people do with the information they access or are made aware of through NMEP or the knowledge they gain. It is also unclear how NMEP information is used to make health plan decisions.

Previous NMEP evaluations have placed little emphasis on evaluating the relative impact and cost-effectiveness of various NMEP activities.

The findings of previous NMEP evaluations suggest that beneficiary access to or awareness of NMEP resources and Medicare information appear to have increased over time; however, at levels lower than what the majority of the beneficiary population would need to successfully navigate through Medicare. Previous studies have not been longitudinal to be able to assess NMEP’s impact over time adequately, such as increase in knowledge. See Appendix A for an overview of prior studies.

The Program Assessment Rating Tool (PART) used by the Office of Management and Budget (OMB) to assess whether federal program and agencies are meeting their goals is a possible design element to consider for evaluating the NMEP. To date, only the Medicare program as a whole has been evaluated under the PART, with no specific mention of the NMEP.

In reviewing previous NMEP evaluations, we found that, since 2003, the focus of the NMEP has appeared to shift with the MMA and the introduction of the Medicare Part D program. Some refer to the NMEP as the Medicare & You campaign, possibly to increase “brand” recognition of the handbook. This changing focus of the NMEP suggests that it is an evolving program, which makes it responsive to the high-priority issues created by new legislation. It is consistent with CMS intentions for the program to be dynamic over time. However, this evolution toward a focus on prescription drug issues makes it challenging (particularly with declining agency resources) to focus on the broader set of Medicare topics that had historically been the focus of the NMEP, such as awareness of insurance option choices and managed care, that underpin the prescription drug program. This is particularly problematic because the MMA also introduced new managed care options and led to greater availability of options initially created under the BBA, such as private FFS plans. As a result, there is now “more to know” about Medicare, which also makes the issue of measuring increases in outcomes, such as awareness and understanding, more complex.

Measurement Issues

Prior to designing the evaluation options, we considered which ou tcome measures to focus on and how to define a successful outcome. We also identified and prioritized specific NMEP activities for evaluation. We discuss each of these issues in this chapter within the broader rubric of measurement.

When evaluating health -related education campaigns such as the NMEP, we recommend reviewing the social marketing literature to assess the degree to which key principles of social marketing and communications have been considered in campaign development and implementation, and how they can guide the evaluation design and actual evaluation. A key criterion for determining whether a campaign has been successful is if it has met its goals (Kotler et al., 2002Kotler et al., 2002). Thus, campaign goals must be clearly articulated and widely disseminated; the goals need to be measurable, recognizing, however, that they may change over time as the campaign matures. Another important social marketing principle to consider in designing and evaluating campaigns is whether an audience-centered approach has been followed (Andreason, 1995Andreason, 1995). Consumers, including Medicare beneficiaries and related stakeholders in this case, should be included in the campaign development and evaluation process to determine whether the information meets their needs and is user-friendly.

It is also important to recognize that Medicare beneficiaries and stakeholders are heterogeneous. Therefore, segmenting the campaign’s audience into smaller, more homogeneous market segments can help gain an understanding of perceptions and information needs. An overarching question focus of an education program or campaign is whether it had the intended impact in promoting behavior change. In health promotion campaigns, behavior change may be measured by eating better or exercising, but in a campaign to educate people about the Medicare program, we argue that the behavior is whether people are making informed decisions. However, we recognize that defining and measuring that concept in the context of Medicare and insurance-related decisions can be challenging.

3.1 Informed Decision Making

Fortunately, the IDM literature offers some useful guidance for campaign design, evaluation, and measurement despite its focus on clinical decision making instead of insurance choices. According to Briss and colleagues (2004), IDM occurs

When an individual understands the nature of the disease or condition being addressed; understands the clinical service and its likely consequences, including risks, limitations, benefits, alternatives, and uncertainties; has considered his or her preferences as appropriate; has participated in decision making at a personally desirable level; and either makes a decision consistent with his or her preferences and values or elects to defer a decision to a later time . (Briss et al., 2004p. 68)

Thus, IDM implies that a person understands the choices he or she faces and the relative advantages and disadvantages of these choices. However, neither a standard definition of an “informed person” nor a widely accepted metric for how knowledgeable a person must be to facilitate IDM exists (Rimer et al., 2004 Rimer et al., 2004).

Mullen and colleagues (2006) characterize IDM as occurring using a multiconstruct approach, which is consistent with the Briss et al. definition. In Table 3-1 , we categorize the constructs offered in Mullen etal. as more and less relevant to health insurance decision making. We propose that many of the more clinically rooted IDM constructs are also relevant and should be considered when applying IDM to health insurance choice, yet we recognize that some are not as relevant. For example, assessing whether a Medicare-related decision was informed could involve assessing the intervention’s impact on knowledge, whether the beneficiaries were involved in the decision-making process to the extent that they wanted to be, whether they considered their values and preferences in making their decision, and to what extent the beneficiaries were certain about the decision they made (e.g., felt that this was definitely the right decision for them versus not sure at all that what they have chosen is right for them).

Table 3-1. Informed Decision-Making Constructs: Applicability of More Clinical-Based Constructs to Insurance-Related Decision Making

Common Construct from Informed Decision Making Literature			More Applicable			Less Applicable or Not Applicable
Demonstrated knowledge			X
Perceived threat						X
Decisional balance (i.e., was decision consistent with values)			X
General attitude toward testing						X
Role preference (i.e., the extent to which an individual wants to make the decision him/herself or defer to others)			X
Utilities/values (i.e., the importance placed on potential benefits or risks of a given course of action)			X
Treatment preference						X
Decisional self-efficacy (i.e., self-confidence in one’s decision and decision-making ability)			X
Discussion with a clinician			X
Test (insurance) preference			X (more applicable if relevant to insurance as opposed to treatment decision)
Screening intention						X
Satisfaction with the decision and decision-making process			X
Decision conflict (i.e., the state of uncertainty about the course of action taken)			X
Source: RTI International based on measures suggested in Mullen et al. (2006)Mullen et al. (2006).

3.2 Specifying Outcomes

Program evaluation typically focuses on measuring processes and outcomes (Kotler et al ., 2002Kotler et al., 2002). Process measures focus on campaign activities and assess how well the campaign was implemented as intended. Outcome measures assess specific results that can be attributed to the campaign (Kotler et al., 2002Kotler et al., 2002). Table 3-2 presents common process and outcome measures used in evaluating health communication programs.

Table 3-2. Examples of Common Process and Outcome Measures

Process Measures			Outcome Measures
Program reach Table 3-2. Examples of Common Process and Outcome Measures			Knowledge
Media coverage			Attitudes and/or beliefs
Number of materials disseminated			Awareness
Changes in program policy or infrastructure			Behavioral intentions
Assessment of implementation			Behavior
Total impressions or Gross Rating Points (GRPs)			Satisfaction
Partner participation and/or contributions			Responses to campaign components
Source: Kotler et al. (2002).

Recognizing that there are limited resources with which to implement health communication campaigns, cost-effectiveness analysis (CEA) can inform policy makers ’ resource allocation decisions (Frick, 2006Frick, 2006). CEA can answer questions such as the following:

What is the cost per person reached by different channels or any channel of the campaign? Which channel produced the greatest change for the cost?

What is the incremental change in outcome expected from a specific increase in cost? What is the incremental cost -effectiveness of different combinations of communication activities?

Which types of health communication interventions are most cost -effective? How does the health communication program rank in cost-effectiveness relative to other programs and interventions seeking to effect behavior change?

What specific factors help or hinder cost-effectiveness of health communication campaigns?

However, to date, it has been rather uncommon for evaluations of health communication campaigns to include CEA (Hutchinson and Wheeler , 2006Hutchinson and Wheeler, 2006). Several challenges to conducting CEA on health communication campaigns include obtaining appropriate estimates of costs, agreeing on a single primary outcome, and measuring the effectiveness of the campaign in bringing about the desired change (Bertrand, 2006Bertrand, 2006; Frick, 2006Frick, 2006). Randomized control trials are the gold standard for evaluating cost-effectiveness. However, randomized experiments are only feasible and appropriate for interventions in which the researcher can control who receives an intervention and at what level of intensity. Exposure to an intervention cannot be easily controlled or randomized in many health communication interventions, especially those that use mass media. Nevertheless, if there is a direct link between the communication program and the outcome (e.g., exposure or program reach), CEA can still be fairly straightforward. On the other hand, CEA can become much more complex if the link between the program and the outcome is less direct. In this case, attributing behavior change to campaign exposure becomes more tenuous because the campaign may be only one of multiple factors that influenced individual behavior change (Guilkey, Hutchinson and Lance, 2006Guilkey, Hutchinson and Lance, 2006). In practice, data for conducting CEA are collected at the same time as other data being collected to evaluate the effectiveness of the health communication program. However, CEA should follow the assessment of program effectiveness because if the program is found ineffective, there is no reason to conduct CEA (Frick, 2006Frick, 2006).

3.3 Defining Successful Outcomes

A successful campaign meets or exceeds its goals and objectives (Kotler et al. , 2002Kotler et al., 2002). Therefore, it is critical to establish specific, measurable, realistic, and meaningful campaign goals and objectives. It follows that successful outcomes would demonstrate that the campaign has met or exceeded its goals and objectives. One challenge in defining successful outcomes for the NMEP is that its goals are not very specific or quantifiable. For example, the goal related to creating awareness does not indicate the percentage of the beneficiary population that should be aware of Medicare’s new choices or an expected change in levels of awareness attributable to the campaign. As a result, it is difficult to define what absolute number or how much of a change is needed for the outcome to be considered successful. Because the NMEP is multifaceted, a true baseline does not really exist. However, the early evaluations of the Medicare & You handbook conducted by McCormack and colleagues can provide baseline measures of some outcomes, most notably Medicare-related knowledge prior to the distribution of the first Medicare & You handbook. Although no specific goals were set for how much the NMEP would be expected to increase Medicare-related knowledge over time, we can at least evaluate whether knowledge has increased or held steady in the years following the implementation of the NMEP. The same approach may be feasible with other measures such as awareness and use of different NMEP components (e.g., toll-free number, Web site).

3.4 Recommended Outcome Measures

Because a comprehensive evaluation of the NMEP should assess how well it is meeting its stated goals, we recommend including outcome measures that assess awareness, knowledge, and use of information. Table 3-3 lists specific outcome measures pertaining to each of the three goals along with potential data sources.

One strategy to be able to better determine the impact of the NMEP is to identify the key messages that are promoted in beneficiary education materials, in particular, the Medicare & You handbook, because it is mailed to beneficiaries each year. The benefit of identifying the key messages is that they can be used to measure awareness and knowledge outcomes. Based on the 104-page 2007 handbook’s table of contents, over half of the handbook focuses on information related to the various Medicare plans. In addition, approximately one-third of the handbook is devoted to explaining beneficiary resources. The handbook’s contents suggest that the key messages are being aware of and understanding what Medicare is; the various Medicare components (Parts A, B, C, and D); Table 3-3. Recommended Outcome Measures and Potential Data Sources; coverage under the various Medicare components; and resources beneficiaries can use for more information.

Table 3-4 highlights the key topics and the number of pages devoted to them in the 2007 handbook.

Table 3-4. Key Topic Areas Addressed in the 2007 Medicare & You Handbook

Topic Area	Number of Pages in Handbook
Index	6
Medicare Parts A and B and What’s Covered	15
Deciding How to Get Your Medicare Benefits	2
Original Medicare Plan	8
Medicare Advantage Plans (Part C)	10
Medicare Prescription Drug Coverage (Part D)	14
Other Medicare Plans, Government, and Private Insurance	6
Help for People with Limited Income and Resources	8
How to Join and Switch Plans	8
Your Medicare Rights	10
Resources and Contacts for More Information	8
Definitions of Terms	4
2007 Medicare Costs	4
Source: 2007 Medicare & You handbook table of contents.

Design Options

One of the first steps in designing comprehensive evaluation options for the NMEP is to select which activities to focus on (among the various NMEP components) as part of this effort. The options proposed in this evaluation design focus on the following components that represent the current core NMEP activities:

Medicare & You handbook

1-800- MEDICARE helpline

http://www.medicare.gov/

SHIPs

Our rational e for selecting these four components took into account the findings from a systematic review of the literature regarding previous NMEP evaluations, potential reach to beneficiaries, and the amount of money allocated for each component. We chose to focus these evaluation options on NMEP components that represent the majority of the NMEP budgetary costs and are likely to reach the greatest number of Medicare beneficiaries and information intermediaries.

On May 31, 2007, we briefed ASPE staff and leadership on key activities and analysis findings to date and introduced several potential design options. Based on the briefing, a determination was made about which evaluation components should be further developed for the evaluation design report. It was determined that the following four study design options would be developed more fully in this report and are discussed in the following sections:

1. collecting and analyzing data using the MCBS
2. case studies of employers, pharmacists, and SHIPs
3. a long-term cohort study of beneficiaires
4. a cost-effectiveness analysis

4.1 Study Design Option 1: Collecting and Analyzing Data from the MCBS

4.1.1 Design Rationale

The MCBS provides a ready source of data on beneficiary knowledge and need for information from a large, nationally representative sample of Medicare beneficiaries. In addition to any new questions, existing questions concerning beneficiary knowledge and behavior that could be used in the future to evaluate the NMEP have been fielded through the MCBS before the NMEP was first implemented and periodically since then, providing the opportunity to make comparisons over time. Survey participants are included for up to 4 years, potentially providing data necessary for longitudinal analyses (changes in individual knowledge or behavior over time). Data also lend themselves to measuring changes in outcomes for Medicare cohorts and their health care decision makers over time. Questions concerning NMEP-related issues have historically been asked once, annually through the Beneficiary Knowledge and Information Needs Supplement.

Because the MCBS survey has already been designed and is conducted annually, it is likely to be less expensive to use for an evaluation than fielding a new separate survey. Previously fielded MCBS questions could be used to evaluate the NMEP (see research questions below), and new questions could be added to the existing beneficiary knowledge survey or other instruments.

4.1.2 MCBS Design

The MCBS is a national probability sample of approximately 12,000 noninstitutionalized Medicare beneficiaries who are interviewed every 4 months for up to 4 years. Very old and disabled beneficiaries under age 65 are oversampled for some rounds of the survey. Proxy respondents answer questions for survey respondents who are unable to do so for themselves because of physical or cognitive impairment. Each year, approximately one-fourth of the sample is rotated out of the survey and replaced with new members, so that each annual MCBS data set represents a cross-section of the Medicare population enrolled in the program continuously since January 1 of that year, as well as members of a longitudinal beneficiary panel.

Each year, beginning in the winter of 1998, just prior to the first national distribution of the Medicare & You handbook, and continuing annually through the winter of 2006, one of the MCBS survey rounds has included a beneficiary knowledge and information needs supplement that is administered to community-dwelling beneficiaries. Through this supplement, the MCBS tracks national trends in (1)beneficiary knowledge and sources of information about Medicare through the periods before and after the education campaign activities were implemented, and (2)beneficiaries’ use and preferences for a variety of sources of information to stay informed about changes in the Medicare program.

A limitation of using the MCBS data to answer specific evaluation questions is that some questions change over time or are not included in some years. Although the exclusion of some questions allows for the collection of new information, it may limit the ability to track changes over time. A second limitation is that some questions are not asked of the entire sample in every year. Therefore, while weights allow for tracking trends in cohorts over time, longitudinal analysis measuring changes over time in individual beneficiaries is not always possible. A final limitation is that there is often a significant time lag between when questions are asked of beneficiaries and when the survey data are available for analysis. Thus, the MCBS is not particularly useful in identifying potential knowledge and information issues in “real time.”

4.1.3 Research Questions

The MCBS can answer the following research questions, addressing various dimensions of an evaluation of the NMEP: access to NMEP information, understanding of Medicare, understanding of NMEP information, use of the NMEP, and disparities between groups in outcomes of interest.

1. Do beneficiaries have access to information from Medicare when they need to make changes in their benefits or coverage? What source(s) of information do beneficiaries have access to? Is access to information increasing over time?

2. Do beneficiaries understand Medicare’s features and options? Does beneficiary understanding improve with longer tenure in the program? Is beneficiary understanding of the program’s features and options increasing over time? Is use of particular NMEP interventions or combination of interventions related to greater understanding of the Medicare program?

3. Do beneficiaries understand the information they receive, can they use it to make informed decisions if they so desire, and is the level of beneficiary understanding changing over time?

4. Are beneficiaries using NMEP information to make choices and is use of the information increasing over time? Which sources of information are beneficiaries using? Are beneficiaries using one or more NMEP sources of information and has the selection of choices changed over time? Are non-NMEP information sources also important sources of information for beneficiaries? Is use of particular NMEP interventions or combination of interventions related to greater awareness and knowledge of key Medicare program concepts or new prescription drug features?

5. Do research question outcomes differ by whether a beneficiary experienced a change in circumstances necessitating a change in coverage (e.g., new enrollee, retirement, relocation, plan elimination), beneficiary age, gender, race, urbanicity, or use of medical services? Do research question outcomes differ among beneficiaries who make their own decisions, those who receive assistance, and the helpers of beneficiaries who cannot make their own decisions?

Some of these questions have been addressed in earlier evaluations, providing historic answers to the research questions. (Our earlier Task 3 literature review provides a summary of methods and findings from earlier evaluations.) A new evaluation can extend earlier findings, as well as provide answers to new questions. Table 4-1 maps the research questions to potential MCBS survey questions, indicating whether the questions have been asked previously or are new. Because of space limitations, questions that have been fielded previously through the MCBS are paraphrased in the table, followed by their related MCBS survey item number. The exact survey questions are included in Appendix B .

4.1.4 Independent Variables/Factors

The MCBS includes a number of descriptive variables of beneficiaries that can be used to examine differences among groups: age, gender, race, Hispanic ethnicity, educational attainment, marital status, self-reported health status, urbanicity, cable TV and Internet access, type of supplemental coverage (e.g., employer-sponsored, privately purchased supplemental coverage, Medicaid), whether coverage eligibility was due to age or disability, and detailed information on charges and use of medical services.

Table 4-1. Research Questions and Potential MCBS Survey Questions

Research Questions	Previously Fielded MCBS Questions (Paraphrased and MCBS Survey Item Number): Outcome Measures	Topics for New Survey Questions (Paraphrased)
Access to Sources of Information
Do beneficiaries have access to information from Medicare when they need to make changes in their benefits or coverage?	Whether beneficiaries received the Medicare & You handbook (KN27). Whether beneficiaries still have the handbook (KN30).	Whether beneficiaries know they can call the 1-800 Helpline with questions. Whether beneficiaries know they can contact their local SHIP with questions. Whether beneficiaries know they can look on the Medicare Web site.
What source(s) of Medicare information do beneficiaries have access to?	Beneficiary information sources identified (both the NMEP and other) concerning cost of particular medical service (KN3 and 4), prescription drug coverage (PD16), new benefits or changes in Medicare (KN6 and 7), what Medicare covers or does not cover (KN9 and 10), availability and benefits of Medicare HMOs and PPOs (KN12 and 13), Medigap or supplemental insurance (KN15 and 16), Medicaid plan (KN18 and 19).	New questions are not needed.
Is access to information increasing over time?	Questions on information sources (see above) have been asked over multiple years and so this question can be answered by measuring changes in the cohort over time.	New questions are not needed.
Knowledge of the Medicare Program
Do beneficiaries understand Medicare’s features and options?	Prior surveys contain a series of questions with responses of true/false/not sure, concerning such topics as Medicare eligibility (KN37), eligibility for prescription drug coverage (KN42), what Medicare HMO coverage (KN43), etc. (see also KN44-46).	Because the true/false format lends itself to guessing, new questions/responses on particular topics of interest could be reworded to more reliably capture beneficiary knowledge.
Does individual beneficiary understanding improve with longer tenure in the program?	Because individual beneficiaries remain in the MCBS for up to 4 years, changes in individual responses can be tracked over time.	New questions must be added if it is felt that beneficiaries learn through responding to existing questions or if topics of interest are not covered through existing questions.
Is beneficiary understanding of the Medicare program’s features and options increasing over time?	Because questions are repeated over time, the percentage of beneficiaries answering correctly can be tracked.	New questions are not needed to track change.
Is use of particular NMEP interventions or combination of interventions related to greater knowledge of key Medicare program concepts or new prescription drug features?	The current survey has no mechanism to directly measure whether understanding of Medicare is related to use of an NMEP interventions. (Prior analyses have attempted to answer this question by correlating understanding with use.)	To answer this question, new questions need to have two parts: (1) establish level of knowledge and (2)recall of where knowledge was obtained. Questions must be limited to newly obtained knowledge to avoid recall problems.
Understanding of NMEP Materials
Do beneficiaries understand the information they receive?	Prior surveys asked about understanding specific aspects of the handbook (KN29 and 29a).	Prior surveys contain data on where beneficiaries said they found information on a topic and whether their questions were answered. These questions could be modified to ask where beneficiaries sought information on a topic and if they understood the information they received.
Can beneficiaries use the information they receive to make informed decisions?	Prior surveys asked whether beneficiaries read the handbook (BK30), used it to get a telephone number (BK31), or to find out about health plan options (BK32) .	New questions could ask whether beneficiaries understood the information they received from the Helpline, Web site, or SHIPs. Other questions could provide an example of print or Web material and determine if the information was understood.
Is beneficiaries’ level of understanding changing over time?	Questions concerning understanding of the handbook have been repeated over time (KN29b).	Questions or exercises examining print material other than the handbook/Web materials could be repeated over time with new survey cohorts to measure whether understanding is increasing.
Using NMEP Materials
To what extent are beneficiaries using NMEP information to make choices?	Information identified above to indicate access could also be used to determine which NMEP materials are used in relation to such topics as the cost of a particular medical service, prescription drug coverage, new benefits or changes in Medicare, what Medicare covers or does not cover, availability and benefits of Medicare HMOs and PPOs, Medigap or supplemental insurance, Medicaid plan	Among beneficiaries who used more than one source of information, additional questions could probe the role of various sources. For example, beneficiaries may rely on the handbook for more general questions, while the Helpline may be primarily used to address concerns that are relatively unique to the individual.
Are the particular NMEP sources of information changing over time?	Questions are repeated over time and can track cohort changes over time.	New questions should be included for more than 1 year to track change over time.
What are important sources of information used by beneficiaries other than those provided through NMEP?	Existing questions concerning information sources include options other than NMEP information sources.	New questions are not needed to address this issue.
Disparities
Identifying differences among demographic groups would be determined through modeling and not survey questions (see Section 4.1.4).	Analysis would be conducted in relation to outcomes of interest.	Analysis would be conducted in relation to outcomes of interest.

In addition, respondents can be grouped by the degree to which they make their own health care decisions (those who say they make their own decisions, those who say they receive assistance, and those who are so infirm that a proxy is used to respond to the survey).

4.1.5 Data Collection Activities

An important advantage of using the MCBS is that the data collection modality has already been developed and implemented. The survey instrument, sample, and data collection strategy have already been firmly established and are collected annually. However, this could also be a disadvantage of this design, if, because of competing survey interests, approval of new NMEP questions cannot be obtained or if there is a reluctance to change existing questions because a long-term tracking strategy would be affected. For example, although eliminating true/false questions may be optimal, there may be a reluctance to discontinue a question design that has been repeated for a number of years.

4.1.6 Data Analysis Plan

MCBS data would be analyzed using both descriptive and multivariate modeling. Given that outcome variables typically are categorical (no continuous outcomes), multivariate models would be estimated using dichotomous, multinomial, or ordered logistic regression equations. To account for the variance calculation complexities of the multistage MCBS sample design, SAS and SUDAAN software will need to be used. Survey weights will be applied specific to the survey rounds being used.

Cross-sectional differences could be examined in relation to the independent variables/factors listed above. Trends over time will be descriptive and include responses to questions that have been repeated for multiple years of the survey. Longitudinal analyses will focus on questions that have repeatedly been asked of the same respondent and can examine factors related to changes over time.

4.1.7 Potential Challenges and Solutions

Using this data source presents two major challenges. The first is determining whether questions of interest have been asked over time so that trends and longitudinal analyses of interest can be conducted. The second (and most likely greater) challenge is adding questions to the survey. Adding questions requires approval from the survey administrators who are concerned about overall survey burden, among other issues, and sufficient lead time for question development and testing. New questions may also require OMB approval.

4.2 Study Design Option 2: Case Studies of Employers, Pharmacists, and SHIPs

4.2.1 Design Rationale

Although the NMEP’s stated goals do not specify the role of employers, SHIP counselors, or pharmacists as potentially important information intermediaries for beneficiaries, we believe that a comprehensive evaluation of the NMEP should assess the perspectives and potential role of these stakeholders. Employers serve as a trusted source of information for beneficiaries. The proportion of employers offering retiree health benefits for Medicare-eligible retirees has declined over the past decade (McCormack et al., 2002 McCormack et al., 2002; McArdle et al., 2004McArdle et al., 2004), but employer-based retiree health benefits continue to be the leading source of supplemental coverage for Medicare beneficiaries (McCormack et al., 2002McCormack et al., 2002). In a 2004 Kaiser-Hewitt survey of private-sector employers, nearly three-quarters of employers said that they are likely to provide educational materials about the Medicare prescription drug benefit to their retirees (Kaiser-Hewitt, 2004Kaiser-Hewitt, 2004). SHIPs provide personalized counseling and answers to beneficiaries’ questions about Medicare largely through their employees and volunteers who staff toll-free lines and/or meet with beneficiaries in person. Finally, with the implementation of Medicare Part D, pharmacists are a potentially important source of information for beneficiaries regarding Medicare prescription drug coverage.

To assess the perspectives of these stakeholders, we recommend conducting case studies with a small, purposeful sample of employers (n = 9), pharmacists (n = 9), and SHIP agency staff at the state and local levels (n =9). Nine case studies of each category is the maximum allowed without seeking OMB clearance for data collection. We recommend using a case study methodology as opposed to a survey methodology for a number of reasons. First, a major advantage of the case study approach is the ability to meet face to face with the target audience and to hear about and see firsthand their experiences with the Medicare program. Qualitative research methods are particularly well suited to answer more contextual questions because they can be used to discover and explore little-understood phenomena or issues; provide increased context and depth to better understand people’s thoughts and experiences; and can be used to gather information to better interpret how and why events have occurred. Second, we anticipate sampling issues and potentially low response rates for surveys of employers and pharmacists. For example, it is extremely challenging and time consuming to identify and contact the appropriate person within an organization to administer a survey about health benefits. In addition, response rates for employer as well as provider surveys tend to be quite low. Finally, the case study approach is a lower-cost option than three, large-scale quantitative surveys.

In addition to the case studies with employers, pharmacists, and SHIP agency staff, we recommend conducting fewer than nine key informant interviews via telephone with leaders from major coalitions and/or associations representing employers and pharmacists to gain a broader perspective about the issues facing their members. There are several large employer-based health care purchasing coalitions such as the National Business Coalition on Health, National Business Group on Health, Pacific Business Group on Health, Buyers Health Action Group, and the Midwest Business Group on Health. In addition, the Society for Human Resources Management and the National Human Resources Association are professional associations for human resources and employee benefits managers. Similarly, there are several professional associations representing pharmacists, including the American Pharmacists Association, National Association of Chain Drug Stores, National Community Pharmacists Association, and National Pharmaceutical Association. These organizations represent the views of thousands of employers and pharmacists, and it is likely that information gained from interviewing leaders of these groups will complement the case study data collected from individual employers and pharmacists.

4.2.2 Research Questions

Research questions addressed through the case studies and the key informant interviews can provide insight into two main domains:

1. Beneficiary awareness, access to information, understanding, and enrollment experiences from the perspective of employers, pharmacists, and SHIP staff and

2. Employer, pharmacist, and SHIP staff awareness; access to information; satisfaction with information; usefulness of information; understanding; experiences assisting beneficiaries with questions and/or the enrollment process; and disseminating information.

A sample of research questions that could be addressed through the use of case studies with employers, pharmacists, and SHIP counselors and key informant interviews with coalitions and professional associations regarding awareness of the NMEP, access to NMEP information, understanding of Medicare, and use of NMEP materials and products include the following:

1. Do employers/pharmacists/SHIP counselors feel that people with Medicare understand the key messages that CMS is trying to convey? Are beneficiaries having problems accessing and understanding the information resources to help them choose a Medicare plan? Are beneficiaries having problems enrolling in plans?

2. What levels of awareness do employers/pharmacists/SHIP staff have about NMEP messages, products, and/or any publicity or advertising campaigns?

3. How knowledgeable are employers/pharmacists/SHIP counselors about Medicare (e.g., prescription drug coverage, plan types, rules, enrollment processes)? Where do they get their information? How completely and accurately do they believe they understand various topics? How comfortable are they in their role, especially pharmacists?

4. What are the most common questions that beneficiaries pose to employers/pharmacists/SHIP counselors?

5. How satisfied are employers/pharmacists/SHIP counselors with the information Medicare provides? What information sources do they find helpful and how can the current information sources be improved?

6. What is the comparative helpfulness of various Medicare informational materials?

7. What topics/sections of Medicare materials/products are most useful?

8. What materials do employers/pharmacists/SHIP counselors disseminate to beneficiaries? How do they disseminate them?

9. What kind of information and support are employers/SHIP pharmacists/counselors getting from CMS? What kind of information and support (e.g., SHIP staff with sufficient computer literacy to be able to help clients with newer options like Part D) do they need?

4.2.3 Data Collection Activities and Suggested Instruments

We recommend purposeful sampling to select the nine employers, nine pharmacists, and nine SHIP programs for the case studies. Employers should be selected based on their size, kind of business, richness of retiree benefits, and location. Pharmacists should be selected to achieve geographic diversity as well as representation of large chains and small, privately owned pharmacies. We also recommend a mix of male and female pharmacists with varying ages. SHIP programs should be selected to achieve geographic diversity, diversity in the type of Medicare beneficiaries they serve, and whether they are operated by the Area Agency on Aging or the State Department of Insurance. We will also sample SHIP case study participants in different communities to ensure that we achieve diversity in internal and external funding and programs with more and less local infrastructure.

Data collection activities will include gathering descriptive information about the employer (e.g., type of benefits offered), SHIP program (e.g., number of volunteers, funding), pharmacists (e.g., patient population), and coalitions and/or professional associations (e.g., number of members, mission) that will enable adequate description of each organization. In addition to descriptive information collected about the organization, most data will come from personal interviews with individuals at the organization. Smaller employers may have just one or only a few human resources or benefits managers. However, larger employers are likely to have more than one person who administers benefits or makes decisions regarding retiree health insurance. If employers have more than one human resources employee or benefits manager, we recommend having discussions with each manager. Such discussions can be accomplished through one-on-one interviews or in a small group. In addition, if the human resources staff hold workshops or seminars for employees about Medicare or retiree benefits, scheduling the visit so data collection personnel could observe the workshops or seminars would be beneficial. We recommend interviewing several selected SHIP counselors and administrators. Finally, depending on the size of the pharmacy, more than one pharmacist may be employed. For large pharmacies, we recommend interviewing multiple pharmacists to get multiple perspectives. For the telephone interviews with employer-based health care coalitions and professional associations, we recommend interviewing one key individual at each selected organization (nine or fewer organizations total).

We recommend developing a form to systematically collect the descriptive or administrative information from each organization. In addition, we recommend developing a semistructured discussion guide to answer the research questions listed above. In conducting qualitative research, refining research tools is valuable as data are collected or new information is revealed or when particular questions have been thoroughly answered. In addition, we recommend developing a case study debriefing tool. This tool will serve as a template in which data can be entered regarding key issues addressed during the data collection to summarize highlights of the site visit either while on site or shortly after returning from the visit. These forms can be used to provide immediate information about key issues uncovered while on site, including any potential revisions to the semistructured discussion guide. We suggest that the interviews be audiotaped and transcribed.

4.2.4 Data Analysis Plan

We recommend using a qualitative analysis software package (e.g., Atlas.ti) to file and organize the data. These activities include performing data reduction, creating data displays, and formulating and verifying conclusions drawn from the data. A variety of qualitative techniques can be employed to draw conclusions from the data (e.g., noting patterns, themes, and plausibility; noting relations between variables; finding intervening variables) and protect against bias by testing and confirming findings (e.g., ensuring the basic quality of the data, checking findings by examining exceptions, testing explanations); (see Miles and Huberman [1994] Miles and Huberman [1994] for an extensive discussion of these techniques). One type of data reduction process involves developing a coding list for expected findings, then revising the coding scheme based on actual interviews. Ideally, the data will be coded and assessed for inter-rater reliability. Finally, we recommend comparing the data obtained by the three different subgroups to identify key similarities and differences by respondent group. For example, are the types of support needed by employers similar or different from that needed by pharmacists and SHIPs?

4.2.5 Potential Challenges and Solutions

The most likely challenge will be gaining the participation of employers and pharmacists. In our prior experience, human resources staff and benefit managers are extremely busy. We recognize that each group may be very busy depending on their business cycle. Therefore, it will be important to get on their calendars well in advance and to be clear about what the site visit will entail, the time commitment involved, and the rationale for conducting the case studies.

4.3 Study Design Option 3: Prospective Cohort Study

4.3.1 Design Rationale

A prospective cohort study is the most in-depth design option available for evaluating the NMEP, and this design has several advantages over other study options. First, the study will identify how key influences on Medicare decision making change over time. By following beneficiaries through the enrollment and plan decision processes, we can examine not only how factors such as values, NMEP materials, and current health care access influence Medicare plan decisions, but also how reliance on these factors increases or decreases over time. This information is key to understanding how beneficiaries arrive at their plan decisions and to determining the role NMEP components play in the decision process. To date, no evaluation of NMEP has examined beneficiary values or any trends in use of NMEP activities throughout enrollment.

Second, the study will identify whether and how multiple NMEP components work together to impact Medicare plan decision making. Currently, almost all NMEP evaluations have been cross-sectional studies that, at best, can identify associations between NMEP components and increased knowledge or plan satisfaction. Conversely, the prospective cohort design will examine how different NMEP components interact and what role each component plays in a beneficiary’s plan decision. This understanding will be crucial to recognizing when and how NMEP activities actually promote IDM among beneficiaries.

Finally, and most importantly, the design allows us to understand how and under what circumstances beneficiaries make active plan decisions. By following beneficiaries throughout enrollment, we can explore the thought processes used to make plan decisions, identify the most influential factors in those decisions, and determine to what extent IDM takes place. These insights are key to promoting IDM because we gain a better understanding of how NMEP activities can influence such decision-making processes. No evaluation to date has examined the decision-making process within the scope of NMEP activities.

4.3.2 Research Questions

The major advantage of this prospective cohort study is the ability to evaluate how Medicare beneficiaries’ enrollment status, plan satisfaction, and plan decisions change over time. By following beneficiaries from pre-enrollment (age 64) through the processes of initial enrollment, change-of-plan decisions, and, if applicable, disenrollment, we can identify the preferences, information, information sources, and experiences that drive those decisions.

Key research questions for evaluating the NMEP will include the following:

1. How do beneficiary preferences (such as plan cost, physician access and selection, specialist access, and continuity of primary care doctor) change over time? How do these values influence plan decisions?

2. How does participant access to and reliance on NMEP information sources (such as the Medicare & You handbook, www.Medicare.gov, 1-800-MEDICARE, and SHIPs) change over time? To what extent are these sources perceived as important in helping beneficiaries make decisions?

3. How does participant access to outside, non-NMEP information sources (such as employer human resources departments, private health insurance companies, and nonprofit groups like AARP) change over time? To what extent are these sources perceived as important in helping beneficiaries make decisions?

4. What do beneficiaries know about health insurance options and the Medicare program before enrollment? What experience do they have with selecting benefits and options? What experience do they have with the benefits and options offered by Medicare?

5. How does reliance on spouses, adult children, and other proxy decision makers change over time? What is the nature and extent of these individuals’ participation in making plan decisions?

6. How does satisfaction with access to information and the Medicare plan decision-making process change over time? Does satisfaction diminish over time following a plan decision? If so, when and how do beneficiaries act on this diminishing satisfaction?

Ultimately, we can use the study to identify which factors such as specific NMEP resourcesare catalysts for enrollment change and which are catalysts for staying enrolled in one’s current plan. More importantly, the study can help us understand beneficiary decision making about Medicare and how values, NMEP resources, and outside information influence those decisions, including fit between expressed preferences and actual choices.

4.3.3 Theoretical Foundation

Because understanding beneficiaries’ decision-making processes is a primary objective for this study, we propose using a theoretical foundation that helps explain IDM the Elaboration Likelihood Model (ELM) (Petty and Cacioppo, 1986Petty and Cacioppo, 1986). The ELM suggests that individuals can use two different cognitive processes to make decisions: a central processing route and a peripheral processing route. Individuals who use central processing try to evaluate information critically and exhaustively, consider the truthfulness and reliability of different arguments, and make a rational decision based on these considerations. This aligns closely with IDM, which entails a clear understanding of the issue and options available and a decision in line with one’s preferences and values (Briss et al., 2004Briss et al., 2004). Conversely, individuals who use peripheral processing make a decision that is not based on critical evaluation. Instead, individuals who use a peripheral decision-making process rely on peripheral cuessuch as likeability of an information source or perceived valueto guide their decision.

ELM also suggests that two constructs encourage central processing motivation and capacity. Motivation refers to an individual’s incentive to be engaged and involved in an issue (i.e., the perceived importance of selecting a Medicare health plan). Capacity refers to an individual’s ability to understand and assess information related to the issue (i.e., the ability to interpret a Medicare plan comparison chart). When individuals are both engaged in an issue and capable of assessing relevant information, they are more likely to use a central processing route and make an informed decision.

The ELM theory has important implications for understanding how Medicare beneficiaries make decisions about enrollment and health care plans. First, ELM can help identify beneficiaries who participate in IDM. Because individuals who use central processing are most likely to evaluate information critically, these same individuals are likely to make informed decisions that are consistent with their values. Second, ELM can help us identify NMEP components that are associated with and promote IDM. For example, the study may find that individuals who review www.Medicare.gov’s plan comparison information are more likely to be motivated to make, and capable of making, enrollment decisions.

4.3.4 Sample Selection and Recruitment

Several sample selection and study duration options are available for the prospective cohort evaluation. We recommend that potential participants be randomly selected from the Social Security Administration files and the Medicare enrollment database (to ensure some representation from those eligible for Medicare due to disability) and that participants be enrolled in the study at least 1 year prior to Medicare enrollment (approximately age 64). This should ensure geographic and demographic representation across key groups and provide adequate time to collect data in advance of any Medicare enrollment decisions.

Ideally, the entire cohort will be selected at one time rather than recruiting participants on a rolling basis each year. This will shorten the overall time frame for the study and ensure that participants experience similar enrollment environments (e.g., cost of private health insurance at time of initial enrollment). To account for attrition (e.g., loss to follow-up, death), we recommend oversampling the initial cohort to ensure both that adequate data exist to answer the proposed research questions and that all participants are representative of the initial cohort.

To examine beneficiary decision-making processes adequately, we recommend the cohort study last several years, a decade or longer if possible, to ensure that beneficiaries have ample opportunity to make several plan decisions, including disenrollment. The major objectives of the study are to understand how beneficiary values and decision making change over time, and to achieve these objectives, the evaluation must extend long enough for numerous plan decisions to be made. The sample size would need to be driven by power calculations, stratification options, and funding availability.

4.3.5 Data Collection and Suggested Instruments

Data for the prospective cohort study will be collected at regular intervals and will include both quantitative and qualitative approaches. Two instruments could be used for primary data collection a more quantitative self-administered, mail-returned survey and a one-on-one telephone discussion guide to collect more qualitative information. The survey will be the most frequently used instrument and will be administered at predetermined intervals.

Telephone discussions of a small subsample will occur less frequently and could be based on key events (i.e., disenrollment), providing more in-depth information to help interpret beneficiaries’ survey responses. This qualitative data will help elaborate on quantitative responses and will help determine whether beneficiaries participated in IDM. In addition to regular interviews, the study team will also interview beneficiaries when certain event triggers occur (e.g., beneficiary indicates on survey that he/she has changed Medicare plans).

We suggest a proposed data collection schedule that involves administering surveys every 6 or 12 months (depending on resources) and a few dozen telephone discussions each year. Because the study seeks to understand beneficiaries’ future intentions, we propose that individuals who disenroll from a certain plan continue to be surveyed as well. This data collection timeline will allow for adequate measurement of all key variables but should limit the burden on participants and minimize data collection costs, especially for telephone interviewing.

4.3.6 Key Measures and Variables

The cohort study will measure multiple outcomes and variables that depict beneficiary decision-making processes and how these processes change over time. The variables are broken down below into two categories outcome measures and independent measures.

Outcome Measures . For the cohort study, outcome measures include knowledge, satisfaction, final enrollment/disenrollment decisions, and decision-making processes. Because these outcomes are likely to change over time, we intend to measure them at several decision pointsinitial enrollment (baseline), change-of-plan decisions, and disenrollmentas well as at regular follow-up intervals. Table 4-2 describes the outcome measures in more detail.

Independent Measures and Other Variables . The study’s independent variables provide insight into what influences beneficiary knowledge, satisfaction, plan decisions, and decision-making processes. As with the outcome measures, independent variables will be measured at decision points as well as at regular follow-up intervals. Table 4-3 describes the independent variables in more detail.

4.3.7 Potential Challenges

Although the prospective cohort study design is one of the most rigorous and appropriate designs for evaluating NMEP activities, it has several challenges. The five primary obstacles for the study include attrition, limited recall of NMEP activities, diminished recall of the decision process, limited decision points during the study, and difficulty collecting data from special populations. Lack of stability in preferences and reports of preferences may be another challenge, given that people construct preferences as they undertake the decision-making process. A description of each challengeas well as the team’s proposed solutionsis detailed below.

Attrition. During the course of the study, a proportion of beneficiaries will likely be lost to attrition. Some of these may simply be lost to follow-up (e.g., change of address, nonresponse), but others may drop out because of death, institutionalization, or an incapacitating illness. Although the team will make every attempt to locate and collect data from participants, the team also recommends oversampling the initial population so that an adequate number of participants remains in the study even after attrition. Alternatively, if resources do not exist to oversample the initial cohort, the study could recruit additional beneficiaries as participants drop out. These replacements would need to be demographically similar to the dropped beneficiaries. However, this approach has several drawbacks, including lack of initial data for the replacement participants and loss of replacement participants to attrition.

Limited Recall of NMEP. Previous studies have demonstrated that beneficiaries may not accurately recall receiving or accessing NMEP resources, which may make it difficult for the study to distinguish between participants who did not access these resources and participants who simply do not recall accessing these resources. For example, approximately 20% to 30% of beneficiaries do not recall receiving the Medicare & You handbook even though the handbook is regularly mailed to all enrolled individuals (Bann et al., 2004Bann et al., 2004; Brant et al., 2001Brant et al., 2001). Telephone interviewers may help beneficiaries recall exposure to NMEP resources by using probing questions and describing the resource in considerable detail.

Table 4-2. Prospective Cohort Outcome Measures

Measure	Description	Instrument
Initial Enrollment
Enrollment Decision	Choice of plan for initial Medicare enrollment (e.g., FFS, HMO)	Survey
Satisfaction Decision	Satisfaction with initial enrollment decision (i.e., how satisfied is beneficiary with the plan he/she chose?)	Survey Discussion Guide
Satisfaction Process	Satisfaction with initial enrollment process (i.e., how satisfied is beneficiary with available information and plan options?)	Survey Discussion Guide
Demonstrated Knowledge	Actual, rather than perceived, knowledge of Medicare and selected plan option	Survey
Decision-Making Process	Central or peripheral decision process used to arrive at decision (based on motivation, capacity, and rationale)	Survey Discussion Guide
Change-of-Plan Decision
Enrollment Decision	Choice of plan (e.g., FFS, HMO)	Survey
Satisfaction Decision	Satisfaction with plan decision (i.e., how satisfied is beneficiary with the plan he/she chose?)	Survey Discussion Guide
Satisfaction Process	Satisfaction with plan change process (i.e., how satisfied is beneficiary with available information, plan options, and ease of switching plans?)	Survey Discussion Guide
Demonstrated Knowledge	Actual, rather than perceived, knowledge of Medicare and new plan option	Survey
Decision-Making Process	Central or peripheral decision process used to arrive at decision (based on motivation, capacity, and rationale)	Survey Discussion Guide
Disenrollment
Satisfaction Decision	Satisfaction with disenrollment decision (i.e., how satisfied is beneficiary with choice to ultimately disenroll?)	Survey Discussion Guide
Satisfaction Process	Satisfaction with disenrollment decision (i.e., how satisfied is beneficiary with available information and ease of disenrollment?)	Survey Discussion Guide
Enrollment Intentions	Intention to enroll in other health insurance program or intention to re-enroll in Medicare in future (i.e., does beneficiary intend to enroll in a private health insurance plan? Does beneficiary intend to re-enroll in Medicare in the future?)	Survey
Decision-Making Process	Central or peripheral decision process used to arrive at decision (based on motivation, capacity, and rationale)	Survey Discussion Guide
Regular Follow-Up (No Plan Change)
Satisfaction Decision	Current satisfaction with enrollment decision (i.e., how satisfied is beneficiary now with enrollment decision?)	Survey Discussion Guide
Demonstrated Knowledge	Actual, rather than perceived, knowledge of Medicare and selected plan	Survey
Enrollment Intentions	Intention to remain in current plan, switch to a new Medicare plan, or disenroll from Medicare in the future	Survey
Decision-Making Process	Central or peripheral decision process used to arrive at decision to remain in current plan (based on motivation, capacity, and rationale)	Survey Discussion Guide
Decision Point (Initial Enrollment, Change-of-Plan, Disenrollment)
Decision Rationale	Reason for enrolling, changing plans, or disenrolling and reason for selecting plan option	Survey Discussion Guide
Beneficiary Values	Preferences and values considered when making decision (e.g., cost, physician access, quality of care)	Discussion Guide
Preference Priority	Priority and importance of articulated values	Discussion Guide
Decision Discussion	Discussion of decision options with spouse, children, or other proxies	Survey Discussion Guide
Proxy Influence	Perceived influence of spouse, children, or other proxies on final decision	Survey Discussion Guide
Health Services Access	Recent use of health services	Survey
Awareness Primary NMEP Resources	Beneficiary awareness of primary NMEP activities (e.g., Medicare & You handbook, www.Medicare.gov, 1-800-MEDICARE, Consumer Assessment of Healthcare Providers and Services [CAHPS])	Survey
Awareness Other NMEP Resources	Beneficiary awareness of other NMEP activities (i.e., SHIPs, REACH activities, HORIZONS activities, plan comparison database, physician directories)	Survey
Use NMEP Resources	Use and extent of use of NMEP resources	Survey Discussion Guide
Trust NMEP Resources	Perceived accuracy of NMEP resources	Survey Discussion Guide
Awareness Outside Resources	Beneficiary awareness of non-NMEP resources (i.e., employer human resource departments, private health insurance companies, nonprofit organizations)	Survey
Use Outside Resources	Use and extent of use of non-NMEP resources	Survey Discussion Guide
Trust Outside Resources	Perceived accuracy of non-NMEP resources	Survey Discussion Guide
Regular Follow-Up (No Plan Change)
Awareness Primary NMEP Resources	Beneficiary awareness of primary NMEP activities (e.g., Medicare & You handbook, www.Medicare.gov, 1-800-MEDICARE, CAHPS)	Survey
Awareness Other NMEP Resources	Beneficiary awareness of other NMEP activities (e.g., SHIPs, REACH activities, HORIZONS activities, plan comparison database, physician directories)	Survey
Use NMEP Resources	Use and extent of use of NMEP resources	Survey Discussion Guide
Trust NMEP Resources	Perceived accuracy of NMEP resources	Survey Discussion Guide
Awareness Outside Resources	Beneficiary awareness of non-NMEP resources (e.g., employer human resource departments, private health insurance companies, nonprofit organizations)	Survey
Use Outside Resources	Use and extent of use of non-NMEP resources	Survey Discussion Guide
Trust Outside Resources	Perceived accuracy of non-NMEP resources	Survey Discussion Guide

Diminished Recall of the Decision Process. Many beneficiaries may have difficulty recalling their decision-making process to enroll in a Medicare health plan, and the more time that elapses between the decision and data collection, the more difficult recall will be. To address this challenge, the study team has recommended frequent data collection intervals to minimize recall problems. Moreover, the team will use the survey data to flag decision points (e.g., decision to enroll in a new Medicare plan) and will follow up within 1 week to conduct a telephone interview. This approach increases the likelihood that beneficiaries will recall their decision process and rationale and that measurement of these variables will be accurate.

Limited Decision Points. Previous studies suggest that only about 10% to 12% of beneficiaries voluntarily change plans or completely disenroll from Medicare in a given year (Mobley et al., 2005Mobley et al., 2005; Lied et al., 2003Lied et al., 2003). These low change and disenrollment rates mean thataside from initial enrollmentonly a limited number of decision points will occur during the course of the study, limiting the opportunities to probe beneficiaries about their decision to change plans or stop using Medicare. While the study also is designed to explore beneficiaries’ decisions to remain in their current plan, such decisions are likely to be less formal and less explicit than the decisions to switch plans. To address this issue, the study team has recommended a multiyear study duration to maximize the number of disenrollment and change-of-plan decisions. The team has also recommended beginning the study prior to initial Medicare enrollment, which will allow the team to examine beneficiaries’ decision-making processes at that stage.

4.4 Study Design Option 4: Cost-Effectiveness Analysis

4.4.1 Design Rationale

CEA is an economic evaluation tool designed to compare the relative costs and effectiveness of different policy mechanisms in achieving an outcome. When alternative policy instruments exist, CEA can be used to identify which policies achieve the largest outcome improvement at the smallest cost.

CEA may identify which major NMEP activities deliver the best outcomes at the lowest cost. The results may help ASPE ascertain the relative efficiency of different NMEP activities and make recommendations that would improve operating efficiency of the NMEP. CEA has been used successfully in several studies of health communication in activities similar to the NMEP ( Hutchinson and Wheeler, 2006Hutchison and Wheeler, 2006).

CEA requires a uniform outcome measures (the objectives of the policy), and data on the cost per outcome for a given policy are compared across several alternatives. The most “cost-effective” policy is the activity that achieves the lowest cost for a comparable improvement in the outcome. Results are expressed in terms of marginal costs (the cost per unit or service “at the margin,” the last unit served). CEA results are also frequently compared with external benchmarks. This is particularly useful if there are no alternative policies or if only one policy can be measured. However, even multiple comparisons can be made, the most “cost-effective” policy may still be assessed as worthwhile or not against some external benchmark. (For example, in health care, $50,000 per quality-adjusted life-year [QALY] is a typical threshold for identifying cost-effective treatments.)

CEA is related to several other forms of economic policy analyses, including cost-benefit analysis (CBA), cost-utility analysis (CUA), and cost analysis. Cost analysis simply involves measuring the costs of a particular policy and is a fundamental input into CEA, CBA, or CUA. CEA involves tying the costs to an observed change in policy outcomes (the effectiveness), but with no other valuation and rescaling. CBA, often confused with CEA, requires that the outcomes of the policy, or the “benefits,” be expressed in dollar terms. CBA is used to identify policies where total benefits exceed total costs, or where the net benefit increase is the largest. However, CBA is likely unsuitable for the NMEP unless an in-depth valuation study of a specific benefit is conducted first. Note that monetizing benefits may be quite difficult for “nonmarket” goods, such as the dollar value of a level of increase in beneficiary knowledge in the NMEP. Finally, CUA is a special form of CEA, used mostly in health care, where outcomes are translated into “utility” levels instead of dollar values using a variety of instruments (Drummond et al., 1997 Drummond et al., 1997).

4.4.2 Research Questions

The proposed CEA will provide answers to the following research questions:

1. What outcome does each NMEP activity achieve?
   1. How many beneficiaries are served by particular NMEP activities?
   2. How much is beneficiary knowledge increased by NMEP activities?
2. How much does each NMEP activity cost?
   1. Specifically, how much is spent on the five major components: beneficiary materials, the 1-800-MEDICARE, the www.Medicare.gov Web site, community outreach programs, and program support services?
3. Finally, what is the cost per outcome?
   1. Cost-effectiveness , defined by having an outcome measure, where the cost per outcome is compared across each major NMEP activity.
   2. Cost consequence, where process measures such as the number of hits on the Web site and the cost per hit are compared to another different activity.

The last of these questions, cost per outcome, is the heart of a CEA. The first two research questions above are of secondary interest and would be measured as part of collecting the necessary data for conducting a CEA study.

4.4.3 Primary Outcome Measures

The primary outcome for a CEA of the NMEP is an estimate of the cost per unit change in an outcome of interest (e.g., $5,000 per one-unit increase in the level of Medicare beneficiary knowledge for activity A, and $10,000 per one-unit increase in the level of knowledge for activity B). Given that the NMEP consists of five major activities, one could generate the estimated cost per unit change for each activity. The methods below discuss changes in an index of Medicare beneficiary knowledge, but other outcomes may be considered.

Another common measure in CEA is the incremental cost-effectiveness ratio (ICER), the ratio of a change in costs (relative to another policy or to doing nothing) over the change in effectiveness (relative to the same policy or to doing nothing). The ICER measures the gains from switching from one policy regime to another. Sensitivity analyses in all CEA estimates could be generated to explore the impact of assumptions in costs or effectiveness data.

4.4.4 Secondary Outcome Measures

A cha llenge for applying CEA to the NMEP is that different activities operate through different channels and may have different intended purposes. For example, the www.Medicare.gov Web site is not a direct substitute for the community outreach programs in the NMEP. Therefore, as a first step in conducting a CEA of the NMEP, one must identify a primary effectiveness measure that can be compared across activities. Improved beneficiary knowledge is one such measure and may be captured by a previously published index of Medicare beneficiary knowledge derived from questions on the MCBS (Uhrig et al., 2006Uhrig et al., 2006). The extent to which different NMEP activities have unique goals that are not shared by other arms of the NMEP will not be captured in a standard CEA framework but may be accommodated as desired by ASPE using a cost-consequence study.

Data on process outcomes may serve as secondary outcome measures for a cost-consequence study useful for external benchmarking. For each activity of the NMEP, one could gather data not only on costs, but also on the volume of services provided, reported by CMS. This may include the number of calls to the 1-800-MEDICARE helpline, the number of unique hits per day or year to the www.Medicare.gov Web site, and so on. These can be matched with corresponding costs to generate simple cost-consequence metrics.

For example, suppose that the primary outcome establishes that the 1-800-MEDICARE helpline is the most cost-effective NMEP activity. A natural extension of this finding is to know how this compares with other services for older adults. Rather than compare the primary CEA measure (e.g., $1,500 per additional correct item in the MCBS knowledge index), which is specific to the NMEP and has no meaning for other programs, cost-consequence facilitates a more natural comparison. If calls to 1 800-MEDICARE cost $1.00/minute, while calls to Social Security’s 1-800-number cost $0.50/minute, the most “cost-effective” NMEP activity clearly does not fare as well in an external comparison in this example. Thus, the two forms of cost analyses may be used in tandem to maximize the utility of the CEA analysis.

4.4.5 Independent Variables/Factors

All economic evaluations of policies or programs require measures of costs. Each activity under consideration in the CEA must be counted separately. For CEA, typically only the direct costs of the program itself are considered; the costs of individuals, such as travel time or waiting time, are excluded. The latter are necessary if a full societal perspective of resources is desired, as is the case for CBA.

For the NMEP, one could collect costs on each of the five major NMEP activities and to restrict attention to programmatic costs. Costs may be classified as either fixed or variable. Fixed costs do not vary in the short run, usually around 1 year, and include items such as buildings, equipment, Web site development, and some wages and salaries. Variable costs depend on the level of activity of the program and include items such as printed materials, telecommunications, and many labor costs. The two types are summed to estimate total costs, which serve as the metric for a CEA study.

4.4.6 Potential Existing Data Sources

Cost data may potentially be obtained from budgeting or accounting records for the NMEP, currently available for FY98 FY07. Although these off-the-shelf numbers are convenient, they do not necessarily contain the level of precision needed for a CEA study. Accounting costs often measure purchased services and materials costs well but may contain insufficient data on labor and on shared materials.

If accounting costs are unsuitable, a simple alternative is to approximate costs with related data. For instance, if CMS records indicate that 100 full-time workers were assigned to the 1-800-MEDICARE helpline and each was paid $40,000 per year, including the value of benefits, then the total labor cost for this activity would be $4,000,000. Per-capita labor costs may be directly available from personnel records from CMS, contract budgets, or may be borrowed from external data sources such as the Bureau of Labor Statistics (for an average worker in a particular occupation and/or industry).

For effectiveness data, the beneficiary knowledge index discussed above generated using MCBS data is suggested. For process measures, all available data and detail that CMS has on utilization of NMEP activities may be useful. This would include number of beneficiaries served, number of Web site visitors, number of Medicare & You handbooks mailed, and so on. These data may serve as inputs into a cost-consequence study, or they may also be used to scale the effectiveness data.

4.4.7 Data Analysis Plan

The CEA will require the effectiveness to be measured separately for each NMEP activity. Analysis of process outcomes (e.g., www.Medicare.gov Web site hits) is straightforward. Data are gathered and collected in a simple database where they may be analyzed for a cost-consequence study.

Analysis of cost data depends on data availability. Budgeting or accounting records may be organized in a simple database. For other forms of costing, one could use a “bottom-up” method, in which individual pieces of an activity are separately valued and summed to estimate total costs. A model of all major components of an activity must be developed and costs attached to each component. The first challenge is to identify a full list of components that will generate an accurate depiction of costs. For example, suppose that total costs of one NMEP activity equal the sum of labor costs, facilities and capital costs, and materials costs. Within these are multiple classes of workers (hourly, salary, and temporary), a single source of capital, several buildings, and three different types of materials used in the activity. These major headings are further broken down to the extent that components can be identified and separately costed. If worker compensation is not available for a finer level of detail (than hourly, salary, and temporary) or if there is little variation within subclasses, then no further gradations are needed. At each level, then, costs are gathered as availableeither directly from the source (e.g., CMS wage and salary records)or from some comparable external source (e.g., Bureau of Labor Statistics average wage data by occupation and industry).

Once effectiveness (outcome) and cost data are available, the cost-effectiveness calculations are quite simple. For a specific NMEP activity, the cost-effectiveness is the costs of that activity divided by the “effect” of that policy. For example, a 10-unit increase in the beneficiary knowledge index may be obtained for an investment of $1 million in a hypothetical activity. To compare and rank activities, estimates are sometimes transformed into single unit changes; in this case, a one-unit increase for a cost of $100,000. It is important to note, however, that transforming the estimates into such changes is not always meaningful or correct. Not all policies can be reduced in size, nor does the effectiveness necessarily change equally. To address these concerns, the incremental ICER is preferred. Moving from policy A to B, the ICER is defined as the change in the costs [(costs of activity B minus costs of activity A)/(effects of activity B minus effects of activity A)].

4.4.8 Potential Challenges and Solutions

Rigorous implementation of the proposed CEA will need to address several challenges in the estimation of costs, outcomes, and effects. Additional cautions about applying and interpreting the methods should also be considered.

Challenges with Data on Outcomes and Effects

Just as NMEP activity costs change widely from year to year, the scope of NMEP activities also changes. If we are studying multiple years of the NMEP, the CEA estimates for one activity in 1 year may not be directly comparable to another year. To address this, one could collect as much information as possible on the subcomponents of each major activity and then determine which components are constant over time and which are not. Potential problems could be discussed in a limitations section.

The discussion above proposes the use of a Medicare beneficiary knowledge index (Uhrig et al., 2006 Uhrig et al., 2006) based on the MCBS as a common outcome measure for the CEA. The extent to which different NMEP activities are captured in the MCBS will dictate what can be measured and used in a CEA. If only three of five NMEP activities are captured, the other two will not be part of a CEA based on this measure. If the MCBS does not measure all activities, ASPE may wish to consider adding additional questions to the MCBS for long-term CEA studies.

Econometric identification of the effect of NMEP activities based on the MCBS knowledge index is also a concern. Analysis of the NMEP will rely on observational data from the MCBS. Econometric methods will be required to identify the effect of engaging in a particular NMEP activity on the outcome variable and to control for confounders, such as sociodemographics. Although econometric estimation should facilitate an unbiased measure of the relationship, direct causal inference is not possible without panel data. However, the MCBS sample design may permit limited panel data analysis. NMEP activities may also have a lagged effect, where participation (such as reading the Medicare & You handbook) affects the knowledge index in future years, not in the contemporaneous year. This could also be tested using panel data. Identification and lagged effects present challenges for further exploration but should not prevent implementation of the CEA.

Challenges with Cost Data

All economic studies require high-quality data on costs to be meaningful. If the numerator of a CEA is measured with error, policy makers will lack confidence in the results. As a result, a CEA of the NMEP will be guided by the availability of cost data. Accounting data represent a recommended approach, if available. Cost models, or a hybrid of approaches, may be possible if accounting data are unavailable or poorly measured.

Accounting detail must be available at a sufficient level of detail to assign costs to one NMEP activity or another with no overlap or double counting. (For example, N MEP staff may work on non-NMEP activities or may share office space with non-NMEP activities. NMEP costs must be separated out from non-NMEP costs.) Data on the proportion of costs attributable to specific activities may be available for shared resources; if not, it is apportioned based on assumption, which may be examined in sensitivity analyses. Despite some apparent advantages of accounting data, they may be incomplete due to errors in reporting or in misallocation of costs.

Alternatively, a modeling approach may be applied if accounting data are poor. As discussed earlier, one could build a model of total costs by identifying all fixed and variable costs related to NMEP activities and staff. Assumptions about staff, labor costs, materials, material costs, and so on are combined to estimate a total. This total can also be calibrated or adjusted to fit an expected total, such as a grand estimate from accounting data.

A major challenge for the NMEP, regardless of the approach to cost measurement, is that summary fiscal budgets for FY98 FY06 indicate that NMEP activity costs have changed substantially between budget years. Thus, studying a single fiscal year may cause a particular NMEP activity to be an outlier. Studying several years of data may help if effects or outcomes data are also available; averaging costs is another possibility, if the same can be done for effects or outcomes.

Challenges in Application and Interpretation of CEA Results

The CEA study will identify the most cost-effective NMEP activity. The units of this CEA are likely to be NMEP specific and not comparable to external benchmarks. Cost-consequence measures, on the other hand, are less useful internally ( since each activity has distinct mechanisms) but may be more useful externally. Given a particular cost-effective policy, cost-consequence data represent a useful metric for comparing to external benchmarks, such as similar activities (e.g., Web sites) used by other programs.

If the MCBS does not contain sufficient data on NMEP activities to generate meaningful comparisons of the relative cost-effectiveness of different activities, a pure costing study and a cost-consequence study appear feasible and would still permit external comparisons of the NMEP to other programs and policies.

Even with good data, CEA results must be used cautiously when recommending major policy changes, such as phasing out one NMEP activity and redirecting those resources to another. First, redistributing resources may have unintended consequences. For instance, the 1-800-MEDICARE helpline may serve many beneficiaries without Internet access. Even if the helpline costs twice as much for the same change in outcome, eliminating the helpline and emphasizing the Web site may not serve all beneficiaries equally well, regardless of the cost savings. Second, the marginal cost measurement in CEA is not appropriate for “large” policy changes. Costs and outcomes may vary considerably from the environment in which they were measured. For example, capacity constraints on infrastructure may limit the ability for large change or affect costs in unplanned ways.

Conclusions

The goal of this project was to identify opportunities and approaches for future comprehensive evaluations of the NMEP program. Future evaluation options should build on extensive existing research on the NMEP. P revious evaluations have predominantly assessed (a) beneficiaries’ knowledge of Medicare and Part D; (b)beneficiaries’ awareness of the various NMEP components; (c)the appropriateness of information provided through various components to help beneficiaries understand their choices; and (d)beneficiaries’ use of and satisfaction with NMEP components. These evaluations have contributed to improvements in the NMEP programs.

Despite these evaluations, some NMEP activities have not been thoroughly assessed, and several critical questions about the program are still unanswered. To date, few evaluations have examined what knowledge beneficiaries need to make informed enrollment decisions, to what extent NMEP activities promote IDM, and what the return on the considerable investment is for such activities. These are crucial questions because, as the Medicare program continues to expand and change, the NMEP and other activities based on original NMEP models likewise continue to expand. Beneficiaries will become increasingly reliant on educational tools and materials in making enrollment decisions. Evaluating the NMEP in a comprehensive, systematic, and ongoing manner will ensure that the program is as effective and cost-efficient as it needs to be to support beneficiaries.

Thus, additional opportunities for evaluating the NMEP definitely exist. Prior NMEP research suggests that what is most lacking among these previous evaluations is an indication of whether the NMEP has led to change in the beneficiary population, particularly whether beneficiaries (a)are more aware of their Medicare choices; (b)better understand and have improved knowledge of their choices; and (c)make informed health care choices at all and/or over time. Finally, little emphasis has been placed on evaluating the relative impact and cost-effectiveness of various NMEP activities in previous evaluations.

Based on this identification of opportunities for future research, in consultation with ASPE and our project consultant, we identified and further developed four specific options for ongoing NMEP evaluation activities. A summary of the pros and cons of these options are summarized in Table 5-1.

Table 5-1. Pros and Cons of Recommended Design Options

Design Option	Pros	Cons
Longitudinal Analysis of MCBS	Existing, well-designed longitudinal survey representative of beneficiary population Less expensive than fielding a separate survey	Limited or no ability to add questions No control over how new questions are asked or how many times questions are administered over time
Case Studies of Employers, Providers, and SHIPs	Assess the perspectives of stakeholders important information intermediaries for beneficiaries	Does not currently align with the stated goals of the NMEP
Prospective Cohort Study of Beneficiaries	New contribution Ability to measure multiple outcomes longitudinally Questions can be augmented over time to correspond to changes in the goals/direction of the NMEP	Could be relatively costly depending on final design and sample size
Cost-Effectiveness Analysis	New contribution	Ability to obtain necessary cost data Chosen outcome should directly link to the program (such as exposure or program reach). If the outcome is less direct (behavior change), the analysis is complex because of other factors that may have influenced the behavior CEA should follow assessment of program effectiveness and be conducted only if the program is determined to be effective

References

Andreasen, A (1995). Marketing Social Change: Changing Behavior to Promote Health, Social Development, and the Environment . San Francisco: Jossey-Bass.

Bann CM, Berkman N, and Kuo TM (2004) Insurance knowledge and decision-making practices among Medicare beneficiaries and their caregivers. Medical Care , 2(11):1091-9.

Bertrand JT (2006). Introduction to the special issue on cost-effectiveness analysis. Journal of Health Communication, 11:3-6.

Brant T, Joseph C, Carlson K, and Piacitelli A (2001) Final report NMEP assessment: Medicare & You 2001 Handbook . CMS Report . August 2001.

Briss P, Rimer B, Reilley B, Coates RC, Lee NC, Mullen P, et al. (2004). Promoting informed decisions about cancer screening in communities and healthcare systems. American Journal of Preventive Medicine , 26(1):67-80.

Centers for Medicare & Medicaid Services (CMS) (2007) Assessment of the National Medicare Education Program. < http://www.cms.hhs.gov/ConsumerResearch/04_NMEPAssessment.asp#TopOfPage
04_NMEPAssessment.asp#TopOfPage>. As obtained February 5, 2007.

Centers for Medicare & Medicaid Services (CMS) (Undated). Information Dissemination Activities in Implementing the National Medicare & You Education Program (NMEP): Fall 1998-Fall 2001.

Drummond MF, O’Brien B, Stoddart GL, and Torrance GW (1997) Methods for the Economic Evaluation of Health Care Programmes , 2 nd edition. New York: Oxford University Press.

Frick KD (2006). Cost-effectiveness studies of behavior change communication campaigns: assessing the state of the science and how to move the field forward. Journal of Health Communication, 11:163-173.

Guilkey DK, Hutchinson P, and Lance P (2006). Cost-effectiveness analysis for health communication programs. Journal of Health Communication, 11:47-67.

Hutchinson P, and Wheeler J (2006). The cost-effectiveness of health communication programs: what do we know? Journal of Health Communication, 11:7-45.

Kaiser Family Foundation and Hewitt Associates (2004). Current Trends and Future Outlook for Retiree Health Benefits: Findings from the Kaiser/Hewitt 2004 Survey on Retiree Health Benefits.

Kotler P, Roberto N , and Lee N (2002). Social Marketing: Improving the Quality of Life. Second Edition. Thousand Oaks: SAGE Publications.

Lied TR, Sheingold SH, Landon BE, Shaul JA, and Cleary PD (2003). Beneficiary reported experience and voluntary disenrollment in Medicare managed care. Health Care Financing Review, 25(1): 55-66.

McArdle F, Neuman P, Kitchman M, Kirland K, and Yamamoto D (2004). Large firms’ retiree health benefits before medicare reform: 2003 survey results. Health Affairs Web exclusive, January 14, 2004.

McCormack L, Garfinkel SA, et al. (2002). Health insurance knowledge among Medicare beneficiaries. Health Services Research 37(1):41-61.

Miles M, and Huberman A (1994). Qualitative Data Analysis (2nd Ed). Thousand Oaks, CA: SAGE Publications.

Mobley LR, Wang J, McCormack L, Lynch J, Squire C, Kenyon A, and Booske B (2005). The Medicare CAHPS Disenrollment Reasons Survey: A summary of findings across survey years 2000 2003. CMS Report . 500-900-0061-TO#5.

Mullen P, Allen J, Glanz K, Fernandez M, Bowen D Pruitt S, Glenn B, and Pignone M (2006). Measures used in studies of informed decision making about cancer screening: a systematic review. Annals of Behavioral Medicine 32(3):188-201.

Petty RE, and Cacioppo JT (1986). Communication and Persuasion: Central and Peripheral Routes to Attitude Change . New York: Springer-Verlag.

Rimer BK, Briss PA, Zeller PK, Chan EC, and Woolf SH (2004). Informed decision making: what is its role in cancer screening? Cancer, 101:1214-1228.

Uhrig JD, Bann CM, McCormack LA, and Rudolph N (2006). Beneficiary knowledge of original Medicare and Medicare managed care. Medical Care, 44:1020-1029.

Appendix A

Summary of Previous NMEP Evaluations

Table A-1 summarizes previous NMEP evaluations (including basic data design, data source and population studied, sample sizes, and study period) first presenting the approaches used in CMS-sponsored analyses, followed by methods used in other government studies, and, finally, the methods used in studies conducted by nongovernmental organizations.

Table A-1. Previous NMEP Evaluations

Study	Design	Population	Sample Size	Study Time Period
Centers for Medicare & Medicaid Services
Aduss, Grass, and Dahlberg, 2003	Video monitoring service Newspaper clipping service	N/A	N/A	2001; 2002
Anderson, McCormack, Berkman et al., 2000	In-person survey (MCBS 1998)	National probability sample of current Medicare beneficiaries who responded to the 1998 MCBS survey (rounds 22 24)	Varies by question. No weighted range is 648,650 to 30,105,201	1998
Bann et al., 2000	Measurement testing	National probability sample of current Medicare beneficiaries who responded to the 1995, 1996, 1997, or 1998 MCBS survey	Varies by survey year and knowledge question. MCBS samples are usually 14,000+ individuals	1995 1998
Barth, Carlson, and Piacitelli, 2001	Telephone survey (NMEP Community Monitoring Survey) In-depth interviews Media monitoring	Medicare beneficiaries Information intermediaries	Beneficiaries: 2,986	2000; 2001
Bearing Point, Sutton Group, Ketchum Public Relations, 2003	Focus groups Telephone interviews (semi-structured) Participant observation	Pharmacists Disabled Medicare beneficiaries (<65) Caregivers of Medicare beneficiaries Medicare beneficiaries (65+) who knew about prescription discount cards and comparison shop Medicare beneficiaries (65+) who were unaware of prescription discount cards and don’t comparison shop Key informants	Not specified	Not specified
Gerteis and Ward, 2003	Media monitoring Telephone surveys Beneficiary diaries In-depth interviews Focus groups Participant observation Key informant interviews	Sample of Medicare beneficiaries who responded to NMEP Community Survey Sample of hard-to-reach Medicare beneficiaries Sample of key informants and information intermediaries	Community survey: 4,628 to 7,732 Beneficiary diaries: 32 Beneficiary interviews: 145 Beneficiary observation activities: 15 Key informant focus groups: 36 Key informant interviews: 89	1998 2003
Goldstein et al., 2001	Review of existing studies	Medicare beneficiaries Upcoming beneficiaries Relatives/friends Health professionals Existing call and Web data	Varies by study Range is 200 to 16,693	2001
Grad and Hassol, 2001	Telephone surveys Focus groups	Randomly selected beneficiaries Randomly selected M+C involuntarily disenrolled beneficiaries	Beneficiaries: 2,036 M+C beneficiaries: 2,048	2006
Greenwald et al., 2006	Review of existing studies	Current and new Medicare beneficiaries	Varies by study	2001
Harris-Kojetin et al., 2001	Focus groups	Current Medicare beneficiaries in Kansas City, MO, area who participated in a previous survey Dually Medicare/Medicaid eligible individuals and individuals eligible for Medicare because of disability	Beneficiaries: 56
James, Neuman, and Kitchman Strollo, 2006	Focus groups	Representatives from the SHIP agencies in 13 states	Not specified	2006
Levesque and Cummins, 2004	Intervention Mailed surveys	National random sample of upcoming enrollees from CMS’ Enrollment File	Upcoming enrollees: 855 Intervention group: 179 Control group: 676	Not specified
McCormack, Anderson, Daugherty et al., 2001 McCormac k, Anderson, Kuo et al., 2001 McCormack, Anderson, Uhrig et al., 2001 McCormack, Garfinkel, Hibbard, Kilpatrick et al., 2001 McCormack, Garfinkel, Hibbard, Norton et al., 2001 McCormack and Uhrig, 2003	Mailed survey Focus groups	National random sample of Medicare beneficiaries from 1998 enrollment database	Total sample: 3,738 Treatment (both groups): 2,563 Control group: 1,175	2000; 2001
McCormack, Garfinkel, Hibbard, Keller et al., 2002	Intervention Telephone survey	New and experienced Medicare beneficiaries in Kansas City area, aged 65 or older (or nearing 65 th birthday)	Beneficiaries: 2,107	Not specified
CMS, 1999	Focus groups In-person survey (MCBS)	Medicare beneficiaries Organizations that work directly with Medicare beneficiaries	Not specified	1996 1997
CMS, 1999	Focus groups Interviews	Medicare beneficiaries (not specified)	Not specified	1999
Pacific Consulting Group, 2006	Telephone survey	Callers assisted by customer service representative (English-speaking) Callers assisted by customer service representative (Spanish-speaking) Callers assisted by interactive voice system (English-speaking)	Customer service callers (English): 18,008 Customer service callers (Spanish): 1,125 Interactive voice system callers: 4,500
Sing et al., 2001 Sing and Stevens, 2005	Mailed survey	New enrollees those who joined an HMO during sample intake period Switchers enrolled in one Medicare HMO and switched to a different HMO FFS enrollees	Beneficiaries: 3,125	1999 2001
Sofaer et al., 2001	Focus groups	Individuals aged 65 or over who had helped another beneficiary	Not specified	2001
Uhrig et al., 2006	In-person survey (MCBS)	Medicare beneficiaries who were administered Medicare Knowledge supplement during Round 36 of MCBS from May to August 2003	Beneficiaries: 2,497	2003
Government Accountability Office
U.S. Government Accountability Office, May 2006	Audit of CMS publications Mystery shopping calls Web site usability testing	N/A	Publications: 70 Calls: 500	2005 2006
U.S. Government Accountability Office, 2001	Audit of NMEP materials and telephone records Audit of CMS-contracted research reports In-depth interviews	Representatives from beneficiary advocacy groups (AARP, Medicare Rights Center, and Center for Medicare Education) Health Care Plan Associations (American Association of Health Plans and Health Insurance Association of America) Officials in two HCFA regional offices	Not specified	1998; 2000
National Institutes of Health
Winter et al., 2006	Internet survey	Medicare-eligible population (aged 65 or older, or those who will be eligible within 2 years)	Seniors: 4,738	2006
Kaiser Family Foundation
Kaiser Family Foundation, 2006	Structured interviews	Not specified	Beneficiaries: 35	2005; 2006
Kaiser Family Foundation, 2006	Public opinion polls	Seniors 65 or older	Varies by poll	2006
Smith et al., 2005	Focus group	Not specified	Medicaid officials: 12	2005
Kaiser Family Foundation, Harvard School of Public Health, 2005	Public opinion poll	Nationally representative sample of U.S. adults aged 65 or older	Seniors: 802	2005
Commonwealth Fund
Leatherman and McCarthy, 2005	Review of existing studies	Current Medicare beneficiaries Medicare beneficiaries enrolled in specific programs (e.g., HMO, FFS) Hospitalized Medicare beneficiaries Community-dwelling elderly adults	Varies by study	1996 2004
Robert Wood Johnson Foundation
Gold et al., 2001	Telephone survey	Current Medicare beneficiaries selected through survey algorithm	Beneficiaries: 6,620	2000
California Health Care Foundation
California Health Care Foundation, 1999	Telephone survey	Random sample of California seniors (65+) covered under Medicare	Seniors: 755	1999
Cihak, 2006	Telephone survey	Random sample of California adults who recently turned 65 or were about to turn 65	Seniors: 288	2006
Cihak, 2006	Telephone survey Focus groups	Random sample of California adults aged 64 Purposive sample of Medicare intermediaries	Upcoming beneficiaries: 417 Intermediaries: 38	2006
MedPac
MedPAC, 2006	Telephone survey Focus groups Structured interview	Beneficiaries Family members of beneficiaries SHIP counselors and coordinators	Survey: 1,411 Focus groups 72 Interviews: 30	2006
American Association of Retired Persons
Hibbard and Jewett, 1998	Telephone survey	Medicare beneficiaries with Part A and Part B aged 65 to 80 living independently in their community	Beneficiaries: 1,673	1998

Appendix B

Final Design Report B-1

Appendix B: MCBS Survey Questions

MCBS Survey Questions

KN3.	In the past year, have you tried to find out how much [you/(SP)] needed to pay for a particular medical service?
	KPYFIND	YES	1	(KN4)
		NO	2	(KN6)
		NEVER CONTACTED ANYONE FOR ANY INFORMATION ON ANY TOPIC	3	(KN21)
		REFUSED	-7	(KN6)
		DON’T KNOW	-8	(KN6)
KN4.	How did you find out about what [you/(SP)] needed to pay for a particular medical service? [CODE ALL THAT APPLY.] [PRESS CTRL/L TO LEAVE SCREEN.]
	KPYNOFND	R DID NOT FIND INFORMATION  THE INSURANCE COMPANY THAT PROCESSES (SP’S)	96	(KN6)
	KPYINSCO	MEDICARE CLAIMS	1
	KPYMEDGP	MEDIGAP/SUPPLEMENTAL INS. COMPANY	2
	KPYMCOFF	MEDICARE OFFICE INCLUDING THE TELEPHONE HOTLINE	3
	KPYSSOFF	SOCIAL SECURITY OFFICE	4
	KPYAARP	AARP/SENIOR CITIZENS ORGS	5
	KPYFAMLY	FAMILY OR FRIENDS	6
	KPYSRCTR	LOCAL SENIOR CENTER	7
	KPYHOSP	LOCAL HOSPITAL	8
	KPYDOCTR	DOCTOR’S OFFICE   MEDICARE COUNSELING PROGRAM, E.G., STATE HEALTH INS.	9
	KPYCOUN	ASSISTANCE PRGM.	10
	KPYPUBL	MEDICARE PUBLICATIONS	11
	KPYZINE	NEWSPAPERS OR MAGAZINES	13
	KPYOGOVT	OTHER GOVERNMENT AGENCY	14
	KPYEMPLR	EMPLOYER OR FORMER EMPLOYER	15
	KPYHMO	HMO	16
	KPYINET	THE INTERNET	17
	KPYSEMNR	HEALTH FAIR/SEMINAR AT AN HMO OR INSURANCE COMPANY	18
	KPYOSMNR	HEALTH FAIR/SEMINAR OTHER THAN HMO/INSURANCE CO	19
	KPYTV	TV	20
	KPYRADIO	RADIO	21
	KPYOTHER	ANY OTHER PERSON OR PLACE (SPECIFY)___________________	91
	KPYOTHOS	REFUSED	-7
	KPYMCBS	DON’T KNOW	-8
KN6.	In the past year, have you tried to find information [for (SP)] about any new benefits or changes in the Medicare program?
	KWBFIND	YES	1	(KN7)
		NO	2	(KN9)
		NEVER CONTACTED ANYONE FOR ANY INFORMATION ON ANY TOPIC	3	(KN21)
		REFUSED	-7	(KN9)
		DON’T KNOW	-8	(KN9)
KN7.	Where did you find the information about new benefits or changes in the Medicare program? [CODE ALL THAT APPLY.] [PRESS CTRL/L TO LEAVE SCREEN.]
	KWBNOFND	R DID NOT FIND INFORMATION   THE INSURANCE COMPANY THAT PROCESSES (SP’S)	96	(KN9)
	KWBINSCO	MEDICARE CLAIMS	1
	KWBMEDGP	MEDIGAP/SUPPLEMENTAL INS. COMPANY	2
	KWBMCOFF	MEDICARE OFFICE INCLUDING THE TELEPHONE HOTLINE	3
	KWBSSOFF	SOCIAL SECURITY OFFICE	4
	KWBAARP	AARP/SENIOR CITIZENS ORGS.	5
	KWBFAMLY	FAMILY OR FRIENDS	6
	KWBSRCTR	LOCAL SENIOR CENTER	7
	KWBHOSP	LOCAL HOSPITAL	8
	KWBDOCTR	DOCTOR’S OFFICE   MEDICARE COUNSELING PROGRAM, E.G., STATE HEALTH INS.	9
	KWBCOUN	ASSISTANCE PRGM.	10
	KWBPUBL	MEDICARE PUBLICATIONS	11
	KWBZINE	NEWSPAPERS OR MAGAZINES	13
	KWBOGOVT	OTHER GOVERNMENT AGENCY	14
	KWBEMPLR	EMPLOYER OR FORMER EMPLOYER	15
	KWBHMO	HMO	16
	KWBINET	THE INTERNET	17
	KWBSEMNR	HEALTH FAIR/SEMINAR AT AN HMO OR INSURANCE COMPANY	18
	KWBOSMNR	HEALTH FAIR/SEMINAR OTHER THAN HMO/INSURANCE CO.	19
	KWBTV	TV	20
	KWBRADIO	RADIO	21
	KWBOTHER	ANY OTHER PERSON OR PLACE (SPECIFY)___________________	91
	KWBOTHOS	REFUSED	-7
	KWBMCBS	DON’T KNOW	-8
KN9.	In the past year, have you tried to find information about what medical services Medicare covers and does not cover?
	KVRFIND	YES	1	(KN10)
		NO	2	(KN12)
		NEVER CONTACTED ANYONE FOR ANY INFORMATION ON ANY TOPIC	3	(KN21)
		REFUSED	-7	(KN12)
		DON’T KNOW	-8	(KN12)
KN10.	Where did you find the information about what medical services Medicare covers and does not cover? [CODE ALL THAT APPLY.] [PRESS CTRL/L TO LEAVE SCREEN.]
	KVRNOFND	R DID NOT FIND INFORMATION THE INSURANCE COMPANY THAT PROCESSES (SP’S)	96	(KN12)
	KVRINSCO	MEDICARE CLAIMS	1
	KVRMEDGP	MEDIGAP/SUPPLEMENTAL INS. COMPANY	2
	KVRMCOFF	MEDICARE OFFICE INCLUDING THE TELEPHONE HOTLINE	3
	KVRSSOFF	SOCIAL SECURITY OFFICE	4
	KVRAARP	AARP/SENIOR CITIZENS ORGS.	5
	KVRFAMLY	FAMILY OR FRIENDS	6
	KVRSRCTR	LOCAL SENIOR CENTER	7
	KVRHOSP	LOCAL HOSPITAL	8
	KVRDOCTR	DOCTOR’S OFFICE MEDICARE COUNSELING PROGRAM, E.G., STATE HEALTH INS.	9
	KVRCOUN	ASSISTANCE PRGM.	10
	KVRPUBL	MEDICARE PUBLICATIONS	11
	KVRZINE	NEWSPAPERS OR MAGAZINES	13
	KVROGOVT	OTHER GOVERNMENT AGENCY	14
	KVREMPLR	EMPLOYER OR FORMER EMPLOYER	15
	KVRHMO	HMO	16
	KVRINET	THE INTERNET	17
	KVRSEMNR	HEALTH FAIR/SEMINAR AT AN HMO OR INSURANCE COMPANY	18
	KVROSMNR	HEALTH FAIR/SEMINAR OTHER THAN HMO/INSURANCE CO.	19
	KVRTV	TV	20
	KVRRADIO	RADIO	21
	KVROTHER	ANY OTHER PERSON OR PLACE (SPECIFY)___________________	91
	KVROTHOS	REFUSED	-7
	KVRMCBS	DON’T KNOW	-8
KN12.	In the past year, have you tried to find out about the availability and benefits of Medicare managed care plans, such as HMOs?
	KVLFIND	YES	1	(KN13)
		NO	2	BOX KN2
		NEVER CONTACTED ANYONE FOR ANY INFORMATION ON ANY TOPIC	3	(KN21)
		REFUSED	-7	BOX KN2
		DON’T KNOW	-8	BOX KN2
KN13.	How did you find out about the availability and benefits of Medicare managed care plans, such as HMOs? [CODE ALL THAT APPLY.] [PRESS CTRL/L TO LEAVE SCREEN.]
	KVLNOFND	R DID NOT FIND INFORMATION THE INSURANCE COMPANY THAT PROCESSES (SP’S)	96 BOX KN2
	KVLINSCO	MEDICARE CLAIMS	1
	KVLMEDGP	MEDIGAP/SUPPLEMENTAL INS. COMPANY	2
	KVLMCOFF	MEDICARE OFFICE INCLUDING THE TELEPHONE HOTLINE	3
	KVLSSOFF	SOCIAL SECURITY OFFICE	4
	KVLAARP	AARP/SENIOR CITIZENS ORGS.	5
	KVLFAMLY	FAMILY OR FRIENDS	6
	KVLSRCTR	LOCAL SENIOR CENTER	7
	KVLHOSP	LOCAL HOSPITAL	8
	KVLDOCTR	DOCTOR’S OFFICE MEDICARE COUNSELING PROGRAM, E.G., STATE HEALTH INS.	9
	KVLCOUN	ASSISTANCE PRGM.	10
	KVLPUBL	MEDICARE PUBLICATIONS	11
	KVLZINE	NEWSPAPERS OR MAGAZINES	13
	KVLOGOVT	OTHER GOVERNMENT AGENCY	14
	KVLEMPLR	EMPLOYER OR FORMER EMPLOYER	15
	KVLHMO	HMO	16
	KVLINET	THE INTERNET	17
	KVLSEMNR	HEALTH FAIR/SEMINAR AT AN HMO OR INSURANCE COMPANY	18
	KVLOSMNR	HEALTH FAIR/SEMINAR OTHER THAN HMO/INSURANCE CO.	19
	KVLTV	TV	20
	KVLRADIO	RADIO	21
	KVLOTHER	ANY OTHER PERSON OR PLACE (SPECIFY)___________________	91
	KVLOTHOS	REFUSED	-7
	KVLMCBS	DON’T KNOW	-8
KN15.	In the past year, have you tried to find information about what (your/SP’s) Medigap or supplemental insurance policy covers? [PROBE IF NECESSARY: That is, information about what (your/SP’s) (READ PLAN NAMES BELOW) policy covers?]
	KGPFIND	YES	1	(KN16)
		NO	2	BOX KN3
		NEVER CONTACTED ANYONE FOR ANY
		INFORMATION ON ANY TOPIC	3	(KN21)
		REFUSED	-7	BOX KN3
		DON’T KNOW	-8	BOX KN3
KN16.	Where did you find the information about what (your/SP’s) Medigap or supplemental insurance policy covers? [CODE ALL THAT APPLY.] [PRESS CTRL/L TO LEAVE SCREEN.]
	KGPNOFND	R DID NOT FIND INFORMATION THE INSURANCE COMPANY THAT PROCESSES (SP’S)	96	BOX KN3
	KGPINSCO	MEDICARE CLAIMS	1
	KGPMEDGP	MEDIGAP/SUPPLEMENTAL INS. COMPANY	2
	KGPMCOFF	MEDICARE OFFICE INCLUDING THE TELEPHONE HOTLINE	3
	KGPSSOFF	SOCIAL SECURITY OFFICE	4
	KGPAARP	AARP/SENIOR CITIZENS ORGS.	5
	KGPFAMLY	FAMILY OR FRIENDS	6
	KGPSRCTR	LOCAL SENIOR CENTER	7
	KGPHOSP	LOCAL HOSPITAL	8
	KGPDOCTR	DOCTOR’S OFFICE MEDICARE COUNSELING PROGRAM, E.G., STATE HEALTH INS.	9
	KGPCOUN	ASSISTANCE PRGM.	10
	KGPPUBL	MEDICARE PUBLICATIONS	11
	KGPZINE	NEWSPAPERS OR MAGAZINES	13
	KGPOGOVT	OTHER GOVERNMENT AGENCY	14
	KGPEMPLR	EMPLOYER OR FORMER EMPLOYER	15
	KGPHMO	HMO	16
	KGPINET	THE INTERNET	17
	KGPSEMNR	HEALTH FAIR/SEMINAR AT AN HMO OR INSURANCE COMPANY	18
	KGPOSMNR	HEALTH FAIR/SEMINAR OTHER THAN HMO/INSURANCE CO.	19
	KGPTV	TV	20
	KGPRADIO	RADIO	21
	KGPOTHER	ANY OTHER PERSON OR PLACE (SPECIFY)___________________	91
	KGPOTHOS	REFUSED	-7
	KGPMCBS	DON’T KNOW	-8
KN18.	In the past year, have you tried to find information about (your/SP’s) Medicaid plan, such as how it works with Medicare?
	KCDFIND	YES	1	(KN19)
		NO	2	(KN22)
		NEVER CONTACTED ANYONE FOR ANY
		INFORMATION ON ANY TOPIC	3	(KN21)
		REFUSED	-7	(KN22)
		DON’T KNOW	-8	(KN22)
KN19.	Where did you find the information? [CODE ALL THAT APPLY.] [PRESS CTRL/L TO LEAVE SCREEN.]
	KCDNOFND	R DID NOT FIND INFORMATION THE INSURANCE COMPANY THAT PROCESSES (SP’S)	96	(KN22)
	KCDINSCO	MEDICARE CLAIMS	1
	KCDMEDGP	MEDIGAP/SUPPLEMENTAL INS. COMPANY	2
	KCDMCOFF	MEDICARE OFFICE INCLUDING THE TELEPHONE HOTLINE	3
	KCDSSOFF	SOCIAL SECURITY OFFICE	4
	KCDAARP	AARP/SENIOR CITIZENS ORGS.	5
	KCDFAMLY	FAMILY OR FRIENDS	6
	KCDSRCTR	LOCAL SENIOR CENTER	7
	KCDHOSP	LOCAL HOSPITAL	8
	KCDDOCTR	DOCTOR’S OFFICE MEDICARE COUNSELING PROGRAM, E.G., STATE HEALTH INS.	9
	KCDCOUN	ASSISTANCE PRGM.	10
	KCDPUBL	MEDICARE PUBLICATIONS	11
	KCDZINE	NEWSPAPERS OR MAGAZINES	13
	KCDOGOVT	OTHER GOVERNMENT AGENCY	14
	KCDEMPLR	EMPLOYER OR FORMER EMPLOYER	15
	KCDHMO	HMO	16
	KCDINET	THE INTERNET	17
	KCDSEMNR	HEALTH FAIR/SEMINAR AT AN HMO OR INSURANCE COMPANY	18
	KCDOSMNR	HEALTH FAIR/SEMINAR OTHER THAN HMO/INSURANCE CO.	19
	KCDTV	TV	20
	KCDRADIO	RADIO	21
	KCDOTHER	ANY OTHER PERSON OR PLACE (SPECIFY)___________________	91
	KCDOTHOS	REFUSED	-7
	KCDMCBS	DON’T KNOW	-8
KN27.	Did [you/(SP)] receive a copy of this book, called Medicare and You 2004, which gives an overview of the Medicare program? *SHOW CARD KN7*
	KBOKRECD	YES	1	(KN28)
		NO	2	BOX KN4B
		REFUSED	-7	BOX KN4B
		DON’T KNOW	-8	BOX KN4B
KN29.	How easy to understand did you find (this book/the parts you looked at) – would you say (it was/they were) very easy to understand, somewhat easy to understand, somewhat difficult to understand, or very difficult to understand? *SHOW CARD KN1*
	KBOKUNDR	VERY EASY	1
		SOMEWHAT EASY	2
		SOMEWHAT DIFFICULT	3
		VERY DIFFICULT	4
		REFUSED	-7
		DON’T KNOW	-8
KN29b.	In general, how easy did you find the plan information chart to understand – would you say it was very easy to understand, somewhat easy to understand, somewhat difficult to understand, or very difficult to understand? *SHOW CARD KN1*
	KBOKUNST	VERY EASY	1
		SOMEWHAT EASY	2
		SOMEWHAT DIFFICULT	3
		VERY DIFFICULT	4
		REFUSED	-7
		DON’T KNOW	-8
KN30.	[Do you/Do you or (SP)] still have this Medicare and You 2004 book? *SHOW CARD KN7*
	KBOKHAVE	YES	1
		NO	2
		REFUSED	-7
		DON’T KNOW	-8
KN37.	Most people covered by Medicare can select among different kinds of health plan options within Medicare. [PROBE: Do you think this is true or false, or are you not sure?]
	KNPLNCHC	TRUE	1
		FALSE	2
		NOT SURE	3
		REFUSED	-7
KN42.	Medicare managed care plans (HMOs) often cover more health services, like prescribed medicines, than Medicare without a supplemental insurance policy. [PROBE: Do you think this is true or false, or are you not sure?]
	KNHMOCOV	TRUE	1
		FALSE	2
		NOT SURE	3
		REFUSED	-7
KN43.	With a Medicare HMO, people can go to any doctor or hospital in the United States for routine care and the visit will be covered. [PROBE: Do you think this is true or false, or are you not sure?]
	KNANYDOC	TRUE	1
		FALSE	2
		NOT SURE	3
		REFUSED	-7
KN44.	If your Medicare HMO stops serving people with Medicare in your area, you can join another HMO if one is available. [PROBE: Do you think this is true or false, or are you not sure?]
	KNJOIHMO	TRUE	1
		FALSE	2
		NOT SURE	3
		REFUSED	-7
KN45.	If your Medicare HMO leaves the Medicare program and you do not choose another one, you will be covered by the Original Medicare plan. [PROBE: Do you think this is true or false, or are you not sure?]
	KNCOVORG	TRUE	1
		FALSE	2
		NOT SURE	3
		REFUSED	-7
KN46.	A Medicare HMO can raise its fees or change its benefits each year. [PROBE: Do you think this is true or false, or are you not sure?]
	KNRAISE	TRUE	1
		FALSE	2
		NOT SURE	3
		REFUSED	-7
PD16.	Please look at this card and tell me which of these sources, if any, you used to find information regarding (your/SP’s) prescription drug coverage options. [CODE ALL THAT APPLY. PRESS CTRL/L TO LEAVE SCREEN.] *SHOW CARD PD3*
	PDNOTLOK	R DID NOT LOOK FOR INFORMATION	96
	PDINFMED	MEDICARE	1
	PDSOSEC	SOCIAL SECURITY OFFICE	2
	PDSTMAGC	STATE MEDICAID AGENCY	3
	PDOTHRGV	OTHER GOVERNMENT AGENCY	4
	PDINFINS	INSURANCE CO (PDP, MEDICARE ADVANTAGE, MEDIGAP, SUPPLEMENTAL)	5
	PDEMPLYR	CURRENT OR FORMER EMPLOYER OR UNION	6
	PDINFFML	FAMILY, FRIENDS, CO-WORKERS	7
	PDINFPCY	PHARMACY	8
	PDINFPRV	HEALTH CARE PROVIDER (DR, HOSP, LAB)	9
	PDINFMDA	MEDIA (TV, RADIO, NEWSPAPER, MAGAZINE)	10
	PDINFORG	SENIOR COUNSELOR OR ORGANIZATION (AARP,SR CENTER, ETC.)	11
	PDNFOTHR	ANY OTHER PERSON OR PLACE (SPECIFY)___________________	91
	PDNFOS	REFUSED	-7
		DON’T KNOW	-8
BK30.	Would you say you have read this (book/bulletin) thoroughly, that you have read parts of it, or that you haven’t read it at all?
	BOOKREAD	READ IT THOROUGHLY	1
		READ PARTS OF IT	2
		HAVEN’T READ IT AT ALL	3
		REFUSED	-7
		DON’T KNOW	-8
BK31.	Have you ever used this (book/bulletin) to look up a telephone number?
	BOOKTELE	YES	1
		NO	2
		REFUSED	-7
		DON’T KNOW	-8
BK32.	*Display “book” if SP resides in Arizona, Florida, Ohio, Oregon, Washington, or the Kansas City MSA. Residence in Arizona, Florida, Ohio, Oregon, or Washington should be determined by the state code of the SP’s primary address. Residence in the Kansas City MSA should be determined by the ZIP code of the SP’s primary address; ZIP codes included in the MSA are listed in Attachment BK1. Otherwise, display “bulletin.”*
BK32.	Have you ever used this (book/bulletin) to find information about health plan options available to [you / (SP)], such as Medicare managed care plans, HMO or supplemental plans?
	BOOKOPT	YES	1
		NO	2
		REFUSED	-7
		DON’T KNOW	-8