Evaluation of Selected Aspects of the National Medicare Education Program: Final Design Report. 2.2 Results of the Environmental Scan

01/10/2008

A key determinant of a successful campaign is how well it achieved its goals. Therefore, an evaluation of the NMEP should assess how well it is meeting its goals. In turn, the measures used in the evaluation of the NMEP should be linked to these goals and be able to adequately assess whether they are being met. The previous evaluations of the NMEP that we reviewed have assessed each of the three goals of the NMEP to some degree (CMS, undatedCMS, undated). The goals are the following:

  1. Create awareness among the beneficiary community of the new choices that are available.
  2. Develop an understanding of those new choices and their ramifications.
  3. Help beneficiaries use new information to make informed health care choices.

Create awareness among the beneficiary community of the new choices that are available

Develop an understanding of those new choices and their ramifications

Help beneficiaries use new information to make informed health care choices

General

  • Awareness of Medicare information sources (2)

  • Perceived knowledge of Medicare (5)

  • General knowledge of health insurance (3)

  • Demonstrated knowledge of Medicare (16)

  • Perceived knowledge of Part D (1)

  • Demonstrated knowledge of Part D (2)

  • Confidence in ability to select best plan (2)

  • Information availability (5)

  • Information needs (3)

  • Information seeking (7)

  • Information usefulness (2)

  • Information satisfaction (5)

  • CMS as a source of managed care information (1)

  • Trust in information from CMS (2)

  • Use of Medicare information sources (2)

Medicare & You handbook

  • Receipt of handbook (10)

  • Awareness of (1)

  • Recognize source (3)

  • Awareness of Spanish language version (1)

  • Understandability of the handbook (5)

  • Ease of use of handbook (5)

  • Perceived knowledge (1)

  • Demonstrated knowledge (7)

  • Accuracy of content (1)

  • Use of/read handbook (12)

  • Satisfaction/usefulness (8)

  • Perceived usefulness (1)

  • Trust in information (1)

  • Frequency of use (1)

  • Use look up phone number (3)

  • Use find plan information (5)

  • Use find services covered (1)

  • Use cost comparison information (4)

  • Use plan decision (3)

1-800-MEDICARE

  • Awareness of (5)

  • Ease of use (2)

  • Accuracy of information provided (4)

  • Operator knowledge (2)

  • Appropriate referrals from operator (1)

  • Satisfaction (4)

  • Use called helpline (12)

  • Use questions answered (1)

  • Use plan decision (1)

Medicare Web site

  • Awareness of (8)

  • Ease of use (3)

  • Content appropriateness (2)

  • Use of Web site (11)

  • Use of low vision group (1)

  • Use of plan decision (1)

  • Usefulness (12)

  • Interest in/likelihood to use (2)

REACH Activities

  • Awareness of (1)

 
  • Use of plan decision (1)

  • Reach/attendance (2)

State Health Insurance Assistance Programs (SHIPs)

   
  • Awareness of (7)

  • Type of information provided (1)

  • Beneficiary use of (5)

  • Beneficiary likelihood to use (1)

  • Use of plan decision (1)

Train-the-trainer program

   
  • Participant satisfaction (1)

Note: Numbers in parentheses indicate the number of studies in which these measures have been used corresponding to the three goals.

Overall, we found a fairly substantial amount of information that described and/or evaluated the NMEP, presented primarily in technical reports by government contractors with highlights repeated in the peer-reviewed literature and, to a lesser extent, in reports produced by nongovernmental organiza tions and distributed via the Web. These studies employed a variety of research methods, including both qualitative and quantitative techniques. Qualitative methods included focus groups and individual in-depth interviews for cognitive and usability testing of NMEP information products and case studies. Among the studies that used quantitative methods, descriptive statistics were more commonly used than multivariate methods. Across studies, quantitative methods were more common than qualitative. Three studies used randomized study designs. A handful of literature reviews and papers described the NMEP program in terms of its goals and approaches. Overall, a small number of larger-scale evaluations took place over the years. They were conducted particularly early in the NMEP program’s history, included more detailed study designs and larger sample sizes, and used more rigorous statistical methods. Numerous other studies presented polling results and were geographically limited; these findings may not be generalizable to the larger population of beneficiaries, and analyses relied only on descriptive statistics.

View full report

Preview
Download

"nmepreport.pdf" (pdf, 401.55Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®