Performance Improvement 2007

Introduction

What This Report Is About

This Performance Improvement 2007 report provides a brief summary of each of the evaluations completed by the Health and Human Services (HHS) during Fiscal Year 2006 (October 1, 2005 to September 30, 2006). All Performance Improvement reports, the full database of summaries or abstracts of the more than 8,000 current, ongoing, and past studies, and links to the full reports produced by these studies are available at http://aspe.hhs.gov/pic/performance.  

Evaluation is essential to successful operation of Federal programs. HHS administers over 330 programs (see them all in the Catalog of Federal Domestic Assistance, www.cfda.gov). In FY 2006, HHS spent $613 billion on these programs. That amount included over $800 million provided through the Evaluation Set-Aside provision of the Public Health Service Act (Section 241) as well as other program funds directed to evaluation activities. The Set-Aside provision allows the Secretary of HHS to use a portion of the amounts appropriated for programs authorized under the Act for the evaluation (directly, or by grants and contracts) of the implementation and effectiveness of these programs. Additionally, Congress annually appropriates program funds for both targeted and broad evaluation activities.

Under Section 241 of the Public Health Service Act, annually about 2.4% of funds appropriated under the Act are used both to conduct evaluations and to fund other activities identified by Congress. The Public Health Service Act also requires that the Secretary report annually, to the Senate Health, Labor and Pensions Committee and the House Energy and Commerce Committee, the findings of evaluations conducted by HHS. This report, the 13^th in this series, provides summaries of recently completed studies funded with set-aside or other program evaluation funds.

Successful program evaluations increase the likelihood of effective delivery of public services through these programs and ensure that programs are efficient, targeted to their intended clients, and well managed. Important questions addressed by program evaluations include: what are the circumstances in which programs exist, who are the people that need services; and which program activities “work.” The initial needs that give rise to programs, the knowledge base on which they are constructed, and the environment in which they operate, are not static. Actions by various groups and individuals and demographic, economic, political, and social conditions change constantly. For programs to be successful, they must stay abreast of these changes, expect new requirements, and support analytical, management, and policy decision-making. Congress and the President are ultimately responsible for the design, modification and implementation of laws and rules governing programs and both branches of government need the results of ongoing research, program evaluation, and policy studies to effectively monitor and appropriately modify programs when necessary.

In order to make the information that this report contains more fully accessible, and the importance of these studies more readily apparent, this report has two important features. First, each study is identified with a title that seeks to capture the central question that prompted the study so that a reader can quickly identify areas of possible interest. Second, the summary of each study provides a brief action narrative describing the research activity itself. The summaries use a minimum of jargon and acronyms, are written in everyday language, and seek to make this information available to a broad readership of interested individuals. The summaries are brief synopses of what the studies were about and why they were conducted, with examples of significant lessons learned, facts gathered, and recommendations made by the investigators. These summaries are intended to provide accomplished experts, proficient novices and the general public a source of interesting factual information and signposts to valuable research.

Structure of this Report

Evaluations summarized in this year’s report are organized under the Department of Health and Human Services’ Fiscal Year 2004-2009 Strategic Goals and Objectives addressing the following eight broad areas:

• Preventing Disease and Illness
• Protecting Our Homeland
• Closing the Gaps in Health Care
• Improving Health Science
• Realizing the Possibilities of 21st Century Health Care
• Working Toward Independence
• Leaving No Child Behind
• Improving Department Management

Chapter 1 highlights major themes and provides examples of interesting findings emerging from the completed studies. Chapter 2 provides the individual summaries of all the studies. Chapter 3 describes the Department’s evaluation responsibilities and offers a conceptual framework of the role of evaluation. Appendix A contains the full set of the Strategic Goals and Objectives introduced above. Appendix B contains statements of each agency’s mission and evaluation program. Appendix C provides a table of the studies by each agency and the Objective in this report where the study may be found.

Chapter I – Overview of Completed Evaluations—Selected Highlights

Chapter I presents highlights from the findings of the studies included in this Performance Improvement 2007 report. These examples were selected to emphasize the breadth of subject areas addressed by the studies and showcase the diverse methodologies used.

The studies examined important management, programmatic, policy, and factual circumstances faced by programs administered by the Department of Health and Human Services. The studies employed a wide range of methods in seeking accurate information about internal and external agency environments, including literature review, focus groups, surveys, microsimulation analyses, field visits, and case studies. Some of the studies were carried out by agency staff; most were completed with the assistance of contract support expertise; all the completed studies reflect an intense, creative and collaborative effort of technical expertise and programmatic knowledge and experience regarding the importance of effective and efficient programs that achieve targeted objectives and that serve the public purposes for which they were established.

While most studies in this collection focus on process and output activites or program environments, some also aim at program outcomes and cost-effectiveness analysis. Several studies, seeking efficient use of resources, were preliminary feasibility or evaluability studies (examinations of the practical means by which a program might be evaluated) rather than full-fledged (and more expensive) evaluations. These studies typically led to advice and recommendations regarding how best to proceed with more in-depth studies.

Studies examined how to communicate important health and human services information to both the professional and general public. Several surveys provided valuable information about the nature of social behaviors or the numbers of institutional clients that are public beneficiaries in order to provide more solid bases for advice to policy decision-makers. Examinations of agency administrative activities also yielded valuable information and recommendations for improving the management of programs.

The studies in Performance Improvement 2007 support the eight important goals articulated in the Department of Health and Human Services’ Fiscal Year 2004-2009 Strategic Plan. The remainder of Chapter I provides highlights of interesting and significant evaluations and findings from the past year, organized under these eight goals:

Preventing Disease and Illness

Several studies focused on ways to better communicate with consumers. A post-cancer treatment education booklet was shown to be useful for both patients and families. A tribally-based diabetes science education program advanced toward operational status. Research supported important actions to advance public health. A study of the cost-effectiveness of routine childhood vaccination for Hepatitis A in the United States resulted in a decision to extend hepatitis A immunization nationwide.

As a result of a research project to evaluate online information used to communicate human immunodeficiency virus (HIV) prevention messages, researchers recommended drafting professionally designed advertisements for various media settings.

The Office of Public Health and Science examined how to better prevent violence in families between intimate partners. The study identified promising practices including: efforts to recognize patients experiencing family or intimate partner violence and helping women safely and confidentially leave abusive situations. Another study, of couples-focused family planning services, found that serving males and building staff capacity to provide these services requires significant effort and more staff development/training. A study of family planning underscored the importance of the program for low-income populations, the uninsured, and teens.

Survey data, essential to budget and administrative planning, was obtained in several studies of drug taking behavior. During 2005, over 13,000 alcohol and drug abuse treatment facilities were operating. At any one time, these facilities served over a million, mostly outpatient, clients. Half of the clients were treated for both alcohol and drug abuse. Two-thirds of the treatment facilities were private non-profits and a quarter were for-profit private facilities. During the same year, illicit drug and alcohol use and binge drinking among youth had declined from the previous years. However, 25 million adults (11 percent of the population) experienced severe psychological distress.

Building Health Care Capacity

Regarding how to best support health professions education, one study provided advice on how to enhance the current system of graduate medical education financing and recommended criteria to use in evaluating the alternatives.

Long-term care needs are expected to grow dramatically; staffing needs are at the center of concerns about this fact. One study concluded that the number of Americans needing long-term care will double from 13 to 27 million during the next 40 plus years, increasing the need to recruit, train, and retain direct care workers, especially considering that some organizations currently experience 100% turnover annually. By 2020, over 300,000 additional direct care staff will be needed to care for individuals with either cognitive or developmental disabilities. Achieving this 37% increase will be difficult because the projected increase in overall labor supply is only 7%. A related study concluded that as many as 50,000 more social workers will be needed to serve in long-term care settings during the coming four decades.

Another study showed the success of small grants for behavioral cancer control research to increase the number of scientists working in the field.

Closing the Gaps in Health Care

A major success revealed by one study supported by the Centers for Disease Control and Prevention of a United States-Mexico bilateral tuberculosis case management and referral project was the project’s formal integration into the detainee screening process at immigration control and enforcement facilities.

Researchers found that residential care patients differ from individuals living at home in a number of ways. They had a greater likelihood of:

• having incomes below $10,000;
• receiving help with activities of daily living;
• being over age 85, female, and widowed; and
• being far more likely to suffer from Alzheimer’s disease or other dementias.

A study exploring what the United States can do to make sure it has a qualified long-term care workforce observed that criminal background checks are valuable during the hiring process to prevent or reduce abuse of vulnerable adults needing long-term care. Another study concluded that greater efforts are needed to more fully advance the use innovative technologies in residential long term care settings. A third study concluded that agreements between patients and assisted living facilities, in which patients or their representatives directly hire individuals to help them, can allow residents a way to be both more independent and safe.

Improving Health Science

To actively advance essential biomedical research, the National Institutes of Health conducts many intramural (internal) clinical research studies for which it needs thousands of willing patient volunteers annually. A feasibility study provided information with which to design an effective evaluation of how best to solicit these new clinical study participants. Protecting volunteer patients is also critical. A study seeking to identify concerns patients may have about potential conflicts of interest on the part of their health care team members found that patients were not worried about this as long as they knew that an oversight system was in place.

Other feasibility studies in the health sciences area included one that provided information to design an evaluation of how most efficiently to operate 15 cancer incidence and treatment tracking systems. Advice was also obtained about how to better evaluate the functioning of bioengineering research partnership grants in the heart, lung and blood institute. In another critical area, consultant advisors concluded that it was feasible and desirable to conduct an outcome study regarding how productive clinical acquired immune deficiency syndrome (AIDS) research is and how it could be improved.

Essential biomedical research discovery and infrastructure depends on expanding the numbers of individuals doing health sciences research and the number of institutions capable of competing for grants. A graduate partnership program was shown to be effective at recruiting and selecting students and providing them with high-quality research training experiences. One study provided guidance on carrying out an evaluation of a program intended to foster health-related research and enhance competitiveness of investigators at institutions located in states which have historically received fewer research grants.

Two studies by the Food and Drug Administration enabled the agency to refine its review and approval processes. The first concluded that the indifferent results of a pilot program to speed drug approvals did not appear to justify the added costs. The second supported  developing a full evaluation of the agency processes through which it achieves efficient first-cycle review approvals.

Realizing the Possibilities of 21st Century Health Care

Several studies for the National Institutes of Health examined how to make informational Web sites more useful, accessible, and effective. Health science information important to both medical professionals, patients, and other consumers underscores the value of Web-based as well as other methods of information dissemination. Online health information on toxic substances was refined based on a study of site users. The usability of an NIH stem cell information Web site was examined and recommendations for improving the site obtained. Collaboration between the National Institute on Aging, the Census Bureau, and the National Center for Health Statistics was examined and advice provided for strengthening this information dissemination project. A plan for evaluating how to improve educational outreach at over 100 conferences each year was developed.

One study confirmed the effectiveness and appeal of a curriculum to train older adults to access health information on the Web for both seniors and trainers. A study about accessing key medical information from the National Library of Medicine via handheld computers found that the information retrieved was useful in answering clinical questions and that special software designed for the purpose, worked well.

A study about providing information on clinical trials to Spanish-speaking users both confirmed the effectiveness of the Web site and identified areas for further attention. An examination of how educational efforts to prevent noise-induced hearing loss could be strengthened recommended focusing on youth. Studies targeted specific groups such as one for the Food and Drug Administration examining information health professionals need in order to talk clearly with their patients about the risks of heart defibrillators.

The success of efforts by state medical boards to discipline physicians was examined to see whether these efforts contributed to safeguarding medical quality and patient safety. Researchers on the study recommended that demonstrations be conducted to see how well some of the more effective strategies worked elsewhere.

A study to determine what could be learned from employer-provided health insurance data concluded that several research questions could be pursued with the available data, including: the magnitude of costs that are passed along to employees and how such increases affect employee participation in employer-offered health insurance plans. A study to determine how the business case for using health information technology in long-term care settings could be evaluated advised that a small retrospective study with 10-20 nursing homes or home health settings could both make relevant information about this subject available and keep research costs low.

A study was supported by the Office of the Assistant Secretary for Planning and Evaluation to determine how barriers to interoperable health information technology in post-acute and long-term care could be reduced and to determine whether patient data could be made to comply with health information technology standards. The study led to the identification of standards that were subsequently recommended for adoption and use. A second study, funded by the Office of Public Health and Science, that explored the role the federal government should play in developing electronic personal health records resulted in several important findings and recommendations, including that the government could be most effective by avoiding two possible pitfalls: sponsoring a superficial, one-size-fits-all public health record, and allowing many agencies to offer independent public health records with little coordination or strategic vision.

Working Toward Independence

The Department explored circumstances that prevented, and explored ways to support, greater independence on the part of both young and old. A study of what might happen if states allowed welfare recipients to keep more of their child support money found that average annual cash income for the families would increase. Another study, looking at whether the risk of legal responsibility for injury increases when disabled Medicaid clients manage their own personal assistants found that, in general, the risks did not increase.

Findings that challenge assumptions about likely program effectiveness are also important. A study by the Administration for Children and Families that looked at a community college approach designed to improve welfare recipients’ employment prospects found that the program did not produce better economic outcomes. Other individual ACF studies ranged from finding positive program effects, as did a study of a program intended to encourage former welfare recipients to obtain jobs, retain employment and advance into better paying jobs, to finding no program effects.  A study analyzing data from 30 welfare-to-work programs identified common aspects of these demonstrations that positively impacted employment outcomes.

A study of adult day services in the context of health care and long-term care systems found that cost and lack of transportation were the two chief barriers to expansion of such services. An examination of the Web site, “My Family Health Portrait,” that helps people record and display their family health history revealed several features that needed to be modified and provided specific recommendations for doing so.  A study of how Texas modified its Medicaid program to have funds move with nursing home residents who chose to live in the community found that this program could easily be incorporated into Medicaid long-term care programs in other states.

Another study revealed three broad strategies for overcoming barriers that Native American Tribes and organizations face trying to access discretionary grants: obtaining information regarding applying for grants; dealing with the grant review process; and managing grants.

Using micro-simulation analyses and a variety of data sources, the Office of the Assistant Secretary for Planning and Evaluation arrived at summary conclusions regarding the extent to which Americans were dependent on social welfare programs. Overall, 3.4 million fewer Americans were dependent on welfare in 2003 than in 1996. Another study about how states pay for services for children and youth with serious emotional disorders arrived at three major findings: sustained program improvement will depend on effective partnerships among community organizations; demonstration projects will continue to play an important role in improving services; and states should replace residential care with more effective home and community services.

Leaving No Child Behind

A study about the characteristics of children being served by the Head Start program found that most children entered the program with math and literacy levels well below national averages. Children made gains toward national averages during the time they spent in Head Start, but still lagged behind national averages.

A study of child welfare agency efforts to identify, locate and involve nonresident fathers found that nearly half the fathers were never contacted by the agency. The study observed that by not reaching out to fathers, caseworkers may overlook potential social connections and resources that could help to achieve permanency for the children.

Improving Department Management

The National Institutes of Health conducted a mission risk assessment that provided the basis for further systematic investigation and development of ways to assure improved risk protection.

More details on the studies highlighted above, and descriptions of other studies completed during Fiscal Year 2006, are provided in Chapter II. In addition, all the studies and final reports issued for these studies can be found online at http://aspe.hhs.gov/pic/performance.

Chapter II – Summaries of Completed Evaluations – What Was Studied and the Key Findings

Evaluation is an essential means of achieving outstanding program performance. Without questions, there would be no need to study how well programs perform, what they accomplish, and what the circumstances are in which they operate. For each of the studies summarized below, the question that motivated each is identified. The summaries then provide a brief but vivid synopsis of the facts about the study, including, as specified in law, its key findings. These studies emerge as significiant signposts along the path of program review and improvement efforts. Evaluation implies critical judgement and reaching conclusions about merit and value. This year’s collection of completed studies include a wide range of significant findings that are potentially of broad interest.

Each study is listed under the strategic goal and objective it most clearly supports (see Appendix A for all the goals and objectives).  Evaluation priorities respond to and are guided by Congressional oversight, Executive Branch decision-making, program management needs and performance measurement systems such as the Performance Assessment Reporting Tool and the Government Performance and Reports Act. Evaluation activities also respond to changing realities in the programs themselves and in the environments in which the programs operate as well as in response to advice and recommendations from earlier evaluations.

To view the studies conducted or supported by specific agencies, see the list in Appendix C. In addition to the studies in this report, available online are previous studes and preliminary entries for studies that are ongoing or too recently concluded to be included here. These can be found at http://aspe.hhs.gov/pic/performance.

Goal 1: Reduce the major threats to the health and well-being of Americans

Objective 1.1: Reduce behavioral and other factors that contribute to the development of chronic diseases

How Can Managers and Evaluators Better Understand, Design, and Perform Cost Effectiveness Evaluations of Community Public Health Prevention Programs?

Summary:

The purpose of this project was to develop a guide stressing the importance of planning for cost effectiveness studies, early in the life of community based prevention programs, for use by program managers and local evaluators.

The Department of Health and Human Services encourages health promotion and disease prevention in a variety of programs targeted to communities. At the community level, managers need to be able to assess whether health promotion and disease prevention interventions are having the desired effect and are a good use of scarce resources. Cost effectiveness analysis is one tool for answering these questions. In cost effectiveness analyses program costs are quantified in dollars and program outcomes are quantified in non-monetary units such as life-years gained. The guide was to offer checklists for the reader, identifying the steps that needed to be taken. It was also to offer advice on how to develop a cost-effectiveness study, including determining the time frame for the intervention and the time frame for the analysis; use of discounting in considering future costs and benefits; and various types of economic studies and their potential use. It also was to offer guidance on identifying outcome measures, citing programs such as the Centers for Disease Control and Prevention's (CDC's) Steps to a Healthier US, and the National Cancer Institute’s American Stop Smoking Intervention Study as examples. It was also to offer criteria for selecting among candidate outcome measures. In a similar vein, it would offer advice on how to quantify program costs, and isolate costs by program component.

Guidance was developed by the contractor, including specific advice on conducting cost-effectiveness analyses, focusing on issues local evaluators are likely to encounter such as how to compare programs that target different health conditions; issues that arise in comparing programs that target the same condition and equity and efficiency concerns. The contractor offered various allocation scenarios that a decision-maker could face and examples of how cost effectiveness studies could inform decision making. The contractor also prepared an annotated bibliography of cost-effectiveness analyses of community prevention strategies from the formal literature as well as worksheets for quantifying program costs.

Report Title: Guide to Analyzing the Cost Effectiveness of Community Public Health Prevention Approaches http://aspe.hhs.gov/health/reports/06/cphpa/index.htm
Agency Sponsor: ASPE-Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Tilson, Wilma, 202-205-8841
Performer: Research Triangle Institute International; Research Triangle Park, NC
PIC ID: 8483

Which Interventions Best Improve Communication, Contraceptive Use, and HIV and Sexually Transmitted Disease Prevention Among Couples?

Summary:

The study assessed use and effectiveness of "couples-focused" family planning program services supported with federal funds. A couples-focused service provider addresses issues related to a relationship by working either directly with both partners or indirectly by working with only one partner. The study examined several questions. What couples-focused approaches were being used in family planning clinics? How were couples-focused approaches viewed? What challenges were couples-focused efforts by family planning clinics coming up against and how could they be overcome? What ways could family planning clinics and programs involve couples in services? The study included: (1) literature review; (2) interviews with key sources, including Federal program administrators, researchers, and grantees working in various clinic settings; (3) site visits to four grantees; and (4) a day-long meeting with federal, academic, and service delivery participants.

There were three key findings. The definition and implementation of couples-focused services isn't uniform. Serving males in family planning clinic settings and building staff capacity to provide couples-focused services requires more effort. Providers of couples-focused services need more training to enable them to better serve their clients' communication, relationship and negotiation development skills.

Report Title: A Collaborative Evaluation of Strategies to Encourage Couples-focused Health Service Delivery in a Sample of Title X-supported Family Planning Clinics http://aspe.hhs.gov/pic/fullreports/06/8278.pdf
Agency Sponsor: OPHS-OPA, Office of Population Affairs
Federal Contact: Panda, Pankaja, 240-453-2820
Performer: Health Systems Research, Inc.; Washington, DC
PIC ID: 8278

What Messages Are Most Effective in Encouraging HIV-Positive Persons and Others to Act in Ways That Will Prevent Further HIV Transmission?

Summary:

This three-phase multi-year study identified and developed human immuno-virus (HIV) prevention strategies and arguments that would be effective in encouraging prevention behaviors among HIV-positive persons. Findings from each phase of the formative research (concept generation, concept screening, and concept testing) were each used to develop the next phase of the study. This research project developed and evaluated online brochures used to communicate HIV prevention messages. The messages were tested in experimental studies to evaluate their efficacy among men who have sex with men. Three separate studies examined different ways of communicating, framing, and displaying the messages. Outcomes examined included behavioral intentions, attitudes, response efficacy, perceived behavioral control, perceived social norms, perceived risk, message appeal, message clarity, perceived effectiveness of the message, and source credibility.

Researchers recommended that the next step involve working with a creative team to further develop the "raw" message concepts into professionally designed advertisements (print, outdoor, radio, television). The advertisements would then need to be further tested with the target audience before being launched as elements of a campaign. (Additional related reports can be found at PIC 8297, 8297.1, and 8297.2)

Report Title: Developing Message Concepts for HIV-Positive Persons; Phase III Final Report http://aspe.hhs.gov/pic/fullreports/06/8297.doc
Agency Sponsor: CDC-OD-OSI, Office of the Director, Office of Strategy Innovation
Federal Contact: Slaton, Terrie, 404-639-7647
Performer: Research Triangle Institute International; Research Triangle Park, NC
PIC ID: 8297.3

What Is the Diabetes-Based Science Education Program for Tribal Schools Curriculum Project?

Summary:

This study continued the multi-year development and evaluation of a K-12 diabetes education curriculum for use in Tribal schools and schools with a significant percentage of American Indian students. Nearly 15% (100,000) American Indian and Alaska Native adults receiving health care from the Indian Health Service have diabetes. It has become one of the most common and serious illnesses found among this population. This Phase III stage of the study documented the development process of the curriculum through the beta and field testing stages. Next, researchers and schools will conduct implementation testing of the (revised) curriculum during the fall of 2007. National launch of the curriculum is expected for the fall, 2008. Beta and field testing consisted of small curriculum components tested at different tribal sites at different times, to accommodate the fact that not all materials were available at the same time or were all sites ready to proceed on a single schedule.

The testing answered three broad questions. Is the program being developed as planned? Are program critical processes being implemented? Has the curriculum been developed into the expected output that meets its general goals? Data sources for the evaluation included: lesson-specific evaluation forms; web-based test forms generalized across several lessons; discussions at quarterly meetings; external advisory committee reviews; and site visits to classrooms. Based on the testing, researchers recommended adding a nationally focused teacher professional development component, developing a strategy for generalizing cultural components and applying these components from within the community where the curriculum will be taught, and maintaining a simple, clean, concise appearance to the curriculum materials.

Report Title: Phase III Evaluation Report: September 2006 Diabetes-Based Science Education Program for Tribal Schools http://aspe.hhs.gov/pic/fullreports/06/8460.pdf
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Garfield, Sanford, 301-594-8803
Performer: PS International, Doug Coulson; Arnold, MD
PIC ID: 8460

Objective 1.2: Reduce the incidence of sexually transmitted diseases and unintended pregnancies

What Were the Short-term Impacts of Providing a Life Skills Education Component to Middle and High School Youth?

Summary:

This study was part of a larger evaluation of a select group of abstinence education programs authorized and funded under Section 510 of the Maternal and Child Health Services Block Grant program in the Social Security Act.

The study examined the short-term effects on the target student audiences of providing a life skills (e.g.: marriage and the family, coping with employment issues, dealing with adversity) education component to middle and high school youth in Edgefield, South Carolina. All the youth participating in the Life Skills Education Component also participated in other components of a broader training program ("Heritage Keepers®"), specifically, the core abstinence education component. This study examined the impact that the program had on students, over and above the impact of other components of the Heritage Keepers® Program.

Researchers found that the Life Skills Education Component had a positive impact on the whether the students viewed the rest of the program services as helpful. However, the education component did not increase participation by the youth in health, family life, and sex education classes. The program generally did not affect the intermediate outcomes thought to be related to teen sexual activity. The one exception was a positive impact on support for abstinence among middle school youths' five closest friends. This last finding was consistent with the program's goals of influencing youth peer groups. However, there were no impacts on this measure for high school youth or on other measures of peer influence and relations examined.

Report Title: First Year Impacts of the Heritage Keepers Life Skills Education Component http://aspe.hhs.gov/hsp/05/abstinence2/index.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Kelsey, Meredith, 202-690-6652
Performer: Mathematica Policy Research, Inc.; Princeton, NJ
PIC ID: 8481

Objective 1.3: Increase immunization rates among adults and children

What Is the Cost-Effectiveness of Routine Childhood Vaccination for Hepatitis A in the United States?

Summary:

Hepatitis A has been one of the most frequently reported vaccine-preventable diseases in the United States. Each year, from 1980 to 1999, there were on average 26,000 cases of acute hepatitis A reported to public health agencies. This corresponds to a rate of approximately 10 cases per 100,000. However, the true number of hepatitis A cases in the United States is believed to be 3-6 times this number. This study developed a detailed report of the model of vaccine prevention including a description of the model, the assumptions of the model, the results of the analysis, a sensitivity analysis, and conclusions. Two manuscripts resulting from this study have been accepted for publication in the journal "Pediatrics." The data from this project were pivotal in the Advisory Committee on Immunization Practices' (ACIP) decision to extend hepatitis A immunization nationwide. ACIP consists of 15 experts in fields associated with immunization who have been selected by the Secretary of the U. S. Department of Health and Human Services to provide advice and guidance on the most effective means to prevent vaccine-preventable diseases.

Report Title: The Cost-Effectiveness of Routine Childhood Vaccination for Hepatitis A in the United States http://aspe.hhs.gov/pic/fullreports/06/8298.htm
Agency Sponsor: CDC-OD-OSI, Office of the Director, Office of Strategy Innovation
Federal Contact: Slaton, Terrie, 404-639-7647
Performer: Research Triangle Institute; Research Triangle Park, NC
PIC ID: 8298

Objective 1.4: Reduce substance abuse

What Were the Characteristics of Individuals Discharged From Substance Abuse Treatment Facilities During 2004?

Summary:

This study examined data on clients discharged from substance abuse treatment in 2004 in the United States, as reported to the Treatment Episode Data Set. In general, this data set includes information on clients treated at facilities that receive State alcohol and/or drug agency funds (including Block Grant funds) for the provision of substance abuse treatment services. In 2004, twenty-eight States provided data on 1,105,644 discharges. The data include demographic and substance abuse characteristics of discharged clients, reasons for discharge (including treatment completion), and length of stay in treatment.

The study found that treatment was completed by 40 percent of the discharged clients in 2004. Twenty-two percent dropped out of treatment, 12 percent were transferred to further treatment, 8 percent had their treatment ended by the facility, 1 percent had treatment ended because of incarceration, less than 1 percent died, 8 percent failed to complete treatment for other reasons, and the reason for discharge was unknown for the remaining 9 percent of discharged clients. Except for discharges from methadone treatment, the median length of stay in treatment was greatest for discharges from outpatient treatment (69 days), followed by long-term residential treatment (46 days), and intensive outpatient treatment (42 days). The median length of stay in treatment for discharges from short-term residential treatment was 20 days. Discharges from detoxification had the shortest median length of stay (3 days).

Report Title: Treatment Episode Data Set (TEDS) 2004: Discharges from Substance Abuse Treatment Services http://wwwdasis.samhsa.gov/teds04/tedsd2k4web.pdf
Agency Sponsor: SAMHSA, Substance Abuse and Mental Health Services Administration
Federal Contact: Trunzo, Deborah, 240-276-1267
Performer: Synectics for Management Decisions, Inc.; Arlington, VA
PIC ID: 8301

What Is the Extent of Non-Medical Use of Prescriptions Drugs in the United States?

Summary:

An analysis was carried out on combined data from the 2002, 2003, and 2004 National Surveys on Drug Use and Health. The analysis examined the non-medical use of prescription psychotherapeutic drugs (pain relievers, tranquilizers, stimulants, and sedatives). Non-medical use is defined as the use of medications without a prescription or merely for the experience or feeling the drug causes, rather than for its intended medical purpose. The annual surveys, including a combined total of over 200,000 respondents, collected information about drug use and health among representative samples of the civilian, non-institutionalized population of the United States aged 12 and older.

Perhaps the most important finding of this study is that the non-medical use of prescription psychotherapeutic drugs—and of pain relievers in particular—is now second only to marijuana use among the Nation's most prevalent drugs. In addition, the annual average number of persons who started using pain relievers non-medically in the past year exceeded the number for marijuana (2.4 million vs. 2.1 million). An estimated 2 million persons were determined to be dependent on or to abuse prescription psychotherapeutics in the past year. Among those who abused or were dependent on prescription psychotherapeutics, only an estimated 290,000 persons received treatment for illicit drug use during the past year.

Report Title: Misuse of Prescription Drugs:Data from the 2002, 2003, and 2004 National Surveys on Drug Use and Health http://oas.samhsa.gov/prescription/toc.htm
Agency Sponsor: SAMHSA, Substance Abuse and Mental Health Services Administration
Federal Contact: Colliver, James, 240-276-1252
Performer: RTI International; Research Triangle Park, NC
PIC ID: 8302

What Were the Characteristics of Substance Abuse Treatment Facilities in the United States in 2005?

Summary:

This survey collected information on the location of alcohol and drug abuse treatment facilities, details about the services offered, and number of clients they treated. The survey is conducted annually of all identified alcohol and drug abuse treatment facilities in the 50 States, the District of Columbia and other U.S. jurisdictions. The number of facilities remained relatively constant between 2000 and 2005. There were 13,428 reporting facilities in 2000 and 13,371 facilities in 2005. The number of clients in treatment on the survey reference date (the date the survey was conducted or from which the data was taken) increased by 8 percent over the same period, to 1,081,049 in 2005. Approximately 8 percent of persons in treatment on the survey reference date were under the age 18. Eighty-nine percent of all clients in treatment were in outpatient treatment, 10 percent were in non-hospital residential treatment, and 1 percent was in hospital inpatient treatment. Nearly half (47 percent) of all clients were in treatment for both alcohol and drug abuse. One-third (34 percent) of clients were in treatment for drug abuse only, and 19 percent were in treatment for abuse of alcohol alone. The majority of facilities in the treatment system (59 percent of all facilities in 2005) were operated by private non-profit organizations. For-profit private facilities made up 27 percent of all facilities.

Report Title: National Survey of Substance Abuse Treatment Services: 2005 -- Data on Substance Abuse Treatment Facilities http://oas.samhsa.gov/DASIS/2k5nssats.cfm
Agency Sponsor: SAMHSA, Substance Abuse and Mental Health Services Administration
Federal Contact: Trunzo, Deborah, 240-276-1267
Performer: Synectics for Management Decisions, Inc.; Arlington, VA
PIC ID: 8303

What Were the Trends in Admissions to Substance Abuse Treatment Facilities in the United States from 1994 to 2004?

Summary:

This study examined data from clients admitted to substance abuse treatment between 1994 and 2004, as reported to the Treatment Episode Data Set. The data include demographic and substance abuse characteristics of the 1.9 million annual admissions to treatment for abuse of alcohol and drugs. This system is based on treatment admissions. An individual who is admitted multiple times in a year would be counted once for each admission.

Five substances accounted for 95 percent of the 1,875,026 treatment admissions in 2004. Alcohol was the primary substance of abuse for 40 percent of admissions. Opiates (primarily heroin) accounted for 18 percent of admissions. Marijuana (16 percent) and cocaine (14 percent) were the next most frequent substances of abuse. Stimulants (primarily methamphetamine) made up another 8 percent of admissions. The alcohol admission rate declined by 28 percent between 1994 and 2004, from 430 to 308 per 100,000 population aged 12 and older. The methamphetamine/amphetamine admission rate increased 182 percent, from 22 to 62 per 100,000 population aged 12 and older. Adolescent (age 12-17) treatment admissions increased 43 percent. The increase in adolescent admissions was largely due to an increase in the number of adolescent admissions for marijuana use (115 percent increase). The proportion of adolescent admissions for marijuana increased from 43 percent of all adolescent admissions in 1994 to 64 percent in 2004.

Report Title: Treatment Episode Data Set (TEDS) 1994-2004 - National Admissions to Substance Abuse Treatment Services http://wwwdasis.samhsa.gov/teds04/TEDSAd2k4TOC.htm
Agency Sponsor: SAMHSA, Substance Abuse and Mental Health Services Administration
Federal Contact: Trunzo, Deborah, 240-276-1267
Performer: Synectics for Management Decisions, Inc.; Arlington, VA
PIC ID: 8304

What Was the Prevalence of Substance Abuse and Mental Health Problems in the United States During 2005?

Summary:

This survey sought information on: use of illicit drugs; use of alcohol; use of tobacco products; initiation of substance use; prevention-related issues; and substance dependence, abuse, and treatment. Measures related to mental health problems also included data on: depression, severe psychological distress, and the co-occurrence of substance use and mental health problems. This is an annual survey of the civilian, non-institutionalized population of the United States aged 12 years old or older. In 2005, approximately 67,500 individuals responded to the survey. The survey collects data in order to produce national prevalence and incidence estimates.

Analysis of 2005 data found that illicit drug use in the past month among youth ages 12-17 continued to decline when compared to survey results from previous years. The rate has declined from 11.6 percent using drugs in the past month in 2002 to 9.9 percent in 2005. Similarly, the rate of past month marijuana use among youth ages 12 to 17 declined from 8.2 percent in 2002 to 6.8 percent in 2005. Drinking among teens also declined, with 16.5 percent of youth ages 12-17 reporting past month alcohol use in 2005, down from 17.6 percent of this age group reporting drinking in 2004. In 2005, 9.9 percent of youth reported past month binge drinking (5 or more drinks at one sitting). This was a decrease from 2004 when 11.1 percent of youth reported this behavior. In 2005 an estimated 24.6 million adults aged 18 or older (11.3 percent of all adults in the nation), met criteria for severe psychological distress in the past year.

Report Title: Results from the 2005 National Survey on Drug Use and Health: National Findings http://oas.samhsa.gov/NSDUH/2k5NSDUH/2k5results.htm
Agency Sponsor: SAMHSA, Substance Abuse and Mental Health Services Administration
Federal Contact: Hughes, Art, 240-276-1261
Performer: RTI International; Research Triangle Park, NC
PIC ID: 8305

What Are the Local Area Prevalence Rates of Substance Use, Abuse and Dependence and Mental Health Problems in the United States?

Summary:

An analysis was carried out on combined 2002, 2003, and 2004 survey data. The annual survey collects information about drug use, abuse and dependence, and mental health based on samples representative at the national, state and local levels of the civilian, non-institutionalized population of the United States aged 12 and older. This multi-year analysis estimates the prevalence of substance use, abuse and dependence, and mental health problems in 357 substate areas. These areas were defined by officials from each State and were typically based on the substance abuse treatment planning areas specified by the States in their applications for the Substance Abuse Prevention and Treatment Block Grant. This analysis includes local level estimates for 22 measures of substance use or mental health problems based on more than 200,000 respondents.

During 2002-2004, 8.1 percent of persons in the United States aged 12 or older used illicit drugs in the month before the survey. Of the 15 areas with the lowest rates of illicit drug use in the previous month, 6 were from four Midwestern States: Iowa, Kansas, Nebraska, and South Dakota. Moreover, South Dakota had three eastern regions among those with the lowest rates of illicit drug use. The highest rate (13.6 percent) of illicit drug use among persons 12 and older was reported by northern California (the 14 northernmost counties). The 15 areas with the highest rates of illicit drug in the past month were dispersed among 10 States, with Alaska accounting for 3 areas and Michigan accounting for 3 areas. Of the top 15 areas, 11 were metropolitan areas or parts of a metropolitan area. Utah County, Utah had the lowest rate of past month alcohol use (19.8 percent), and Ward 3 in the District of Columbia had the highest rate (74.7 percent). Tobacco is the second most commonly used substance in the United States after alcohol. The 15 substate areas with the lowest rates of past month tobacco use were in California and Utah, and Maryland (Montgomery County). The two highest areas were in Kentucky.

Report Title: Substate Estimates from the 2002-2004 National Surveys on Drug Use and Health http://oas.samhsa.gov/substate2k6/toc.cfm
Agency Sponsor: SAMHSA, Substance Abuse and Mental Health Services Administration
Federal Contact: Wright, Douglas, 240-276-1259
Performer: RTI International; Research Triangle Park, NC
PIC ID: 8306

How Many Drug-Related Emergency Department Visits Were There in the United States During 2004?

Summary:

Data on drug-related emergency department visits are collected on an ongoing basis as part of the Drug Abuse Warning Network from a national sample of general, non-Federal hospitals operating 24-hour emergency departments. National estimates for 2004 are based on data submitted by 417 hospitals. In participating hospitals, medical records are reviewed retrospectively to find emergency department visits that were related to recent drug use. All emergency department visits where a drug was implicated are included, regardless of the reason the patient took the drug. Emergency department visits related to drug abuse/misuse, as well as adverse reactions to drugs taken as prescribed/directed, drug-related suicide attempts, accidental ingestions, underage drinking, and malicious poisonings are captured in this data system. All types of drugs (illegal, prescription, over-the-counter, etc.) are included.

The study found that there were an estimated 106 million emergency department visits in the United States during 2004. Of these, almost 2 million were drug-related and nearly 1.3 million emergency department visits were associated with drug misuse or abuse. Of those involving drug misuse or abuse, 30 percent involved illicit drugs only, 25 percent involved pharmaceuticals (prescription or over-the-counter medications) only, 15 percent involved illicit drugs and alcohol, 8 percent involved illicit and pharmaceuticals, and 14 percent involved illicit drugs and/or alcohol and pharmaceuticals.

Report Title: Drug Abuse Warning Network, 2004: National Estimates of Drug-Related Emergency Department Visits http://dawninfo.samhsa.gov/files/DAWN2k4ED.htm
Agency Sponsor: SAMHSA, Substance Abuse and Mental Health Services Administration
Federal Contact: Ball, Judy, 240-276-1250
Performer: RTI International; Research Triangle Park, NC
PIC ID: 8307

What Were the Characteristics of Discharges from Substance Abuse Treatment Facilities During 2003?

Summary:

This study examined data on clients discharged from substance abuse treatment in 2003 in the United States, as reported to the Treatment Episode Data Set. In 2003, twenty-six States provided data on 937,499 discharges. The data include information on the demographic and substance abuse characteristics of discharged clients, reasons for discharge (including treatment completion), and length of stay in treatment. The study found that treatment was completed by 41 percent of the clients discharged in 2003. Twenty-six percent dropped out of treatment, 11 percent were transferred to further treatment, 8 percent had their treatment ended by the facility, 1 percent had treatment ended because of incarceration, less than 1 percent died, 6 percent failed to complete treatment for other reasons, and the reason for discharge was unknown for the remaining 8 percent of discharged clients. Except for discharges from methadone treatment, the median length of stay in treatment was greatest for discharges from outpatient treatment (64 days), followed by long-term residential treatment (42 days), and intensive outpatient treatment (41 days). The median length of stay in treatment for discharges from short-term residential treatment was 20 days. Discharges from detoxification had the shortest median length of stay (3 days).

Report Title: Treatment Episode Data Set (TEDS) 2003: Discharges from Substance Abuse Treatment Services http://wwwdasis.samhsa.gov/teds03/tedsdischweb2k3.pdf
Agency Sponsor: SAMHSA, Substance Abuse and Mental Health Services Administration
Federal Contact: Trunzo, Deborah, 240-276-1267
Performer: Synectics for Management Decisions, Inc.; Arlington, VA
PIC ID: 8308

What Were the Characteristics of Individuals Admitted to Substance Abuse Treatment Facilities in 2004?

Summary:

This study presents summary data on clients admitted to substance abuse treatment facilities in 2004 in the United States, as reported to the Treatment Episode Data Set. In general, these data include information on clients treated at facilities that receive State alcohol and/or drug agency funds (including Block Grant funds) for the provision of substance abuse treatment services. Information on the demographic and substance abuse characteristics is included on the 1.9 million annual admissions to treatment for abuse of alcohol and drugs. This system is based on treatment admissions only. An individual who is admitted multiple times in a year would be counted once for each admission.

Five substances accounted for 95 percent of the 1,875,026 treatment admissions in 2004. Alcohol was the primary substance of abuse for 40 percent of admissions. Opiates (primarily heroin) accounted for 18 percent of admissions. Marijuana (16 percent) and cocaine (14 percent) were the next most frequent substances of abuse. Stimulants (primarily methamphetamine) made up another 8 percent of admissions. In 2004, admissions to substance abuse treatment involving methamphetamine and narcotic pain medications continued to rise compared with previous years. The number of methamphetamine admissions rose 25 percent between 2002 and 2004 to 129,179. For opiates other than heroin (largely prescription pain medications), there was a 42 percent increase in the number of admissions to treatment between 2002 and 2004 to 63,243. Between 1994 and 2004, the proportion of admissions for primary alcohol and cocaine abuse declined while the proportion of admissions for marijuana, stimulants, and opiates increased.

Report Title: Treatment Episode Data Set (TEDS) Highlights - 2004 http://oas.samhsa.gov/TEDS/2k4TEDS/TEDShi2k4toc.htm
Agency Sponsor: SAMHSA, Substance Abuse and Mental Health Services Administration
Federal Contact: Trunzo, Deborah, 240-276-1267
Performer: Synectics for Management Decisions, Inc.; Arlington, VA
PIC ID: 8309

What Are the Prevalence Rates of Substance Use, Abuse and Dependence, and Mental Health Problems by State?

Summary:

An analysis was carried out on combined 2003 and 2004 survey data. The annual survey collects information about substance use, abuse and dependence and mental health from samples representative at national, state and local levels of the civilian, non-institutionalized population of the United States aged 12 and older. This analysis estimates the prevalence of substance use and mental health problems in the 50 States and District of Columbia. Twenty-two measures of substance use, abuse and dependence, and mental health problems were estimated for each State based on approximately 135,500 respondents. Estimates for four age groups (12 to 17, 18 to 25, 26 or older, and all persons 12 or older) were produced for each measure.

Use, in the month before the survey, of any illicit drug in 2003 and 2004 ranged from a low of 5.8 percent in Mississippi to a high of 11.8 percent in Alaska for persons aged 12 or older. Among youths aged 12 to 17, the percentage of past month drug use decreased from 11.4 percent in 2002-2003 to 10.9 percent in 2003-2004. While the national rate did not change, four States showed decreases from 2002-2003 to 2003-2004 in the percentage of persons 12 or older that used an illicit drug in the past month: the District of Columbia, Florida, Nevada, and Washington. Six States showed decreases in past month illicit drug use among youths 12 to 17: Illinois, Nebraska, North Carolina, South Dakota, Vermont, and Virginia. There were no increases among States in any of the age groups. In 2003-2004, the rate of previous month alcohol use in States among persons aged 12 or older ranged from a low of 29.3 percent in Utah to a high of 62.1 percent in Wisconsin. Although national alcohol use rates remained unchanged, four States experienced changes in alcohol use among persons 12 or older. Missouri, Nevada, and New Jersey had decreases. Wisconsin had the only increase in alcohol use for those 12 or older. In 2003-2004, West Virginia had the highest rate of serious psychological distress in the last year (12.7 percent), while Hawaii had the lowest (7.1 percent). Ten states showed increases between 2002-2003 and 2003-2004.

Report Title: State Estimates of Substance Use from the 2003-2004 National Surveys on Drug Use and Health http://oas.samhsa.gov/2k4State/toc.htm
Agency Sponsor: SAMHSA, Substance Abuse and Mental Health Services Administration
Federal Contact: Wright, Douglas, 240-276-1259
Performer: RTI International; Research Triangle Park, NC
PIC ID: 8310

Objective 1.6: Reduce the incidence and consequences of injuries and violence

How Can Public Health Clinics Better Prevent Violence in Families and Between Intimate Partners?

Summary:

This study identified promising practices in family and intimate partner violence prevention. These practices can be used by reproductive health clinics funded under Title X (“Population Research and Voluntary Family Planning Programs”) of the Public Health Act. The clinics are important access points because they typically serve young, low-income women. This population is at-risk for violence. The clinics may be the only contact that the women have with a health care provider. The clinics often have resource constraints. To be promising, a practice must be effective, possible and financially sustainable.

To identify promising practices, evaluators explored family and intimate partner violence prevention programs at nine clinics. Clinic directors, nurse practitioners, physicians, social workers, and health educators were interviewed. They were asked about policies, training materials and resources, and relationships with other organizations. They were also asked about barriers to and ways to facilitate violence prevention activities. The research team toured the clinics and reviewed materials and resources used by them. The team conducted a literature review of the topic under study and interviewed state and federal staff overseeing the clinics.

The study identified promising practices in four areas: training, screening and intake, clinical evaluation, and follow-up and referral. Examples of these promising practices identified included: role playing as part of clinician prevention training, recognizing patients experiencing family or intimate partner violence, methods to help women safely and confidentially leave abusive situations, and ways to identify and work with community organizations to extend clinic services.

Report Title: A Collaborative Evaluation of Family and Intimate Partner Violence Prevention Activities in Title X Clinics http://aspe.hhs.gov/pic/fullreports/06/8277.doc
Agency Sponsor: OPHS-OPA, Office of Population Affairs
Federal Contact: Panda, Pankaja, 240-453-2820
Performer: Battelle; Arlington, VA
PIC ID: 8277

Goal 2: Enhance the ability of the Nation's health care system to effectively respond to bioterrorism and other public health challenges

Objective 2.1: Build the capacity of the health care system to respond to public health threats, especially bioterrorism threats, in a more timely and effective manner

How Is Influenza Vaccine Produced and Disseminated in the United States?

Summary:

This project clarified our understanding of the system by which the United States produces and disseminates influenza vaccine. The system is marked by a complex set of relationships between the vaccine makers, sellers, buyers, healthcare providers and consumers as well as federal and state governments. The contractor prepared five issue briefs regarding these relationships. The intended audiences for the briefs included policymakers and others with an interest in, but limited knowledge of, these topics. For the initial issue brief, "Influenza Vaccine: Summary and Assessment," the contractor reviewed the subject area broadly and identified a number of important subtopics. It examined the roles and responsibilities of the influenza vaccine stakeholders. It described key issues relevant to influenza vaccine manufacturing and supply and outlined factors impacting purchasing and distribution of influenza vaccine. The contractor also presented a case study of the influenza seasons from 1999 through 2005 highlighting the important challenges that arose during each season. It discussed the impact of bioterrorism preparedness planning on influenza planning during that period, and presented key issues for consideration by policymakers.

For the remaining four issues briefs, the contractor looked in-depth at key areas summarized in the first issue brief. For the second issue brief, "Influenza Vaccine Economics," the contractor analyzed the characteristics of the influenza vaccine market; industry regulatory requirements for vaccine production; and factors affecting the profitability of vaccine production compared to producing other biologic products or pharmaceuticals. The contractor also considered factors affecting the decisions of individual suppliers about how much vaccine to produce in a given year and possible policy approaches to prevent future shortages of influenza vaccine. In the third issue brief, "Influenza Vaccine: Who Buys It and Who Sells It," the contractor considered the vaccine supply chain as well as potential means for facilitating vaccine purchase and distribution. For the fourth issue brief, "Influenza Vaccine Manufacturing," the contractor identified the unique characteristics of the influenza vaccine and discussed the structure of the vaccine manufacturing industry, the nature of the market, decisions about the amount of vaccine produced each year, investments in new technologies to produce vaccine, the implications of relying on global manufacturers for the U.S. domestic vaccine supply, and the factors that influence manufacturers' decisions to enter or exit the market. To prepare the final issue brief, "Influenza Vaccine Demand: The Chicken and the Egg," the contractor looked at the factors that impact the demand for influenza vaccine and how these factors can affect supply.

Report Title: Understanding the Dynamics of Influenza Vaccine Supply and Demand -- Five Issue Briefs http://aspe.hhs.gov/pic/fullreports/06/8476.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Nevel, Amy, 202-690-7795
Performer: RTI International; Research Triangle Park, NC
PIC ID: 8476

How Should Graduate Medical Education in the United States Best Be Financed?

Summary:

This study evaluated the current system, and assessed the strengths and weaknesses of different methods for financing graduate medical education. Researchers performed a literature review and held discussions with experts in the United States and abroad. The study identified a number of ways in which the Medicare program's nearly $8 billion annual support for graduate medical education could be improved. Most of these improvements centered on changes to the formulas used to determine the level of payments Medicare makes to teaching hospitals. The researchers observed that the current system for financing graduate medical education is complicated and the costs of the activities it supports are difficult to measure.

The study identified five elements on which to evaluate any alternatives to the current system: equity, adequacy, efficiency, accountability, and administrative feasibility. Three different models identified in the study for financing graduate medical education were: market-, incentive-, and regulatory-based approaches. Researchers did not recommend replacing the current system, but suggested changes to address limitations in the present system and to achieve particular policy goals. The study cited two policy directions for making refinements for Medicare policies: either emphasizing that Medicare's purpose is to pay for patient care provided to Medicare beneficiaries or using graduate medical education payments to achieve health care workforce needs. The study concluded that any changes in funding policies should occur gradually in order to allow program sponsors and affiliated teaching institutions time to adjust to any redistribution of funds.

Report Title: Alternative Ways of Financing Graduate Medical Education http://aspe.hhs.gov/health/reports/06/AltGradMedicalEdu/
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Nonnemaker, Lynn, 202-690-7807
Performer: Rand Corporation; Santa Monica, CA
PIC ID: 8294

Why Do Individuals Choose and Remain in Nursing Assistant Careers and Work in Nursing Homes?

Summary:

This survey was the first national survey of certified nursing assistants working in nursing homes. It represents a major advance in the data available about certified nursing assistants and has the potential for numerous studies to improve our understanding of the direct care workforce. This survey provided a basis for future evidence-based policy, practice, and applied research to address the shortage of long-term direct care works. The survey involved a stratified, multistage probability design sampling nursing facilities and nursing assistants within these facilities. A sub-sample of 790 facilities was selected. Of those eligible, 187 did not participate in all or a portion of the survey. This resulted in a facility response rate of 76%. A total of 582 facilities agreed to participate, and from these facilities, 4,542 nursing assistants were selected to participate. Among these, 4,274 were eligible and 3,017 completed an interview, yielding a response rate of 71%.

The total number of Americans in need of long-term care is projected to more than double from 13 million in 2000 to 27 million in 2050. Long-term care providers face tremendous challenges each day trying to provide high-quality care to clients. One of the greatest challenges is staff retention among direct care workers -- nursing assistants, personal care attendants and home health aides -- who provide hands-on services to clients. Annual turnover among direct care workers is high and can reach rates of over 100% in some organizations. If left unaddressed, this emerging care gap could severely restrict the ability of providers to deliver adequate long-term care. This introductory study examined the relevant research that led to federal interest in sponsoring the survey, provided a comprehensive overview of the procedures undertaken, addressed the advantages of combining nursing home facility and nursing home worker surveys, and highlighted the potential uses of the data. Subsequent papers will report on survey findings to inform improved planning, decision-making and policy development.

Report Title: An Introduction to the National Nursing Assistant Survey http://aspe.hhs.gov/daltcp/reports/2006/NNASintro.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Squillace, Marie, 202-690-6250
Performer: Department of Health and Human Services (HHS); Washington, DC
PIC ID: 8317

Did Small Grants for Behavioral Cancer Control Research Increase the Number of Scientists Working in This Field?

Summary:

This study assessed the effectiveness of a cancer control behavioral research grant funding program intended to facilitate the growth of a nationwide cohort of scientists with a high level of expertise in behavioral cancer control research. The study found that the program was important in providing early stage investigators with funding opportunities for independent, innovative research. The program offered critical support for career development, and encouraged investigation of new cancer control research questions. The program also prompted more behavioral-focused research, and opportunities to interact with researchers in the field. Grantees were successful in continuing research in the field. For example, researchers were able to obtain additional funding as principal investigators for their research questions. Overall, the findings supported the claim that the program fulfilled its original intent. The grant program advanced the field of behavioral research in cancer control. The program's success provided a strong basis for its continuation in the subsequent budget year.

Report Title: Outcome Evaluation of the Small Grants Program for Behavioral Research in Cancer Control http://aspe.hhs.gov/pic/fullreports/06/8355.doc
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Chollette, Veronica, 301-435-2837
Performer: Battelle Center for Research and Evaluation; Arlington, VA
PIC ID: 8355

Goal 3: Increase the percentage of the Nation's children and adults who have access to health care services, and expand consumer choices

Objective 3.2: Strengthen and expand the health care safety net

How Can the United States Make Sure That It Has a Qualified Workforce to Provide Services to Those Needing Long-Term Care?

Summary:

This study assessed the contributions and effectiveness of existing federal mandates and state and provider based strategies for preventing or reducing the abuse of vulnerable adults. The study pursued this objective in three ways. It examined the usefulness of nurse aide registries and criminal background checks as ways to reduce abuse of nursing facility residents. The study also determined the relationship between past criminal background (or previous incidences of abuse) and the proclivity toward subsequent abuse, neglect or exploitation. Finally, the study identified how states and nursing facilities could improve screening and monitoring processes to ensure a qualified workforce.

The study found that federal and state governments, education and training centers, and employers have created a variety of formal ways to prevent abuse, neglect and exploitation in patients in nursing homes and other long-term care settings. These diverse mechanisms include: certification and licensure of paraprofessional long-term care workers, pre-employment screening (e.g., nurse aide registries, criminal background checks and drug tests) systems, and on-the-job training and personnel monitoring. Using data from two states, a correlation was found between criminal history and incidence of abuse. The study determined that while criminal background checks are only one component of preventing abuse, such checks are valuable during the hiring process. Use of background checks did not appear to limit the pool of potential job applicants. Finally, the study found that there are few federal, state or employer policies in place that support or reinforce post-hiring strategies to ensure a qualified workforce.

Report Title: Ensuring a Qualified Long-Term Care Workforce: From Pre-Employment Screens to On-the-Job Monitoring http://aspe.hhs.gov/daltcp/reports/2006/LTCWqual.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Rosenoff, Emily, 202-690-6443
Performer: The Lewin Group; Falls Church, VA
PIC ID: 8330

What Is the Current Supply of Professional Social Workers in Long-Term Care Settings and What Is the Expected Future Supply?

Summary:

This study assessed the availability of professional social workers in long-term care settings. The study quantified the current and future supply and demand for professional social workers serving older adults across the continuum of long-term care services. The study prepared recommendations for addressing any identified future shortage areas, including cooperative strategies involving Federal agencies, professional associations, and schools of social work. Generally, there were limited research findings across the range of long-term care services that demonstrated the efficacy of using professional social workers (and which could be considered in projecting supply or demand). In the absence of data to the contrary, the researchers assumed that the current conditions vis-à-vis substitution, regulation, licensure, and reimbursement of professional social workers in long-term care would stay constant going forward. Using data from the most recent National Industry-Occupation Employment Matrix, the 2004 Current Population Survey, and the 1% Public Use Microdata Sample from the 2000 Census, it was estimated that approximately 36,100 to 44,200 professional social workers were employed in long-term care settings.

If the ratio of professional social workers to the population age 65 and older remains constant, then approximately 82,000 to 98,400 professional social workers will be needed in long-term care settings by 2050. While their educational preparation, training and clinical orientation may be unique, social workers in long-term care settings often perform tasks, including assessment, psychosocial support, active treatment, and case management that may also be performed by other disciplines. There are efforts underway to include additional aging-related content in social work curriculum and to encourage professional social workers to seek employment in long-term care settings. It is unclear whether these efforts will make a substantial impact on the future supply of professional social workers in long-term care. Because of the uncertainties noted above, it is difficult to determine whether or not the future number of professional social workers will be adequate to care for an increasingly older American population. However, the likely increased demand for social work services will provide a strong impetus for continued job growth in the future.

Report Title: The Supply and Demand of Professional Social Workers Providing Long-Term Care Services: Report to Congress http://aspe.hhs.gov/daltcp/reports/2006/SWsupply.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Bergofsky, Linda, 202-690-6443
Performer: Department of Health and Human Services (Office of the Assistant Secretary for Planning and Evaluation in consultation with Centers for Medicare and Medicaid Services, Administration on Aging and Health Resources and Services Administration), the Department of Veterans Affairs and the Department of Labor Bureau of Labor Statistics.; Washington, DC
PIC ID: 8348

Objective 3.4: Eliminate racial and ethnic health disparities

What Can Hospitals Do to Assure Their Policies and Practices Are Culturally and Linguistically Appropriate?

Summary:

This project developed a survey instrument for use by hospitals to better determine whether their practices are culturally and linguistically appropriate. The assessment tool, the Cultural Competency Assessment Tool for Hospitals (CCATH) was based on the National Standards for Culturally and Linguistically Appropriate Services (CLAS) in Health Care that were developed by the Office of Minority Health. The survey instrument enables hospital administrators, other health care managers, and health policymakers to assess whether their policies and practices are culturally and linguistically appropriate, compare cultural and linguistic quality improvement and monitoring efforts to regional and national benchmarks or norms, and examine the relationship between the standards and key outcomes that may be related to diversity management/cultural competency. Researchers described the protocols used to develop the assessment tool and conducted a comprehensive review of cultural competence assessment tools, pilot tested the instrument in six hospitals, and qualitatively tested the instrument using focus groups and cognitive interviews.

This project successfully initiated development of a survey to assess cultural competency in the provision of hospital care. Additionally, the project provided a solid support for administering the survey broadly. Key findings included: enthusiasm for the assessment instrument by hospital administrators but a recommendation that participating hospitals be provided comparative data to assess their performance; variability in the preferred method of distribution and survey administration, and difficulty on the part of respondents in completing it; and the need to develop a completion guide to facilitate the administration of the survey. Future use of the survey by the same hospitals should be easier. As a result of this study, the Office of Minority Health intends to continue refining and shortening the survey further to the extent possible in order to expedite data collection further. In addition, , a follow-up study was funded by the Commonwealth Fund that will allow the survey to be administered in a large scale field test. This next step will provide an opportunity to conduct extensive empirical analyses of the items in the Survey.

Report Title: Cultural Competency Assessment Tool for Hospitals: An Application of the Culturally and Linguistically Appropriate Services (CLAS) Standards in Health Care http://aspe.hhs.gov/pic/fullreports/06/7865.htm
Agency Sponsor: OPHS-OMH, Office of Minority Health
Federal Contact: Pacheco, Guadalupe, 240-453-2882
Performer: Rand Corporation; Santa Monica, CA
PIC ID: 7865

Do the Benefits That Result from Using Languages Familiar to Patients Outweigh Costs Health Care Providers Face by So Doing?

Summary:

This project developed and tested a model for examining costs associated with medical errors related to (A) barriers to physician-patient communications and understanding versus (B) the provision of a variety of language appropriate services to promote communication and understanding in clinical encounters. An expert panel with expertise in physician-patient communications, language barriers, medical errors, health care quality, cost-benefit analyses and econometrics collaborated in the development of an approach to examining language barriers in various clinical health care settings. The panel determined that the project should address health care service utilization and established measures of both quality and medical error. The panel developed a conceptual model that could serve as a research framework for studies of the impacts of provider-patient language barriers on health care quality and costs. A limited study conducted to test the model, showed that the model worked.

The expert panel subsequently made 38 recommendations relative to the scope of future research, conceptual model, localized study and its findings, further research, and other programmatic and policy-relevant actions. Examples of panel recommendations include: broadening research to include all the patient-provider interactions; expanding examination of impact to include health care outcomes; examining different approaches to providing language support; broadening research beyond mere linguistic issues (to culture, etc.); applying quality indicators identified by the Institute of Medicine; and considering using randomized control trials in future research. The project demonstrated that the model could be used to conduct research to answer questions about costs and other impacts of language barriers in health care. More well-controlled studies utilizing the model are needed to investigate the impacts of these barriers in order to better inform health care policy and programmatic decision-making for racial, ethnic, and linguistic minorities in the United States. Panel members expect that future studies will likely produce findings that will build support for the notion that understanding and being understood is a critical component of health care quality, regardless of primary language spoken or English proficiency status.

Report Title: Assessing the Impact of Provider-Patient Language Barriers on Health Care Costs and Quality http://aspe.hhs.gov/pic/fullreports/06/7711.htm
Agency Sponsor: OPHS-OMH, Office of Minority Health
Federal Contact: Hawks, Betty, 240-453-2882
Performer: Cosmos Corporation; Bethesda, MD
PIC ID: 7711

Objective 3.5: Expand access to health care services for targeted populations with special health care needs

How Successful Has the U.S.-Mexico Tuberculosis Referral and Case Management Project Been and What Changes Are Needed?

Summary:

Established in 2003, the goal of the US-Mexico Binational Tuberculosis Referral and Case Management Project was to build a system to ensure continuity of care for tuberculosis patients who cross the US-Mexico border. The project was a partnership between the Centers for Disease Control and Prevention and the National Tuberculosis Program in Mexico. In the project, referral systems in each country registered eligible patients in a central database and provided data about their treatment to one another. An important outcome of the Project was the effective collaboration between health officials in the United States and Mexico. Patients were not asked about their immigration status. Health officials were able to bring infected tuberculosis patients to treatment regardless of their immigration status. Ongoing dialogue among health and government officials across the border ensured an understanding of the prevailing conditions affecting the health of people in the border area.

A major success of the Project was its formal integration into the detainee screening process at immigration control and enforcement facilities. The evaluators recommended expanding the Project to other areas, agencies, and diseases. Public health professionals and other involved public and private participants urged providing more information to the community about tuberculosis, the Project, and the health benefits of the Project. The majority of participants and community members interviewed felt that more funding was needed. Community participants agreed that efforts should be made to increase funding for program expansion, improve communication, support fieldwork, and bring additional sites into the Project.

Report Title: Evaluation of the U.S.-Mexico Binational TB Referral and Case Management Project http://aspe.hhs.gov/pic/fullreports/06/7985.1.doc
Agency Sponsor: CDC-OD-OSI, Office of the Director, Office of Strategy Innovation
Federal Contact: Slaton, Terrie, 404-639-7647
Performer: Battelle; Arlington, VA
PIC ID: 7985.1

What Are the Barriers to Implementing Innovative Technology in Residential Long-Term Care Settings?

Summary:

Approximately two million older Americans live in nursing homes and other residential care settings. Innovative technology can play a vital role in the lives of these individuals. Persons in residential long-term care settings have physical or cognitive limitations that may be compensated through use of technologies. Technology may offer ways to provide lower cost interventions and address the needs of a growing number of individuals who are expected to need long-term care residence. This study sought to: (1) describe a range of existing and emerging technological solutions for select care issues appropriate for residential care settings; (2) identify barriers to the successful implementation of technology in residential care settings; and (3) propose initial steps to address the barriers. To identify technologies, the project team conducted literature and internet searches, consulted long-term care buyer's guides, and spoke with technology manufacturers and vendors. To identify barriers, the project team conducted a literature review and spoke with experts representing four distinct perspectives in long-term care--regulators, providers, technology manufacturers, and other experts--about their experience implementing technology in long-term care settings.

Assistive technologies are being developed to help older adults remain more independent, safer, and connected to the rest of the world. There are a small but growing number of technologies designed mainly to help caregivers in residential care settings. Five aspects of care in residential settings--wander management, fall prevention, incontinence care, assistance calls, and bathing--were identified as key areas in which to investigate technological developments.

The study identified a complex set of circumstances contributing to barriers to the implementation of technology in long-term care settings, including: lack of information about technologies and the residential long-term care market; perceived lack of financial resources to develop and buy residential long-term care technologies; failure of the regulatory process to keep pace with technological advances; industry's lack of standards for technologies in residential long-term care; and providers' lack of experience implementing and managing technological changes. In light of these findings, a series of educational and collaborative activities should be developed to: remedy existing gaps in knowledge about technologies in residential long-term care settings; explore ways to encourage development and implementation of cost-effective technological innovations in residential long-term care settings; explore how best to reduce regulatory barriers to innovation; encourage development of industry standards for residential care technologies; and educate providers about implementing and managing technological change.

Report Title: Barriers to Implementing Technology in Residential Long-Term Care Settings http://aspe.hhs.gov/daltcp/reports/techbarr.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Aykan, Hakan, 202-690-6443
Performer: Polisher Research Institute; North Wales, PA
PIC ID: 8319

What Role Do Negotiated Risk Agreements Play in Assisted Living?

Summary:

Negotiated risk agreements were developed as a mechanism to enable older persons residing in regulated assisted living settings to make choices, even when they entail some risk. Negotiated risk agreements are written documents that list: the behavior or resident preference of concern to the provider; the potential or actual risk; the resident preferences and potential provider accommodations or suggested alternatives to the behavior that reduce risk while meeting resident preferences; a negotiated resolution; and the resident's acknowledgement and acceptance of the potential negative consequences of his or her actions. The study examined states' statutory and regulatory provisions related to negotiated risk agreements and analyzed the policy objectives that the agreements were designed to meet. The study also analyzed the debate surrounding these agreements; and gained an understanding of how assisted living providers, staff and residents viewed and used such agreements. This study was designed to inform state policymakers, assisted living providers, and key stakeholders about agreements and issues related to their use.

Balancing the need to assure both autonomy and safety is a major challenge when providing long-term care services to older persons who live in licensed group settings. Provider policies and state regulations intended to ensure safety can conflict with individuals' ability to make the choices they prefer. Negotiated risk was conceived to help assisted living programs provide residential alternatives to institutional care. Residential assisted living programs provide frail and cognitively impaired older persons an option that maximizes their privacy, independence, choice, and enables them to maintain a normal lifestyle. These qualities are generally lacking in institutional long-term care settings.

Over the past decade, many assisted living providers have adopted negotiated risk agreements, and several states have regulatory provisions regarding their use. However, their use is not without controversy. Assisted living providers, policy makers, aging advocates, and long-term care experts have defined negotiated risk agreements as a mechanism to enhance resident choice by providing a rigorous process designed to balance autonomy and risk for residents and providers in assisted living. While the sample is small and not representative, the findings suggest that negotiated risk agreements can be a useful means to help residents and providers achieve a balance between desires for autonomy and concerns about safety. At the same time, they suggest that the negotiated risk agreement concept is proving difficult to broadly and consistently apply.

Report Title: Study of Negotiated Risk Agreements in Assisted Living Facilities http://aspe.hhs.gov/daltcp/reports/2006/negrisk.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Kennedy, Gavin, 202-690-6443
Performer: Research Triangle Institute; Research Triangle Park, NC
PIC ID: 8345

Who Lives in Residential Care Facilities and How Disabled Are They?

Summary:

This study examined estimates from three major national surveys of the number of persons living in residential care facilities, often called "assisted living facilities." The surveys identified such facilities through a number of questions about housing and services provided. Estimates from the surveys of the older population examined were in substantial agreement. About 6.5 percent of persons 65 or older, about 2.2 million persons, lived in some type of residential care other than settings for special populations (such as persons with mental illness or intellectual disabilities). The estimates indicated that most, about 1.45 million, lived in nursing homes, but more than 750,000 lived in alternative residential care settings.

In recent years, an increasing number of persons with disabilities have chosen to live in residential care settings. For the elderly, these facilities offer help with activities of daily living (bathing, taking medication, etc.) as well as providing oversight and opportunities for social activities and interaction. It has been hypothesized that people in these facilities, which range in size from small homes to large campuses, were more disabled than persons living in traditional houses or apartments, but were less disabled than persons living in nursing homes. However, relatively little has been known about the characteristics of persons in these facilities for three reasons: (1) the national Census Bureau classifies such persons as living in housing units and no data are collected to indicate that they are receiving services; (2) people typically pay for their care themselves, so there is no record of their care in a government database, as there would be in the case of a nursing home; and (3) if the facilities are regulated at all, they are regulated by the states, and each state has its own system.

The three surveys provided a consistent picture of the characteristics of the residential care population. Persons living in residential care facilities were more likely to be over age 85, female, and widowed, than were persons residing in their own homes. Residential care facilities served a broad income range. Persons living in those facilities were more likely to have incomes below $10,000, roughly approximating the federal poverty level for older couples. Relative to older persons remaining in traditional private housing, the residential care population was far more likely to receive help with activities of daily living, and to suffer from Alzheimer's disease or other dementias. Estimates from the two surveys that included the facility or institutional population, indicated that the prevalence of disability and dementias was dramatically higher among those living in residential care facilities than in community care settings, and highest in nursing homes.

Report Title: The Size and Characteristics of the Residential Care Population: Evidence from Three National Surveys http://aspe.hhs.gov/daltcp/reports/2006/3natlsur.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Drabek, John, 202-690-6443
Performer: Urban Institute; Washington, DC
PIC ID: 8347

Who Has an Intellectual or Developmental Disability, Who Provides Their Care, and How Will That Change in the Future?

Summary:

This study provided a comprehensive view of the supply and demand for "direct service professionals," individuals who help persons with intellectual or developmental disabilities. Intellectual disability consists of having low intellectual functioning and limitations in conceptual, social, and practical skills needed for everyday life. Individuals with developmental disabilities have limited functioning in three or more "activities of daily living." These activities are: capacity for independent living, economic self-sufficiency, learning, mobility, receptive and expressive language, self-care, and self-direction. Direct support professionals are paid to provide a wide range of day-to-day services to persons with intellectual or developmental disabilities. These professionals include direct care staff, aides, home health assistants, respite care providers, and personal care attendants. By providing services such as habilitation, health needs, personal hygiene, employment, transportation, recreation, house-keeping, and home management support, they enable individuals to live, work and function in their communities.

Direct support professionals are essential to the quality of life, health and safety of more than 1 million Americans with intellectual or developmental disabilities in need of long-term services and support. An estimated 4.3 million Americans with intellectual and/or developmental disabilities live in residential settings, their own homes or with other members of their families. Ensuring access to and quality of professional support is key to realizing goals established in various statements of national purpose with regard to full citizenship and inclusion of individuals with intellectual or developmental disabilities. In June 2003 there were an estimated 874,000 full-time equivalent direct support personnel assisting intellectually and/or developmentally disabled individuals in group residential settings, family homes, their own homes, community jobs, vocational and day training settings, and other service settings.

The current study estimated that by 2020, 1.2 million direct service professionals will be needed in order to provide an estimated 1.4 million individuals with intellectual or developmental disabilities with needed residential, vocational and other support. This is an increase of about 323,000 jobs, roughly 37% more such professionals than in 2003. Most of the increase will be due to increases in population, life expectancy among persons with the disabilities, aging of family caregivers, and use of increased types of home and community-based services. This increased demand will occur at a time when the labor supply of adults age 18-39 years, who traditionally have filled these jobs, is expected to increase only by 7%. Meeting future demand for support professionals will be difficult to achieve through regular recruitment alone. Attracting new personnel and improving retention will be essential.

Many industries will compete for the direct care workers, including the rapidly growing long term services and supports systems for aging persons with disabilities. Compensation, working conditions, career opportunities and job design will be key to retaining personnel and attracting new workers. Providing adequate levels of high quality, committed and stable direct support will need sustained effort by many actors. Work will be needed across federal, state and local governments, education and training institutions, workforce investment systems, faith-based organizations, service provider organizations and support professionals themselves. These groups and institutions will need to shape support roles into ones that offer those who enter these positions enough opportunity and respect to consider them viable careers.

Report Title: The Supply of Direct Support Professionals Serving Individuals with Intellectual Disabilities and Other Developmental Disabilities: Report to Congress http://aspe.hhs.gov/daltcp/reports/2006/DSPsupply.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Drabek, John, 202-690-6443
Performer: Department of Health and Human Services (HHS); Washington, DC
PIC ID: 8349

Goal 4: Enhance the capacity and productivity of the Nation's health science research enterprise

Objective 4.1: Advance the understanding of basic biomedical and behavioral science and how to prevent, diagnose, and treat disease and disability

What Evaluation Design Would Best Measure the Success of Different Ways to Recruit Patients to Participate in National Institutes of Health Clinical Trials?

Summary:

This feasibility study was the first of a multi-phase, multi-year evaluation seeking to answer the question: what patient recruitment strategies result in the greatest number of contacts, referrals and enrollments in National Institutes of Health (NIH) clinical trials for various categories of clinical studies? NIH needs several thousand new patient participants each year for clinical research it conducts. Clinical research conducted by NIH itself ("intramural" research) at the Mark O. Hatfield Clinical Center, is essential to the innovative excellence sought by NIH. At any given time, the 15 Institutes of NIH have nearly 900 clinical studies ("protocols") underway for which patient and non-patient volunteer participants are being actively recruited. While over 5,000 new participants are enrolled in these clinical trials each year from the direct efforts of NIH, several times more participants are needed. NIH clinicians do not provide regular routine care to a patient population from which they can draw clinical trial participants. They must recruit from external sources, such as referrals from other physicians. Patient recruitment is one of the most significant bottlenecks to treatment development. Failed or delayed trials extract significant costs, wasting money, patient participants' time, and discouraging care professionals from cooperating with future research. On a national basis, it has been estimated that 78 percent of all clinical studies fail to enroll the needed number of patients on time.

To facilitate its recruitment efforts, NIH established two centralized recruiting offices: the Patient Recruitment and Public Liaison (PRPL) serving investigators across institutes, and the Clinical Studies Support Center (CSSC) recruiting patients for National Cancer Institute studies. These two offices provide NIH with some centralized aid in recruiting subjects, yet personnel for individual studies often must recruit patients on an ad hoc basis, and little guidance for successful recruitment exists. These two offices are able to provide only a fraction of the total number of study participants needed. The study focused on gathering and evaluating PRPL and CSSC recruitment data to assess recruitment strategies for NIH intramural early phase clinical trials.

This study arrived at four major findings: It identified a number of key variables of interest in each data set (for PRPL and CSSC) as well as a more limited number of key variables common to both. Researchers determined that the primary definition of "success" in recruiting patients into clinical trials should be referral to a study protocol. Several basic categories of patients, diseases, and protocols were developed. Finally, researchers examined how often different recruitment strategies were used as well as the source of information reported by the patients (leading them to participate) across these categories. The goal of the overall project is to develop an evidence-based systematized approach to help both the PRPL and the CSSC and individual investigators in selecting patient recruitment strategies that will result in optimum patient enrollment in particular studies. Results of this first, feasibility, study will serve as a baseline for the development of a prospective study and of methods to collect improved recruitment outcome data.

Report Title: Evaluation of Patient Recruitment Strategies (for NIH Intramural Early Phase Clinical Trials) -- Phase 1 Feasibility Study Final Report http://aspe.hhs.gov/pic/fullreports/06/8384.pdf
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Dominguez, Dinora, 301-402-6072
Performer: CSR Incorporated; Sponsored by: National Institutes of Health: Mark O. Hatfield Clinical Center and the National Cancer Institute; Arlington, VA
PIC ID: 8384

How Could the National Heart, Lung and Blood Institute Bioengineering Research Partnerships Program Be Evaluated?

Summary:

This study sought how best to evaluate the success, and to recommend methodology and measurement criteria for assessing the performance of Bioengineering Research Partnership Program grants. The program is sponsored by the National Heart, Lung, and Blood Institute. The National Institutes of Health (NIH) established the Bioengineering consortium in 1997 to focus on bioengineering in public health and, a year later, the Bioengineering Research Partnership Program to solicit multidisciplinary bioengineering research teams applying an integrated approach to developing knowledge and methods focused on important biological or medical problems. Since 1999, the National Heart, Lung and Blood Institute has funded 42, or more than 25%, of the 160 grants in this area that are supported by NIH. The study examined nine case studies and summarized existing award and application data to provide an overview of project goals and progress. The study included online interviews to collect information about nine key measurement criteria. The study resulted in a logic model, metrics that differentiate between successful and less successful awardees, and recommended methods for collecting evaluation data.

The study found that: the typical grant was crafted specifically for this particular program; awardees believed that this type of research was unlikely to be funded elsewhere in NIH; the typical project includes lead investigators from both medical and science fields; design-driven research make it difficult to identify a group with whom to compare performance of these grants; outcome evaluations would be preferable, but the extent of published results may serve as a rough proxy in the meantime.

Report Title: Bioengineering Research Partnership Program Feasibility Study http://aspe.hhs.gov/pic/fullreports/06/8457.pdf
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Lundberg, Martha, 301-435-0513
Performer: Stephanie Karsten/Humanitas, Inc; Silver Spring, MD
PIC ID: 8457

Is It Feasible to Evaluate the Centers for AIDS Research Program?

Summary:

This study explored whether an outcome evaluation of the Centers for AIDS Research was warranted and feasible and made recommendations regarding the design of such an evaluation. The Centers for AIDS Research is a long-standing program funded by the National Institute of Allergy and Infectious Diseases. The Centers provides administrative aid, shared research support, and coordination of AIDS research projects funded by other NIH grants and contracts. In 1998, the program was redesigned as a cross-NIH effort that provided both fiscal and scientific flexibility for the grantee institution. The scientific focus was expanded to include all basic, translational (taking research from the laboratory to the community), and clinical AIDS research. The feasibility study included consulting with key workers at the Centers (including the steering committee and principal investigators) and program staff at the National Institute of Allergy and Infectious Diseases. The investigators developed a provisional logic model that described inputs, activities, outputs, outcomes, impacts, and external influences of the Centers for AIDS Research program. Investigators reviewed existing data on the Centers and potential comparison groups.

The study concluded that an outcome evaluation was needed and would be useful based on strong support, availability of investigator progress reports that could be used as a primary data source, and descriptive and anecdotal data useful for providing context. Challenges to the design of an outcome evaluation would include variations of individual Centers' grant awards. This was the possible explanation for the finding that there were detected no differences in research productivity or research character between Centers institutions and likely comparison groups. The feasibility study supported conducting an outcome evaluation using a cross-sectional design based on data about the program, institutions, and individual awards. No distinct comparison with external institutions or programs was recommended. The results of such comparisons could not be interpreted with confidence. Data collection would likely require Office of Management and Budget clearance. The proposed design also recommended a case study component in order to gather qualitative evidence on how Centers for AIDS Research activities lead to outputs, outcomes, and impacts.

Report Title: Feasibility Study for an Evaluation of the Centers for AIDS Research http://aspe.hhs.gov/pic/fullreports/06/8462.pdf
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Halula, Madelon, 301-402-2636
Performer: Science and Technology Policy Institute; Arlington, VA
PIC ID: 8462

Objective 4.2: Accelerate private sector development of new drugs, biologic therapies, and medical technology

Did the “Continuous Marketing Application” Pilots Increase the Efficiency and Effectiveness of the New Drug Review Process?

Summary:

This study evaluated the Food and Drug Administration’s (FDA's) Continuous Marketing Application (CMA) Pilots 1 and 2. The two pilot programs were established under the Prescription Drug User Fee Act III (PDUFA III). PDUFA is a series of laws that allows the agency to help fund the review of new drugs through fees paid by the companies that submit new drug applications. PDUFA was first enacted in 1992 and has been reauthorized twice, each time for five years, in 1997 and 2002, respectively. The two pilots were designed to test whether providing early review of selected applications and additional feedback and advice to sponsors during drug development for selected products can further shorten drug development and review times. Pilot 1 involved a commitment on the part of FDA to review and provide feedback to the sponsor within 6 months of submission of "reviewable units" of an application in advance of the submission of the complete application.

This pilot program represented an extension of the "rolling review" program begun under the Food and Drug Act Modernization Act of 1997 and was limited to applications that had received a "Fast Track" designation. Pilot 2 involved a commitment on the part of FDA to provide more structured and extensive interaction and feedback to sponsors for up to one Fast Track application per review division during drug development. This pilot represented an extension of the usual interactions between FDA and sponsors during drug development. To evaluate the costs and benefits of these pilots, FDA commissioned an independent assessment. The evaluation included interviews of FDA employees and product sponsors who participated in the pilots. Also interviewed were sponsors with eligible drug products who did not participate in the pilots. The non-pilot, eligible products sponsors formed a comparison cohort against which the pilots were measured. In addition to interviews, the evaluation examined data from product action packages, FDA's time tracking system, the PDUFA goal date tracking system, and the document filing system.

The study did not reach any conclusions about whether either pilot program should be continued. However, after review of the findings, FDA and industry representatives agreed that although the pilots demonstrated value in some areas, the overall added benefits of the programs did not justify their costs to FDA. Therefore, FDA is proposing to recommend that the pilot programs not be continued in the upcoming reauthorization of the Prescription Drug User Fee Act (PDUFA IV). Recommendations regarding the upcoming reauthorization were published in the Federal Register on January 16, 2007 (Vol. 72, No. 9/Page 1743).

Report Title: Independent Evaluation of the FDA's Continuous Marketing Application Pilot 1 and Pilot 2: Initiatives & Evaluations http://aspe.hhs.gov/pic/fullreports/06/8312.pdf
Agency Sponsor: FDA, Food and Drug Administration
Federal Contact: Bobolis, Mary A., 301-827-5253
Performer: Booz Allen Hamilton; McLean, VA
PIC ID: 8312

Can the FDA Improve the Efficiency and Effectiveness of New Drug Application and Biologics Licensing Application First Cycle Reviews?

Summary:

This retrospective study examined the impact of the Food and Drug Administration's (FDA's) initiatives to improve the likelihood that applications would be approved by the agency during the first round of review ("first cycle review") for New Dug Applications, New Molecular Entities and Biologic Licensing Applications during the five-year period of the Prescription Drug User Fee Act ("PDUFA III"). A New Molecular Entity is an active ingredient that has never before been marketed in the United States in any form. The primary goal of this evaluation was to determine the factors that prevent applications from being approved on first review and that result in them being subject to more than one ("multi-cycle") review. The study did not evaluate the merit of the underlying science or quality of reviews (discipline reviews), but rather investigated process issues that might cause multiple review cycles. The analysis was based on data originating from FDA. Under the Prescription Drug User Fee Act, FDA committed to achieving specific performance goals to improve the effectiveness and efficiency of new drug application and biologics licensing application reviews.

Several goals sought to improve the review process that occurs between initial submission of an application and subsequent FDA action. Action Packages (product review documents including correspondence, meeting notes, discipline reviews, etc.) were analyzed relating to all new molecular entity drug and biologics licensing applications submitted during fiscal years 2002-2004 that reached the first step in agency action by December 1, 2004 (a total of 77 applications). These packages generally contained the critical information required for Office and/or Division Directors to decide whether to approve the application or whether it contains sufficient information to be either approvable or not approvable. Where feasible during the study, comments from FDA Regulatory Project Managers involved in the product reviews was solicited to fill in missing information. The factors that influenced the speed and effectiveness of reviews requiring multiple review cycles were found to include product and sponsor characteristics, quality of the design and execution of the drug development program, variations in review processes, and development of post-marketing commitments. One major finding was that priority and fast-track products had higher first-cycle approval rates. Beyond the unmet medical need however, increased regulatory and sponsor attention throughout the drug development and review process may have contributed to the timely identification and resolution of issues. Most products that failed to receive first-cycle approval had key deficiencies in only one or two categories, with an even breakdown between the categories of safety, efficacy, and chemistry.

FDA reviewer team members agreed that early on-going discussions with sponsors was the most important factor in order to identify issues and provided an opportunity for timely resolution, ideally before the first formal advisory letter was sent to the applicant ("first action"). Effective communication and responsiveness to FDA inquiries by applicants was prevalent in applications that were approved during the first-cycle reviews while persisting disagreements over issue resolution were associated with approval delays. FDA's Good Review Management Principles and Practices guidance (published on March 31, 2005) recommended specific timelines for New Drug/Biologics Licensing Applications review procedures. Further, early and open communication with the sponsors would allow sponsors to address/resolve issues in a timely manner, potentially within the first review cycle. A number of the suggested recommendations may have resource implications for the FDA and sponsors. The final step of the first cycle study is a prospective evaluation which began in 2006 and is expected to be completed by 2008.

Report Title: Independent Evaluation of the FDA's First Cycle Review Performance -- Retrospective Analysis Final Report http://aspe.hhs.gov/pic/fullreports/06/8313.pdf
Agency Sponsor: FDA, Food and Drug Administration
Federal Contact: Bobolis, Mary A., 301-827-5253
Performer: Booz Allen Hamilton; McLean, VA
PIC ID: 8313

Objective 4.3: Strengthen and diversify the pool of qualified health and behavioral science researchers

How Has the Graduate Partnerships Program Enhanced Student Education and Participation in Research?

Summary:

This process evaluation assessed the operations of a program started in 2000 that pays for graduate level training. The program operates through partnerships with universities dedicated to biomedical, basic, and clinical research. The study found that the "Graduate Partnerships" program was effective in recruiting and selecting students and providing them with a high-quality research training experience. Overall, student satisfaction was high and most of the students were moving toward degree completion. The use and refinement of a novel collaborative approach to dissertations allowed students to be co-mentored by both a National Institutes of Health and a university scientist. Challenges faced by the program included unpredictable funding and insufficient administrative help for the program directors. Other challenges included competing with traditional PhD programs for candidates and monitoring progress of over 400 students toward completion of their degrees. Features the evaluators recommended could be added to the program included creation of a process for periodic review of each partnership and developing the ability to add new partnerships over time. The program began implementing recommendations from the study in spring of 2006.

Report Title: Graduate Partnerships Program (GPP) Five-Year Self-Study http://aspe.hhs.gov/pic/fullreports/06/8390.pdf
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: McGee, Richard, 301-594-9604
Performer: N/A; Bethesda, MD
PIC ID: 8390

How Should a Process Evaluation of the Centers of Biomedical Research Excellence Program Be Designed and Conducted?

Summary:

This study examined the feasibility of and designed an optimum process evaluation to assess the operations and outputs of the Centers of Biomedical Research Excellence (COBRE) program during its first five years and to measure success of junior investigators supported during this period. COBRE, begun in 2000, is a part of the Institutional Development Award (IDeA) program administered by the National Institutes of Health (NIH). IDeA assures broader geographic distribution of NIH funds for biomedical and behavioral research. The program fosters health-related research and enhances the competitiveness of investigators at institutions located in states which have historically received fewer research grants. COBRE does this by supporting thematic multidisciplinary centers that strengthen the research capacity of biomedical institutions. It does so by expanding and developing faculty research capability and research infrastructure, including establishing facilities needed to carry out multidisciplinary, collaborative research.

The study concluded that an outcome evaluation of the COBRE program was not appropriate at present due to the program's short existence. The feasibility study created a conceptual framework (logic model) illustrating how COBRE should work. The framework showed how funding and other resources, baseline characteristics, and specific program activities influence achievement of program goals. The results of this feasibility study were used as a basis for the process evaluation of the COBRE program, which began in September 2006. The evaluation will use the framework during the feasibility study to assess seven baseline characteristics of COBRE that could affect each center's success. The evaluation will also examine resources of the National Center for Research Resources that are allocated to COBRE and external factors over which the centers had no control.

Report Title: Feasibility Study for a Process Evaluation of the COBRE Program http://aspe.hhs.gov/pic/fullreports/06/8456.htm
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Newman, Patricia, 301-435-0864
Performer: Carlyn Consulting; Poolesville, MD
PIC ID: 8456

Goal 5: Improve the quality of health care services

Objective 5.2: Increase the appropriate use of effective health care services by medical providers

What Do We Know About Medicare Advantage Plans?

Summary:

Most of the 43 million elderly and disabled people on Medicare (81%) have their health bills aid by the traditional fee-for-service program; 19% (8.3 million) receive their Medicare benefits through private health plans that receive payments from Medicare, now generally known as "Medicare Advantage" plans. This study identified a baseline of timely, policy-relevant information about Medicare Advantage health care service delivery products, analyzed how these products compared to past offerings when only local Medicare Advantage options were available, and examined the initial planning decisions and experiences under the new statutorily-specified competitive bidding process through which health insurance organizations are approved to offer Medicare Advantage.

The Medicare Prescription Drug, Improvement, and Modernization Act of 2003 (MMA) made many changes to the Medicare Advantage program, including establishing regional preferred-provider organization plans, and expanding the number of private fee-for-service plans. In addition, regional and local Medicare Advantage plans were approved to provide Medicare beneficiaries with a comprehensive set of benefits, including a new prescription drug benefit. The study included both quantitative analyses of existing administrative data available primarily from the Centers for Medicare and Medicaid Services and qualitative analysis based on discussions with individuals in a number of diverse health insurance firms. These discussions focused learning how insurers made coverage decisions and arrived at insurance marketing strategies. This study found that the growth in Medicare Advantage contracts in 2006 made these plans more available to beneficiaries, especially in rural areas.

Although many firms participated in the Medicare Advantage program, a small number dominated. Health maintenance organizations still accounted for most enrollments in Medicare Advantage. However, other health insurance and health care services products, especially private fee-for-service health care plans also drove much of the enrollment growth.

Report Title: Analysis of the Characteristics of Medicare Advantage Plan Participation: Final Report http://aspe.hhs.gov/pic/fullreports/06/8477.pdf
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Cox, Donald, 202-690-6597
Performer: Mathematica Policy Research, Inc.; Princeton, NJ
PIC ID: 8477

How Do Integrated Care Programs for the Frail Elderly Use Community-Based Primary Care Physicians?

Summary:

This was the most recent of several studies leading up to a final report evaluating the "Program of All-Inclusive Care for the Elderly" (PACE) that will be released at a later date. PACE is an innovative model that seeks positive outcomes and cost savings by providing a range of integrated preventative, acute care, and long-term care services to manage the often complex medical, functional, and social needs of frail elderly individuals. The over-all study is examining the expected quality and cost of providing PACE program services under the Medicare and Medicaid programs when it is operated as a permanent (as opposed to a demonstration) program. The study is also comparing the cost, quality, and access to services provided by private, for-profit entities operating under demonstration project waivers with the costs, quality, and access to services of public and not-for-profit providers. This project seeks to expand the understanding gained in previous evaluations of PACE by predicting costs beyond the first year of enrollment, assessing the impact of higher end-of-life costs and long term nursing home care, and assessing the impact of local treatment practices.

The current study examined how three programs, in California, New York, and Wisconsin, used physicians. The studies included site visits to the local programs and interviews of participants. None of the sites preferred working with community-based physicians; they considered this a way to get care to more people who need their services. To counteract the concern that community-based physicians would not communicate with or work as well with the PACE team as PACE physicians do, and thus cause the program to not be able to care as effectively for its patients, programs at the three sites had implemented ways that, based on staff's perception, addressed these concerns.

Report Title: How Integrated Care Programs Use Community-Based Primary Care Physicians http://www.cms.hhs.gov/reports/downloads/cheh.pdf
Agency Sponsor: CMS, Centers for Medicare and Medicaid Services
Federal Contact: Thomas, Fred G., 410-786-6675
Performer: Mathematica Policy Research, Inc.; Princeton, NJ
PIC ID: 7667

What Are Private Sector "Best Practice" Approaches to Control Costs of Prescription Drug Benefit Programs?

Summary:

This study identified and described in case studies the effective strategies and best practices developed by private sector organizations to manage the costs of prescription drugs. While state Medicaid and other public-sector efforts to contain drug benefit costs have been well documented, information on the nature and effectiveness of private sector approaches has been limited. Case studies were formulated for five private organizations that provide, manage, or fund prescription drug benefits which have achieved positive results to control the growth of drug costs. The organizations chosen were Tufts-New England Medical Center, Pitney Bowes Inc., Medco Health Solutions Inc., Provider Service Network, and Kaiser Foundation Health Plan.

This study identified cross-cutting themes used by the private sector and outlined opportunities for further research. Several cost savings strategies used by these companies were identified. These included: the use of clinical pharmacists to educate physicians and recommend cost-savings strategies, establishing tiered and evidence based formulary systems to limit high-cost branded drugs, employing hospital information technology systems to identify high-cost practices and suggest lower-cost drug alternatives (such as generics, pill-splitting), limiting detailing and sampling of drug representatives, encouraging mail-order dispensing, and participating on pharmacy and therapeutic (P&T) committees (multi-disciplinary health care provider groups reviewing medical services) and hospital committees. Benefits these best practices yielded included: greater use of generic drugs and improved compliance with drug formularies. These practices resulted, in turn, significant immediate cost savings and slower drug cost increases.

Report Title: Identification and Description of Industry Best Practices to Manage the Costs of Prescription Drugs http://aspe.hhs.gov/sp/cost/cost.pdf
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Bush, Laina, 202-260-7329
Performer: Abt Associates Inc; Bethesda, MD
PIC ID: 8474

How Do Health Care Professionals Look for Medical Information on Medical Web Sites?

Summary:

This extensive feasibility study investigated how health care professionals look for medical information on publicly available Web sites and how they browse on-line medical resources including multi-media files and education materials. The study was conducted to provide information for future Web application designs that will be developed by the National Library of Medicine. The study sought answers to a variety of questions regarding human-computer interactions, including: what usability criteria should be used in selecting search engines; how do users cope with increasing amounts of information, especially medical and health information, and what is the desired format and level of language for medical and health information? The study included a review of the literature, user interviews, and focus groups.

The study found that skills necessary to successfully negotiate government medical and health information Web sites included: familiarity with medical terminology, English language literacy, and the ability to formulate searches. Researchers were also reminded of the fact that health care workers of all kinds are time-starved professionals who generally don't spend long periods on-line and need to be alerted to important information they may need rather than having to initiate their own searches. In the medical area, useful information is perishable and the date that information is published is of great interest to information "finders" and needs to be highlighted. The authority/source of information is likewise of significant interest in this subject area. Effective media presentations and interesting graphic images can also contribute to improved communication of information. The study also produced observations regarding future processes for evaluating how effective Web information was. For instance, researchers observed that individual interviews together with participant's hands-on activities yielded better data regarding interface functionality while focus group activities yielded better information regarding content. A concluding caution provided by the study was pointing out that all the participants in the study had access to modern information systems but that there are still relatively large numbers of National Library of Medicine information consumers who do not have access to such technology. These latter groups will need intermediaries to help them access good quality information.

Report Title: On-Line Search Behavior for Medical Information http://aspe.hhs.gov/pic/fullreports/06/8430.pdf
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Ma, Wei, 301-496-8436
Performer: Quotient Inc; Columbia, MD
PIC ID: 8430

Objective 5.3: Increase consumer and patient use of health care quality information

How Can the Effectiveness of Efforts to Detect Breast and Cervical Cancer Early Be Measured and Improved?

Summary:

This program management project developed a comprehensive 5-year evaluation plan for the National Breast and Cervical Cancer Early Detection Program. The evaluation plan was developed to be used to gauge accomplishments and better coordinate efforts within the offices responsible for administering the cancer detection program. The evaluation plan was intended to detect broad national impact of Federal program efforts. It was not intended to measure performance of individual grantees.

 The project team identified broad health outcomes and program/administrative outcomes to be measured to determine overall success. Health outcome priorities included increasing both the number of women screened and appropriate cancer diagnosis of women with abnormal test results. Additional health outcome priorities included increasing the percent of women: who begin treatment within two months of diagnosis and those with precancerous cervical lesions who begin treatment within three months. The evaluation plan included measurement of characteristics of women being served and the quality of services provided. The Centers for Disease Control and Prevention will conduct a strategic planning effort and will prioritize evaluation activities. The agency will refocus program activities to better achieve these priorities.

Report Title: Strategic Management and Evaluation Plan of the National Breast and Cervical Cancer Early Detection Program http://aspe.hhs.gov/pic/fullreports/06/7973.1.pdf
Agency Sponsor: CDC-OD-OSI, Office of the Director, Office of Strategy Innovation
Federal Contact: Slaton, Terrie, 404-639-7647
Performer: Research Triangle Institute; Research Triangle Park, NC
PIC ID: 7973.1

Who Is Aware of, Uses, and How Good Is the Information in a Community Preventive Services Guide?

Summary:

Through a survey, researchers obtained information on awareness, use, and appraisal of information in the Guide to Community Prevention Services. The Guide summarizes what is known about the effectiveness, efficiency, and feasibility of interventions to promote community health and prevent disease. The Guide provides public health practitioners and decision makers with recommendations regarding population-based interventions to promote health and to prevent disease, injury, disability, and premature death, appropriate for use by communities and health care systems. The information in the Guide is developed by an independent, non-federal Task Force appointed by the Director of the Centers for Disease Control and Prevention (CDC). The Task Force makes recommendations for the use of various interventions based on the evidence gathered in rigorous and systematic scientific reviews of published studies. The findings from the reviews are published in peer-reviewed journals and also available through the internet at http://www.thecommunityguide.org/.

This evaluation obtained survey responses from state health directors, state chronic disease directors, Healthy People 2010 coordinators, and a sample of local/county health directors. More than half, 53 percent, of the decision-makers surveyed were familiar with the Guide. State-level decision-makers were more likely than local decision-makers to be familiar with the Guide. Of those familiar with the Guide, 61 percent had used it for their work and tended to encourage others to use it. The overall importance of relying on systematic scientific reviews of successful interventions supports the need to revise/update existing portions of the Guide and to develop additional sections for new prevention activities.

Report Title: Surveillance Survey for the Guide to Community Preventive Services http://aspe.hhs.gov/pic/fullreports/06/8295.pdf
Agency Sponsor: CDC-OD-OSI, Office of the Director, Office of Strategy Innovation
Federal Contact: Slaton, Terrie, 404-639-7647
Performer: Research Triangle Institute; Research Triangle Park, NC
PIC ID: 8295

Who Wants to Know What About Sexually Transmitted Disease?

Summary:

This comprehensive qualitative evaluation looked at who was using sexually transmitted disease (STD) surveillance data and the best ways of getting this information out to current and prospective data users. Various state, local, and private offices, organizations and individuals implement prevention, treatment, and control programs to preserve and promote sexual health. Fundamental program components include surveillance activities such as screening, counseling, and reporting; contacting and treating sex partners of infected individuals to decrease further transmission; and conducting community outreach. The Centers for Disease Control and Prevention (CDC) recognized that changes in technology and norms regarding information dissemination required a reevaluation of the methods used to distribute sexually transmitted disease surveillance data. The evaluation included interviews, web-based discussions, and analysis of surveillance report subscriber data.

The research revealed widespread agreement that the CDC office disseminating the information targets health state or local health directors, sexually transmitted disease program managers, and epidemiologists. Participants also agreed that students in various disciplines use the report. Very few people, however, said that they thought the surveillance report is meant for non-public health audiences. The participants suggested a wide range of potential users of STD surveillance data that may be classified into several broad categories: maternal/child health/family planning organizations, corrections agencies, policy makers, managed care organizations, and educational institutions.

Report Title: National STD Surveillance Data Dissemination-Who Wants to Know What? http://aspe.hhs.gov/pic/fullreports/06/8296.pdf
Agency Sponsor: CDC-OD-OSI, Office of the Director, Office of Strategy Innovation
Federal Contact: Slaton, Terrie, 404-639-7647
Performer: Research Triangle Institute; Research Triangle Park, NC
PIC ID: 8296

How Easy Is It for Health Researchers, College Students, and the General Public to Use the Proposed Stem Cell Information Web Site?

Summary:

A usability test was conducted of the proposed National Institutes of Health (NIH) Stem Cell Information Web site. The three target audiences for information provided through the Web site, each represented by five participants in the study, were health researchers, college students, and the general public. The study found that health researchers were the most proficient at finding information on the site, possibly due to having more specific needs and greater familiarity with the subject. College students and the general public quickly found information for which there were on-screen links but they were unsure where to look for information on other topics. Recommendations for improving the site included: expanding the section termed "Fast Answers" and associating it with the frequently asked questions (FAQs) section; renaming, simplifying or eliminating misleading navigation links; further developing different navigation styles for general public/college students versus health researchers, and avoiding making graphical images links unless clearly marked.

Report Title: Analysis of NIH Stem Cell Web Site Usability Testing http://aspe.hhs.gov/pic/fullreports/06/8351.pdf
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Blessing, Patricia, 301-496-7243
Performer: Eagle Design & Management; Bethesda, MD
PIC ID: 8351

Can a Curriculum to Train Older Adults to Access Health information on the Web Be Both Senior Friendly and Trainer Friendly?

Summary:

This study examined whether a curriculum to train older adults to access health information on the Internet was easy for older students to grasp and easy for trainers to teach. The study provided recommendations to the National Institute on Aging on ways to improve the training program and distribute it. Researchers collected data on content, instructions, format, usage, and content distribution. The study consisted of review by a focus group of original lessons, pilot testing the lessons in computer classes with seniors, mid-study and end-of-study curriculum review sessions, and an online survey and lesson review. The online usage and distribution survey confirmed high interest by potential users in the training program. It also showed an inclination by recipients to use the program when it was distributed. Usage may also be affected by availability of train-the-trainer materials and by how easily the materials can be downloaded. The results of the study were used to improve the content, format, and layout of the curriculum. The preferred method of distribution was to download from online sources.

Report Title: The National Institute on Aging's "Guide for Training Older Adults to Access Health Information on the Web" http://aspe.hhs.gov/pic/fullreports/06/8397.htm
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Dailey, Stephanie, 301-496-1752
Performer: PS International; Arnold, MD
PIC ID: 8397

How Can Information on Clinical Trials Most Effectively Be Provided to Spanish-Speaking Users?

Summary:

The National Library of Medicine (NLM) maintains a Web site with information about clinical trials, http://clinicaltrials.gov/. This study assessed how well a Spanish language prototype of ClinicalTrials.gov was received by Spanish-speaking users. The study included practical usability testing and interviews. The study was performed for NLM's Consumer Health Research Program. This program's objective is to develop tools that assess consumer health texts in order to better match documents to consumers' abilities. The study included a series of task scenarios to assess whether users were able to navigate easily through the site and comprehend the content of the site.

The study determined overall how satisfied users were with the clinical trials Spanish-language Web site. It also included one-on-one interviews with 16 participants from the eight target groups. Participants liked that the information provided was available in Spanish, was short and was easy to comprehend. Most participants found the information helpful but not sufficient to completely understand the purposes of individual clinical trials or what they entailed. Participants found the site easy to navigate and liked that the search engine was well located and easy to find. Participants with lower literacy levels had more difficulty. Participants liked that the home page provided several options to finding information. Several participants commented that the site and the information were well translated.

Report Title: Process Evaluation to Assess the Usefulness and Effectiveness of Spanish Language ClinicalTrials.gov Prototype for Spanish-Speaking Consumers, and of Abbreviated Displays of Records http://aspe.hhs.gov/pic/fullreports/06/8398.pdf
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Tse, Tony, 301-402-7789
Performer: Bearing Point, Inc.; McLean, VA
PIC ID: 8398

How Is Money From the National Institute on Aging Being Used by the National Center for Health Statistics and the U.S. Census Bureau?

Summary:

Evaluators reviewed the interagency agreements that the Behavioral and Social Research Program of the National Institute on Aging (NIA) has with two Federal agencies. The objective was to revitalize and focus these activities and identify new goals and objectives for the next five years. Agreements with the US Census Bureau (Census) and the National Center for Health Statistics (NCHS) have supported publications, dataset development and dissemination, data enclaves (data sites with restricted access), and linkages of datasets with administrative data. Background materials and data were reviewed and a site visit to each agency conducted.

The review committee recommended that NIA support to aging-related activities at NCHS and the Census Bureau continue, with modifications, and that peer review of these agreements occur every five years. The committee provided specific recommendations for the interagency agreements with each agency. For both agreements, the committee recommended that the agencies increase communications, seek additional ways to cooperate on new ventures that enhance the awareness of and interest in aging-related data, and seek to increase use of data among the research community. For the agreement between NIA and NCHS, the committee recommended that the parties: (1) review status of ongoing projects, clarify expectations, and plan future activities; (2) define more carefully the target audiences for all products and programs; (3) consider producing life tables by cohort and education; and (4) develop proposals that make innovative and intellectually exciting use of data. For the agreement between NIH and Census, the committee recommended that the parties: (1) develop clear accountability and mutually agreeable timetables for delivery of products; (2) enhance outreach to scholars and the research community; (3) develop mechanisms to encourage young researchers to use Census datasets and to facilitate exploratory research; (4) take advantage of approaches that would permit access to data sets while maintaining confidentiality; (5) convene experts to examine linkage studies and offer suggestions for improving efficiency of these activities; and (6) continue to expand interaction with the media.

Report Title: Review of NIA-Funded Activities at the National Center for Health Statistics and at the U.S. Census Bureau -- Review Committee Reports http://aspe.hhs.gov/pic/fullreports/06/8431.htm
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Hamilton, Elizabeth, 301-402-4582
Performer: Rose Li and Associates; Bethesda, MD
PIC ID: 8431

How Should the National Library of Medicine Evaluate Its Exhibit Program?

Summary:

This project developed recommendations and an evaluation plan the National Library of Medicine could use to assess the effectiveness of its conference exhibit program. The Library exhibits at over 100 conferences each year; this study is based on 40 conferences managed by one library division. Informing professionals, minority and underserved populations, and the public about its health information resources is an essential role of the Library. Researchers examined current conference materials, interviewed Library conference staff, and conducted a review of conference exhibit literature. This initial study concluded that future evaluation efforts should focus on research questions pertaining to two dimensions: conference selection and interactions with visitors to the exhibit booths. Business literature pointed out the importance of speaking one-on-one to as many people as possible at conference exhibits. Questions to be asked about conference attendance and exhibits include: what percentage of conference attendees are members of National Library of Medicine target audiences; what pre-conference promotional efforts are made; and what are the number and interests of booth visitors. Library staff is currently implementing elements of the evaluation plan.

Report Title: Evaluation of Exhibit Program; Evaluation of New Web Resources http://aspe.hhs.gov/pic/fullreports/06/8454.pdf
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Love, Cindy, 301-496-5306
Performer: Lockheed Martin Aspen Systems; Rockville, MD
PIC ID: 8454

How Can Educational Efforts to Prevent Noise-Induced Hearing Loss Be Strengthened?

Summary:

This study analyzed how information about noise-induced hearing loss is being made available to the public, how a program called "WISE EARS!" has contributed, and provided information for future campaign efforts for prevention. WISE EARS! is a campaign of national, regional, and local organizations, volunteers, and state and local government agencies. The coalition's objectives are to increase awareness and motivate audiences to take action by understanding the problem and its solutions. Researchers reviewed scientific literature and media coverage and examined WISE EARS! materials. Researchers analyzed publication distribution information, reviewed non-governmental programs related to noise-induced hearing loss, and interviewed coalition members. The study produced recommendations in three categories.

Program recommendations included giving the campaign higher priority within the National Institute on Deafness and Other Communication Disorders and conducting a comprehensive evaluation. Strategic recommendations included focusing on a specific target audience and selecting partners and delivery channels with the highest potential to attract and engage this audience. Researchers further recommended reaching additional audiences by supporting programs conducted by other organizations. Tactical recommendations included conducting more aggressive and responsive media outreach and making greater use of communication technology. Coalition members recommended that the Institute develop hearing loss-prevention materials for a more targeted audience. They also recommended focusing the campaign on youth, ages 9-14, and distributing materials more aggressively through the Web, mass media, and through partner organizations.

Report Title: WISE EARS!® Campaign: Coalition Member Interviews/ WISE EARS!® Campaign: Environmental Scan http://aspe.hhs.gov/pic/fullreports/06/8455.htm
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Blessing, Patricia, 301-496-7243
Performer: ORC Macro; Rockville, MD
PIC ID: 8455

How Can a Web Site Be Made Easier for the Public to Use?

Summary:

This study assessed the performance of the National Institute of Biomedical Imaging and Bioengineering Web site (http://www.nibib.nih.gov) by measuring audience satisfaction and ease-of-use. The Web site provides mission-related information to federal agency staff, extramural researchers, health professionals, educators, students, and patients. Researchers observed 16 federal agency staff, researchers, health professionals, educators, students and patients as they performed tasks on the web site, and recorded specific barriers to task performance. The contractor asked participants to rate their satisfaction with the site, its ease-of use and measured how quickly they were able to perform the tasks. The researchers used coding forms to tally the occurrence and frequency of repetitive issues and key usability problems as they arose.

Participants in this small sample found that the site was easy-to-use and provided valuable information. The contractor provided specific recommendations for enhancing the website and a list of prioritized action items. Suggested website improvements included: restructuring the footer to include the organization's mailing address, including an "About Us" section linked to from the home page, and re-writing content using more plain language.

Report Title: Result of Usability Testing of the NIBIB Redesigned Web Site http://aspe.hhs.gov/pic/fullreports/06/8463.pdf
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Kester, Mary Beth, 301-402-1465
Performer: Z-Tech; Rockville, MD
PIC ID: 8463

Did an Education Booklet for Survivors Increase Their Knowledge and Coping Skills Following Cancer Treatment?

Summary:

This evaluation examined whether reading a particular cancer treatment education booklet, "Facing Forward: Life After Cancer Treatment," increased patients' knowledge about post-cancer treatment issues and their ability to cope with these issues. The booklet, part of a series, was produced by the Office of Special Initiatives as a patient and caregiver resource for the six-month period after cancer treatment. Three surveys were given to patients at four cancer centers who had recently completed a course of treatment. Many respondents read the booklet and indicated that it was helpful, specifically for getting follow-up care, identifying common side effects, and better understanding feelings and changes in relationships. On the other hand, respondents reported that they did not know how to handle certain symptoms or side effects, even though the booklet specifically mentioned ways to relieve them. A larger percentage of the patients reported an increase in knowledge of issues than confidence in handling those issues. Also, a larger percentage of the patients reported increased confidence in handling issues than an intention to act upon those issues. Although an initial hypothesis of the study was that the impact of reading the booklet would diminish over time, there was no general indication that this happened. Study findings were being used to revise the booklet.

Report Title: Evaluation of the "Facing Forward: Life After Cancer Treatment" Booklet http://aspe.hhs.gov/pic/fullreports/06/8358.pdf
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Solomon, Felicia, 301-594-8989
Performer: ORC Macro; Calverton, MD
PIC ID: 8358

Objective 5.4: Improve consumer and patient protections

What Role Does the Physician Discipline Function of State Medical Boards Play in Assessing Medical Quality?

Summary:

The study sought to describe how State Medical Boards generally addressed physician discipline and sought to identify innovative practices that Boards believe improve their functioning and enhance patient safety. The study included a literature review and development of a concept paper that described the physician discipline process, discussions with national experts, a technical advisory group, and cross-state analysis of Board structure and disciplinary performance data. Further information on Board structure and operations came from case study interactions in six states. Finding ways to assess medical boards' performance is important. States have given Boards a key role in safeguarding medical quality and patient safety. Medical litigation cases have also drawn attention to the importance of Medical Boards. Other private and public efforts also attempt to protect quality, but all depend on Medical Boards to address practitioners who don't meet minimum standards.

The study found that discipline of physicians beyond initial licensure was the Boards' main activity. Discipline was largely complaint-driven, meaning that Boards proactively began few cases. Almost 5% of complaints resulted in some level of sanction, which might be negotiated before a hearing or imposed after one. Board managers were asked what outcomes they intended to achieve and how they measured them. Only two performance measures were noted in all sites. The first was the number of disciplinary sanctions imposed. The second was timeliness of complaint resolution--and avoidance of a lengthy backlog of open cases. All stages of discipline cases were costly, particularly for cases concerning quality of medical care. Managers sought to improve Board effectiveness by: targeting resources more effectively, identifying the important cases from first contact, improving access to medical expertise, and taking advantage of information technology. Insufficient funds were cited by all Board managers. Many Board members and managers also wanted to do more for safety rather than merely react to complaints. For policy research, the next steps may be to more rigorously study existing state interventions or conduct demonstrations to see how well some of them may work in other states.

Report Title: State Discipline of Physicians: Assessing State Medical Boards through Case Studies http://aspe.hhs.gov/daltcp/reports/2006/stdiscp.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Bergofsky, Linda, 202-690-6443
Performer: Urban Institute; Washington, DC
PIC ID: 8344

What Information Do Health Professionals Need in Order to Talk Clearly with Their Patients About the Risks of Heart Defibrillators?

Summary:

This focus group study explored what health professionals believe they need to know in order to be able to talk to their patients about the risks of implantable heart defibrillators and pacemakers. Four focus groups, two of doctors and two of allied health professionals, were gathered from attendees of the 2006 annual meeting of the Heart Rhythm Society. Participants said that a manufacturer's recall of a device typically meant serious possible consequences to their patient's health, possibly having to remove the device and return it to the manufacturer. Focus group members said that factors they considered when deciding whether to remove a recalled device included the patient's dependence on the device and whether the risks of removing it were greater than the risks of leaving it in. Often, recall information would first appear in popular media. The health professionals believed that this would undermine the patient's confidence in the doctor or cause them undue alarm. Some focus group participants felt that the Food and Drug Administration should regulate the process of the producers' communications about device recall. Some felt that written information explaining existing and potential problems was very important. However, it is a challenge to find simple and digestible language they could use with their patients. The participants reported that they preferred receiving information about recalls via e-mail, express mail, or fax.

Report Title: Physicians' and Allied Health Professionals' Perceptions about Communicating Implantable Cardiac Defibrillator and Pacemaker Risks to Patients: Focus Group Research http://aspe.hhs.gov/pic/fullreports/06/8311.pdf
Agency Sponsor: FDA, Food and Drug Administration
Federal Contact: Bobolis, Mary A., 301-827-5253
Performer: Olchak Market Research; Greenbelt, MD
PIC ID: 8311

Objective 5.5: Accelerate the development and use of an electronic health information infrastructure

What Role Should the Federal Government Play in Developing Electronic Personal Health Records?

Summary:

This study explored the federal government's role in the emerging technology of personal health records. The study team interviewed 15 experts in key positions in the federal government and health care and health information technology fields. The team drew from presentations at two hearings on personal health record models. There was general agreement that the federal government should facilitate standards-based approaches to health data collection and exchange and finance and help disseminate findings from a wide range of experiments to find the most successful public health record models. The government could be most effective by avoiding two possible pitfalls: sponsoring a superficial, one-size-fits-all public health record, and allowing many agencies to offer independent pubic health records with little coordination or strategic vision. The real test for public health records would be whether they made it easier for ordinary people to engage more actively in maintaining their own health and health care with better communication, improved safety, enriched knowledge and confidence, and trusted safeguards of their privacy. The study team concluded that the federal government should take a lead role as personal health records evolve in order to help promote this new technology becoming a trusted, widely used tool.

Report Title: Identifying Appropriate Federal Roles in the Development of Electronic Personal Health Records: Results of a Key Informant Process http://odphp.osophs.dhhs.gov/projects/PHRecords/default.htm
Agency Sponsor: OPHS-ODPHP, Office of Disease Prevention and Health Promotion
Federal Contact: Harris, Linda, 240-453-8262
Performer: David Lansky, Susan Kanaan, Josh Lemieux; working under auspices of FACCT, Foundation for Accountability; Portland, OR
PIC ID: 8467

What Can We Learn from Employer-Provided Health Insurance Data?

Summary:

This study assessed the value of using data from the Department of Labor's Pension and Welfare Benefits Administration for determining whether employees received employer-provided health care. The study used administrative records from benefit plans filed annually by firms (on IRS Form 5500, that provides information on employee benefit plans, including data on different types of health insurance plans) when merged with individual and employer data from the Census Bureau's Longitudinal Employer Household Dynamics data program. The study explored what proportion of the health benefit plans on file in the Form 5500 data can accurately be linked to specific employers. Further, the study examined how the proportion of firms offering health plans observed in these data coincides with what was observed in other data sources. The study looked at individual workers and whether their employment by firms that had and had not filed health plans was consistent with their self-reporting of access to health benefits as reported in the Current Population Survey and the Survey of Income and Program Participation. Finally, the study examined the types of questions that these data could inform, and identified some descriptive findings on changes in employer health benefits that illustrate what the first steps of such a research agenda might look like.

The study found that while the data are not well-suited to examine the proportion of workers receiving employer-sponsored health care or the proportion of firms offering health care, they could contribute significantly to our understanding of the types of firms that offer health plans, as well as how offers and cost-sharing change over time within specific firms. Several types of research questions could be pursued with the data systems identified. How much of the increases in employer health care costs are passed along to employees? How do such increases affect employee participation in employer-offered health insurance plans? How does turbulence in an economic sector affect the kinds of benefits being offered by businesses in that sector? Do reports by workers accurately reflect changes in employer offerings?

Report Title: Employer Provided Health Insurance: What can be Learned from the 5500 Data? http://aspe.hhs.gov/hsp/05/admin-data-emp-ins/index.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Hauan, Susan, 202-690-8698
Performer: U.S. Census Bureau; Washington, DC
PIC ID: 8289

What Features and Capabilities of Electronic Medical Records Best Serve Doctors in Small Practices?

Summary:

The study examined what physicians in small practices (nine or fewer members) believe are the elements of electronic medical records (EMRs) that will improve the quality of their health care and provide them with the greatest financial return. The study also sought to identify the most efficient, effective and successful implementation strategies for EMRs in these small practice offices. This, the first part of a two part study, focused on understanding the factors influencing adoption of electronic health records in small practices and developing an explanatory model incorporating these factors. Electronic medical records are viewed as a means to achieve improved health care quality and reduced costs. A number of public and private sector initiatives focus on promoting the adoption of health information technology such as community- and physician-focused initiatives. Despite these initiatives, the adoption of EMRs has been limited. Adoption rates vary widely across care settings; for instance, rates in ambulatory settings range between 15 and 18 percent. With approximately 75 percent of physician practices employing fewer than nine physicians, such low adoption rates among small practices does not bode well for a national goal of achieving broad use of electronic medical records. Low rates of EMR adoption have been attributed to a variety of forces including: misaligned financial incentives, lack of standard system applications, and the high turnover of health information technology vendors. Few studies have examined, at a microeconomic level, the various economic and non-economic factors that promote or deter EMR adoption in small practice settings. The analysis revealed that physicians are motivated to adopt electronic medial records by both financial and non-financial factors. Physicians care about quality and patient safety in addition to their income and leisure.

Report Title: Assessing the Economics of Electronic Medical Record Adoption and Successful Implementation in Physician Small Practice Settings http://aspe.hhs.gov/sp/07/adoption/index.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Burke-Bebee, Suzie, 202-401-8266
Performer: Moshman Associates, Inc.; Bethesda, MD
PIC ID: 8475

How Can Barriers to Interoperable Health Information Technology in Post-Acute and Long-Term Care Be Reduced; Can Patient Data Be Made to Comply with Health Information Technology Standards?

Summary:

This study provided recommendations regarding the use of health information technology content and messaging standards that could be applied to existing and emerging health assessment requirements (such as the assessments that nursing homes, home health agencies, and in-patient rehabilitation facilities are required to complete). Health information technology is expected to improve the quality of health care and patient safety, decrease costs, and increase efficiencies in health care delivery. To facilitate the effective and efficient use of health information technology generally, voluntary consensus standards are gradually being developed and adopted by a wide range of public and private individuals and organizations. To most efficiently support health information exchange and re-use, health information technology solutions must be "interoperable." Interoperability means the ability of different health information technology systems, software, and networks to exchange and reuse data accurately, effectively, securely, and consistently in various settings, including preserving the meaning of the data. Nursing homes, home health agencies, and in-patient rehabilitation providers face challenges in acquiring and implementing such interoperable health information technology solutions. Research has found that the health information technology products used by these providers are not interoperable with electronic health records systems used in other health delivery settings (for example, acute care hospitals and physician practices). Federal requirements for patient assessments are a significant barrier to interoperability. These assessments are not linked with health information technology standards that would support standardized exchange and reuse of assessment content. Federal law and Centers for Medicare and Medicaid Services' regulations require that Medicare and Medicaid certified nursing home providers complete, and electronically submit, patient-specific health and functional assessment information several times during each patient's stay. In nursing homes, the required assessment is the "Minimum Data Set" (MDS). Medicare, and sometimes Medicaid, uses this data to determine provider payments, report on various quality activities, and for other purposes.

The study , conducted in collaboration with the Centers for Medicare and Medicaid Services, examined how to apply health information technology standards to the Minimum Data Set. The study identified possible matches of standardized vocabulary terms and concepts with the Minimum Data Set concepts. The study also represented the Minimum Data Set in a format known as "Logical Observation Identifiers Names and Codes" (LOINC). The use of the LOINC format enables the standardized exchange using health information technology messaging standards of assessment results. The study constructed sample messages using data elements from the Minimum Data Set that had been represented in the LOINC format and linked standardized vocabulary content. The standards identified in this study were endorsed by the Consolidated Health Informatics (CHI) Initiative, one of the electronic government activities under the Office of Management and Budget and recommended for adoption and use by the National Committee for Vital and Health Statistics.

Report Title: Making the Minimum Data Set Compliant with Health Information Technology Standards http://aspe.hhs.gov/daltcp/reports/2006/MDS-HIT.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Harvell, Jennie, 202-690-6443
Performer: Apelon; Alameda, CA
PIC ID: 8338

Can Accessing MEDLINE Via Handheld Computers Be Useful in Answering Clinical Questions for Physicians?

Summary:

This study assessed the role of MEDLINE when accessed via wireless handheld computers in real clinical scenarios. MEDLINE is a database of over 15 million references to articles published in more than 5,000 current biomedical journals from the United States and over 80 foreign countries. It is an indexing service for research in medicine and related fields. "MD on Tap" is a software application enabling individuals, using personal digital assistants or smartphones to search and retrieve MEDLINE citations in real time directly through a wireless connection to the Internet. MEDLINE and MD on Tap are sponsored by the National Library of Medicine. For this study medical residents accompanied teams on teaching rounds in intensive care units and general medical wards of teaching hospitals, recording clinical questions in real clinical scenarios and searching MEDLINE for answers. The study found MEDLINE to be useful in answering clinical questions and found that MD on Tap was an effective interface to MEDLINE in clinical settings. The system was easy to use and allowed clinicians to quickly find relevant citations. The study analyzed the search and retrieval results from 363 clinical questions asked by members of clinical teams during 77 daily rounds. Answers to 246 (68%) questions were found in MEDLINE by five physicians during rounds using the handheld devices. Strategies that significantly (at the p<.05 statistical confidence level) facilitated timely retrieval of results included using spell check, and using three to four search terms. Possible improvements identified by the study included expanded automatic topical and evidence-level clustering and providing patient outcome information along with the titles of the retrieved articles.

Report Title: Evaluating MD on Tap and MEDLINE via Handheld Computers as a Resource for Mobile Clinicians. http://aspe.hhs.gov/pic/fullreports/06/8393.htm
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Hauser, Susan, 301-435-3209
Performer: University Clinical, Education and Research Associates; Honolulu, HI
PIC ID: 8393

Goal 6: Improve the economic and social well-being of individuals, families, and communities, especially those most in need

Objective 6.1: Increase the proportion of low-income individuals and families, including those receiving welfare, who improve their economic condition

Is Substance Abuse a Barrier to Employment and How Effective Are Substance Abuse Treatment Interventions for Welfare Recipients?

Summary:

Research findings on substance abuse disorders among welfare recipients were examined. Reviewers offered suggestions for how past research findings could inform future policy and research questions.

Research on four questions was examined: (1) What is the prevalence of substance use and substance use disorders among Temporary Assistance to Needy Families (TANF) recipients? (2) To what extent are substance use disorders and co-occurring problems a barrier to employability? (3) Can screening strategies in welfare settings successfully identify and refer clients to substance abuse treatment? (4) What types of services do these clients need to attain self-sufficiency?

The review found that 8 to 20 percent of women on TANF have a substance abuse problem that probably interferes with their functioning. Women with substance abuse disorders experience substantially more barriers to employment and are less likely to become employed and more likely to lose welfare benefits as a result of a sanction. Screening procedures are being used in many States to identify individuals receiving Temporary Assistance to Needy Families benefits who have substance use disorders. However, such screening identifies only a minority of those with substance use disorders. Evidence from a rigorous random assignment study indicated that more intensive interventions could yield better treatment participation rates and substance use outcomes.

Report Title: Welfare Reform and Substance Abuse Treatment for Welfare http://pubs.niaaa.nih.gov/publications/arh291/63-67.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Radel, Laura, 202-690-5938
Performer: National Center on Addiction and Substance Abuse, Columbia University; New York, NY
PIC ID: 8478

What Lessons Can Be Learned from Evaluating a Community College Approach Designed to Improve Employment Prospects of Welfare Recipients Already in the Workforce?

Summary:

This demonstration project tested the impact of a particular education program intended to improve the job prospects of welfare recipients who worked at least twenty hours each week. The education program provided the welfare recipients had been designed to meet their special needs for flexible schedules for an employed population of recipients and for remedial education. Those participating in the program were expected to maintain their employment at a level of at least 20 hours per week. The education program and evaluation was conducted in Riverside, California. It was designed by and implemented on the campus of Riverside Community College. The core pieces of the program, lasting twenty-four weeks, consisted of classes in English, math, reading , office computer software, and career guidance. The core courses emphasized basic communication and computer skills critical for work. After graduating from the core program, students were encouraged to enroll in a variety of courses at the Riverside Community College designed to help them enter or advance in specific careers. Evaluation of the program was carried out by randomly assigning eligible recipients who were interested in the program to either a treatment or a control group. Only subjects in the treatment group could receive the core training. Recipients in the control group were encouraged to pursue some other form of education or training.

Recruiting students and keeping them in the program proved to be very difficult. Those in the treatment group participated more in education and training than those in the control group. The treatment group earned more college credits. However, a large percentage of those in the treatment group either did not show up after enrolling, or if they did, failed to complete the core program. Of those that did complete the core program, virtually none enrolled in the courses aimed at specific jobs that were to follow it. The program did not produce better economic or welfare outcomes. Rather, outcomes were less favorable for treatment than for control group members. For each of the three years after assignment to the treatment group, there was a statistically significant but negative impact on earnings. These participants earned less and received more welfare benefits than the control group members, though for welfare benefits the difference was statistically significant in only the third year. However, this program tested only one particular education initiative aimed at improving the jobs prospects of recipients already in the work force. Whether other approaches might work better would need to be the subject of further studies.

Report Title: College as a Job Advancement Strategy: Final Report on the New Visions Self-Sufficiency and Lifelong Learning Project http://www.acf.dhhs.gov/programs/opre/welfare_employ/new_visions/reports/clg_job_adv/clg_title.html
Agency Sponsor: ACF, Administration on Children and Families
Federal Contact: Sternbach, Leonard, 415-437-7671
Performer: Abt Associates Inc.; Cambridge, MA
PIC ID: 8275

What Lessons Can Be Learned by Combining Evaluation Data from Several Welfare-to-Work Programs?

Summary:

This study analyzed data from 30 welfare-to-work research demonstrations implemented during the 1980s and 1990s. The demonstrations had sought to improve outcomes related to moving recipients of welfare toward greater self-sufficiency. The demonstration evaluations measured progress of welfare recipients toward increased employment and earnings and toward a reduction in receipt of cash welfare benefits. This study analyzed how aspects of the demonstrations influenced meeting these self-sufficiency goals. This study also determined which aspects of the demonstrations had positive impacts on child well-being outcomes. All demonstrations included in the study had been evaluated using random assignment methods. Recipient participation was mandatory in the welfare-to-work programs in all but four of the demonstrations (thus, this study said little about voluntary programs).

Aspects of the demonstrations positively impacting employment outcomes included: job search activities, time limits and sanctions on recipients for not meeting various work requirements. These demonstrations tended to work better in strong labor markets. There was some evidence that positive impacts were stronger for recipients considered disadvantaged. Disadvantaged persons include those that are long term recipients and that have little recent work history. Evidence was weak regarding the affect of education or work experience on employment outcomes. The study also found small impacts, with regard to receipt of welfare benefits as measured by a reduction in the amount of cash benefits or a reduction in the time spent receiving benefits. In this regard, sanctions were found to have had positive affects, and increases in earned income disregards were found to have had negative impacts. Since, for the most part, the demonstrations included in this study were not focused on child outcomes, the data regarding impacts on children were less extensive and less definitive. The impacts these demonstrations had on emotional/behavioral problems in children were small and varied greatly among the demonstrations. To the extent data were available, however, the impacts on emotional/behavioral problems were found to be less positive for school-age children than for younger children or for adolescents. The affect of sanctions, participation of adults in basic education, and participation in unpaid work appeared to be positive while higher earned income disregards (income not counted for determining cash eligibility) and time limits was associated with negative impacts on children.

Report Title: Report on a Meta-analysis of Welfare-to-Work Programs http://www.acf.hhs.gov/programs/opre/welfare_employ/meta_analysis/reports/report_wtw/meta_title.html
Agency Sponsor: ACF, Administration on Children and Families
Federal Contact: Sternbach, Leonard, 415-437-7671
Performer: University of Maryland; College Park, MD
PIC ID: 8276

How Can Former Welfare Recipients Be Successfully Encouraged to Obtain and Retain Employment, and Advance to Better Paying Jobs?

Summary:

This study evaluated the effectiveness of South Carolina's "Move Up" program. Move Up sought to help former welfare recipients get jobs, work steadily, and advance in the labor market. Much is known about helping welfare recipients find jobs but not about helping them keep jobs or advance. Move Up (one of 15 related programs being tested across the country) operated during 2001-2005 in six rural counties. It sought to contact and help individuals who had left welfare between 1997 and 2000. Nonworking participants received help finding a job; working participants received help staying in their job or moving up. The core of the program was one-on-one case management. Staff provided or connected participants with services, including job search assistance, vocational training, and support services. The program provided some financial incentives to encourage participation and employment. For the evaluation, a random assignment research design was used; eligible individuals were assigned, through a lottery-like process, either to a program group, whose members were recruited for the program, or to a control group, whose members were not recruited or eligible for program services but who could use other services in the community. The program's effects were estimated by comparing how the two groups fared over time. Staff located about three-fourths of the program group, most of whom had been off welfare for several years when they entered the study. Staff had to persuade individuals who were not required to do so to participate. Often they did not want or need the services. About half of the program group engaged in services during the year after they entered the study, few of them intensively.

The target population increased its participation in some employment-related services, such as vocational training, but only modestly. Overall, Move Up had little effect on employment rates, earnings, employment retention, or advancement. During the year after entering the study, members of the program and control groups had similar employment patterns. It appears unlikely that effects will emerge during the second year after study entry or later. The program did not affect welfare or food stamp receipt or income overall. The program did have positive effects on employment for three subgroups of sample members: those who become unemployed shortly before entering the study, those who left welfare less than two and a half years before entering the study, and those who left welfare because of a sanction or the state's time limit on benefit receipt. One county's program produced positive effects on employment; the other five did not. The research contractor will continue to track employment outcomes for the study's participants. Many the individuals in the study remain poor and in need of support.

Report Title: The Employment Retention and Advancement Project: Results from the South Carolina ERA Site http://www.acf.hhs.gov/programs/opre/welfare_employ/employ_retention/reports/era_sc/era_sc_toc.html
Agency Sponsor: ACF, Administration on Children and Families
Federal Contact: Richards, Patrice, 202-205-8324
Performer: MDRC; New York, NY
PIC ID: 8279

How Can States Encourage Welfare Applicants, Recipients, and Those Leaving Welfare, to Seek, Retain and Advance in Employment?

Summary:

This study assessed implementation and two-year follow-up effects of a job placement, employment retention, and advancement program for applicants and recipients in the Temporary Assistance for Needy Families (TANF) program in Texas. The Texas program is part of the national Employment Retention and Advancement (ERA) demonstration project, which is testing 15 such programs across the country. To encourage employment retention and advancement among working persons leaving welfare, the Texas program provided job search assistance and a monthly stipend of $200. The program was evaluated in Corpus Christi, Fort Worth, and Houston starting. The evaluation used random assignment to assign eligible individuals to a program group whose members participated in the program or to a control group, whose members participated in Texas’ standard welfare-to-work program (called “Choices”). The control group’s outcomes showed what would have happened in the absence of the ERA program. The ERA program was well implemented in Corpus Chris ti but experienced operational difficulties in the other two sites. Across all sites, the control group participated in a relatively strong welfare-to-work program.

The study findings reinforced the view that promoting employment retention and advancement among welfare recipients presents challenging implementation issues. The Texas ERA program did not produce consistent or large effects on employment and retention outcomes during the first two years of the study period. Program and control group results were similar during the pre-employment phase but had larger treatment differences during the post-hiring phase, primarily due to the stipend. A significant effort was needed to market the stipend, and program staff increasingly made a good-faith effort to do so. However, this effort may have been insufficient; people did not become eligible for the stipend until the expiration of a four-month period during which their earnings were not counted, in accord with “earned income disregard” welfare rules. Among those assigned to ERA, the percentage of individuals receiving stipends ranged from 20 to 30 percent at the different sites. Among those who found jobs and received the entire earned income disregard, estimated stipend receipt rates ranged from 40 to 55 percent. At one site, there were a few modest impacts on employment and retention that were concentrated among those who entered the program during the early phases of the study period; the extra income from the stipend was enough to generate a statistically significant effect on income. At another site, the program’s impacts were on initial employment rather than on employment retention. Researchers will continue to track employment outcomes for the study’s participants.

Report Title: The Employment Retention and Advancement Project: Results from the Texas ERA Site http://www.acf.hhs.gov/programs/opre/welfare_employ/employ_retention/reports/results_texas/results_tx_title.html
Agency Sponsor: ACF, Administration on Children and Families
Federal Contact: Richards, Patrice, 202-205-8324
Performer: Manpower Demonstration Research Corporation; New York, NY
PIC ID: 8281

How Can States Implement Healthy-Marriage Programs for Unmarried Couples with Children?

Summary:

This study is the first installment of the Building Strong Families (BSF) Demonstration and Evaluation. BSF is a multi-year, multi-site test of voluntary, marriage and relationship education programs for low-income unmarried persons over 18 who are expecting, or have just had, a child and who are not involved in domestic violence. The study will provide important new information about whether marriage and relationship education and links to other needed support services can improve relationship/marital quality, relationship/marital stability and child well-being for economically disadvantaged couples. The demonstrations are being conducted in sites in Florida, Georgia, Indiana, Louisiana, Maryland, Oklahoma, and Texas. Using a random assignment evaluation design, the study will track and assess both the short (15 months) and longer-term (36 months) effectiveness of the BSF programs. Subsequent study components will examine the experiences of agencies involved in setting up and running the programs, as well as the experiences and opinions of couples. During the initial pilot phase, sites successfully enrolled couples both during pregnancy and after delivery of their child.

This initial study phase found that certain characteristics of host agencies and staff may be particularly helpful. These characteristics include: the strength of agency commitment to the importance of healthy marriage; program staff's organizational experience in delivering services in a group format; use of both male and female staff and teams; and employing staff with cultural backgrounds similar to their participants. For pilot sites, the maternal health care system was a major and efficient recruitment source for the target population of pregnant and new parents. The early data indicate the challenge of maintaining participation by couples who are pregnant or have a new baby in a program that requires ongoing weekly attendance over several months. An early lesson was that although there were significant drop-offs in attendance, the overall pattern was one of steady but intermittent participation. Most often, couples who began attending returned to group sessions after missing some sessions rather than dropping out entirely. Overall, couples responded positively to the programs, valued the group instruction format and learning from other couples' experiences, and demonstrated a basic understanding of what they had been taught.

Report Title: Implementing Healthy Marriage Programs for Unmarried Couples with Children: Early Lessons from the Building Strong Families Project. http://www.acf.hhs.gov/programs/opre/strengthen/build_fam/reports/implementing_healthy/impl_title.html
Agency Sponsor: ACF, Administration on Children and Families
Federal Contact: Campbell, Nancye, 202-401-5760
Performer: Mathematica Policy Research, Inc.; Princeton, NJ
PIC ID: 8282

What Might Happen If States Allow Current and Former Welfare Recipients to Keep More of Their Child Support Money?

Summary:

Predictions about current and former welfare recipients' receipt of child support under varying conditions were made using historical information about their household income. The analysis was carried out in order to provide States with cost and benefit data about different ways to increase child support distributions to both groups of welfare recipients. The analysis provided a general examination of possible effects of several types of state policy options similar to options that were recently made available to states through the Deficit Reduction Act of 2005. The analysis examined effects of changes to the amount of child support money states gave welfare recipients and former recipients rather than retaining it (that is, changes to the "pass-through" amounts). The analysis assumed that this additional child support money would not count (would be disregarded) for purposes of determining a family's eligibility for cash and other support. The study estimated the effects of program variables, such as changes in other public benefit payments and administrative costs, and some individual variables, such as non-custodial parents' payment of child support, to determine potential effects of state policy changes on family income, the poverty gap, self-sufficiency, and government costs.

There were several major findings. More generous child support pass-through and disregard policies might have several effects. Average annual cash income for welfare families and former welfare families who have child support collection made on their behalf would increase by up to 10%. The increase in average annual cash income for such families would be greater in states that do not currently have the pass-through. About 2/3 of the added income would reduce the poverty gap. The remaining 1/3 of the added income would replace benefits like food stamp and housing subsidies. A "behavioral response" to the more generous pass-through and disregard would increase the number of families with a child support collection. This would further reduce the poverty gap. Government costs would increase, but not as much as the net cash to families if there is a behavioral response to the policy changes.

Report Title: Benefits and Costs of Increased Child Support Distribution to Current and Former Welfare Recipients http://aspe.hhs.gov/hsp/05/cs-dist-TANF/index.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Burnszynski, Jennifer, 202-690-8651
Performer: The Urban Institute; Washington, DC
PIC ID: 8292

Objective 6.2: Increase the proportion of older Americans who stay active and healthy

What Is the Role for Adult Day Services in the Health Care and Long-Term Care Systems?

Summary:

Adult day services are non-residential community services provided outside an individual's home. This study examined the role such services play in health care and long-term care. The study identified operational and regulatory issues facing adult day service providers, and provided information to guide future research and policy analysis. Although promoted as community-based service for older persons, little was known about the provision, use, or outcomes of adult day services. Adult day services provide therapeutic activities, assistance with activities of daily living and meet caregivers' needs for respite care or to enable them to work. States are interested in the potential of adult day services to reduce health care costs, and prevent or delay nursing home placement.

Adult day services are relatively inexpensive compared to home care or nursing home care. However, the study identified a number of barriers to the use of these services. High cost or the lack of transportation is a major impediment to the use of adult day services. The study also found that adult day service providers have difficulty covering their costs solely through private payments and public program reimbursements. Nearly all providers receive a significant portion of their operating revenue through Medicaid or other public funding sources, but these reimbursements generally do not cover providers' costs. Many programs rely on volunteers, in-kind contributions and charitable donations to subsidize their operations.

Report Title: Regulatory Review of Adult Day Services: Final Report http://aspe.hhs.gov/daltcp/reports/2006/keyADS.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Rosenoff, Emily, 202-690-6443
Performer: Research Triangle Institute; Research Triangle Park, NC
PIC ID: 8343

How Do States Regulate Adult Day Care and Adult Health Care Services?

Summary:

This study examined State approaches to regulating adult day care and adult day health care, generically called adult day services. Adult day services are non-residential community services provided outside the home. The purposes of the study were to learn what role such services can and do play in health care and long-term care, to identify operational and regulatory issues facing adult day service providers, and to guide future research and policy analysis. Although promoted as community-based service for older persons, little is known about the provision, use, or outcomes of adult day services. States either license or certify adult day care providers. This study found that states vary in how they regulate these services. A majority require inspections annually and several have unannounced visits. There may be additional Medicaid requirements for Medicaid-funded adult day service facilities. Most states have minimum mandatory staff-to-participant ratios. Nearly two-thirds require one staff for every six to eight persons. States are interested in the potential of adult day services to reduce health care costs, and prevent or delay nursing home placement.

Report Title: Regulatory Review of Adult Day Services http://aspe.hhs.gov/daltcp/reports/adultday.pdf
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Rosenoff, Emily, 202-690-6443
Performer: Research Triangle Institute; Research Triangle Park, NC
PIC ID: 8343.1

How Can the Online "My Family Health Portrait" Tool Be Improved?

Summary:

The United States Surgeon General has an online program, "My Family Health Portrait" (from http://www.hhs.gov/familyhistory/) that helps people record their family health history and display the information visually. This study assessed how well the Web site functioned. The study included usability testing with an early version of the software, an assessment of compliance with Federal requirements for product accessibility for people with disabilities, and a post-launch survey of users. The software tool was developed by the Communications and Public Liaison Branch of the National Human Genome Research Institute. The study identified changes that could improve the software program.

The study concluded that a clearer explanation of its purpose was needed for the tool; for instance, some participants expected a risk assessment at the end. The tool also needed more descriptive navigation support, particularly once participants completed the initial process of entering health history information for their immediate family. Usability test participants made a number of specific recommendations: 1) add a "don't know" option for the deceased status of relatives; 2) during the initial process of entering immediate family members, make it possible to "go back" to family members previously entered in order to make changes; 3) make the order of relatives on different screens consistent; 4) fix the process for adding family members (aunts, uncles, nieces, nephews) so that it is not error-prone; 5) reconsider the request for weight and height information as it was a deterrent for some female participants; 6) make the language used to describe the "Save" process less technical and confusing and more intuitive; and 7) explain the purpose of the "Save My Health Information as Another Relative."

Report Title: National Family History Day http://aspe.hhs.gov/pic/fullreports/06/8364.doc
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Bartlett, Maggie, 301-594-0632
Performer: Lynn Baumeister; Silver Spring, MD
PIC ID: 8364

Objective 6.3: Increase the independence and quality of life of persons with disabilities, including those with long-term care needs

How Did Texas Modify Its Medicaid Program In Order to Have "Money Follow the Person" When Nursing Home Residents Chose to Live in the Community?

Summary:

This study examined the transition process, participant characteristics, and service utilization and costs for a program in Texas that allows institutionalized individuals receiving Medicaid to live in the community. In 2001, the state enacted legislation to promote choice, independence, and community integration for nursing home residents who expressed a desire to live in the community. In part, this was a response to the 1999 Supreme Court finding in Olmstead vs. L.C. that unnecessary segregation of individuals with disabilities in institutions may constitute discrimination based on disability. The state legislation ("Rider 37" and subsequently reauthorized as "Rider 28"), allows Medicaid funds spent for a person living in a nursing facility to be moved to the Texas community-based care budget when an individual elects to move back to the community. Any interested Medicaid nursing facility resident who meets medical and functional eligibility criteria for one of the state's community care waivers is eligible, regardless of his/her length of stay under Medicaid. The study of the first years of this program included qualitative field research and quantitative analysis of state administrative data. The study sought to understand the experience Texas had in implementing this state-initiated effort to rebalance the long-term care system.

 The initiative in Texas demonstrates that a money-follows-the-person component can be incorporated relatively easily into an existing Medicaid long-term care program. This far-reaching change in the state's approach to long-term care was accomplished through authorization of an accounting mechanism allowing funds to be shifted provisionally from the nursing facility component of the Medicaid budget to the community care component. Because community care programs were in place in Texas before this accounting change, the only change "money-follows-the-person" made to the Medicaid long-term care program was, in effect, to identify the nursing facility population as a separate eligibility category for community care programs and to make this eligibility group exempt from any upper limits ("caps") on enrollment in these programs. The comprehensive nature of the waiver benefits package allowed a diverse population of nursing facility residents to move back to residential settings that were more integrated in the community. The accounting provisions later became one component in a comprehensive funding policy, which defined a Medicaid long-term care program and incorporated lessons from the initiative transition process.

In all three regions where the research team conducted site visits state staff caseworkers and social workers at nursing facilities served key roles in identifying clients who were interested in or suitable for transition to the community. Local ombudsmen and advocates for the elderly and disabled were an important additional source of information, In some regions, they advocated for and assisted residents' transitions. As the initiative matured, word of mouth became a source of information for potential clients.

Report Title: Examination of Texas Rider 37: A Medicaid "Money Follows the Person" Long-Term Care Initiative http://aspe.hhs.gov/daltcp/reports/2006/TXltc.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Bergofsky, Linda, 202-690-6443
Performer: Urban Institute; Washington, DC
PIC ID: 8332

Objective 6.4: Improve the economic and social development of distressed communities

What Approaches Best Help Welfare Clients Living in Rural Places Move from Welfare to Work?

Summary:

The study examined the impact of an Illinois welfare-to-work program (“First Steps”) on the employment, self-sufficiency, and well-being of work-ready low-income people in rural Illinois, including mandatory Temporary Assistance for Needy Families and food stamps recipients, as well as low-income volunteers.  The program teamed up staff of the welfare agency with members of a regional community college; drew on the local connections of program staff to help clients gain access to services, resources, and jobs; and provided each case manager with very small caseloads.  This study, which constituted an interim evaluation conducted after the first 18 months of program operation, did not find evidence that the program had improved employment and earnings or that it had reduced welfare dependency up to that point in time.  At the 18-month follow-up, a little more than half of the sample participants were employed, and close to two-thirds lived in poverty.  Employer-focused job readiness and vocational training was never implemented.  The program cost $2,900 per participant.  Some clients received few services or insufficient ones, partly reflecting the limited training and support given case managers.  The program was unable to capitalize on the employer connections and job-training resources that its community college partner offered.  The Illinois program is one of two rural welfare-to-work demonstration projects being evaluated by ACF.  Results of a second study, focusing on another promising program in Nebraska, will be available in the online data base in late 2007.

 

Report Title: Paths to Work in Welfare Rural Places: Key Findings and Lessons from the Impact Evaluation of the Future Steps Rural Welfare-to-Work Program http://www.acf.hhs.gov/programs/opre/welfare_employ/rural_wtw/reports/paths_to_work/path_title.html
Agency Sponsor: ACF, Administration on Children and Families
Federal Contact: Dubinsky, Michael, 202-401-3442
Performer: Mathematica Policy Research, Inc.; Princeton, NJ
PIC ID: 8274

What Barriers to Accessing Discretionary Grants Do Native American Tribes and Organizations Face?

Summary:

This study gathered the views of program officials of the Department of Health and Human Services (HHS) and tribal representatives on program and regulatory barriers to American Indian, Alaska Native, and other Native American tribes and communities accessing HHS discretionary grants. The study identified for HHS the most significant barriers to and strategies for improving grants access for Native American tribes and organizations. The limited resources of Native American tribes and organizations were identified by both their representatives and by HHS program staff as a major barrier to such access. Limited resources make it difficult for Native American tribes and communities to: (1) learn about grant opportunities; (2) apply for grants that have matching requirements or limits on indirect costs; (3) prepare a successful grant application; and (4) develop and implement the infrastructure necessary to meet grants management requirements.

There was considerable agreement among study respondents on barriers and on strategies to reduce those barriers. Numerous strategies for overcoming these barriers were identified in three broad areas: strategies for obtaining grant announcement information, deciding to apply for, and preparing an application; strategies for grant review processes; and strategies for grant management. Within HHS, there are currently initiatives underway at the Department level or within specific agencies that are similar to several of the suggested strategies.

Report Title: Barriers to American Indian/Alaska Native/Native American Access to HHS Programs http://aspe.hhs.gov/hsp/06/barriers2access/index.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Landey, Alana, 202-401-6636
Performer: Westat, Inc.; Rockville, MD
PIC ID: 8286

To What Extent Were Americans Dependent on Social Welfare Programs?

Summary:

This analysis of indicators of welfare dependence for the ninth annual report to Congress examined the extent to which Americans were dependent on social welfare programs. The analysis identified welfare dependence indicators through 2003, reflecting changes that had taken place since enactment of major welfare reform legislation in 1996, the Personal Responsibility and Work Opportunity Reconciliation Act. The analysis used a variety of data sources, including the Current Population Survey and the Survey of Income and Program Participation. Current Population Survey data were analyzed using the Transfer Income Model (TRIM3) microsimulation model, which simulates the major governmental tax, transfer, and health programs that affect the U.S. population and helps to correct for under-reporting of benefit receipt.

Indicators from these data sources showed a significant decline in the percentage of individuals dependent upon welfare since 1996, concurrent with the more widely reported declines in the percentage of individuals receiving welfare benefits. In 2003, 3.6 percent of the total population was dependent upon welfare benefits, in that they received more than half of their total family income from the Temporary Assistance for Needy Families (TANF), Food Stamp, and/or Supplemental Security Income (SSI) programs. While marginally higher than the 3.2 percent dependency rate measured in 2002, the 2003 rate was much lower than the 5.2 percent rate measured in 1996. Overall, 3.4 million fewer Americans were dependent on welfare in 2003 compared to 1996.

Report Title: Indicators of Welfare Dependence Annual Report to Congress 2006 http://aspe.hhs.gov/hsp/indicators06/index.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Hauan, Susan, 202-690-8698
Performer: ASPE Staff; Washington, DC
PIC ID: 8288

Objective 6.5: Expand community and faith-based partnerships

How Do States Pay for Services for Children and Youth with Serious Emotional Disorders and Why Do They Choose These Methods?

Summary:

The mechanisms states use to pay for intensive home and community services for children and youth with serious emotional disorders were studied. The study asked several questions. How do states choose among--and what are the benefits and challenges of--the different financial strategies? Do demonstration projects influence selection of particular financing mechanisms? What are ways to promote coordination among agencies? What is the role of residential treatment centers? There are four major funding mechanisms states use to pay for these services to children and youth. Section 1915(c) of the Social Security Act permits states to waive certain spending restrictions and offer community services to persons who otherwise would qualify for services in a hospital, nursing home, or intermediate care facility for the mentally retarded. Section 1905(a)(13) of the Act allows optional Medicaid coverage of rehabilitative services and related administrative costs. The third funding mechanism is known as "case rates." The final mechanism (the "Katie Beckett" Act) comes from the Tax Equity and Fiscal Responsibility Act of 1982 and provides for special exception to the requirement of institutionalization by allowing payment for care of children in their homes. Officials in each of three sets of states were interviewed.

One group of states had conducted comprehensive statewide reforms of their mental health systems including their financing strategies. A second had made reforms incrementally. The third were contemplating future reform. The strengths and weaknesses of four major financing mechanisms were identified. The study focused especially on the Medicaid home and community-based service waiver (the "1915(c) waiver"). States with such waivers have greater flexibility to provide an expanded set of services under Medicaid to children at serious mental health risk. States may control costs by limiting who can be enrolled in the program.

The study found that state officials typically seek to combine a variety of mechanisms and funding sources because no one mechanism provides the flexibility and breadth needed to coordinate and pay for a comprehensive set of intensive home and community services. A state's legislative and political history of seeking to deal with high-risk youth influences what approaches are likely to succeed. States that carefully manage access to residential treatment services and psychiatric hospitals tend to have more resources for intensive home and community services. Administrative and budgetary procedures can support cost-sharing of services among agencies that serve children. Allowing emotionally disturbed near-poor child in families with limited insurance coverage for mental health care to access intensive services (even if not Medicaid eligible) is critically important. There are three major implications of the study. Sustained improvements in financing intensive home and community services for emotionally disturbed youth depends on the development of effective partnerships between key agencies at the state and local levels. Demonstration projects play an important role in laying the groundwork for carrying out further collaborative systems and financing for intensive services for youth. States should save both federal and state Medicaid money by replacing residential care with more effective home and community services.

Report Title: Public Financing of Home and Community Services for Children and Youth with Serious Emotional Disturbances: Selected State Strategies http://aspe.hhs.gov/daltcp/reports/2006/youthSED.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Kennedy, Cille, 202-690-6443
Performer: Mathematica Policy Research, Inc.; Princeton, NJ
PIC ID: 8342

Goal 7: Improve the stability and healthy development of our Nation's children and youth

Objective 7.1: Promote family formation and healthy marriages

What Evaluations of the Family Planning Program Have Been Conducted and What Did They Find?

Summary:

The project included three parts. The study identified, reviewed and synthesized published and unpublished evaluations relevant to the family planning program. The study also identified, reviewed and synthesized family planning indicators. Finally, the study assessed issues affecting family planning program performance and management at the regional, state, and clinic levels. This was achieved through interviews and focus groups with federal program staff who administered the program, grant recipients and family planning clients. This project was a collaboration between federal Public Health Service regional offices, which administer and monitor family planning service delivery grants (authorized by Title X of the Public Health Service Act), the Office of HIV/AIDS Policy, and the Office of Minority Health.

The findings, including program needs and gaps, were summarized under three broad evaluation headings: scope, quality, and identification. Of the 68 evaluations reviewed, 2/3 focused on "supply level issues" (such as cost of delivering services and structural and operational factors of providing services) and the remainder focused on "demand-level issues" (such as service utilization and characteristics of those using or needing services). The published evaluations were superior to the unpublished studies. Problems identifying relevant evaluations limited the review. For each state, the researchers prepared a brief summary table of state-level data for the 1998-2002 period. Each summary included (for the nation, each region and for each state) the number of: women in need of family planning services and the number of all individuals using family planning. Regarding users of family planning services, each summary also identified, when available, distribution data regarding income, gender, age, clinic usage, and estimates of mistimed and unwanted pregnancy. All key informants agreed that the purpose of the program was to provide family planning services and prevent unintended pregnancies. Respondents agreed that the program addressed an important need for comprehensive family planning and related preventive health services in a manner that was not addressed by any other federal programs.

For both 1995 and 2000, over 16 million women were in need of publicly supported contraceptive services and supplies. Low-income women between 20 and 44 comprised over 2/3 of this population. Limited program funding is one of the most important factors preventing the program from addressing unmet needs among those unable to obtain or afford this health service. In general, there was strong agreement that the program's decentralized structure allows it to have a significant impact. Respondents identified low-income populations, the uninsured, and teens as the groups with the greatest need for subsidized family planning services.

Report Title: Review of the Title X Family Planning Program Evaluation Activities and Assessment of Current Evaluation Needs http://aspe.hhs.gov/pic/fullreports/06/8285.pdf
Agency Sponsor: OPHS-OPA, Office of Population Affairs
Federal Contact: Panda, Pankaja, 240-453-2820
Performer: Research Triangle Institute; Research Triangle Park, NC
PIC ID: 8285

Objective 7.2: Improve the development and learning readiness of preschool children

What Are the Characteristics of the Children and Families Being Served by the Head Start Program?

Summary:

This study analyzed data from a survey begun in year 2000 of a nationally representative sample of children and families served by the Head Start program. The study examined information on children's cognitive and social-emotional development over the Head Start year(s), measures of classroom quality including teacher education, credentials and knowledge of child development as well as direct observations of teacher-child interactions. The study predicted kindergarten performance based on skills and behavior in Head Start.

The study found that the population being served by Head Start is comprised of roughly equal numbers of white, African-American, and Hispanic children from poor families. Most children entered Head Start with math and literacy levels well below national averages. Children made gains toward national averages during the time they spent in Head Start, but still lagged behind national averages. As required in the program performance standards, nearly all Head Start programs used a curriculum; 60 percent of Head Start teachers reported that they used either "Creative Curriculum" or "High/Scope" curriculum. Head Start teachers tended to be qualified and experienced although as a group they did not have the same level of credentials as public school teachers. Teachers with higher levels of education had more positive attitudes and knowledge of early childhood education practices. This was associated with of higher classroom quality (for example: availability of more learning materials, improved language and reasoning activities, and teacher sensitivity and responsiveness). Surprisingly, greater entry-graduation achievement gains for some children were associated with higher, rather than lower child to staff ratios, although the range of variation in these ratios in Head Start is small. In addition, higher teacher scores on a measure assessing their knowledge of early childhood education practices did not translate into higher child cognitive outcomes.

Family incomes tended to rise modestly during their children's Head Start years. A significant portion of caregivers of Head Start children exhibited symptoms of depression or indicated having witnessed violent crime. Half the children lived in households where at least one person smoked and 2/3 lived in households where at least one person drank alcohol. Children whose parents read to them were more likely to show educational gains. A high proportion of families, 2/3, attended Head Start parent-teacher conferences. The measures used in the survey demonstrate strong predictive value for how well children will do in elementary school after their Head Start experiences.

Report Title: Head Start Performance Measures Center Family and Child Experiences Survey (FACES 2000) Technical Report http://www.acf.hhs.gov/programs/opre/hs/faces/reports/technical_2000_rpt/tech2k_final2.pdf
Agency Sponsor: ACF, Administration on Children and Families
Federal Contact: Woolverton, Maria, 202-205-4039
Performer: Westat, Inc.; Rockville, MD
PIC ID: 6331.3

Objective 7.4: Increase the percentage of children and youth living in a permanent, safe environment

What Efforts Do Child Welfare Agencies Engage in to Identify, Locate and Involve Nonresident Fathers?

Summary:

The study: examined the extent to which child welfare agencies involved nonresident fathers of foster children in casework and permanency planning; described the various methods used by local agencies to identify fathers of children in foster care, establish paternity, and locate nonresident fathers; identified challenges to involvement; identified practices and initiatives that may increase father involvement; and explored how child support agencies' information resources may help child welfare agencies to identify and locate nonresident fathers.

Most children in foster care were not living with their fathers at the time they were removed from their homes. Once in foster care, these children may experience even less contact with their nonresident fathers. This study sought to assess typical child welfare practice with respect to nonresident fathers of children in foster care. The study also examined the potential utility of expanding the use of child support enforcement data sources in these efforts. Local agency caseworkers were interviewed by phone about nearly 2,000 children in foster care in four study states (Arizona, Massachusetts, Minnesota and Tennessee) to examine front-line practices related to nonresident fathers.

The study documented that nonresident fathers of children in foster care were not often involved in case planning efforts, and nearly half were never contacted by the child welfare agency. By not reaching out to fathers, caseworkers may overlook potential social connections and resources that could help to achieve permanency for the child.

Report Title: What About the Dads? Child Welfare Agencies' Efforts to Identify, Locate and Involve Nonresident Fathers http://aspe.hhs.gov/hsp/06/CW-involve-dads/index.htm
Agency Sponsor: ASPE, Office of the Assistant Secretary for Planning and Evaluation
Federal Contact: Radel, Laura, 202-690-5938
Performer: Urban Institute; Washington, DC
PIC ID: 8361

Goal 8: Achieve excellence in management programs

Objective 8.2: Improve the strategic management of human capital

What Should the National Institutes of Health Do to Prevent Potential Threats to Its Research Mission?

Summary:

This study provided a framework for developing an enterprise-wide effort to assess and mitigate threats to the ability of the National Institutes of Health (NIH) to be able to carry out its various research-related missions successfully. Threats or risks include results due to actions/inaction of people, processes, systems, technology, science, or external events. The study provided a basis for prioritizing these risks, identifying where and how they might occur, and providing information needed to better allocate resources to prevent or mitigate risks. The study addressed three questions: what is the best method to use to evaluate risk for scientific, administrative, and financial programs; what is the most economical way to collect data needed to evaluate high risk areas; and what was the existing state of risk preparation/avoidance arrangements?

Researchers concluded that several methods existed and have been used by various organizations to assess risk. But, at NIH, several constraints would make scientific study of risk management methods challenging. These included: the unknowable value of potential losses, non-repeating organizational events, the alteration of circumstances by the very act of assessing risk, and the relatively long lifespan of agency activities. The study resulted in a conceptual design to begin evaluating risks that included: reviewing risk literature and existing risk assessment guidance, exploring risk assessment practices and examining case examples of recent risk assessment, and interviewing agency executives. Senior NIH managers were beginning to engage in emergency risk management processes. Some offices were engaged in systematic risk assessment activities.

There was a clear perception among agency leaders of the importance of enterprise-wide solutions. Researchers made several recommendations. The NIH risk advisory committee should annually review and approve risk management policy. Management should lead an effort to integrate risk management into the NIH culture. Internal and external subject matter specialists should be involved as appropriate in the assessment of risk. Management should determine priorities, taking into account such factors as speed of risk onset, urgency, cost of mitigation compared to expected benefit, degree of difficulty and time required to implement. Lessons learned should be identified and communicated to appropriate personnel on a timely basis. The study concluded that risk assessment and mitigation efforts should be embedded into the regularly occurring NIH strategy, operations, scientific, financial, budgeting, and administrative processes.

Report Title: A Feasibility Study for the Evaluation of NIH's Enhanced Internal Management Control (IMC) Program http://aspe.hhs.gov/pic/fullreports/06/8464.doc
Agency Sponsor: NIH, National Institutes of Health
Federal Contact: Hardy, Karen, 301-402-3510
Performer: Deloitte & Touche; Reston, VA
PIC ID: 8464

Chapter III - Program Evaluation at the Department of Health & Human Services

The Department of Health and Human Services (HHS) funds or conducts many evaluations some required by statute, others considered essential by an individual agency, the Department, or the President.  Evaluation is a core Federal management responsibility with strategic planning, policy and budget development, and operating programs (Figure 1, Performance Management System). 
 

Performance Management System

As currently listed in the Catalog of Federal Domestic Assistance (www.cfda.gov), the Department is responsible for more than 330 separate programs.  In FY 2006, HHS spent $613 billion.  Of this amount, $2 billion was for demonstration and evaluation activities, including $832 million provided through the Public Health Service Act Set Aside provision [Section 241(a) of the Act]. Successful evaluation increases the likelihood of effective delivery of public services through these programs and insures that programs are efficient, targeted to their intended clients, and well managed.

 

Role of Evaluation

Programs need to provide good results for the individuals served, spend tax dollars wisely, and achieve the goals intended by Congress and the President. This obligatory report to Congress on Performance Improvement continues the effort to provide a strategic and analytic presentation of evaluation studies.  With the implementation of a unified Strategic Plan, as required by the Government Performance and Results Act of 1993, the Department recognizes its responsibility  to evaluate programs and to assure that evaluation funds are targeted to address the core goals and objectives of both the Congress and Executive branch.  This report reflects the important role evaluations have to test, weigh, measure and judge the success of management performance, program outputs, and social outcomes.

Evaluations are essential to successful achievement of the HHS mission. Assessing various aspects of agency functioning allows staff to identify means of improving individual program performance.  HHS evaluations directly support several efforts. Evaluations help government officials and members of the Congress make decisions related to programs, policies, budgets, and strategic planning. Evaluations enable managers to improve their program operations and performance. Evaluation results and methodological tools are useful to the larger health and human services community of state and local officials, researchers, advocates, and practitioners  to improve the performance of their programs.

 

Types of Evaluation

At their best, HHS evaluations assess the performance (efficiency, effectiveness, and responsiveness) of programs or strategies through the analysis of data or information collected systematically and ethically; the effective use of resulting information in strategic planning, program or policy decision-making and program improvement.  Evaluations serve one or more of the following objectives (Figure 2, Types of Evaluation):

Enhance Program Effectiveness and Support Policy Analysis — Determine the impact of HHS programs on achieving intended goals and objectives and examine the impact of alternative policies on the future direction of HHS programs or services.

Improve Performance Measurement — Monitor annual progress in achieving departmental strategic and performance goals.  We invest evaluation funds to develop and improve performance measurement systems and improve the quality of the data that support those systems.  Performance measurement is a high priority for HHS agencies.  The emphasis during development, implementation, and refinement of programs is on results and specific measurements are required under the Government Performance and Results Act.

Assess Environmental Factors — Seek to understand the forces of change in the health and human services environment that influence the success of our programs.  Such understanding allows us to adjust our strategies and continue to deliver effective health and human services.

Strengthen Program Management and Development — Address the need of program managers to obtain information or data that will help them effectively design and manage programs more efficiently and ensure successful results. Focus on developmental or operational aspects of program activities and provide understanding of services delivered and populations served.

 

Types of Evaluation

 

Evaluation Resources Broadly

Evaluation activities of the various HHS agencies are supported with both general program funding and with some of the funds appropriated under the Public Health Service Act “set-aside” authority. Under the first, program managers, operating under discretionary authority, use program funds to support contracts to design and carry out evaluation studies and analyze evaluation data. In some cases, a program’s legislative authority calls for specially mandated evaluations, and program funds are used directly to support these studies.
 

Public Health Service Section Set-Aside Authority

The Public Health Services Section 241 set-aside authority was originally established in 1970, when the Congress amended the Act to permit the HHS Secretary to use up to 1 percent of appropriated funds to evaluate authorized programs.  Section 241 limited the base from which funds could be reserved for evaluations to programs authorized by the PHS Act.  Excluded were funds appropriated for the Food and Drug Administration, the Indian Health Service, and certain other programs that were managed by PHS agencies but not authorized by the Act (e.g., HRSA’s

Maternal and Child Health Block Grant and CDC’s National Institute for Occupational Safety and Health).[1]  In addition, programs may have other authorized sources of funding for evaluation activities.[2]  Section 207 of the Departments of Labor, Health and Human Services, and Education, and Related Agencies Appropriations Act, 2006, authorized the Secretary to use up to 2.4 percent of the amounts appropriated for programs authorized by the Public Health Service Act for the evaluation of these programs. 

For Fiscal Year 2006, the year reflected in the studies here reported, agencies were budgeted a total of $831 million from the set-aside authority, a portion of which was used for evaluation activities:

• Administration for Children and Families (ACF) -- $11 million
• Agency for Healthcare Research and Quality (AHRQ) -- $319 million
• Centers for Disease Control and Prevention (CDC) -- $267 million
• Health Resources and Services Administration (HRSA) -- $28 million
• National Institutes of Health (NIH) -- $24 million
• Substance Abuse and Mental Health Services Administration (SAMHSA) -- $121 million
• Three staff components in the Office of the Secretary received a total of $40 million:
• Office of the Assistant Secretary for Planning and Evaluation (ASPE)
• Office of Public Health and Science (OPHS)
• Office of the Assistant Secretary for Resources and Technology (ASRT)
• In addition, the Office of the National Coordinator for Health Information Technology (ONC) received $19 million.

Most evaluation studies are started in one budget year, carried out in one or more subsequent years, and final reports, marking the completion of each study, may be delivered and available for the public, in a third or subsequent year. Therefore, the studies completed in a particular year cannot be equated to the funds appropriated for the same year.

This Performance Improvement 2007 report includes studies funded through the Public Health set-aside authority as well as studies supported with other appropriated funds.

Evaluation Management

Management of evaluations carried out by HHS agencies and offices and coordinated by ASPE involves: (1) planning and coordination, (2) project oversight, (3) quality assurance, and (4) dissemination of results (Figure 3, Evaluation Management Cycle).  A description of each function follows.

Figure 3: Evaluation Management Cycle

Evaluation Planning and Coordination

HHS agencies, ASPE, the Office of the Inspector General (OIG), and several other offices develop evaluation plans annually in concert with HHS program planning, legislative development, and budgeting cycles. Plan development is coordinated by ASPE.  Each agency or office evaluation plan generally states the evaluation priorities or projects under consideration for implementation. Typically, HHS evaluation priorities include: congressionally-mandated program evaluations, evaluations of Secretarial program or policy initiatives, assessments of new programs and ones that are candidates for reauthorization, and evaluations that support program performance management and accountability.

HHS evaluation planning activities are coordinated with three department-wide planning initiatives. First, HHS evaluation activities support the Department’s strategic planning and performance management activities in several ways.  Completed evaluation studies are used in shaping specific HHS strategic goals and objectives.  Evaluation findings provide important sources of information and evidence about the success of various HHS programs or policies. The HHS Strategic Plan highlights evaluations that document efficacy or effectiveness of strategic programs or policies and lists future evaluations that will benefit strategic planning.  HHS agencies use findings from their evaluations to support the Government Performance and Results Act (GPRA) annual performance reporting to Congress and program budget justifications.

Second, Congress requests that HHS coordinate and report to Congress regarding all of its research, demonstration, and evaluation (RD&E) programs to ensure that the results of these projects address HHS program goals and objectives.  ASPE and the Assistant Secretary for Resources and Technology work together with HHS agencies to provide the Congress with a special annual research, demonstration, and evaluation budget plan that coincides with the preparation of the President’s fiscal year budget.  The plan outlines HHS agency research, demonstration, and evaluation priorities as related to the Department’s strategic goals and objectives (Figure 4, Evaluation Reporting Cycle).

Figure 4: Evaluation Reporting Cycle

Third, the Secretary report to the Congress his plan for using PHS evaluation set-aside funds before implementing the plan (Figure 4). 

Project Oversight

HHS agencies, ASPE, OASH, and the OIG execute annual evaluation plans that involve developing evaluation contracts and disseminating and applying evaluation results.  All agencies and their subunits (centers, institutes, and bureaus) coordinate with each other on research and evaluation project planning and release of final reports that relate to work of other HHS agencies.  While there is some oversight responsibility and execution capability in the Office of the Director or Administrator for each agency, the various agency subunits conduct much of the day-to-day evaluation activity.

The OIG performs independent evaluations through its Office of Evaluations and Inspections (OEI).  OEI’s mission is to improve HHS programs by conducting inspections that provide timely, useful, and reliable information and advice to decision makers.  Findings of deficiencies or vulnerabilities and recommendations for corrective action are usually disseminated through inspection reports issued by the Inspector General.  A summary of individual inspection reports and other OIG reports can be viewed on the World Wide Web at http://oig.hhs.gov/reports.html. OEI provides technical assistance to HHS agencies in conducting their evaluations.

Quality Assurance and Improvement

Most evaluation projects are developed at the program or office level.  A committee of agency- or office-level policy and planning staff members generally conducts the initial quality review.  Before a project is approved, a second committee reviews it for technical quality with expertise in evaluation methodology.  Technical review committees generally follow a set of criteria for quality evaluation practice established by each agency.  ASPE, for example, has a peer review committee that serves to improve the technical merits of ASPE proposals before final approval. Some HHS agencies have external evaluation review committees composed of evaluation experts from universities and research centers.

Since HHS began reporting to Congress in 1995 on completed evaluations through the Performance Improvement report series, the Department has focused attention on improving the quality of evaluation studies performed.  In the past, Evaluation Review Panels, convened periodically, have contributed insights to HHS evaluation officers on the strengths and challenges of ensuring quality evaluation studies.  HHS evaluation officers have had opportunities to discuss these strengths and challenges and identify steps to improve agency evaluation projects.

Dissemination of Evaluation Reports

Maintaining online electronic report libraries and distributing information on evaluation results is an important component of HHS evaluation management.  The Department’s information and reports on major evaluations are available through the Web site of the HHS Policy Information Center (PIC), located at: http://aspe.hhs.gov/pic/performance.  ASPE’s PIC Web site offers users an opportunity to search – by key word, selected program, or policy topics – the departmental evaluation report database and electronic report library maintained by ASPE. The PIC contains over 8,500 completed and in-progress evaluation and policy research studies conducted by the Department of Health and Human Services, as well as some studies completed outside of it by others.

During the past year, the Policy Information Center online database has been modified so that evaluation study project officers or other key agency staff now directly submit information online about current and project evaluations. This means that, as regards the online database, there is no delay in making information avalible to evaluation peers in other parts of the Department, and to the public at large. Researchers may now search to see what studies have been funded and are currently underway that may be relevant to their own research or planning activities. New entries in the online database are intended to focus on effective and clear summaries answering the basic questions: what was the study, why was it conducted, and what was learned. Through the online database, several months before annual reports are due to the Congress, much of the information regarding the work of evaluation underway can be known both to Congressional and Executive Branch staff and to the public as well, speeding the dissemination of important factual information regarding work of the Department. A positive result is reduced chances of duplication of efforts and speedier application of policy implications of evaluation work carried out.

Additionally, the results of HHS evaluations are disseminated through targeted distribution of final reports, articles in referenced journals, and presentations at professional meetings and conferences. Although individual HHS agencies have primary responsibility for disseminating results, ASPE continues its Department-wide efforts to expand dissemination of evaluation results to the larger research and practice communities through email lists, e-newsletters, and publications available on the PIC Web site.

 

Appendix a HHS FY 2004-2009

“PREVENTING DISEASE AND ILLNESS”

“PROTECTING OUR HOMELAND”

“CLOSING THE GAPS IN HEALTH CARE”

“IMPROVING HEALTH SCIENCE”

“REALIZING THE POSSIBILITIES OF 21^ST CENTURY HEALTH CARE”

“WORKING TOWARD INDEPENDENCE”

“LEAVING NO CHILD BEHIND”

 

 

“IMPROVING DEPARTMENT MANAGEMENT”

 

Appendix B Agency Mission and Evaluation Program Statements

This appendix provides brief statements for each agency or office, of the Department of Health and Human Services, that conducts evaluation activities. The majority received funding this past Fiscal Year under Section 241 of the Public Health Service Act; others, (notably, Administration on Aging,  Centers for Medicare and Medicaid Services, Food and Drug Administration, and the Indian Health Service) expended other discretionary or mandatory program funds on evaluation activities. The statements indicate both the broad mission of each agency or office and the general objectives and focus of its evaluation activities.

Mission

Promote the economic and social well-being of families, children, individuals, and communities.

Evaluation Program

The Administration for Children and Families (ACF) administers a broad range of formula and discretionary programs, including family self-sufficiency (Temporary Assistance for Needy Families), child support, children and family services (Head Start, Child Welfare, Family Preservation and Support, and youth programs), and special programs for targeted populations, such as the developmentally disabled, refugees, and Native Americans.

ACF’s evaluation objectives are to: furnish information on designing and operating effective programs; test new service delivery approaches capitalizing on the success of completed demonstrations; apply evaluation data to policy development, legislative planning, budget decisions, program management, and strategic planning and performance measures development; and disseminate findings of completed studies and promote application of results by state and local governments.

ACF stays current on emerging issues affecting its programs and identifying questions for evaluation studies by actively engaging other federal agencies, state and local policy and program officials, national organizations, foundations, professional groups and practitioners, and consumers. 

Studies are often funded as joint ventures with the Office of the Assistant Secretary for Planning and Evaluation (ASPE) and other federal agencies and foundations. Such collaborations permit large-scale efforts that are better informed and more representative of varying perspectives. Multidisciplinary experts review proposals. Evaluation study designs are carefully developed in collaboration with project partners and technical experts in order to address specific research questions. Work groups of various kinds are used to monitor the progress of projects and to advise on design refinements and the presentation of findings.

 

Mission

Foster the development of services to help older persons maintain their independence.

Evaluation Program

The Administration on Aging (AoA) is the federal focal point and advocate agency for the concerns of older persons. The agency administers key federal programs mandated under various titles of the Older Americans Act. These programs help vulnerable older persons remain in their own homes by providing supportive services. Other programs offer opportunities for older Americans to enhance their health and to be active contributors to their families, communities, and the nation through employment and volunteer programs. AoA works closely with its nationwide network of regional offices and state and area agencies on aging to plan, coordinate, and develop community‑level systems of services that meet the unique needs of individual older persons and their caregivers. AoA collaborates with other federal agencies, national organizations, and representatives of business to ensure that, whenever possible, their programs and resources are targeted to the elderly and coordinated with those of the network on aging. As the responsibilities of this nationwide network of state and area agencies on aging continue to grow, it is essential that they have the necessary information to meet these responsibilities.

The overall evaluation priorities of the AoA are to support studies that provide information on:  (1) Successful program implementation in meeting the goals of the Older Americans Act; (2) Design and operation of effective programs; (3) Issues relevant to policy development, legislative planning, and program management.

 

Mission

Improve the quality, safety, efficiency and effectiveness of health care for all Americans.

Evaluation Program

Evaluation activities within the Agency for Healthcare Research and Quality (AHRQ) provide executive management, program officers and audiences external to the Agency with evaluative findings concerning the Agency’s effectiveness and efficiency in meeting its Government Performance Results Act (GPRA), Program Assessment Rating Tool (PART), and other performance goals.  The work is conducted by external, independent evaluators.  Evaluation components are built into virtually all major AHRQ programmatic or portfolio activities beginning at the design phase.  Among evaluation mechanisms used by the Agency are targeted evaluation studies undertaken through contracts, using a variety of quantitative and qualitative methods, and that tend increasingly to provide more real-time monitoring feedback.  Evaluation activities also include satisfaction feedback from AHRQ customers regarding the usefulness of its research findings and dissemination products. 

 

Mission

Serve the public by using the best science, taking responsible public health actions, and providing trusted health information to prevent harmful exposures and disease related to toxic substances.

Evaluation Program

The Comprehensive Environmental Response, Compensation, and Liability Act (CERCLA), more commonly known as Superfund, created the Agency for Toxic Substances and Disease Registry (ATSDR) as a federal agency. ATSDR was created to carry out the health‑related sections of CERCLA and other laws that protect the public from hazardous waste and environmental spills of hazardous substances. The ATSDR evaluation program is coordinated with the HHS-wide strategic planning process to carry out requirements of the Government Performance and Results Act (GPRA), the Program Assessment Rating Tool (PART), and the President’s Management Agenda (PMA). ATSDR’s strategic goals and its annual performance plan are the result of an interactive process that reflects a long‑term commitment by Agency staff to develop stronger relationships among external clients and stakeholders, to assess products and services using relevant data, and to improve our processes and systems for more efficient accomplishment of its mission.

 

Mission

Promote health and quality of life by preventing and controlling disease, injury, and disability.

Evaluation Program

The Centers for Disease Control and Prevention (CDC) conducts evaluation studies designed to provide essential information about its programs, goals, and priorities.  These projects support the assessment of CDC’s strategies, which are to protect the health and safety of Americans provide credible information to enhance health decisions, and promote health through strong partnerships.

CDC emphasizes evaluations that advance its health protection goals, answer policy, program, and strategic planning questions related to the goals and objectives of Healthy People 2010.  Performance improvement studies, such as those focusing on the development of key performance indicators consistent with the Government Performance and Results Act (GPRA) and the Office of Management and Budget’s Program Assessment Rating Tool (PART) are of particular interest and import to the Agency.

In addition, CDC supports a variety of activities to enhance evaluation quality, use, and understanding.  An example of one such activity completed during FY2006 was an economic evaluation of various strategies to expand childhood hepatitis A immunization in the U.S.  The data from this project were pivotal in the Advisory Committee on Immunization Practices' decision to extend hepatitis A immunization nationwide.

 

Mission

Assure health care security for beneficiaries.

Evaluation Program

 

The research arm of the Centers for Medicare & Medicaid Services (CMS), the Office of Research, Development, and Information (ORDI), performs and supports research and evaluations of demonstrations (through intramural studies, contracts and grants) to develop and carry out new health care financing policies and provide information on the impact of CMS’ programs.  ORDI’s activities embrace all areas of health care: costs, access, quality, service delivery models, and financing.  ORDI’s responsibilities include evaluations of ongoing Medicare and Medicaid programs and demonstration projects testing new health care financing and delivery approaches. Examples of research themes include state program flexibility, the future of Medicare, provider payment and delivery, and vulnerable populations and dual-eligibles.

 

Mission

 

To protect the public health by assuring the safety, efficacy, and security of human and veterinary drugs, biological products, medical devices, our nation’s food supply, cosmetics, and products that emit radiation; to advance public health by helping to speed innovations that make medicines and foods more effective, safer, and affordable; and helping the public get the accurate, science-based information they need to use medicines and foods to improve their health.

 

Evaluation Program

The Office of Planning’s Evaluation Staff plays an integral role in carrying out the FDA’s mission. Assessing various aspects of Agency program performance allows staff to identify means of improving that performance. The Food and Drug Administration evaluation function has three goals: 1) provide information on FDA programs that helps Agency officials, the Department, and members of Congress make decisions related to programs, policies, budgets, and strategic planning; 2) help FDA managers improve program operations and performance; and 3) disseminate evaluation results and methodological tools useful to FDA program managers and, in some cases, to the larger public health community.

Evaluation in FDA can be defined as the assessment of the performance (efficiency, effectiveness, and responsiveness) of FDA programs or strategies through the analysis of data or information collected and the effective use of the resulting information. Evaluation activities serve one or more of the following purposes:

• Program evaluation
• Performance measurement
• Environmental assessment
• Process assessment

 

Mission

Provide national leadership, program resources and services needed to improve access to culturally competent, quality health care.

Evaluation Program

The Health Resources and Services Administration (HRSA) supports a wide array of very different programs and activities that promote access to needed health care for all, including primary health care centers, the National Health Service Corps, HIV/AIDS programs, maternal and child health activities, health professions training, rural health programs, organ donation and transplantation initiatives, and telehealth activities.  To provide underpinning for these efforts, HRSA’s evaluation program is designed to enhance strategic planning, strengthen budget and legislative development, and improve program performance.

 

Mission

In partnership with American Indian and Alaska Native people, raise their physical, mental, social, and spiritual health to the highest level.

Evaluation Program

The goal of the Indian Health Service (IHS) is to assure that comprehensive, culturally acceptable, personal and public health services are available and accessible to American Indian and Alaska Native people. The importance of evaluation in supporting this goal has increased significantly in recent years and includes American Indians and Alaska Natives as the primary stakeholders in defining the purpose, design, and execution of evaluations. The stakeholders use the end product of the evaluations, and are the population or groups most likely to be affected by the findings. The IHS has formally adopted the principle of a responsive evaluation practice to address the needs and concerns of Native Americans and Alaska Natives.

Each year IHS selects high‑priority health care and management studies for funding through the submission of proposals to headquarters and Area Offices. These proposals are reviewed and rated by a panel of subject‑matter experts, evaluation experts, and IHS staff for concurrence with IHS strategic goals, objectives, and priority areas. The proposals are then prioritized and forwarded to the IHS Director, who reviews the projects that are recommended for funding and determines the respective funding levels.

The evaluation needs of the IHS service components are coordinated using two major types of short‑term studies: policy assessments and program evaluations. Policy assessments contribute to decision making about budget, legislation, and program modifications including information to support the Agency’s initiatives. Evaluations are focused at the program level, or Area Offices, and focus on specific needs.

The evaluation program of the IHS is managed by the Office of Public Health Support, Staff Office of Planning, Evaluation, and Research, which provides national leadership and consultation for IHS and Area Offices on strategic and tactical planning, program evaluation and assessment, public health and medical services, research agendas, and special public health initiatives for the Agency.

 

Mission

Uncover new knowledge that will lead to better health for everyone.

Evaluation Program

The National Institutes of Health (NIH) pursue new knowledge about the prevention, detection, diagnosis, and treatment of disease and disability.  To that end, NIH has a wide range of programs to support health-related research and training and professional development.  Evaluating these numerous and diverse programs is one important tool that NIH administrators use to determine the extent to which these programs are operating efficiently and achieving their intended outcomes.

NIH Institutes and Centers (ICs) and components within the Office of the Director (OD), NIH, use program evaluations and evaluation-related activities to improve decision-making and, ultimately, enhance program performance.  Many NIH activities are crosscutting in nature and require trans-NIH program evaluations (i.e., program evaluations that involve more than one IC or OD office) to be examined effectively.  Program evaluations are professional systematic investigations or studies that evaluate the merit of particular programs, or contribute to making such an evaluation possible.  In most cases, the purpose of program evaluations is to help NIH administrators improve a program or make other programmatic decisions (e.g., how to allocate resources).  A “program” is broadly defined as any set of activities funded by the NIH to achieve one or more predefined goals (also referred to as “program goals”).

The NIH recognizes that results-based management as a basic principle for the sound and productive operation of government agencies and their programs.  The most notable evidence of this is passage of the Government Performance and Results Act (GPRA), and the use of the Program Assessment Rating Tool (PART).  With additional efforts to increase public sector accountability (such as passage of the Chief Financial Officers Act and the Government Management Reform Act), interest in evaluation has increased steadily among program administrators.

A distinguishing feature of the NIH Evaluation Program is its position within a larger institutional framework of several evaluation strategies including the use of national advisory councils, boards of scientific counselors, consensus development conferences, and ad hoc committees. This framework helps to chart scientific directions and select the most promising research to support.

 

Mission

Provide the Secretary analyses and advice for policy development, and help the development and coordination of department-wide program planning and evaluations.

Evaluation Program

The Office of the Assistant Secretary for Planning and Evaluation (ASPE) has three evaluation-related responsibilities: funding or conducting necessary policy and evaluation research; in partnership with others, especially HHS agencies, planning and carrying out evaluations; and providing oversight and advice to the Secretary regarding evaluation across the Department.  ASPE functions as a principal advisor to the Secretary on policy development and conducts a variety of health and human services evaluation and policy research studies on issues of national importance. ASPE also is responsible for department wide coordination of planning, policy review, and legislative activities. In its evaluation coordination role, ASPE has the following tasks: (1) Provide annual guidance to all HHS agencies and staff offices regarding evaluation priorities, procedures, and review requirements. (2) Review evaluation priorities proposed by HHS agencies, providing advice about the focus or method of proposed projects and identifying opportunities for collaboration and effective use of resources. (3) Prepare planning and summary reports on evaluation activities as required by Congress.

Through the departmental evaluation planning process, ASPE has the capacity to identify crosscutting health and human services program or policy issues of particular concern to the Secretary and specific program and policy areas not covered by the HHS Agency evaluation plans. In these instances, ASPE initiates evaluations or collaborates with the agencies to conduct evaluations or policy assessments.

Another continuing evaluation objective of ASPE is to support and promote the development and improvement of databases that HHS agencies and ASPE use to evaluate health care programs and health trends. ASPE  co-chairs and provides support to the HHS Data Council, which is charged with integrating key national surveys, such as linking health status indicators with indicators of well being.

Finally, ASPE uses evaluation funds to promote effective use of evaluation-generated information in program management and policymaking. The latter is accomplished through the dissemination of evaluation findings and other activities, such as providing technical assistance to agencies in the development of performance measures.

Mission

The mission of the Office of the Assistant Secretary for Resources and Technology (ASRT) is to provide advice and guidance to the Secretary on budget, financial management, and information technology, and grants management and to provide for the direction and coordination of these activities throughout the Department.

 

Evaluation Program

ASRT systematically evaluates the effectiveness of HHS programs and strategies, including examination of program purpose and design, strategic planning systems, program management, and program results.  ASRT also oversees the development of the HHS annual performance plans and reports pursuant to the requirements of the Government Performance Results Act (GPRA).

Mission

The Office of the National Coordinator for Health Information Technology (ONC) provides counsel to the Secretary of HHS and Departmental leadership for the development and nationwide implementation of an interoperable health information technology infrastructure. Use of this infrastructure will improve the quality, safety and efficiency of health care and the ability of consumers to manage their health information and health care.

Evaluation Program

The Office of the National Coordinator for Health Information Technology is engaged in funding research and programs to foster the development of interoperable nation-wide health information exchange. Initiatives are being funded to harmonize standards for interoperability, certify electronic health record systems, evaluate the variation of organization-level business practices, policies, and state law that relate to privacy and security, and develop best practices and consensus-based policies for health information exchange.  

 

Mission

Provide advice to the Secretary on public health and science, provide executive direction to program offices within the Office of Public Health and Science, and, at the Secretary’s direction, coordinate crosscutting public health and science initiatives in the Department.

Evaluation Program

The Office of Public Health and Science (OPHS) provides advice, policy and program coordination, and leadership in the implementation, management, and development of activities related to public health and science, as directed by the Secretary.  OPHS helps HHS conduct broad‑based public health assessments to better address and solve public health problems.  It assists other components of HHS in anticipating future public health issues and helps ensure that HHS designs and carries out appropriate approaches, interventions, and evaluations that will maintain, sustain, and improve the health of the Nation. OPHS provides leadership and policy recommendations on population‑based public health and science and, at the Secretary’s direction, leads or coordinates initiatives that cut across agencies and operating divisions. In addition, OPHS communicates and interacts, on behalf of the Secretary, with professional and constituency organizations on matters of public health and science. Finally, OPHS’ unique role allows it to use its resources to link important HHS programs or fill gaps in areas needing better policy formulation and coordination.

OPHS’ evaluation strategy focuses on public health and science issues that cut across multiple interests of the operating divisions and requires a coordinated approach to achieve the most effective results.  OPHS evaluations support the Assistant Secretary for Health as the Secretary’s senior advisor for public health and science.  OPHS also conducts evaluations specific to the needs of the programs operated from the offices located within OPHS, such as women’s health, minority health, disease prevention and health promotion, and research integrity.  Some evaluation funds are made available to the ten HHS Regional Health Administrators.

 

Mission

Build resilience and facilitate recovery for people with, or at risk for, substance abuse and mental illness.

Evaluation Program

The Substance Abuse and Mental Health Services Administration (SAMHSA) conducts evaluations to ensure accountability for federal funds and to measure results toward its programmatic and policy objectives.  In compliance with the Government Performance and Results Act (GPRA), SAMHSA is improving performance management and results by identifying annual, long-term and cost-efficiency performance measures to manage its programs.

SAMHSA has an combined evaluation and planning process.  Strategic planning identifies priorities that drive the development of grant programs and evaluations. The formulation of programmatic and evaluation priorities includes consultation with SAMHSA Center Advisory Councils, with other HHS agencies, and with experts in the fields of evaluation and service delivery.  Early and continuous coordination of program planning and evaluation design results in the articulation of program objectives that may be evaluated. Evaluations measure achievement of grant programs overall objectives, and these results are used for program and policy development. The strategic planning and policy development processes then use these results to refine SAMHSA’s priorities and performance objectives.

The specific type of evaluation required depends on the type and purpose of the particular grant program. To the greatest extent possible, SAMHSA encourages the use of comparable data elements and instruments across its evaluations to carry out a comprehensive evaluation system and to minimize respondent burden.  Efforts to improve evaluation are continuing and SAMHSA is committed to using systematic approaches in using data to accomplish its mission.

Appendix C Studies by Agency

AGENCY	ID	TITLE	G/O	OBJECTIVE	PROJECT OFFICER
ACF	8282	How Can States Implement Healthy-Marriage Programs for Unmarried Couples with Children?	6.1	Increase the proportion of low-income individuals and families, including those receiving welfare, who improve their economic condition	Campbell, Nancye
ACF	8274	What Approaches Best Help Welfare Clients Living in Rural Places Move from Welfare to Work?	6.4	Improve the economic and social development of distressed communities	Dubinsky, Michael
ACF	8281	How Can States Encourage Welfare Applicants, Recipients, and those Leaving Welfare, to Seek, Retain and Advance in Employment?	6.1	Increase the proportion of low-income individuals and families, including those receiving welfare, who improve their economic condition	Richards, Patrice
ACF	8279	How Can Former Welfare Recipients Be Successfully Encouraged to Obtain and Retain Employment, and Advance to Better Paying Jobs?	6.1	Increase the proportion of low-income individuals and families, including those receiving welfare, who improve their economic condition	Richards, Patrice
ACF	8276	What Lessons Can be Learned By Combining Evaluation Data From Several Welfare-to-Work Programs?	6.1	Increase the proportion of low-income individuals and families, including those receiving welfare, who improve their economic condition	Sternbach, Leonard
ACF	8275	What Lessons Can Be Learned from Evaluating a Community College Approach Designed to Improve Employment Prospects of Welfare Recipients Already in the Workforce?	6.1	Increase the proportion of low-income individuals and families, including those receiving welfare, who improve their economic condition	Sternbach, Leonard
ACF	6331.3	What Are the Characteristics of the Children and Families Being Served by the Head Start Program?	7.2	Improve the development and learning readiness of preschool children	Woolverton, Maria
ASPE	8292	What Might Happen if States Allow Current and Former Welfare Recipients to Keep More of Their Child Support Money?	6.1	Increase the proportion of low-income individuals and families, including those receiving welfare, who improve their economic condition	Burnszynski, Jennifer
ASPE	8477	What Do We Know About Medicare Advantage Plans?	5.2	Increase the appropriate use of effective health care services by medical providers	Cox, Donald
ASPE	8289	What Can We Learn from Employer-Provided Health Insurance Data?	5.5	Accelerate the development and use of an electronic health information infrastructure	Hauan, Susan
ASPE	8288	To What Extent Were Americans Dependent on Social Welfare Programs?	6.4	Improve the economic and social development of distressed communities	Hauan, Susan
ASPE	8481	What Were the Short-term Impacts of Providing a Life Skills Education Component to Middle and High School Youth?	1.2	Reduce the incidence of sexually transmitted diseases and unintended pregnancies	Kelsey, Meredith
ASPE	8478	Is Substance Abuse a Barrier to Employment and How Effective Are Substance Abuse Treatment Interventions for Welfare Recipients?	6.1	Increase the proportion of low-income individuals and families, including those receiving welfare, who improve their economic condition	Radel, Laura
ASPE-ODALTCP	8319	What are the Barriers to Implementing Innovative Technology in Residential Long-Term Care Settings?	3.5	Expand access to health care services for targeted populations with special health care needs	Aykan, Hakan
ASPE-ODALTCP	8348	What is the Current Supply of Professional Social Workers in Long-Term Care Settings and What is the Expected Future Supply?	3.2	Strengthen and expand the health care safety net	Bergofsky, Linda
ASPE-ODALTCP	8344	What Role Does the Physician Discipline Function of State Medical Boards Play in Assessing Medical Quality?	5.4	Improve consumer and patient protections	Bergofsky, Linda
ASPE-ODALTCP	8332	How Did Texas Modify its Medicaid Program In Order to Have "Money Follow the Person" When Nursing Home Residents Chose to Live in The Community?	6.3	Increase the independence and quality of life of persons with disabilities, including those with long-term care needs	Bergofsky, Linda
ASPE-ODALTCP	8349	Who Has an Intellectual or Developmental Disability, Who Provides Their Care, and How Will That Change in the Future?	3.5	Expand access to health care services for targeted populations with special health care needs	Drabek, John
ASPE-ODALTCP	8347	Who Lives in Residential Care Facilities and How Disabled are They?	3.5	Expand access to health care services for targeted populations with special health care needs	Drabek, John
ASPE-ODALTCP	8338	How Can Barriers to Interoperable Health Information Technology in Post-Acute and Long-Term Care be Reduced; Can Patient Data be Made to Comply with Health Information Technology Standards?	5.5	Accelerate the development and use of an electronic health information infrastructure	Harvell, Jennie
ASPE-ODALTCP	8342	How Do States Pay for Services for Children and Youth With Serious Emotional Disorders and Why Do They Choose These Methods?	6.5	Expand community and faith-based partnerships	Kennedy, Cille
ASPE-ODALTCP	8345	What Role Do Negotiated Risk Agreements Play in Assisted Living?	3.5	Expand access to health care services for targeted populations with special health care needs	Kennedy, Gavin
ASPE-ODALTCP	8343	What is the Role for Adult Day Services in the Health Care and Long-Term Care Systems?	6.2	Increase the proportion of older Americans who stay active and healthy	Rosenoff, Emily
ASPE-ODALTCP	8330	How Can the United States Make Sure That It Has a Qualified Workforce To Provide Services to Those Needing Long-Term Care?	3.2	Strengthen and expand the health care safety net	Rosenoff, Emily
ASPE-ODALTCP	8317	Why Do Individuals Choose and Remain in Nursing Assistant Careers and Work in Nursing Homes?	2.1	Build the capacity of the health care system to respond to public health threats, especially bioterrorism threats, in a more timely and effective manner	Squillace, Marie
ASPE-OHP	8294	How Should Graduate Medical Education in the United States Best Be Financed?	2.1	Build the capacity of the health care system to respond to public health threats, especially bioterrorism threats, in a more timely and effective manner	Nonnemaker, Lynn
ASPE-OHSP	8286	What Barriers to Accessing Discretionary Grants Do Native American Tribes and Organizations Face?	6.4	Improve the economic and social development of distressed communities	Landey, Alana
ASPE-OHSP	8361	What Efforts do Child Welfare Agencies Engage in to Identify, Locate and Involve Nonresident Fathers?	7.4	Increase the percentage of children and youth living in a permanent, safe environment	Radel, Laura
ASPE-OPPS	8343.1	How Do States Regulate Adult Day Care and Adult Health Care Services?	6.2	Increase the proportion of older Americans who stay active and healthy	Rosenoff, Emily
ASPE-OPPS	8483	How Can Managers and Evaluators Better Understand, Design, and Perform Cost Effectiveness Evaluations of Community Public Health Prevention Programs?	1.1	Reduce behavioral and other factors that contribute to the development of chronic diseases	Tilson, Wilma
ASPE-OSDP	8475	What Features and Capabilities of Electronic Medical Records Best Serve Doctors in Small Practices?	5.5	Accelerate the development and use of an electronic health information infrastructure	Burke-Bebee, Suzie
ASPE-OSDP	8474	What are Private Sector "Best Practice" Approaches to Control Costs of Prescription Drug Benefit Programs?	5.2	Increase the appropriate use of effective health care services by medical providers	Bush, Laina
ASPE-OSDP	8476	How Is Influenza Vaccine Produced and Disseminated in the United States?	2.1	Build the capacity of the health care system to respond to public health threats, especially bioterrorism threats, in a more timely and effective manner	Nevel, Amy
CDC-OD-OSI	8298	What Is the Cost-Effectiveness of Routine Childhood Vaccination for Hepatitis A in the United States?	1.3	Increase immunization rates among adults and children	Slaton, Terrie
CDC-OD-OSI	8297.3	What Messages Are Most Effective In Encouraging HIV-Positive Persons and Others To Act In Ways That Will Prevent Further HIV Transmission?	1.1	Reduce behavioral and other factors that contribute to the development of chronic diseases	Slaton, Terrie
CDC-OD-OSI	8296	Who Wants to Know What About Sexually Transmitted Disease?	5.3	Increase consumer and patient use of health care quality information	Slaton, Terrie
CDC-OD-OSI	8295	Who Is Aware of, Uses, and How Good Is the Information in a Community Preventive Services Guide?	5.3	Increase consumer and patient use of health care quality information	Slaton, Terrie
CDC-OD-OSI	7985.1	How Successful Has the U.S.-Mexico Tuberculosis Referral and Case Management Project Been and What Changes Are Needed?	3.5	Expand access to health care services for targeted populations with special health care needs	Slaton, Terrie
CDC-OD-OSI	7973.1	How Can the Effectiveness of Efforts to Detect Breast and Cervical Cancer Early be Measured and Improved?	5.3	Increase consumer and patient use of health care quality information	Slaton, Terrie
CMS	7667	How Do Integrated Care Programs for the Frail Elderly Use Community-Based Primary Care Physicians?	5.2	Increase the appropriate use of effective health care services by medical providers	Thomas, Fred G.
FDA	8313	Can FDA Improve the Efficiency and Effectiveness of New Drug Application and Biologics Licensing Application First Cycle Reviews?	4.2	Accelerate private sector development of new drugs, biologic therapies, and medical technology	Bobolis, Mary A.
FDA	8312	Did the "Continuous Marketing Application" Pilots Increase the Efficiency and Effectiveness of the New Drug Review Process?	4.2	Accelerate private sector development of new drugs, biologic therapies, and medical technology	Bobolis, Mary A.
FDA	8311	What Information Do Health Professionals Need in Order to Talk Clearly with Their Patients About the Risks of Heart Defibrillators?	5.4	Improve consumer and patient protections	Bobolis, Mary A.
NIH	8364	How Can the Online "My Family Health Portrait" Tool Be Improved?	6.2	Increase the proportion of older Americans who stay active and healthy	Bartlett, Maggie
NIH	8455	How Can Educational Efforts to Prevent Noise-Induced Hearing Loss Be Strengthened?	5.3	Increase consumer and patient use of health care quality information	Blessing, Patricia
NIH	8351	How Easy Is It For Health Researchers, College Students, and the General Public to Use the Proposed Stem Cell Information Web Site?	5.3	Increase consumer and patient use of health care quality information	Blessing, Patricia
NIH	8355	Did Small Grants for Behavioral Cancer Control Research Increase the Number of Scientists Working In this Field?	2.1	Build the capacity of the health care system to respond to public health threats, especially bioterrorism threats, in a more timely and effective manner	Chollette, Veronica
NIH	8397	Can a Curriculum to Train Older Adults to Access Health information on the Web be both Senior Friendly and Trainer Friendly?	5.3	Increase consumer and patient use of health care quality information	Dailey, Stephanie
NIH	8384	What Evaluation Design Would Best Measure the Success of Different Ways to Recruit Patients to Participate in National Institutes of Health Clinical Trials?	4.1	Advance the understanding of basic biomedical and behavioral science and how to prevent, diagnose, and treat disease and disability	Dominguez, Dinora
NIH	8460	What is the Diabetes-based Science Education Program for Tribal Schools Curriculum Project?	1.1	Reduce behavioral and other factors that contribute to the development of chronic diseases	Garfield, Sanford
NIH	8462	Is it Feasible to Evaluate the Centers for AIDS Research Program?	4.1	Advance the understanding of basic biomedical and behavioral science and how to prevent, diagnose, and treat disease and disability	Halula, Madelon
NIH	8431	How Is Money From the National Institute on Aging Being Used By the National Center for Health Statistics and the U.S. Census Bureau?	5.3	Increase consumer and patient use of health care quality information	Hamilton, Elizabeth
NIH	8464	What Should the National Institutes of Health Do to Prevent Potential Threats to its Research Mission?	8.2	Improve the strategic management of human capital	Hardy, Karen
NIH	8393	Can Accessing MEDLINE Via Handheld Computers be Useful in Answering Clinical Questions for Physicians?	5.5	Accelerate the development and use of an electronic health information infrastructure	Hauser, Susan
NIH	8463	How Can A Web Site Be Made Easier for the Public to Use?	5.3	Increase consumer and patient use of health care quality information	Kester, Mary Beth
NIH	8454	How Should The National Library of Medicine Evaluate Its Exhibit Program?	5.3	Increase consumer and patient use of health care quality information	Love, Cindy
NIH	8457	How Could the National Heart, Lung and Blood Institute Bioengineering Research Partnerships Program Be Evaluated?	4.1	Advance the understanding of basic biomedical and behavioral science and how to prevent, diagnose, and treat disease and disability	Lundberg, Martha
NIH	8430	How do Health Care Professionals Look for Medical Information on Medical Web Sites?	5.2	Increase the appropriate use of effective health care services by medical providers	Ma, Wei
NIH	8390	How Has the Graduate Partnerships Program Enhanced Student Education and Participation in Research?	4.3	Strengthen and diversify the pool of qualified health and behavioral science researchers	McGee, Richard
NIH	8456	How Should a Process Evaluation -- of the Centers of Biomedical Research Excellence Program -- be Designed and Conducted?	4.3	Strengthen and diversify the pool of qualified health and behavioral science researchers	Newman, Patricia
NIH	8358	Did an Education Booklet for Survivors Increase Their Knowledge and Coping Skills Following Cancer Treatment?	1.1	Reduce behavioral and other factors that contribute to the development of chronic diseases	Solomon, Felicia
NIH	8398	How Can Information on Clinical Trials Most Effectively Be Provided to Spanish-Speaking Users?	5.3	Increase consumer and patient use of health care quality information	Tse, Tony
OPHS-ODPHP	8467	What Role Should the Federal Government Play In Developing Electronic Personal Health Records?	5.5	Accelerate the development and use of an electronic health information infrastructure	Harris, Linda
OPHS-OMH	7711	Do the Benefits that Result from Using Languages Familiar to Patients Outweigh Costs Health Care Providers Face By So Doing?	3.4	Eliminate racial and ethnic health disparities	Hawks, Betty
OPHS-OMH	7865	What Can Hospitals Do To Assure There Policies and Practices are Culturally and Linguistically Appropriate?	3.4	Eliminate racial and ethnic health disparities	Pacheco, Guadalupe
OPHS-OPA	8285	What Evaluations of the Family Planning Program Have Been Conducted and What Did They Find?	1.1	Reduce behavioral and other factors that contribute to the development of chronic diseases	Panda, Pankaja
OPHS-OPA	8278	Which Interventions Best Improve Communication, Contraceptive Use, and HIV and Sexually Transmitted Disease Prevention Among Couples?	1.1	Reduce behavioral and other factors that contribute to the development of chronic diseases	Panda, Pankaja
OPHS-OPA	8277	How Can Public Health Clinics Better Prevent Violence in Families and Between Intimate Partners?	1.1	Reduce behavioral and other factors that contribute to the development of chronic diseases	Panda, Pankaja
SAMHSA	8307	How Many Drug-Related Emergency Department Visits Were There in the United States During 2004?	1.4	Reduce substance abuse	Ball, Judy
SAMHSA	8302	What is the Extent of Non-Medical Use of Prescriptions Drugs in the United States?	1.4	Reduce substance abuse	Colliver, James
SAMHSA	8305	What Was the Prevalence of Substance Abuse and Mental Health Problems in the United States During 2005?	1.4	Reduce substance abuse	Hughes, Art
SAMHSA	8309	What Were the Characteristics of Individuals Admitted to Substance Abuse Treatment Facilities in 2004?	1.4	Reduce substance abuse	Trunzo, Deborah
SAMHSA	8308	What Were the Characteristics of Discharges from Substance Abuse Treatment Facilities During 2003?	1.4	Reduce substance abuse	Trunzo, Deborah
SAMHSA	8304	What Were the Trends in Admissions to Substance Abuse Treatment Facilities in the United States from 1994 to 2004?	1.4	Reduce substance abuse	Trunzo, Deborah
SAMHSA	8303	What Were the Characteristics of Substance Abuse Treatment Facilities in the United States in 2005?	1.4	Reduce substance abuse	Trunzo, Deborah
SAMHSA	8301	What Were the Characteristics of Individuals Discharged from Substance Abuse Treatment Facilities During 2004?	1.4	Reduce substance abuse	Trunzo, Deborah
SAMHSA	8310	What are the Prevalence Rates of Substance Use, Abuse and Dependence, and Mental Health Problems by State?	1.4	Reduce substance abuse	Wright, Douglas
SAMHSA	8306	What are the Local Area Prevalence Rates of Substance Use, Abuse and Dependence and Mental Health Problems in the United States?	1.4	Reduce substance abuse	Wright, Douglas