Performance Improvement 2007. Chapter I – Overview of Completed Evaluations—Selected Highlights


Chapter I presents highlights from the findings of the studies included in this Performance Improvement 2007 report. These examples were selected to emphasize the breadth of subject areas addressed by the studies and showcase the diverse methodologies used.

The studies examined important management, programmatic, policy, and factual circumstances faced by programs administered by the Department of Health and Human Services. The studies employed a wide range of methods in seeking accurate information about internal and external agency environments, including literature review, focus groups, surveys, microsimulation analyses, field visits, and case studies. Some of the studies were carried out by agency staff; most were completed with the assistance of contract support expertise; all the completed studies reflect an intense, creative and collaborative effort of technical expertise and programmatic knowledge and experience regarding the importance of effective and efficient programs that achieve targeted objectives and that serve the public purposes for which they were established.

While most studies in this collection focus on process and output activites or program environments, some also aim at program outcomes and cost-effectiveness analysis. Several studies, seeking efficient use of resources, were preliminary feasibility or evaluability studies (examinations of the practical means by which a program might be evaluated) rather than full-fledged (and more expensive) evaluations. These studies typically led to advice and recommendations regarding how best to proceed with more in-depth studies.

Studies examined how to communicate important health and human services information to both the professional and general public. Several surveys provided valuable information about the nature of social behaviors or the numbers of institutional clients that are public beneficiaries in order to provide more solid bases for advice to policy decision-makers. Examinations of agency administrative activities also yielded valuable information and recommendations for improving the management of programs.

The studies in Performance Improvement 2007 support the eight important goals articulated in the Department of Health and Human Services’ Fiscal Year 2004-2009 Strategic Plan. The remainder of Chapter I provides highlights of interesting and significant evaluations and findings from the past year, organized under these eight goals:

Preventing Disease and Illness

Several studies focused on ways to better communicate with consumers. A post-cancer treatment education booklet was shown to be useful for both patients and families. A tribally-based diabetes science education program advanced toward operational status. Research supported important actions to advance public health. A study of the cost-effectiveness of routine childhood vaccination for Hepatitis A in the United States resulted in a decision to extend hepatitis A immunization nationwide.

As a result of a research project to evaluate online information used to communicate human immunodeficiency virus (HIV) prevention messages, researchers recommended drafting professionally designed advertisements for various media settings.

The Office of Public Health and Science examined how to better prevent violence in families between intimate partners. The study identified promising practices including: efforts to recognize patients experiencing family or intimate partner violence and helping women safely and confidentially leave abusive situations. Another study, of couples-focused family planning services, found that serving males and building staff capacity to provide these services requires significant effort and more staff development/training. A study of family planning underscored the importance of the program for low-income populations, the uninsured, and teens.

Survey data, essential to budget and administrative planning, was obtained in several studies of drug taking behavior. During 2005, over 13,000 alcohol and drug abuse treatment facilities were operating. At any one time, these facilities served over a million, mostly outpatient, clients. Half of the clients were treated for both alcohol and drug abuse. Two-thirds of the treatment facilities were private non-profits and a quarter were for-profit private facilities. During the same year, illicit drug and alcohol use and binge drinking among youth had declined from the previous years. However, 25 million adults (11 percent of the population) experienced severe psychological distress.

Building Health Care Capacity

Regarding how to best support health professions education, one study provided advice on how to enhance the current system of graduate medical education financing and recommended criteria to use in evaluating the alternatives.

Long-term care needs are expected to grow dramatically; staffing needs are at the center of concerns about this fact. One study concluded that the number of Americans needing long-term care will double from 13 to 27 million during the next 40 plus years, increasing the need to recruit, train, and retain direct care workers, especially considering that some organizations currently experience 100% turnover annually. By 2020, over 300,000 additional direct care staff will be needed to care for individuals with either cognitive or developmental disabilities. Achieving this 37% increase will be difficult because the projected increase in overall labor supply is only 7%. A related study concluded that as many as 50,000 more social workers will be needed to serve in long-term care settings during the coming four decades.

Another study showed the success of small grants for behavioral cancer control research to increase the number of scientists working in the field.

Closing the Gaps in Health Care

A major success revealed by one study supported by the Centers for Disease Control and Prevention of a United States-Mexico bilateral tuberculosis case management and referral project was the project’s formal integration into the detainee screening process at immigration control and enforcement facilities.

Researchers found that residential care patients differ from individuals living at home in a number of ways. They had a greater likelihood of:

  • having incomes below $10,000;
  • receiving help with activities of daily living;
  • being over age 85, female, and widowed; and
  • being far more likely to suffer from Alzheimer’s disease or other dementias.

A study exploring what the United States can do to make sure it has a qualified long-term care workforce observed that criminal background checks are valuable during the hiring process to prevent or reduce abuse of vulnerable adults needing long-term care. Another study concluded that greater efforts are needed to more fully advance the use innovative technologies in residential long term care settings. A third study concluded that agreements between patients and assisted living facilities, in which patients or their representatives directly hire individuals to help them, can allow residents a way to be both more independent and safe.

Improving Health Science

To actively advance essential biomedical research, the National Institutes of Health conducts many intramural (internal) clinical research studies for which it needs thousands of willing patient volunteers annually. A feasibility study provided information with which to design an effective evaluation of how best to solicit these new clinical study participants. Protecting volunteer patients is also critical. A study seeking to identify concerns patients may have about potential conflicts of interest on the part of their health care team members found that patients were not worried about this as long as they knew that an oversight system was in place.

Other feasibility studies in the health sciences area included one that provided information to design an evaluation of how most efficiently to operate 15 cancer incidence and treatment tracking systems. Advice was also obtained about how to better evaluate the functioning of bioengineering research partnership grants in the heart, lung and blood institute. In another critical area, consultant advisors concluded that it was feasible and desirable to conduct an outcome study regarding how productive clinical acquired immune deficiency syndrome (AIDS) research is and how it could be improved.

Essential biomedical research discovery and infrastructure depends on expanding the numbers of individuals doing health sciences research and the number of institutions capable of competing for grants. A graduate partnership program was shown to be effective at recruiting and selecting students and providing them with high-quality research training experiences. One study provided guidance on carrying out an evaluation of a program intended to foster health-related research and enhance competitiveness of investigators at institutions located in states which have historically received fewer research grants.

Two studies by the Food and Drug Administration enabled the agency to refine its review and approval processes. The first concluded that the indifferent results of a pilot program to speed drug approvals did not appear to justify the added costs. The second supported  developing a full evaluation of the agency processes through which it achieves efficient first-cycle review approvals.

Realizing the Possibilities of 21st Century Health Care

Several studies for the National Institutes of Health examined how to make informational Web sites more useful, accessible, and effective. Health science information important to both medical professionals, patients, and other consumers underscores the value of Web-based as well as other methods of information dissemination. Online health information on toxic substances was refined based on a study of site users. The usability of an NIH stem cell information Web site was examined and recommendations for improving the site obtained. Collaboration between the National Institute on Aging, the Census Bureau, and the National Center for Health Statistics was examined and advice provided for strengthening this information dissemination project. A plan for evaluating how to improve educational outreach at over 100 conferences each year was developed.

One study confirmed the effectiveness and appeal of a curriculum to train older adults to access health information on the Web for both seniors and trainers. A study about accessing key medical information from the National Library of Medicine via handheld computers found that the information retrieved was useful in answering clinical questions and that special software designed for the purpose, worked well.

A study about providing information on clinical trials to Spanish-speaking users both confirmed the effectiveness of the Web site and identified areas for further attention. An examination of how educational efforts to prevent noise-induced hearing loss could be strengthened recommended focusing on youth. Studies targeted specific groups such as one for the Food and Drug Administration examining information health professionals need in order to talk clearly with their patients about the risks of heart defibrillators.

The success of efforts by state medical boards to discipline physicians was examined to see whether these efforts contributed to safeguarding medical quality and patient safety. Researchers on the study recommended that demonstrations be conducted to see how well some of the more effective strategies worked elsewhere.

A study to determine what could be learned from employer-provided health insurance data concluded that several research questions could be pursued with the available data, including: the magnitude of costs that are passed along to employees and how such increases affect employee participation in employer-offered health insurance plans. A study to determine how the business case for using health information technology in long-term care settings could be evaluated advised that a small retrospective study with 10-20 nursing homes or home health settings could both make relevant information about this subject available and keep research costs low.

A study was supported by the Office of the Assistant Secretary for Planning and Evaluation to determine how barriers to interoperable health information technology in post-acute and long-term care could be reduced and to determine whether patient data could be made to comply with health information technology standards. The study led to the identification of standards that were subsequently recommended for adoption and use. A second study, funded by the Office of Public Health and Science, that explored the role the federal government should play in developing electronic personal health records resulted in several important findings and recommendations, including that the government could be most effective by avoiding two possible pitfalls: sponsoring a superficial, one-size-fits-all public health record, and allowing many agencies to offer independent public health records with little coordination or strategic vision.

Working Toward Independence

The Department explored circumstances that prevented, and explored ways to support, greater independence on the part of both young and old. A study of what might happen if states allowed welfare recipients to keep more of their child support money found that average annual cash income for the families would increase. Another study, looking at whether the risk of legal responsibility for injury increases when disabled Medicaid clients manage their own personal assistants found that, in general, the risks did not increase.

Findings that challenge assumptions about likely program effectiveness are also important. A study by the Administration for Children and Families that looked at a community college approach designed to improve welfare recipients’ employment prospects found that the program did not produce better economic outcomes. Other individual ACF studies ranged from finding positive program effects, as did a study of a program intended to encourage former welfare recipients to obtain jobs, retain employment and advance into better paying jobs, to finding no program effects.  A study analyzing data from 30 welfare-to-work programs identified common aspects of these demonstrations that positively impacted employment outcomes.

A study of adult day services in the context of health care and long-term care systems found that cost and lack of transportation were the two chief barriers to expansion of such services. An examination of the Web site, “My Family Health Portrait,” that helps people record and display their family health history revealed several features that needed to be modified and provided specific recommendations for doing so.  A study of how Texas modified its Medicaid program to have funds move with nursing home residents who chose to live in the community found that this program could easily be incorporated into Medicaid long-term care programs in other states.

Another study revealed three broad strategies for overcoming barriers that Native American Tribes and organizations face trying to access discretionary grants: obtaining information regarding applying for grants; dealing with the grant review process; and managing grants.

Using micro-simulation analyses and a variety of data sources, the Office of the Assistant Secretary for Planning and Evaluation arrived at summary conclusions regarding the extent to which Americans were dependent on social welfare programs. Overall, 3.4 million fewer Americans were dependent on welfare in 2003 than in 1996. Another study about how states pay for services for children and youth with serious emotional disorders arrived at three major findings: sustained program improvement will depend on effective partnerships among community organizations; demonstration projects will continue to play an important role in improving services; and states should replace residential care with more effective home and community services.

Leaving No Child Behind

A study about the characteristics of children being served by the Head Start program found that most children entered the program with math and literacy levels well below national averages. Children made gains toward national averages during the time they spent in Head Start, but still lagged behind national averages.

A study of child welfare agency efforts to identify, locate and involve nonresident fathers found that nearly half the fathers were never contacted by the agency. The study observed that by not reaching out to fathers, caseworkers may overlook potential social connections and resources that could help to achieve permanency for the children.

Improving Department Management

The National Institutes of Health conducted a mission risk assessment that provided the basis for further systematic investigation and development of ways to assure improved risk protection.

More details on the studies highlighted above, and descriptions of other studies completed during Fiscal Year 2006, are provided in Chapter II. In addition, all the studies and final reports issued for these studies can be found online at

View full report


"PerformanceImprovement2007.pdf" (pdf, 717.63Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®