Skip to main content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.


The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Performance Improvement 2003

Publication Date
Dec 31, 2002


Performance Improvement 2003: Evaluation Activities of the U.S. Department of Health and Human Services is the ninth annual report to Congress summarizing prior fiscal year evaluation efforts. The purpose of this report is to provide Congress with outcome-oriented evaluation findings for the Department’s programs, policies, and strategies. The report contains brief summaries of evaluation results. Additional information can be obtained from the Agency Sponsor, Federal Contact, Performer, the cited web site, or the Policy Information Center (PIC)

Evaluations summarized in this report contribute to performance improvement in one or more of the following ways:

•        Program effectiveness

•        Performance measurement

•        Environmental assessment

•        Program management

Performance Improvement 2003 includes HHS evaluation projects completed during fiscal year (FY) 2002. The report is organized into two chapters. Chapter I provides evaluations that were selected by an outside review panel as outstanding for their potential use by the larger health and human services community. Chapter II presents summaries of all other evaluations completed by the HHS agencies and the Office of the Secretary during FY 2002.

The appendices contain a description of the HHS evaluation program (Appendix A), acknowledgments of the HHS officials who contributed to the report (Appendix B), and the external evaluation experts who recommended evaluations for highlighting (Appendix C) and the criteria the panel used to review the 26 reports nominated for outstanding evaluations for FY 2002 (Appendix D).

This report is required by section 241(b) of the Public Health Service (PHS) Act, as amended by the Preventive Health Amendments of 1993. HHS is directed to submit an annual report summarizing the findings of the evaluations under section 241(a) to the Committee on Health, Education, Labor and Pensions of the Senate and the Committee on Energy and Commerce of the House of Representatives.

We hope that you will find this report useful and informative.

Chapter 1 - Results in Brief: Outcome-Oriented Program Evaluations

This chapter consists of summaries of six evaluation projects selected by an outside review panel as outstanding based on rigor of evaluation methodology, importance of findings, and clarity of presentation.

  • The Panel reviewed twenty-six reports nominated by the HHS agencies and selected the reports to be highlighted in this chapter on the basis of the following criteria:Is the report useful? Does it address a significant policy issue or problem? Are the findings likely to be useful for a broader audience?
  • Is the report conceptually sound? Is it clearly based upon and integrated with previous research? Are assumptions clearly stated? Is the research appropriately linked to the program? Are relevant stakeholders identified and included?
  • Is the report methodologically sound? Are its concepts, designs, data collection, and analyses conducted and reported in a competent manner?
  • Are the recommendations practical? Do the conclusions and recommendations logically follow from the data and analyses, and are they relevant to the questions asked?

A full statement of the criteria is found in Appendix D.

Performance measurement or assessment, program management and development, and policy analysis and development represent the three most common uses of HHS evaluation resources. Performance measurement or assessment is a high priority for HHS agencies. The development, implementation, and refinement of programs are more results oriented than in the 1990’s and specific measurements are required under the Government Performance and Results Act (GPRA) of 1993. Program management and development reflects the kind of evaluation projects that program managers initiate to obtain information or data that will help them manage a program more efficiently and ensure successful results. Policy analysis and development includes the evaluation projects conducted by HHS agencies to examine the impact of alternative policies, on the future direction of HHS programs or services.

It is important to note that the context in which an evaluation was conducted may have changed since the evaluation was initiated. It should also be recognized that many evaluations are bounded by policy and resources from the funding agency. In addition, the panel considered whether the timing of the evaluation is appropriate for inclusion – evaluation reports that reflect a mid-program assessment of an ongoing evaluation may be included if the policy it addresses is currently under discussion, if the program is large and recommendations potentially include a need for change in program direction, or for other reasons that indicate a midcourse assessment is appropriate.

Each summary includes a brief abstract; a description of the study, including its purpose, background, methods, findings, and use of results; and the name and phone number of the HHS official to contact for additional information.

Following are the six highlighted reports:

Identifying a Uniform Data Set of Information/Data to Assess the Impacts of Activities Funded by the Office of Minority Health


This report examined obstacles to designing a uniform data set (UDS) to assess the impact of activities funded by the Office of Minority Health (OMH), which administers a broad array of grants and contracts aimed at eliminating disparities in health outcomes and access to services for minority groups. Taking into account the obstacles, as well as guidelines, legislation, performance measurement methodology, and databases, the evaluators described a recommended data system. The literature review attended to a variety of important contextual issues (e.g., recommendations regarding the evaluation of programs  like those administered by OMH). Site visits and focus groups with relevant players were carried out, and the UDS was then pilot-tested over a 3-month period with nine OMH grantees. In addition, a Technology Survey was conducted by telephone with 22 participants to assess the capacity and preference for gathering data for the UDS. Based on analyses summarized in the report, the evaluators recommended a web-based database and report generating system that would be user-friendly. In addition, it would allow the various projects around the country to enter their data into a national system, obtain needed technical support, and generate reports of data that were useful locally.


Due to the increased emphasis on data-based evaluation performance indicators, which are often difficult for community based organization (CBOs) to implement, a standard data set of performance measures was developed and proposed. The feasibility of implementation of the data set was evaluated through a pilot study.


The Office of Minority Health (OMH) funds a variety of grant and standard cooperative agreement programs, under its mandate to eliminate disparities in health for minorities. The program activities generally involve a mixture of health promotion, screening, access to services; treatment, training and education; materials development; case management; capacity-building; and development of community linkages. These programs are required to provide program evaluation data related to performance standards. At the same time, unique program contributions are often overlooked by standard reporting mechanism. The goals of the OMH UDS were to capture the unique program factors while incorporating the required uniform measures.


A Project Advisory Group (PAG) composed of representatives from Federal agencies and private organizations involved in similar efforts was organized. Current and past agency efforts to develop uniform data sets and performance measures, and major data set configurations and approaches that may be applicable to OMH were identified and reviewed. Focus groups, interviews, and site visits with current and past OMH grantees were conducted to identify specific needs with respect to a data set tailored to the unique characteristics of OMH grantees. A draft data set was developed, and a pilot test was conducted with nine grantees to assess efficacy and applicability of the data set for all OMH grantees.


The final UDS combined multiple data types, including numbers (e.g., counts of persons served or how many times a service was provided); specific results (e.g., of language interpretation) or pre- and post- tests for training sessions; and qualitative impacts using brief case studies. The results of the pilot test of the system indicated the system was easy to use and that data could be readily obtained. However, additional data collection was required at several sites in order to fully complete the forms. The major recommendation was to put the UDS on line.

Use of Evaluation Results

Implementation of a reporting system such as the UDS would enable collection of standardized data across multiple types of organizations. An Internet-based system, potentially supported by a data coordinating center, would ensure the greatest access to the system.

AGENCY SPONSOR: Office of Public Health and Science, Office of Minority Health FEDERAL CONTACT: Violet Ryo-Hwa Woo, 301-443-9923

PIC ID: 7097

PERFORMER: Systems Development Group

The Impact of the State Children’s Health Insurance Program (SCHIP) on Community Health Centers


The State Children’s Health Insurance Program (SCHIP) was enacted in 1997 to expand health insurance coverage for children. This evaluation focused on the impact of the enhanced insurance status of children served by two HRSA sponsored programs: the community health centers (CHCs) and Title V maternal and child health programs, and the impact on HRSA grantee organizations. The study included site visits; interviews with former CHC users; and an analysis of CHCs transactional encounter data.

Three groups of children were evaluated 1) children who continue using HRSA sites after enrolling in SCHIP; 2) children who are new to the HRSA sites; and 3) children who discontinued use of the HRSA sites.

Use of Evaluation Results

CHC strategies that increased SCHIP enrollment included the involvement of all site staff, especially clinicians, in encouraging SCHIP enrollment; establishing a protocol for uninsured patients; cultivating a relationship with the local SCHIP administrative staff; and follow-up on submitted applications. The recommendations for CHCs included dedicating staff to outreach and enrollment activities, instituting formal processes for referring uninsured patients to outreach staff; and conducting follow-up on SCHIP/Medicaid applications. State-based recommendations included investment in CHC-based outreach and education activities; implementation of presumptive eligibility for enrollment into SCHIP; passive re- determination after 12 months in the SCHIP program; and requiring SCHIP health plans to contract with safety-net providers.

AGENCY SPONSOR: Health Resources Services Administration FEDERAL CONTACT: Jerilynn Regan, 301-594-4283

PIC ID: 7125

PERFORMER: Center for Health Service Research and Policy, George Washington University


The study addressed two primary goals: 1) document health insurance volatility experienced by users of the health centers covered by SCHIP vs. Medicaid; and 2) determine whether and how SCHIP has affected safety net providers such as community health centers (CHCs) and Title V maternal and child health (MCH) programs. The analytic framework examined the effect of the SCHIP in areas such as parents’ ability/willingness to seek health services for their children at a CHC site or from other providers; children’s health insurance volatility; and SCHIP’s effect on CHC sites.


Over 12 million children lack health insurance coverage, and therefore may lack access to health care. This report evaluated the State Children’s Health Insurance Program (SCHIP), which was enacted in 1997 to expand the scope of state-provided services as an adjunct for children not eligible for Medicaid. The study focused on the experience of CHCs and examined three groups of children: those who continued using the Health Resources Services Administration (HRSA) program sites after enrolling in SCHIP; those who were new to the HRSA sites; and those who were previous users but were no longer visiting the HRSA site.


The analysis included quantitative and qualitative components. Site visits to CHC programs were conducted, former CHC users were interviewed, and CHC encounter data were analyzed. In addition, state and county officials, representatives of safety net providers, community-based organizations, and other stakeholders were interviewed to provide contextual information. Fourteen health centers and sixteen individual health care delivery sites in six states were evaluated. Patient encounter data was extracted from an insurance transaction database.


SCHIP had very little impact on CHC sites; SCHIP/Medicaid enrollment was slow; and once children obtained SCHIP, few lost coverage over a 12-month period. Bureaucratic obstacles to SCHIP application and factors that promoted successful application (e.g., establishing a protocol for uninsured patients) were identified. Current CHC users reported being long-time users, and few reported taking their children to other non-health centers for care. Nearly all focus group participants reported that with insurance coverage, they were less apt to delay care. Based on study findings, the report provided useful, targeted recommendations for CHCs and states.

Early Head Start Research: Making a Difference in the Lives of Infants and Toddlers and Their Families


Early Head Start provides services to low-income pregnant women and families with infants and toddlers with the goals of improving children’s development (cognitive, language, social-emotional, and health); fostering close supportive relationships between parents and their infants and toddlers; and encouraging the development of community partnerships. This report summarizes the interim results of an evaluation initiated in 1995 that includes about 3,000 children and families across 17 sites selected to represent a variety of environments (regional, rural, and urban), ethnic and racial compositions, and types of program approaches. At each site, children and families were randomly assigned to program and control groups; follow-up was carried out over three years of program participation, through a child’s third birthday. There was a consistent pattern of statistically significant (although modest) favorable impacts across a range of outcomes related to children and their families. The results found at the interim evaluation (conducted at age 2) were sustained through the child’s third birthday.


This evaluation carried out the recommendation of the Advisory Committee on Services for Families with Infants and Toddlers to include an evaluation component within the Early Head Start program that would facilitate continuous improvement. The 17 programs selected for evaluation were geographically diverse, and represented all of the major programmatic approaches under the Early Head Start program.


Early Head Start was first funded in 1995, and the performance standards of Head Start were applied to this extension of these two-generation programs. The programs provide services to low-income pregnant women and families with infant and toddlers through a wide variety of program options. Individual sites have the latitude to create their own programs; however, these programs must meet the requirements to provide child development services directly and support child development through parenting and or family development services.


The evaluation of Early Head Start was carried out in 17 selected local programs across the country that agreed to randomly assign eligible families to either the program or to a comparison group. Sites were selected to ensure geographic and program diversity. Center-based, home-based, and mixed approaches to service delivery were examined. Data sources included periodic parent interviews; field interviewers’ child and family assessments; Head Start Family Information System data; site visit observations; and program staff ratings of families’ engagement. Implementation data were collected in three rounds of site visits. A panel assessed the degree of implementation independently and for different areas.



The level of implementation (i.e., early, late, or incomplete) was associated with stronger results. Although most Early Head Start families received some services, fewer than half were involved intensively for the full period of eligibility. Nevertheless, program families and children generally received substantially more services than those in the comparison group. With respect to outcomes, positive impacts of Early Head Start were observed on children’s cognitive development, language development, and several aspects of socio-emotional development.  In addition, the assessment of Early Head Start parents demonstrated some favorable impacts on parenting outcomes (e.g., reading daily to the child) and on some aspects of progress towards self-sufficiency (education, job training activities). Early Head Start mothers were less likely to have subsequent births within two years.

Use of Results

Results of the evaluation suggest that implementing key services fully is important for program success. Specific recommendations for center-based and home-based services were provided, and it was noted that flexibility in service options seems to have the greatest positive impact. Two ‘difficult-to-serve’ subgroups (parents at risk for depression and teenage parents) were reached by the program. The study validated the importance of meeting Head Start Program Performance Standards and supported the value of monitoring programs regularly.

AGENCY SPONSOR: Administration for Children and Families, Administration on Children, Youth and Families

FEDERAL CONTACT: Rachel Chazan Cohen, 202-205-8810 PIC ID: 3570

PERFORMER: Mathematica Policy Research, Inc., Princeton, NJ

The Impact of Welfare Reform on Children: The Indiana Welfare Reform Evaluation


This evaluation was part of a five state evaluation of welfare reform’s impact on children. An in-home survey was conducted over a nine-month period during 2000. The survey was administered to 1,679 single parent families with children between the ages of 5 and 12. An additional set of outcomes for 1,126 adolescent children (between 13 and 17 years old) was also presented. The survey addressed child outcomes related to educational performance, social and emotional adjustment, and health. Other aspects in the children’s environment that were potentially affected by welfare reform measures, such as childcare and family stability, were also measured.  Groups were randomly assigned to either the traditional welfare environment or the welfare reform program.  The welfare reform groups were encouraged to find work as soon as possible, with a 24-month time limit imposed on receipt of Temporary Assistance for Needy Families. Personal responsibility provisions were also incorporated.

The welfare reform program did not appear to have a significant impact, either positive or negative, on elementary school-age children or on adolescents.


This evaluation was part of a five state study (the Project on State-Level Child Outcomes) that examined the effects of welfare reform on children; this evaluation focused on reforms in Indiana. Differences in outcome measures for two groups— one randomly assigned to participate in Indiana’s welfare reform program and the other assigned to a traditional welfare environment — were examined.


Federal and state welfare reform programs, particularly those that emphasize a ‘Work First’ approach, are aimed at adults. However, it was recognized that the policies were likely to have an effect on the children of welfare recipients. Critics of reform are concerned that there may be a decrease in parental supervision and a decline in the quality of parenting due to single mothers returning to work. Home environments may deteriorate if benefit reductions were not made up by earnings gains. Proponents of reform believe maternal employment may have positive benefits due to increases in family income and more regular family routines. In addition, it is believed the personal responsibility provisions included in the welfare reform policies would lead to better quality parenting.



An in-home survey that lasted two hours was conducted with a representative statewide sample of single-parent families with children who had entered the welfare reform evaluation in its first year. A seventy percent response rate was obtained, and statistical adjustments were made to control for non- response bias. Primary results were based on a sample of families with a focal child between 5 and 12 years of age; results were also presented for a sample of adolescent children between 13 and 17 years of age. The primary analysis method was regression, adjusting for (modest) baseline differences and for a set of baseline characteristics.


Findings indicated limited effects for elementary school-age children; for instance, there was no  significant program impact on school performance, level of engagement in school, or receipt of special education. There were no effects on the measures of social behavior and emotional well-being, nor were there differences between the welfare reform group and the traditional welfare group in terms of children’s health status, frequency of injuries or accidents, or medical and dental care. Fewer outcome variables were measured for adolescents. Although negative effects of welfare reform were reported for parental ratings of adolescents’ school performance, no differences were observed in other areas, such as school dropout rates and teen births. Moreover, welfare reform was not associated with differences on measures of the home environment (e.g., cognitive stimulation, emotional support); children’s involvement with nonresident fathers; or most measures of household stability. Additional findings showed that children in the welfare reform condition spent somewhat more time in child care, and that welfare reform was associated with lowered maternal depression and a small but significant reduction on a parenting aggravation scale. The families continued to face significant challenges; most remained financially insecure with multiple family problems in both groups.

Use of Evaluation Results

The findings were consistent with other evaluations of the impact of welfare reform on children. There were generally few effects on elementary school children, and some potentially negative effects on adolescent children. The authors suggested that increasing household income, through earnings disregard or other approaches, may help with other family problems associated with single parent families on welfare.

AGENCY SPONSOR: Administration for Children and Families; Administration on Children, Youth and Families

FEDERAL CONTACT: Alan Yaffe, 202-401-4537 PIC ID: 6847

PERFORMER: ABT Associates Inc.

Final Report on Vermont's Welfare Restructuring Project


The Vermont Welfare Restructuring Project (WRP) was initiated in 1994 and continued through 2001. Requirements of the program included a 30-month time limit on cash assistance before work in a wage- paying job was required. Small financial incentives were also offered to encourage work.  An evaluation was conducted of applicants and recipients of welfare by assigning them randomly to the WRP or to the Aid to Needy Families with Children groups which operated under the prior welfare rules. A third group received the financial incentives but was not subject to the work requirement.

The WRP program appeared to increase employment, save taxpayer money, had little effect on income, and did not require creation of community service positions.


States have new flexibility in creating their own rule for governance of welfare programs due to the conversion of federal welfare funding into a fixed block grant. States developed a variety of approaches to welfare reform. Vermont’s approach had more lenient standards than other programs; the intent of this evaluation was to determine if less stringent standards still resulted in an increase in employment and reduced welfare receipt.


Since it preceded Temporary Assistance for Needy Families by two years, the Vermont provisions for welfare reform differed from those of most other states. In Vermont, all welfare applicants from July 1994 through December 1996 were randomly assigned to one of three treatment groups. Members of the Welfare Restructuring Project group were subject to time-triggered work requirements (required to work in wage-paying jobs after receiving cash assistance for 30 months) and financial work incentives (enhanced earnings disregard, and being able to hold more savings and personal effects). This was the final report on Vermont’s welfare reform program, with previous reports having been provided at 21 and 42 months. The evaluation covered an eight-year time period, from 1994 through 2001.


The final outcome evaluation was based on a variety of data sources, including baseline data from a background information from and private opinion survey, field research, staff surveys, administrative records, and program expenditure data. While all welfare recipients from July 1994 to December 1996 were randomly assigned to one of three conditions, this impact study was based on a subset of the cases assigned in the research districts from July 1994-June 1995. This “report sample” represented about 70 percent of the full research sample (N=7,691), of which approximately one third were surveyed for a 42- month (post assignment) assessment, and approximately 80 percent (N=1,872) responded.


Selected findings indicated that the full WRP program increased employment and reduced reliance on  cash assistance for single-parent families, but had little effect in areas such as material hardship, children’s school performance, or other family and child outcomes. Results also showed that increases in employment and income were largest for the most disadvantaged members, and that the net cost of the program was low and more than offset by reduced public assistance payments. There were few program effects for two-parent households.

Use of Evaluation Results

This evaluation helps inform the debate over the best approach to welfare reform.  As one of the earliest statewide welfare reform programs initiated under waivers, it provides a longer time period of evaluation. However, the contribution of the healthy economy to the program’s success is not yet clear.

AGENCY SPONSOR: Administration for Children and Families FEDERAL CONTACT: Girley Wright, 202-401-5070

PIC ID: 6764

PERFORMER: Manpower Demonstration Research Corporation

How Effective are Different Welfare-to-Work Approaches? Five-Year Adult and Child Impacts for Eleven Programs


Eleven mandatory welfare-to-work programs were evaluated for their long-term effects on welfare recipients and their children. This multi-year study of alternative approaches to helping welfare recipients find employment concentrated on answering the question of ‘What works best, and for whom’.  Welfare recipients were randomly assigned to one of two or three research groups, depending on the site. Three sites conducted two different programs simultaneously.  One site used a three-group random assignment design (to two case management models or to a control group). The remaining three sites used assignment to existing programs versus no program enrollment as the comparison. All welfare-to-work programs increased participation in employment-related activities, but had little impact on total combined income (as earnings were substituted for public assistance). Employment–focused programs generally produced larger employment gains than educational programs.


The intent of this multi-program evaluation was to determine what alternative approaches to welfare reform help welfare recipients find and hold jobs and move off of public assistance. Two broad types of pre-employment strategies were compared—programs that emphasized short-term job search assistance (‘employment-focused’ programs) and programs that emphasized basic education and skill building (‘education-focused’ programs).


Welfare reform has been legislated for the past 30 years, but the knowledge of what type of reform works best in moving people off welfare to work is still unknown. Although research in the 1980s demonstrated that provision of job search programs sped up the entry of welfare recipients into the labor market, the jobs were often short term and/or low-paying. The severely disadvantaged were not impacted. Some states began offering alternative programs that emphasized basic education and skill development.  These programs are more costly, but may be more cost-effective. The implementation of the TANF program made the question of which welfare-to-work strategy was most effective more urgent.


From 1991-1999, approximately 40,000 single parents (primarily mothers) and their children were followed for a five-year period after random assignment to either a welfare reform program or to traditional welfare groups. Eleven different programs in seven different sites across the U.S. were included in the evaluation. Three of the sites simultaneously operated two different programs – one program focused on immediate employment (a Labor Force Attachment (LFA) program) and one based on education and skills building (a Human Capitol Development (HCD) program) – expressly for evaluation purposes. Different samples were included in different analyses, but all samples exceeded 1,000 individuals.


All welfare-to-work programs increased participation in employment-related activities, but had little impact on total combined income (as earnings were substituted for public assistance). Employment– focused programs generally produced larger employment gains than educational programs. Portland, Oregon’s program, which tailored education versus employment first to client needs, outperformed the other 10 programs in employment and earnings gains, as well as in return on investment to the government. However, the researchers noted that these results might have been affected by factors such as the favorable employment climate in Portland. Control group members changed their work behavior as well: by the end of the five-year study period, most had participated in an employment- related activity, without the welfare-to-work program assistance. The welfare-to-work programs had minimal effects on health insurance coverage, family structure, or on young children’s well-being, and a small negative effect for adolescent members of program families. On the positive side, program participants were less likely to report physical abuse during the last year of follow-up. The report also provided a cost analysis showing that net costs were higher for education-focused programs and for clients who already had high school credentials. Reduction of welfare payments was offset by program costs, and only three programs—all employment-focused—produced gains to government budgets.


Use of Evaluation Results

The most cost-effective, program utilized a mixed approach, blending employment search and education or training. The challenge of the future is to identify other types of programs or initiatives that could provide welfare recipients with better and more stable jobs, increase their income, and improve the well- being of their children.

AGENCY SPONSOR: Office of the Assistant Secretary for Planning and Evaluation; Administration for Children and Families; US Department of Education

FEDERAL CONTACT: Audrey Mirsky-Ashby, 202-401-6640 PIC ID: 7089.7

PERFORMER: Manpower Demonstration Research Corporation

Chapter 2 - Compendium of HHS Agency FY 2002 Evaluations Completed

Administration for Children and Families


To promote the economic and social well-being of families, children, individuals, and communities.

Evaluation Program

The Administration for Children and Families (ACF) administers a broad range of formula and discretionary programs, including family self-sufficiency (Temporary Assistance for Needy Families); child support; children and family services (Head Start, Child Welfare, Family Preservation and Support, and youth programs); and special programs for targeted populations, such as the developmentally disabled, refugees, and Native Americans.

The objectives of ACF’s evaluations are to furnish information on designing and operating effective programs; to test new service delivery approaches capitalizing on the success of completed demonstrations; to apply evaluation data to policy development, legislative planning, budget decisions, program management, and strategic planning and performance measures development; and to disseminate findings of completed studies and promote application of results by state and local governments.

ACF actively engages with other federal agencies, state and local policy and program officials, national organizations, foundations, professional groups and practitioners, and consumers to stay current on emerging issues affecting its programs and to identify questions for evaluation studies. Systems changes and how they affect vulnerable populations, particularly children, are of primary concern. The movement toward devolving responsibility for health and human services to state and local organizations offers both tremendous opportunities and unprecedented challenges in redefining and implementing services for families.

Studies often are funded as joint ventures with the Office of the Assistant Secretary for Planning and Evaluation and other federal agencies and foundations. Such collaborations permit large-scale efforts that are better informed and more representative of varying perspectives. Proposals are reviewed by multidisciplinary experts. Evaluation study designs are carefully developed in collaboration with project partners and technical experts in order to address specific research questions. Work groups of various kinds are used to monitor the progress of projects and to advise on design refinements and the presentation of findings.

Summary of Findings

Administration on Developmental Disabilities

Profile of Three EI² Pilot Sites

The purpose of this project was to develop a preliminary conceptual framework for a realistic, community-based, and person-centered model of comprehensive and holistic long-term support services for persons with severe or multiple developmental disabilities who are entering or are currently in the workforce. The research and evaluation activities of the Economic Independence and Inclusion (EI²)

project consisted of: a review and analysis of research and pertinent federal and state policies and programs; the development of 22 case studies of past and present approaches and “best practices”; design, field testing (in three sites), and finalizing the desired model; and utilization of an expert advisory panel to review all products. The EI² Model addresses the following questions: What attempts have previously been made or are currently underway to design and implement a person-centered directed comprehensive service model that results in valued employment outcomes for the target population?

What are the common elements and unique features of these programs and the barriers they encounter? What would be a realistic model for the target population, and how can it best be implemented?

Recommendations were for: (1) each state’s Olmstead Planning Group to identify options within their state (HCB) Medicaid waiver and to create a state level working group to review state oversight plans for commitment of resources to expand opportunities for production of income; (2) commitment at state and local levels to cross agency training and capacity building; (3) additional training, research, and technical assistance nationwide to build understanding and knowledge transfer of promising practices; (4) commitments from state and local government to part- and full-time employment of the target population with job sharing and additional natural and paid supports in the workplace; and (5) additional research to monitor the impact of benefit counseling services and results achieved with implementation.

FEDERAL CONTACT: Hossein Farris, 202-205-4922 PIC ID: 7544

PERFORMER: Diana McIver & Associates, Inc., Austin, TX

Office of Planning, Research, and Evaluation

Agency for Healthcare Research and Quality


To improve the quality, safety, efficiency and effectiveness of health care for all Americans.

Evaluation Program

Evaluation activities within the Agency for Healthcare Research and Quality (AHRQ) provide executive management, program officers and audiences external to the Agency with evaluative findings concerning the Agency's effectiveness and efficiency in meeting its GPRA, PART and other performance goals. The work is conducted moreover by external, independent evaluators and complies with (OMB) Paperwork Reduction Act requirements. Evaluation is built into (beginning at the design phase) virtually all major AHRQ programmatic or portfolio activities. Among evaluation mechanisms used by the Agency are targeted evaluation studies undertaken through contracts that use a variety of quantitative and qualitative methods and that are increasingly tending toward providing more real-time monitoring feedback. Evaluation activities also include satisfaction feedback from AHRQ "customers" regarding the usefulness of AHRQ research findings and dissemination products.

Summary of Findings

Agency for Healthcare Research and Quality

An Assessment of the Centers for Education and Research on Therapeutics (CERTS) Initiative

This study provided an assessment of the progress CERTs has made in meeting its programmatic objectives. The purpose of this study was to assess the capability of the current CERTs programmatic structure to adequately address all of the objectives stipulated in the authorizing legislation.

Recommendations include the following: (1) CERTs need an effective mechanism to develop communication channels with state and local public health agencies; (2) long term communication strategy should include identification of key target audiences and the most effective means to reach those audiences; (3) additional professional editorial resources should be provided through the Coordinating Center; (4) existing public relations resources should be utilized to reach its various publics; and (5) the design and implementation of public affairs campaigns should be assessed.

FEDERAL CONTACT: Lynn Bosco, 301-427-1490 PIC ID: 7686

PERFORMER: Washington Consulting Group, Bethesda, MD


Feasibility Study for Developing a Web-based Redesigned and Enhanced CONQUEST and Integrating it with an Enhanced National Guideline Clearinghouse

This project sought to provide advice and recommendations to AHRQ about the design of a Web-based database of clinical performance measures that could be appropriately and functionally integrated with an enhanced National Guideline Clearinghouse database. CONQUEST, the second of a suite of Web-based quality improvement tools to be developed by AHRQ, is operated as a clearinghouse for clinical performance measurement protocols. The contractor evaluated the features that a clearinghouse for performance measures should embody and explored alternative approaches to CONQUEST’s design and management. The study found that the construction of a compendium of clinical performance measures was feasible and highly valuable. There was strong evidence that quality cannot be improved unless effects and outcomes are measured, and there was also evidence that measurement tends to influence the priorities of health care institutions and individual providers. The study also found that by putting these measures on the Web, AHRQ could very disseminate information about measures and maintain them so that they remain up-to-date in this rapidly changing scientific and clinical environment.

FEDERAL CONTACT: Larry Patton, 301-427-1486 PIC ID: 7010.1

PERFORMER: Lewin Group, San Francisco, CA


Implementation Planning Study for the (1) Integration of Medical Event Reporting Input and, (2) Data Structure for Reporting to Implementation [AHRQ, CDC, CMS, and FDA]

This study analyzed existing US, DHHS data systems, a select number of other systems, and factors affecting the potential integration of systems. Although efforts to measure and reduce the risk of harm to patients are underway, there was no national, centralized system for the collection and analysis of information pertaining to adverse events. Rather, an array of disparate systems collected information about different types of events (e.g., device-related, medication-related), from different sources (e.g., consumers, practitioners, manufacturers). Individually, each of these systems provided value to those who submit information and those who analyzed the collected data. However, linking data or combining databases promises efficiencies for the collection of data and could allow for early identification of systemic safety problems that would otherwise be impossible to detect. Our investigation suggested that there are several possible integration options, and this report details two specific candidates. Both approaches lent themselves to a phased implementation with the chief integration efforts occurring early in Phase 1 and expanding to other domains or user communities. One approach initially addressed the issue of “burden of reporting” for several systems, and it subsequently developed rich analytic facilities. Another approach began by integrating two systems “end-to-end,” including data entry and analysis components. FEDERAL CONTACT: James Battles, 301-427-1332

PIC ID: 7681


Center for Cost and Financing Studies

Health Services Research Education; Assessing Customer Satisfaction and Program Needs: Survey Update

This report analyzes the responses to a mailed, self-administered survey to assess customer satisfaction among trainees who participated in institutionally based and other health services research training programs funded by the Agency for Healthcare Research and Quality. The survey was designed to generate information about training program activities and needs. The survey was conducted in two waves. The first was conducted during the summer of 1999 and included 251 former trainees from the AHRQ-funded institutionally based (T-32) program. The second wave included an additional 113 former trainees. Of the 364 former trainees sent a survey, there was an 87 percent response rate (317).

Respondents represented a variety of training program types: 35 percent had been in pre-doctoral programs; 56 percent had been in postdoctoral programs, and 3 percent had participated in both types. The vast majority of respondents are currently employed (88%), with many in positions that were research-oriented or included opportunities for research (67%). Global observations about trainees satisfaction included: (1) Respondents were very satisfied with the quality of their research training, and (2) Large numbers of respondents were satisfied with many elements of the infrastructure of their programs, including the structure of the training experience, knowledge and skills learned, mentoring, stipend duration, and the academic program.

FEDERAL CONTACT: Debbie Rothstein, 301-427-1525 PIC ID: 7397

PERFORMER: Battelle Corporation, Arlington, VA

Center for Organization and Delivery Studies

AHRQ Market File: Feasibility Project Report

This project examined the feasibility of developing and maintaining a data system that could be used by researchers to study healthcare markets. The research consultant conducted an evaluation of data sources used in research on healthcare markets. That initial effort resulted in a report that identified and described over 80 different data sources that could potentially be used to conduct research on the characteristics of healthcare markets. The report described a healthcare market data system and described the feasibility of establishing such a system. The report included five chapters. Chapter 1 outlines the tasks required for system development. Chapter 2 presents staffing requirements and estimated total ‘loaded’ personnel costs for the first year of operation. Chapter 4 contains a discussion of possible methods for addressing data issues that will need to be resolved if AHRQ decides to move forward and develop the healthcare market data system. The final chapter, Chapter 5, is a discussion of recommendations regarding the development and implementation of a healthcare market data system. Recommendations included: having ready access to some subset of data; the need for wide coverage in the types of data sets that can be accessed through the system; achieving high quality data set documentation and related data resources; and assuring an intuitive and easy to use web site, a streamlined process of ordering data files, and responsive technical assistance.

FEDERAL CONTACT: Michael Hagan, 301-427-1409 PIC ID: 7484

PERFORMER: Center for Health Policy Studies, Columbia, MD


Validating the Healthcare Cost and Utilization Project Quality Indicators

In 1994, the Agency for Healthcare Research and Quality developed the Healthcare Cost and Utilization Project (HCUP) Quality Indicators (QIs) in response to the increasing demand for information regarding the quality of health care. These measures, based on discharge data, were intended to flag potential quality problems in hospitals or regions. The purpose of this project was to refine the original set of HCUP QIs (HCUP I) and recommend a revised indicator set (HCUP II). Specifically, this project aimed to: (1) identify quality indicators reported in the literature and in use by health care organizations, (2) evaluate HCUP I and other indicators using literature review and novel empirical methods, and (3) make recommendations for the HCUP II set and further research. The project deferred evaluation of indicators of complications to a separate study and report. This project identified 45 indicators that were promising for use as quality screens, demonstrating through literature review and empirical analyses that useful information regarding quality of health care can be gleaned from routinely collected administrative data. However, these indicators had important limitations and could benefit from further research. Techniques such as risk adjustment and multivariate smoothing could reduce the impact of some of these limitations, but other limitations remained. There were two major recommendations for further action and research:

(1)  the need to improve HCUP data and subsequently the HCUP QIs to address some of the noted limitations, and (2) the need for further research into quality measurement and the reality of these limitations. The HCUP QIs could benefit from the inclusion of additional data, some of which can be routinely obtained in some states.

FEDERAL CONTACT: Anne Elixhauser, 301-427-1411 PIC ID: 7671

PERFORMER: Research Triangle Institute, Research Triangle Park, NC


Vision Rehabilitation Within Models of Care and Benefit Plans

AHRQ undertook this study to: (1) assess the scientific evidence on the effectiveness of services to improve functioning of persons with vision impairment, (2) survey and analyze how rehabilitation services are included in health delivery systems and health benefit plans, and (3) identify opportunities for AHRQ, to contribute to advancement of knowledge, practices, and policy deliberations with regard to vision rehabilitation services. This report was based on an extensive literature synthesis supplemented by interviews with clinical experts, provider organizations and researchers in the field. The preliminary results were critiqued in conference discussions with researchers, service delivery leaders, and leaders of consumer organizations. The report provides some background on the etiologies of low vision, some discussion of the apparent under use of clinical preventive services and treatments to halt or slow progression of disease. The main focus was on rehabilitation services that were poorly coordinated with clinical providers and relied on scattered sources of financing. The limited likelihood for cure or rehabilitation for many case of vision deterioration led the study team to suggest that stronger prevention guidelines be discussed by the Preventive Services Task force, and that AHRQ encourage greater patient awareness of benefits of early detection of eye disease, which in many cases could benefit from early intervention. However, for the rapidly growing incidence of age-related macular degeneration more progress in knowledge about prevention and intervention is needed.

FEDERAL CONTACT: Bernard Friedman, 301-427-1404 PIC ID: 7495

PERFORMER: Lewin Group, San Francisco, CA

Center for Outcomes and Effectiveness Research

Innovative Approaches to Training Clinicians for Bioterrorist Attacks: Final Report

The goal of this project—which included an advisory committee of clinicians, medical educators, health systems leaders, and others—was to develop two prototype approaches for training clinicians to recognize and respond appropriately to a possible bioterrorist attack. Early in the project, a two-day workshop was held with the Technical Advisory Committee to discuss the bioterrorism threat and develop approaches for training clinicians for bioterrorism events. At that time, the Committee considered uses of biologic agents as a bioterrorist attack an unlikely, not even a rare, event. The events of the Fall of 2001 markedly changed the nation’s views, and the health community’s opinions about bioterrorism preparedness. New information about bioterrorism disease recognition, treatment, and precautionary measures has evolved with each new patient and event. Two prototype approaches were developed for training clinicians to recognize and respond appropriately to a possible bioterrorism attack. The first approach was a website that has four main segments: the home page, bioterrorism agents, emerging infectious agents, and notifications and key web links. The second approach involved creating simulated bioterrorism patients using virtual reality technologies previously developed by the research contractor.

The Web-based learning materials and the simulation-based learning and evaluation materials were assessed by practicing clinicians (in primary care or infectious disease). The clinicians endorsed the broad approach taken by the website providing historical and clinical information for emerging infections as well as bioterrorism diseases. In our usability evaluation, users testing the software ranked it moderately high to very high.

FEDERAL CONTACT: Carolyn M. Clancy, 301-427-1200 PIC ID: 7680

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Center for Practice and Technology Assessment

Evidence Report/Technology Assessment: Criteria for Determining Disability in Speech-Language Disorders

The study included a systematic review of the literature to address two key questions about evaluating and diagnosing speech and language disorders in adults and children of concern to the Social Security Administration in making disability eligibility determinations: (1) What instruments have demonstrated reliability, validity, and normative data? (2) Do these instruments have predictive validity for an individual’s communicative impairment, performance? Approximately 42 million Americans have some type of communication disorder, which annually costs the nation $30 billion to $154 billion for lost productivity, special education, and medical care. The quality of the numerous evaluation procedures and instruments for clinical decision-making about language, speech, or voice disorders influences decisions about access to services and funding (e.g., special education services, Social Security disability income). The study found the following: (1) Reliability and validity data for the majority of instruments rarely came from peer-reviewed literature; instrument manuals yielded most such data. (2) Some manuals provided comprehensive data from well-conducted standardization studies; most did not. (3) Because normative data were usually not derived from nationally representative samples, generalizing results beyond the populations studied was difficult. (4) Sample size and representativeness problems limited the predictive validity studies. (5) Evidence about diagnostic or predictive properties of instruments addressing language, speech, and voice disorders is weak and incomplete at this time. The sparse evidence base suggested an outline and need for advancing a substantial methodological, clinical, and policymaking research agenda.

FEDERAL CONTACT: Kevin Murray, 301-427-1853 PIC ID: 7688

PERFORMER: Research Triangle Institute, Research Triangle Park, NC


Systems to Rate Strength of Scientific Evidence

As a key part in its strategy for meeting its legislative mandate, AHRQ undertook a systematic review and analysis of methods of rating the quality of scientific studies. The Healthcare Research and Quality Act requires that AHRQ, in collaboration with experts from the public and private sector, identify methods or systems to assess health care research results, particularly “methods or systems to rate the strength of the scientific evidence underlying health care practice, recommendations in the research literature, and technology assessments.” The University of North Carolina’s Evidence-based Practice Center identified 19 generic systems that fully addressed their key quality domains and identified seven systems that fully addressed their three domains for grading the strength of a body of evidence. The report also provides a research agenda including questions relevant for assessing the quality of scientific evidence.

FEDERAL CONTACT: David Introcaso 301-427-1213 PIC ID: 7676

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Center for Primary Care Research

Assessing the Information Technology Infrastructure in Integrated Delivery Systems

There is evidence of increased investment in information technology (IT) by the health care industry, but little is known about how integrated delivery systems (IDSs) acquire, implement, use, and evaluate IT. How extensive is the use of IT in IDSs, and what capabilities do IDSs have to integrate data from different IT system components? How are the special needs of vulnerable populations addressed using IT systems? To address these questions and others, the Agency for Healthcare Research and Quality funded the University of North Carolina’s IDS Research Network. Information was obtained about the IT infrastructures of the six IDSs in the RTI-UNC IDSRN through a replicated series of in-depth case studies that involved a series of six two-day site visits. The study found that decisions to adopt, and subsequent implementation of, these systems in IDS settings are extremely complex processes. The financial investment and cultural implications for implementing this type of innovative clinical process are different from those typically encountered with traditional IT systems developed for billing or registration  purposes. Key observations from the analysis of case study data suggest that: (1) with experience, IDSs moved toward centralized decision making, (2) IDSs took a phased approach to implementation that involved sequential installation of IT systems across care delivery sites and maintenance of parallel data entry, storage, and retrieval systems, and (3) there was a necessity for organizational infrastructure (i.e., sufficient staffing, dedicated IT department) to support implementation and maintenance of IT solutions.

FEDERAL CONTACT: Kelly Morgan, 301-427-1570

PIC ID: 7674

PERFORMER: Research Triangle Institute, Research Triangle Park, NC


Evaluating AHRQ’s Low-Income Research Portfolio, 1989-2000: Final Report

The purpose of this report was to provide an understanding of the breadth and depth of AHRQ’s low- income populations research portfolio and to identify potential directions for continued research in this area for the next decade. This report was the final deliverable for a project representing the first phase in low-phase evaluation of AHRQ’s low-income research portfolio. This phase of the evaluation answered the following two questions: (1) what contributions has AHRQ’s sponsored research made to knowledge about health care for low-income populations and (2) comparing AHRQ’s research portfolio to the body of literature in the field, in what areas has the agency made the greatest contributions and in what areas have agency investments been low? This report presented findings from the first phase of the evaluation. Studies of access to prenatal and obstetric care found that it is possible to improve provider participation in obstetrics for Medicaid patients with a comprehensive Medicaid expansion program. Also, low income is associated with significantly elevated risks of untimely prenatal care, but the increased risk of untimely care is not confined to women in absolute poverty or to those lacking private insurance. Studies that focused on pregnancy outcomes found that density of obstetricians appeared to have no impact on pregnancy outcomes. Also, heightened levels of stress were not associated with earlier deliveries. Studies focused on prenatal quality of care found that patients’ ideas of what constitutes quality in prenatal care did not appear to differ by ethnicity. In another study, satisfaction also did not appear to vary according to personal characteristics.

FEDERAL CONTACT: Robin Weinick, 301-427-1573 PIC ID: 7687

PERFORMER: Center for Health Policy Studies, Columbia, MD

Office of Healthcare Information

Tracking Research Impact: Final Report

This report provided a conceptual framework for understanding the research process and used the framework to synthesize information collected from structured discussions with staff from the Agency for Healthcare Research and Quality, the Veteran’s Health Administration and four nonprofit health services research organizations. The report describes the efforts of each organization to collect, translate and disseminate research findings and assesses their impact on the knowledge pool, policy and practice. Key findings and recommendations are that AHRQ should: (1) consider improving patient outcomes as its principal indicator of research impact and adopt “priority conditions” as a fundamental organizing   principle for its ongoing research process, (2) develop a more structured process for obtaining feedback from end users of research and consider increasing funding for projects designed specifically to help the agency synthesize larger bodies of research results, and (3) foster a closer and more deliberate interaction between the Office of Health Care Information (OHCI) and the project officers and Center   staff to improve dissemination efforts. Most organizations studied do not systematically use the results of current and past research efforts to explicitly determine priorities and allocate funding for the next round of research. Rather, they target their research at achieving a different type of impact--influencing health care policy or improving clinical and delivery system performance.

FEDERAL CONTACT: Sandra Isaacson, 301-427-1253 PIC ID: 7494

PERFORMER: Lewin Group, San Francisco, CA

Centers for Disease Control and Prevention


To promote health and quality of life by preventing and controlling disease, injury, and disability.

Evaluation Program

The Centers for Disease Control and Prevention (CDC) conducts evaluations designed to provide information essential for CDC’s strategies to achieve its mission:

  • Conduct public health research, including epidemiology, laboratory, behavior, and social sciences.
  • Develop and implement ongoing evaluation of scientific research.
  • Assure that scientific information is communicated to the general public.
  • Develop and implement public health information systems for monitoring and promoting he health of the Nation.
  • Deploy multidisciplinary teams to detect and investigate health threats.
  • Develop and implement a system for establishing CDC scientific and programmatic priorities.
  • Routinely evaluate the effectiveness and cost of CDC programs.
  • Demonstrate the value of investment in health prevention.
  • Collaborate with diverse partners.
  • Design, implement, and evaluate prevention programs based on community needs.
  • Prepare the public health leaders (present and future) through training in management and public health science.

CDC places high priority on evaluations to answer policy, program, and strategic planning questions related to the goals and objectives of Healthy People 2010. Performance improvement studies, such as those focusing upon the development of indicators consistent with the Government Performance and Results Act (GPRA), are of interest and import to the Agency. With the support of 1-percent evaluation funds, the GPRA planning process began at CDC in FY 1995 and is continuing.

Summary of Findings

Centers for Disease Control

Evaluation of Expanding Pharmaceutical Data in the National Health Care Survey

This project evaluated current methods used in the National Health Care Survey to collect pharmaceutical data in the National Hospital Discharge Survey (NHDS). This study is the first step in examining the feasibility of including pharmaceuticals and had implications for the potential to include additional clinical data in the NHDS. During Phase I of the project, various practical issues of applying National Hospital Ambulatory Medical Care Survey (NHAMCS) methods to the NHDS were studied, and field procedures (forms, manuals, training) were developed to be used in th4e Phase II field test. Phase II consisted of a field test of procedures to collect the names of drugs administered to inpatients and to evaluate the use of NHAMCS methods as applied to the NHDS. Results of the study support the use of a NHAMCS type method to collect pharmaceutical data for inpatients in the NHDS. The study overview includes a list of next steps envisioned by NCHS.

FEDERAL CONTACT: Robert Pokras 301-458-4439 PIC ID: 7745

PERFORMER: CODA Inc., Silver Spring, MD


Formative Evaluation of Physician Practices Regarding Prostate Cancer

The purpose of this research project was to better understand physician behavior related to prostate cancer screening practices, factors affecting these practices, and the feasibility of using educational materials on this topic in clinical practice. The project explored whether and why physicians recommend prostate cancer screening to their patients, what physicians tell men about screening, why they do or do not discuss the pros and cons of screening with patients, and whether education materials regarding screening are relevant. Prostate cancer is the second leading cause of cancer deaths in men. Although screening tests and treatment exist, there is no clear evidence that treatment for prostate cancer reduced mortality. Professional and medical organizations have produced conflicting and confusing recommendations for clinicians. Many guidelines either advise no prostate cancer screening or recommend that physicians give men information on the pros and cons of screening and encourage men to decide whether or not to screen. Despite the controversy, limited survey data indicate that may clinicians continue to recommend prostate cancer screening. Battelle did not develop findings for this project. CDC will analyze the coded data and develop the findings.

FEDERAL CONTACT: Nancy Cheal, 404-639-7222 PIC ID: 7853

PERFORMER: Battelle, Seattle, WA

National Center for Chronic Disease Prevention and Health Promotion

Evaluation of Public Health Care Providers’ Training, Screening, and Referral Practices for Pregnancy-Related Violence

This study focused on clinics that provided family planning services because healthy women who may not have other regular medical care typically used these services. The purpose of this study of family  planning clinic managers and clinicians was to determine awareness and availability of clinical guidelines for identifying women who experience intimate partner violence and to assess current practice and perceived barriers to identifying and referring abused clients identified in clinics that provided publicly- funded family planning services. Physical, sexual, and emotional violence against women is increasingly recognized as an important public health issue. An estimated 1.8 million women (3% of all women) are severely assaulted by male partners in the United States each year. The Survey of Family Planning Clinics on Intimate Partner Violence was designed to obtain information from a nationally representative sample of Title X funded clinics in the continental United States. Clinicians were asked about how their clinic addressed the issue of intimate partner violence, about violence training they had received, about their own practices with patients regarding intimate partner violence, attitudes about screening for violence in the health care setting, availability of community resources, and personal characteristics. Preliminary findings included the following: (1) most of the clinics in the sample were affiliated with health departments (58%) and were located in rural settings (60.1%); (2) clinic informants tended to be female (96.7%), Caucasian (87%), middle-aged, and were directors of their respective clinics; (3) clinic directors reported that 83.3 percent of clinics routinely screen for violence against women at intake exams and 78.5 percent screen during periodic visits; and (4) forty five percent of clinics offered formal trainings for their staff and 36.7 percent provided training opportunities elsewhere.

FEDERAL CONTACT: Mary Goodwin, 770-488-6232 PIC ID: 6712

PERFORMER: Battelle Corporation, Arlington, VA


Evaluation of the CDC Cardiovascular Health State Health Program: Final Report

This contract called for: creating a logic model to guide activities of cardiovascular health (CVH) state programs; conducting evaluation training; developing an evaluation framework; and conducting a meeting with Comprehensive State program staff to discuss the Evaluation Framework and State evaluation activities. In 1998 Congress made funded a national, state-based cardiovascular disease prevention program starting with eight states and it increased funding for the CVH State Program each subsequent year. The CVH State Program was designed to increase the leadership of State health departments in cardiovascular health promotion, cardiovascular disease prevention and control, and expand and direct efforts to establish a national CVH program. The outcomes indicate that participants felt that the logic model provided them with a clear picture of Federal expectations for the CVH State program and evaluation activities. The participants felt that the program evaluation training was useful and it provided a good introduction to conducting evaluations. The content of the Evaluation Framework was well received by the States and they reported that the content was useful for long-term outcomes, but suggested including more expectations for short-term and intermediate outcomes.

FEDERAL CONTACT: Sheree Williams, 770-488-8007 PIC ID: 7703

PERFORMER: Macro International, Inc., Calverton, MD

National Center for Environmental Health

Evaluating a Free Vitamin Supplement Distribution Program in State-Funded Family Planning Clinics

The purpose of this evaluation was to assess the impact of the Georgia Folic Acid Initiative (GFAI) on family planning clients’ knowledge about the benefits of folic acid, their attitude toward supplementation, and their use of folic acid supplements. Six Title X County Family Planning Clinics participated in the evaluation. Three clinics provided educational materials and super-fortified cereal and one clinic provided educational materials only. A survey of women aged 18 to 45 who use the clinics was conducted.

Additionally focus groups were conducted to gather data regarding: (1) participants’ experiences with   the GFAI, (2) barriers and facilitators for implementing the GFAI, (3) factors that encouraged or discouraged clients’ use of free folic acid supplements, and (4) recommendations for improving the intervention. Four focus groups were conducted with clients who participated in the evaluation and two focus groups were conducted with staff from nine Georgia family planning sites participating in the GFAI. Focus group participants largely viewed the GFAI as a successful and beneficial intervention. Client and staff participants appreciated that folic acid supplements were provided free. As a daily supplement, folic acid pills were preferred to cereal. Most of the clients in the focus groups reported taking at least some of the pills given to them. However, very few took folic acid pills on a daily basis.

FEDERAL CONTACT: Katherine Lyon-Daniel, 404-498-3965 PIC ID: 7042/7748

PERFORMER: Battelle Corporation, Arlington, VA

National Center for HIV, STD, and TB Prevention

Discharge Planning and Continuity of Care for HIV-Infected State Prison Inmates as They Return to the Community: A Study of Ten States

The purpose of this study was to examine discharge planning programs for HIV-infected inmates in ten states. The study’s goals were to describe the discharge planning systems for HIV-infected releases currently in place in the ten states, and the key policy and programmatic elements that influenced continuity of care for this population. Currently, many HIV-infected inmates do not have access to transitional services for continuity of care, and many states do not designate resources for this purpose. Failure to provide inmates with transitional support may have public health and public safety consequences. A total of 96 respondents--representing correctional health care departments, parole agencies, state and local public health departments, and community based organizations--participated in the study. The study found that services for HIV-infected inmates were often fragmented because Federal and state benefits programs did not cover individuals served by other government programs.

Incarcerated individuals were under the care of the department of corrections and therefore generally could not apply for or receive services until after they had been released. Many community service providers also had eligibility criteria (e.g., current homelessness) that made it difficult for inmates to develop a transitional plan before release.

FEDERAL CONTACT: Karina Rapposelli, 404-639-3641 PIC ID: 7749

PERFORMER: Program Support Center, Rockville, MD


Evaluation Adherence to CDC’s Sexually Transmitted Diseases (STD) Treatment Guidelines

The objective of this project was to: evaluate the impact of STD Treatment Guidelines in public and private settings, identify factors associated with compliance with the guidelines, and assess the relationship between adherence to the guidelines and the associated health outcomes. Plans were developed to assess adherence to treatment for one bacterial STD [acute, uncomplicated infection with Chlamydia trachomatic (CT)] and one viral STD (genital warts, a manifestation of infection with human papilloma virus). The task order was supplemented in 1999 to conduct four focus groups of providers at HealthPartners (HP) about knowledge, attitudes, and practices concerning CDC and other guidelines for STD treatment and treatment of CT and genital warts. The focus groups’ findings were intended to guide the development of the survey and to help interpret the survey findings. The findings indicated that adherence to CDC-recommended treatment for uncomplicated CT infection in these two medical care organizations was very high during the study period. Adherence to CDC recommendations for the treatment of genital warts was also very high.

FEDERAL CONTACT: Nancy Cheal 404-639-7222 PIC ID: 7047

PERFORMER: The HMO Group, New Brunswick, NJ


HIV Vaccine Efficacy Trial Site and Community-Level Factors Associated with Enrollment, Retention, and Protocol Compliance

This project evaluated how trial site and community-level variables were associated with vaccine trial enrollment, retention, and protocol compliance using the VaxGen trial data and additional data collected as part of this project. The primary objectives of this evaluation were to: (1) identify individual-, site-, and community-level factors associated with vaccine trial enrollment, retention, and protocol compliance,

(1) establish evaluation criteria that will inform the development and selection of future HIV vaccine efficacy trial sites, and (3) identify “best practices” and provide recommendations that will inform the conduct of future HIV vaccine efficacy trials including trial-site development and selection. It was found that sites having mainstream and gay media coverage enrolled participants reported a higher level of unprotected sex at baseline compared to sites without such media coverage. Recommendations include the following: (1) media events and opinion leader endorsement were associated with the enrollment of higher-risk efficacy trial participants. (2) Trial sites should work with the media and community opinion leaders to ensure that balanced information about trials is disseminated. (3) Alternatives to advertising such as outreach should be considered for enrolling younger and minority participants at risk of infection.

FEDERAL CONTACT: Bradford Bartholow, 404-639-6164

PIC ID: 7850

PERFORMER: Research Triangle Institute, Research Triangle Park, NC


Phase IV: National Physician Survey of STD Diagnosis, Treatment and Control Practices

The purpose of this study was to measure STD diagnosis, treatment, and control practices among a nationally representative sample of physicians in five specialties that diagnose most STDs: obstetrics and gynecology, family and general practice, general internal medicine, pediatrics, and emergency medicine. The objective was to provide the baseline data necessary to characterize infection control practices, especially partner notification practices for syphilis, gonorrhea, HIV, and chlamydia, and to measure the contextual factors that influence those practices. The survey data helped CDC to better focus STD control and partner notification program efforts and to allocate program resources. Completed surveys were received from 4,226 physicians. The STD contact survey data was used by CDC to describe and assess STD diagnosis experience, STD management and treatment practices, and STD partner management. It was also used to identify differences between subgroups of physicians in STD diagnosis, management and treatment practices, and STD reporting and partner management practices. It identified factors such as physician specialty, practice characteristics, and physician beliefs and attitudes that are associated with STD management, treatment, reporting, and partner notification practices. Finally, it summarized protocol, education, practices, and needs regarding STD control in a national sample of physicians.

Completed surveys were received from 4,226 physicians. The cumulative response rate was 70.2 percent. Respondents came from all 50 states and the District of Columbia, with a balanced regional distribution.

FEDERAL CONTACT: Janet St. Lawrence, 601-632-5636 PIC ID: 7054

PERFORMER: Battelle Corporation, Arlington, VA


Telephone Survey Measuring HIV/STD Risk Behavior: Pretest Final Report

This is the second part of a methodological investigation that was designed to determine the effectiveness of a survey administration procedure--Telephone-Audio Computer-Assisted Self Interviewing (T-ACASI)- to collect sensitive information on sexual behavior, drug use, and HIV testing status. Under sections of a survey interview administered via T-ACASI, a telephone-facility-based computer reads the survey question and records the respondent’s keypad-tone-based response, bypassing the need for human interviewer involvement. The focus of the current investigation was sensitive sections of several questionnaires that were appropriate for general-population administration. Three issues were examined:

(1) item non-response due to refusals, (2) the frequency with which sensitive behaviors were reported, and (3) respondent self-report concerning comfort and truthfulness within the survey environment. It was found that item non-response was infrequent, leading to the conclusion that respondents who completed the interview were not at all resistant to answering the sensitive survey questions from the HIV/STD Standard Set. Almost no one reported engaging in risk behavior, leading to the conclusion either that the survey did not reach a high-risk population; that it did so but that high-risk individuals selected out through survey non-response; or that it reached high-risk individuals, but they simply chose not to report such behaviors.

FEDERAL CONTACT: John Anderson, 404-639-2046 PIC ID: 7775

PERFORMER: Research Triangle Institute, Research Triangle Park, NC


The Development of a National HIV/AIDS Prevention Intervention Taxonomy for Program Evaluation

This project: (1) identified known taxonomies of interventions, (2) collected and organized categories of interventions used in CDC-funded health departments, (3) convened a group of experts from a variety of applicable fields and backgrounds to discuss the feasibility of a systematic taxonomy of HIV/AIDS interventions that can be integrated into and used by local, state, and national groups and organizations for program development, improvement, and evaluation, and (4) suggested a chronology of necessary activities CDC and its partners could undertake to develop a common vocabulary of HIV prevention interventions. Without a standard taxonomy of HIV prevention interventions, CDC and its prevention partners struggle to scientifically determine how CDC funding is affecting designated and whether CDC- funded programs are having the intended impact on the HIV/AIDS epidemic. The more consistent CDC and its partners were in classifying a type of HIV/AIDS intervention program or strategy, the more successful CDC would have been in providing higher quality data to Congress, federal agencies, state and local health departments and other constituencies for decision-making regarding funding allocation and targeted program development, improvement and evaluation. This is especially critical in making the link between prevention services provided and impacts on the epidemic assessed through surveillance systems and other data sources. The final report summarized the methods and results for each of the activities and concluded with recommendations synthesized from these findings.

FEDERAL CONTACT: Gary Uhl, 404-639-0950 PIC ID: 7707

PERFORMER: Macro International, Inc., Calverton, MD

National Center for Health Statistics

Evaluation of Non-response to the National Health and Nutrition Examination Survey

The objective of this study was to reduce the non-response bias in the National Health and Nutrition Examination Survey (NHANES). The central foci of this program assessment were the conduct of interviews and/or focus groups of respondents and non-respondents to evaluate potential reasons for response and non-response. The results of the ESP NHANES study mirror results from similar non- response studies. Sample members who chose to participate in the study were generally pleased to be a part of what they viewed as an important study. They were receptive to providing data on their health status and appreciated the opportunity to help society. The free medical examination was perceived as a very tangible benefit and many were surprised to learn they would also be paid for completing the interview and examination process. In contrast, non-respondents were more wary of the request for participation. Even aspects of the study meant to alleviate concerns or encourage participation were met with suspicion. Non-respondents also expressed great reluctance to commit the time necessary to complete the interview process. A number of recommendations were offered by the researchers, including: (1) providing greater detail about the study in the advance letter, (2) providing tailored interviewer training on refusal avoidance, and (3) developing a higher profile for the National Centers for Health Statistics, which tends to be an unfamiliar federal agency.

FEDERAL CONTACT: Vicki Burt, 301-458-4127 PIC ID: 7435

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

National Center for Infectious Disease

Definition of Public Health Laboratory Core Capacities for Infectious Disease Surveillance: Final Report

The purpose of this study was to ease the process of developing a consensus definition of core capacities for infectious disease surveillance starting with public health laboratories, public health units, and core capabilities for infectious disease surveillance. Health departments have used grants from the Centers for Disease Control and Prevention to hire and train additional epidemiologists and laboratorians, acquire laboratory technology, and upgrade communications and information technology. Assessing the adequacy of these investments has been constrained by a lack of consensus as to what state and local health departments required to conduct infectious disease surveillance. The consensus-building workshop format implemented in this study proved effective in eliciting constructive discussion among a diverse array of stakeholders regarding surveillance for drug-resistant Streptococcus pneumoniae. Workshop participants were generally supportive of the format as a mechanism for discussing and arriving at consensus regarding surveillance options. The report concludes that the consensus-building workshop methodology would likely prove as effective in similar consensus-building discussions of the set of core capacities required for other emerging infectious disease surveillance systems.

FEDERAL CONTACT: Deborah Deppe, 404-639-4668 PIC ID: 7763

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

National Center for Injury Prevention and Control

Evaluation of the PICARD Software - Final Evaluation Report

The goal of this evaluation was to determine the usefulness, user friendliness, and feasibility of using Preventable Injuries, Costs, and Related Deaths (PICARD) software to generate state data on preventable injury and associated costs by states. It was also used to determine if the use of PICARD contributed to the formation of new bicycle helmet promotion programs, policies, or laws, and if National Center for Injury Prevention and Control should develop additional PICARD modules for other priority injury prevention topics. The vendor piloted tests in five states that were previously funded for bicycle helmet programs and evaluated the first module of the PICARD software focusing on bicycle helmet-preventable head injuries. The state environment for injury prevention, which provided the context for the introduction of PICARD into a state, included the network of stakeholders in injury prevention, bicycle safety programs and legislation, and data resources. All site visit states had numerous bicycle safety programs at the state and local level. These programs involved bicycle helmet promotion, skills building, law enforcement and awareness, infrastructure improvements, and driver awareness of bicyclists. It was concluded that the current version of the PICARD software was a user-friendly, useful tool that might have broader application to other preventable injuries. Further development of PICARD could provide public health professionals with a tool for estimating the incidence and direct and indirect costs of many types of preventable injuries. When these estimates are presented to legislators and other policy and decision-makers in injury prevention, they can facilitate design and implementation of intervention to reduce the occurrence of preventable injuries.

FEDERAL CONTACT: Nancy Cheal, 404-639-7222 PIC ID: 7743

PERFORMER: Battelle Corporation, Arlington, VA


Links Among Different Forms of Adolescent Violence

The purpose of this review was to lay the groundwork to better understand the prevalence, correlations, and consequences of different types of aggressive behaviors (physical, sexual, verbal, and coercive), delineate the association between dating violence and other forms of peer violence, and synthesize how patterns of aggressive behavior vary by sex, development stage, and other individual-level factors. An examination of the literature regarding violence among adolescents reveals problems with obtaining reliable estimates of the prevalence of different forms of violence. Family of origin, peer group, and other affiliations associated with prior victimization play an integral role in the perpetration of violence. The research found that child maltreatment, lack of parental supervision, victimization by peers, perceived social support among peers, and witnessing peer violence was predictive of and/or correlated with perpetration of date and other forms of violence.

FEDERAL CONTACT: Joyce McCurdy 770-488-4266 PIC ID: 7698

PERFORMER: Battelle Memorial Institute, MD, VA


The Role of Power and Control in Intimate Partner Violence Perpetration

The purpose of this project was to design a comprehensive research study concerning the role of power and control in the perpetration of intimate partner violence (IPV). The project produced a literature review synthesis and instrument review to measure factors predisposing men to IPV and a research protocol for a case-control study. The purpose of the study was to increase understanding of psychological and sociological factors that are potentially modifiable causes of IPV. These included substance abuse, poor impulse control, depression, patriarchal attitudes toward women, victim blaming, and socioeconomic status. Three dimensions of IPV were considered: theories linking IPV to power and control; risk and protective factors that mediate IPV by men; and research on programs that address power and control with perpetrators of IPV. The literature review and instrument review deliverables were designed to inform development of a research protocol for a case-control study. New insights gained from the proposed case-control study provided a valuable opportunity to enhance understanding of factors that contributed to IPV perpetration and were amenable to early intervention.

FEDERAL CONTACT: Nancy Cheal 404-639-7222 PIC ID: 7697

PERFORMER: Battelle Memorial Institute, MD, VA

National Immunization Program

Evaluation of the Distribution of the Influenza Vaccine by Health Care Providers in the U.S.

This study sought to better understand patterns of influenza vaccine distribution that occurred following advice to providers that they focus on high risk populations. Immunization tops the list of the ten great public health achievements during the last century. Despite the efficacy of immunization, vaccine- preventable diseases such as influenza are still a significant cause of mortality. During 11 of 23 annual influenza seasons between 1972-1995 in the United States, approximately 20,000 deaths, mostly among persons aged 65 or older, were attributed to complications of influenza. The objective for Healthy People 2000 was to have achieved influenza vaccination levels of greater than 60 percent among persons aged 65 or older, as well as among younger persons with high-risk medical conditions for whom the vaccine is recommended. The vaccination coverage objectives for influenza vaccine reported in Healthy People 2010 among persons aged 65 or older was elevated to 90 percent, while the goal remains at 60 percent for younger high-risk persons. In order to achieve these goals, it is essential that fundamental issues, such  as the ready availability of influenza vaccine be assured. Many providers who found themselves without vaccine at the time they had already prepared to provide influenza vaccine viewed the resulting four-to- eight week delay in the distribution of influenza vaccines as a de facto shortage. Preliminary findings revealed a variety of responses to the CDC recommendations to focus immunization efforts on those at high risk: some non-traditional providers, such as large pharmacy chains, canceled immunization clinics altogether because they were unable to determine who was at high risk and did not want to have to triage their customers; mass immunizers, such as those who contracted their services to large organizations, often rearranged their schedules in an effort to first assure coverage for high-risk persons; and some distributors made an effort to distinguish among their customers who were likely to be immunizing high-risk persons and distribute limited vaccines accordingly.

FEDERAL CONTACT: Marika Iwane, 404-639-8769 PIC ID: 7704

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Centers for Medicare and Medicaid Service


We assure health security for beneficiaries.

Evaluation Program

The research arm of the Centers for Medicare and Medicaid Services (CMS), the Office of Research, Development and Information (ORDI), performs and supports research and evaluations of demonstrations (through intramural studies, contracts and grants) to develop and implement new health care financing policies and to provide information on the impact of CMS’ programs. The scope of ORDI’s activities embraces all areas of health care: costs, access, quality, service delivery models, and financing approaches. ORDI’s research responsibilities include evaluations of the ongoing Medicare and Medicaid programs and of demonstration projects testing new health care financing and delivery approaches.

Examples of research themes include state program flexibility, the future of Medicare, provider payment and delivery, and vulnerable populations and dual eligibles.

Summary of Findings

Centers for Medicare and Medicaid Services

The Cost and Benefit of the Health Insurance Portability and Accountability Act

This was a multi-phase study that focused on groups that have been in the forefront of interacting with the population affected by the Health Insurance Portability and Accountability Act of 1996 (HIPAA) and three HIPAA related provisions: MHPA (Mental Health Parity Act of 1996), NMPHA (Newborns and Mothers’ Health Protection Act of 1996) and WHCRA (Women’s Health and Cancer Rights Act of 1998). The groups, i.e., state agencies, consumer advocacy groups, or individual researchers, are being contacted and benefits identified. Since strategy for implementation of HIPAA was technically based on state insurance regulatory models, the project delineated and described similar and/or differing effects as this model was applied at the Federal level. Evaluation results led to assisting CMS in the planning of future endeavors in private health insurance regulation. This report presented findings on whether HIPAA, which was intended to be a positive influence on healthcare portability and availability, led to negative changes in the healthcare market. Specifically, the areas of premium fluctuations, modifications to coverage, employer waiting periods and job-lock were addressed. Conclusions, based on interviews with Departments of Insurance, health plans, employers, employer advocacy groups and review of human resource literature, include: (1) healthcare costs have increased markedly in the last several years; (2) it is difficult to attribute premium increases and benefit plan modifications directly to HIPAA, however, they are more prevalent in the small group marketplace, which has been more directly impacted by HIPAA’s guaranteed issue and pre-existing conditions limitations; and (3) among the changes most commonly seen are increases in cost sharing features such as deductibles and copays, and pharmaceutical coverage cost sharing.

FEDERAL CONTACT: James Fuller, 410-786-3365 PIC ID: 7420

PERFORMER: Arthur Andersen and Company, Washington, DC

Office of Clinical Standards and Quality

A Normative Standards Framework For Home Health Agency Performance Enhancement

This study explored the possible further development and testing of a normative standards approach through the Peer Review Organization (PRO) program in conjunction with the Home Health PRO Pilot Project. Normative standards were developed to assist Medicare Fiscal Intermediaries in making home health care payment denials under the Medicare cost-based payment. With the shift to the Prospective Payment System (PPS) for Medicare home health care in October 2000, normative standards were envisaged as a potential aid in identifying possible under-provision of services in a prospective payment environment. During the same period that the payment system was moving toward PPS, plans for the national implementation of outcome reporting and outcome-based quality improvement (OBQI) also were underway. As OBQI and PPS evolve, a promising role for normative standards is as a tool to assist providers and others to improve patient outcomes and the efficiency with which care is delivered. This report presented the normative standards approach developed to date and proposed for further testing. The essence of the proposed approach was to classify agencies into utilization and outcome categories and then tailor performance enhancement on already-developed home health outcome measurement, outcome reporting, and OBQI methods by enabling agencies to incorporate utilization information into their performance improvement activities.

FEDERAL CONTACT: Tricia L. Rodgers 410-786-1833 PIC ID: 7175

PERFORMER: Center for Health Policy Research, Denver, CO

Office of Research, Development, and Information

Development and Psychometric Evaluation of Beneficiary Knowledge Indices from the Medicare Current Beneficiary Survey

This Phase 3 project evaluated the impact of the National Medicare Education Program (NMEP) and assessed changes in beneficiary knowledge from immediately before to immediately after national distribution of the Medicare & You (2000) handbook. This report expanded on previous work by developing knowledge measures from questions included in the Beneficiary Knowledge (BK) supplemental rounds 23 & 26 and Beneficiary Needs (BN) supplemental rounds 24 and 27 of the 1998 and 1999 Access To Care files, which were administered immediately before and after national distribution of the first Medicare & You (2000) handbook. In addition, this report evaluated the psychometric properties of each knowledge measure, including internal consistency reliability and construct validity. One of the most consistent findings across all years and interview types was a strong relationship between higher educational achievement and higher knowledge scores. Another finding was that, as hypothesized, the relationship between knowledge scores and enrollment in managed care was strongest for the seven- item quiz. This quiz included four questions concerning managed care plans, while the three-item quiz and the perceived knowledge index each contained only one question on this topic. The perceived knowledge index seemed to have the best psychometric properties. This index performed well in the reliability and validity analyses. On the basis of content considerations and the psychometric analysis results, the seven-item quiz appeared to be the most useful measure of beneficiary knowledge.

FEDERAL CONTACT: Tricia L. Rodgers 410-786-1833 PIC ID: 7802

PERFORMER: Research Triangle Institute, Research Triangle Park, NC


Early Experience Under Medicare+Choice: Final Summary Report

This report summarized the results that have been produced for the monitoring system of the Medicare+ Choice (M+C) program. The analysis focused on the experience of managed care organizations (MCOs) and beneficiaries across 69 study markets. Of all Metropolitan Statistical Areas (MSAs) with Medicare managed care, the study chose those with a population of at least 1.5 million or a Medicare managed care penetration rate of at least 30 percent. Sixty-nine MSAs met the criteria, and together they accounted for 74 percent of all Medicare managed care enrollees in 1998. The study found that while the changes implemented by the Balanced Budget Act in 1997 (increasing the payment rates in some mainly rural counties and expanding the types of organizations eligible to participate) brought problems, M+C MCO benefits still compared favorably to traditional Medicare supplemented with Medigap coverage and performance on quality indicators was generally good. Across the majority of study markets, M+C MCOs continued to deliver health care services of solid quality and to offer prescription drug coverage at a reasonable monthly premium. The study concluded that Medicare managed care, remains an important source of supplemental coverage, particularly for Medicare beneficiaries who lack employer-based coverage and who do not have access to Medicaid.

FEDERAL CONTACT: Dave Skellan, 410-786-0699 PIC ID: 7169

PERFORMER: Mathematica Policy Research, Inc., Washington, DC


Evaluation of Medicare’s Competitive Bidding Demonstration for DMEPOS: Second-Year Annual Evaluation Report

The Balanced Budget Act of 1997 (BBA 97) authorizes the Department of Health and Human Services to implement up to five demonstration projects of competitive bidding for Medicare Part B items and services, except physician services. On the basis of this authority, to test the use of competitive bidding to set prices for durable medical equipment and prosthetics, orthotics, and supplies (DMEPOS). the Centers for Medicare & Medicaid Services planned and implemented the DMEPOS Competitive Bidding Demonstration. BBA 97 required that the demonstration be evaluated for its impact on Medicare program payments, access, diversity of product selection, and quality. The purpose of this report was to describe the results to date of the evaluation of the DMEPOS Competitive Bidding Demonstration. The impact of the demonstration was evaluated on (1) Medicare expenditures, (2) beneficiary access to care,

(3) quality of care (including diversity of product selection), (4) competitiveness of the market, and (5) the reimbursement system. Based on approximately 2 years of operation, CMS’s Competitive Bidding Demonstration for DMEPOS showed the potential to decrease Medicare expenditures. Competitive bidding has lowered the prices paid by Medicare for the large majority of DMEPOS products and services. Because there is not yet data on utilization, it cannot be definitively concluded that total DMEPOS allowed    charges (the product of price times utilization) fell. It is estimated that Medicare-allowed charges for demonstration products would fall by nearly $8.5 million over the course of the demonstration, a reduction of 20 percent.

FEDERAL CONTACT: Ann Meadow, 410-786-6602

PIC ID: 7173

PERFORMER: University of Wisconsin, Madison, WI


Evaluation of the EverCare Demonstration Program

The EverCare demonstration attempted to reduce medical complications and dislocation trauma resulting from hospitalization, and to save the expense of hospital care when patients can be managed safely in nursing homes with expanded services. The EverCare evaluation combined data from site case studies, a network analysis of nurse practitioners (NPs), participant and caregiver surveys and participant utilization data to examine: (1) a comparison of enrollees and non-enrollees, (2) the process of implementation and operation of EverCare changes in the care process, as well as quality of care, (3) effects of the demonstration on enrollees’ health and health care utilization, (4) satisfaction of enrollees and their families, and (5) effects of the demonstration on the costs of care, as well as payment sources. The report provides a description of the methods used in data collection, descriptive analysis of the study population, and analyses from the resident surveys of health status, function and satisfaction, the NP time study and the analysis of hospital utilization. The main findings from the nurse practitioner time study were NPs spent about 35 percent of their working day on direct patient care and another 26 percent in indirect care activities. The mean time spent on a given patient per day was 42 minutes; of this     time, 20 minutes was direct care. Finally, it was found that NPs’ activities are varied. Much of their time was spent in communicating with vital parties, an important function that supports the physicians’ primary care role and should enhance families’ satisfaction with care. The nurse practitioners in this study were actively engaged in clinical work. By simply being present in the facility on such a frequent basis,  the NP may develop relationships with nursing home staff that ease the identification of early changes in nursing home residents’ status and monitoring of on-going treatments.

FEDERAL CONTACT: John Robst, 410-786-1217 PIC ID: 7185

PERFORMER: University of Minnesota, Minneapolis, MN


Questionnaire Development and Cognitive Testing Using Item Response Theory

The purpose of this project was to design a pool of survey questions to measure beneficiary knowledge of the Medicare program in future rounds of the Medicare Current Beneficiary Survey (MCBS). The intent of the project was to assist the Centers for Medicare and Medicaid Services in assessing how well the National Medicare Education Program is meeting its consumer information goals. The researchers developed questionnaire items using a comprehensive multi-step process that included: (1) background research, (2) review of Medicare informational materials and Medicare knowledge surveys, including the Medicare Current Beneficiary Survey (MCBS), and (3) multiple meetings and discussions with CMS and  the project’s seven-member Technical Advisory Panel (TAP). Knowledge scores calculated after the first round of interviews showed that respondents answered between 15 and 43 (out of 51) knowledge items correctly, or 29 to 84 percent. Based on feedback from the TAP and CMS, the survey questions were revised for a second round of testing. The language was simplified and more consistent terminology was used throughout. Fewer changes were suggested by the second round testing than by the first, suggesting that the instrument worked better. The report determined that using Item Response Theory (IRT) would produce a more precise measure of beneficiary knowledge, allowing CMS to more accurately determine areas in which beneficiaries lack knowledge and could use more information. In addition, an IRT-based knowledge scale would allow CMS to track knowledge, thereby allowing for the evaluation of the effectiveness of interventions and education programs.

FEDERAL CONTACT: Tricia L. Rodgers 410-786-1833 PIC ID: 7786

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Food and Drug Administration


The FDA is responsible for protecting the public health by assuring the safety, efficacy, and security of human and veterinary drugs, biological products, medical devices, our nation's food supply, cosmetics, and products that emit radiation. The FDA is also responsible for advancing the public health by helping to speed innovations that make medicines and foods more effective, safer, and more affordable; and helping the public get the accurate, science-based information they need to use medicines and foods to improve their health.

Evaluation Program

The Food and Drug Administration's Evaluation Program ultimately reflects on the goals established and promulgated by the Department of Health and Human Services (HHS) for which FDA has responsibility. HHS's goals are products of its strategic performance planning process, and FDA uses its own strategic framework to accomplish these goals. This process also satisfies the implementation requirements of the Government Performance and Results Act (GPRA) and the Food and Drug Administration Modernization Act of 1997 (FDAMA). The strategic and performance process is an evolving set of program directions for FDA as changes occur in FDA's dynamic environment. FDA's challenges rest on its ability to leverage its efforts in that environment, which grows increasingly complex and more institutionally networked. The Agency will strive to maintain the scientific knowledge base necessary to achieve greater effectiveness in assuring the quality and availability of the products it regulates.

Summary of Findings

Center for Drug Evaluation and Research

CDER 2001 Report to the Nation: Improving Public Health Through Human Drugs

The Center promotes and protects public health by assuring safety and effectiveness of drugs used by Americans. The Food and Drug Administration (FDA) Modernization Act of 1997 affirmed the center’s public health protection role, clarified the FDA’s mission and called for the FDA to do the following: (1) promote public health by promptly and efficiently reviewing clinical research and taking appropriate action on the marketing of human drugs in a timely manner, (2) protect the public health by ensuring that human drugs are safe and effective, (3) participate through appropriate processes with representatives of other countries to reduce the burden of regulation, harmonize regulatory requirements and achieve appropriate reciprocal arrangements, and (4) carry out its mission in consultation with experts in science, medicine and public health and in cooperation with consumers, users, manufacturers, importers, packers, distributors, and retailers of human drugs. Among many findings, this report states that: (1) people with cancer, heart disease, HIV, AIDS, and other serious conditions have benefited from FDA approvals in 2001; FDA met its obligations to Congress for prompt and thorough review of drug applications supported by user fees; (2) with modern, state-of-the-art tools and techniques, FDA was  able to detect rare and unexpected risks more rapidly and take corrective action more quickly; and FDA augmented its risk assessment ability by gaining access to user fee data.

FEDERAL CONTACT: Debbie Henderson, 301-443-5281 PIC ID: 7890

PERFORMER: Food and Drug Administration, Rockville MD

Office of Policy, Planning and Legislation

Assessment of the Availability of Mammography Services

This analysis provided an overview of the availability of mammography services and of the changes in the demand for and supply of mammograms in recent years. It has been suggested that closures and  service curtailments by mammography providers have made it more difficult for women to schedule and obtain mammograms within a reasonable length of time. The possible shortage of mammography services has raised concerns that some women are not receiving needed mammograms due to limitations in access. The data from the FDA mammography database, in combination with population data, showed that the number of mammography facilities has declined in most states while the population of potential recipients of mammography services has increased. This suggested an increased demand relative to the supply of mammography services. The number of facilities did not capture the effective capacity of mammography providers. The CDC data failed to support claims that closures of mammography facilities or increased demand for mammography services have negatively affected the number of women obtaining mammograms. Data from CDC surveys, when combined with FDA data, show that providers conducted substantially more mammograms per facility in 2000 than they did in 1997. Other CDC survey data describing the prevalence of women who received clinical breast exams and mammograms showed that an increasing percentage of women in the 40 and older age group had recently obtained a mammogram. Furthermore, when only women receiving clinical breast exams were considered, an increased percentage received a mammogram in the same year.

FEDERAL CONTACT: Steve Tucker, (301) 827-5339

PIC ID: 6082.1

PERFORMER: Eastern Research Group, Lexington, MA

Cost Analysis of the Labeling and Related Testing Requirements for Medical Glove Manufacturers

The FDA is proposed regulations to reclassify surgeons’ and patient examination gloves as Class II medical devices, and placed controls and restrictions on the manufacturers and on some users of these products.  The reclassified gloves, including those made of natural rubber latex or synthetic material, were regulated in four categories: (1) powdered surgeons’ gloves, (2) powder-free surgeons’ gloves, (3) powdered patient examination gloves, and (4) powder-free patient examination gloves. This report showed that under the proposed regulation, the newly recommended protein and powder limits were lower than current average levels. In addition, the regulation required manufacturers to place expiration dates on glove labels to ensure adequate barrier protection (i.e., to prevent the transmission of pathogens to and from the patient). This report described the labeling requirements put forth by the FDA, the labeling cost model, and the various testing costs that determine total industry costs.

FEDERAL CONTACT: Steve Tucker, (301) 827-5339

PIC ID: 7889

PERFORMER: Eastern Research Group, Lexington, MA

Health Resources and Service Administration


To improve the Nations health by assuring equitable access to comprehensive, culturally competent, quality health care for all.

Evaluation Program

The Health Resources and Services Administration (HRSA) supports a wide array of very different programs and activities that promote access to needed health care for all, including primary health care centers, the National Health Service Corps, HIV/AIDS programs, maternal and child health activities, health professions training, rural health programs, organ donation and transplantation initiatives, and telehealth activities.  To provide underpinning for these efforts, HRSA’s evaluation program is designed to enhance strategic planning, strengthen budget and legislative development, and improve program performance.

HRSA also supports activities to enhance the quality of evaluation agency-wide, such as funding short courses in evaluation for staff and expanding agency staff’s access to technical assistance on the conceptualization, design and implementation of evaluation activities. The broad dissemination of evaluation products and results is also an area of emphasis.

Summary of Findings

Bureau of Primary Health Care

Assessment of Selected Domestic Violence Programs in Primary Health Care Settings

This assessment profiled in-depth the domestic violence programs and protocols of nine Bureau of Primary Health Care (BPHC) funded community-based primary health care centers. The protocol, the foundation of a clinical domestic violence program, served as the plan for identifying and responding to individuals who have experienced domestic violence. Primary health care providers across the country are beginning to acknowledge family and intimate partner violence (FIPV), or domestic violence, as one of the most prevalent and serious public health issues affecting millions. Although domestic violence has devastating consequences for women, men, and their families, it disproportionately affects women. The domestic violence programs identified as complete in this assessment shared features, and developed other unique elements to enhance their program’s service mix and ability to respond to the needs of the communities they serve. Most programs profiled in this assessment served racially, ethnically, and culturally diverse populations, and were located in rural and urban settings. Attempts to highlight programs that were geographically dispersed across the United States were also made during the program selection process. To prevent domestic violence and change social attitudes regarding its acceptance, the majority of domestic violence programs profiled in the assessment conducted some form of outreach or education for their clients and the general public regarding domestic violence.

FEDERAL CONTACT: Dr. Shari Campbell, 301-594-4251 PIC ID: 7284

PERFORMER: North American Management Company, Alexandria, VA


Enabling Services and Perinatal Care: Final Report

Providing enabling services -- key strategy used by all Bureau of Primary Health Care grantees, may include transportation, translation, case management, health education, nutrition counseling and outreach--are not typically reimbursed under managed care. The purpose of this study was to: analyze the types and levels of enabling services provided by Community Migrant Health Centers (C/MHCs), determine how these services have changed, and analyze whether enabling services improve outcomes and reduce costs. The study drew upon aggregate data from the BPHC Uniform Data System, the annual report submitted by all grantees. A total of 650 grantees filed reports in 1996 and 1997. In 1997,  enabling service personnel accounted for almost one-quarter of total direct care staff, and sample grantees expended $211 million on these services (about 13% of total direct service cost). About 95 percent of grantees provided case management and health education services. At the other end of the spectrum, under 20 percent provided child care on-site or operated food banks and/or delivered meals. It was found that health center prenatal care users are twice as likely to be teenagers, and twice as likely to be Hispanic, than is the case for the US prenatal population. As a whole, about 72 percent of C/MHC grantees have specialized obstetrical staff; the average grantee with specialized staff has 1.3 OB- GYN/Certified Nurse Midwife full-time employees. The report concludes that the breadth of perinatal services, coupled with staffing that promotes continuity of care, contribute to appropriate and timely use of prenatal and after-delivery services by mothers and infants.

FEDERAL CONTACT: Fred Butler, 301-549-4281 PIC ID: 7126

PERFORMER: MDS Associates, Inc., Wheaton, MD


Assessing the Impact of Increased Medicaid Dental Reimbursement Rates On the Utilization and Access of Dental Services in South Carolina

In January 2000, the South Carolina Medicaid program set new fees at a level such that each procedure would reimburse 75 percent of the dentists at their full, usual, and customary level. This was a stark departure from the usual approach taken, where state Medicaid agencies would approve modest increases in fees --normally a proportion of the full, usual, and customary level. Given that the South Carolina reimbursements were set such that most dentists would pay no financial penalty for filling a time slot with a Medicaid patient instead of a private pay patient, policy makers hoped that dentists would respond to the fee changes. The reform increased the number of: (1) Medicaid children receiving services; (2) services Medicaid children received; and (3) participating dentists, but not the Medicaid load per participating dentist. Consequently, the results indicate that the January 2000 reform had the desired results. It is recommended that South Carolina policy makers consider maintaining their commitment to a Medicaid reimbursement that reflects the 75th percentile of private pay fees for dental procedures.

FEDERAL CONTACT: John Kehoe, 404-562-7983 PIC ID: 7196

PERFORMER: Medical University of South Carolina, Charleston, SC

Maternal and Child Health Bureau

Application of a Distance Learning Evaluation Plan to Data Skills Online

This study examined the contribution of different distance learning approaches to continuing education and professional training to the Health Resources and Services Administration’s (HRSA’s) mission. This document demonstrated how this framework can be applied to a specific distance learning training program, Data Skills Online, a Web-based, self-instructional training program targeting public health professionals at state and local levels, as part of an Maternal and Child Health Bureau--initiated evaluation that targeted training objectives and components of Data Skills Online that related to distance learning. Recommendations included the following: (1) collect data from diverse pool using a variety of methods and, (2) build process evaluation activities into grant requirements and provide grantees with a standard set of required data elements as well as a template for data entry.

FEDERAL CONTACT: Jacob Tenenbaum, 301-443-9011 PIC ID: 7114

PERFORMER: The Lewin Group, Falls Church, VA


Discontinuous Coverage in Medicaid and the Implications of 12-Month Continuous Coverage for Children

This report analyzes the extent to which a policy of continuous coverage improves the continuity of Medicaid coverage and decreases the incidence of gaps in coverage that result when children temporarily lose Medicaid eligibility. The Balanced Budget Act of 1997 gave states the option of providing up to 12 months of continuous coverage for children through age 18 enrolled in Medicaid and SCHIP. Using 1994- 1995 Medicaid enrollment and payment data from four states--California, Michigan, Missouri, and New Jersey--this study examined the implications of a policy of 12-month continuous coverage. Findings on the impacts of 12-month continuous coverage indicate: (1) the number of children eligible for continuous coverage and ever enrolled during a year would increase, (2) the total number of months during which children are covered would increase, (3) the average cost per enrollee month would decline slightly, (4) among children who would qualify, payments would increase, (5) administrative costs associated with disenrollments, re-enrollments, and redeterminations in states using a six-month redetermination period would fall substantially, (6) staff costs associated with those categories in #5 are only a small portion of administrative expenses associated with the cost of operating public health insurance, and (7) effects on emergency room use and payments were not conclusive.

FEDERAL CONTACT: Jacob Tenenbaum, 301-443-9011 PIC ID: 7774

PERFORMER: Mathematica Policy Research, Inc. Plainsboro, NJ and, Boston University Medical Center Boston, MA

Office of the Administrator

Big Cities Health Inventory

This report is the fourth in a series from the Chicago Department of Public Health presenting city-to-city comparisons of leading measures of health. This edition established an advisory group to help guide the reports’ focus and content. Based on the committee’s recommendations, the report includes, for the first time, Healthy People 2010 objectives to serve as benchmarks. The report focuses on 20 indicators of health: five indicators of communicable diseases, nine causes of mortality, and six indicators of maternal and child health. The data presented here and in other studies analyzing urban health suggest that there is a unique urban health profile influenced by the dynamics peculiar to large cities. For example, the health outcome of the nation’s largest cities are less favorable than those of smaller urban and rural areas. The three largest cities in the U.S., New York, Los Angeles and Chicago had considerably different health profiles. New York ranked among the top ten cities for incidence of HIV, TB and HIV-related mortality. Chicago ranked highest in syphilis and homicide. Los Angeles ranked in the middle for most indicators. A better understanding of what caused such different health outcomes may improve health policies and programs in these cities.

FEDERAL CONTACT: Michael Millman, 301-443-0368 PIC ID: 7241

PERFORMER: Chicago Center for Health Systems Development, Chicago, IL

Indian Health Service


The mission of the Indian Health Service, in partnership with American Indian and Alaska Native people, is to raise their physical, mental, social, and spiritual health to the highest level.

Evaluation Program

The goal of the Indian Health Service (IHS) is to assure that comprehensive, culturally acceptable, personal and public health services are accessible to American Indian and Alaska Native people. The importance of evaluation in supporting this goal has increased significantly in recent years and includes American Indians and Alaska Natives as the primary stakeholders in defining the purpose, design, and execution of evaluations. The stakeholders use the end product of the evaluations, and are typically the population or groups most likely to be affected by the evaluation findings. The IHS has formally adopted the principles of a responsive evaluation practice to address the needs and concerns of American Indians and Alaska Natives.

The evaluation program of the IHS is managed by the Office of Public Health, Staff Office of Planning, Evaluation, and Research, which provides national leadership and consultation for IHS and Area Offices on strategic and tactical planning, program evaluation and assessment, public health and medical services, research agendas, and special public health initiatives for the Agency.

The evaluation needs of the IHS service programs are coordinated using two major types of short-term studies: policy assessments and program evaluations. The IHS policy assessments contribute to decision making about budget, legislation, and program modifications and include background information to support the Agency’s initiatives. Evaluations are focused at the program level, or Area Offices, and focus on specific program needs.

Each year IHS selects high-priority health care and management studies for funding through the submission of proposals to headquarters and Area Offices. These proposals are reviewed and rated by a panel of subject-matter experts, evaluation experts, and IHS staff for concurrence with IHS strategic goals, objectives, and priority areas. The proposals are then prioritized and forwarded to the IHS Director, who reviews the projects that are recommended for funding and determines the respective funding levels.

Summary of Findings

Office of Public Health

Evaluating the Impact of Primary Intervention Techniques on the Dental Caries Rate in Children Living in Southwest Alaska Native Villages

This study evaluated how the dental staff analyzed potential reasons for variation in caries rates between villages in an attempt to reverse the trend of increasing caries rates in all villages. In the 1990s, the Bristol Bay Area Health Corporation dental staff noticed a persistent increase in caries (dental decay) rate in all Bristol Bay village children, particularly in the youngest ages (0-4 years old). The staff concluded that 3 factors contributed the most to lowering caries rates in a given village. First was the presence of a prominent individual in the village, such as a community health aide or teacher, who encouraged good oral hygiene habits in the children. Second was the exposure to fluoride through community water fluoridation or with a consistent fluoride mouth rinse program. Third was a low sugar diet among the children.

FEDERAL CONTACT: Debbie Melton, 301-443-2417 PIC ID: 7138

PERFORMER: Indian Health Service, Rockville MD


Quantifying the Unmet Need in IHS/Tribal EMS

The purpose of this report was to present data collected from Tribal Emergency Medical Services (EMS) programs, report their unmet needs, and make suggestions for future resource allocation by the Indian Health Service. It was found that the 562 federally recognized Tribes do not have their own EMS programs. While it was not known how the members of these Tribes obtain emergency medical services, it was assumed that they relied on local public (city or county) and private ambulance services to access EMS. Injury deaths among Native American children were caused by motor vehicle crashes, pedestrian related mother vehicle crashes, drowning, fires, and suicide. It was reasonable to assume these fatality rates reflected a significant portion of the EMS run volume by tribal Emergency Medical Technicians (EMTs) within their communities. It was also reasonable to conclude that these same EMTs do, or could contribute to Service Unit injury prevention activities through their role as participating members of Injury Prevention Committees. These childhood injury fatalities had significant impact on the direction of community and service area Emergency Medical Services for Children activities.

FEDERAL CONTACT: Dr. Eric B. Broderick, 301-443-3024 PIC ID: 7860

PERFORMER: Office of Program Planning and Evaluation, Bethesda MD


Study of the Impact of a Full-Time Community Health Nurse on the Health of Native American (Sioux) Patients with Hypertension & Prevention

The goal of this project was to hire a community health nurse to follow hypersensitive patients. The nurse was responsible for identification, education, and following the progress of the hypertensive patients at the Sisseton Service Unit. Statistics were gathered on the patients in order to know more about them and to better plan their treatment. The goal of the project was to achieve a blood pressure control rate of 50 percent for patients with hypertension. Control was defined as a systolic blood pressure below than 140 and a diastolic lower than 90. This report presents preliminary results of the Community Health Nurse on the care of the hypertensive population. It also presents the process by which a pharmacist influenced the treatment of hypertension and outcomes. The impact made on the treatment of hypertension is expected to be more fully demonstrated through lower illness and death from Cardiovascular Disease in the Native American Population.

FEDERAL CONTACT: Elizabeth A. Fowler, 301-443-1270 PIC ID: 6413

PERFORMER: Aberdeen Area Indian Health Service, Aberdeen, SD

National Institutes of Health


To sponsor and conduct medical research that leads to better health for all Americans.

Evaluation Program

The National Institutes of Health (NIH) generates scientific knowledge that leads to improved health. This is done by conducting medical research in its intramural laboratories and by supporting research in universities, medical and health professional schools, and other health research organizations. NIH fosters the widespread dissemination of the results of medical research, facilitates the training of research investigators, and ensures the viability of the research infrastructure. The NIH Evaluation Program is an integral part of how NIH sponsors and conducts medical research.

Results based management is recognized as a basic principle for the sound and productive operation of government agencies and their programs. This is evidenced most notably by passage of the Government Performance and Results Act (GPRA) and by the considerable effort across the federal government to implement results based management mechanisms. With GPRA and other initiatives aimed at increasing public sector accountability (such as the Chief Financial Officers Act and the Government Management Reform Act), interest in the use of evaluation has increased steadily among NIH administrators and others, such as officials within the General Accounting Office, the Office of Management and Budget and the Department of Health and Human Services.

Philosophy and Priorities

The NIH Evaluation Program provides information to assist the NIH Director and the NIH Institute and Center (IC) Directors in determining whether NIH goals and objectives are being achieved and to help guide policy development and program direction. Evaluations are planned and conducted from two sources of funds: 1 percent evaluation set-aside funds used to fund trans-NIH projects, and IC program funds used for program evaluations for use by various committees, working groups, task forces, workshops, conferences, and symposia to assist the ICs in program management and development. This approach ensures that planning and priority setting specific to the mission of each IC are fully developed and implemented and that there is central leadership for developing crosscutting initiatives and promoting collaboration among the ICs.

NIH’s major evaluation priority areas fall within three broad program areas: basic research, research training and career development, and facilities. NIH conducts evaluations in these areas to assess strategies and goals, develop performance measures and improve operations.

Policies and Operations

A distinguishing feature of the NIH Evaluation Program is the utilization of a variety of evaluation strategies that include the use of national advisory councils, boards of scientific counselors, consensus development conferences, and ad hoc committees that help to chart scientific directions and select the most promising research to support.

A two-tier system is used to review project requests that will use 1-percent evaluation set-aside funding. The first tier involves a review and recommendations by the NIH Technical Merit Review Committee (TMRC) on the technical aspects of project proposals and whether a project fits within HHS guidelines for use of the set-aside fund. The second tier involves the NIH Evaluation Policy Oversight Committee, which considers TMRC recommendations, conducts policy level reviews, and makes final funding recommendations to the NIH Director or his designee.

Summary of Findings

National Center for Research Resources

Scientific and Engineering Research Facilities

This report consists of a collection of tables prepared biennially through the National Science Foundation’s congressionally mandated Survey of Scientific and Engineering Research Facilities (Facilities survey). The survey originated in 1986 in response to Congress’ concern about the state of research facilities at the Nation’s colleges and universities. The survey was sent to research-performing colleges and universities in the United States. Research-performing colleges and universities are defined as meeting one of three criteria: (1) offer doctorates in S&E fields, (2) report at least $150,000 in research and development (R&D) expenditures for fiscal year 2000, and (3) are an Historically Black College or University with an R&D expenditure. The survey was also sent to nonprofit biomedical research organizations. The 2001 survey was mailed to academic institutions in April of 2001 and data collection ended on July 24, 2001.

Of the 580 eligible institutions, 90 percent returned surveys.

FEDERAL CONTACT: Jeff Sussman 301-496-5617

PIC ID: 6863

PERFORMER: National Science Foundation, Arlington, VA


Survey of NIH-funded Investigators Who Use Nonhuman Primates: Report on Survey Findings

The Center commissioned the National Survey of Nonhuman Primate Research Use to learn about investigator access to nonhuman primate resources at the eight Regional Primate Research Centers (RPRCs) and to assess current and future nonhuman primate needs. Humanitas Incorporated, the survey contractor, conducted the survey from June through September 2001 via the Internet and used mail and telephone follow-up procedures. The survey queried investigators who were identified in NIH databases as having been awarded NIH funds in FY99 for research nonhuman primates. The survey found that: (1) nonhuman primate survey findings can promote understanding of RPRC access and nonhuman primate needs; (2) RPRCs provided access to valuable nonhuman primate resources – all investigators who used an RPRC gave the highest satisfaction ratings possible to the quality of the nonhuman primate resources (freedom of animals and specimens from disease) and to the availability of desired services (specimens, animal services, and collaborative consultations); (3) obtaining nonhuman primate resources was challenging – especially regarding animal-, species- and characteristics-specificity; (4) rhesus macaques were most often used and most needed, more than half of responding investigators needed rhesus macaques for their research in the next three years; and (5) respondents recommended increasing the availability of nonhuman primates and improving nonhuman primate allocation systems.

FEDERAL CONTACT: Patricia Newman, 301-435-0864 PIC ID: 7639

PERFORMER: Humanitas, Inc., Silver Spring, MD

National Institute of Child Health and Human Development

First Phase of the Evaluation of WISE EARS! Campaign in Industrial Workers, Hispanic/Latino/Latina Individuals, and Native American Youth Under 17 Years Old

The Institute and its partner, the National Institute of Occupational Safety and Health/Centers for Disease Control, were joined by a coalition of national, regional, and local organizations, voluntaries, and state and local government agencies in a national campaign called WISE EARS! to prevent noise-induced hearing loss (NIHL). The “WISE EARS!” goals included: (1) increasing awareness about NIHL among all audiences, and (2) motivating all audiences to take action against NIHL by understanding the problem and its solutions. An evaluation was conducted consisting of a literature review to determine how similar programs were developed to reach specific populations, focus group testing to discuss the strengths and limitations of the strategic approaches to these populations, and consultation with experts in the field prior to final, informed decisions being made. Several significant findings include: (1) although coal mining representatives and American Indian nurses all understood the term “noise-induced hearing loss,” participants in both groups in Houston, all female, had difficulty with the term but did understand that exposure to loud noise is bad for hearing, (2) while coal miners received information equally from television/radio, newspaper/magazines and doctor’s office or clinic, and also used material from the federal government, none of them identified “family and friends” as a source of health information, and (3) while Hispanic/Latina mothers received the most health information from family/friends and television/radio, they also received information from doctor’s office or clinic and only one identified newspaper and magazines as a source of health information.

FEDERAL CONTACT: Jeff Sussman 301-496-5617 PIC ID: 7630

PERFORMER: Caliber Associates, Fairfax, VA

National Institute of General Medical Sciences

Measuring Up: Research & Development Counts for the Chemical Industry

This study was undertaken to measure the impact, and return, or payoff, of chemical research and development. Macroeconomic studies have addressed and documented R&D contributions to the nation’s economic growth; this study examined chemical research. Key accomplishments of this study included the development of a measurement that quantified the chemical R&D impact, and a methodology to show linkages between chemical literature citations and science innovation. The study revealed that: (1) every dollar invested in chemical R&D today produced $2 in operating income over six years; (2) the chemical industry’s share of total U.S. R&D had declined from 11 percent in 1956 to 8 percent in 1992; (3) the industry still appeared to be holding strong despite growing globalization; and (4) predictability of government funding was more important to industry than the level of funding. The study concluded that U.S. Chemicals technology was strong and getting better, but it is not the leading technology among economic indicators. Its quality is clearly increasing when compared to foreign-invented Chemical patents, and its science linkage was increasing rapidly, especially when compared to foreign Chemical patents. It was more highly linked to science than other major U.S. technology, except for the Life Sciences, and it was also highly linked to public science and to local science.

FEDERAL CONTACT: Dr. James Onken, 301-594-2762 PIC ID: 7659

PERFORMER: Council for Chemical Research, Washington DC

National Institute of Aging

Evaluation Feasibility Study for the National Institute on Aging’s Book and Videotape, Exercise: A Guide from the National Institute on Aging

Since 1998, the National Institute on Aging’s (NIA) Office of Communications and Public Liaison (OCPL) has promoted and distributed the book, “Exercise: A Guide from the National Institute on Aging,” and an accompanying video. The materials were developed in response to an increasing number of requests about exercise for older people from the public, the media, and private-sector health and fitness providers. A kick-off event in 1998, including representatives from NIA, NASA, DHHS Office on Women’s Health, and featuring Senator/Astronaut John Glenn, marked the initial availability of the Guide. Since then, the Guide has become one of NIA’s most popular publications. NIA distributed 72,994 copies of the Guide in 2001 alone. Based on the clear public health need, NIA plans to update the Guide and continue its efforts to promote the beneficial effects of exercise among older Americans.

FEDERAL CONTACT: Freddi Karp, 301-496-1752 PIC ID: 7642

PERFORMER: Academy for Educational Development, Washington, DC

National Eye Institute

Designing Pilot Study to Enhance Referrals to Vision Rehabilitation Services

The Institute conducts a public education program on low vision designed to increase awareness of the benefits of vision rehabilitation. A key part of this program will be to develop strategies that will increase referrals from eye care professionals to vision rehabilitation services. In 2001, the NEI conducted over 25 focus groups with eye care professionals and office staff to identify barriers, opportunities, and methods for enhancing the referral process. NEI seeks to determine the nature and extent of problems a referral education program should address, what the goals should be, and how a pilot program should be developed to achieve those goals. Recommendations included: (1) print media such as the proposed Physician’s Vision Rehabilitation Services Desk Reference, vision rehabilitation services fast facts, other brief print materials, (2) audiovisual media such as Physician’s Vision Rehabilitation Services Audio Tape Series and Patient Videos Addressing Low Vision Referral, and (3) Web-based Enhancements and Referral Tools, such as Web links and the Low Vision Services Locator.

FEDERAL CONTACT: Rosemary Janiszewski, 301-496-5248

PIC ID: 7891

PERFORMER: ORC Macro, Calverton, MD

Office of the Director

Evaluation of Personnel Delegations: Year Five Report

It has been five years since the Secretary of the Department of Health and Human Services agreed to delegate to the Director of NIH all the Secretary’s personnel authorities, except those reserved to the Secretary by law and regulation. The agreement designated NIH as a pilot project and gave the director authority to use the delegations and redelegate them to develop human resource management policies, processes and systems that would best meet mission needs in recruiting and retaining the highest quality workforce. This report addressed in detail the effects of the delegation on the accomplishment of two goals: (1) advance superior biomedical and behavioral science research and (2) efficiently manage the resources provided to the NIH by the American public. Findings included: (1) the delegations to the NIH Director and their redelegation to the senior executives of the Institutes and Centers of NIH are empowering them to become more responsible and accountable for the human capital of their respective institutes and centers, (2) the redelegations have not reached the managers--division director and branch and equivalent levels--whose competencies, tools and leadership have the broadest direct impact on the hiring, development, utilization and nurturing of the people who do the NIH mission work, (3) some of the human resource system elements were so critical to the supporting manager that actions have become more flexible and easy-to-use during the life of this pilot redelegation project, and (4) the three strategies NIH has advocated remained crucial to moving the reinvention of NIH human resource management forward.

FEDERAL CONTACT: Jeff Sussman 301-496-5617 PIC ID: 6862

PERFORMER: National Academy of Public Administration, Washington, DC


Survey Options for Estimating Expenditure Weights for the Extramural Activities Component of the Biomedical Research and Development Price Index

This report presents options for the estimation of expenditure weights for the extramural component of the Biomedical Research and Development Price Index by means of a survey of institutions receiving funding from NIH. A survey is needed because institutions are no longer asked to submit data on research expenditures by budget categories during the grant application or renewal process for many types of awards. A number of sampling strategies will provide excellent estimates of the expenditure weights, especially for academic institutions that account for almost 80 percent of total extramural funding. Because of the skewed nature of the distribution of NIH funding, a focus on institution size, as measured by total dollars awarded, is essential. For example, the nine largest academic institutions alone account for nearly 25 percent of total funding to academic institutions. Awards received by large institutions are also representative of the population of awards. A similar pattern is observed for non- academic institutions as well. Thus, small samples of institutions are sufficient to provide adequate coverage of extramural funding and accurate estimates of population expenditure shares. It is recommended that 9 to 18 large academic institutions and 9 to 27 large non-academic institutions be selected for the expenditure survey.

FEDERAL CONTACT: James A. Schuttinga 301-496-2229 PIC ID: 7629

PERFORMER: Joel Popkin & Company, New York, NY

Office of the Assistant Secretary for Planning and Evaluation


To provide analytical support and advice to the Secretary on policy development and assist the Secretary with the development and coordination of department wide program planning and evaluation activities.

Evaluation Program

The Assistant Secretary for Planning and Evaluation (ASPE) functions as a principal advisor to the Secretary on policy development and, in this capacity, conducts a variety of evaluation and policy research studies on issues of national importance. ASPE also is responsible for department wide coordination of legislative, planning, and evaluation activities. In its evaluation coordination role, ASPE has the following tasks:

  • Provide annual guidance to all HHS agencies and staff offices regarding evaluation priorities, procedures, and review requirements.
  • Review evaluation priorities proposed by HHS agencies, providing advice about the focus or method of proposed projects and identifying opportunities for collaboration and effective use of resources.
  • Prepare planning and summary reports on evaluation activities as required by Congress.

Through the departmental evaluation planning process, ASPE has the capacity to identify crosscutting program or policy issues of concern to the Secretary and specific program and policy areas not covered by the HHS Agency evaluation plans. In these instances, ASPE initiates evaluations or collaborates with the agencies to conduct evaluations or policy assessments.

Another continuing evaluation objective of ASPE is to support and promote the development and improvement of databases that HHS agencies and ASPE use to evaluate health care programs and health trends. For example, ASPE has been the major initiator, in collaboration with the National Center for Health Statistics at the Centers for Disease Control and Prevention, of the first comprehensive survey of people with disabilities in the United States. Collection of the first new data was completed in FY 1996 along with national prevalence data on disability.

The ASPE chairs the Research Coordination Council (RCC), which will evaluate Department-wide research priorities to ensure that efficiencies are realized and research funding priorities are consistent with the Administration’s priorities. The ASPE also co-chairs and provides support to the HHS Data Council, which is charged with integrating key national surveys, such as linking health status indicators with indicators of well-being.

Finally, ASPE uses evaluation funds to promote effective use of evaluation-generated information in program management and policymaking. The latter is accomplished through the dissemination of evaluation findings and other activities, such as providing technical assistance to agencies in the development of performance measures.

Summary of Findings

Office of Disability, Aging, and Long-Term Care Policy

Constrained Innovation in Managing Care for High-Risk Seniors in Medicare + Choice Risk Plans

This case study examined four well-regarded managed care organizations (MCOs). While the innovations utilized by the MCOs appear to have improved care and produced high levels of satisfaction among high-risk seniors, some unmet needs remain. The study found that the four MCOs made numerous innovations to improve care delivery for elderly Medicare beneficiaries with chronic illnesses and disabilities. These MCOs used the flexibility provided by capitation to add new services, including screening and other programs to identify high-risk seniors, care management and disease management, network credentialing, occasional provision of off-policy benefits, and better coordination and flexibility in the delivery of inpatient, subacute, and home health services. The MCOs’ contracts focused on the primarily medical services covered by Medicare and do not obligate (or pay) the MCOs to address seniors’ needs for long-term support services, housing, transportation to routine care, or the myriad of other types of nonmedical assistance high-risk seniors may need to maintain their functioning and independence. The report recommends more comprehensive or intensive methods to address the full spectrum of needs such as more expansive contracts, new payment strategies, and strong evidence of effectiveness.

FEDERAL CONTACT: Jennie Harvell, 202-690-6443 PIC ID: 6391.1

PERFORMER: Mathematica Policy Research, Inc., Plainsboro, NJ

Designing and Conducting a Survey of High Risk Seniors: Methodology from an Evaluation of Managed Care Organizations

This was a survey of Medicare beneficiaries enrolled in three managed care organizations (MCOs) that participated in the Medicare+Choice program: Keystone MCO East, Kaiser Permanente--Colorado, and Aspen Medical Group. The study identified three groups of seniors with severe limitations or multiple chronic conditions who had been identified as high risk by their MCO. These beneficiaries had been enrolled in their MCO’s care management program, had attained advanced age (over 84 years old), and had experienced a recent hip fracture or stroke. This group provided a convenient way of illustrating the experiences of seniors whose high-risk status is known to their MCO.

FEDERAL CONTACT: Jennie Harvell, 202-690-6443 PIC ID: 6391

PERFORMER: Mathematica Policy Research, Inc., Plainsboro, NJ

How Managed Care Has Affected Working-Age Medicaid Beneficiaries with Disabilities and Chronic Illnesses: A Synthesis of Literature, 1995-2001

The purpose of this study was to synthesize recent research on the performance of managed care plans in caring for persons with disabilities, summarize research on current trends in enrollment of disabled persons into Medicaid managed care programs, and recommend areas for future research. Since the mid- 1990s, a number of research institutions initiated projects to assess how well disabled persons are cared for in managed care settings. Past research has focused on many different aspects of managed care and disability and used a wide variety of research designs. As a step toward developing a new research agenda, areas of concentration were identified. A literature review found that progress in these areas would enhance managed care systems for working-age Medicaid beneficiaries with disabilities and chronic illnesses. These areas of needed concentration are: (1) define needs more clearly for purposes of program planning -- the heterogeneity of needs, capacities, and attitudes within the large population suggests that a range of managed care practices and products is needed to promote positive health outcomes, (2) establishing system goals -- moving toward a shared understanding of the goals for the system of care will be a critical step in establishing a foundation for a national research agenda, (3) improved understand of how to change systems-- given a consensus on important goals, some changes in current practices and procedures will be required. Cutting across all of these areas is the need for adequate data; many critical questions will be answered only with better encounter data for working-age adults in Medicaid managed care plans with databases that shed light on patterns of cost and service use over time and across multiple service providers.

FEDERAL CONTACT: Hunter McKay, 202-690-6443 PIC ID: 7560

PERFORMER: Mathematica Policy Research, Inc., Plainsboro, NJ

Medicaid Buy-In Programs: Case Studies of Early Implementer States

Today, individuals with significant disabilities have greater opportunities for employment than ever before in the history of our Nation. Improved public understanding of disability and innovations in assistive technology, medical treatment, and rehabilitation aided these opportunities. The project had several purposes: to examine and describe the early implementation experiences of states that have opted for  the Medicaid Buy-In for working disabled persons. It also used the descriptive information to inform and provide technical assistance to state policy makers. The report informed federal policy makers regarding states’ experiences implementing Medicaid Buy-In programs. It also used the descriptive information to inform others in the state, including persons with disabilities, service providers, and employers about the range of policy options and tradeoffs that can be made in developing and improving Medicaid Buy-In programs as part of efforts to improve systems that support the employment of persons with disabilities. A case study discussed the design features of each state’s Medicaid Buy-In program within the context of the state’s Medicaid program and other initiatives. The study included descriptions of the: (1) SSI state supplementation policies and the regular Medicaid eligibility categories, (2) Medicaid Buy-In program and comprehensive work incentive initiatives, and (3) relationship between the SSI state supplementation program, regular Medicaid eligibility categories and the Medicaid Buy-In program. The findings in these three sections built on the fact that every state with a Medicaid Buy-In program and every state contemplating such a program started from a different baseline against which to measure impact and change.

FEDERAL CONTACT: Andreas Frank, 202-690-6443 PIC ID: 7556

PERFORMER: George Washington University Medical Center, Washington, DC

Medication Use By Medicare Beneficiaries Living in Nursing Homes and Assisted Living Facilities

This study compared medication use by Medicare beneficiaries living in nursing homes and assisted living facilities. Descriptions of medication use included the average number of drug mentions per month of stay (scheduled and PRN drugs), and prevalence and duration of therapy by major drug classes. Characteristics of institutionalized beneficiaries that were studied include: demographic, income, coverage, and residence, level of care, morbidity/mortality, and activities of daily living. Estimates of institutional drug use nationally served as the first benchmarks of prescribing patterns in nursing homes and assisted living facilities. Medicare beneficiaries in these settings typically received many different kinds of medications and at least a third had monthly drug regimens that include more than nine different medications. Future research should assess the appropriateness of these drug therapies. Associations detected between having Medicare supplemental coverage and prescribing practices in assisted living facilities prompt questions about sufficient drug coverage that deserve further investigation.

FEDERAL CONTACT: Andreas Frank, 202-690-6443 PIC ID: 7768

PERFORMER: University of Maryland, College Park, MD

The Contribution of Medication Use to Recent Trends in Old-Age Functioning

The purpose of this study was to explore the extent to which changes in medication use accounted for improvements in functioning among older Americans. In order to understand the consequences of disability declines for such programs, better insight into the causes driving the trend was needed. After nearly a decade of debate among academics and policy makers, a consensus has emerged that disability rates among older Americans have declined over the last fifteen years. However, the implications of such trends for publicly funded programs such as Medicare, Medicaid, and Social Security remain far from clear. Using several waves of the Health and Retirement Study (HRS), a nationally representative survey of non-institutionalized Americans ages 51-61, we examined changes during the 1990s in the prevalence of functional limitations and medication use associated with five highly prevalent and often debilitating chronic conditions: hypertension, diabetes, lung disease, stroke, and arthritis. For Americans of pre- retirement age (51-61) and for subgroups of this age group with specific chronic conditions the study explored the following questions: (1) has functioning improved, (2) has medication use increased, (3) do changes in medication use account for improvements in functioning, (4) have improvements over time been greater for those groups reporting medication use? We found that the average number of functional limitations declined. Statistically significant improvements were found among those reporting hypertension, diabetes, and arthritis. When we limited the analysis to 53-63 year olds in 1994 and 2000, we found no significant improvement in functioning, except among those reporting no arthritis, those who were not married, and those with no liquid assets.  Finally, we found no evidence that improvements in functioning were larger for Americans taking medications.

FEDERAL CONTACT: William Marton, 202-690-6443 PIC ID: 7770

PERFORMER: Philadelphia Geriatric Center, Philadelphia, PA

The Medicaid Buy-In Program: Lessons Learned from Nine “Early Implementer” States

This report discusses findings from case studies of nine states operating Medicaid Buy-In programs for working persons with disabilities. It is the second of three reports. The first included in-depth case studies of nine early implementer states entitled, “Medicaid Buy-In Programs: Case Studies of Early Implementer States.” For many individual Social Security Disability Insurance (SSDI) and Supplemental Security Income (SSI) recipients, the risk of losing Medicaid coverage linked to their case benefits was a powerful work disincentive. Eliminating barriers to health care and creating incentives to work can greatly improve financial independence and well-being. This paper examines state decisions concerning program eligibility, their approaches to estimating program enrollment and costs, and the patterns of program enrollment. The report seeks to assist stakeholders to design and implement Medicaid Buy-In programs and related work incentive initiatives. Medicaid Buy-In programs typically are managed by state Medicaid agencies with significant input from consumers and assistance from other state agencies. It was found that (1) stakeholder involvement was important in program design, (2) the Medicaid Buy-In program was linked to other employment supports, and (3) the state Medicaid agency usually worked with other state agencies to support persons with disabilities in the workplace. It was also found that eligibility standards and cost-sharing policies show considerable variation across the states and may have a significant impact on program enrollment. Finally, state policies on general Medicaid eligibility, SSI, and state SSI supplementation and federal policies on SSDI affected Medicaid access for working persons with disabilities.

FEDERAL CONTACT: Andreas Frank, 202-690-6443 PIC ID: 7556.1

PERFORMER: George Washington University, Washington, DC

Office of Health Policy

Getting In, Not Getting In, and Why? Understanding SCHIP Enrollment

This study focused on the findings related to enrollment in the State Children’s Health Insurance Program (SCHIP); findings from the study of retention are examined in a companion report. The Balanced Budget Act of 1997 established Title XXI in the Social Security Act, creating SCHIP. Title XXI provided states the authority and funding to increase health insurance coverage to low-income children by either expanding Medicaid or developing new “separate” child health programs. During the first three years of SCHIP, considerable policy attention was directed at state efforts to enroll eligible children. Yet the program was often criticized for getting off to a slow start and enrolling a small percentage of the target population. States have designed and implemented numerous strategies to streamline the application process with the goal of achieving higher enrollment. This study found that: (1) states have implemented many similar strategies for simplifying the SCHIP enrollment process, but simplifications to Medicaid policies and procedures are less extensive, (2) inconsistencies between SCHIP and Medicaid eligibility rules and requirements make the enrollment process more difficult and confusing for families, (3) in most states, less than 50 percent of applicants were approved for SCHIP eligibility, however, a large proportion of applicants appeared to be Medicaid eligible and were referred to that program, (4) large portions of SCHIP applications are denied for “procedural” reasons, yet this may be the unexpected “down side” of a simplified application process, (5) SCHIP programs are asking families about existing health insurance coverage as part of the application process, and are denying coverage to those who possess it, however, it appears that only a small proportion of applicants already have insurance, and (6) state SCHIP and Medicaid data systems are highly variable in their capacity to report eligibility outcome data.

FEDERAL CONTACT: Caroline Taplin, 202-690-7906 PIC ID: 7895.2

PERFORMER: Urban Institute, Washington, DC


Data Needs for the State Children’s Health Insurance Program

This report summarizes discussions at a workshop which brought together state SCHIP officials and researchers to share findings and methods that would inform the design, implementation, and evaluation of SCHIP at the state and national levels. The State Children’s Health Insurance Program (SCHIP) was established by Congress to provide health insurance to uninsured children whose family income was too high for Medicaid coverage but too low to allow the family to obtain private health insurance coverage. There are SCHIP programs in all of the states and the District of Columbia. The National Research Council, through the Committee on National Statistics, was asked to explore some of the ways in which data analysis could be used to promote achievement of the SCHIP goal of expanding health insurance coverage for uninsured children from low-income families. The panel concluded that data are insufficient in the individual states to provide a clear picture of the impact of SCHIP on the number of children who are eligible for the program, and the rate at which they are retained in the program once enrolled. This situation is due, in part, to the fact that sample sizes in national surveys are too small to provide detailed data for individual states. Some practices which could improve functioning of SCHIP include: (1) developing more uniform ways of estimating eligibility and health insurance coverage among the states, (2) sharing among the states effective methods for outreach, and (3) implementing longitudinal studies to track the movement of children among the various insurance statutes.

FEDERAL CONTACT: Caroline Taplin, 202-690-7906 PIC ID: 7893

PERFORMER: National Science Foundation Washington, D.C. and, Abt Associates Inc. Cambridge, MA


Is There a Hole in the Bucket? Understanding SCHIP Retention

The Balanced Budget Act of 1997 established Title XXI in the Social Security Act, creating the State Children’s Health Insurance Program (SCHIP). Title XXI provided states the authority and funding to expand health insurance coverage to low-income children by expanding Medicaid, developing new “separate” child health programs, or a combination of both approaches. During the first three years of SCHIP, considerable policy attention was directed at state efforts to enroll eligible children and states implemented numerous strategies to streamline the application process with the goal of achieving higher enrollment. This study focused on state efforts to enroll and retain children in SCHIP. As state SCHIP programs have matured, national enrollment has steadily increased. The Urban Institute analyzed information about states’ application and eligibility redetermination processes under SCHIP, as well as data on the outcomes of these processes, collected data. The major findings of this study and their implications fro future policy include: (1) states’ procedures for conducting SCHIP eligibility redetermination are similar to one another, however, these processes have not undergone the same level of reform in the interest of simplification, as have initial enrollment processes, (2) less than 50 percent of children appeared to be retaining SCHIP eligibility at redetermination, but further research is needed to understand what is reasonable to expect for this program, (3) high rates of parents who do not respond to renewal notices nor submit renewal applications for their children may be cause for concern, (4) denial of eligibility for “failure to pay premiums” may or may not address whether SCHIP cost sharing is affordable, and (5) state SCHIP and Medicaid data systems are highly variable in their capacity to report eligibility and redetermination outcome data.

FEDERAL CONTACT: Caroline Taplin, 202-690-7906 PIC ID: 7895.1

PERFORMER: Urban Institute, Washington, DC


The Cost of Medigap Prescription Drug Coverage

This study examined the price of standard Medigap policies that offer coverage for prescription drugs. Three of the ten standard Medigap policy forms (H, I, and J) cover prescription drugs. In most states, new Medicare beneficiaries who are either buying Medigap coverage for the first time, or looking to change their policy or insurer, have access to at least one of these plans. In every state, combined enrollment in Medigap plan that covers prescription drugs was very low. Such low enrollment may be related to non-price problems of access and also to the much higher price of Medigap policies that cover prescription drugs. This study considered the average cost of Medigap policies that covered prescription drugs in 1999 and also their marginal cost relative to other standard Medigap policies. This study focused on individual, not group, Medigap plans and found that premiums for Medigap policies covering prescription drugs are expensive by most measures. The cost of Medigap policies that cover prescription drugs is also high relative to other standard Medigap policies. Therefore, the Medigap market is neither a simple nor inexpensive place for Medicare beneficiaries to obtain prescription drug coverage.

FEDERAL CONTACT: Stephen Finan, 202-690-7387 PIC ID: 7348

PERFORMER: Mathematica Policy Research, Inc., Plainsboro, NJ


Premium Assistance Programs Under SCHIP: Not for the Faint of Heart?

For this study, three states--Massachusetts, Mississippi, and Wisconsin--were selected for in-depth examination because, until recently, they were the only states with federally approved State Children’s Health Insurance Program (SCHIP) premium assistance programs. Under The State Children’s Health Insurance Program (SCHIP), states have the option to subsidize employer premiums for low income children, and in some cases, their parents. Providing families with premium assistance may be a viable means of covering more uninsured children because this country’s health insurance system is dominated by employer-based coverage. This study included a literature review, a study of relevant SCHIP regulations, and a review and analysis of relevant information collected from the Urban Institute’s SCHIP evaluation--conducted as part of the “Assessing the New Federalism” project. The majority of information for this study was collected during telephone interviews and the protocol primarily focused on the impact of SCHIP regulations, state implementation experiences and lessons learned. The premium assistance programs examined in this study suggested several different findings that may be useful to other states considering such programs. They included the following: (1) outreach -- although states have recently gained considerable experience in targeting public program outreach efforts to families and community- based organizations, premium assistance programs pose a new outreach challenge to states as they directly involve and require the cooperation of employers, (2) enrollment -- Wisconsin and Massachusetts have learned they must engage in complex, time-consuming, and challenging processes for enrolling children (and often their parents) into premium assistance programs, and (3) cautious optimism -- Massachusetts and Wisconsin believe their premium assistance programs are worthwhile endeavors, but caution other states considering such programs from being overly optimistic about initial enrollment.

FEDERAL CONTACT: Caroline Taplin, 202-690-7906 PIC ID: 7895

PERFORMER: Urban Institute, Washington, DC

Office of Human Services Policy

13 Indicators of Quality Child Care: Research Update

This research brief highlighted the latest research studies related to the 13 indicators that have been completed since the publication of the “National Health and Safety Standards” in 1992. In some cases, research going back further than the last decade was used because of the classic nature of the studies and their significance to the 13 key indicators. The 13 indicators are: child abuse reporting and clearances, proper immunizations, staff: child ratio and group size, director and teacher qualifications, staff training, supervision/discipline, fire drills, administration of medication, emergency contact/plan, outdoor playground safety, inaccessibility of toxic substances, and hand washing/diapering. This review incorporated the latest research into an empirically demonstrated list of key regulatory indicators that statistically predicted positive outcomes for young children. This review offered the reader substantial evidence regarding the critical importance of these key indicators. The research literature over the past 20 years has demonstrated that these indicators do two things: statistically predict compliance with regulations in some states, and, demonstrate a relationship between compliance with these indicators and positive outcomes for young children. These key indicators supported and embraced the research literature related to child care quality.

FEDERAL CONTACT: Colleen Rathgeb 202-690-5937 PIC ID: 7804

PERFORMER: University of Colorado, Denver, CO


Understanding the Costs of the DOL Welfare-to-Work Grants Program

This report examined the costs of selected Welfare-to-Work (WtW) programs that operated with federal grant support. The main objectives of the WtW cost analysis were to understand the cost structure of these programs and factors that influenced their costs. The WtW grants program was one of several major federally funded initiatives to help welfare recipients and other low-income parents get jobs. In 1997, the Balanced Budget Act authorized the U.S. Department of Labor (DOL) to award $3 billion in  WtW grants to states and local organizations. These grants were intended to help the hardest-to-employ recipients of Temporary Assistance for Needy Families (TANF), and non-custodial parents, prepare for, find, keep, and advance in employment. It was found that WtW costs per participant reflected differences in program design. Also, on average, WtW programs cost more than WIN, less than Supported Work, and about the same as JOBS programs. And finally, future efforts could cost as much as, or more than, the current WtW program has. The WtW process and cost analyses leave considerable uncertainty about how more emphasis on basic or occupational training would affect costs. Integrating education and training into structured services could increase participation and costs of programs that target hard-to-employ individuals. However, to the extent that new policies require participants to pursue education and training activities concurrent with employment, participation and costs may continue to be limited (as it has been in WtW programs).

FEDERAL CONTACT: Alana Landey, 202-401-6636 PIC ID: 7868

PERFORMER: Mathematica Policy Research, Inc., Princeton, NJ


America’s Children: Key National Indicators of Well-Being 2002

This study, developed by the Federal Interagency Forum on Child and Family Statistics, is the sixth annual synthesis of information on the status of the Nation’s most valuable resource, our children. This      report presented key indicators of the well-being of children. These indicators were monitored through official Federal statistics covering children’s economic security, health, behavior and social environment, and education. The report presented data on eight key contextual measures and included a special feature showing children of at least one foreign-born parent. The 20 agencies of the Forum introduced improvements in the measurement of several of the indicators presented the year before. The report provided a broad annual summary of national indicators of child well-being and monitored changes in these indicators over time. According to the report, the infant mortality rate decreased since 1983. As in previous years, the report showed that most children are in very good or excellent health. However, children living in poverty were less likely than children in higher-income families to be in very good or excellent health. Nevertheless, the gap in health status by income narrowed over the past few years. The adolescent birth rate also declined. As a result of language and cultural barriers confronting children and their parents, children with foreign-born parents may need additional resources at school and at home to successfully progress in school and transition to adulthood. There is still more work to be done, but the chances of the poorest children have been improved and they can share in the advances in health.

FEDERAL CONTACT: Martha Moorehouse, 202-690-6939 PIC ID: 7297

PERFORMER: Westat, Inc., Rockville, MD


Assessing the Family Circumstances of TANF Applicants and Leavers in Contra Costa and Alameda Counties

Limited information existed about the well-being of families affected by welfare reform. To add to our knowledge in this area, Alameda and Contra Costa counties, in California, initiated a study to describe the circumstances of three groups of families: (1) Leavers--families leaving CalWORKs (California’s Welfare Reform Program) and remaining off aid at least two consecutive months, (2) Informally Diverted--families applying for but denied CalWORKs assistance for non-financial reasons and not receiving CalWORKs for at least two months following the denial, and (3) Transition to Child Only--families transitioning from a CalWORKs case with aided adults and children, to one with aided children only. Conditions are improving for leavers and the informally diverted. Leavers were doing much better than the transition to child-only cases. Even though median income was well above poverty, some leavers and informally diverted families were very poor, and most of these families were not receiving CalWORKs. The finding that a very high proportion of child-only status families had been long-term aid recipients prior to the point of transition could be useful in efforts to identify CalWORKs families at risk of being sanctioned for purposes of developing preventive policies, such as targeted home-visiting programs designed to uncover and   address non-compliance factors.

FEDERAL CONTACT: Julie Isaacs, 202-690-6805 PIC ID: 7777

PERFORMER: Sphere Institute, Burlingame, CA


Disaggregating the TANF Child-Only Caseload in Three States

The purpose of this study was to provide more detailed information about the make-up and trends of the Temporary Assistance for Needy Families (TANF) child-only population in three States. This study used administrative data, case file reviews, and interviews with program and policy staff. Child-only cases were those in which benefits are paid only on behalf of minor children, and there were no adults in the household receiving assistance. In some cases the children lived with adults other than their parents: in such cases, the caretaker received assistance on behalf of the child, but not for himself/herself. In other cases, a parent was in the household but was ineligible for benefits for one of several reasons, including: (1) receipt of SSI, (2) unqualified alien status, or (3) imposed sanctions. The letter cases make up 21 percent of the TANF caseload, a proportion that has grown significantly in recent years.

FEDERAL CONTACT: Laura Radel, 202-690-5938 PIC ID: 7188

PERFORMER: The Lewin Group, Fairfax, VA


Final Synthesis Report of Findings from ASPE’s “Leavers” Grants

The synthesis of fifteen studies provided in this report included information on welfare leavers’ employment and earnings, public assistance program participation, income and poverty status, material hardships, and child well-being. The studies showed many differences in specific measures of families’ post-TANF experiences, reflecting in part the differences in context across these areas, such as welfare policies, economic conditions, and the characteristics of leavers. Despite these differences, a number of clear general patterns emerged. Employment and Earnings: A little more than a third held jobs in all four quarters after exiting TANF. No single barrier to work consistently affected a majority of leavers; however, a substantial minority of leavers must have overcome childcare and health-related problems in order to work. Program Participation: A quarter to a third of families who left welfare returned to TANF at some point in the first year after exit. Household Income: Average monthly family income for leavers generally hovered near the poverty line. In the four studies that explicitly examined poverty rates of leaver families, on average, over half of leavers were poor. Child Well-Being: One-tenth to one-quarter of leaver families have children without health insurance. For childcare, a substantial percentage of leaver families relied on parental care. The fifteen ASPE-funded leaver studies reviewed here provided a considerable amount of information on the status of families leaving welfare. This synthesis focuses on key outcomes and measures of well-being that are commonly reported in these studies. In addition to these common elements, the individual studies also contain a rich array of information and subgroup analyses pertinent to understanding the status of former welfare recipients in their respective geographic areas.

FEDERAL CONTACT: Julia Isaacs, 202-690-7882

PIC ID: 7368

PERFORMER: Urban Institute, Washington, DC


How are Immigrants Faring After Welfare Reform? Preliminary Evidence from Los Angeles and New York City

Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) enacted in 1996, the  eligibility of legally-admitted immigrants for means-tested federal benefits significantly limited, particularly for immigrants entering the United States after the law was passed. Findings from a survey of immigrants in Los Angeles County and New York City (NYC) that yield new insights about the status of immigrants and eligibility. The report summarized data from a survey of 3,447 immigrant families. The survey was conducted in late 1999 and early 2000 by the Survey Research Center of the University of California at  Los Angeles. The survey described the living conditions of about 4.8 million people in Los Angeles County and 3.5 million people in New York City who lived in immigrant families. The study found that many immigrants in LA County and NYC, particularly those who were not citizens, lived in families experiencing economic hardship. The hardship measures examined included poverty, food insecurity, moderate  hunger, housing problems, and lack of health insurance. When compared with native citizen families, the immigrant families in the survey had consistently lower incomes and higher hardship levels, despite relatively high employment rates. About 80 percent of the children in these immigrant families were native-born citizens, and they shared economic hardship with their immigrant parents and non-native born siblings. The findings showed reduced benefit use and substantial levels of need among immigrant families in program areas directly affected by welfare reforms’ immigrant eligibility restrictions.

FEDERAL CONTACT: David Nielsen, 260-615-2612 PIC ID: 6747

PERFORMER: Urban Institute, Washington, DC


Low-Income and Low-Skilled Workers’ Involvement in Nonstandard Employment

The role of alternative work arrangements--temporary help, independent contractors, on-call workers, and contract company workers--has caught the attention of policy makers and academic researchers. Current research indicates that 1 in 10 workers are employed in one of these four alternative work arrangements and employment in the temporary help services industry grew five times as fast as non- farm employment between 1972 and 1997. This growth is likely to have important implications for low- income workers, since the establishment of the Temporary Assistance for Needy Families (TANF) block grant, authorized by the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996, which dramatically transformed the nation’s welfare system. This welfare reform resulted in an increasing number of low-income individuals entering the labor force. Thus, alternative work arrangements, especially for those with limited work histories, might be expected to be a natural pathway to work for such workers. Findings from the final report include: (1) workers who were at risk of welfare dependency were more than twice as likely to be in alternative work arrangements as other workers; (2) the number of industries utilizing temporary workers has increased, particularly among industries where the median education level of temporary workers is very high--suggesting that low-skilled workers would be increasingly less able to compete for these jobs; (3) at-risk workers fared worse in nonstandard jobs than did other workers in such arrangements across a variety of dimensions (wages, incidence of part- time jobs, job duration, and employer-provided benefits). FEDERAL CONTACT: Alana Landey, 202-401-6636 PIC ID: 7367

PERFORMER: Urban Institute, Washington, DC


Moving People from Welfare to Work: Lessons from the National Evaluation of Welfare-to-Work Strategies

How to increase employment among welfare recipients has long been debated. Over the past three decades, federal and state policymakers have created a variety of programs with the common goal of moving people from welfare to work By laying out the lessons learned from the National Evaluation of Welfare-to-Work Strategies (NEWWS), this research synthesis provided answers to critical questions in the welfare-to-work policy discussion. NEWWS examined the long-term effects on welfare recipients and their children of 11 mandatory welfare-to-work programs, operated in seven sites, that took different approaches to helping welfare recipients find jobs, advance in the labor market, and leave public assistance. A central question of the evaluation was: “What program strategies work best, and for whom?” Under study were two primary pre-employment approaches--one that emphasized short-term job search assistance and encouraged people to find jobs quickly and one that emphasized longer-term skill-building activities (primarily basic education) before entering the labor market--and a third approach that mixed elements of the other two. The findings from NEWWS provided compelling evidence that these programs succeeded in achieving many program goals. All the programs increased people’s employment and earnings and decreased their receipt of welfare, thus resulting in gains in people’s self- sufficiency. Notably, mothers who were single parents achieved these benefits with few indications of either harm or benefit of their children’s well-being. However, none of the programs met the implicit goal of making people materially better off. The lessons from NEWWS remain highly relevant in the current welfare reform environment and beyond.

FEDERAL CONTACT: Audrey Mirsky-Ashby, 202-401-6640 PIC ID: 7089.11

PERFORMER: Manpower Demonstration Research Corporation, New York, NY


National Survey of Homeless Assistance Providers and Clients: Supplemental Analysis

The National Survey of Homeless Providers and Clients conducted by the U.S. Census Bureau on behalf of 12 sponsoring federal agencies, was designed to provide data to support policy development and  research initiatives. During the survey, the Census Bureau drew a representative sample of all known providers of services for the homeless in the United States. It randomly selected clients from that sample: more than 4,000 clients of the selected service providers were interviewed. That sample of clients supported analyses of subgroups of interest to the sponsoring agencies (e.g., clients living with mental health problems). The findings and methods of the initial study were described in the technical report “Homeless Programs and the People They Serve.” The study’s rich data had limitations. Nevertheless, the data allowed analysts to estimate problem prevalence among a broad population of homeless people. Analysts can also use client reports of current symptoms to identify factors associated with problems among homeless individuals.

FEDERAL CONTACT: Anne Fletcher, 202-690-5739 PIC ID: 7849.1

PERFORMER: Westat, Inc., Rockville, MD


State Policies to Promote Marriage

The 1996 Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) stated four broad goals for Temporary Assistance for Needy Families (TANF): (1) provide assistance to needy families so that children may be cared for at home or in the homes of relatives; (2) end the dependence of needy parents on government benefits by promoting job preparation, work, and marriage; (3) prevent and reduce the incidence of out-of-wedlock pregnancies and establish annual goals for preventing and reducing the incidence of these pregnancies; and (4) encourage the formation and maintenance of two- parent families. As TANF was reauthorized in 2002, an important topic was whether the program was meeting its goals. States have passed laws or proposed legislation in every area studied for this project. Some states had activities in multiple areas. It was found that it was important that state officials publicly focus on marriage-related issues through statewide campaigns, commissions, and proclamations. Nine states have undertaken an activity in this area. State campaigns included media projects that extolled the virtue of marriage and larger-scale initiatives, such as a statewide effort to curb divorce rates. Commissions included “summits” that brought together diverse groups to discuss marriage-strengthening policies and commissions charged with implementing specific policies. Finally, issuing proclamations recognizing the importance of marriage and reaffirming marriage’s special status as the foundation for healthy families promoted marriage.

FEDERAL CONTACT: Kelleen Kaye, 202-401-6634 PIC ID: 7756

PERFORMER: Lewin Group Fairfax, VA and The Johns Hopkins University, Baltimore, MD


Studies of Welfare Populations: Data Collection and Research Issues

The Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) was enacted in 1996. The Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs of the National Research Council was formed in 1998 to review the evaluation methods and data that are needed to study the effects of welfare reform. The panel realized that the database for conducting studies of welfare reform had many deficiencies and required attention by policy makers and research analysts. The final report concluded that welfare reform evaluation imposed significant demands on the data infrastructure for welfare and low-income populations and that inadequacies in the nation’s data infrastructure for social welfare program study constituted the major barrier to good monitoring and evaluation of the effects of reform. The panel concluded that national-level surveys were being put under great strain for PRWORA research given their small sample sizes, limited welfare policy-related content, and, often, high rates of non-response. The panel concluded that major new investments are needed in the data infrastructure for analysis of welfare and low-income populations.

FEDERAL CONTACT: Susan Hagan, 202-690-8698 PIC ID: 7145.2

PERFORMER: National Academy of Sciences, Washington, DC


The 1996 National Survey of Homeless Assistance Providers and Clients: A Comparison of Faith-Based and Secular Non-Profit Programs

This study examined data from National Survey of Homeless Assistance Providers and Clients (NSHAPC) to determine more thoroughly the role that faith-based programs played in the larger context of   homeless assistance. The study had an explicit focus on comparing homeless assistance programs administered by faith-based versus secular non-profit service agencies. One of the most dramatic findings to have emerged from the 1996 NSHAPC was the tremendous growth in the number and variety of homeless assistance programs during the late 1980s and early 1990s. While much of this growth was fueled by new investments of public funds, most faith-based non-profits operated with little or no government funding, yet they played a critical role in helping homeless people. The study provided a  basic but comprehensive picture of the numbers and characteristics of the two types of homeless assistance programs. NSHAPC documented just fewer than 40,000 homeless assistance programs operating on an average day in February 1996. Faith-based non-profits ran about a third of these programs, including the majority of all food programs and one-quarter of all shelters and drop-in centers. Faith-based providers served a more diverse group of clients than did secular non-profits. The proportion of programs serving each client group--single men, single women, females with children, other households with children, and youth--was higher among faith-based programs than it was among secular non-profits. Over all programs, faith-based providers were much less likely to have a special focus than were secular providers. Several factors may have accounted for this, including differences in the types of programs run by faith-based versus secular non-profits, as well as the types and diversity of their clients. The NSHAPC data analyzed here provided yet more evidence of the continuing importance of faith-based organizations in serving people who were homeless or on the brink of homelessness.

FEDERAL CONTACT: Brenda Benesch, 202-260-0382 PIC ID: 7849

PERFORMER: Urban Institute, Washington, DC


The Evaluation of Abstinence Education Programs Funded Under Title V Section 510: Interim Report

This report presented interim findings from an independent, federally funded evaluation of the abstinence education programs authorized under PRWORA and defined under Title V, Section 510 (b)(2)(A-H) of the Social Security Act. This report drew most heavily on four years of implementation experiences in a selected group of abstinence education programs. Later reports from the evaluation presented estimates of short- and long-term program impacts, as well as studies on special topic areas. The methodology consisted of an implementation and process analysis which used program documents, program observations, focus groups with program participants and parents, and interviews with program staff and community leaders. The impact analysis used longitudinal survey data for groups of youth randomly assigned to program and control groups. Enrollment in the impact evaluation study samples spanned three school years and was completed in fall of 2001. Early lessons indicated that: (1) abstinence funds were changing the local landscape of approaches to teenage pregnancy prevention and youth risk avoidance, (2) most abstinence education programs offered more than a single message of abstinence, (3) most participants reported favorable feelings about their program experience, (4) abstinence education programs faced real challenges addressing peer pressure and the communication gulf between parents and children, and (5) local schools were valuable program partners, but establishing these partnerships was sometimes difficult.

FEDERAL CONTACT: Meredith Kelsey, 202-690-6652

PIC ID: 7491

PERFORMER: Mathematica Policy Research, Inc., Plainsboro, NJ


The Evaluation of the Tribal Welfare-to-Work Grants Program: Initial Implementation Findings

Over the past 20 years, the federal government has increasingly supported tribal self-governance and self-determination. Indian tribes and tribal consortia have been explicitly included in federal welfare reform initiatives such as TANF, the Child Care Development Fund, WtW, and NEW. Congress and federal agencies administering these programs have supported Indian self-determination and tribal consultation in formulating legislative provisions and in developing policies and regulations. Legislation and regulations permit tribes to operate programs and, in recognition of their special circumstances, allow some degree  of flexibility in program operation. The challenge of ending welfare is nowhere more daunting in the United States than on Indian reservations and in Alaska Native villages. While the circumstances of each tribe are unique, most tribes face economic, education, housing, health, and other problems at levels of severity rarely seen in most other American communities. The experiences of the 10 tribal grantees included in this study highlight the challenges commonly faced in Indian country and suggest some of the following lessons: (1) improving coordination with other programs, especially TANF, is critical to successful program implementation, (2) states can be an important source of support and technical assistance, and (3) cooperation with states can strengthen child support enforcement and fatherhood initiatives.

FEDERAL CONTACT: Alana Landey, 202-401-6636

PIC ID: 7785

PERFORMER: Mathematica Policy Research, Inc., Princeton, NJ


The Implementation of the Welfare-to-Work Grants Program

This report presented findings from the process and implementation analysis part of the evaluation of implementation of the Welfare-to-Work grants program, and described the service delivery operations of programs funded with Welfare-to-Work grants in eleven study sites. This report was based on information collected through two rounds of site visits in 1999 and 2001, and management information system data maintained by the programs on participants and services. The $3 billion Welfare-to-Work (WtW) grants program established by Congress as part of the Balanced Budget Act (BBA) of 1997 provided funds to over 700 state and local grantees. Congress appropriated funds for FY1998 and FY1999, and grantees were allowed five years to spend their grant funds. The intent of the grants program, administered at the national level by the U.S. Department of Labor (DOL), was to supplement the welfare reform funds included in the Temporary Assistance for Needy Families (TANF) block grants to states, which were authorized under the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996. WtW funds were intended to support programs, especially in high-poverty communities, to assist the least employable, most disadvantaged welfare recipients and non-custodial parents (Naps) make the transition from welfare to work. This part of the multi-year evaluation found that three general program models for delivering services to the hard-to-employ were implemented in programs in the study sites. WtW grantees focused on the most disadvantaged, as specified in congressionally established provisions, but most programs have faced difficulties enrolling eligible individuals. Also, WtW programs went beyond job readiness and self-directed job search assistance in the sense that they provided intensive individualized case management, coaching or support; and many programs also included more intensive developmental components and activities.

FEDERAL CONTACT: Alana Landey, 202-401-6636 PIC ID: 7868.1

PERFORMER: Mathematica Policy Research & Urban Institute, Washington, DC

Office of Planning and Policy Support

Evaluation Synthesis: Tobacco Control Programs in Communities of Color

This study assessed the extent to which Communities of Color were being targeted and summarized the key findings for Communities of Color. The study was conducted utilizing published scientific literature, state reports, and tobacco control Web sites for eight state case studies. The eight states included in this evaluation were: Arizona, California, Florida, Massachusetts, Maryland, Mississippi, Washington, and Texas. These states were selected based on their demonstrated success in reducing rates of initiation of smoking, increasing smoking cessation, or reducing environmental tobacco smoke (ETS). They also had a presence of large numbers of minority populations or multiple, small segments of racially and ethnically diverse communities. Also, these states allotted funds for tobacco prevention and control. The key findings indicated that: (1) all eight states had adopted the goal of prevention and have or will soon have cessation as a goal, (2) published evaluations of interventions pertaining to  Communities of Color were few and focus on adolescents, (3) outcome evaluations of interventions pertinent to Communities of Color were difficult to find, (4) a published evaluation supports the importance of mass media campaigns in combination with other interventions in preventing the initiation of tobacco use in Communities of Color, (5) the published literature supported the effectiveness of community education in Communities of Color for smoking cessation and reduction of ETS and was suggestive of an effect in decreasing smoking initiation, and (6) one study provided evidence that legislation prohibiting sales to minors was effective in all ethnic neighborhoods.

FEDERAL CONTACT: Vijaya ChannahSorah, 202-260-3815

PIC ID: 7815

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Office of Public Health and Science


To provide advice on public health and science to the Secretary of Health and Human Services, to provide executive direction to program offices within the Office of Public Health and Science (OPHS), and, at the direction of the Secretary, to coordinate crosscutting public health and science initiatives in the Department.

Evaluation Program

The Office of Public Health and Science (OPHS) provides advice, policy and program coordination, and leadership in the implementation, management, and development of activities related to public health and science, as directed by the Secretary. OPHS helps HHS conduct broad-based public health assessments to better address and solve public health problems. It assists other parts of HHS in anticipating future public health issues and helps ensure that HHS designs and implements appropriate approaches, interventions, and evaluations that will maintain, sustain, and improve the health of the Nation. OPHS provides leadership and policy recommendations on population-based public health and science and, at the Secretary’s direction, leads or coordinates initiatives that cut across agencies and operating divisions. In addition, OPHS communicates and interacts, on behalf of the Secretary, with national and international professional and constituency organizations on matters of public health and science. Finally, OPHS’s unique role allows it to use its resources to link important HHS programs or fill gaps in areas needing better policy formulation and coordination.

OPHS’ evaluation strategy focuses on public health and science issues that cut across multiple interests of the operating divisions and requires a coordinated approach to achieve the most effective results. OPHS evaluations support the Assistant Secretary for Health as the Secretary’s senior advisor for public health and science. OPHS also conducts evaluations specific to the needs of the programs operated from the offices located within OPHS, such as emergency preparedness, women’s health, minority health, disease prevention and health promotion, and research integrity. The ten HHS Regional Health Administrators have access to some evaluation funds. In keeping with its role within the Department, OPHS does not undertake evaluations more appropriately undertaken by operating divisions of HHS or by the Assistant Secretary for Planning and Evaluation (ASPE).

Summary of Findings

Office of Minority Health

Consumers’ Experiences with Culturally and Linguistically Appropriate Services (CLAS) Study Design Report

The availability of culturally and linguistically appropriate services (CLAS) was widely regarded as essential to the delivery of efficient, quality health care in the United States. The Office of Minority Health (OMH) had addressed CLAS through numerous efforts, including this current study. OMH contracted with  Health Systems Research Associates to assess the feasibility of conducting a study of consumers’ experiences with CLAS. The feasibility assessment and consultation process included these main parts: (1)  developing a consultant work plan, (2) conducting a literature review/scan and key informant interviews within and outside the Federal government, (3) designing three study options for OMH consideration, with recommendation for one option, and (4) developing a detailed study design for that option. The report recommended the following study option (out of a set of three): Market Area Case Studies--Multiple Languages, Providers. This option addressed consumers’ experiences by conducting a set of 4-6 case studies of consumers and other stakeholders in diverse markets. This option, according to the report, provided the OMH with considerable flexibility with regard to study design, policy focus, and budget, and allows participation by the OMH in the study process. In addition, this option provided the potential for a modest exploratory study to provide a foundation for further research, based on review and analysis of related research and studies, key informant interviews, and case study observations and findings.

FEDERAL CONTACT: Valerie Welsh, 301-443-9923 PIC ID: 7710

PERFORMER: Health Systems Research Associates, Chevy Chase, MD


National Youth Sports Program Evaluation Project

The National Youth Sports Program (NYSP), initiated in 1968 as an outreach program of the National Collegiate Athletic Association, was designed to enhance the quality of life for youth from low-income families by providing academic and athletic activities through a five-week summer camp. Key NYSP objectives include the utilization of sports and educational activities for the purpose of exposing youth to educational opportunities, physical fitness, life skills, citizenship, interactive athletic skills, and health issues. The purpose of this funded project with the Inter-University Program for Latino Studies was to develop tools and resources that could be used by NYSP managers and staff to assess the program’s potential effects on knowledge, attitudes, and behaviors of youth participating in the summer program. Primary activities that were conducted included: a qualitative review of program activities via site visits at selected NYSP locations; development and pilot-testing of pre- and post-survey instruments designed to assess NYSP participants’ level of exposure to risk behaviors, their knowledge and beliefs about health and risk factors, and short-term impacts of the health-related curriculum and activities on participant knowledge; and revisions to the instruments based on pilot results and feedback on program performance. A comprehensive Health Resource Guide was compiled listing health education materials covering a wide range of health topics. The tools and resource guide can be used at NYSP sites to conduct self-assessments to determine whether program objectives are being met and to improve the program’s structure and content as indicated.

FEDERAL CONTACT: Joyce Heinonen, 301-594-0769 PIC ID: 7709

PERFORMER: The Inter-University Program for Latino Research, University of Notre Dame, South Bend, IN


Patients Who Don’t Speak English: Improving Language Minorities’ Health Care with Professional Interpreters

This was a study of the effects of professional interpreter services on health visit levels and patterns of a sample of limited English proficient (LEP) patients. LEP patients are those who speak a primary language other than English and who cannot speak English at all or speak English so poorly that they cannot communicate in English without assistance. Language services, including the use of professional interpreters, were seen as a facilitator variable in conventional health care models because they help patients with limited English to communicate with health care providers, and therefore access medical care. It was expected that language services were necessary to significantly reduce language barriers to health care. Findings provide support for the hypothesized role of interpreter services. LEP patients increased their health visits after the implementation of interpreter services. The increased utilization was larger for groups that can be considered to be most vulnerable to language barriers in accessing health care, including the elderly, the poor, and patients who had below average health visits prior to the implementation of interpreter services. Given the scarcity of systematic research on the effects of interpreter services on LEP patients’ health care, this study’s findings represented new and welcome evidence. This study has shown that organizations involved in health care, such as those providing health insurance, medical services, and language services, can play critical roles in gathering the kind of information that may be productively analyzed in further research.

FEDERAL CONTACT: Guadalupe Pacheco, 301-443-5084 PIC ID: 7858

PERFORMER: Department of Sociology, Portland State University, Portland, OR

Substance Abuse and Mental Health Services Administration


SAMHSA’s mission is to build resilience and facilitate recovery for people with or at risk for substance abuse and mental illness.

Evaluation Program

The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its programs and individual grant projects to assess the effectiveness of prevention, treatment, and rehabilitation approaches and systems of care. Evaluation is also conducted to ensure accountability for federal funds and to measure results toward SAMHSA’s programmatic and policy objectives. In compliance with the Government Performance and Results Act (GPRA), SAMHSA is improving performance management and results by identifying annual, long-term and cost-efficiency performance measures to manage its programs.

SAMHSA has an integrated model of evaluation and planning. Strategic planning identifies priorities that drive the development of grant programs and evaluations. The formulation of programmatic and evaluation priorities includes consultation with SAMHSA Center Advisory Councils, with other HHS agencies, and with experts in the fields of evaluation and service delivery. Early and continuous coordination of program planning and evaluation design results in the articulation of program objectives that may be evaluated. Planned evaluations demonstrate the extent to which the grant programs have achieved their objectives, and SAMHSA translates these results into information that can be used for program and policy development. The strategic planning and policy development processes then use these results to refine SAMHSA’s priorities and performance objectives.

The specific type of evaluation required for a grant program is dependent on the type and purpose of the program. To the greatest extent possible, SAMHSA encourages the use of comparable data elements and instruments across its evaluations to implement a comprehensive evaluation system and to minimize respondent burden. Efforts to improve evaluation are continuing and SAMHSA is committed to using systematic approaches in using data to accomplish its mission.

Summary of Findings

Center for Mental Health Services

Evaluation of the Community Action Grant Program, All Phase I Grantees

The study assessed the impact of the Community Action Grant Program. Variables of interest include implementation status of all programs adopted by Phase I grantees; description of consensus building processes and implementation strategies; and barriers to consensus building and practice implementation. The evaluation included a special focus on the specific experiences of grantees targeting racial/ethnic minority populations. The contractor collected qualitative and quantitative data in order to make recommendations on the grantee reporting system, data needed from the grantees to advance CMHS’ understanding of knowledge development and the adoption of evidence-based practices, and the management of the grant program by project officers.

FEDERAL CONTACT: Neal Brown, 240-276-1960 PIC ID: 7719.1

PERFORMER: Association for the Study and Development of Community, NYD


Overview of the ACCESS Program

The Access to Community Care and Effective Services and Supports (ACCESS) program, evaluated the integration of service systems and their impact on outcomes for homeless persons with severe mental illness. The ACCESS program provided funds and technical assistance to nine community sites to implement strategies for system change that would promote systems integration. These experimental sites, along with nine comparison sites, received funds to support outreach and assertive community treatment for 100 clients a year for four years at each site. Data on the implementation of system change strategies were collected from 1994 to 1998 during annual visits to the sites. Data on changes in systems integration were obtained from interviews with key informants from relevant organizations in each community. Client outcome data were obtained at program entry and three and 12 months later from 7,055 program participants across the four annual client cohorts at all sites. Detailed findings from the ACCESS evaluation are presented in four journal articles discussed below

FEDERAL CONTACT: Frances Randolph, 240-276-1892 PIC ID: 4980.1

PERFORMER: R.O.W. Sciences, Inc., Rockville, MD


Integration of Service Systems for Homeless Persons With Serious Mental Illness Through the ACCESS Program

This study assessed core questions used in the ACCESS evaluation: Does implementation of system- change strategies lead to better integration of service systems? The study was part of the five-year service demonstration program seeking to enhance integration of services’ delivery for seriously mentally ill homeless persons. Data were gathered -- from nine randomly selected experimental sites and nine comparison sites in 15 large cities -- on the extent to which each site implemented systems integration strategies (and systems integration among community agencies) across five service sectors: mental health, substance abuse, primary care, housing, and social welfare and entitlement services. Integration was measured across all organizational relationships in the local service networks (overall systems integration) and across relationships involving the primary ACCESS grantee organization (project-centered integration). Contrary to expectations, the nine experimental sites did not demonstrate significantly greater systems integration than the nine comparison sites. However, the experimental sites demonstrated better project-centered integration than the comparison sites. Moreover, more extensive implementation of strategies for system change was associated with higher levels of systems integration as well as project-centered integration at the experimental sites and the comparison sites. The ACCESS demonstration was successful in terms of project-centered integration but not system integration.

FEDERAL CONTACT: Frances Randolph, 240-276-1892 PIC ID: 4980.2

PERFORMER: R.O.W. Sciences, Inc., Rockville, MD


Service Systems Integration and Outcomes for Mentally Ill Homeless Persons in the ACCESS Program

The authors evaluated the second of the two core questions around which the ACCESS evaluation was designed: Does better integration of service systems improve the treatment outcomes of homeless persons with severe mental illness? The program provided technical support and about $250,000 a year for four years to nine sites to implement strategies to promote systems integration. These sites, along with nine comparison sites, also received funds to support outreach and assertive community treatment programs to assist 100 clients a year at each site. Outcome data were obtained at baseline and three and 12 months later from 7,055 clients across four annual cohorts at all sites. Clients at all sites demonstrated improvement in outcome measures. However, the clients at the experimental sites showed no greater improvement on measures of mental health or housing outcomes across the four cohorts than those at the comparison sites. Implementation of systems integration strategies was unrelated to these outcomes. Clients of sites that became more integrated, regardless of degree or whether the sites were experimental or comparison, had progressively better housing outcomes. Interventions designed to increase systems integration in the ACCESS demonstration did not result in better client outcomes.

FEDERAL CONTACT: Frances Randolph, 240-276-1892 PIC ID: 4980.3

PERFORMER: R.O.W. Sciences, Inc., Rockville, MD


Lessons From the Evaluation of the ACCESS Program

The authors summarized the main findings of the ACCESS program and offer lessons for policy makers. Data from studies at the site level and the client level are summarized and synthesized with the authors’ collective experience with the ACCESS program. The results of the evaluation suggest that although service systems integration can be improved, targeted efforts to implement strategies for integration do not produce better client outcomes. Efforts to integrate service systems can be supported by their effects on some organizational relationships within the mental health service system but not by their widespread effects across human services or their direct effects on clients.

FEDERAL CONTACT: Frances Randolph, 240-276-1892 PIC ID: 4980.4

PERFORMER: R.O.W. Sciences, Inc., Rockville, MD

Center for Substance Abuse Prevention

Cross-site Study of the Children of Substance Abusing Parents Initiative

A cross-site study was conducted on the Children of Substance Abusing Parents (COSAPs) initiative that targeted COSAPs and their caregivers. Each study site was designed to test substance abuse prevention strategies for a specific age cohort of COSAPs and their substance-abusing caregivers. Age cohorts for the target children included 6- to 8-year-olds, 9- to 11-year-olds, and 12- to 14-year olds. An outcome and a process evaluation were conducted. The outcome study used a quasi-experimental design that involved nonequivalent comparison groups and repeated measures. Each study site formed appropriate comparison groups of index COSAPs and index caregivers. In addition to using dosage data, the process evaluation used program-level data collected by means of site visits. The findings showed the Center for Substance Abuse Prevention programs had many positive effects on risk and protective factors for the COSAPs. Programs for younger COSAPs resulted in increased self-control, school bonding, school performance, and perception of alcohol, tobacco, and other drugs (ATOD) harm. Positive outcomes for older COSAPs began to emerge 6-months post intervention when there was an increased perception of ATOD harm, and reductions in problem behaviors. Although the programs did not affect substance use among the COSAPs, the rate of ATOD use among the children were very low at baseline, making it difficult to measure reductions in use. The analysis found no significant negative program outcomes.

FEDERAL CONTACT: Soledad Sambrano, 240-276-2416 PIC ID: 7888

PERFORMER: EMT Associates, St. Louis, MO

Center for Substance Abuse Treatment

Cost Effectiveness and Cost Benefit Analysis of Substance Abuse Treatment:  A Bibliography

This bibliography lists books, published articles and research/evaluations, and government documents (including “Web” publications) published since 1980 that focus on the costs of substance abuse treatment, methods for estimating the costs of treatment, and studies of the cost effectiveness and cost benefits of substance abuse treatment. This document also identifies trends and areas where there are gaps in the literature.

FEDERAL CONTACT: Nita Fleagle, 240-276-2823 PIC ID: 7811.2

PERFORMER: Caliber Associates, Fairfax, VA


Cost Effectiveness and Cost Benefit Analysis of Substance Abuse Treatment:  A Literature Review

This literature review summarizes the major findings from books, published articles, research and evaluations, and government documents (including “Web” publications) published since 1980, which focus on the cost effectiveness and cost benefits of substance abuse treatment. Cost effectiveness and cost benefit studies play an important role in evaluating existing and alternative substance abuse approaches and in assessing new treatment methods.  Evaluating the outcomes and costs of treatment is necessary in order to determine how to more efficiently allocate scarce resources.  This document is intended to assist policymakers, researchers/evaluators, and treatment providers to identify and acquire evidence-based information. Policymakers and researchers/evaluators can use this information to help inform decisions of whether or not an increase in effectiveness justifies an increase in cost of a treatment. Treatment providers need this information when they seek funding from public agencies.

This review also explores trends and areas where there are gaps in the literature. FEDERAL CONTACT: Nita Fleagle 240-276-2823

PIC ID: 7811.1

PERFORMER: Caliber Associates, Fairfax, VA

Appendix A - Evaluation in the Department of Health and Human Services

Evaluation plays an integral role in carrying out the HHS mission. Assessing various aspects of agency program performance allows staff to identify means of improving that performance. The HHS evaluation function has three goals: (1) to provide information on HHS programs that helps government officials and members of the Congress make decisions related to programs, policies, budgets, and strategic planning; (1)  to help HHS managers improve program operations and performance; and (3) to disseminate evaluation results and methodological tools useful to the larger health and human services community of state and local officials, researchers, advocates, and practitioners for improving the performance of their programs.

For HHS, evaluation is the assessment of the performance (efficiency, effectiveness, and responsiveness) of HHS programs or strategies through the analysis of data or information collected systematically and ethically, and the effective use of resulting information in strategic planning, program or policy decision making and program improvement. As mentioned in the foreword of this report, evaluations serve one or more of the following:

  • Program effectiveness determines the impact of HHS programs on achieving intended goals and objectives.
  • Performance measurement monitors annual progress in achieving departmental strategic and performance goals. To support performance measurement, we are investing evaluation funds to develop and improve performance measurement systems and the quality of the data that supports those systems.
  • Environmental assessment seeks to understand the forces of change in the health and human services environment that influence the success of our programs and the achievement of our goals and objectives. Such understanding allows us to adjust our strategies and continue to deliver effective health and human services.
  • Program management reflects the need of program managers to obtain information or data  helpful for designing and managing programs. These evaluations generally focus on developmental or operational aspects of program activities and provide understanding of services delivered and populations served.

Evaluation Resources

Evaluation activities of the various HHS agencies are largely supported through two funding mechanisms: direct use of program funds and use of special legislative set-aside authorities for evaluation. The first is a common mechanism by which programs managers have discretionary authority to use appropriated program funds to support contracts that will design, implement, and analyze evaluation data. In some cases, a program’s legislative authority calls for a specially mandated evaluation, and program funds are used directly to support the evaluation.

The second mechanism for evaluation funding is the legislative set-aside authority which permits the Secretary of HHS to use a proportion of program funds for evaluation purposes. The largest of such set- aside authorities at HHS is one established for evaluations conducted by several agencies of the U.S. Public Health Service (AHRQ, CDC, HRSA, NIH, and SAMHSA), ASPE, and the Office of Public Health and Science (OPHS) in the Office of the Secretary. The mechanism is called the PHS evaluation set-aside

legislative authority, which is provided for in Section 241 of the Public Health Service (PHS) Act. This authority was established in 1970, when the Congress amended the Act to permit the HHS Secretary to use up to 1 percent of appropriated funds to evaluate authorized programs. Section 206 of the FY 2002 Labor, HHS, and Education Appropriations Act increased the amount the Secretary could use for evaluation to 1.25 percent. Section 241 limited the base from which 1.25 percent of appropriated funds could be reserved for evaluations of programs authorized by the PHS Act. Excluded were funds appropriated for FDA, IHS1 and other programs that were managed by PHS agencies but not authorized by the Act (e.g., HRSA’s Maternal and Child Health Block Grant and CDC’s National Institute for Occupational Safety and Health).

1FDA programs are principally authorized by legislation other than the PHS Act, specifically the authority of the Agriculture, Rural Development, Food and Drug Administration and Related Agencies Appropriations Act. IHS programs are authorized under the Indian Health Care Improvement Act and the Indian Self-Determination Act, and appropriated under the Department of the Interior and Related Agencies Appropriations.

Evaluation Management

The management of HHS evaluations, which are carried out on a regular basis by the HHS agencies and offices and coordinated by ASPE, involves four basic functions: (1) evaluation planning and coordination, (2) project management, (3) quality assurance, and (4) dissemination of evaluation reports:

Evaluation Planning and Coordination

The HHS agencies, ASPE, the Office of the Inspector General (OIG), and OPHS develop evaluation plans annually in concert with HHS’s program planning, legislative development, and budgeting cycles. Plan development is coordinated by ASPE. Each agency or office plan generally states the evaluation priorities, or projects under consideration for implementation. Typically, HHS evaluation priorities include: congressionally-mandated program evaluations, evaluations of Secretarial program or policy initiatives, assessments of new programs and ones that are candidates for reauthorization, and evaluations that support program performance management and accountability.

HHS evaluation planning activities are coordinated with three department-wide planning initiatives. First, HHS evaluation activities support the Department’s strategic planning and performance management activities in several ways. Completed evaluations are used in shaping the specific HHS strategic goals and objectives. Evaluation findings provide an important source of information or evidence about the success of various HHS programs or policies that collectively make up the strategies to achieve the goals and objectives. The HHS Strategic Plan highlights evaluations that document efficacy or effectiveness of strategic programs or policies and lists future evaluations that will benefit strategic planning. HHS agencies also use findings from their evaluations to support GPRA annual performance reporting to Congress and program budget justifications.

Second, Congress requests that HHS coordinate all of its research, demonstration, and evaluation (RD&E) programs to ensure that the results of these projects address HHS program goals and objectives. ASPE and the Assistant Secretary for Budget, Technology and Finance work together with HHS agencies to provide the Congress with a special annual research, demonstration, and evaluation budget plan that coincides with the preparation of the President’s fiscal year budget. The plan outlines HHS agency research, demonstration, and evaluation priorities as related to the Department’s strategic goals and objectives.

The newly-formed Research Coordination Council (chaired by the ASPE, and containing representatives of HHS agencies), seeks to foster greater interactions among its research programs. The Council’s work includes streamlining research and evaluating Department-wide research priorities to ensure greater efficiencies in research, demonstration, and evaluation. The Council seeks to strengthen HHS research coordination and planning around key Departmental priorities and themes.

Third, those agencies and offices that use the PHS evaluation set-aside authority–AHRQ, CDC, HRSA, NIH, ASPE, OPHS, SAMHSA–submit a formal plan to ASPE, which coordinates and develops the individual plans into the HHS report to the Congress on the use of the PHS authority. The statute requires that this report be submitted to the Congress before HHS implements the plan.

Project Management

HHS agencies, OIG, and ASPE all are responsible for executing annual evaluation plans, developing evaluation contracts, and disseminating and applying evaluation results. All agencies and their subunits are encouraged to coordinate with each other on research/evaluation project planning and release of final reports that are likely to relate to work of other HHS agencies. Within agencies— while there is some oversight responsibility and execution capability in the Office of the Director or Administrator—the various subunits (centers, institutes, and bureaus) conduct much of the day-to-day evaluation activity.

The OIG performs independent evaluations through its Office of Evaluations and Inspections (OEI). The OEI’s mission is to improve HHS programs by conducting inspections that provide timely, useful, and reliable information and advice to decision makers. This information (findings of deficiencies or vulnerabilities and recommendations for corrective action) is usually disseminated through inspection reports issued by the Inspector General. A summary of individual inspection reports and other OIG reports can be viewed on the Internet at OEI also provides technical assistance to HHS agencies in conducting their evaluations.

Quality Assurance and Improvement

Most evaluation projects are developed at the program or office level. The initial quality review is generally conducted by a committee of agency- or office-level policy and planning staff members. Before a project is approved, it is also reviewed for technical quality by a second committee with expertise in evaluation methodology. Technical review committees follow a set of criteria for quality evaluation practice established by each agency. Some HHS agencies also have external evaluation review committees composed of evaluation researchers and policy experts from universities and research centers.

Since HHS began reporting to Congress in 1995 on evaluations completed and in-progress through the Performance Improvement report series, the Department has focused more attention on improving the quality of evaluations performed every year. The HHS Evaluation Review Panel, convened to recommend evaluations for highlighting in Chapter I of this report, has contributed insights to HHS evaluation officers on the strengths and challenges of ensuring quality evaluations. HHS evaluation officers have had opportunities to discuss these strengths and challenges with the Panel and to identify steps to improve agency evaluation capacity.

Based on these discussions, ASPE initiated two evaluation capacity-building projects. The first was to establish an evaluation consultant Blanket Purchase Agreement (BPA) contract mechanism to assist agency program managers in obtaining short term assistance in developing evaluation designs or monitoring the performance quality of evaluation contractors. The second was to conduct a department- wide evaluation training needs assessment survey, which indicated training needs in various areas.

Dissemination of Evaluation Reports

Maintaining report libraries and distributing information on evaluation results is an important part of HHS evaluation management. Interested individuals can access the Department’s information and reports on major evaluations through the HHS Policy Information Center (PIC) website: The website offers users an opportunity to search (by key word or by selected program or policy topics) the departmental evaluation database and electronic report library maintained by ASPE. As an information database and library resource, the PIC contains over 8,000 completed and in-progress evaluation and policy research studies conducted by the Department, as well as key studies completed outside of HHS by the U.S. General Accounting Office (GAO) and private foundations.

Typically, the results of HHS evaluations are disseminated through targeted distribution of final reports, articles in refereed journals, and presentations at professional meetings and conferences. Although individual HHS agencies have primary responsibility for disseminating results, the ASPE will continue its Department-wide efforts to expand dissemination of evaluation results to the larger research and practice communities through centralized computer communications and publications like the PIC website and project information database.

Appendix B - Acknowledgements of HHS Officials

Performance Improvement 2003: Evaluation Activities of the U.S. Department of Health and Human Services describes the continuous efforts of the various HHS agencies to examine service and research programs for the efficiency of their operations and their effectiveness in achieving objectives. The following persons from HHS agencies contributed to preparing information on evaluation projects:

Administration for Children and Families Office of Planning, Research, and Evaluation

Mary Ann MacKenzie


Centers for Medicare and Medicaid Services Office of Strategic Planning

William Saunders

Sydney Galloway


Administration on Aging Office of Planning and Evaluation

Frank Burnsi


Health Resources and Services Administration Office of Planning and Evaluation

Willine Carr


Agency for Healthcare Research and Quality Office of the Administrator

Kathleen Kendrick

David Introcaso

Indian Health Service Office of Public Health

Debbie Melton



Agency for Toxic Substances and Disease Registry Office of Policy and External Affairs

Larry Young

National Institutes of Health Office of Evaluation

Diane Buckley


Centers for Disease Control and Prevention Office of Program Planning and Evaluation

Susan Cleveland


Office of Public Health and Science

Omar Passons

Valerie Welsh


Food and Drug Administration Office of Policy, Planning, and Legislation

Theresa Mullin

John Uzzell

Mary Bobolis


Substance Abuse and Mental Health Services Administration Office of Policy and Program Coordination

George Fitzelle




Preparation of this report was managed by the Office of the Assistant Secretary for Planning and Evaluation, Office of Planning and Policy Support. Project Directors: Andrew Rock and Vijaya ChannahSorah. Project Assistant: Anna Sierra. Richard Silva provided technical and other support. Annette Johnson provided clerical support.

Susan Belsinger of Library Associates, Inc. provided database management as well as support in the development and production of the report, and Binnet Shumburo of Sanad Support Technologies and Jana Liebermann of Library Associates, Inc., provided database updating and support.

Appendix C - HHS Evaluation Review Panel

The following individuals served on the evaluation review panel that made recommendations on the reports highlighted in Chapter I of the Performance Improvement 2003: Evaluation Activities of the U.S. Department of Health and Human Services:

Douglas Barnett, Ph.D.
Department of Psychology
Wayne State University
Detroit, MI

Kenneth McLeroy, Ph.D.
Texas A & M
Associate Dean for Academic Affairs
Bryan, TX

Heather Becker, Ph.D.
University of Texas-Austin
School of Nursing
Austin, TX

Marv Mandell, Ph.D.
Policy Sciences Graduate Program
University of Maryland, Baltimore County
Baltimore, MD

Leslie J. Cooksy, Ph.D.
Center for Community Development & Family Policy
College of Human Services, Education, and Public Policy
University of Delaware
Newark, DE

Melvin Mark, Ph.D.
Professor of Psychology
Penn State University
University Park, PA


Holly Korda, Ph.D.
Consulting Principal
Health Systems Research Associates
Chevy Chase, MD

Donna Mertens, Ph.D.
Gallaudet University
Washington, DC


John Kralewski, Ph.D.
University of Minnesota
School of Public Health
Minneapolis, MN

Joy Quill
C. J. Quill & Associates, Inc.
Annapolis, MD


Contractors for Review Panel Process:

Cheryl J. McDonnell, Ph.D.
James Bell Associates
Arlington, VA 22201

Norma I. Gavin, Ph.D.
Research Triangle Institute
Research Triangle Park, NC

Project Manager: Vijaya ChannahSorah, Ph.D.
Office of the Assistant Secretary for Planning and Evaluation, HHS


Appendix D - Program Evaluation Review Criteria

The following core criteria were developed to address four types of evaluation conducted by HHS agencies, offices, and programs:

  • Program Effectiveness (PrE) evaluations provide a way to determine the impact of HHS programs on achieving intended goals and objectives.
  • Performance Measurement (PfM) projects are intended to assist in the development of data systems for monitoring progress on departmental strategic or agency performance goals.
  • Environmental Assessments (EnA) provide a means for understanding the forces of change in the health and human services environment that will influence HHS programs and achievement of its goals and objectives.
  • Program Management (PrM) studies reflect the needs of program managers to obtain information or data helpful for designing or managing a program.

The criteria are intended to be generally applicable to each of the four types of evaluations. However, some special considerations are noted for specific elements when applied to particular study types. All criteria are intended as general guidelines; not all elements may apply to every evaluation.

    • The study addresses a significant issue of policy relevance.
    • Evaluation findings are likely to be useful; the study’s usefulness to the stakeholder audience(s) is described.
    • Conceptual Foundations
      • The study is based on either theory, or conceptual models; it builds on previous relevant research, is logically based on previous findings (as appropriate), cites relevant literature, or otherwise justifies the study focus and utility.2
      • The program and/or study assumptions are stated.
      • When the report is linked with a program, policy, or issue, the report describes this context.
      • The timing is appropriate because the program/policy/issue is ready for study; evaluation methods used are appropriate to the program/policy/issue stage.3
    • Study Questions
      • The research questions addressed are clearly stated, measurable, and clearly specified.4
      • The questions are feasible, significant, linked to the program or issue, appropriate for the resources and audience, and derive logically from the conceptual foundations.
    • Evaluation/Study Design
      • Design considerations should include all of the following deemed appropriate: feasibility, funding and time constraints, generalizability5, applicability for cultural diversity, assessment of program delivery, validity, feasibility for data collection, reliability of selected measurements, use of multiple measures of key concepts, appropriateness of the sample, and assessment of statistical power prior to data collection.
      • Variables and/or methods used are clearly specified and fit with the questions and concepts.
      • The design matches the study questions; the design permits measurement of program implementation, as appropriate.
      • Multiple methods are used as appropriate and should be well integrated to support triangulation of results.6
    • Data Collection
      • Data collection includes: a data collection plan; data collection that is faithful to the plan; attention to and cooperation with the relevant community; data integrity; project confidentiality; ethical handling of data; and consistency.7
      • Data are collected appropriate to the evaluation questions, use appropriate units of measurement, and appropriately handle missing data and attrition.8
      • The quality of the data (including the quality of extant data sets used in the study), training of interviewers/data collection staff, and justification of the sampling frame are addressed.
    • Data Analysis
      • The data analysis addresses: handling of attrition, matching of the analysis to the design, use of appropriate statistical controls and techniques, use of methodology and levels of measurement appropriate to the type of data, estimation of effect size, and confidence intervals.9
      • If multiple methods are used to collect data on the same phenomenon, the data are compared to assess the extent to which the findings are consistent (triangulation).
      • The analysis shows sensitivity to cultural categories, if applicable.
      • The analysis procedures are appropriate and correctly applied.10
    • Findings
      • The study questions are answered/addressed, or if not, explanation is provided.
      • The interpretation of results is linked to and assesses the study’s conceptual foundation.11
      • The report notes that the findings are either consistent with or deviate from the relevant literature and findings in the field.12
      • Program implementation is assessed, as appropriate.13
      • Generalizability inferences are addressed.
    • Conclusions
      • The summary does not go beyond what the data will support; conclusions are justified by the analyses; qualifiers are stated as needed.
      • Equivocal findings are addressed appropriately.
      • The results have practical significance.
    • Recommendations
      • The programmatic/policy recommendations follow from findings; are worth carrying out; are affordable, timely, feasible, useful and appropriate; and are relevant to  stakeholders.
      • Any recommendations for future studies, and/or program improvements are clearly presented and justified.14
    • The following are cross-cutting factors that are likely to be important at all stages of a report: presentation is well written, clear and understandable; use of state-of-the-art approaches where possible; methodology is clearly described and defended; innovative, efficient, logical relationships, and discussion of the report’s limitations. The report presents multiple perspectives, as appropriate; relevant stakeholders are consulted and involved. The report should also address ethical issues, possible perceptual bias, cultural diversity, and gaps in study execution.

2 This element may be less relevant to Program Management and Performance Measurement studies.

3 For Program Effectiveness studies, timing should be appropriate to implementation stage; for Performance Measurement projects, timing should be appropriate to establishment of strategic goals and objectives; for Environmental Assessments, timing should be appropriate due to changing or uncertain environments; for Program Management studies, timing should be appropriate to management priorities.

4 Program Effectiveness studies should specify research questions and testable hypotheses.

5 For Performance Measurement and Program Management studies, generalizability is not a key consideration, given study design and foci for these study types.

6 This element may be less relevant to Performance Measurement studies.

7 Program Effectiveness and Program Management studies that examine person-level outcomes should also address use of an appropriate comparison or control group; and adequate sample size, response rate, and information about the sample.

8 Program Effectiveness studies and Environmental Assessments that address person-level outcomes should control for participant selection and assignment bias.

9 For Program Management studies that may use other secondary data and methods, data analysis should be appropriate to the kinds of data collected.  Estimation of effect size may not apply to Environmental Assessments, where results may be descriptive.

10 For Program Effectiveness studies, appropriate sensitivity analyses are conducted, and  uncertainty in key parameter estimates is addressed, as appropriate.

11 This element may not apply to Program Management studies, given their focus and design.

12 This element may not apply to Program Management studies, which may not have a conceptual foundation in the research literature.

13 This element may not apply to Environmental Assessments, which do not directly assess program implementation issues.

14 Performance Measurement studies should consider the context of the strategic goals and objectives of interest to the study.