Performance Improvement 2003

Foreword

Performance Improvement 2003: Evaluation Activities of the U.S. Department of Health and Human Services is the ninth annual report to Congress summarizing prior fiscal year evaluation efforts. The purpose of this report is to provide Congress with outcome-oriented evaluation findings for the Department’s programs, policies, and strategies. The report contains brief summaries of evaluation results. Additional information can be obtained from the Agency Sponsor, Federal Contact, Performer, the cited web site, or the Policy Information Center (PIC) http://aspe.hhs.gov/pic.

Evaluations summarized in this report contribute to performance improvement in one or more of the following ways:

•        Program effectiveness

•        Performance measurement

•        Environmental assessment

•        Program management

Performance Improvement 2003 includes HHS evaluation projects completed during fiscal year (FY) 2002. The report is organized into two chapters. Chapter I provides evaluations that were selected by an outside review panel as outstanding for their potential use by the larger health and human services community. Chapter II presents summaries of all other evaluations completed by the HHS agencies and the Office of the Secretary during FY 2002.

The appendices contain a description of the HHS evaluation program (Appendix A), acknowledgments of the HHS officials who contributed to the report (Appendix B), and the external evaluation experts who recommended evaluations for highlighting (Appendix C) and the criteria the panel used to review the 26 reports nominated for outstanding evaluations for FY 2002 (Appendix D).

This report is required by section 241(b) of the Public Health Service (PHS) Act, as amended by the Preventive Health Amendments of 1993. HHS is directed to submit an annual report summarizing the findings of the evaluations under section 241(a) to the Committee on Health, Education, Labor and Pensions of the Senate and the Committee on Energy and Commerce of the House of Representatives.

We hope that you will find this report useful and informative.

Chapter 1 - Results in Brief: Outcome-Oriented Program Evaluations

This chapter consists of summaries of six evaluation projects selected by an outside review panel as outstanding based on rigor of evaluation methodology, importance of findings, and clarity of presentation.

• The Panel reviewed twenty-six reports nominated by the HHS agencies and selected the reports to be highlighted in this chapter on the basis of the following criteria:Is the report useful? Does it address a significant policy issue or problem? Are the findings likely to be useful for a broader audience?
• Is the report conceptually sound? Is it clearly based upon and integrated with previous research? Are assumptions clearly stated? Is the research appropriately linked to the program? Are relevant stakeholders identified and included?
• Is the report methodologically sound? Are its concepts, designs, data collection, and analyses conducted and reported in a competent manner?
• Are the recommendations practical? Do the conclusions and recommendations logically follow from the data and analyses, and are they relevant to the questions asked?

A full statement of the criteria is found in Appendix D.

Performance measurement or assessment, program management and development, and policy analysis and development represent the three most common uses of HHS evaluation resources. Performance measurement or assessment is a high priority for HHS agencies. The development, implementation, and refinement of programs are more results oriented than in the 1990’s and specific measurements are required under the Government Performance and Results Act (GPRA) of 1993. Program management and development reflects the kind of evaluation projects that program managers initiate to obtain information or data that will help them manage a program more efficiently and ensure successful results. Policy analysis and development includes the evaluation projects conducted by HHS agencies to examine the impact of alternative policies, on the future direction of HHS programs or services.

It is important to note that the context in which an evaluation was conducted may have changed since the evaluation was initiated. It should also be recognized that many evaluations are bounded by policy and resources from the funding agency. In addition, the panel considered whether the timing of the evaluation is appropriate for inclusion – evaluation reports that reflect a mid-program assessment of an ongoing evaluation may be included if the policy it addresses is currently under discussion, if the program is large and recommendations potentially include a need for change in program direction, or for other reasons that indicate a midcourse assessment is appropriate.

Each summary includes a brief abstract; a description of the study, including its purpose, background, methods, findings, and use of results; and the name and phone number of the HHS official to contact for additional information.

Following are the six highlighted reports:

Identifying a Uniform Data Set of Information/Data to Assess the Impacts of Activities Funded by the Office of Minority Health

Highlights

This report examined obstacles to designing a uniform data set (UDS) to assess the impact of activities funded by the Office of Minority Health (OMH), which administers a broad array of grants and contracts aimed at eliminating disparities in health outcomes and access to services for minority groups. Taking into account the obstacles, as well as guidelines, legislation, performance measurement methodology, and databases, the evaluators described a recommended data system. The literature review attended to a variety of important contextual issues (e.g., recommendations regarding the evaluation of programs  like those administered by OMH). Site visits and focus groups with relevant players were carried out, and the UDS was then pilot-tested over a 3-month period with nine OMH grantees. In addition, a Technology Survey was conducted by telephone with 22 participants to assess the capacity and preference for gathering data for the UDS. Based on analyses summarized in the report, the evaluators recommended a web-based database and report generating system that would be user-friendly. In addition, it would allow the various projects around the country to enter their data into a national system, obtain needed technical support, and generate reports of data that were useful locally.

Purpose

Due to the increased emphasis on data-based evaluation performance indicators, which are often difficult for community based organization (CBOs) to implement, a standard data set of performance measures was developed and proposed. The feasibility of implementation of the data set was evaluated through a pilot study.

Background

The Office of Minority Health (OMH) funds a variety of grant and standard cooperative agreement programs, under its mandate to eliminate disparities in health for minorities. The program activities generally involve a mixture of health promotion, screening, access to services; treatment, training and education; materials development; case management; capacity-building; and development of community linkages. These programs are required to provide program evaluation data related to performance standards. At the same time, unique program contributions are often overlooked by standard reporting mechanism. The goals of the OMH UDS were to capture the unique program factors while incorporating the required uniform measures.

Methods

A Project Advisory Group (PAG) composed of representatives from Federal agencies and private organizations involved in similar efforts was organized. Current and past agency efforts to develop uniform data sets and performance measures, and major data set configurations and approaches that may be applicable to OMH were identified and reviewed. Focus groups, interviews, and site visits with current and past OMH grantees were conducted to identify specific needs with respect to a data set tailored to the unique characteristics of OMH grantees. A draft data set was developed, and a pilot test was conducted with nine grantees to assess efficacy and applicability of the data set for all OMH grantees.

Findings

The final UDS combined multiple data types, including numbers (e.g., counts of persons served or how many times a service was provided); specific results (e.g., of language interpretation) or pre- and post- tests for training sessions; and qualitative impacts using brief case studies. The results of the pilot test of the system indicated the system was easy to use and that data could be readily obtained. However, additional data collection was required at several sites in order to fully complete the forms. The major recommendation was to put the UDS on line.

Use of Evaluation Results

Implementation of a reporting system such as the UDS would enable collection of standardized data across multiple types of organizations. An Internet-based system, potentially supported by a data coordinating center, would ensure the greatest access to the system.

AGENCY SPONSOR: Office of Public Health and Science, Office of Minority Health FEDERAL CONTACT: Violet Ryo-Hwa Woo, 301-443-9923

PIC ID: 7097

PERFORMER: Systems Development Group

The Impact of the State Children’s Health Insurance Program (SCHIP) on Community Health Centers

Highlights

The State Children’s Health Insurance Program (SCHIP) was enacted in 1997 to expand health insurance coverage for children. This evaluation focused on the impact of the enhanced insurance status of children served by two HRSA sponsored programs: the community health centers (CHCs) and Title V maternal and child health programs, and the impact on HRSA grantee organizations. The study included site visits; interviews with former CHC users; and an analysis of CHCs transactional encounter data.

Three groups of children were evaluated 1) children who continue using HRSA sites after enrolling in SCHIP; 2) children who are new to the HRSA sites; and 3) children who discontinued use of the HRSA sites.

Use of Evaluation Results

CHC strategies that increased SCHIP enrollment included the involvement of all site staff, especially clinicians, in encouraging SCHIP enrollment; establishing a protocol for uninsured patients; cultivating a relationship with the local SCHIP administrative staff; and follow-up on submitted applications. The recommendations for CHCs included dedicating staff to outreach and enrollment activities, instituting formal processes for referring uninsured patients to outreach staff; and conducting follow-up on SCHIP/Medicaid applications. State-based recommendations included investment in CHC-based outreach and education activities; implementation of presumptive eligibility for enrollment into SCHIP; passive re- determination after 12 months in the SCHIP program; and requiring SCHIP health plans to contract with safety-net providers.

AGENCY SPONSOR: Health Resources Services Administration FEDERAL CONTACT: Jerilynn Regan, 301-594-4283

PIC ID: 7125

PERFORMER: Center for Health Service Research and Policy, George Washington University

Purpose

The study addressed two primary goals: 1) document health insurance volatility experienced by users of the health centers covered by SCHIP vs. Medicaid; and 2) determine whether and how SCHIP has affected safety net providers such as community health centers (CHCs) and Title V maternal and child health (MCH) programs. The analytic framework examined the effect of the SCHIP in areas such as parents’ ability/willingness to seek health services for their children at a CHC site or from other providers; children’s health insurance volatility; and SCHIP’s effect on CHC sites.

Background

Over 12 million children lack health insurance coverage, and therefore may lack access to health care. This report evaluated the State Children’s Health Insurance Program (SCHIP), which was enacted in 1997 to expand the scope of state-provided services as an adjunct for children not eligible for Medicaid. The study focused on the experience of CHCs and examined three groups of children: those who continued using the Health Resources Services Administration (HRSA) program sites after enrolling in SCHIP; those who were new to the HRSA sites; and those who were previous users but were no longer visiting the HRSA site.

Methods

The analysis included quantitative and qualitative components. Site visits to CHC programs were conducted, former CHC users were interviewed, and CHC encounter data were analyzed. In addition, state and county officials, representatives of safety net providers, community-based organizations, and other stakeholders were interviewed to provide contextual information. Fourteen health centers and sixteen individual health care delivery sites in six states were evaluated. Patient encounter data was extracted from an insurance transaction database.

Findings

Early Head Start Research: Making a Difference in the Lives of Infants and Toddlers and Their Families

Highlights

Early Head Start provides services to low-income pregnant women and families with infants and toddlers with the goals of improving children’s development (cognitive, language, social-emotional, and health); fostering close supportive relationships between parents and their infants and toddlers; and encouraging the development of community partnerships. This report summarizes the interim results of an evaluation initiated in 1995 that includes about 3,000 children and families across 17 sites selected to represent a variety of environments (regional, rural, and urban), ethnic and racial compositions, and types of program approaches. At each site, children and families were randomly assigned to program and control groups; follow-up was carried out over three years of program participation, through a child’s third birthday. There was a consistent pattern of statistically significant (although modest) favorable impacts across a range of outcomes related to children and their families. The results found at the interim evaluation (conducted at age 2) were sustained through the child’s third birthday.

Purpose

This evaluation carried out the recommendation of the Advisory Committee on Services for Families with Infants and Toddlers to include an evaluation component within the Early Head Start program that would facilitate continuous improvement. The 17 programs selected for evaluation were geographically diverse, and represented all of the major programmatic approaches under the Early Head Start program.

Background

Early Head Start was first funded in 1995, and the performance standards of Head Start were applied to this extension of these two-generation programs. The programs provide services to low-income pregnant women and families with infant and toddlers through a wide variety of program options. Individual sites have the latitude to create their own programs; however, these programs must meet the requirements to provide child development services directly and support child development through parenting and or family development services.

Methods

 

Findings

The level of implementation (i.e., early, late, or incomplete) was associated with stronger results. Although most Early Head Start families received some services, fewer than half were involved intensively for the full period of eligibility. Nevertheless, program families and children generally received substantially more services than those in the comparison group. With respect to outcomes, positive impacts of Early Head Start were observed on children’s cognitive development, language development, and several aspects of socio-emotional development.  In addition, the assessment of Early Head Start parents demonstrated some favorable impacts on parenting outcomes (e.g., reading daily to the child) and on some aspects of progress towards self-sufficiency (education, job training activities). Early Head Start mothers were less likely to have subsequent births within two years.

Use of Results

Results of the evaluation suggest that implementing key services fully is important for program success. Specific recommendations for center-based and home-based services were provided, and it was noted that flexibility in service options seems to have the greatest positive impact. Two ‘difficult-to-serve’ subgroups (parents at risk for depression and teenage parents) were reached by the program. The study validated the importance of meeting Head Start Program Performance Standards and supported the value of monitoring programs regularly.

http://www.mathematica-mpr.com/PDFs/ehsfinalvol1.pdf

AGENCY SPONSOR: Administration for Children and Families, Administration on Children, Youth and Families

FEDERAL CONTACT: Rachel Chazan Cohen, 202-205-8810 PIC ID: 3570

PERFORMER: Mathematica Policy Research, Inc., Princeton, NJ

The Impact of Welfare Reform on Children: The Indiana Welfare Reform Evaluation

Highlights

This evaluation was part of a five state evaluation of welfare reform’s impact on children. An in-home survey was conducted over a nine-month period during 2000. The survey was administered to 1,679 single parent families with children between the ages of 5 and 12. An additional set of outcomes for 1,126 adolescent children (between 13 and 17 years old) was also presented. The survey addressed child outcomes related to educational performance, social and emotional adjustment, and health. Other aspects in the children’s environment that were potentially affected by welfare reform measures, such as childcare and family stability, were also measured.  Groups were randomly assigned to either the traditional welfare environment or the welfare reform program.  The welfare reform groups were encouraged to find work as soon as possible, with a 24-month time limit imposed on receipt of Temporary Assistance for Needy Families. Personal responsibility provisions were also incorporated.

The welfare reform program did not appear to have a significant impact, either positive or negative, on elementary school-age children or on adolescents.

Purpose

This evaluation was part of a five state study (the Project on State-Level Child Outcomes) that examined the effects of welfare reform on children; this evaluation focused on reforms in Indiana. Differences in outcome measures for two groups— one randomly assigned to participate in Indiana’s welfare reform program and the other assigned to a traditional welfare environment — were examined.

Background

 

Methods

An in-home survey that lasted two hours was conducted with a representative statewide sample of single-parent families with children who had entered the welfare reform evaluation in its first year. A seventy percent response rate was obtained, and statistical adjustments were made to control for non- response bias. Primary results were based on a sample of families with a focal child between 5 and 12 years of age; results were also presented for a sample of adolescent children between 13 and 17 years of age. The primary analysis method was regression, adjusting for (modest) baseline differences and for a set of baseline characteristics.

Findings

Findings indicated limited effects for elementary school-age children; for instance, there was no  significant program impact on school performance, level of engagement in school, or receipt of special education. There were no effects on the measures of social behavior and emotional well-being, nor were there differences between the welfare reform group and the traditional welfare group in terms of children’s health status, frequency of injuries or accidents, or medical and dental care. Fewer outcome variables were measured for adolescents. Although negative effects of welfare reform were reported for parental ratings of adolescents’ school performance, no differences were observed in other areas, such as school dropout rates and teen births. Moreover, welfare reform was not associated with differences on measures of the home environment (e.g., cognitive stimulation, emotional support); children’s involvement with nonresident fathers; or most measures of household stability. Additional findings showed that children in the welfare reform condition spent somewhat more time in child care, and that welfare reform was associated with lowered maternal depression and a small but significant reduction on a parenting aggravation scale. The families continued to face significant challenges; most remained financially insecure with multiple family problems in both groups.

Use of Evaluation Results

The findings were consistent with other evaluations of the impact of welfare reform on children. There were generally few effects on elementary school children, and some potentially negative effects on adolescent children. The authors suggested that increasing household income, through earnings disregard or other approaches, may help with other family problems associated with single parent families on welfare.

http://www.abtassoc.com/reports/Indy_Child_Final_Report.pdf

AGENCY SPONSOR: Administration for Children and Families; Administration on Children, Youth and Families

FEDERAL CONTACT: Alan Yaffe, 202-401-4537 PIC ID: 6847

PERFORMER: ABT Associates Inc.

Final Report on Vermont's Welfare Restructuring Project

Highlights

The WRP program appeared to increase employment, save taxpayer money, had little effect on income, and did not require creation of community service positions.

Purpose

States have new flexibility in creating their own rule for governance of welfare programs due to the conversion of federal welfare funding into a fixed block grant. States developed a variety of approaches to welfare reform. Vermont’s approach had more lenient standards than other programs; the intent of this evaluation was to determine if less stringent standards still resulted in an increase in employment and reduced welfare receipt.

Background

Since it preceded Temporary Assistance for Needy Families by two years, the Vermont provisions for welfare reform differed from those of most other states. In Vermont, all welfare applicants from July 1994 through December 1996 were randomly assigned to one of three treatment groups. Members of the Welfare Restructuring Project group were subject to time-triggered work requirements (required to work in wage-paying jobs after receiving cash assistance for 30 months) and financial work incentives (enhanced earnings disregard, and being able to hold more savings and personal effects). This was the final report on Vermont’s welfare reform program, with previous reports having been provided at 21 and 42 months. The evaluation covered an eight-year time period, from 1994 through 2001.

Methods

The final outcome evaluation was based on a variety of data sources, including baseline data from a background information from and private opinion survey, field research, staff surveys, administrative records, and program expenditure data. While all welfare recipients from July 1994 to December 1996 were randomly assigned to one of three conditions, this impact study was based on a subset of the cases assigned in the research districts from July 1994-June 1995. This “report sample” represented about 70 percent of the full research sample (N=7,691), of which approximately one third were surveyed for a 42- month (post assignment) assessment, and approximately 80 percent (N=1,872) responded.

Findings

Selected findings indicated that the full WRP program increased employment and reduced reliance on  cash assistance for single-parent families, but had little effect in areas such as material hardship, children’s school performance, or other family and child outcomes. Results also showed that increases in employment and income were largest for the most disadvantaged members, and that the net cost of the program was low and more than offset by reduced public assistance payments. There were few program effects for two-parent households.

Use of Evaluation Results

This evaluation helps inform the debate over the best approach to welfare reform.  As one of the earliest statewide welfare reform programs initiated under waivers, it provides a longer time period of evaluation. However, the contribution of the healthy economy to the program’s success is not yet clear. http://www.acf.hhs.gov/programs/opre/vermont/vt_title.html

AGENCY SPONSOR: Administration for Children and Families FEDERAL CONTACT: Girley Wright, 202-401-5070

PIC ID: 6764

PERFORMER: Manpower Demonstration Research Corporation

How Effective are Different Welfare-to-Work Approaches? Five-Year Adult and Child Impacts for Eleven Programs

Highlights

Eleven mandatory welfare-to-work programs were evaluated for their long-term effects on welfare recipients and their children. This multi-year study of alternative approaches to helping welfare recipients find employment concentrated on answering the question of ‘What works best, and for whom’.  Welfare recipients were randomly assigned to one of two or three research groups, depending on the site. Three sites conducted two different programs simultaneously.  One site used a three-group random assignment design (to two case management models or to a control group). The remaining three sites used assignment to existing programs versus no program enrollment as the comparison. All welfare-to-work programs increased participation in employment-related activities, but had little impact on total combined income (as earnings were substituted for public assistance). Employment–focused programs generally produced larger employment gains than educational programs.

Purpose

The intent of this multi-program evaluation was to determine what alternative approaches to welfare reform help welfare recipients find and hold jobs and move off of public assistance. Two broad types of pre-employment strategies were compared—programs that emphasized short-term job search assistance (‘employment-focused’ programs) and programs that emphasized basic education and skill building (‘education-focused’ programs).

Background

Welfare reform has been legislated for the past 30 years, but the knowledge of what type of reform works best in moving people off welfare to work is still unknown. Although research in the 1980s demonstrated that provision of job search programs sped up the entry of welfare recipients into the labor market, the jobs were often short term and/or low-paying. The severely disadvantaged were not impacted. Some states began offering alternative programs that emphasized basic education and skill development.  These programs are more costly, but may be more cost-effective. The implementation of the TANF program made the question of which welfare-to-work strategy was most effective more urgent.

Methods

From 1991-1999, approximately 40,000 single parents (primarily mothers) and their children were followed for a five-year period after random assignment to either a welfare reform program or to traditional welfare groups. Eleven different programs in seven different sites across the U.S. were included in the evaluation. Three of the sites simultaneously operated two different programs – one program focused on immediate employment (a Labor Force Attachment (LFA) program) and one based on education and skills building (a Human Capitol Development (HCD) program) – expressly for evaluation purposes. Different samples were included in different analyses, but all samples exceeded 1,000 individuals.

Findings

 

Use of Evaluation Results

The most cost-effective, program utilized a mixed approach, blending employment search and education or training. The challenge of the future is to identify other types of programs or initiatives that could provide welfare recipients with better and more stable jobs, increase their income, and improve the well- being of their children.

http://aspe.hhs.gov/hsp/NEWWS/5yr-11prog01/

AGENCY SPONSOR: Office of the Assistant Secretary for Planning and Evaluation; Administration for Children and Families; US Department of Education

FEDERAL CONTACT: Audrey Mirsky-Ashby, 202-401-6640 PIC ID: 7089.7

PERFORMER: Manpower Demonstration Research Corporation

Chapter 2 - Compendium of HHS Agency FY 2002 Evaluations Completed

Administration for Children and Families

Mission

To promote the economic and social well-being of families, children, individuals, and communities.

Evaluation Program

The Administration for Children and Families (ACF) administers a broad range of formula and discretionary programs, including family self-sufficiency (Temporary Assistance for Needy Families); child support; children and family services (Head Start, Child Welfare, Family Preservation and Support, and youth programs); and special programs for targeted populations, such as the developmentally disabled, refugees, and Native Americans.

The objectives of ACF’s evaluations are to furnish information on designing and operating effective programs; to test new service delivery approaches capitalizing on the success of completed demonstrations; to apply evaluation data to policy development, legislative planning, budget decisions, program management, and strategic planning and performance measures development; and to disseminate findings of completed studies and promote application of results by state and local governments.

ACF actively engages with other federal agencies, state and local policy and program officials, national organizations, foundations, professional groups and practitioners, and consumers to stay current on emerging issues affecting its programs and to identify questions for evaluation studies. Systems changes and how they affect vulnerable populations, particularly children, are of primary concern. The movement toward devolving responsibility for health and human services to state and local organizations offers both tremendous opportunities and unprecedented challenges in redefining and implementing services for families.

Studies often are funded as joint ventures with the Office of the Assistant Secretary for Planning and Evaluation and other federal agencies and foundations. Such collaborations permit large-scale efforts that are better informed and more representative of varying perspectives. Proposals are reviewed by multidisciplinary experts. Evaluation study designs are carefully developed in collaboration with project partners and technical experts in order to address specific research questions. Work groups of various kinds are used to monitor the progress of projects and to advise on design refinements and the presentation of findings.

Summary of Findings

Administration on Developmental Disabilities

project consisted of: a review and analysis of research and pertinent federal and state policies and programs; the development of 22 case studies of past and present approaches and “best practices”; design, field testing (in three sites), and finalizing the desired model; and utilization of an expert advisory panel to review all products. The EI² Model addresses the following questions: What attempts have previously been made or are currently underway to design and implement a person-centered directed comprehensive service model that results in valued employment outcomes for the target population?

What are the common elements and unique features of these programs and the barriers they encounter? What would be a realistic model for the target population, and how can it best be implemented?

Recommendations were for: (1) each state’s Olmstead Planning Group to identify options within their state (HCB) Medicaid waiver and to create a state level working group to review state oversight plans for commitment of resources to expand opportunities for production of income; (2) commitment at state and local levels to cross agency training and capacity building; (3) additional training, research, and technical assistance nationwide to build understanding and knowledge transfer of promising practices; (4) commitments from state and local government to part- and full-time employment of the target population with job sharing and additional natural and paid supports in the workplace; and (5) additional research to monitor the impact of benefit counseling services and results achieved with implementation.

FEDERAL CONTACT: Hossein Farris, 202-205-4922 PIC ID: 7544

PERFORMER: Diana McIver & Associates, Inc., Austin, TX

Office of Planning, Research, and Evaluation

Agency for Healthcare Research and Quality

Mission

To improve the quality, safety, efficiency and effectiveness of health care for all Americans.

Evaluation Program

Evaluation activities within the Agency for Healthcare Research and Quality (AHRQ) provide executive management, program officers and audiences external to the Agency with evaluative findings concerning the Agency's effectiveness and efficiency in meeting its GPRA, PART and other performance goals. The work is conducted moreover by external, independent evaluators and complies with (OMB) Paperwork Reduction Act requirements. Evaluation is built into (beginning at the design phase) virtually all major AHRQ programmatic or portfolio activities. Among evaluation mechanisms used by the Agency are targeted evaluation studies undertaken through contracts that use a variety of quantitative and qualitative methods and that are increasingly tending toward providing more real-time monitoring feedback. Evaluation activities also include satisfaction feedback from AHRQ "customers" regarding the usefulness of AHRQ research findings and dissemination products.

Summary of Findings

Agency for Healthcare Research and Quality

FEDERAL CONTACT: Larry Patton, 301-427-1486 PIC ID: 7010.1

PERFORMER: Lewin Group, San Francisco, CA

 

Implementation Planning Study for the (1) Integration of Medical Event Reporting Input and, (2) Data Structure for Reporting to Implementation [AHRQ, CDC, CMS, and FDA]

This study analyzed existing US, DHHS data systems, a select number of other systems, and factors affecting the potential integration of systems. Although efforts to measure and reduce the risk of harm to patients are underway, there was no national, centralized system for the collection and analysis of information pertaining to adverse events. Rather, an array of disparate systems collected information about different types of events (e.g., device-related, medication-related), from different sources (e.g., consumers, practitioners, manufacturers). Individually, each of these systems provided value to those who submit information and those who analyzed the collected data. However, linking data or combining databases promises efficiencies for the collection of data and could allow for early identification of systemic safety problems that would otherwise be impossible to detect. Our investigation suggested that there are several possible integration options, and this report details two specific candidates. Both approaches lent themselves to a phased implementation with the chief integration efforts occurring early in Phase 1 and expanding to other domains or user communities. One approach initially addressed the issue of “burden of reporting” for several systems, and it subsequently developed rich analytic facilities. Another approach began by integrating two systems “end-to-end,” including data entry and analysis components.

http://www.ahcpr.gov/downloads/pub/rfp020015/MERIP2.pdf FEDERAL CONTACT: James Battles, 301-427-1332

PIC ID: 7681

PERFORMER: MEDSTAT Group, Cambridge, MA

Center for Cost and Financing Studies

Health Services Research Education; Assessing Customer Satisfaction and Program Needs: Survey Update

This report analyzes the responses to a mailed, self-administered survey to assess customer satisfaction among trainees who participated in institutionally based and other health services research training programs funded by the Agency for Healthcare Research and Quality. The survey was designed to generate information about training program activities and needs. The survey was conducted in two waves. The first was conducted during the summer of 1999 and included 251 former trainees from the AHRQ-funded institutionally based (T-32) program. The second wave included an additional 113 former trainees. Of the 364 former trainees sent a survey, there was an 87 percent response rate (317).

Respondents represented a variety of training program types: 35 percent had been in pre-doctoral programs; 56 percent had been in postdoctoral programs, and 3 percent had participated in both types. The vast majority of respondents are currently employed (88%), with many in positions that were research-oriented or included opportunities for research (67%). Global observations about trainees satisfaction included: (1) Respondents were very satisfied with the quality of their research training, and (2) Large numbers of respondents were satisfied with many elements of the infrastructure of their programs, including the structure of the training experience, knowledge and skills learned, mentoring, stipend duration, and the academic program.

FEDERAL CONTACT: Debbie Rothstein, 301-427-1525 PIC ID: 7397

PERFORMER: Battelle Corporation, Arlington, VA

Center for Organization and Delivery Studies

FEDERAL CONTACT: Bernard Friedman, 301-427-1404 PIC ID: 7495

PERFORMER: Lewin Group, San Francisco, CA

Center for Outcomes and Effectiveness Research

Innovative Approaches to Training Clinicians for Bioterrorist Attacks: Final Report

The goal of this project—which included an advisory committee of clinicians, medical educators, health systems leaders, and others—was to develop two prototype approaches for training clinicians to recognize and respond appropriately to a possible bioterrorist attack. Early in the project, a two-day workshop was held with the Technical Advisory Committee to discuss the bioterrorism threat and develop approaches for training clinicians for bioterrorism events. At that time, the Committee considered uses of biologic agents as a bioterrorist attack an unlikely, not even a rare, event. The events of the Fall of 2001 markedly changed the nation’s views, and the health community’s opinions about bioterrorism preparedness. New information about bioterrorism disease recognition, treatment, and precautionary measures has evolved with each new patient and event. Two prototype approaches were developed for training clinicians to recognize and respond appropriately to a possible bioterrorism attack. The first approach was a website that has four main segments: the home page, bioterrorism agents, emerging infectious agents, and notifications and key web links. The second approach involved creating simulated bioterrorism patients using virtual reality technologies previously developed by the research contractor.

The Web-based learning materials and the simulation-based learning and evaluation materials were assessed by practicing clinicians (in primary care or infectious disease). The clinicians endorsed the broad approach taken by the website providing historical and clinical information for emerging infections as well as bioterrorism diseases. In our usability evaluation, users testing the software ranked it moderately high to very high.

FEDERAL CONTACT: Carolyn M. Clancy, 301-427-1200 PIC ID: 7680

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Center for Practice and Technology Assessment

FEDERAL CONTACT: Kevin Murray, 301-427-1853 PIC ID: 7688

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

 

Systems to Rate Strength of Scientific Evidence

As a key part in its strategy for meeting its legislative mandate, AHRQ undertook a systematic review and analysis of methods of rating the quality of scientific studies. The Healthcare Research and Quality Act requires that AHRQ, in collaboration with experts from the public and private sector, identify methods or systems to assess health care research results, particularly “methods or systems to rate the strength of the scientific evidence underlying health care practice, recommendations in the research literature, and technology assessments.” The University of North Carolina’s Evidence-based Practice Center identified 19 generic systems that fully addressed their key quality domains and identified seven systems that fully addressed their three domains for grading the strength of a body of evidence. The report also provides a research agenda including questions relevant for assessing the quality of scientific evidence.

FEDERAL CONTACT: David Introcaso 301-427-1213 PIC ID: 7676

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Center for Primary Care Research

FEDERAL CONTACT: Robin Weinick, 301-427-1573 PIC ID: 7687

PERFORMER: Center for Health Policy Studies, Columbia, MD

Office of Healthcare Information

Tracking Research Impact: Final Report

This report provided a conceptual framework for understanding the research process and used the framework to synthesize information collected from structured discussions with staff from the Agency for Healthcare Research and Quality, the Veteran’s Health Administration and four nonprofit health services research organizations. The report describes the efforts of each organization to collect, translate and disseminate research findings and assesses their impact on the knowledge pool, policy and practice. Key findings and recommendations are that AHRQ should: (1) consider improving patient outcomes as its principal indicator of research impact and adopt “priority conditions” as a fundamental organizing   principle for its ongoing research process, (2) develop a more structured process for obtaining feedback from end users of research and consider increasing funding for projects designed specifically to help the agency synthesize larger bodies of research results, and (3) foster a closer and more deliberate interaction between the Office of Health Care Information (OHCI) and the project officers and Center   staff to improve dissemination efforts. Most organizations studied do not systematically use the results of current and past research efforts to explicitly determine priorities and allocate funding for the next round of research. Rather, they target their research at achieving a different type of impact--influencing health care policy or improving clinical and delivery system performance.

FEDERAL CONTACT: Sandra Isaacson, 301-427-1253 PIC ID: 7494

PERFORMER: Lewin Group, San Francisco, CA

Centers for Disease Control and Prevention

Mission

To promote health and quality of life by preventing and controlling disease, injury, and disability.

Evaluation Program

The Centers for Disease Control and Prevention (CDC) conducts evaluations designed to provide information essential for CDC’s strategies to achieve its mission:

• Conduct public health research, including epidemiology, laboratory, behavior, and social sciences.
• Develop and implement ongoing evaluation of scientific research.
• Assure that scientific information is communicated to the general public.
• Develop and implement public health information systems for monitoring and promoting he health of the Nation.
• Deploy multidisciplinary teams to detect and investigate health threats.
• Develop and implement a system for establishing CDC scientific and programmatic priorities.
• Routinely evaluate the effectiveness and cost of CDC programs.
• Demonstrate the value of investment in health prevention.
• Collaborate with diverse partners.
• Design, implement, and evaluate prevention programs based on community needs.
• Prepare the public health leaders (present and future) through training in management and public health science.

CDC places high priority on evaluations to answer policy, program, and strategic planning questions related to the goals and objectives of Healthy People 2010. Performance improvement studies, such as those focusing upon the development of indicators consistent with the Government Performance and Results Act (GPRA), are of interest and import to the Agency. With the support of 1-percent evaluation funds, the GPRA planning process began at CDC in FY 1995 and is continuing.

Summary of Findings

Centers for Disease Control

FEDERAL CONTACT: Nancy Cheal, 404-639-7222 PIC ID: 7853

PERFORMER: Battelle, Seattle, WA

National Center for Chronic Disease Prevention and Health Promotion

FEDERAL CONTACT: Sheree Williams, 770-488-8007 PIC ID: 7703

PERFORMER: Macro International, Inc., Calverton, MD

National Center for Environmental Health

Evaluating a Free Vitamin Supplement Distribution Program in State-Funded Family Planning Clinics

The purpose of this evaluation was to assess the impact of the Georgia Folic Acid Initiative (GFAI) on family planning clients’ knowledge about the benefits of folic acid, their attitude toward supplementation, and their use of folic acid supplements. Six Title X County Family Planning Clinics participated in the evaluation. Three clinics provided educational materials and super-fortified cereal and one clinic provided educational materials only. A survey of women aged 18 to 45 who use the clinics was conducted.

Additionally focus groups were conducted to gather data regarding: (1) participants’ experiences with   the GFAI, (2) barriers and facilitators for implementing the GFAI, (3) factors that encouraged or discouraged clients’ use of free folic acid supplements, and (4) recommendations for improving the intervention. Four focus groups were conducted with clients who participated in the evaluation and two focus groups were conducted with staff from nine Georgia family planning sites participating in the GFAI. Focus group participants largely viewed the GFAI as a successful and beneficial intervention. Client and staff participants appreciated that folic acid supplements were provided free. As a daily supplement, folic acid pills were preferred to cereal. Most of the clients in the focus groups reported taking at least some of the pills given to them. However, very few took folic acid pills on a daily basis.

FEDERAL CONTACT: Katherine Lyon-Daniel, 404-498-3965 PIC ID: 7042/7748

PERFORMER: Battelle Corporation, Arlington, VA

National Center for HIV, STD, and TB Prevention

 

Completed surveys were received from 4,226 physicians. The cumulative response rate was 70.2 percent. Respondents came from all 50 states and the District of Columbia, with a balanced regional distribution. http://aspe.hhs.gov/pic/pdf/7054.pdf

FEDERAL CONTACT: Janet St. Lawrence, 601-632-5636 PIC ID: 7054

PERFORMER: Battelle Corporation, Arlington, VA

 

Telephone Survey Measuring HIV/STD Risk Behavior: Pretest Final Report

This is the second part of a methodological investigation that was designed to determine the effectiveness of a survey administration procedure--Telephone-Audio Computer-Assisted Self Interviewing (T-ACASI)- to collect sensitive information on sexual behavior, drug use, and HIV testing status. Under sections of a survey interview administered via T-ACASI, a telephone-facility-based computer reads the survey question and records the respondent’s keypad-tone-based response, bypassing the need for human interviewer involvement. The focus of the current investigation was sensitive sections of several questionnaires that were appropriate for general-population administration. Three issues were examined:

(1) item non-response due to refusals, (2) the frequency with which sensitive behaviors were reported, and (3) respondent self-report concerning comfort and truthfulness within the survey environment. It was found that item non-response was infrequent, leading to the conclusion that respondents who completed the interview were not at all resistant to answering the sensitive survey questions from the HIV/STD Standard Set. Almost no one reported engaging in risk behavior, leading to the conclusion either that the survey did not reach a high-risk population; that it did so but that high-risk individuals selected out through survey non-response; or that it reached high-risk individuals, but they simply chose not to report such behaviors.

FEDERAL CONTACT: John Anderson, 404-639-2046 PIC ID: 7775

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

 

The Development of a National HIV/AIDS Prevention Intervention Taxonomy for Program Evaluation

This project: (1) identified known taxonomies of interventions, (2) collected and organized categories of interventions used in CDC-funded health departments, (3) convened a group of experts from a variety of applicable fields and backgrounds to discuss the feasibility of a systematic taxonomy of HIV/AIDS interventions that can be integrated into and used by local, state, and national groups and organizations for program development, improvement, and evaluation, and (4) suggested a chronology of necessary activities CDC and its partners could undertake to develop a common vocabulary of HIV prevention interventions. Without a standard taxonomy of HIV prevention interventions, CDC and its prevention partners struggle to scientifically determine how CDC funding is affecting designated and whether CDC- funded programs are having the intended impact on the HIV/AIDS epidemic. The more consistent CDC and its partners were in classifying a type of HIV/AIDS intervention program or strategy, the more successful CDC would have been in providing higher quality data to Congress, federal agencies, state and local health departments and other constituencies for decision-making regarding funding allocation and targeted program development, improvement and evaluation. This is especially critical in making the link between prevention services provided and impacts on the epidemic assessed through surveillance systems and other data sources. The final report summarized the methods and results for each of the activities and concluded with recommendations synthesized from these findings. http://aspe.hhs.gov/pic/pdf/7707.pdf

FEDERAL CONTACT: Gary Uhl, 404-639-0950 PIC ID: 7707

PERFORMER: Macro International, Inc., Calverton, MD

National Center for Health Statistics

Evaluation of Non-response to the National Health and Nutrition Examination Survey

The objective of this study was to reduce the non-response bias in the National Health and Nutrition Examination Survey (NHANES). The central foci of this program assessment were the conduct of interviews and/or focus groups of respondents and non-respondents to evaluate potential reasons for response and non-response. The results of the ESP NHANES study mirror results from similar non- response studies. Sample members who chose to participate in the study were generally pleased to be a part of what they viewed as an important study. They were receptive to providing data on their health status and appreciated the opportunity to help society. The free medical examination was perceived as a very tangible benefit and many were surprised to learn they would also be paid for completing the interview and examination process. In contrast, non-respondents were more wary of the request for participation. Even aspects of the study meant to alleviate concerns or encourage participation were met with suspicion. Non-respondents also expressed great reluctance to commit the time necessary to complete the interview process. A number of recommendations were offered by the researchers, including: (1) providing greater detail about the study in the advance letter, (2) providing tailored interviewer training on refusal avoidance, and (3) developing a higher profile for the National Centers for Health Statistics, which tends to be an unfamiliar federal agency.

FEDERAL CONTACT: Vicki Burt, 301-458-4127 PIC ID: 7435

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

National Center for Infectious Disease

Definition of Public Health Laboratory Core Capacities for Infectious Disease Surveillance: Final Report

The purpose of this study was to ease the process of developing a consensus definition of core capacities for infectious disease surveillance starting with public health laboratories, public health units, and core capabilities for infectious disease surveillance. Health departments have used grants from the Centers for Disease Control and Prevention to hire and train additional epidemiologists and laboratorians, acquire laboratory technology, and upgrade communications and information technology. Assessing the adequacy of these investments has been constrained by a lack of consensus as to what state and local health departments required to conduct infectious disease surveillance. The consensus-building workshop format implemented in this study proved effective in eliciting constructive discussion among a diverse array of stakeholders regarding surveillance for drug-resistant Streptococcus pneumoniae. Workshop participants were generally supportive of the format as a mechanism for discussing and arriving at consensus regarding surveillance options. The report concludes that the consensus-building workshop methodology would likely prove as effective in similar consensus-building discussions of the set of core capacities required for other emerging infectious disease surveillance systems.

FEDERAL CONTACT: Deborah Deppe, 404-639-4668 PIC ID: 7763

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

National Center for Injury Prevention and Control

FEDERAL CONTACT: Nancy Cheal, 404-639-7222 PIC ID: 7743

PERFORMER: Battelle Corporation, Arlington, VA

 

Links Among Different Forms of Adolescent Violence

The purpose of this review was to lay the groundwork to better understand the prevalence, correlations, and consequences of different types of aggressive behaviors (physical, sexual, verbal, and coercive), delineate the association between dating violence and other forms of peer violence, and synthesize how patterns of aggressive behavior vary by sex, development stage, and other individual-level factors. An examination of the literature regarding violence among adolescents reveals problems with obtaining reliable estimates of the prevalence of different forms of violence. Family of origin, peer group, and other affiliations associated with prior victimization play an integral role in the perpetration of violence. The research found that child maltreatment, lack of parental supervision, victimization by peers, perceived social support among peers, and witnessing peer violence was predictive of and/or correlated with perpetration of date and other forms of violence.

FEDERAL CONTACT: Joyce McCurdy 770-488-4266 PIC ID: 7698

PERFORMER: Battelle Memorial Institute, MD, VA

 

The Role of Power and Control in Intimate Partner Violence Perpetration

The purpose of this project was to design a comprehensive research study concerning the role of power and control in the perpetration of intimate partner violence (IPV). The project produced a literature review synthesis and instrument review to measure factors predisposing men to IPV and a research protocol for a case-control study. The purpose of the study was to increase understanding of psychological and sociological factors that are potentially modifiable causes of IPV. These included substance abuse, poor impulse control, depression, patriarchal attitudes toward women, victim blaming, and socioeconomic status. Three dimensions of IPV were considered: theories linking IPV to power and control; risk and protective factors that mediate IPV by men; and research on programs that address power and control with perpetrators of IPV. The literature review and instrument review deliverables were designed to inform development of a research protocol for a case-control study. New insights gained from the proposed case-control study provided a valuable opportunity to enhance understanding of factors that contributed to IPV perpetration and were amenable to early intervention.

FEDERAL CONTACT: Nancy Cheal 404-639-7222 PIC ID: 7697

PERFORMER: Battelle Memorial Institute, MD, VA

National Immunization Program

Evaluation of the Distribution of the Influenza Vaccine by Health Care Providers in the U.S.

This study sought to better understand patterns of influenza vaccine distribution that occurred following advice to providers that they focus on high risk populations. Immunization tops the list of the ten great public health achievements during the last century. Despite the efficacy of immunization, vaccine- preventable diseases such as influenza are still a significant cause of mortality. During 11 of 23 annual influenza seasons between 1972-1995 in the United States, approximately 20,000 deaths, mostly among persons aged 65 or older, were attributed to complications of influenza. The objective for Healthy People 2000 was to have achieved influenza vaccination levels of greater than 60 percent among persons aged 65 or older, as well as among younger persons with high-risk medical conditions for whom the vaccine is recommended. The vaccination coverage objectives for influenza vaccine reported in Healthy People 2010 among persons aged 65 or older was elevated to 90 percent, while the goal remains at 60 percent for younger high-risk persons. In order to achieve these goals, it is essential that fundamental issues, such  as the ready availability of influenza vaccine be assured. Many providers who found themselves without vaccine at the time they had already prepared to provide influenza vaccine viewed the resulting four-to- eight week delay in the distribution of influenza vaccines as a de facto shortage. Preliminary findings revealed a variety of responses to the CDC recommendations to focus immunization efforts on those at high risk: some non-traditional providers, such as large pharmacy chains, canceled immunization clinics altogether because they were unable to determine who was at high risk and did not want to have to triage their customers; mass immunizers, such as those who contracted their services to large organizations, often rearranged their schedules in an effort to first assure coverage for high-risk persons; and some distributors made an effort to distinguish among their customers who were likely to be immunizing high-risk persons and distribute limited vaccines accordingly.

FEDERAL CONTACT: Marika Iwane, 404-639-8769 PIC ID: 7704

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Centers for Medicare and Medicaid Service

Mission

We assure health security for beneficiaries.

Evaluation Program

The research arm of the Centers for Medicare and Medicaid Services (CMS), the Office of Research, Development and Information (ORDI), performs and supports research and evaluations of demonstrations (through intramural studies, contracts and grants) to develop and implement new health care financing policies and to provide information on the impact of CMS’ programs. The scope of ORDI’s activities embraces all areas of health care: costs, access, quality, service delivery models, and financing approaches. ORDI’s research responsibilities include evaluations of the ongoing Medicare and Medicaid programs and of demonstration projects testing new health care financing and delivery approaches.

Examples of research themes include state program flexibility, the future of Medicare, provider payment and delivery, and vulnerable populations and dual eligibles.

Summary of Findings

Centers for Medicare and Medicaid Services

The Cost and Benefit of the Health Insurance Portability and Accountability Act

This was a multi-phase study that focused on groups that have been in the forefront of interacting with the population affected by the Health Insurance Portability and Accountability Act of 1996 (HIPAA) and three HIPAA related provisions: MHPA (Mental Health Parity Act of 1996), NMPHA (Newborns and Mothers’ Health Protection Act of 1996) and WHCRA (Women’s Health and Cancer Rights Act of 1998). The groups, i.e., state agencies, consumer advocacy groups, or individual researchers, are being contacted and benefits identified. Since strategy for implementation of HIPAA was technically based on state insurance regulatory models, the project delineated and described similar and/or differing effects as this model was applied at the Federal level. Evaluation results led to assisting CMS in the planning of future endeavors in private health insurance regulation. This report presented findings on whether HIPAA, which was intended to be a positive influence on healthcare portability and availability, led to negative changes in the healthcare market. Specifically, the areas of premium fluctuations, modifications to coverage, employer waiting periods and job-lock were addressed. Conclusions, based on interviews with Departments of Insurance, health plans, employers, employer advocacy groups and review of human resource literature, include: (1) healthcare costs have increased markedly in the last several years; (2) it is difficult to attribute premium increases and benefit plan modifications directly to HIPAA, however, they are more prevalent in the small group marketplace, which has been more directly impacted by HIPAA’s guaranteed issue and pre-existing conditions limitations; and (3) among the changes most commonly seen are increases in cost sharing features such as deductibles and copays, and pharmaceutical coverage cost sharing.

FEDERAL CONTACT: James Fuller, 410-786-3365 PIC ID: 7420

PERFORMER: Arthur Andersen and Company, Washington, DC

Office of Clinical Standards and Quality

FEDERAL CONTACT: Tricia L. Rodgers 410-786-1833 PIC ID: 7175

PERFORMER: Center for Health Policy Research, Denver, CO

Office of Research, Development, and Information

FEDERAL CONTACT: John Robst, 410-786-1217 PIC ID: 7185

PERFORMER: University of Minnesota, Minneapolis, MN

 

Questionnaire Development and Cognitive Testing Using Item Response Theory

The purpose of this project was to design a pool of survey questions to measure beneficiary knowledge of the Medicare program in future rounds of the Medicare Current Beneficiary Survey (MCBS). The intent of the project was to assist the Centers for Medicare and Medicaid Services in assessing how well the National Medicare Education Program is meeting its consumer information goals. The researchers developed questionnaire items using a comprehensive multi-step process that included: (1) background research, (2) review of Medicare informational materials and Medicare knowledge surveys, including the Medicare Current Beneficiary Survey (MCBS), and (3) multiple meetings and discussions with CMS and  the project’s seven-member Technical Advisory Panel (TAP). Knowledge scores calculated after the first round of interviews showed that respondents answered between 15 and 43 (out of 51) knowledge items correctly, or 29 to 84 percent. Based on feedback from the TAP and CMS, the survey questions were revised for a second round of testing. The language was simplified and more consistent terminology was used throughout. Fewer changes were suggested by the second round testing than by the first, suggesting that the instrument worked better. The report determined that using Item Response Theory (IRT) would produce a more precise measure of beneficiary knowledge, allowing CMS to more accurately determine areas in which beneficiaries lack knowledge and could use more information. In addition, an IRT-based knowledge scale would allow CMS to track knowledge, thereby allowing for the evaluation of the effectiveness of interventions and education programs.

http://aspe.hhs.gov/pic/pdf/7786.pdf

FEDERAL CONTACT: Tricia L. Rodgers 410-786-1833 PIC ID: 7786

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Food and Drug Administration

Mission

The FDA is responsible for protecting the public health by assuring the safety, efficacy, and security of human and veterinary drugs, biological products, medical devices, our nation's food supply, cosmetics, and products that emit radiation. The FDA is also responsible for advancing the public health by helping to speed innovations that make medicines and foods more effective, safer, and more affordable; and helping the public get the accurate, science-based information they need to use medicines and foods to improve their health.

Evaluation Program

The Food and Drug Administration's Evaluation Program ultimately reflects on the goals established and promulgated by the Department of Health and Human Services (HHS) for which FDA has responsibility. HHS's goals are products of its strategic performance planning process, and FDA uses its own strategic framework to accomplish these goals. This process also satisfies the implementation requirements of the Government Performance and Results Act (GPRA) and the Food and Drug Administration Modernization Act of 1997 (FDAMA). The strategic and performance process is an evolving set of program directions for FDA as changes occur in FDA's dynamic environment. FDA's challenges rest on its ability to leverage its efforts in that environment, which grows increasingly complex and more institutionally networked. The Agency will strive to maintain the scientific knowledge base necessary to achieve greater effectiveness in assuring the quality and availability of the products it regulates.

Summary of Findings

Center for Drug Evaluation and Research

CDER 2001 Report to the Nation: Improving Public Health Through Human Drugs

The Center promotes and protects public health by assuring safety and effectiveness of drugs used by Americans. The Food and Drug Administration (FDA) Modernization Act of 1997 affirmed the center’s public health protection role, clarified the FDA’s mission and called for the FDA to do the following: (1) promote public health by promptly and efficiently reviewing clinical research and taking appropriate action on the marketing of human drugs in a timely manner, (2) protect the public health by ensuring that human drugs are safe and effective, (3) participate through appropriate processes with representatives of other countries to reduce the burden of regulation, harmonize regulatory requirements and achieve appropriate reciprocal arrangements, and (4) carry out its mission in consultation with experts in science, medicine and public health and in cooperation with consumers, users, manufacturers, importers, packers, distributors, and retailers of human drugs. Among many findings, this report states that: (1) people with cancer, heart disease, HIV, AIDS, and other serious conditions have benefited from FDA approvals in 2001; FDA met its obligations to Congress for prompt and thorough review of drug applications supported by user fees; (2) with modern, state-of-the-art tools and techniques, FDA was  able to detect rare and unexpected risks more rapidly and take corrective action more quickly; and FDA augmented its risk assessment ability by gaining access to user fee data. http://www.fda.gov/cder/reports/rtn/2001/rtn2001.pdf

FEDERAL CONTACT: Debbie Henderson, 301-443-5281 PIC ID: 7890

PERFORMER: Food and Drug Administration, Rockville MD

Office of Policy, Planning and Legislation

Assessment of the Availability of Mammography Services

This analysis provided an overview of the availability of mammography services and of the changes in the demand for and supply of mammograms in recent years. It has been suggested that closures and  service curtailments by mammography providers have made it more difficult for women to schedule and obtain mammograms within a reasonable length of time. The possible shortage of mammography services has raised concerns that some women are not receiving needed mammograms due to limitations in access. The data from the FDA mammography database, in combination with population data, showed that the number of mammography facilities has declined in most states while the population of potential recipients of mammography services has increased. This suggested an increased demand relative to the supply of mammography services. The number of facilities did not capture the effective capacity of mammography providers. The CDC data failed to support claims that closures of mammography facilities or increased demand for mammography services have negatively affected the number of women obtaining mammograms. Data from CDC surveys, when combined with FDA data, show that providers conducted substantially more mammograms per facility in 2000 than they did in 1997. Other CDC survey data describing the prevalence of women who received clinical breast exams and mammograms showed that an increasing percentage of women in the 40 and older age group had recently obtained a mammogram. Furthermore, when only women receiving clinical breast exams were considered, an increased percentage received a mammogram in the same year. http://www.fda.gov/cdrh/mammography/mamavail01.pdf

FEDERAL CONTACT: Steve Tucker, (301) 827-5339

PIC ID: 6082.1

PERFORMER: Eastern Research Group, Lexington, MA

Cost Analysis of the Labeling and Related Testing Requirements for Medical Glove Manufacturers

The FDA is proposed regulations to reclassify surgeons’ and patient examination gloves as Class II medical devices, and placed controls and restrictions on the manufacturers and on some users of these products.  The reclassified gloves, including those made of natural rubber latex or synthetic material, were regulated in four categories: (1) powdered surgeons’ gloves, (2) powder-free surgeons’ gloves, (3) powdered patient examination gloves, and (4) powder-free patient examination gloves. This report showed that under the proposed regulation, the newly recommended protein and powder limits were lower than current average levels. In addition, the regulation required manufacturers to place expiration dates on glove labels to ensure adequate barrier protection (i.e., to prevent the transmission of pathogens to and from the patient). This report described the labeling requirements put forth by the FDA, the labeling cost model, and the various testing costs that determine total industry costs.

FEDERAL CONTACT: Steve Tucker, (301) 827-5339

PIC ID: 7889

PERFORMER: Eastern Research Group, Lexington, MA

Health Resources and Service Administration

Mission

To improve the Nation’s health by assuring equitable access to comprehensive, culturally competent, quality health care for all.

Evaluation Program

The Health Resources and Services Administration (HRSA) supports a wide array of very different programs and activities that promote access to needed health care for all, including primary health care centers, the National Health Service Corps, HIV/AIDS programs, maternal and child health activities, health professions training, rural health programs, organ donation and transplantation initiatives, and telehealth activities.  To provide underpinning for these efforts, HRSA’s evaluation program is designed to enhance strategic planning, strengthen budget and legislative development, and improve program performance.

HRSA also supports activities to enhance the quality of evaluation agency-wide, such as funding short courses in evaluation for staff and expanding agency staff’s access to technical assistance on the conceptualization, design and implementation of evaluation activities. The broad dissemination of evaluation products and results is also an area of emphasis.

Summary of Findings

Bureau of Primary Health Care

FEDERAL CONTACT: Fred Butler, 301-549-4281 PIC ID: 7126

PERFORMER: MDS Associates, Inc., Wheaton, MD

HIV/AIDS Bureau

Assessing the Impact of Increased Medicaid Dental Reimbursement Rates On the Utilization and Access of Dental Services in South Carolina

In January 2000, the South Carolina Medicaid program set new fees at a level such that each procedure would reimburse 75 percent of the dentists at their full, usual, and customary level. This was a stark departure from the usual approach taken, where state Medicaid agencies would approve modest increases in fees --normally a proportion of the full, usual, and customary level. Given that the South Carolina reimbursements were set such that most dentists would pay no financial penalty for filling a time slot with a Medicaid patient instead of a private pay patient, policy makers hoped that dentists would respond to the fee changes. The reform increased the number of: (1) Medicaid children receiving services; (2) services Medicaid children received; and (3) participating dentists, but not the Medicaid load per participating dentist. Consequently, the results indicate that the January 2000 reform had the desired results. It is recommended that South Carolina policy makers consider maintaining their commitment to a Medicaid reimbursement that reflects the 75th percentile of private pay fees for dental procedures.

FEDERAL CONTACT: John Kehoe, 404-562-7983 PIC ID: 7196

PERFORMER: Medical University of South Carolina, Charleston, SC

Maternal and Child Health Bureau

 

Discontinuous Coverage in Medicaid and the Implications of 12-Month Continuous Coverage for Children

This report analyzes the extent to which a policy of continuous coverage improves the continuity of Medicaid coverage and decreases the incidence of gaps in coverage that result when children temporarily lose Medicaid eligibility. The Balanced Budget Act of 1997 gave states the option of providing up to 12 months of continuous coverage for children through age 18 enrolled in Medicaid and SCHIP. Using 1994- 1995 Medicaid enrollment and payment data from four states--California, Michigan, Missouri, and New Jersey--this study examined the implications of a policy of 12-month continuous coverage. Findings on the impacts of 12-month continuous coverage indicate: (1) the number of children eligible for continuous coverage and ever enrolled during a year would increase, (2) the total number of months during which children are covered would increase, (3) the average cost per enrollee month would decline slightly, (4) among children who would qualify, payments would increase, (5) administrative costs associated with disenrollments, re-enrollments, and redeterminations in states using a six-month redetermination period would fall substantially, (6) staff costs associated with those categories in #5 are only a small portion of administrative expenses associated with the cost of operating public health insurance, and (7) effects on emergency room use and payments were not conclusive.

FEDERAL CONTACT: Jacob Tenenbaum, 301-443-9011 PIC ID: 7774

PERFORMER: Mathematica Policy Research, Inc. Plainsboro, NJ and, Boston University Medical Center Boston, MA

Office of the Administrator

Big Cities Health Inventory

This report is the fourth in a series from the Chicago Department of Public Health presenting city-to-city comparisons of leading measures of health. This edition established an advisory group to help guide the reports’ focus and content. Based on the committee’s recommendations, the report includes, for the first time, Healthy People 2010 objectives to serve as benchmarks. The report focuses on 20 indicators of health: five indicators of communicable diseases, nine causes of mortality, and six indicators of maternal and child health. The data presented here and in other studies analyzing urban health suggest that there is a unique urban health profile influenced by the dynamics peculiar to large cities. For example, the health outcome of the nation’s largest cities are less favorable than those of smaller urban and rural areas. The three largest cities in the U.S., New York, Los Angeles and Chicago had considerably different health profiles. New York ranked among the top ten cities for incidence of HIV, TB and HIV-related mortality. Chicago ranked highest in syphilis and homicide. Los Angeles ranked in the middle for most indicators. A better understanding of what caused such different health outcomes may improve health policies and programs in these cities.

FEDERAL CONTACT: Michael Millman, 301-443-0368 PIC ID: 7241

PERFORMER: Chicago Center for Health Systems Development, Chicago, IL

Indian Health Service

Mission

The mission of the Indian Health Service, in partnership with American Indian and Alaska Native people, is to raise their physical, mental, social, and spiritual health to the highest level.

Evaluation Program

The goal of the Indian Health Service (IHS) is to assure that comprehensive, culturally acceptable, personal and public health services are accessible to American Indian and Alaska Native people. The importance of evaluation in supporting this goal has increased significantly in recent years and includes American Indians and Alaska Natives as the primary stakeholders in defining the purpose, design, and execution of evaluations. The stakeholders use the end product of the evaluations, and are typically the population or groups most likely to be affected by the evaluation findings. The IHS has formally adopted the principles of a responsive evaluation practice to address the needs and concerns of American Indians and Alaska Natives.

The evaluation program of the IHS is managed by the Office of Public Health, Staff Office of Planning, Evaluation, and Research, which provides national leadership and consultation for IHS and Area Offices on strategic and tactical planning, program evaluation and assessment, public health and medical services, research agendas, and special public health initiatives for the Agency.

The evaluation needs of the IHS service programs are coordinated using two major types of short-term studies: policy assessments and program evaluations. The IHS policy assessments contribute to decision making about budget, legislation, and program modifications and include background information to support the Agency’s initiatives. Evaluations are focused at the program level, or Area Offices, and focus on specific program needs.

Each year IHS selects high-priority health care and management studies for funding through the submission of proposals to headquarters and Area Offices. These proposals are reviewed and rated by a panel of subject-matter experts, evaluation experts, and IHS staff for concurrence with IHS strategic goals, objectives, and priority areas. The proposals are then prioritized and forwarded to the IHS Director, who reviews the projects that are recommended for funding and determines the respective funding levels.

Summary of Findings

Office of Public Health

 

Quantifying the Unmet Need in IHS/Tribal EMS

The purpose of this report was to present data collected from Tribal Emergency Medical Services (EMS) programs, report their unmet needs, and make suggestions for future resource allocation by the Indian Health Service. It was found that the 562 federally recognized Tribes do not have their own EMS programs. While it was not known how the members of these Tribes obtain emergency medical services, it was assumed that they relied on local public (city or county) and private ambulance services to access EMS. Injury deaths among Native American children were caused by motor vehicle crashes, pedestrian related mother vehicle crashes, drowning, fires, and suicide. It was reasonable to assume these fatality rates reflected a significant portion of the EMS run volume by tribal Emergency Medical Technicians (EMTs) within their communities. It was also reasonable to conclude that these same EMTs do, or could contribute to Service Unit injury prevention activities through their role as participating members of Injury Prevention Committees. These childhood injury fatalities had significant impact on the direction of community and service area Emergency Medical Services for Children activities.

FEDERAL CONTACT: Dr. Eric B. Broderick, 301-443-3024 PIC ID: 7860

PERFORMER: Office of Program Planning and Evaluation, Bethesda MD

 

Study of the Impact of a Full-Time Community Health Nurse on the Health of Native American (Sioux) Patients with Hypertension & Prevention

The goal of this project was to hire a community health nurse to follow hypersensitive patients. The nurse was responsible for identification, education, and following the progress of the hypertensive patients at the Sisseton Service Unit. Statistics were gathered on the patients in order to know more about them and to better plan their treatment. The goal of the project was to achieve a blood pressure control rate of 50 percent for patients with hypertension. Control was defined as a systolic blood pressure below than 140 and a diastolic lower than 90. This report presents preliminary results of the Community Health Nurse on the care of the hypertensive population. It also presents the process by which a pharmacist influenced the treatment of hypertension and outcomes. The impact made on the treatment of hypertension is expected to be more fully demonstrated through lower illness and death from Cardiovascular Disease in the Native American Population.

FEDERAL CONTACT: Elizabeth A. Fowler, 301-443-1270 PIC ID: 6413

PERFORMER: Aberdeen Area Indian Health Service, Aberdeen, SD

National Institutes of Health

Mission

To sponsor and conduct medical research that leads to better health for all Americans.

Evaluation Program

The National Institutes of Health (NIH) generates scientific knowledge that leads to improved health. This is done by conducting medical research in its intramural laboratories and by supporting research in universities, medical and health professional schools, and other health research organizations. NIH fosters the widespread dissemination of the results of medical research, facilitates the training of research investigators, and ensures the viability of the research infrastructure. The NIH Evaluation Program is an integral part of how NIH sponsors and conducts medical research.

Results based management is recognized as a basic principle for the sound and productive operation of government agencies and their programs. This is evidenced most notably by passage of the Government Performance and Results Act (GPRA) and by the considerable effort across the federal government to implement results based management mechanisms. With GPRA and other initiatives aimed at increasing public sector accountability (such as the Chief Financial Officers Act and the Government Management Reform Act), interest in the use of evaluation has increased steadily among NIH administrators and others, such as officials within the General Accounting Office, the Office of Management and Budget and the Department of Health and Human Services.

Philosophy and Priorities

The NIH Evaluation Program provides information to assist the NIH Director and the NIH Institute and Center (IC) Directors in determining whether NIH goals and objectives are being achieved and to help guide policy development and program direction. Evaluations are planned and conducted from two sources of funds: 1 percent evaluation set-aside funds used to fund trans-NIH projects, and IC program funds used for program evaluations for use by various committees, working groups, task forces, workshops, conferences, and symposia to assist the ICs in program management and development. This approach ensures that planning and priority setting specific to the mission of each IC are fully developed and implemented and that there is central leadership for developing crosscutting initiatives and promoting collaboration among the ICs.

NIH’s major evaluation priority areas fall within three broad program areas: basic research, research training and career development, and facilities. NIH conducts evaluations in these areas to assess strategies and goals, develop performance measures and improve operations.

Policies and Operations

A distinguishing feature of the NIH Evaluation Program is the utilization of a variety of evaluation strategies that include the use of national advisory councils, boards of scientific counselors, consensus development conferences, and ad hoc committees that help to chart scientific directions and select the most promising research to support.

A two-tier system is used to review project requests that will use 1-percent evaluation set-aside funding. The first tier involves a review and recommendations by the NIH Technical Merit Review Committee (TMRC) on the technical aspects of project proposals and whether a project fits within HHS guidelines for use of the set-aside fund. The second tier involves the NIH Evaluation Policy Oversight Committee, which considers TMRC recommendations, conducts policy level reviews, and makes final funding recommendations to the NIH Director or his designee.

Summary of Findings

National Center for Research Resources

Scientific and Engineering Research Facilities

This report consists of a collection of tables prepared biennially through the National Science Foundation’s congressionally mandated Survey of Scientific and Engineering Research Facilities (Facilities survey). The survey originated in 1986 in response to Congress’ concern about the state of research facilities at the Nation’s colleges and universities. The survey was sent to research-performing colleges and universities in the United States. Research-performing colleges and universities are defined as meeting one of three criteria: (1) offer doctorates in S&E fields, (2) report at least $150,000 in research and development (R&D) expenditures for fiscal year 2000, and (3) are an Historically Black College or University with an R&D expenditure. The survey was also sent to nonprofit biomedical research organizations. The 2001 survey was mailed to academic institutions in April of 2001 and data collection ended on July 24, 2001.

Of the 580 eligible institutions, 90 percent returned surveys. http://www.nsf.gov/sbe/srs/nsf02307/pdf/nsf02307.pdf

FEDERAL CONTACT: Jeff Sussman 301-496-5617

PIC ID: 6863

PERFORMER: National Science Foundation, Arlington, VA

 

Survey of NIH-funded Investigators Who Use Nonhuman Primates: Report on Survey Findings

The Center commissioned the National Survey of Nonhuman Primate Research Use to learn about investigator access to nonhuman primate resources at the eight Regional Primate Research Centers (RPRCs) and to assess current and future nonhuman primate needs. Humanitas Incorporated, the survey contractor, conducted the survey from June through September 2001 via the Internet and used mail and telephone follow-up procedures. The survey queried investigators who were identified in NIH databases as having been awarded NIH funds in FY99 for research nonhuman primates. The survey found that: (1) nonhuman primate survey findings can promote understanding of RPRC access and nonhuman primate needs; (2) RPRCs provided access to valuable nonhuman primate resources – all investigators who used an RPRC gave the highest satisfaction ratings possible to the quality of the nonhuman primate resources (freedom of animals and specimens from disease) and to the availability of desired services (specimens, animal services, and collaborative consultations); (3) obtaining nonhuman primate resources was challenging – especially regarding animal-, species- and characteristics-specificity; (4) rhesus macaques were most often used and most needed, more than half of responding investigators needed rhesus macaques for their research in the next three years; and (5) respondents recommended increasing the availability of nonhuman primates and improving nonhuman primate allocation systems.

FEDERAL CONTACT: Patricia Newman, 301-435-0864 PIC ID: 7639

PERFORMER: Humanitas, Inc., Silver Spring, MD

National Institute of Child Health and Human Development

FEDERAL CONTACT: Jeff Sussman 301-496-5617 PIC ID: 7630

PERFORMER: Caliber Associates, Fairfax, VA

National Institute of General Medical Sciences

Measuring Up: Research & Development Counts for the Chemical Industry

This study was undertaken to measure the impact, and return, or payoff, of chemical research and development. Macroeconomic studies have addressed and documented R&D contributions to the nation’s economic growth; this study examined chemical research. Key accomplishments of this study included the development of a measurement that quantified the chemical R&D impact, and a methodology to show linkages between chemical literature citations and science innovation. The study revealed that: (1) every dollar invested in chemical R&D today produced $2 in operating income over six years; (2) the chemical industry’s share of total U.S. R&D had declined from 11 percent in 1956 to 8 percent in 1992; (3) the industry still appeared to be holding strong despite growing globalization; and (4) predictability of government funding was more important to industry than the level of funding. The study concluded that U.S. Chemicals technology was strong and getting better, but it is not the leading technology among economic indicators. Its quality is clearly increasing when compared to foreign-invented Chemical patents, and its science linkage was increasing rapidly, especially when compared to foreign Chemical patents. It was more highly linked to science than other major U.S. technology, except for the Life Sciences, and it was also highly linked to public science and to local science.

FEDERAL CONTACT: Dr. James Onken, 301-594-2762 PIC ID: 7659

PERFORMER: Council for Chemical Research, Washington DC

National Institute of Aging

Evaluation Feasibility Study for the National Institute on Aging’s Book and Videotape, Exercise: A Guide from the National Institute on Aging

Since 1998, the National Institute on Aging’s (NIA) Office of Communications and Public Liaison (OCPL) has promoted and distributed the book, “Exercise: A Guide from the National Institute on Aging,” and an accompanying video. The materials were developed in response to an increasing number of requests about exercise for older people from the public, the media, and private-sector health and fitness providers. A kick-off event in 1998, including representatives from NIA, NASA, DHHS Office on Women’s Health, and featuring Senator/Astronaut John Glenn, marked the initial availability of the Guide. Since then, the Guide has become one of NIA’s most popular publications. NIA distributed 72,994 copies of the Guide in 2001 alone. Based on the clear public health need, NIA plans to update the Guide and continue its efforts to promote the beneficial effects of exercise among older Americans.

FEDERAL CONTACT: Freddi Karp, 301-496-1752 PIC ID: 7642

PERFORMER: Academy for Educational Development, Washington, DC

National Eye Institute

Designing Pilot Study to Enhance Referrals to Vision Rehabilitation Services

The Institute conducts a public education program on low vision designed to increase awareness of the benefits of vision rehabilitation. A key part of this program will be to develop strategies that will increase referrals from eye care professionals to vision rehabilitation services. In 2001, the NEI conducted over 25 focus groups with eye care professionals and office staff to identify barriers, opportunities, and methods for enhancing the referral process. NEI seeks to determine the nature and extent of problems a referral education program should address, what the goals should be, and how a pilot program should be developed to achieve those goals. Recommendations included: (1) print media such as the proposed Physician’s Vision Rehabilitation Services Desk Reference, vision rehabilitation services fast facts, other brief print materials, (2) audiovisual media such as Physician’s Vision Rehabilitation Services Audio Tape Series and Patient Videos Addressing Low Vision Referral, and (3) Web-based Enhancements and Referral Tools, such as Web links and the Low Vision Services Locator.

FEDERAL CONTACT: Rosemary Janiszewski, 301-496-5248

PIC ID: 7891

PERFORMER: ORC Macro, Calverton, MD

Office of the Director

FEDERAL CONTACT: James A. Schuttinga 301-496-2229 PIC ID: 7629

PERFORMER: Joel Popkin & Company, New York, NY

Office of the Assistant Secretary for Planning and Evaluation

Mission

To provide analytical support and advice to the Secretary on policy development and assist the Secretary with the development and coordination of department wide program planning and evaluation activities.

Evaluation Program

The Assistant Secretary for Planning and Evaluation (ASPE) functions as a principal advisor to the Secretary on policy development and, in this capacity, conducts a variety of evaluation and policy research studies on issues of national importance. ASPE also is responsible for department wide coordination of legislative, planning, and evaluation activities. In its evaluation coordination role, ASPE has the following tasks:

• Provide annual guidance to all HHS agencies and staff offices regarding evaluation priorities, procedures, and review requirements.
• Review evaluation priorities proposed by HHS agencies, providing advice about the focus or method of proposed projects and identifying opportunities for collaboration and effective use of resources.
• Prepare planning and summary reports on evaluation activities as required by Congress.

Through the departmental evaluation planning process, ASPE has the capacity to identify crosscutting program or policy issues of concern to the Secretary and specific program and policy areas not covered by the HHS Agency evaluation plans. In these instances, ASPE initiates evaluations or collaborates with the agencies to conduct evaluations or policy assessments.

Another continuing evaluation objective of ASPE is to support and promote the development and improvement of databases that HHS agencies and ASPE use to evaluate health care programs and health trends. For example, ASPE has been the major initiator, in collaboration with the National Center for Health Statistics at the Centers for Disease Control and Prevention, of the first comprehensive survey of people with disabilities in the United States. Collection of the first new data was completed in FY 1996 along with national prevalence data on disability.

The ASPE chairs the Research Coordination Council (RCC), which will evaluate Department-wide research priorities to ensure that efficiencies are realized and research funding priorities are consistent with the Administration’s priorities. The ASPE also co-chairs and provides support to the HHS Data Council, which is charged with integrating key national surveys, such as linking health status indicators with indicators of well-being.

Finally, ASPE uses evaluation funds to promote effective use of evaluation-generated information in program management and policymaking. The latter is accomplished through the dissemination of evaluation findings and other activities, such as providing technical assistance to agencies in the development of performance measures.

Summary of Findings

Office of Disability, Aging, and Long-Term Care Policy

FEDERAL CONTACT: William Marton, 202-690-6443 PIC ID: 7770

PERFORMER: Philadelphia Geriatric Center, Philadelphia, PA

The Medicaid Buy-In Program: Lessons Learned from Nine “Early Implementer” States

This report discusses findings from case studies of nine states operating Medicaid Buy-In programs for working persons with disabilities. It is the second of three reports. The first included in-depth case studies of nine early implementer states entitled, “Medicaid Buy-In Programs: Case Studies of Early Implementer States.” For many individual Social Security Disability Insurance (SSDI) and Supplemental Security Income (SSI) recipients, the risk of losing Medicaid coverage linked to their case benefits was a powerful work disincentive. Eliminating barriers to health care and creating incentives to work can greatly improve financial independence and well-being. This paper examines state decisions concerning program eligibility, their approaches to estimating program enrollment and costs, and the patterns of program enrollment. The report seeks to assist stakeholders to design and implement Medicaid Buy-In programs and related work incentive initiatives. Medicaid Buy-In programs typically are managed by state Medicaid agencies with significant input from consumers and assistance from other state agencies. It was found that (1) stakeholder involvement was important in program design, (2) the Medicaid Buy-In program was linked to other employment supports, and (3) the state Medicaid agency usually worked with other state agencies to support persons with disabilities in the workplace. It was also found that eligibility standards and cost-sharing policies show considerable variation across the states and may have a significant impact on program enrollment. Finally, state policies on general Medicaid eligibility, SSI, and state SSI supplementation and federal policies on SSDI affected Medicaid access for working persons with disabilities.

FEDERAL CONTACT: Andreas Frank, 202-690-6443 PIC ID: 7556.1

PERFORMER: George Washington University, Washington, DC

Office of Health Policy

 

The Cost of Medigap Prescription Drug Coverage

This study examined the price of standard Medigap policies that offer coverage for prescription drugs. Three of the ten standard Medigap policy forms (H, I, and J) cover prescription drugs. In most states, new Medicare beneficiaries who are either buying Medigap coverage for the first time, or looking to change their policy or insurer, have access to at least one of these plans. In every state, combined enrollment in Medigap plan that covers prescription drugs was very low. Such low enrollment may be related to non-price problems of access and also to the much higher price of Medigap policies that cover prescription drugs. This study considered the average cost of Medigap policies that covered prescription drugs in 1999 and also their marginal cost relative to other standard Medigap policies. This study focused on individual, not group, Medigap plans and found that premiums for Medigap policies covering prescription drugs are expensive by most measures. The cost of Medigap policies that cover prescription drugs is also high relative to other standard Medigap policies. Therefore, the Medigap market is neither a simple nor inexpensive place for Medicare beneficiaries to obtain prescription drug coverage.

FEDERAL CONTACT: Stephen Finan, 202-690-7387 PIC ID: 7348

PERFORMER: Mathematica Policy Research, Inc., Plainsboro, NJ

 

Premium Assistance Programs Under SCHIP: Not for the Faint of Heart?

For this study, three states--Massachusetts, Mississippi, and Wisconsin--were selected for in-depth examination because, until recently, they were the only states with federally approved State Children’s Health Insurance Program (SCHIP) premium assistance programs. Under The State Children’s Health Insurance Program (SCHIP), states have the option to subsidize employer premiums for low income children, and in some cases, their parents. Providing families with premium assistance may be a viable means of covering more uninsured children because this country’s health insurance system is dominated by employer-based coverage. This study included a literature review, a study of relevant SCHIP regulations, and a review and analysis of relevant information collected from the Urban Institute’s SCHIP evaluation--conducted as part of the “Assessing the New Federalism” project. The majority of information for this study was collected during telephone interviews and the protocol primarily focused on the impact of SCHIP regulations, state implementation experiences and lessons learned. The premium assistance programs examined in this study suggested several different findings that may be useful to other states considering such programs. They included the following: (1) outreach -- although states have recently gained considerable experience in targeting public program outreach efforts to families and community- based organizations, premium assistance programs pose a new outreach challenge to states as they directly involve and require the cooperation of employers, (2) enrollment -- Wisconsin and Massachusetts have learned they must engage in complex, time-consuming, and challenging processes for enrolling children (and often their parents) into premium assistance programs, and (3) cautious optimism -- Massachusetts and Wisconsin believe their premium assistance programs are worthwhile endeavors, but caution other states considering such programs from being overly optimistic about initial enrollment.

FEDERAL CONTACT: Caroline Taplin, 202-690-7906 PIC ID: 7895

PERFORMER: Urban Institute, Washington, DC

Office of Human Services Policy

 

 

 

FEDERAL CONTACT: Alana Landey, 202-401-6636 PIC ID: 7868.1

PERFORMER: Mathematica Policy Research & Urban Institute, Washington, DC

Office of Planning and Policy Support

Evaluation Synthesis: Tobacco Control Programs in Communities of Color

This study assessed the extent to which Communities of Color were being targeted and summarized the key findings for Communities of Color. The study was conducted utilizing published scientific literature, state reports, and tobacco control Web sites for eight state case studies. The eight states included in this evaluation were: Arizona, California, Florida, Massachusetts, Maryland, Mississippi, Washington, and Texas. These states were selected based on their demonstrated success in reducing rates of initiation of smoking, increasing smoking cessation, or reducing environmental tobacco smoke (ETS). They also had a presence of large numbers of minority populations or multiple, small segments of racially and ethnically diverse communities. Also, these states allotted funds for tobacco prevention and control. The key findings indicated that: (1) all eight states had adopted the goal of prevention and have or will soon have cessation as a goal, (2) published evaluations of interventions pertaining to  Communities of Color were few and focus on adolescents, (3) outcome evaluations of interventions pertinent to Communities of Color were difficult to find, (4) a published evaluation supports the importance of mass media campaigns in combination with other interventions in preventing the initiation of tobacco use in Communities of Color, (5) the published literature supported the effectiveness of community education in Communities of Color for smoking cessation and reduction of ETS and was suggestive of an effect in decreasing smoking initiation, and (6) one study provided evidence that legislation prohibiting sales to minors was effective in all ethnic neighborhoods.

FEDERAL CONTACT: Vijaya ChannahSorah, 202-260-3815

PIC ID: 7815

PERFORMER: Research Triangle Institute, Research Triangle Park, NC

Office of Public Health and Science

Mission

To provide advice on public health and science to the Secretary of Health and Human Services, to provide executive direction to program offices within the Office of Public Health and Science (OPHS), and, at the direction of the Secretary, to coordinate crosscutting public health and science initiatives in the Department.

Evaluation Program

The Office of Public Health and Science (OPHS) provides advice, policy and program coordination, and leadership in the implementation, management, and development of activities related to public health and science, as directed by the Secretary. OPHS helps HHS conduct broad-based public health assessments to better address and solve public health problems. It assists other parts of HHS in anticipating future public health issues and helps ensure that HHS designs and implements appropriate approaches, interventions, and evaluations that will maintain, sustain, and improve the health of the Nation. OPHS provides leadership and policy recommendations on population-based public health and science and, at the Secretary’s direction, leads or coordinates initiatives that cut across agencies and operating divisions. In addition, OPHS communicates and interacts, on behalf of the Secretary, with national and international professional and constituency organizations on matters of public health and science. Finally, OPHS’s unique role allows it to use its resources to link important HHS programs or fill gaps in areas needing better policy formulation and coordination.

OPHS’ evaluation strategy focuses on public health and science issues that cut across multiple interests of the operating divisions and requires a coordinated approach to achieve the most effective results. OPHS evaluations support the Assistant Secretary for Health as the Secretary’s senior advisor for public health and science. OPHS also conducts evaluations specific to the needs of the programs operated from the offices located within OPHS, such as emergency preparedness, women’s health, minority health, disease prevention and health promotion, and research integrity. The ten HHS Regional Health Administrators have access to some evaluation funds. In keeping with its role within the Department, OPHS does not undertake evaluations more appropriately undertaken by operating divisions of HHS or by the Assistant Secretary for Planning and Evaluation (ASPE).

Summary of Findings

Office of Minority Health

FEDERAL CONTACT: Valerie Welsh, 301-443-9923 PIC ID: 7710

PERFORMER: Health Systems Research Associates, Chevy Chase, MD

 

National Youth Sports Program Evaluation Project

The National Youth Sports Program (NYSP), initiated in 1968 as an outreach program of the National Collegiate Athletic Association, was designed to enhance the quality of life for youth from low-income families by providing academic and athletic activities through a five-week summer camp. Key NYSP objectives include the utilization of sports and educational activities for the purpose of exposing youth to educational opportunities, physical fitness, life skills, citizenship, interactive athletic skills, and health issues. The purpose of this funded project with the Inter-University Program for Latino Studies was to develop tools and resources that could be used by NYSP managers and staff to assess the program’s potential effects on knowledge, attitudes, and behaviors of youth participating in the summer program. Primary activities that were conducted included: a qualitative review of program activities via site visits at selected NYSP locations; development and pilot-testing of pre- and post-survey instruments designed to assess NYSP participants’ level of exposure to risk behaviors, their knowledge and beliefs about health and risk factors, and short-term impacts of the health-related curriculum and activities on participant knowledge; and revisions to the instruments based on pilot results and feedback on program performance. A comprehensive Health Resource Guide was compiled listing health education materials covering a wide range of health topics. The tools and resource guide can be used at NYSP sites to conduct self-assessments to determine whether program objectives are being met and to improve the program’s structure and content as indicated.

FEDERAL CONTACT: Joyce Heinonen, 301-594-0769 PIC ID: 7709

PERFORMER: The Inter-University Program for Latino Research, University of Notre Dame, South Bend, IN

 

Patients Who Don’t Speak English: Improving Language Minorities’ Health Care with Professional Interpreters

This was a study of the effects of professional interpreter services on health visit levels and patterns of a sample of limited English proficient (LEP) patients. LEP patients are those who speak a primary language other than English and who cannot speak English at all or speak English so poorly that they cannot communicate in English without assistance. Language services, including the use of professional interpreters, were seen as a facilitator variable in conventional health care models because they help patients with limited English to communicate with health care providers, and therefore access medical care. It was expected that language services were necessary to significantly reduce language barriers to health care. Findings provide support for the hypothesized role of interpreter services. LEP patients increased their health visits after the implementation of interpreter services. The increased utilization was larger for groups that can be considered to be most vulnerable to language barriers in accessing health care, including the elderly, the poor, and patients who had below average health visits prior to the implementation of interpreter services. Given the scarcity of systematic research on the effects of interpreter services on LEP patients’ health care, this study’s findings represented new and welcome evidence. This study has shown that organizations involved in health care, such as those providing health insurance, medical services, and language services, can play critical roles in gathering the kind of information that may be productively analyzed in further research.

FEDERAL CONTACT: Guadalupe Pacheco, 301-443-5084 PIC ID: 7858

PERFORMER: Department of Sociology, Portland State University, Portland, OR

Substance Abuse and Mental Health Services Administration

Mission

SAMHSA’s mission is to build resilience and facilitate recovery for people with or at risk for substance abuse and mental illness.

Evaluation Program

The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its programs and individual grant projects to assess the effectiveness of prevention, treatment, and rehabilitation approaches and systems of care. Evaluation is also conducted to ensure accountability for federal funds and to measure results toward SAMHSA’s programmatic and policy objectives. In compliance with the Government Performance and Results Act (GPRA), SAMHSA is improving performance management and results by identifying annual, long-term and cost-efficiency performance measures to manage its programs.

SAMHSA has an integrated model of evaluation and planning. Strategic planning identifies priorities that drive the development of grant programs and evaluations. The formulation of programmatic and evaluation priorities includes consultation with SAMHSA Center Advisory Councils, with other HHS agencies, and with experts in the fields of evaluation and service delivery. Early and continuous coordination of program planning and evaluation design results in the articulation of program objectives that may be evaluated. Planned evaluations demonstrate the extent to which the grant programs have achieved their objectives, and SAMHSA translates these results into information that can be used for program and policy development. The strategic planning and policy development processes then use these results to refine SAMHSA’s priorities and performance objectives.

The specific type of evaluation required for a grant program is dependent on the type and purpose of the program. To the greatest extent possible, SAMHSA encourages the use of comparable data elements and instruments across its evaluations to implement a comprehensive evaluation system and to minimize respondent burden. Efforts to improve evaluation are continuing and SAMHSA is committed to using systematic approaches in using data to accomplish its mission.

Summary of Findings

Center for Mental Health Services

 

PERFORMER: R.O.W. Sciences, Inc., Rockville, MD

 

Lessons From the Evaluation of the ACCESS Program

The authors summarized the main findings of the ACCESS program and offer lessons for policy makers. Data from studies at the site level and the client level are summarized and synthesized with the authors’ collective experience with the ACCESS program. The results of the evaluation suggest that although service systems integration can be improved, targeted efforts to implement strategies for integration do not produce better client outcomes. Efforts to integrate service systems can be supported by their effects on some organizational relationships within the mental health service system but not by their widespread effects across human services or their direct effects on clients.

FEDERAL CONTACT: Frances Randolph, 240-276-1892 PIC ID: 4980.4

PERFORMER: R.O.W. Sciences, Inc., Rockville, MD

Center for Substance Abuse Prevention

Cross-site Study of the Children of Substance Abusing Parents Initiative

A cross-site study was conducted on the Children of Substance Abusing Parents (COSAPs) initiative that targeted COSAPs and their caregivers. Each study site was designed to test substance abuse prevention strategies for a specific age cohort of COSAPs and their substance-abusing caregivers. Age cohorts for the target children included 6- to 8-year-olds, 9- to 11-year-olds, and 12- to 14-year olds. An outcome and a process evaluation were conducted. The outcome study used a quasi-experimental design that involved nonequivalent comparison groups and repeated measures. Each study site formed appropriate comparison groups of index COSAPs and index caregivers. In addition to using dosage data, the process evaluation used program-level data collected by means of site visits. The findings showed the Center for Substance Abuse Prevention programs had many positive effects on risk and protective factors for the COSAPs. Programs for younger COSAPs resulted in increased self-control, school bonding, school performance, and perception of alcohol, tobacco, and other drugs (ATOD) harm. Positive outcomes for older COSAPs began to emerge 6-months post intervention when there was an increased perception of ATOD harm, and reductions in problem behaviors. Although the programs did not affect substance use among the COSAPs, the rate of ATOD use among the children were very low at baseline, making it difficult to measure reductions in use. The analysis found no significant negative program outcomes.

FEDERAL CONTACT: Soledad Sambrano, 240-276-2416 PIC ID: 7888

PERFORMER: EMT Associates, St. Louis, MO

Center for Substance Abuse Treatment

This literature review summarizes the major findings from books, published articles, research and evaluations, and government documents (including “Web” publications) published since 1980, which focus on the cost effectiveness and cost benefits of substance abuse treatment. Cost effectiveness and cost benefit studies play an important role in evaluating existing and alternative substance abuse approaches and in assessing new treatment methods.  Evaluating the outcomes and costs of treatment is necessary in order to determine how to more efficiently allocate scarce resources.  This document is intended to assist policymakers, researchers/evaluators, and treatment providers to identify and acquire evidence-based information. Policymakers and researchers/evaluators can use this information to help inform decisions of whether or not an increase in effectiveness justifies an increase in cost of a treatment. Treatment providers need this information when they seek funding from public agencies.

This review also explores trends and areas where there are gaps in the literature. FEDERAL CONTACT: Nita Fleagle 240-276-2823

PIC ID: 7811.1

PERFORMER: Caliber Associates, Fairfax, VA

Appendix A - Evaluation in the Department of Health and Human Services

Evaluation plays an integral role in carrying out the HHS mission. Assessing various aspects of agency program performance allows staff to identify means of improving that performance. The HHS evaluation function has three goals: (1) to provide information on HHS programs that helps government officials and members of the Congress make decisions related to programs, policies, budgets, and strategic planning; (1)  to help HHS managers improve program operations and performance; and (3) to disseminate evaluation results and methodological tools useful to the larger health and human services community of state and local officials, researchers, advocates, and practitioners for improving the performance of their programs.

For HHS, evaluation is the assessment of the performance (efficiency, effectiveness, and responsiveness) of HHS programs or strategies through the analysis of data or information collected systematically and ethically, and the effective use of resulting information in strategic planning, program or policy decision making and program improvement. As mentioned in the foreword of this report, evaluations serve one or more of the following:

• Program effectiveness determines the impact of HHS programs on achieving intended goals and objectives.
• Performance measurement monitors annual progress in achieving departmental strategic and performance goals. To support performance measurement, we are investing evaluation funds to develop and improve performance measurement systems and the quality of the data that supports those systems.
• Environmental assessment seeks to understand the forces of change in the health and human services environment that influence the success of our programs and the achievement of our goals and objectives. Such understanding allows us to adjust our strategies and continue to deliver effective health and human services.
• Program management reflects the need of program managers to obtain information or data  helpful for designing and managing programs. These evaluations generally focus on developmental or operational aspects of program activities and provide understanding of services delivered and populations served.

Evaluation Resources

legislative authority, which is provided for in Section 241 of the Public Health Service (PHS) Act. This authority was established in 1970, when the Congress amended the Act to permit the HHS Secretary to use up to 1 percent of appropriated funds to evaluate authorized programs. Section 206 of the FY 2002 Labor, HHS, and Education Appropriations Act increased the amount the Secretary could use for evaluation to 1.25 percent. Section 241 limited the base from which 1.25 percent of appropriated funds could be reserved for evaluations of programs authorized by the PHS Act. Excluded were funds appropriated for FDA, IHS¹ and other programs that were managed by PHS agencies but not authorized by the Act (e.g., HRSA’s Maternal and Child Health Block Grant and CDC’s National Institute for Occupational Safety and Health).

---

¹FDA programs are principally authorized by legislation other than the PHS Act, specifically the authority of the Agriculture, Rural Development, Food and Drug Administration and Related Agencies Appropriations Act. IHS programs are authorized under the Indian Health Care Improvement Act and the Indian Self-Determination Act, and appropriated under the Department of the Interior and Related Agencies Appropriations.

Evaluation Management

The management of HHS evaluations, which are carried out on a regular basis by the HHS agencies and offices and coordinated by ASPE, involves four basic functions: (1) evaluation planning and coordination, (2) project management, (3) quality assurance, and (4) dissemination of evaluation reports:

Evaluation Planning and Coordination

Project Management

HHS agencies, OIG, and ASPE all are responsible for executing annual evaluation plans, developing evaluation contracts, and disseminating and applying evaluation results. All agencies and their subunits are encouraged to coordinate with each other on research/evaluation project planning and release of final reports that are likely to relate to work of other HHS agencies. Within agencies— while there is some oversight responsibility and execution capability in the Office of the Director or Administrator—the various subunits (centers, institutes, and bureaus) conduct much of the day-to-day evaluation activity.

The OIG performs independent evaluations through its Office of Evaluations and Inspections (OEI). The OEI’s mission is to improve HHS programs by conducting inspections that provide timely, useful, and reliable information and advice to decision makers. This information (findings of deficiencies or vulnerabilities and recommendations for corrective action) is usually disseminated through inspection reports issued by the Inspector General. A summary of individual inspection reports and other OIG reports can be viewed on the Internet at http://oig.hhs.gov/reports.html. OEI also provides technical assistance to HHS agencies in conducting their evaluations.

Quality Assurance and Improvement

Most evaluation projects are developed at the program or office level. The initial quality review is generally conducted by a committee of agency- or office-level policy and planning staff members. Before a project is approved, it is also reviewed for technical quality by a second committee with expertise in evaluation methodology. Technical review committees follow a set of criteria for quality evaluation practice established by each agency. Some HHS agencies also have external evaluation review committees composed of evaluation researchers and policy experts from universities and research centers.

Since HHS began reporting to Congress in 1995 on evaluations completed and in-progress through the Performance Improvement report series, the Department has focused more attention on improving the quality of evaluations performed every year. The HHS Evaluation Review Panel, convened to recommend evaluations for highlighting in Chapter I of this report, has contributed insights to HHS evaluation officers on the strengths and challenges of ensuring quality evaluations. HHS evaluation officers have had opportunities to discuss these strengths and challenges with the Panel and to identify steps to improve agency evaluation capacity.

Based on these discussions, ASPE initiated two evaluation capacity-building projects. The first was to establish an evaluation consultant Blanket Purchase Agreement (BPA) contract mechanism to assist agency program managers in obtaining short term assistance in developing evaluation designs or monitoring the performance quality of evaluation contractors. The second was to conduct a department- wide evaluation training needs assessment survey, which indicated training needs in various areas.

Dissemination of Evaluation Reports

Typically, the results of HHS evaluations are disseminated through targeted distribution of final reports, articles in refereed journals, and presentations at professional meetings and conferences. Although individual HHS agencies have primary responsibility for disseminating results, the ASPE will continue its Department-wide efforts to expand dissemination of evaluation results to the larger research and practice communities through centralized computer communications and publications like the PIC website and project information database.

Appendix B - Acknowledgements of HHS Officials

Performance Improvement 2003: Evaluation Activities of the U.S. Department of Health and Human Services describes the continuous efforts of the various HHS agencies to examine service and research programs for the efficiency of their operations and their effectiveness in achieving objectives. The following persons from HHS agencies contributed to preparing information on evaluation projects:

 

 

Preparation of this report was managed by the Office of the Assistant Secretary for Planning and Evaluation, Office of Planning and Policy Support. Project Directors: Andrew Rock and Vijaya ChannahSorah. Project Assistant: Anna Sierra. Richard Silva provided technical and other support. Annette Johnson provided clerical support.

Susan Belsinger of Library Associates, Inc. provided database management as well as support in the development and production of the report, and Binnet Shumburo of Sanad Support Technologies and Jana Liebermann of Library Associates, Inc., provided database updating and support.

Appendix C - HHS Evaluation Review Panel

The following individuals served on the evaluation review panel that made recommendations on the reports highlighted in Chapter I of the Performance Improvement 2003: Evaluation Activities of the U.S. Department of Health and Human Services:

 

Appendix D - Program Evaluation Review Criteria

The following core criteria were developed to address four types of evaluation conducted by HHS agencies, offices, and programs:

• Program Effectiveness (PrE) evaluations provide a way to determine the impact of HHS programs on achieving intended goals and objectives.
• Performance Measurement (PfM) projects are intended to assist in the development of data systems for monitoring progress on departmental strategic or agency performance goals.
• Environmental Assessments (EnA) provide a means for understanding the forces of change in the health and human services environment that will influence HHS programs and achievement of its goals and objectives.
• Program Management (PrM) studies reflect the needs of program managers to obtain information or data helpful for designing or managing a program.

The criteria are intended to be generally applicable to each of the four types of evaluations. However, some special considerations are noted for specific elements when applied to particular study types. All criteria are intended as general guidelines; not all elements may apply to every evaluation.

1. OVERALL SIGNIFICANCE
   • The study addresses a significant issue of policy relevance.
   • Evaluation findings are likely to be useful; the study’s usefulness to the stakeholder audience(s) is described.
2. CONCEPTUAL CRITERIA
   • Conceptual Foundations
     • The study is based on either theory, or conceptual models; it builds on previous relevant research, is logically based on previous findings (as appropriate), cites relevant literature, or otherwise justifies the study focus and utility.²
     • The program and/or study assumptions are stated.
     • When the report is linked with a program, policy, or issue, the report describes this context.
     • The timing is appropriate because the program/policy/issue is ready for study; evaluation methods used are appropriate to the program/policy/issue stage.³
   • Study Questions
     • The research questions addressed are clearly stated, measurable, and clearly specified.⁴
     • The questions are feasible, significant, linked to the program or issue, appropriate for the resources and audience, and derive logically from the conceptual foundations.
3. METHODS
   • Evaluation/Study Design
     • Design considerations should include all of the following deemed appropriate: feasibility, funding and time constraints, generalizability⁵, applicability for cultural diversity, assessment of program delivery, validity, feasibility for data collection, reliability of selected measurements, use of multiple measures of key concepts, appropriateness of the sample, and assessment of statistical power prior to data collection.
     • Variables and/or methods used are clearly specified and fit with the questions and concepts.
     • The design matches the study questions; the design permits measurement of program implementation, as appropriate.
     • Multiple methods are used as appropriate and should be well integrated to support triangulation of results.⁶
   • Data Collection
     • Data collection includes: a data collection plan; data collection that is faithful to the plan; attention to and cooperation with the relevant community; data integrity; project confidentiality; ethical handling of data; and consistency.⁷
     • Data are collected appropriate to the evaluation questions, use appropriate units of measurement, and appropriately handle missing data and attrition.⁸
     • The quality of the data (including the quality of extant data sets used in the study), training of interviewers/data collection staff, and justification of the sampling frame are addressed.
   • Data Analysis
     • The data analysis addresses: handling of attrition, matching of the analysis to the design, use of appropriate statistical controls and techniques, use of methodology and levels of measurement appropriate to the type of data, estimation of effect size, and confidence intervals.⁹
     • If multiple methods are used to collect data on the same phenomenon, the data are compared to assess the extent to which the findings are consistent (triangulation).
     • The analysis shows sensitivity to cultural categories, if applicable.
     • The analysis procedures are appropriate and correctly applied.¹⁰
4. INTERPRETATION OF RESULTS
   • Findings
     • The study questions are answered/addressed, or if not, explanation is provided.
     • The interpretation of results is linked to and assesses the study’s conceptual foundation.¹¹
     • The report notes that the findings are either consistent with or deviate from the relevant literature and findings in the field.¹²
     • Program implementation is assessed, as appropriate.¹³
     • Generalizability inferences are addressed.
   • Conclusions
     • The summary does not go beyond what the data will support; conclusions are justified by the analyses; qualifiers are stated as needed.
     • Equivocal findings are addressed appropriately.
     • The results have practical significance.
   • Recommendations
     • The programmatic/policy recommendations follow from findings; are worth carrying out; are affordable, timely, feasible, useful and appropriate; and are relevant to  stakeholders.
     • Any recommendations for future studies, and/or program improvements are clearly presented and justified.¹⁴
5. CROSS-CUTTING FACTORS
   • The following are cross-cutting factors that are likely to be important at all stages of a report: presentation is well written, clear and understandable; use of state-of-the-art approaches where possible; methodology is clearly described and defended; innovative, efficient, logical relationships, and discussion of the report’s limitations. The report presents multiple perspectives, as appropriate; relevant stakeholders are consulted and involved. The report should also address ethical issues, possible perceptual bias, cultural diversity, and gaps in study execution.

---

² This element may be less relevant to Program Management and Performance Measurement studies.

³ For Program Effectiveness studies, timing should be appropriate to implementation stage; for Performance Measurement projects, timing should be appropriate to establishment of strategic goals and objectives; for Environmental Assessments, timing should be appropriate due to changing or uncertain environments; for Program Management studies, timing should be appropriate to management priorities.

⁴ Program Effectiveness studies should specify research questions and testable hypotheses.

⁵ For Performance Measurement and Program Management studies, generalizability is not a key consideration, given study design and foci for these study types.

⁶ This element may be less relevant to Performance Measurement studies.

⁷ Program Effectiveness and Program Management studies that examine person-level outcomes should also address use of an appropriate comparison or control group; and adequate sample size, response rate, and information about the sample.

⁸ Program Effectiveness studies and Environmental Assessments that address person-level outcomes should control for participant selection and assignment bias.

⁹ For Program Management studies that may use other secondary data and methods, data analysis should be appropriate to the kinds of data collected.  Estimation of effect size may not apply to Environmental Assessments, where results may be descriptive.

¹⁰ For Program Effectiveness studies, appropriate sensitivity analyses are conducted, and  uncertainty in key parameter estimates is addressed, as appropriate.

¹¹ This element may not apply to Program Management studies, given their focus and design.

¹² This element may not apply to Program Management studies, which may not have a conceptual foundation in the research literature.

¹³ This element may not apply to Environmental Assessments, which do not directly assess program implementation issues.

¹⁴ Performance Measurement studies should consider the context of the strategic goals and objectives of interest to the study.