Performance Improvement 2001. Program Effectiveness



Reforming Welfare and Rewarding Work: Final Report on the Minnesota FamilyInvestment Program (MFIP).

Volume I: Impact on Adults
Volume II: Impact on Children


This final report on an experimental program in Minnesota that combined financial incentives to work with participation or work requirements for long-term welfare recipients provides insight into four major policy issues. The issues addressed were the following:

  • How can states increase the probability that long-term welfare recipients have a way to support themselves before they reach their time limit on welfare benefits?
  • How should policy makers support the efforts of low-income workers to stay in their jobs and provide for their families in this era of time-limited welfare?
  • How can social policies avoid penalizing marriage?
  • How do the policy changes that states have made in moving their welfare systems from Aid to Families with Dependent Children (AFDC) to Temporary Assistance to Needy Families (TANF) affect families and children?

The evaluation, conducted from April 1994 and March 1996, utilized a random assignment research design to evaluate the MFIP success. More than 14,000 families in seven Minnesota counties were assigned to either the MFIP program or to the traditional AFDC program. The two groups were compared on employment, welfare receipt, and outcome measures related to family life, child well-being and other factors of interest.

Findings indicate single-parent, long-term welfare recipients benefitted most from the MFIP program as evidenced by increases in employment and earnings compared to AFDC participants. However, total costs were higher, since families retained their welfare benefits even as their earnings increased. MFIP families showed improvements in family well-being, as evidenced by parental reports on children’s behavior, a decrease in reported domestic abuse, and a modest increase in marriage rates. For two-parent recipient families, there was an increase in the proportion of parents who stayed married, compared to AFDC, and the program enabled second earners to work less. Results were mixed for recent welfare applicants.


This evaluation was designed to assess the costs and benefits of the MFIP project, a program designed to encourage work and reduce dependence on welfare, while simultaneously reducing poverty. Under the rules of the MFIP, recipients of welfare remained eligible for assistance until their income reached 140 percent of the poverty line. Child care costs for working parents were paid directly to child care providers, eliminating the need for a parent to pay for child care up front. In addition, the rules and procedures related to obtaining welfare benefits were simplified.


A new vision of welfare that encourages “making work pay” for low-income families while reducing their dependence on public assistance has emerged over the past decade. This philosophy is embodied in the Personal Responsibility and Work Opportunities Reconciliation Act (PRWORA), which was enacted in 1996. Traditional welfare programs tend to provide disincentives for single mothers to enter the workplace, as benefits are often better than the income that could be obtained. Two-parent families are required to meet stringent eligibility rules that tend to discourage marriage. At the same time poverty rates for families with children in the U.S. have increased since the mid- 1970s. To meet the requirements of the Personal Responsibility and Work Opportunities Reconciliation Act (PRWORA), states have attempted to make work pay by ensuring the transition to work does not result in a decline into poverty

The MFIP program, which was implemented prior to the passage of the PRWORA, demonstrates the results of implementing two complementary policies that previously have been implemented as single approaches: financial incentives that reward work and reduce poverty, and for long-term welfare recipients, mandatory participation in employment-focused services to reduce long-term welfare dependence.

The MFIP program was first implemented as a field trial in April 1994 in three urban and four rural counties in Minnesota. An interim evaluation of the project was provided in 1997. This final evaluation, presented in two volumes and a summary report, covers the period from 1994 through 1998.


The evaluation design had two components, one directed primarily at the welfare recipients and the other examining the specific effects on children. Between April 1994 and March 1996 over 14,000 families were assigned, using a lottery approach, to the MFIP group or to the traditional AFDC program. A sub-sample of single-mother families with preschool and school-age children who entered the evaluation between April and October 1997 were selected to provide information on impacts of the program on children. MFIP differed from AFDC in three key ways: while in both MFIP and AFDC welfare benefits decreased as earned income rose, the working families in MFIP were allowed to keep more of their monthly financial benefits and to continue to receive these benefits over a longer period of time. Child-care expenses were paid directly to providers, leaving no up-front costs for the parent. MRIP required long-term welfare recipients to participate in employment and training services, while participation in training programs was voluntary under the AFDC program. The MFIP program combined AFDC, Minnesota’s Family General Assistance and Food Stamps into a single monthly payment. In addition, the program rules were simplified, particularly for two-parent families.

There were two sub-groups of MFIP recipients in both the single and two-parent families; those considered long-term welfare recipients (mothers on welfare at least 24 of 36 months prior to random assignment) and recent applicants. Recent applicants were not required to participate in the mandatory employment aspects of the program.

The effects of MFIP were evaluated by comparing outcomes for families randomly assigned to MFIP with outcomes for families randomly assigned to the AFDC system. For the subsample, data on family and child well-being were collected from a survey administered to the families three years after they entered the program. The survey collected information on child and family well-being, including family employment and income; parents’ psychological well-being and parenting practices; and children’s health, behavior and school progress. Data were also collected from administrative case records.


Throughout the 3 year period following their assignment, single-parent long-term recipients (SPLTR) on MFIP had increased employment rates and increased average earnings. These effects were smaller in the rural counties. MFIP increased the number of SPLTR on welfare, but reduced the number relying solely on welfare. MFIP increased incomes and reduced measured poverty. SPLTR MFIP recipients were more likely to be married than AFDC recipients at the end of the 3 year period.

Single parent, recent applicants (SPRA) on MFIP demonstrated modestly increased employment but not increased earnings during the same 3 year period. MFIP actually increased the number of SPRA on welfare, but reduced the number relying solely on welfare. Two-parent long-term recipients (TPLTR ) were less likely than AFDC families to have both parents working, leading to lower family earnings. While MFIP increased the proportion of TPLTR receiving welfare benefits, program participants also showed increased marital stability, income, and home ownership. MFIP had little effect on employment, earnings, or income for two-parent recent applicants (TPRA) families.

For both SPLTR and SPRA the MFIP financial incentives increased employment and earnings, but they also caused some working parents to reduce their hours. Increases in income resulting from financial incentives were responsible for antipoverty effects. The program cost more per family per year than did the AFDC system, because of this increased support for working families.

Compared to financial incentives alone, the complete MFIP program increased full-time work and earnings. SPLTR in the sub-sample also reported improvements in child-well being and declines in domestic abuse. Their children exhibited fewer behavioral problems and did better in school than children of mothers on AFDC. Children were also more likely to be placed in child care, particularly in child care centers, and were more likely to have continuous health coverage. The mothers were more likely to work and had higher incomes. Mothers of these children were also more likely to marry and less likely to experience domestic abuse than recent applicants.

Findings for children of more recent applicants (for whom job preparation services were not mandatory) reported had similar levels of behavioral problems and school progress for their children compared to children of AFDC mothers. Their children were more likely to have had continuous health coverage. The mothers were only slightly more likely to work, did not have higher incomes, and experienced few other changes in their well-being compared to AFDC participants.

In summary, MFIP has been most efficient for SPLTR, the group that is the largest and that was the major target for the program.

Use of Results

The evaluation addresses three goals that have emerged as high priorities under PRWORA: ensuring that long-term welfare recipients make substantial strides toward self-sufficiency before realizing their time limit on welfare receipt, supporting the efforts of low-income workers to advances in their jobs while providing adequately for their families, and assuring that social policies do not discourage marriage. Lessons learned from this evaluation provide important information on types of families for which this approach is most effective; which components of the program; financial incentives, mandatory employment and training B contribute to the effects; and the non-economic effects on both families and children;

AGENCY SPONSOR: Office of Planning, Research and Evaluation and the Office of Child Support Enforcement at the Administration for Children and Families (ACF)


PHONE NUMBER: 202-401-5084

PIC ID: 6824

PERFORMER ORGANIZATION: Manpower Demonstration Research Corporation,
New York, NY

* * * * * *

The Outcome of Outcomes Research at AHCPR: Final Report


This report was prepared for use in discussions of strategic directions for the Outcomes and Effectiveness Research (OER) program at AHCPR, now the Agency for Healthcare Research and Quality (AHRQ). It covers three topics: 1) a conceptual framework for understanding and communicating the impact of OER on health practice and outcomes, 2) a critical analysis of accomplishments and lessons learned over the past decade, and 3) a list of recommendations intended to maintain AHCPR strengths while increasing the measurable impact of future research. It examines issues of how the OER program at AHCPR can most effectively advance the field of health services research (HSR), fulfill its unique role in HSR, contribute to public health, and address the expectations of policymakers and stakeholders. Research findings in OER were classified according to four levels of impact: 1) findings that contribute to an area’s knowledge base and help focus subsequent research (e.g., new analytic methods), 2) research that prompts creation of a new policy or program (e.g., incorporating medical guidelines into practice), 3) a change in what clinicians and patients do, and 4) actual changes in health outcomes.

Conclusions and future directions identified based on these findings suggest that OER must move beyond hypothesis testing and unchannelled intellectual explorations to address knowledge gaps and address areas of application. It suggests that OER should leverage resources through partnerships with HCFA and other public and private organizations, moving from a tool using-culture to a problem-solving culture. Also, OER should develop and refine the “conceptual infrastructure” for conducting effectiveness trials, building on the effectiveness initiatives and PORTS of the 1980s. The study also identifies a need for AHCPR to develop a “dedicated knowledge transfer function” to help engage policymakers and other stakeholders in understanding the links between basic research and clinical practice.


This report is intended both for use within the Agency for Health Care Policy and Research (AHCPR) to guide internal discussions, and for external researchers involved in outcomes and effectiveness research (OER). The key question addressed by the report is how the OER program at AHCPR can most effectively advance the field of health services research, contribute to public health, and address the expectations of policymakers and stakeholders. Through a review of accomplishments within the agency, the report provides lessons learned and recommendations for improvement.


Since program inception in 1989, the AHCPR has had the primary responsibility for implementing theEffectiveness Initiative. The theoretical model underlying effectiveness research in health care is that the retrospective study of patterns of care leads to the definition of optimal treatments; this in turn could lead to substantial economic savings in the subsequent delivery of health care. AHCPR was established as an embodiment of this hypothesis that guidance for optimal medical practice can be obtained from analysis of data that is routinely gathered in the process of delivering and paying for patient care. The intent of this report is to review the evidence that this is a valid approach.


The study is based on a review of articles from grants, original concept papers for launching the “outcomes movement” and critiques of the same, analysis of private sector involvement in OER, a survey of Principal Investigators (PIs) and interviews with selected PIs, interviews with a former the Director of the Center for Outcomes and Effectiveness Research (COER), discussions with the COER staff, and recommendations made by investigators and stakeholders at two expert panel meetings. The PI survey was based on the premise that researchers should have a clear understanding of the impact of their own research. It achieved a 64% responserate, including 61 of 95 PIs.

The impact of AHCPR OER research was categorized into four levels: (1) findings that contribute to a change in practice or policy by adding knowledge or helping to focus subsequent research; (2) research findings that are incorporated into the creation of a new policy or program; (3) an actual change in what clinicians or patients do; and (4) a change in a health outcome. The first item is essentially descriptive research focused on descriptions of what occurs in the health area and in developing tools for measuring costs of care and patient reported outcomes, and identifying topics for future research.


Results suggest that PIs have been most successful in providing descriptions of what actually occurs in health care, developing tools for measuring costs and outcomes, and identifying topics for future research. Other study findings include evidence that OER has increased recognition in the medical community that evidence, not opinion, should guide clinical decision-making; acceptance that a broader range of patient outcomes needs to be measured to understand the benefits of interventions; and the perspective that research priorities should be guided in part by public health needs. The study also found that the most concrete accomplishments in 10-15 years of OER are the tools and methods developed.

Use of Evaluation Results

Although it is increasingly evident that observational designs, such as those that predominate at OER, cannot provide definitive answers regarding clinical effectiveness, these studies do produce information about benefits, risks and costs of health care. Change is dependent upon use of the information by decision makers. Suggested options to enhance the positive impact of the research include:

  • setting research priorities based on likelihood of impact;
  • increasing emphasis on problem-solving by requiring the incorporation of strategies for change as part of the study design;
  • promoting improvement in strategies and methods used to study comparative clinical effectiveness and cost effectiveness;
  • promoting development and use of practical, patient-oriented outcome measures in routine clinical practice;
  • identifying research findings that should be implemented and provide assistance to change agents; and
  • creating a mechanism to document changes in practice that may be related to AHCPR work.

AGENCY SPONSOR: The Center for Outcomes Effectiveness Research at the Agency for Healthcare Research and Quality (AHRQ),

FEDERAL CONTACT: Carolyn M. Clancy

PHONE NUMBER: 301-594-1485

PIC ID: 5979


* * * * * *

Infant Immunization with Acellular Pertussis Vaccines in the United States: Assessment of the first Two Years’ Data From the Vaccine Adverse Event Reporting System (VAERS)


The safety of the new acellular pertussis vaccines was evaluated utilizing the Vaccine Adverse Event Reporting System (VAERS) a passive national surveillance system. Reports of adverse events reported to VAERS in conjunction with an immunization against pertussis (whooping cough) between January 1, 1995 (when the whole-cell vaccine was used exclusively) and June 30, 1998 (when the acellular vaccine predominated) were analyzed. Differences in reported events, categorized according to level of seriousness (fatal, non-fatal serious, less serious), were evaluated for three types of vaccines: Diptheria-tentanus-pertussis (DTP) diptheria-tetanus-whole-cell pertussis-Haemophilus influenzae type b (DPTH) and diptheria-tetanus-acellular (DtaP). The results showed a decline in total adverse events reported in the less serious and non-fatal serious report categories. There was not a concomitant decrease in fatal events; in general these events are rare. The authors conclude that the decrease in the number of less serious reports after vaccination with pertussis-containing vaccines while utilization of the DtaP form of the vaccine was increasing, is consistent with information obtained from clinical trials of the acellular vaccine. In addition, an analysis of the proportional distribution of specific events within each category did not show a substantial increase in any event. No new safety concerns surrounding the administration of acellular pertussis vaccine were identified. .


The purpose of this study was to evaluate the safety of infant immunization with acellular pertussis vaccines in the United States utilizing the VAERS data. This particular type of vaccine has been licensed for infants recently and is intended to replace the whole-cell vaccines. Although pre- licensure studies were conducted, post-licensure surveillance on a larger scale is required to ensure safety.


The development of an acellular pertussis vaccine resulted from concerns about the safety of whole- cell pertussis vaccines that arose during the 1970s and 1980s. The acellular pertussis vaccine was initially developed and licensed for older children in 1991, and was more recently (1996) instituted for infants. Although clinical trials conducted during the pre-licensing phase of drug development provide initial information on safety, sample sizes are necessarily limited. This in turn limits the ability to detect the occurrence of very rare adverse events during prelicensure trials. Post licensure surveillance systems, such as the VAERS, provide the opportunity to review data for millions of doses.

The VAERS was established in 1990 to provide information on adverse events associated with vaccinations. Reporting to VAERS is mandated by the National Childhood Vaccine Injury Act for specific adverse events occurring within specified time intervals after listed vaccines. The VAERS reporting form contains information on demographics, vaccinations(s) administered, description of the adverse event, treatment, and severity. Reporting may be done by physicians, nurses, pharmacists, parents, vaccines, and others. The reports to VAERS do not necessarily represent causal relations between vaccinations and adverse events, although they are temporally related events.

Although passive surveillance systems such as the VAERS have their limitations, they do provide the ability to: detect previously unrecognized adverse events, provide additional clinical insight into recognized adverse events, and assess the safety of newly licensed vaccines


The VAERS data encompassing domestic reports from January 1 1995 through June 1998 were obtained for infants (less than 1 year of age at time of vaccination). Time trends in number of VAERS reports according to severity of adverse event and type of pertussis vaccine were presented. The coding of severity for adverse events was based on definitions from the Code of Federal Regulations: reports of death, non fatal serious events (initial hospitalization, prolongation of hospitalization, life-threatening illness, or permanent disability), and reports of less serious events. In addition, selected reports of adverse events associated with DTP vaccinations were individually reviewed to verify that the reported diagnoses were correctly coded.


Trend analyses showed a sharp decline in the total number of reported serious and less serious adverse events associated with pertussis vaccine administration during the transition from use of whole-cell pertussis vaccine to predominant use of acellular pertussis vaccine. A similar clear-cut decline was not observed for reports of deaths. An analysis of the proportional distributions of adverse events within the three types of vaccines indicated no specific event was occurring at a substantially higher proportional rate for the acellular vaccine versus the other vaccines. Urticaria (hives) was noted in 7.7% of the DTP vaccine, compared to about 3% for both DTP and DHTP vaccines, but this was not associated with anaphylaxis, and may have been due to the proportional decrease in reports of fever, which are proportionally higher for both the DTP and DHTP vaccines.

Use of Evaluation Results

The safety of the acellular vaccine demonstrated by the pre-licensure clinical trials is confirmed by the current surveillance study.

AGENCY SPONSOR: Center for Biologics Evaluation and Research at the Food and Drug Administration (FDA)


PHONE NUMBER: 301-827-3974

PIC ID: 7251

PERFORMER ORGANIZATION: Center for Biologics Evaluation and Research at the Food and Drug Administration (FDA), Rockville, MD and the National Immunization Program at the Centers for Disease Control and Prevention (CDC), Atlanta, GA

* * * * * *

National Health Service Corps


The National Health Service Corps (NHSC) is a program intended to improve access to health care in underserved areas. The program is conducted under the auspices of the Bureau of Primary Health Care within the Health Resources and Services Administration (HRSA). This evaluation of the NHSC examined the factors that are hypothesized to lead to clinician retention in underserved communities. The study consisted of in-depth surveys of three groups: NHSC alumni clinicians, current clinicians, and NHSC site administrators.

The NHSC program provides either scholarships to currently enrolled students or a loan repayment program for those who recently completed their training. Providers in both programs are then obligated to practice in a high-priority health professional shortage area, or in the case of the loan repayment program, eligible facilities from the Loan Repayment Vacancy List. The agency asserts that many of their providers, in addition to caring for patients, contribute in other ways to medical care at their host sites. This assertion has not been previously studied and became a major focus of this evaluation.

Questionnaires were sent to a stratified sample of 2,160 alumni physicians, advanced practice nurses, dentists, and physicians assistants, who have participated in the program over the past twenty years, as well as to 1,143 currently serving providers. Seventy five percent of current providers and 59% of alumni responded to the survey. (Sixty-two percent of NHSC site administrators also responded to a mailed survey.) Descriptive statistics, univariate, and multi- variate analyses of the data were presented. The predictors included program characteristics, individual provider background characteristics, and self-reported attitudes. Findings were many and complex, given the multiple outcome measures, but generally supported the conclusion that participants in the loan repayment program exhibited better outcomes than did providers in the scholarship program. Other programmatic, attitudinal and background characteristics such as gender, discipline, type of site, site match with provider choice and motivation for entering the program were also associated with various outcomes. Survival analysis using Cox regression models were employed to investigate retention over time.

Main findings were that satisfaction with the program varied according to numerous demographic factors as well as with the type of program (scholarship vs loan repayment) the provider was enrolled in. Providers, primarily physicians, in rural communities have had an impact on their communities in areas other than direct patient care.


The purpose of the current evaluation was review the NHSC program in 3 areas: assess the extent to which the NHSC is fulfilling its mission, improving clinical practice standards in underserved areas, and contributing to the community in other areas; identify factors that influence the quality of service and retention; and assess the impacts of changes in program objectives and expectations on development of the NHSC.


The NHSC was established in 1971 and is part of a larger Federal effort designed to improve the delivery of health services to persons in currently underserved areas. Primary health care professionals, including physicians, dentist, nurse practitioners, physician assistants, certified nurse midwives, and others, are recruited and placed in underserved areas through two methods: 1) scholarship programs, which require assignment to a specific area for the length of service equal to the scholarship term; and 2) loan repayment, which requires assignment to a wider variety of sites while NHSC assists with repayment of loans incurred during clinical training. Although the mission of the program has changed only slightly, the approach and levels of funding available to the program have varied greatly over the 25 years since inception. As of August 1999, when the study was conducted, there were approximately 1,863 NHSC clinicians in the field; 70% were federal loan repayment participants and 30% were scholarship recipients. More than half of the clinicians in the field at that time period were physicians (51%), 14% were dentists, 14% were physician assistants, 11% were nurse practitioners, and 3% were nurse midwifes. The remaining 7% were in disciplines considered to be out of the scope of the current evaluation (mental health workers, marriage therapists, etc.). Previous evaluations have focused primarily on physicians, rural areas, and scholarship programs. The current effort broadens the scope of the program evaluation.


Information was obtained via mail surveys of NHSC current and former clinicians, and NHSC sites administrator. The surveys were self-administered and contained both structured and open-ended questions. The objectives of the surveys were to assess clinician impact on the sites and communities, and to assess factors that influence both retention at the site and subsequent career objectives of the NHSC clinician. The clinician surveys evaluated experience before NHSC service; NHSC service; service impacts; satisfaction with practice setting; education and employment history; and socio-demographic characteristics. The site administration survey covered the site historical involvement with the NHSC program; site clinical services; compensation approach; and NHSC recruitment and retention record.


Response rates for survey returns were 75.2% for current clinicians, 58.9 for alumni clinician, and 62.1% for site administrators. Important motivations for joining the corps were financial need and a desire to work with underserved populations based on altruistic motivation (as measured by a composite of three questions). Overall satisfaction levels among alumni and current clinicians are not high, and were found to vary by such factors as discipline, type of community, gender, age at start of service, and ethnicity. Selected findings are: Scholarship program participants are less satisfied than loan repayment clinicians, possibly due to less choice in sites for work effort. Non- physician primary care workers are more satisfied with both compensation and job situation than are physicians. Rurally located clinicians are less satisfied than those at other sites; female clinicians and their families report lower satisfaction compared to males, as do younger participants.

The study also revealed that NHSC clinicians have had an impact on their communities in areas other than direct patient care. These include accessing new patient populations; increasing and adding services, initiating community-oriented primary care programs, participating in quality assurance activities, and engaging in leadership activities. This is more true for those clinicians in rural communities than those in urban areas.

Use of Evaluation Results

Results can be utilized to assist the NHSC program select applicants who are well suited to carry out the mission of the corps. The selection and recruitment process can be improved by matching NHSC clinicians to communities that provide a good fit, leading to higher performance and increased retention.

AGENCY SPONSOR: Bureau of Primary Health Care at the Health Resources and Services Administration (HRSA)


PHONE NUMBER: 301-594-4204

PIC ID: 6357

PERFORMER ORGANIZATION: Mathematica Policy Research, Inc., Washington, DC