Policy Information Center Highlights: Vol. 8, No. 1




Evaluation of AHCPR Minority Health Services Research

In 1978, the American Association of Medical Colleges (AAMC) established a goal of “population parity” for minority individuals in medicine and health sciences. AAMC’s goal was to see the number of minority individuals practicing medicine reflect the representation of these minorities in the community-at-large. Since then, many organizations have committed themselves to increasing the overall numbers of minority individuals in health services research. The largest source of Federal funding for health services research and education is the Agency for Health Care Policy and Research (AHCPR). AHCPR sponsors research and training programs in health services research through National Research Service Award (NRSA) training grants to 22 institutions in 15 States. These training grants promote the involvement of minority researchers in the health services research community. AHCPR began the Minority Research Supplement Program to attract and support underrepresented minority individuals at each of the undergraduate, graduate, and investigator levels. Additionally, AHCPR wanted to encourage health research on issues affecting minorities. Through the Minority Supplements Program, AHCPR makes supplemental funds available to grantees to increase the involvement of minority health professionals in ongoing health services research and to emphasize research on issues affecting minorities and ensure that the resultant research information is made available to the appropriate audiences. Several non-Federal programs have also made commitments to building the supply of minority health services researchers. Until this report, there had not been a systematic external review and synthesis of factors that are critical to the success of these programs.

The report, Evaluation of AHCPR Minority Health Services Research Training Activities, qualitatively reviews AHCPR initiatives and examines their impact on increasing minority participation in health services research. In addition to evaluating the success of the initiatives, the study seeks to identify “best practices” among several non-Federal programs. The report examines two types of 1. initiatives: (1) predoctoral and postdoctoral training provided through NRSA grants; and (2) supplementary funding for individual AHCPR grantees to provide training to individual minority candidates or for individuals investigating an area related to minority issues in health services.

Telephone discussions and informal interviews with program directors, faculty, and trainees in a sample of NRSA programs, and recipients of AHCPR supplemental grants were undertaken for this report. A convenience sample was used and information was gathered from 9 of the 22 institutions and 13 principal investigators holding AHCPR supplemental grants. Further information was obtained from a literature review on minority training in the health professions and interviews with individuals from three comparable minority training programs.

The report finds that there has been an increase in the number of minority researchers pursuing health services research. In 1986, there were 7 AHCPR NRSA institutional grants, and by 1994, the number of NRSA programs had increased to 22. The number of NRSA traineeships grew from 33 in 1986 to 114 in 1994. Funding for NRSA programs and supplements increased from $1.1 million in 1986 to $4.3 million in 1994. Since 1993, program directors have been required to gather information on the race and ethnicity of trainees. In 1993, 18 percent of the trainees were minorities, and in 1994, 21 percent were minorities.

However, there was no clear consensus of how successful individual programs considered their efforts to be, since some programs did not currently have even a single minority trainee. Others had two or three trainees, among a total of six or seven. According to participants, the most significant factor in assessing Agency success in this respect, is that AHCPR funding was essential to each of the trainees who were funded. Without AHCPR funding, these trainees could not have afforded to pursue the study of health services research.

The report recognizes that an increase in minority investigators in health services research alone is not a sufficient indicator of program success. General workforce literature and a recent evaluation of Pew Health Policy Programs identify five factors that “work in unison to generate success”. The five factors are: recruitment, leadership, networking, mentoring, and building a community of scholars. The degree to which the activities that support these factors are in place is a far more comprehensive measure of success than numbers alone. While the number of graduates may not alone prove “success” the program can consider itself “successful” if all of these factors are in place. The report evaluates both the NRSA programs and Minority Supplemental Grants in terms of their efforts in these areas.

The American Nurses Association Health Services Research Fellowship, the Association of American Medical Colleges Minority Junior Faculty Program, and the Commonwealth Fund/Harvard University Minority Faculty Development Program outline the “best practices” of their minority training programs. According to these programs, the following practices are important: (1) developing sufficient infrastructure to support training; (2) providing sufficient financial support to attract the brightest trainees; and (3) providing necessary support for faculty mentors involved in the program.

This initial evaluation provides a preliminary set of criteria for determining the success of these programs and concludes with a new set of research questions:

  • Are minority principal investigators funded as frequently and at the same levels as comparable majority investigators on comparable minority projects, given their relative representation in the health services community? If not, why not?
  • Are minority investigators mainly pursuing minority-related research?
  • What roles are minority investigators typically performing in research projects?

The report recommends a comprehensive and longitudinal analysis of minority health services researchers, to determine both the short and longer-term impacts of the AHCPR minority training initiatives.

The report was sponsored by the Agency for Health Care Policy and Research. It was prepared by Barents Group LLC. The report’s project officer, Dr. Morgan Jackson, may be reached at 301-594-1406, ext. 1477. Copies of the executive summary of this report, PIC ID No. 6384, are available from the Policy Information Center.

FDA Product Review Performance

The Prescription Drug User Act of 1992 (PDUFA) provided new time frames for the approval of new human drugs and biologics and provided additional funding to the Food and Drug Administration (FDA), through user fees, for review of these products. In response to PDUFA and requirements under the Federal Food, Drug and Cosmetic Act, the FDA has developed clear performance goals for reviewing product applications. Setting these goals has proven valuable for identifying performance expectations, assessing achievements, and for meeting the requirements of the Government Performance and Results Act of 1993 (GPRA). The FDA has also developed performance measures and reporting conventions that can be used to report product review performance across product categories.

Product Review Performance Report: Food and Drug Administration, is the first report analyzing the FDA’s performance in reviewing applications for human drugs, biological products, medical devices, food and color additives, and veterinary drugs. In the past, the FDA quantified the length of product review using measures that varied from product category to product category, making it difficult to provide an Agencywide assessment of the FDA’s performance. Published in July 1997, this report tracks the FDA’s performance in meeting performance goals for the review of product applications, established by statute, internally or by agreement with the regulated industry; and provides the public with an assessment of the Agency’s performance using measures that are consistent across the Agency.

The report examines four measures of the FDA’s performance: (1) how often the FDA is meeting statutory and nonstatutory time frames for review of each category of products (on-time performance); (2) the median length of time to approval for each product category (approval times); (3) the total number of products approved in each category (applications approved); and (4) the number of applications for which an action was overdue at the end of the fiscal year (overdue applications). The report provides the Agency’s on-time performance, approval times, and the number of applications approved and overdue, for fiscal years (FY) 1993, 1994, and 1995.

The FDA has four categories of applications: original applications; 510 (k) applications; supplemental applications and generic drug applications. Original applications are submitted for new products or for new uses of food and color additives; 501(k) applications are submitted to establish medical devices as “substantially equivalent” to devices already on the market; supplemental applications are submitted for changes in an already approved product, such as changes in the way a product is manufactured, or a new indication; and generic drug applications are submitted to establish that a firm can manufacture a copy of a brand name drug.

The report finds that in most product categories, the FDA has improved its performance since FY 1993. On-time performance improved for 11 of 15 categories of product reviews, median approval times decreased for 11 of 15 categories, the number of overdue applications decreased for 13 of 15 categories, and the total number of applications reviewed and improved increased in every category.

The greatest improvements in performance have occurred for product reviews covered by PDUFA. PDUFA has provided additional resources that have benefited the review processes for new human drugs and biologics. The performance improvement in product reviews covered by PDUFA shows that the combination of realistic time frames, management improvements, and additional resources is an effective means of improving the FDA’s performance in reviewing new products.

Significant progress has been made in reviewing applications for 501(k)’s for medical devices. On-time performance has increased from 42 percent in FY 1993 to 94 percent in FY 1996, median approval times have steadily declined, and the number of overdue applications has dropped to zero.

The report concludes that reviews of non-PDUFA applications need to improve their on-time performance. Premarket approval applications for medical devices and for food and color additive petitions improved only slightly in their on-time performance. For new animal drug applications and for abbreviated new drug applications the on-time performance declined.

This study was sponsored by the Food and Drug Administration’s Office of Planning and Evaluation. For more information contact William Hagan at 301-827-5212. Copies of the report, PIC ID No. 6731, are available from the Policy Information Center.

Health Promotion in Elder Care

As the American population ages, the health and safety of the aging population presents new concerns to the elderly and their families. Public health agencies are able to provide support to families and providers as they seek ways to maintain the health and well-being of the elderly in society.

The report, Disease and Injury Prevention and Health Promotion in Elder Care: Needs and Opportunities as Perceived by Elder Care Providers, the Elderly, and Their Families, identifies interests of the elderly, their families, and elder care providers, in selected areas of health and safety, to inform the development of a handbook delineating the Centers for Disease Control and Prevention (CDC) policies and recommendations in disease prevention and health promotion in elder care settings.

Eight focus groups were conducted in Atlanta, Georgia, and in Floyd County, a rural area in the northwest part of the State. Focus group participants were the elderly, their family members, and elder care providers who either live independently or care for elderly relatives or clients who are still ambulatory provide the data for this report.

Key areas addressed by the study include: (1) topics of interest in preventing disease and injury and promoting healthy behaviors; (2) the role of public health agencies in encouraging health and safety through consultation, inspection, regulation, and similar mechanisms; (3) opportunities to educate the elderly, their families, and elder care providers in disease prevention and health promotion; and (4) opportunities for direct delivery of prevention and public health services through elder care settings, especially for the elderly in medically undeserved areas of the population.

The report concludes that the elderly, their families, and health care providers are eager for current, practical information on health and safety. Currently, there is no single resource that provides this kind of information or support. The study recommends that a modular format could be developed, with common sections on basic health and safety information and specific sections geared to the needs of the elderly and their caregivers. While no single document could address the variety and scope of this task, a compendium could address major categories and suggest other organizations that could provide support.

Participants in the focus groups unanimously suggested that the format of such a document should contain large print, easily accessible information, and clear graphics. Classes on health and safety information were well-received by only those focus group participants caring for the elderly, not by the elderly themselves. Prevention services, in particular public health services such as flu shots, pneumonia shots, and TB skin tests were considered worthwhile and important to provide the elderly. Screening services, such as blood pressure, cholesterol, and diabetes were considered by most focus group participants as most appropriately and adequately provided by private physicians. However, providing these services to the disadvantaged elderly and those without transportation is an important service.

The lack of awareness about public health’s potential usefulness as a source of information was an unfortunate theme in the focus groups. In general, the public’s lack of awareness about public health agencies seems to be the result of limited exposure, and not the result of negative experiences. Any effort to consolidate useful, relevant information on the health and safety of the elderly represents an opportunity for the CDC, and its partners at the State and local levels, to raise awareness of the ongoing role of public health.

This report was sponsored by the Centers for Disease Control and Prevention, National Center for Infectious Diseases. The report’s project officer, Cyndy Hale, may be reached at 404-639-6477. Copies of the executive summary of the report, PIC ID No. 6708, may be obtained from the Policy Information Center.

Evaluating Welfare Reform

The Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996 eliminated the Aid to Families with Dependent Children (AFDC) entitlement program and replaced it with block grants to the States. States now design, implement, and administer their own programs. Before PRWORA, many States obtained waivers from AFDC and Food Stamp Program rules in order to implement welfare reform. Between 1993 and 1996, the Clinton administration approved waivers for 43 States.

The report, Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations, compiles information about the approaches used in the evaluation of State welfare reforms undertaken as part of the waiver process. This report identifies the principle research issues surrounding the design and execution of waiver evaluations; assesses the appropriateness of various technical approaches to those issues; and documents those issues and approaches so they can be used in designing and conducting welfare evaluations outside of the Federal waiver context.

This project examines waiver demonstration projects in five States: (1) the California Assistance Payments and Work Pays Demonstration Projects; (2) the Colorado Personal Responsibility and Employment Program; (3) To Strengthen Michigan Families; (4) the Minnesota Family Investment Program; and (5) the Wisconsin Work not Welfare demonstration.

The report reviews and presents recommendations concerning selected issues in five areas: the choice between an experimental and quasi-experimental evaluation design; sample design; implementation of experimental evaluations; data collection; and analysis methods.

The report finds that: (1) most evaluations of State welfare reform demonstrations used an experimental design which involved random assignment of cases to an experimental group subject to welfare reform or to a control group subject to pre-reform policies; (2) only one waiver (Wisconsin’s) was approved with a quasi-experimental design, but this may become more common in a block grant environment; and (3) when designing a study sample, evaluation planners should ensure that the sample size is adequate and should design the sample so that applicant and recipient subgroups can support separate impact estimates; the sample should also be representative of the State as a whole. Four aspects of the implementation of an experimental evaluation require special care. These aspects are: the timing of the random assignment; the method of the random assignment; ensuring that control group policies remain unchanged; and preventing experimental and control group cases from changing status.

This report was sponsored by the Office of the Assistant Secretary for Planning and Evaluation. The report’s project officer, Audrey Mirsky-Ashby, may be reached at 202-690-7148. Copies of the executive summary, PIC ID No. 6050, are available from the Policy Information Center.

Income and Wealth of the Elderly: Implications for Long-Term Care

Debate surrounding the Federal role in financing long term care (LTC) facilities centers on the economic status of the elderly (those 65 years of age and older). A key issue is whether the elderly can afford to pay for their own long-term care costs directly from personal resources or through the purchase of private LTC insurance. If many elderly persons can afford to pay for it privately then many analysts would argue that there is less need for government involvement.

The report, Estimates of the Income and Wealth of the Elderly Using the Panel Study of Income Dynamics and the Survey of Consumer Finances and Their Implications for Long-Term Care, is a two-part study of the income and wealth of the elderly. The first part of the study was a detailed literature review. As part of the review, the study identified gaps in the current empirical literature investigating the wealth and income of the elderly. The second part of the study used two recent data sources, the 1983 and 1992 Survey of Consumer Finances (SCF), and the 1983 to 1991 Panel Study of Income Dynamics (PSID) to provide updated estimates of the income and wealth of the elderly. This report presents the results of the second part of the study, which is divided into three chapters.

Chapter One is an overview of the literature on the economic status of the elderly. The review examines both the empirical and the theoretical literature. Most empirical studies of the elderly conclude that, on average, the elderly’s economic status (i.e., their income and wealth) is high in comparison with other age groups, previous elderly cohorts, or to the current elderly’s status at some point in the past. Generally, the elderly are found to have smaller incomes than the non-elderly or than the elderly themselves had prior to retirement. However, the elderly often need to consume less and usually have higher levels of assets than the non-elderly which helps them maintain their standard of living. The elderly are at greater risk than the non-elderly to live in poverty and face economic risks that are not faced by other age groups.

The theoretical literature contains three primary models that explain the current wealth of the elderly. Life cycle models assume the elderly hold wealth to increase the level of consumption they can maintain. The model implies that individuals smooth their consumption through time in order to maintain a constant standard of living. There are two other elderly savings models that help to explain why the elderly might choose not to consume their savings. Precautionary models recognize that the elderly face certain risks that may affect their current consumption. These risks include death and catastrophic illness. The bequest motive models assume that the elderly do not consume their savings because they want to bequest money when they die.

Chapter Two of the report provides estimates of income and assets of the elderly. Much of the previous information on the income and wealth of the elderly is outdated. This study produces a series of new estimates of elderly income and wealth measures using two recent data sets that follow individuals through time: the SCF and PSID. The research indicates that as the elderly and near-elderly age, their incomes decline. For example, the median household income for those age 75 and above in 1989 (PSID) was $11,500, much less than median incomes for those age 65 to 74 ($17,500) or those age 55 to 64 ($29,900). Additionally, elderly married couples had incomes that were 2.5 times greater than the nonmarried elderly, and black households had median incomes that were approximately half that of nonblack households.

The longitudinal estimates of wealth and assets for both the PSID and SCF were generally consistent with the life cycle theory. There was also a strong correlation between income and assets in SCF and PSID. Families with higher household incomes had much greater total net asset holdings. The detailed income and asset data in both the PSID and SCF allows the study to estimate that 15 percent of elderly households in 1983 and 1989 could have afforded to purchase a LTC insurance policy that costs $1,500 to $2,000. For a household, the study defines a LTC insurance policy as “affordable” if the cost of the policy is less than 5 percent of the household’s income.

Chapter Three reports the results of a simulation model that predicts changes in elderly assets using data from PSID. The model is a life cycle model that incorporates a precautionary motive for saving. The modeling effort shows the underlying heterogeneity in the savings and consumption patterns of the elderly. The report concludes that a much broader research effort is required to forecast changes in elderly assets.

The recommendations for future research include: (1) incorporating a target savings motive; (2) including decedents; (3) allowing the consumption and savings decisions of married couples to differ from those of the single elderly; (4) allowing key model parameters to vary across households and/or through time, and (5) including the nonelderly directly in the model.

This report was sponsored by the Health Care Financing Administration and was prepared by Lewin-VHI, Inc. The report’s project officer, Carolyn Rimes, may be reached at 410-786-6620. Copies of the executive summary, PIC ID No. 6613, are available from the Policy Information Center.

Results-Based Systems For Public Health Programs

In 1993, the Vice President’s report on National Performance Review intensified interest in enhancing the performance and accountability of government programs. Since then, policymakers have been contemplating changes in Federal grant making and oversight authorities. The Department of Health and Human Services (HHS) has been considering performance partnership grants for some of its public health programs for several years. These arrangements would combine various categorical grants into performance partnerships and would reshape several of HHS’s block grant programs. The Office of the Assistant Secretary for Planning and Evaluation asked the Office of the Inspector General (OIG) to identify and examine State initiatives that use outcome measures to assess the performance of their public health programs

The report, Results-Based Systems for Public Health Programs: Volume One: Lessons From State Initiatives, reviews results-based initiatives in the preventive health, maternal and child health, substance abuse, and mental health programs of 11 States. The OIG conducted onsite interviews in seven of these States: Florida, Illinois, Massachusetts, Nebraska, New York, North Carolina, and Washington. The OIG also conducted telephone interviews with officials in the four other States: Georgia, Minnesota, Ohio, and Oregon. The companion volume contains a brief description of these States’ initiatives.

The report finds that results-based accountability initiatives were generally of two types: (1) broad efforts at statewide strategic planning and priority setting; and (2) systems focused on targeted populations and specific program interventions. The States, in initiating their results-based systems, have several characteristics in common: public pressure for better government, top-level commitment, and extensive stakeholder involvement. State officials see many benefits from development and implementation of results-based systems and are using data in various ways.

States face several significant challenges in their attempt to develop and implement results-based systems, including issues with the measures themselves, data concerns, the use of the results, and system capacities.

Challenges also face HHS as it considers future directions for its performance partnership grant initiative. These challenges include: ensuring that the performance partnership approach will be integrated with the State’s own performance management efforts; determining what kind of administrative infrastructure can best support the interagency partnerships; and finding the best way to support an information exchange among Federal and State agencies and between the research and academic communities.

The report documents a number of benefits seen by State officials in their efforts to develop results-based systems. These benefits include: (1) obtaining information efficiently, which can be used to improve program performance; (2) empirically demonstrating program results; and (3) obtaining a consensus among stakeholders on program missions and goals. Ultimately, the biggest determinant of whether these systems succeed, may be their usefulness to those most vested in the programs: legislators, administrators, providers, advocates, and consumers. The challenge for these stakeholders will be to strike an appropriate balance when using data from these systems to enhance decisionmaking about program improvements, budgets and contracts, and feedback to providers and consumers.

The report was sponsored by the Office of the Inspector General and was prepared by the OIG with support from Management Evaluation Training, LLC. The report’s project officer, Suzanne Johnson, may be reached at 312-353-9867. Copies of the report, PIC ID No. 6672, are available from the Policy Information Center.


The Policy Information Center (PIC) is a centralized source of information on in-process, completed, and on-going evaluations; short-term evaluative research and; policy-oriented projects conducted by HHS as well as other Federal departments and agencies. The PIC on-line database provides project descriptions of these studies. It is available on-line at http://aspe.hhs.gov/pic/. Inquiries regarding PIC services should be directed to Juanita Butler, Technical Information Specialist, at 202-690-6445. Or E-mail PIC at: webmaster.aspe@hhs.gov.

Return to Index