MISSION: Improve the health of the Nation by assuring quality health care to underserved and vulnerable populations and by promoting primary care education and practice.
HRSA Evaluation Program
The purposes of the HRSA evaluation program are to enhance strategic planning, budget decisions, and legislative planning, and to improve program management. Consequently, major emphases during fiscal 1996 and beyond will be performance measurement and assessment of program implementation and policy.
Performance measurement includes (1) technical and training activities to strengthen HRSA's capacity to assess program performance, and (2) studies to assess the outcomes of individual programs or groups of programs. Projects to enhance measurement capacity have assumed greater emphasis in relation to the Government Performance and Results Act. During fiscal 1995, HRSA established a performance measurement baseline for all operating programs; the ultimate objective is to ensure that HRSA establishes valid and useful indicators and measures for all programs by the end of fiscal 1998. Over the past year, HRSA has made significant progress in performance management that provides operational linkages between strategic planning, program activity, and the budget process--as intended by GPRA.
Assessment of program implementation and policy includes a broad range of efforts to assist HRSA during a time of significant change caused by new policy directions initiated by the Administration or Congress. Specifically, HRSA-funded programs are entering into new arrangements for delivering services, providing health professions education, and encouraging the development of systems reform within seven program priority areas. These priorities reflect HRSA's focus on underserved populations. They include academic and community partnerships to foster clinical training in community-based settings; new arrangements to bring poor, uninsured, rural, and chronically ill persons into the mainstream of managed care; assistance to communities in strengthening their health care infrastructure; and new activities with States. HRSA's studies relate to one or more of these priorities and include efforts to describe and assess the initial or later implementation of a program; compare alternative approaches to delivering services; assess the benefits of a current or potential policy; examine the effectiveness or efficiency of resources management; and conduct evaluation syntheses.
The objective of HRSA's evaluation work is to provide useful and timely information to the Administrator, the four bureaus, and the Office of Rural Health Policy. To ensure that this objective is met, all study proposals are reviewed first by a committee composed of the bureau directors and four other senior executives who consider the proposals' relevance to important policy, budgetary, or legislative issues; potential to answer questions about program effectiveness or impact; and degree of attention to crosscutting topics. This committee makes recommendations to the Administrator about study approval and in so doing establishes relative priorities for funding. Subsequently, expanded materials needed for developing contracts for approved studies are reviewed by a committee of senior analysts from HRSA, the Agency for Health Care Policy and Research (AHCPR), the National Center for Health Statistics (NCHS), and the Office of the Assistant Secretary for Planning and Evaluation (ASPE). The role of the latter group is to offer suggestions, as needed, on ways to strengthen study design and/or data sources.
HRSA places major emphasis on the effective dissemination of findings and information about the use of completed studies. Dissemination is done in a number of ways and is especially designed to ensure that evaluation information promptly reaches the Administrator and other agency executives. In addition, attention is given to a broader dissemination through articles in professional literature and presentations at professional conferences. An annual report on completed studies describes purpose, findings, and uses; a second annual volume provides brief summaries of studies initiated the preceding fiscal year.
Summary of Fiscal 1995 HRSA Evaluations
HRSA completed an agencywide review of performance measurement capacity for all line programs last year through a project that is highlighted in chapter II, "Performance Indicators for GPRA: Initial Assessment of HRSA Programs." This work is the foundation for current component-specific efforts and for developing the fiscal 1998 budget submission. Three projects reflect differing aspects of health professions work force issues. A study of user satisfaction found that information from the National Practitioner Data Bank was clearly important in identifying practitioners who had not voluntarily revealed their "problem records" to hiring and certifying authorities. HRSA has used the study to develop the improved data bank system implemented in June 1995 and to strengthen the reporting process. Another health professions study, "Development of Integrated Requirements for PAs, NPs, CNMs, and Physicians (MDs and DOs)," has produced a model for estimating the national demands for primary care practitioners by year through 2020. HRSA advisory groups (the Council on Graduate Medical Education and the National Advisory Council on Nursing Education and Practice) are using the model to form recommendations for data and education program development. Another external user is the Utah State Health Department, which is adapting the model to project State-level needs.
A third example is a study that describes factors influencing decisions about the numbers and types of primary care personnel used by health maintenance organizations. The growth of managed care is altering the ways in which members of the health care work force are used and thus is modifying the demand for, and the costs of, various groups of health care professionals. Understanding how primary care staffing decisions are made in managed care organizations will improve health professions' forecasts by helping determine how to adjust to the influence of managed care. The study also investigates ways in which managers alter their primary care staffing patterns to accommodate the needs of Medicaid enrollees. It has given an initial insight into changes in the demand for primary care personnel that will accelerate as more States move into managed Medicaid.
Major priority is placed on improving access to high-quality services for prevention and treatment of HIV/AIDS. A study of Ryan White CARE Act Title I assessed grantee strategies used to reach African-Americans not in care in four metropolitan areas. Among the strategies described in the report are establishing and maintaining broad planning council representation from all populations affected by epidemic; enhancing the capacity of community-based agencies to successfully compete for local funding; adapting case management systems to changing priorities; recruiting African-American physicians as clinicians in community-oriented primary care; and building community awareness and participation in planning through African-American institutions, such as communities of faith. Two key findings of the report were that the methods used here also could be used to assess strategies for other underserved populations and that developing effective strategies requires the grantee, the planning body, and constituents of services to recognize that each acts on distinctive underlying values and principles.
Another study, "Impact of Ryan White CARE Act Title I on Capacity Building in Latino Community-Based Organizations," developed and pilot tested a methodology for evaluating whether Title I funds have influenced the development, expansion, or enhancement of HIV/AIDS services in San Diego and Boston. (This methodology also can be used to study capacity building in organizations serving other populations.)
A careful analysis was conducted of the cost-effectiveness of the Community Health Centers (CHCs) Program. It showed that AFDC recipients who used a CHC as their main source of primary care incurred lower Medicaid costs and used fewer hospital days than other recipients living in the service areas of selected CHCs in California and New York. Study findings, which reflect only care reimbursed on a fee-for-service basis, have been used in presentations to the Office of Personnel Management and Congress. Building on this study, HRSA has contracted for a national evaluation of the effectiveness and impact of CHCs, including examination of the experience of centers involved in managed care.
Representatives of the Substance Abuse and Mental Health Services Administration (SAMHSA) participated in planning and oversight of an additional CHC study, "Community and Migrant Health Centers and the Assessment of and Response to Mental Health and Developmental Needs in Primary Care Patients." The final report has been distributed to State primary care organizations (which administer the primary care cooperative agreements with States) and to primary care associations for use in advising member organizations on improving their mental health services.
Essential to achieving more effective program evaluation is the production of useful data. To help produce this data, a third primary care project developed a Uniform Data System (UDS) covering five programs, including Community and Migrant Health Centers and Health Care for the Homeless, and a user manual for the UDS. This new system provides uniformly defined data for related programs and eliminates duplication and inconsistency in reporting, thus reducing grantee burden.
Also in the services area, a maternal and child health study assessed the outcomes of grants that were intended to foster creation of a permanent infrastructure for child and adolescent injury prevention in seven States. This study, which analyzed and compared the State strategies, found that recipients in general exhibited a considerable increase in injury prevention capacity several years after the expiration of the special incentive grant funding. Results will be used by policymakers in considering future funding for injury prevention and will influence the design of such efforts.
HRSA Evaluations in Progress
Ongoing studies include a range of projects concerned with enhancing performance measurement capacity and assessing the effectiveness or implementation of specific programs. Illustrative studies in particular topic areas follow.
In the AIDS area, HRSA is continuing a longitudinal examination of the effects of Ryan White CARE Act Title I funding on services for active or recovering drug users with HIV. Staff of the National Institute of Drug Abuse are collaborating in the design and oversight of this study. Another AIDS-related project provides a synthesis of local evaluations sponsored by Ryan White grantees to facilitate dissemination nationwide of findings and experience with various methodologies as applied to locally identified issues.
A study concerning organ transplantation, titled "Reasons African-American and White Waiting List Patients Are Unavailable for an Organ Offer," reflects HRSA's concern with cultural competence in health service delivery. An Inspector General's report prepared in 1991 showed that African-American patients on the waiting list for a kidney transplant waited nearly twice as long as did Caucasian patients. Only part of this difference can be explained by biological and medical factors. HRSA, through the Organ Procurement and Transplantation Network, is responsible for ensuring that the U.S. organ allocation system operates equitably. This current study will help determine whether changes in Federal policy governing the organ procurement system are needed.
Continued major attention is directed toward community-based service programs. "Effectiveness of the National Health Service Corps" is a 3-year study to assess the Corps' performance by using such indicators as satisfaction of communities where Corps members are working; increases in numbers of people served because of placement of a Corps member in a site; and long-term retention of former Corps assignees in a primary care or related profession.
As noted, HRSA is conducting a national study of the effectiveness and impact of Community and Migrant Health Centers through a sample of 50 centers in 10 States. The data will be analyzed separately for users served under managed care arrangements. The "Community Health Center User and Visit Survey" involves interviews with 2,000 users and a review of 3,000 medical records for visits at the same centers to provide information about the demographics of CHC users, their reasons for seeking care, their diagnoses, services used, and outcomes of care. Another study is assessing the impact of Medicaid waivers on HRSA-funded, federally qualified health centers in States that have instituted mandatory managed care for Medicaid beneficiaries.
In addition, "The Future of Primary Care" was intended to define the place of primary care in the changing health care environment and to develop normative goals for primary care. Results will be used to assess the adequacy of the primary care system in the United States and strategies needed to influence the supply and distribution of primary care providers.
An emerging policy issue concerns the management of the J-1 visa program, a matter that cuts across health professions and primary care issues, therefore calling for a collaborative effort with the Educational Commission for Foreign Medical Graduates and the Appalachian Regional Commission. This project, "Tracking of J-1 Visa Exchange Students," is examining the postresidency experience of exchange students who have secured waivers (to the requirement to return home for 2 years following completion of residency) to remain in the United States. Experience of those remaining will be compared with experience of those who return to their home country. Results, expected in 1997, will provide information for guiding U.S. policy about such waivers for physicians.
The "National Evaluation of the Healthy Start Program" continues as a 5-year longitudinal study of the development, implementation, and outcomes of comprehensive, coordinated perinatal care systems in the initial 15 Healthy Start demonstration sites. This project, the largest study funded by HRSA, includes an assessment of changes in the health status of pregnant women and infants across the sites. The study is designed to answer four questions: Did the Healthy Start initiative succeed? If so, why? If not, why not? What would be required for a similar intervention to succeed in other settings?
Finally, with the continued and projected emphasis on technology, HRSA is asking whether investment in telemedicine improves the availability and quality of care to underserved populations and provides easier access to continuing education and consultation for providers in isolated settings. "Rural Applications of Telemedicine" is constructing a broad base of knowledge about telemedicine upon which further assessment of the HRSA telemedicine grant program will be built. The four main objectives are (1) to determine the current status of telemedicine in rural health; (2) to explore the effects of telemedicine on access to care, practitioner isolation, and the development of health care networks; (3) to explore the organizational factors that aid or impede the successful development and implementation of telemedicine systems; and (4) to develop, test, and refine data-collection instruments that can be used in subsequent evaluation efforts. Representatives of several other Federal departments and agencies are participating in the conduct of this study.
New Directions for HRSA Evaluation
Major evaluation priorities in fiscal 1996 include managed care; such primary care programs as Community and Migrant Health Centers and the National Health Service Corps; care for mothers and children, exemplified by the Healthy Start initiative; HIV/AIDS services, including new approaches to delivering and financing services through the Special Projects of Regional and National Significance; health professions efforts to foster community-based training for primary care practitioners; and strengthening of the health care infrastructure at the community level, partly through collaboration with States and external organizations such as the American Hospital Association. Underlying the work in all of these topical areas will be continuing efforts to further strengthen HRSA's performance measurement capacity through projects targeted to the needs of particular components and programs and to the integration of performance management approaches agencywide in the context of budget decisionmaking and strategic planning. Finally, HRSA will continue efforts to broaden the dissemination of evaluation results and information about ongoing studies to the public health community and to Congress.