MISSION: To improve health by assuring quality health care to underserved, vulnerable populations and by promoting a primary care and public health workforce.
HRSA Evaluation Program The objectives of the Health Resources and Services Administration (HRSA) evaluation program are to improve program management and policy development and to provide information that will enhance strategic planning, budget decisions, and legislative planning. Consequently, a high priority is given to studies that will (1) assess program effectiveness or outcomes, or (2) enhance the Agency's capacity to measure performance by identifying meaningful performance indicators, creating or refining databases, and developing study designs or approaches. Over the past several years, HRSA has made considerable progress in expanding such capacity.
The evaluation program in HRSA is designed to ensure that the four Bureaus and the Office of Rural Health Policy (ORHP) identify their particular information needs within a framework established by the Administrator each spring through a memorandum on evaluation planning guidance. Other staff offices may also propose studies. After study proposals are reviewed for clarity by staff of the Office of Planning, Evaluation, and Legislation (OPEL), proposals are reviewed for technical merit by a committee of senior evaluators representing each Bureau, ORHP, the Agency for Health Care Policy and Research, the National Center for Health Statistics (NCHS), the Office of the Assistant Secretary for Health, and the Office of the Assistant Secretary for Planning and Evaluation. Authors receive written comments on methodological or other design issues. Proposals are then reviewed by a committee of the Bureau Directors and other senior HRSA executives, whose role is to consider the proposals for relevance to important policy, budgetary, or legislative issues; potential to answer questions about program effectiveness or impact; and degree of attention to crosscutting topics. Proposals and committee recommendations are submitted to the Administrator for decision. OPEL staff work closely with program staff to develop a strong scope of work and other required documentation.
HRSA places major emphasis on the dissemination of information about findings and utilization of completed studies. Developing articles for publication in the professional literature and presentation at professional meetings is strongly encouraged, and draft manuscripts are typically called for as a contractor product. Briefings on completed studies of broad interest are scheduled periodically for the Administrator and other senior officials. An annual report on completed evaluation studies describes study purposes and findings, and tells how study results have been disseminated and used. Another annual publication provides brief summaries that describe the purpose and proposed methodology for studies initiated in the preceding fiscal year.
Summary of FY 1994 Evaluations Two significant studies were completed that examined aspects of the community and migrant health centers. "Evaluation Design: Evaluation of the Effectiveness of Community Health Centers" developed a study methodology to examine the efficacy and effectiveness of the centers. This work has provided a way to assess the impact these providers have on the health status of their users. "Linkage Programs: Survey of Mental Health Services" examined a joint National Institute on Drug Abuse/HRSA demonstration program designed to test community-based approaches for linking substance abuse and primary care services. The study found that the projects had been successful in creating supportive environments for the assessment, referral, and delivery of mental health services. Findings from this study will help other providers of primary care services establish the necessary alliances and treatment networks to reduce delivery system fragmentation and develop a continuum of mental health services.
"HIV Service Networks in Four Rural Areas," highlighted in chapter II, is a series of case studies undertaken to help States identify ways to improve the quality, availability, and organization of services for rural residents with HIV. The case studies described two distinct approaches to providing medical care to rural residents with HIV: (1) cooperative working relationships between HIV clinics and physicians in private practice, and (2) linkages between physicians and a medical school or rural-based regional care facility for training and backup consultation. Findings are being used to formulate HRSA policies on HIV care in rural areas and to provide technical assistance to States with large rural populations. The findings are also being used to design a rural component, funded by HRSA, for a large national study of AIDS services sponsored by the Agency for Health Care Policy and Research.
Three other AIDS-related studies were completed by HRSA in FY 1994. Two of these address the priority area of community infrastructure building. "Implementation of Title I of the Ryan White CARE Act of 1990" established a preimplementation baseline in Baltimore, Maryland, and Oakland, California, covering the organization and funding of HIV-related services, the functioning of planning councils, and the perceptions of HIV patients regarding service delivery. The Henry J. Kaiser Family Foundation subsequently funded followup surveys on these issues. "The Participation of People With HIV in Title I HIV Health Services Planning Councils" studied planning councils in Atlanta, Philadelphia, San Diego, and Seattle, and identified factors supporting the initial and sustained involvement of HIV-positive individuals in these councils. It also identified alternative methods employed by the councils to gain input from and provide feedback to HIV-positive populations in the absence of their direct participation on the councils. "Evaluation of Methods for Estimating Unit Costs of HIV Health and Support Services" developed common definitions for service, cost, and expenditure categories, and identified potential indicators of cost savings attributable to coordinated, comprehensive, community-based care systems.
"The Demographic and Treatment Characteristics of the Hill-Burton Population" obtained data on a sample of patients receiving services under the Hill-Burton Uncompensated Services Program and compared these with data on the general population. The study found that, compared with the total U.S. population, the Hill-Burton population was more likely to be female, young adult, and unmarried, with a very low household income.
Three studies dealt with health professions programs. "Survey of Beneficiaries of Nursing Education Projects" concluded that Federal support of these projects has enhanced the supply of advanced practice nurses. "The Training and Practice of Preventive Medicine Specialists: An Underutilized Resource" studied the area and, based on its findings, recommended incorporating preventive medicine into the medical school curriculum. In light of the increased attention being given to improved management practices, "Evaluation of Bureau of Health Professions Strategic Directions--Phase I" developed a set of outcome indicators and identified potential data sources and gaps relating to programs under Titles VII and VIII of the Public Health Service Act.
Evaluations in Progress The 41 studies that were ongoing as of September 1994 include four types: (1) measurement of program outcomes, effectiveness, or impact; (2) development of performance indicators, and/or data systems or databases; (3) design and/or testing of study approaches; and (4) provision of other types of information for program management or policy development.
The largest study of program performance and outcomes is the National Evaluation of the Healthy Start initiative. Begun in September 1993, this 5-year prospective study includes both process and outcomes components. The process evaluation is assessing the development and implementation of comprehensive, coordinated systems of perinatal care in the 15 initially funded Healthy Start sites. The outcomes component concerns changes in the health status of pregnant women and infants across the sites. Client-specific data from a standard data set will be used to examine the relationship among clients' sociodemographic characteristics, project-specific service use, and maternal and infant health outcomes. Secondary data, including linked birth and death records and Medicaid claims data, will be used to compare outcomes of clients residing in Healthy Start project areas with those of women living in matched comparison areas. The study also will employ a series of more qualitative methodologies to gain information about the unique features of each of the 15 projects. Four questions are being addressed. Did the Healthy Start initiative succeed? If so, why? If not, why not? And, what would be required for a similar intervention to succeed in another setting?
Two other studies are assessing the effectiveness of the community and migrant health centers programs, using national samples of grantees and including both quantitative and qualitative techniques. Additional studies concern effectiveness and outcomes of the National Practitioner Data Bank, the Healthy Schools/Healthy Communities program, and the Injury Prevention Implementation Incentive Grants program. One further study examines the effects of Ryan White CARE Act Title I funding on services for active and recovering HIV-infected drug users.
Among studies of the second type are projects to assess data systems for the Ryan White CARE Act Titles I and II programs, and to develop national integrated requirements estimates for physician assistants, nurse practitioners, certified nurse-midwives, and physicians.
The Agency also has several ongoing studies to design and test study approaches. One effort is intended to design a method to evaluate the validity of current estimates of the reported mortality rate among infants of Mexican descent residing in the United States and examine factors contributing to the reported low rate. The study includes nontraditional sources of data concerning births that occur outside the confines of the infant tracking systems. Another study is developing a self-assessment tool for use by HIV health planning councils in Title I cities and by care consortia in Title II states. Grantees and their planning councils and care consortia have expressed interest in methodologies and resources that will help them know what they are accomplishing. The tool will be sensitive to the limited resources available for planning and evaluation, providing a cost-effective method for evaluating performance. An additional study is developing a design for assessing variations in the procurement rates of Organ Procurement Organizations (OPOs). Identification of the factors associated with low procurement rates should help increase these rates.
Finally, several studies are intended to provide other types of information for program management or policy development. One exploratory project on the nature of telemedicine systems serving rural populations is providing baseline information in connection with the new HRSA Rural Telemedicine Grant program. An additional study is examining the impact of certain Medicaid waivers on Federally Qualified Health Centers. Another study is examining the relationship of medical school characteristics to graduates who choose primary care and ultimately provide care to the underserved. Still another study is exploring the impact of the Ryan White CARE Act on strategies for serving African Americans. One of the study products will be a report providing detailed examples of effective and ineffective processes and strategies for enhancing access for African Americans. The report will also identify performance indicators being used and approaches for gathering data to measure performance. Finally, a 2-year study by the Institute of Medicine is defining the place of primary care in the health care environment of the future and developing normative goals for primary care.
New Directions for Evaluation The major priority of HRSA evaluations in FY 1995 and 1996 will be studies to continue the development of meaningful measures and data systems to facilitate performance assessment. During FY 1995, HRSA is conducting a project to review the status of performance indicators for all line organizations and major programs and determine the degree to which data sources are currently available for applying the appropriate indicators. The impetus for this effort is the implementation of the GPRA, which will begin with the FY 1997 budget.
In FY 1995 and 1996, HRSA's other evaluation studies will continue to reflect the Agency's mission of expanding access to care for the underserved and such crosscutting priority areas as academic/community partnerships in health professions education, community infrastructure building, managed care, expanded roles for States, information technology, and HIV/AIDS. HRSA also expects to develop multiyear evaluation goals for selected program areas in connection with new authorizing legislation and to initiate or continue studies of effectiveness, outcomes, and/or impact in selected program areas.