MISSION: To improve the Nation's health by assuring equitable access to comprehensive, quality health care for all.
The purposes of the Health Resources and Services Administration (HRSA) evaluation program are to enhance strategic and performance planning, strengthen budget and legislative development, and improve program management. The Agency's evaluation priorities are to develop and support performance measurement, assess program implementation, and conduct crosscutting policy analysis and research.
Performance measurement includes technical and training activities to strengthen the Agency's capacity to assess program performance, as well as studies to assess program outcomes. These activities, mandated by the Government Performance and Results Act (GPRA), will provide data for ongoing program monitoring and evaluation and for developing annual performance plans and reports. HRSA completed a study in September 1995 to establish a performance measurement baseline for all operating programs, with the intent of specifying valid and useful indicators and measures for all programs by September 1998. Over the past three years, HRSA has made significant progress in performance management, such as in strengthened linkages among strategic planning, evaluation, and budget decisionmaking.
Assessment of program implementation includes a broad range of efforts to assist the Agency in adjusting program development and management as the priorities of the Executive Branch and the Congress change. HRSA programs are entering into new arrangements for delivering services, providing health professions education, and encouraging the development of systems reform.
Crosscutting policy analysis and research includes efforts to build capacity and conduct studies to clarify the environmental shifts affecting HRSA programs and to improve HRSA's ability to document performance and impact through better measurement tools and data sources. The term, "crosscutting," refers to issues that bear on the programs of two or more HRSA Bureaus, such as trends in numbers and characteristics of the uninsured, and the viability of the health safety net. Development of this capacity at the Agency level began in FY 1997; work in FY 1998 and beyond will build on this foundation.
HRSA is striving to improve the quality of evaluation products. Potential approaches for this initiative include (1) an increase in the level and amount of technical assistance available to staff across the Agency; (2) expansion in the proportion of work done intramurally or with short-term help from individual experts; (3) encouragement of HRSA staff to participate in evaluation courses offered by local universities; and (4) establishment of an external evaluation advisory committee to consult on evaluation strategies in terms of content, approaches, and dissemination. Evaluation capacity is being strengthened first in the central evaluation unit, with the intent of making similar improvements in evaluation resources of the Bureaus in the future.
HRSA is continuing to place a major emphasis on dissemination of findings and information about the utilization of study results. A broader dissemination strategy, including an assessment of opportunities for using electronic media more effectively, will also be an important component of the quality improvement activities now under development.
Summary of Fiscal Year 1997 Evaluations
Studies related to the three key areas of HRSA's mission are described first, followed by crosscutting projects.
Eliminating Barriers to Care
One significant study, Exploratory Evaluation of Rural Applications of Telemedicine (5749), provides national baseline data on the status of telemedicine in health care for rural populations. The project included a mail survey of all non-Federal hospitals located outside metropolitan areas in the summer of 1995, plus a short survey of hospitals doing only teleradiology and a longer survey of hospitals applying telemedicine for purposes beyond radiology. The major findings were that (1) rural telemedicine is in the early stages of development but is rapidly expanding; (2) telemedicine networks are complex, with an average of four "spoke sites," two hubs, and four facilities that provided and received consults; and (3) Federal and State grants and support from hospitals are common sources of direct funding, but fewer than 25 percent of hub facilities had negotiated payment with insurance carriers and many had not begun negotiations. Study results have been used in forming the FY 1997 guidelines for the Rural Telemedicine Grant Program, in part to encourage the use of telemedicine in community settings. As requested, early findings were shared with congressional staff, who then specified priority areas in the authorizing legislation. The report was distributed to Federal agencies funding telemedicine, to State Offices of Rural Health, and through the World Wide Web. Finally, findings are informing the design of a common telemedicine evaluation instrument for HRSA grantees and for consideration by other Federal sponsors.
Another study relating to access is Institutional Factors That Influence the Likelihood of Living-Related Kidney Transplant Operations--Phase II (4917.1). The purpose of this project was to determine (1) the willingness to conduct, and frequency with which the transplant community does conduct, kidney transplants from living-related donors; (2) the medical and non-medical institutional factors influencing the likelihood that transplant professionals will suggest and conduct such transplants; and (3) the circumstances and conditions under which a transplant team that does not encourage these transplants would refer a patient to another transplant center for evaluation. One finding was that over half of kidney transplant centers lacked patient education materials about live kidney donors. A summary report has been sent to all kidney transplant centers. In addition, discussions have been held with groups such as the National Kidney Foundation about ways to apply the findings, such as through development of educational materials for national distribution.
In the health professions area, a major statistical study produced a technical and an analytic report; the title of the latter is The Registered Nurse Population, March 1996: Findings from the National Sample of Registered Nurses (6367). A contractor prepared the sampling design and conducted the survey, which is the most comprehensive source of statistics on registered nurses (RNs) with current licenses to practice in the United States. The survey resulted in a data base of information on 29,950 nurses, which was then analyzed by HRSA staff. The report provides information on the number of nurses; their educational background and specialty areas; their employment status, including type of employment setting, position level and salary; their geographic distribution; and their personal characteristics, including gender, race and ethnicity, age, and family status. HRSA is using the study products to guide nursing education programs, to prepare biennial reports to Congress on characteristics of the RN population, and for projecting the supply and demand for nurses at a national level. The data base and analytic report have been shared with the nursing community, which looks to the Department to maintain and update this data base.
Eliminating Disparities in Health Status
Another major quantitative effort was the Community Health Center User and Visit Study: Survey Methodology Report (5737). This project gathered, for the first time, nationally representative data about Community Health Center (CHC) users and the services provided to them. The study had two components: (1) personal interviews with 1,932 patients selected randomly from a representative sample of 48 CHCs, and (2) data extraction from 2,878 medical records concerning randomly selected encounters during 1994 at the same centers. Data on users were gathered in a manner to permit comparison with results from the National Health Interview Survey. Similarly, information from the visit (encounter) survey will be compared with results from the National Hospital Ambulatory Medical Care Survey. The CHC data are being analyzed to answer questions on such topics as the race and ethnicity of users, health-risk behaviors presented, most prevalent diagnoses, services utilized, and the extent of monitoring of compliance with treatment prescribed for chronic conditions. In addition, the data are being reviewed to form a profile of CHC visits in comparison with studies of the general population in terms of conditions, diagnoses, type of provider seen, and disposition. Analysis of data from the initial surveys continues intramurally, as well as through a current study noted below.
Assuring Quality of Care
A major concern in recent years has been the spread of tuberculosis (TB) among persons with HIV/AIDS. Examining Relationships Between HIV/AIDS and Tuberculosis Programs in Title I Eligible Metropolitan Areas (6359) was designed (1) to determine the extent to which Ryan White Comprehensive AIDS Resources Emergency (CARE) Act grantees were giving guidance to providers on TB screening and prophylactic and therapeutic regimes, and (2) to describe models of collaboration between HIV and TB control programs within Eligible Metropolitan Areas (EMAs). The study found that TB services varied across EMAs and across care settings within them. As of 1995/1996, Federal and State funding policies and staffing patterns hampered collaboration between HIV and TB programs. However, the study identified mechanisms to promote TB-HIV services collaboration, fostered in some places by outbreaks of TB and multidrug-resistant TB. HRSA has used the results to inform Title I program guidance and to develop the training and technical assistance plan for FY 1998. The report has been sent to Title I grantees and to the Centers for Disease Control and Prevention (CDC) TB control grantees. Results were also presented at the 1995 International AIDS conference in Vancouver, Canada, and in September 1997 at the CDC conference entitled, "Issues for HIV and TB Co-Infected People."
The increasing adoption of managed care by public insurance programs, such as Medicaid, has raised concerns about whether capitated reimbursement affects quality of care provided to people with chronic illnesses such as HIV/AIDS, or with other special health care needs. One way to address these concerns is to create risk-adjusted capitation rates to provide equitable reimbursement for the care of patients with costly illnesses, an approach several States had proposed or implemented by 1997. In May of last year, HRSA convened an invited conference to examine issues surrounding risk adjustment for HIV/AIDS, in collaboration with other HHS organizations (the Office of the Assistant Secretary of Planning and Evaluation and the Health Care Financing Administration), the Henry J. Kaiser Family Foundation, and the National Academy of State Health Policy. The HIV Capitation Risk Adjustment Conference Report (6894) presents perspectives of participants (representing persons living with HIV/AIDS, providers, managed care organizations, and States); methodological issues; and research needs. Conference findings have been used in technical assistance to States on coordinating CARE Act programs with managed care. HRSA has distributed the report to all CARE Act grantees and State Medicaid directors.
One recently completed study concerned managed care and its relevance for vulnerable populations. The Availability of Primary Care Services Under Medicaid Managed Care: How 14 Health Plans Provide Access and the Experience of 23 Safety Net Providers and Their Communities (6366) examined strategies used by health maintenance organizations (HMOs) to generate access to health care services for Medicaid enrollees, who often live in inner-city and rural, underserved areas. The project also explored the implications of these strategies for overall access in underserved areas and for existing care providers. Third, the study considered how these strategies affect traditional Medicaid providers in the community. The study concluded that Medicaid managed care has had an overall positive effect on the availability of primary care to low-income residents in the 14 communities, and that traditional safety net providers have not had to reduce services. But the future viability and range of services of public sector-supported providers might be endangered in a more competitive health system. The findings will be useful to policymakers concerned with the provision of care for vulnerable populations, providers in underserved areas, and those attempting to assure an adequate supply of health personnel. The study has been noted in the professional press and will be referenced in a journal article now under preparation.
Evaluations in Progress
Projects in progress are presented first according to HRSA's three mission goals; crosscutting projects follow.
Eliminating Barriers to Care
Begun in September 1994, the Evaluation of the Effectiveness of Community Health Centers: Implementation Phase (5736) is describing CHC delivery model(s), measuring the impacts of the model(s), and identifying factors that contribute to the observed impacts. The study is examining the experience of CHC users and non-users in both fee-for-service and managed care arrangements though a sample of 50 CHCs in 10 States. The project has three components: (1) an analysis of utilization and expenditures, based on HCFA State Medicaid Research Files for eight States; (2) an analysis of outcomes and content of care for four conditions (otitis media, asthma, hypertension, and diabetes) based on CHC medical records and the literature on medical treatment norms; and (3) a description of CHC delivery systems using primary data collection (a mail survey) and secondary sources. The analysis of utilization and expenditures is addressing casemix and differences between exclusive and regular CHC users.
Effectiveness of the National Health Service Corps (6357) is examining the various mechanisms for training and recruiting providers, placing them in underserved areas, and retaining them in primary care and service to vulnerable populations. The project includes mail surveys of three random samples: (1) National Health Services Corps (NHSC) alumni clinicians; (2) current NHSC clinicians; and (3) administrators of sites staffed with NHSC clinicians. (Some of the sites are CHCs and others are free-standing.) The study also involves review of secondary data on individual NHSC participants, sites that are eligible for NHSC assignees, characteristics of designated shortage areas, and site vacancies available for assignment of obligated scholarship providers.
In another critical policy area, HRSA is conducting a study entitled Enrollment, Utilization and Cost-Forecasting Models for Use in State AIDS Drug Assistance Programs (6806). This study is exploring the effect of the introduction of protease inhibitors on medical service utilization for use in State AIDS Drug Assistance Programs (ADAP). During 1996, five protease inhibitors became available; this class of drugs has been shown to have superior outcomes in treating HIV. However, although pharmaceutical interventions represent a sizeable portion of the cost of care for HIV-infected patients, the efficacy of these drugs has not been shown outside of controlled clinical trials. This study is using client-specific data from a geographically diverse population to identify personal factors that may modify the link between the introduction of protease inhibitors and services use.
Eliminating Disparities in Health Status
The National Evaluation of the Healthy Start Program (5610) continues as a longitudinal study of the development, implementation, and outcomes of the comprehensive maternal and child health delivery systems at the original 15 Healthy Start sites, which were funded initially as 5-year demonstration projects. A sixth year was added in FY 1996. This study includes an assessment of changes in the health status of pregnant women and infants across the sites. The study was designed to answer four questions: Did the Healthy Start initiative succeed? If so, why? If not, why not? What would be required for a similar intervention to succeed in other settings? The study includes both process and outcome components. Findings from both will contribute to analyses bearing on replication of successful approaches for reducing infant mortality, and on improving program effectiveness in general. Study results will be used to guide the development and implementation of Healthy Start projects funded subsequently, and will be widely shared with the public health community for application in settings funded under other auspices.
Analysis of CHC User Survey: Selected Conditions (6805) is assessing portions of the survey data from the Community Health Center User and Visit Study (see PIC ID No. 5737). The purpose is to describe the self-reported process of care and outcome measures for CHC users with certain conditions: hypertension, high serum cholesterol, diabetes, asthma, and pregnancy. The analysis includes the demographic characteristics, income and educational levels, health status, and activity limitations of these patients as compared with (1) CHC users without the selected conditions, and (2) the general population. Data on the general U.S. population will be from the National Health Interview Survey and the National Hospital Ambulatory Medical Care Survey. Findings will be used, in part, to determine whether care provided at CHCs is appropriate and conforms to the specifications of Healthy People 2000.
Assuring Quality of Care
One of HRSA's critical concerns is cultural competence of care. A study begun in FY 1997, Development and Refinement of Tools for Monitoring Cultural Competence in Managed Care (6817), is analyzing successful experiences of managed care organizations in California in meeting the needs of culturally and linguistically diverse Medicaid populations. A second purpose of this study is to plan for the validation of tools for measuring cultural competence among individual providers. The new study builds on Tools for Monitoring Cultural Competence in Health Care (6352), which was completed in FY 1996.
Direct Support for Conduct of Evaluation Studies at Selected CARE Act Pilot Sites (6807) is providing assistance to six Title I and Title II grantees in analyzing data sets and developing models to assure the effectiveness of their primary care and supportive services. These local studies constitute a first step in developing future "sentinel" studies to create a picture of the national impact of these CARE Act programs. These initial projects include three types: (1) examining the association of CARE Act-funded services with clinical and other outcomes for patients with HIV; (2) developing models to assess the effects of managed care on fund allocations and service delivery; and (3) developing and pilot testing models to analyze the effectiveness of Title II care consortia in evaluating needs, setting priorities, and delivering services.
In addition to studies relating mainly to one mission area, HRSA has ongoing projects focused on the capacity of two HRSA Bureaus to measure performance across the organization. The first of these projects, which concerns the Maternal and Child Health Bureau (MCHB), is the MCHB GPRA Maturity Project (6811). This study, building on a series of program-specific efforts, is providing coordination of performance measurement activities to produce coherent performance data for the Bureau as a whole. The project also provides technical assistance in working with grantees and others to promote acceptance and implementation of performance measures throughout the maternal and child health community, and training for MCHB program staff in the ongoing use of performance measures. The intent is to complete the development of a fully operational performance management system.
A Bureau of Health Professions (BHPr) project also illustrates the use of evaluation to develop performance measures that encompass all Bureau programs. Development of BHPr's Comprehensive Performance Monitoring System (6800) is helping the Bureau implement its monitoring system. In preparation for meeting GPRA requirements, BHPr has developed goals, outcomes, and indicators with respect to workforce quality, supply, diversity, and distribution. This project, begun in FY 1997, is using a data collection instrument that was tested through an earlier pilot study to begin nationwide implementation of the data collection. The study includes technical assistance to applicants and grantees in gathering the required data and completing the data collection instrument. This project will involve grantees for the first time in submitting data electronically using the new instrument. The study includes development of software, training in its use for Bureau staff and grantees, and a help desk to provide ongoing technical assistance during implementation of the reporting system. The study will also include an evaluation of the data collection instrument and electronic implementation, the training and technical assistance, and the quality of data collected. Results of the project will be used to determine whether changes are needed in the current approach for collecting and analyzing program data, and will be shared with the public through an article in a professional journal.
Through Managed Care and the Safety Net Providers (6815), HRSA is examining the impact of Medicaid managed care on the fiscal and operational viability of safety net providers in primarily ambulatory and primary care settings. Among the types of providers to be included are Community and Migrant Health Centers, Maternal and Child Health programs, Ryan White CARE Act grantees, and rural health centers. A key topic being explored is the effect of a more competitive, market-oriented health care system on capacity to serve vulnerable populations. Policy and program interventions to deal with possible shortfalls will be identified.
A collaborative effort with NCHS, Person-Place Approximations and Utilization vs. Health Status Analysis of the National Health Interview Survey (5614) is addressing three questions: (1) What common characteristics are shared by communities with problems in access to health care? (2) Can these characteristics/
problems be measured directly, or through good proxies from data that can be easily obtained? (3) What is the appropriate aggregation of geographical units to use in describing small areas? The intent is to gain a better understanding of the ways in which the characteristics of particular areas influence the behavior of respondents to the National Health Interview Survey in order to use the knowledge for predicting access problems in small areas for the entire U.S. population.
Finally, HRSA Crosscutting Community Case Studies (6814) is examining the nature, extent, and impact of the interaction of HRSA programs in selected "sentinel" communities in order to determine whether inflows of HRSA resources at the local level combine in a mutually reinforcing way, operate independently, or work at cross-purposes. This project builds on two ongoing studies sponsored by other organizations. The Robert Wood Johnson Foundation is developing baseline information for tracking changes in the health care market in 12 metropolitan statistical areas. The other project, sponsored by the HHS Office of Disease Prevention and Health Promotion, is addressing how market-driven changes are affecting the level of effort of public health departments in providing ten essential public health services. Sites for the HRSA study are three urban communities (Boston, Cleveland, and Phoenix) in the Johnson Foundation study. This project is HRSA's first attempt to test a methodology for measuring the interactions and collective impacts of the Agency's programs on communities, which can then to used for conducting prospective and long-term evaluations.