Performance Improvement 1997. Health Resources and Services Administration


MISSION: Improve the health of the Nation by assuring quality health care to underserved, vulnerable, and special-need populations and by promoting appropriate health professions workforce capacity and practice, particularly in primary care and public health.

Evaluation Program

The purposes of the Health Resources and Services Administration (HRSA) evaluation program are to enhance strategic planning, budget decisions, legislative planning, and program management. Consequently, major emphases during FY 1997 and beyond will be (1) performance measurement, (2) assessment of program implementation, and (3) cross-cutting policy analysis and research.

Performance measurement includes technical assistance and training activities to strengthen the agency's capacity to assess program performance, as well as studies to assess program outcomes. These activities, mandated by the Government Performance and Results Act (GPRA), will provide data for ongoing program monitoring and evaluation and for developing annual performance plans and budgets. A study completed in September 1995 established a performance measurement baseline for all operating programs, with the ultimate objective of ensuring that the HRSA specifies valid and useful indicators and measures for all programs by September 1998. Over the past two years, the HRSA has made significant progress in performance management, such as strengthening linkages among strategic planning, program activity, and the budget process.

Assessment of program implementation includes a broad range of efforts to assist the agency during a time of new directions from the Administration and Congress. HRSA programs are entering into new arrangements for delivering services, providing health professions education, and encouraging the development of systems reform within seven Program Priority areas. These priorities reflect the HRSA's focus on the communities where underserved populations live, and include academic and community partnerships to foster clinical training in community-based settings; new arrangements to bring poor, uninsured, rural, and chronically ill persons into the mainstream of managed care; assistance to communities in strengthening their health care infrastructure; and new activities with States. All agency studies relate to one or more of these Priorities, which also include improved comprehensiveness and integration of HIV/AIDS programs; school-based care for children and adolescents; and activities to enhance the health of people living along the U.S.-Mexican border.

Cross-cutting policy analysis and research includes efforts to build capacity and conduct studies to clarify the environmental shifts within which HRSA programs operate and to improve the HRSA's ability to document performance and impact through better measurement tools and data sources. Developing this capacity at the agency level and initiating some studies are special priorities for FY 1997; work in later years will build on these beginnings. "Cross-cutting" implies policy issues that bear on the programs of two or more HRSA Bureaus, such as trends in numbers and characteristics of the uninsured and the health of the safety net.

The main purpose of the HRSA's evaluation program is to provide accurate and relevant information on a timely basis to the administrator and other senior line managers. In working toward this objective, HRSA subjects study proposals to a formal review process, which begins with consideration by a committee comprised of senior line officials and is chaired by the Director of the Office of Planning, Evaluation and Legislation (OPEL). This committee's task is to assess the relevance of the proposals to important policy, budgetary, or legislative issues; its potential to answer questions about program effectiveness or impact; its degree of attention to cross-cutting topics; and its relative priority for funding. Following administrator decisions on approval of the recommended studies, a committee of senior analysts from the HRSA, the Agency for Health Care Policy and Research, the National Center for Health Statistics, and the Office of the Assistant Secretary for Planning and Evaluation (ASPE) reviews scopes of work and other substantive portions of documents needed for development of contracts. The role of this technical review group is to suggest ways to strengthen study design or data sources.

HRSA is continuing to place major emphasis on disseminating findings and information about use of study results. This is done through inclusion of abstracts in the ASPE evaluation data base (on the Internet), submission of articles to peer-reviewed professional journals, presentations by HRSA staff at professional conferences, and placement of final reports with the National Technical Information Service. In addition, the HRSA prepares annual volumes of summaries of studies begun or completed the preceding fiscal year. These are distributed within the department and their availability is noted on the HRSA OPEL web page at

Summary of Fiscal Year 1996 Evaluations

During FY 1996, HRSA completed evaluations in the following areas.

Performance Measurement

Projects to build on the baseline study on performance measurement noted above were a major emphasis during the past year; they included technical assistance tailored to the needs of each of the four Bureaus. Illustrative of this work was the Evaluation of the Bureau of Health Professions Strategic Directions: Phase II, which developed a set of outcome-based performance measures and a plan for a performance-monitoring system to support ongoing program management and compliance with GPRA requirements. This study is featured in chapter II.

Relevant to HRSA's continued efforts to identify appropriate health outcomes indicators, the agency conducted a Consensus Conference on Health Status Gaps of Low-Income and Minority Populations. The report recommends potential measures and data sources for eight conditions, sets of conditions, or preventive approaches: diabetes; hypertension, cardiovascular, and cerebrovascular conditions; breast cancer; cervical cancer; infectious diseases, including AIDS, sexually transmitted diseases, and tuberculosis; immunization; asthma; and pregnancy outcomes.

Another HRSA effort developed quality assurance procedures for the Uniform Reporting System for Titles I and II of the Ryan White Comprehensive AIDS Resources Emergency (CARE) Act, which provide grants to disproportionately affected metropolitan areas and States. The purpose of the funding is to improve the availability and coordination of services for persons with HIV/AIDS. Under the Uniform Reporting System, grantees submit provider-level administrative data and data on client demographics and services received. Uniform data are used to assess the extent to which grantees are achieving the goals of the Act and to help metropolitan planning councils, State agencies, and State consortia target and monitor the provision of services to specific population groups. Through this study, data quality targets were established and manuals were written to help grantees compile high-quality data. HRSA and grantee staff are using the results to improve the quality of data submissions, which will support national and local program monitoring and evaluation.

Managed Care

Managed care will continue to have a major influence on HRSA programs. This topic was looked at in two studies: An Evaluation of the Impact of the Social Security Act Section 1115(a) Waivers on Federally Qualified Health Centers was designed to investigate how federally qualified health centers (FQHC's) have fared in Hawaii and Oregon, both of which have waivers involving placement of Medicaid eligibles in managed care organizations. The study found that the waiver programs expanded access to primary care and dental services, and that FQHC's continued to provide types and levels of services not included in the capitation rates, as well as to serve patients not covered under the waiver program. Study results are being used by State and Federal policymakers concerned with Medicaid managed care programs and have been disseminated to managed care organizations that are part of Medicaid managed care programs.

Additionally, tools were developed for monitoring the cultural competence of primary care providers practicing in managed care systems. The work was carried out through physician and patient panels in two California HMO's serving predominantly Latino or Chinese populations, and through a literature review. "Cultural competence" was defined as the level of knowledge-based skills required to provide effective clinical care to patients from a particular ethnic or racial group. A related concept, "cultural sensitivity," was defined as the psychological propensity to adjust one's practice style to the needs of different ethnic or racial groups. Study products include a patient satisfaction questionnaire, which a managed care plan could use to assess the degree to which enrollees with limited English believe their care is culturally competent; a provider self-assessment questionnaire; and behavioral ethnic identifiers that can be used in enrollment processes as a cue for designing culturally appropriate services.

Primary Care

Primary Care: America's Health in a New Era was a major undertaking carried out by the Institute of Medicine to define and describe the place of primary care in the changing health care environment and to develop a set of normative goals for primary care. Findings focus on interdisciplinary primary care teams, roles of the different providers that make up the teams, and ways in which team members could interact with each other and with providers outside the team. The study report is being used to inform the health care community about primary care issues for the next decade.

Assessing the Impact of Public Spending on the Health of Vulnerable Populations: A Framework for Evaluating HRSA Programs provides a structure for describing the linkages among HRSA dollars and HRSA-funded services, HRSA-enabled resource- and infrastructure-building programs, and the populations affected in terms of health outcomes. The analytic approaches developed here could be used by the HRSA or other operating divisions to explain the difficulties in establishing causal factors, to highlight areas with the greatest need for surveillance and monitoring, and to develop data systems to provide the needed information.

A study entitled Impact of Case Management on Health Status in Community and Migrant Health Centers (CMHC's) examined case management practices and their impact on health status of participants in the Comprehensive Perinatal Care Program (CPCP). The study consisted of three components: a control-site study of paired CMHC's, intensively using and not intensively using case management; a model-site study of seven CMHC's serving high-risk perinatal populations; and a two-part mail and telephone survey of a sample of centers receiving CPCP funding. The study found that case management had a positive impact on the continuity and adequacy of care, on the content of care in some areas (psychological risk assessment and health promotion), and on users' perceptions of care. The HRSA is using the findings to integrate case management in all primary care projects serving perinatal patients. The study also developed a self-assessment tool that grantees can use to evaluate their case management activities.

Finally, Technical Support for Need Designation and State Primary Care Access Plan Initiatives assessed the process for designating areas of primary health care shortage. Results are being used in finalizing new designation criteria for shortage, underservice, and access barrier areas.

Care for Mothers and Children

The Emergency Medical Services for Children (EMSC) program was begun in 1984 to help States develop and institutionalize emergency medical services for critically ill and injured children. This past year, HRSA completed a study to assess the extent to which seven States had sustained services funded with an EMSC grant, to identify attributes associated with successful institutionalization of EMSC in State emergency medical services systems, to characterize barriers to institutionalization, and to recommend approaches for Federal actions to improve EMSC systems. Recommendations, which include three areas (strategic planning and funding; grant guidance, including an increased emphasis on evaluation; and program development and intergovernmental relations), will be used by program managers in setting future directions and goals.

Health Professions

One project in the health workforce area developed the Report to Congress on the Process by Which International Medical Graduates Are Licensed To Practice in the United States. This study assessed the time required to verify the credentials of physicians applying for licenses and differences in the application process for international versus U.S. medical graduates. The study found that there is a time differential between these groups of graduates and that a national credentials verification system is needed to assist State medical boards. The study also recommended that uniformity among States in licensure requirements be expanded to a maximum degree.

Another study assessed the feasibility of increasing the production of advanced practice nurses by investigating the potential for nursing educational institutions to expand their capacity. "Advanced practice nurses" include nurse practitioners, clinical nurse specialists, nurse anesthetists, and certified nurse midwives. The study found that the single most significant factor in determining an institution's ability to expand capacity is the availability of sites for clinical education, although the availability of clinical faculty is also important. HRSA is using the results to guide future directions for advanced practice nursing programs and to provide technical assistance to grantees. The report has also been shared with schools of nursing with advanced practice programs.

A third health professions study was an exploratory effort to describe arrangements between primary care residency programs and ambulatory training sites affiliated with them. The most important findings are that (1) residency programs do not significantly offset the costs of ambulatory training experienced by the sites nor are there many written agreements in place; (2) residency programs are not aggressively seeking managed care training sites; and (3) cost pressures on ambulatory training sites may make decisionmakers less willing to assume or continue to absorb the cost of training. HRSA will use these findings, in conjunction with findings from two other studies on site-specific costs of ambulatory training, to expand understanding of the costs of ambulatory education and help HCFA in its deliberation on new approaches to GME funding.

Evaluations in Progress

During FY 1996, HRSA had evaluations under way in the following areas.

Performance Measurement

A study entitled Health Care Status Outcome Measures for the Bureau of Primary Health Care: The Assessment of Ambulatory Care Sensitive Conditions Through State Medicaid Recipient Files is comparing the relative risk of inpatient hospitalizations of community health center users with that of nonusers for ambulatory care-sensitive conditions. This project builds on the Consensus Conference noted above. The third study identified below under Managed Care, Meeting Industry Standards for Performance Measurement Through Clinical Outcome Measures at Community and Migrant Health Centers, also applies here.

Managed Care

A number of studies will continue several of the major themes and initiatives. For example, Evaluation of the Response to Medicaid Managed Care Initiatives by Federally Qualified Health Centers will describe strategies that FQHC's are using to incorporate Medicaid managed care into their practices and assess the impact of these strategies on FQHC organization, delivery of services, and market position. Findings will guide policy for CMHC's. Building on a study completed in FY 1995, the Performance Assessment of Managed Care in Community and Migrant Health Centers will develop case studies in seven centers. Center performance will be compared with that of other providers in the same HMO network on selected industry performance indicators.

Work will also continue to examine the extent to which CMHC's are using clinical outcome measures to assess quality of health care, to meet center needs in a managed care environment, and to improve and monitor the quality of care locally and nationally. Findings will be used to identify performance issues needing further research and development and to facilitate collaboration among private and governmental organizations focused on health outcome measures and underserved populations.

Two studies are examining staffing issues in a managed care context: HMO Staffing Strategies in Underserved Areas is describing strategies to ensure access to services for enrollees in underserved areas, as well as the implications of these strategies for overall access in these areas. The study involves interviews and other data collection at nine HMO's with significant Medicaid enrollment. Selected Aspects of Education in Managed Care is surveying managed care organizations to identify their practices and determine the potential for these organizations to provide high-quality learning experiences for medical students and residents.

Primary Care

The Community Health Center User and Visit Survey, which involves personal interviews of users at a representative national sample of centers and a review of patient records at the same centers, continues from FY 1994. One of the purposes is to gather data on characteristics of users and types of services they received on a basis that will permit comparison with the results of the National Health Interview Survey. A related study begun during FY 1996 is expanding the analysis of the User and Visit Survey data to compare managed care users with users of centers that are not part of managed care arrangements.

Casemix Differences in Health Centers and Other Providers and Their Relationship to Cost will recommend a method for measuring casemix (severity of illness) in ambulatory populations; develop a methodology for creating "casemix profiles" of patient characteristics and expenditure data; and test the methodology with Medicaid data from several States. This study should also provide insights into the relative importance of casemix compared with other factors that may affect expenditures for primary care in the ambulatory setting.

HRSA is continuing to work with the American Hospital Association Hospital Research and Educational Trust (HRET) on evaluation of the HRET-sponsored Community Care Network, a series of demonstrations in 25 sites across the United states. HRSA support for this collaboration was begun through a project completed last year, National Demonstration of the Community Care Network Vision: Development of a Program Evaluation. HRSA will use the findings from the evaluation as a guide for selecting model types or community characteristics that deserve detailed exploration to form hypotheses about attributes that may lead to the successful launching of collaborative activities and to assess implications for cost-effective approaches to measurement and data collection in future evaluations of HRSA programs.

Another study concerns the ability of HRSA-supported community-based organizations to integrate services for individuals with multiple risk factors, such as HIV/AIDS, substance abuse, and homelessness. This project has explored the use of multiple funding sources to provide integrated care, barriers to integration of services supported by separate funding sources, successful integration strategies, and ways in which HRSA might reduce barriers created through multiple funding streams. The results will inform policy on primary care and AIDS programs and will be shared with other operating divisions.

Attention to bilingual and multilingual services is also continuing through studies directed to services at CMHC's, such as the project, An Assessment of Bi/Multicultural Services Offered at Community and Migrant Health Centers. Bilingual Assistance Program: Evaluation of Strategies for Reducing Cultural and Linguistic Barriers to Health Care for Hispanic and Asian Pacific Islander Populations concerns bilingual services in the context of small grants to State and local health departments.

Health Care for Mothers and Children

The National Evaluation of the Healthy Start Program continues as a 5-year longitudinal study of the development, implementation, and outcomes of the comprehensive perinatal delivery systems at the initial 15 Healthy Start sites. This project includes an assessment of changes in the health status of pregnant women and infants across the sites. The study is designed to answer four questions: Did the Healthy Start initiative succeed? If so, why? If not, why not? What would be required for a similar intervention to succeed in other settings?


A major priority of the HRSA is to improve access to a comprehensive continuum of care for persons with HIV/AIDS, through the Ryan White CARE Act programs. One ongoing study concerns the management of people infected with both AIDS and TB in metropolitan areas with Title I grants to identify models of successful collaboration between HIV/AIDS and TB control programs. The project is also determining the extent to which specific Title I grantees are providing guidance to funded providers on TB screening and prophylactic and therapeutic regimes. Another study is reviewing and synthesizing HIV-related consumer-level evaluations conducted by CARE Act grantees across the country. The intent is to identify useful and replicable models of studies for use in providing technical assistance to grantees, and to identify topics on which further studies are needed.

Organ Transplantation

One of the HRSA's roles is to oversee the national system for distributing organs for transplantation. An ongoing study is examining the reasons why African Americans remain on waiting lists longer than white patients. Anecdotal evidence indicates that African Americans are more difficult to contact within allowable time periods than whites and that African Americans are more likely to decline an organ offer. Studies on this subject have been inconclusive, however. Findings will be used to assess whether changes are needed in HRSA policy concerning the transplantation network.


Another significant policy area involving ongoing work is the rural applications of telemedicine. In FY 1994, the HRSA began a baseline project to determine the current status of telemedicine in rural health; to explore the effects of telemedicine on access to care, practitioner isolation, and development of health care networks; to identify organizational factors that aid or impede the successful implementation of telemedicine systems; and to develop and refine data collection instruments for use in subsequent evaluations.

Health Professions

The Sixth National Sample of Registered Nurses will provide new data on the characteristics of the registered nurse workforce and will contribute to the data base underlying models for projecting national nurse supply and requirements. Data from the survey will be used by the HRSA to guide nurse education programs and to prepare biennial reports to Congress on characteristics of the registered nurse population. Results will also be used by the nursing community, which looks to the Department to maintain and update this unique data base.

The cost and location of training of physicians and others to provide primary care in ambulatory settings is a continuing concern. One project is obtaining quantitative data on the costs involved in training medical students and residents in over 30 ambulatory sites across the country.

A collaborative study with the Educational Commission on Foreign Medical Graduates and the Appalachian Regional Commission is tracking exchange students granted a J-1 visa to participate in graduate medical education in the United States. The HRSA portion of the project will examine the postresidency experience of exchange visitors who have secured waivers to remain in the United States following completion of training for the purpose of providing services in an underserved area.

New Directions for Evaluation

As in FY 1996, the HRSA will emphasize performance measurement, managed care, and such primary care programs as CMHC's. Care for mothers and children (exemplified by Healthy Start initiative), HIV/AIDS (including new approaches to financing and delivering care tested through the Special Projects of Regional and National Significance), and strengthening of the health care infrastructure at State and local levels will also be major areas of attention. Ongoing and new efforts that involve collaboration with other operating divisions and with such external entities as foundations and the American Hospital Association will also be continued or developed. Finally, the HRSA will ensure the further integration of evaluation with strategic planning and budget decisions and presentation, as called for in GPRA.