Performance Improvement 1997. Office of the Assistant Secretary for Planning and Evaluation


MISSION: To provide analytical support and advice to the Secretary on policy development and to assist the Secretary with the development and coordination of Departmentwide program planning and evaluation activities.

Evaluation Program

The Office of the Assistant Secretary for Planning and Evaluation (ASPE) functions as a principal advisor to the Secretary on policy development and, in this capacity, conducts a variety of evaluation and policy research studies on issues of national importance. ASPE is also responsible for Departmentwide coordination of legislative, planning, and evaluation activities. In its evaluation coordination role, ASPE has the following tasks:

  • Provide annual guidance to all HHS agencies and staff offices regarding evaluation priorities, procedures, and review requirements.
  • Review evaluation priorities proposed by the HHS agencies, providing advice about the focus or method of proposed projects, identifying opportunities for collaboration, and ensuring effective use of resources.
  • Prepare planning and summary reports on evaluation activities as required by Congress.

Through the departmental evaluation planning process, ASPE has the capacity to identify cross-cutting program or policy issues of particular concern to the Secretary and specific program and policy areas not covered by the HHS agency evaluation plans. In these instances, ASPE initiates evaluations or collaborates with the agencies to conduct evaluations or policy assessments. For example, in recent years ASPE has initiated projects to develop cost estimates for health financing issues in general, and specifically for the Medicare and Medicaid programs; the effects of managed care expansion on public health infrastructure; welfare-to-work approaches; long-term care alternatives; and studies evaluating alternative services for children at risk of harm from drug abuse, crime, child abuse, and other pathologies.

Another continuing ASPE evaluation objective is to support and promote the development and improvement of data bases that HHS agencies and ASPE use to evaluate health care programs and health trends. For example, ASPE has been the major initiator on collaboration with the National Center for Health Statistics at the Centers for Disease Control and Prevention (CDC) of the first comprehensive survey of people with disabilities in the United States. The first component of these new data was completed in FY 1996, and national prevalence data on disability are now available. ASPE provides support to the HHS Data Council, which is charged with integrating key national surveys, such as linking health status indicators with indicators of well-being. The Department needs more comprehensive data sources to assess anticipated transformations in health and human services.

Finally, ASPE uses evaluation funds to promote effective use of evaluation-generated information in program management and policy making. The latter is accomplished through the dissemination of evaluation findings and other activities, such as providing technical assistance to agencies in the development of performance measures.

Summary of Fiscal Year 1996 Evaluations

During FY 1996, ASPE completed a wide range of studies and reports on a number of issues, providing information useful to the Secretary and the HHS divisions for purposes of program planning and budget and legislative development.

Health Policy

  • Health Care Technology Assessment--This project, also described in chapter II, contains current information on the demand for technology assessment. The study examined the performers of technological assessment, the methods of conducting assessments, the uses of the results, and the unmet needs that might be met by further cooperation between the public and private sectors of health care. In the past 5 years, the evaluation of clinical practice for its cost-effectiveness has assumed greater market value. The use of technology assessment in the managed care sector is primarily to support coverage decision making. This study also examined the appropriate role of the Federal Government in technology assessment and the appropriate division of labor between the public and private sector.
  • Information to Guide Physician Practice--The purpose of this study was to understand the relationship between the increasing amounts of information that are being produced for physicians and the types of information that physicians actually use. In particular, this study assessed physician use of computerized health and medical information sources available, such as those on the World Wide Web and the Internet, and how access to such information influences medical practice. Among the findings are the following: (1) The wealth of information available to physicians causes them to aggressively screen information sources; this screening process depends significantly on the characteristics of the physician, on the financial and legal practice environment, and on the source and type of information being provided; (2) technology advances are making it more feasible for rural physicians to communicate with colleagues, but financial constraints prevent them from taking full advantage of resources; (3) potential barriers to use of the On-line Medical Networked Information resources include the fact that information resources are not of enough value to justify the time needed to access them.
  • Assumptions Underlying Insurance Participation Modeling--This report provides background information to analyze policies designed to expand health insurance coverage, especially models that seek to estimate the impact of incremental health care reforms on the size and composition of the uninsured population. The study found that the number of uninsured persons increased from 30.5 million in 1979 to 39.6 million in 1994 (about one-third due to increases in the population); disagreement over estimates of the numbers of uninsured persons can be traced to several factors, including different survey methods and disparities between point-in-time and longitudinal estimates. The report concludes that the major contributor to the problem of uninsured workers lies in the area of insurance market supply, but the lack of demand on the part of some workers also contributes to the problem.

Disability, Aging, and Long-Term Care Policy

  • Board and Care Homes--Nearly one million elderly or disabled individuals are dependent on personal services and supervision provided by board and care homes. ASPE carried out an analysis of the effect of State regulation on the quality of care provided in board and care homes in 10 States. The study, also described in chapter II, found that board and care residents are significantly older and more frail than a decade ago, making it a more complex challenge for board and care operators and staff. The study findings indicate that States with extensive regulatory and licensing requirements are more able to cope with this challenge on a number of dimensions including greater availability of supportive services in licensed homes, more operator training, and lower use of psychotic drugs and inappropriate medications.
  • Subacute Care: Market Analysis, Cost, and Quality--There has been a huge increase in health care services provided in postacute care settings, as providers struggle to reduce inpatient hospital care. This exploratory study, also presented in chapter II, examined the extent to which subacute care appears to be a low-cost, appropriate alternative to hospital care. It found that even "state-of-the-art" subacute care programs had difficulty documenting their success in reducing costs and improving outcomes compared with traditional nursing home care. The study concludes that systematic evaluation is needed before the cost-effectiveness of subacute care is demonstrated.
  • Research and Demonstration Strategy to Test Models for Integrating Acute and Long-Term Care for Children With Disabilities--This study documented "state-of-the-art" practices in developing managed care for children with disabilities and how States are coordinating managed care strategies with their Title V programs for children with special needs. The study concludes that there is little operational experience to guide States in developing managed care approaches for children with special needs, particularly programs that integrate acute and long-term care services. It also found that Medicaid managed care providers could benefit from the experience of Title V providers in serving special needs children as they create new managed care models.
  • Americans With Disabilities--This project provided funds to the Census Bureau to analyze the 1990 Census questions on disability and to create a national baseline on disability data. Data were collected for the population age 16 and over on several types of disabilities: limitations in work, ability to work, mobility limitations, and self-care limitations. The tabulations are unique disability data for States, counties, and large metropolitan areas, including persons with disabilities living in the community and in institutions. The results--which include tables, graphs and maps--are available on the HHS home page available through the Internet at
  • Interdisciplinary Education and Training of Professionals Caring for Persons With Disabilities--People with disabilities generally use more health care services than people without disabilities and often require a range of both health and human service professionals. If managed care arrangements are to serve such populations effectively, providers must learn to integrate the services of a number of different professional disciplines. This study identified a number of promising programs where professionals receive interdisciplinary instruction and practicum experience in providing care to people with disabilities. It also found that few training programs address the needs of managed care organizations and that managed health care providers often lack the specialized knowledge and skills to serve people with significant disability.

Human Services Policy

  • Alcohol and Other Drugs Treatment for Parents and Welfare Recipients: Outcomes, Costs, and Benefits--While alcohol and drug treatment have been shown to be effective for reducing crime and health care costs, very little is known about this relationship for welfare clients. This study analyzed the outcomes, benefits, and costs of substance abuse treatment for two overlapping groups: parents of children under 18 years of age and recipients of public income support such as Aid to Families with Dependent Children (AFDC). The study found that many people in treatment are both parents and welfare recipients. The data documented that the treatment benefits for these individuals exceeded the treatment costs.
  • Coordinated Community Responses to Domestic Violence in Six Communities: Beyond the Justice System--Awareness of domestic violence has extended past the justice and domestic violence service systems to health care providers, child welfare and substance abuse service agencies, churches, and business. This study examined the coordinated responses to domestic violence in six U.S. communities. The study found that a strong community response requires well-coordinated policies and consistent application among community criminal justice agencies. A need was found for health care providers to develop screening protocols to identify women who are battered and to develop a system for referring victims to the most appropriate agency.

Special Populations

In addition, ASPE completed two evaluation projects during FY 1996 that focused on the health needs of special populations. The first project was an evaluation design effort with the Office of Minority Health in the Office of Public Health and Science, titled Family and Community Violence Prevention Program: Technical Assistance in Program Evaluation and Capacity Building Assessment. The project produced a design for the Family and Community Violence Prevention Program, created by the Male Minority Consortium, a group of 19 historically black colleges and universities (HBCU). The report recommends several evaluation strategies that individual HBCU's can employ to assess the program, such as measuring attitudes and beliefs about violence, identifying communication channels and influences, and measuring community participation.

The second project was an effort to synthesize knowledge about the demography of American Indian and Alaska Native populations, with a focus on the major health problems they face and their use of health care services. The report, Changing Numbers, Changing Needs: American Indian Demography and Public Health, documents the difficulties in tracing demographic and public health trends for Native American populations. The report synthesizes data on population growth since the turn of the century, including information on fertility rates; reduction of infant mortality; persistence of high death rates for American Indian youths; declines in infectious diseases; and increases in rates of chronic disease, injuries, and alcohol and drug abuse. One analysis presented in the report showed that although IHS facilities are well suited to serve rural Native American population, problems continue with adequate access to care in rural areas and sufficient levels of budgeted resources.

Evaluations in Progress

Health Policy

  • Monitoring Changes in Health and Human Service Programs--With the rapid changes in the health and human services systems, ASPE is studying the effects of State Medicaid waivers and changes in the State welfare laws on HHS programs such as Medicaid, foster care, and child support enforcement. ASPE evaluations will help define key measurements with which to monitor the effects of both program changes and changes in the environment, such as the impact of managed care on nonelderly individuals with disabilities, a project being done in collaboration with the Health Care Financing Administration (HCFA), and the effect of welfare-to-work programs on the well-being of children, a project being done in collaboration with the Administration for Children and Families (ACF) and foundations.
  • Understanding Private-Sector Changes--ASPE's objective is to understand the private-sector environments in which HHS programs operate and to be able to describe and monitor the ongoing changes. A joint project with HCFA on managed care outcome measures and collaborative efforts with some of the Public Health Service (PHS) agencies on health promotion, prevention services, and benefits coverage in a changing private-sector marketplace were initiated during FY 1996. ASPE also continues to analyze data and information on health insurance coverage on an intramural basis.
  • ASPE Data Initiative--This is a series of projects directed at filling gaps in the data HHS needs to analyze and monitor the health and well-being of the population, particularly the potential policy changes in health and human service programs. These gaps include State-level data; improved data on providers and the supply of health services; data that allow the measurement of child well-being; and better linkages between health and human services data.
  • Performance Data--The Federal grants relationships to States in public health are evolving into outcomes-based performance management. A health outcomes-based monitoring approach requires data system development. ASPE is coordinating three projects with HHS agencies to accomplish this data development. One, with the National Academy of Sciences, is working to identify options for a performance-based approach that States and others want to achieve. The project will recommend specific steps that can be taken to improve State and local community data capabilities. A related project is developing approaches to obtain comprehensive baseline and trend data on public health infrastructure. A third study is building on a Robert Wood Johnson Foundation project to assess the quality of data that States collect and determine whether there is a foundation on which to build a multistate health data system that can be shared among States to improve health policy decision making.
  • HHS Agency-Related Priority Health Services Evaluations--During FY 1996 ASPE has collaborated with HHS agencies on several program evaluations and technical assistance projects in program areas such as CDC's domestic violence initiatives, Indian Health Service alcohol treatment programs, the FDA's food additive review process, Medicare and Medicaid payment policies, and the Minority Male Consortium for Violence Prevention with the Office of Minority Health.

Disability, Aging, and Long-Term Care Policy

  • Managed Care and People With Disabilities--Managed care can provide unique opportunities, but it also has potential pitfalls for people with disabilities. ASPE has projects under way to determine the impact of managed care on access, quality, and satisfaction for people of all ages who experience disabilities. Of particular interest are two policy questions. First is a concern about the impact of extending Medicaid managed care to the population eligible for Supplemental Security Income. Second, the impact of managed care on elderly persons with physical and cognitive impairments needs to be examined, and a comprehensive program for capitating acute and long-term care services for children with disabilities must be initiated and evaluated.
  • Assisted Living--ASPE has commissioned a national study of assisted living, or the residential settings that combine adapted housing, assistive technologies, personal assistance, and other supportive services for persons with disabilities. Assisted living is considered an important component of services to the aging American population and an effective response to the rising costs of nursing home care.

Human Services Policy

  • Evaluation of Family Preservation Services--ASPE and ACF are now examining family preservation programs aimed at preventing children from entering substitute care and reunification services to speed the return of children to their homes after they enter substitute care. Measures of program success will include prevention of placement of children into substitute care (for preplacement services), successful reunification (for reunification services), reduction of the recurrence of child abuse and neglect, improved child psychological well-being, improved child behavior, and improved family functioning.
  • Moving Welfare Recipients to Work--ASPE is also working with ACF to conduct a comprehensive, multiyear study of the Government's principal program for moving people off welfare and into employment--the Job Opportunities and Basic Skills Training (JOBS) Program. The evaluation addresses the long-term effects of different welfare reform approaches, including whether any approach significantly improves the economic circumstances of the most disadvantaged recipients; the effects of welfare-to-work programs on the children of welfare recipients; and the cost-effectiveness of different approaches.
  • Promoting Father Involvement--Three evaluations are looking at various aspects of paternal involvement, especially among fathers who are not living with their children. One project is examining the relationship between various child custody, visitation, and support payment patterns and the effect that payment and contact have on child well-being. The second is developing a design for the evaluation of community-based programs for vulnerable fathers. The third project will develop a theoretical framework to assist in understanding and implementing programs and activities to promote responsible fathering. The last two projects are being conducted in collaboration with ACF.
  • Trends in the Well-Being of America's Children and Youth--In FY 1995, ASPE provided funds for the first annual report on the health and well-being of America's children. The report presented up-to-date trends on how our Nation's children are faring, pulling together information on all available national trends in the lives of children and youth. This volume fills a crucial gap and is now an annual series to keep the Nation apprised of the well-being of its children and youth. ASPE is developing the second annual report, to be released in 1997.

In addition, human services policy assessments are under way to examine child welfare and domestic violence policies, the health and mental health of immigrant children, and factors in the growth and decline of AFDC caseloads.

New Directions for Evaluation

In upcoming years, ASPE will pursue the following evaluation priorities.

Impact of Program Transformation

The public health system and public health programs supported by HHS are an essential part of the health care safety net for the uninsured and are a vital element in protecting the health of all Americans. An increasing challenge for HHS agencies is to understand how health care system changes, such as managed care, interact with the dramatic changes taking place in welfare policies to affect the public health system, public health programs, and the clients those programs serve. In future studies, ASPE will compliment the efforts of the CDC, the Health Resources and Services Administration, and the Substance Abuse and Mental Health Services Administration by evaluating how health and welfare reforms create new challenges and responsibilities for the public health system and programs.

Managed Care and Disability

ASPE will continue to support evaluations to determine the effect of managed care on individuals with disabilities who have both health and long-term care needs. As mentioned above, ASPE, in collaboration with PHS and other HHS agencies, initiated a series of studies to identify and understand how State and community health care reform efforts are affecting people with disabilities. New studies will continue to examine the impact of selected State and community managed care initiatives on people with disabilities, including how such arrangements affect access to needed services, health status, well-being, and satisfaction. Investigations will also focus on developing better outcome indicators of health care quality for elderly persons with physical and cognitive impairments, including individuals with Alzheimer's disease and other dementias. In addition, a collaborative evaluation effort will be initiated with PHS and HCFA to investigate the effectiveness of new initiatives to integrate health and long-term care services.

Impact of Welfare Reform

ASPE will be initiating new studies in upcoming years to evaluate the combined changes taking place in health care delivery systems and in income supports for low-income children and families who often have special needs. One issue that will be studied is how to help substance abusers and victims of domestic violence whose entitlements and access to services will change under the terms of new welfare reform legislation. Under the Personal Responsibility and Work Opportunity Reconciliation Act, welfare recipients generally will be required to work to receive benefits and will face overall time limits on welfare receipt. In the past, many clients identified as substance abusers or victims of domestic violence were not required to participate in work activities. While some of these clients will be exempted under the new law, others must enter the workforce. Case managers need better tools to assess clients' barriers to sustaining employment and to assist these clients in addressing those barriers.

Other ASPE efforts to study the combined effects of changes in health care and income support policies on the public health system and its clients will focus on developing better State and national data on outcomes and the impact of the changes on the health and well-being of children and youth. Studies that address teen pregnancy prevention will build on work already undertaken and will examine how communities are addressing this issue as required in the new welfare law.

Quality-Related Studies

A major goal of the ASPE evaluation program, in relationship to HHS's ongoing responsibilities in the area of consumer protection and health quality standards, is both to understand the measurement of quality of care issues in health plans and providers and to improve the public availability and the comparability of such information. Working in conjunction with HCFA's Bureau of Health Standards and Quality, the Agency for Health Care Policy and Research, and other related PHS agencies, ASPE will develop projects to better measure health outcomes, relate health outcomes data to critical process of care data, evaluate the feasibility of developing new outcomes-based measures as part of administrative monitoring systems, and provide comparative-based health information to health consumers.

Improving Evaluation Data and Analytic Capability

HHS has taken a number of steps to strengthen its capacity in the data policy area and to work more effectively in collaborative data efforts with the health industry, the research community, and State and local governments. In the future, ASPE, working with the HHS Data Council, expects to initiate projects to support (1) development of State data strategies to monitor the impact of health and welfare transformations and to support evaluation of public health programs; (2) development of health data standards to support evaluation and monitoring in health care and public health programs; and (3) development and evaluation of HHS-wide information-privacy policy. These projects are largely responsive to new data policy requirements that the Kennedy-Kassebaum law places on HHS.