Performance Improvement 2008. Appendix D - glossary of evaluation terms, terms-of-art used by some offices or agencies, and acronyms for the organizational units of HHS.


The glossary, new with this 2008 issue of the report, provides some assistance to readers who may be unfamiliar with the terms and acronyms found in the report.

1915(c) waivers – refers to section 1915(c) of the Medicaid program allowing the Secretary of HHS to waive certain program requirements in the law. Waivers permit States greater flexibility to target program eligibility and provide home and community based services for the disabled and/or elderly populations.

Accountability – responsibility for the expenditure, administrative and programmatic activities that occur in organizational units over which one has formal authority.

ACF – Administration for Children and Families.

AOA – Administration on Aging.

AHRQ – Agency for Healthcare Research and Quality.

ATSDR – Agency for Toxic Substances and Disease Registry.

Appreciative inquiry – Appreciative inquiry rejects the more traditional ´problem-focused´ approach and instead seeks to identify what is working well or opportunities for positive change.

Arrearages – in ACF, past-due child support payments.

Balanced Budget Act of 1997 (BBA) – designed to balance the federal budget by 2002. Among many other things, the Act contained major Medicare reforms.

Balanced Budget Refinement Act of 1999 (BBRA) – moved to reinstate some of the funding that the Balanced Budget Act (BBA) of 1997 had cut. It provided financial relief to Medicare and

Medicaid providers and State Children´s Health Insurance Programs (SCHIPs).

Bayesian – a statistical approach that assesses the probability of a hypothesis being correct (for example, whether an association is valid) by incorporating the prior probability of the hypothesis and the experimental data supporting the hypothesis (Named after the Reverend Thomas Bayes, 1702-1761).

Benchmark – standard or point of reference (often some standard of best practices) against which program processes or outcomes can be compared.

Best practices – program models or activities for which effectiveness in achieving specified goals or objectives has been demonstrated or suggested, across a number of implementations and evaluations.

Benefits Improvement and Protection Act of 2000 (BIPA) – provided $35 billion over a 5-year period to hospitals, nursing homes, managed care plans, home health agencies, hospices, and DME providers to reinstate some of the reimbursements that the Balanced Budget Act of 1997 cut.

Bias – systematic distortion in a measurement instrument which results in data that tend to be either too high or too low in relation to the true value of a measure.

Biologic – a virus, serum, toxin, antitoxin, vaccine, blood, blood component or derivative, allergenic product, or other similar product used to prevent, treat or cure disease or injury.

Capitation – the system of payment for each customer served, rather than by service performed.

Case studies – methods of inquiry that focus on intensive data collection and analysis that investigates only a few units of analysis.

CCF – see Compassion Capital Fund.

CDC – Centers for Disease Control and Prevention.

Charge compression – the tendency of hospitals to hospitals tend to markup high cost items less than they markup low cost items (the term is used by the Centers for Medicare and Medicaid Services).

Clinical efficacy – In a medical context it indicates that the therapeutic effect of a given intervention (e.g. intake of a medicine, an operation, or a public health measure) is acceptable.

CMS – Centers for Medicare and Medicaid Services.

Commissioned Corps – a force of more than 6,000 well-trained, highly qualified public health professionals dedicated to protecting, promoting, and advancing the health and safety of the Nation. The Surgeon General implements Corps policy and manages operations of the Corps including training and assignment of officers, deployment of special response teams to public health emergencies, and allocation of officers to underserved communities and populations. Commissioned Corps officers serve throughout Office of Public Health and Science (OPHS), in assignments across the Department of Health and Human Services, as well as in other agencies and programs, providing essential public health leadership and fulfilling service roles.

Community-Level Social Indicator – a social indicator that can be collected, reported and meaningfully interpreted for geo-political units such as neighborhoods, towns or cities, metropolitan areas, or regions (see also, social indicator).

Community-Level Social Indicator System – a compilation of community-level social indicators based on data from one or more sources such as archival, surveillance, or administrative data developed for other purposes and, in some cases, other information including data developed from surveys implemented specifically for tracking local indicators.

Comparison group, control group – people who are not exposed to the program and who are compared to the program group.

Compassion Capital Fund (CCF) – Federal grant opportunities––for the purpose of organizational capacity building––that are of interest to faith-based and community groups. Managed by the Administration for Children and Families (ACF).

Cost-benefit (or benefit-cost) ratio – the ratio of total discounted program benefits to total discounted program costs.

Cost-benefit analysis – process of comparing values of all benefits less those of related costs when benefits can be valued in dollars the same way as costs. A cost-benefit analysis is performed in order to select the alternative that maximizes the benefits of the program.

Cost-effectiveness analysis – comparison of the relative costs and benefits of two or more approaches to a problem.

Cost-utility analysis – a form of cost-effectiveness analysis of alternative interventions in which costs are measured in monetary units and outcomes are measured in terms of their utility, usually to the patient, e.g., using QALYs.

Diabetes – The World Health Organization recognizes three main forms of diabetes mellitus: type 1, type 2, and gestational diabetes (occurring during pregnancy), which have different causes and population distributions. While, ultimately, all forms are due to the beta cells of the pancreas being unable to produce sufficient insulin to prevent hyperglycemia, the causes are different. Type 1 diabetes is usually due to autoimmune destruction of the pancreatic beta cells. Type 1 diabetes is usually diagnosed in children and young adults, and was previously known as juvenile diabetes. In type 1 diabetes, the body does not produce insulin. Insulin is a hormone that is needed to convert sugar (glucose), starches and other food into energy needed for daily life.

Type 2 diabetes is characterized by insulin resistance in target tissues. This causes a need for abnormally high amounts of insulin and diabetes develops when the beta cells cannot meet this demand. Gestational diabetes is similar to type 2 diabetes in that it involves insulin resistance; the hormones of pregnancy can cause insulin resistance in women genetically predisposed to developing this condition.

Diagnosis-Related Groups (DRG) – classification system that groups patients according to diagnosis, type of treatment, age, and other relevant criteria. Under the prospective payment system, hospitals are paid a set fee for treating patients in a single DRG category, regardless of the actual cost of care for the individual.

Discounted, discounting – process of determining the net present value of a dollar amount of costs or benefits.

Discount rate – rate of interest used in discounting costs and benefits, that is, converting all costs and benefits over the life of the policy, program, or project into net present values.

DMEPOS – Medicare durable medical equipment, prosthetics, orthotics, and supply suppliers

DRG – see Diagnosis Related Groups.

Drug risk-adjustment – the process by which the Centers for Medicare and Medicaid Services (CMS) modifies medical insurance payments to drug insurance plans for prescription drugs to account for known differences in health status of the enrollees served––to take into account the higher costs of individuals who have medical conditions which require greater use of pharmaceuticals––in order to neutralize incentives drug plans have to enroll only healthy individuals with low drug utilization in order to maximize their profits.  

Dual eligibles – persons entitled to Medicare (Part A and/or Part B) and who are also eligible for Medicaid.

Effectiveness – extent to which the observed outcomes are consistent with the intended objectives.

Enrollment growth – increases in the number of people eligible for and receiving benefits.

Environmental factors – factors in the surroundings of a program that may have an effect on it and on the intended outcomes.

Evaluability Assessment – systematic process used to determine the feasibility of a program evaluation. It also helps determine whether conducting a program evaluation will provide useful information that will help improve the management of a program and its overall performance.

Evidence-based – a philosophy that emphasizes the importance of using defensible evidence as the basis for actions/decisions ("evidence-based decision making is sometimes associated with performance management).

Experimental/impact design – a research design involving one or more treatment (program) and control groups, where program and control participants are randomly assigned to the groups, ensuring that the groups are equal except for the program itself.

FDA – Food and Drug Administration.

Fee-For Service – a system in health care by which particular services are paid for individually rather than provided as part of a comprehensive plan

FFS – see Fee-For-Service.

Focus group – group of persons selected for their relevance for a particular evaluation question.

Formative evaluation -- an assessment of program efforts prior to their completion for the purpose of improving the efforts; examples include developmental and implementation evaluation.

HCBS – see Home and Community-Based Services.

HIE – see Health Information Exchange.

HIT – see Health Information Technology.

HRSA – Health Resources and Services Administration.

Home and Community Based Services (HCBS) – programs that offer different choices to some people with Medicaid; if a person qualifies, they receive care in their home and community so they can stay independent and close to family and friends. HCBS programs help the elderly and disabled, mentally retarded, developmentally disabled, and certain other disabled adults. These programs give quality and low-cost services.

Health Information Exchange (HIE) – the mobilization of healthcare information electronically across organizations within a region or community.

Health Information Technology (HIT) – software and infrastructure used in the clinical practice of medicine to support documentation, storage and exchange of patient data.

Health Maintenance Organization (HMO) – form of health insurance combining a range of coverages on a group basis. A group of doctors and other medical professionals offer care through the HMO for a flat monthly rate with no deductibles. However, only visits to professionals within the HMO network are covered by the policy. All visits, prescriptions and other care must be cleared by the HMO in order to be covered. A primary physician within the HMO handles referrals.

HEDIS – Health Plan Employer Data and Information Set, a list of about 60 standardized performance measures developed and maintained by the National Committee for Quality Assurance (NCQA) in Washington, DC. Currently, most health plans report their results directly to NCQA as well as to their larger customers.

HHS – standard acronym used in this report designating the United States Department of Health and Human Services.

HMO – see Health Maintenance Organization.

IHS – Indian Health Service.

Impact evaluation -- Focuses on the long-range results of the program or project, and changes or improvements as a result (for e.g., long-term maintenance of desired behavior, reduced absenteeism from work, reduced morbidity and mortality).  Because such evaluations are the most comprehensive and focus on long-term results of the program and changes or improvements in health status, they are the most desirable.  However, impact evaluations are rarely possible because they are frequently costly and involve extended commitment.  Also, the results often cannot be directly related to the effects of a program, project, or activity because of other (external) influences on the target audience, which occur over time. 

Information Technology (IT) – a general term encompassing the use of hardware, software and services to create, store, retrieve, transfer, process and present information. IT projects typically involve the introduction or enhancement of systems or technology to meet a particular business need.

Input Measure: measure of what an agency or manager has available to carry out the program or activity to produce an output or outcome.

Interoperability – ability of two or more systems or components to exchange information and to use the information that has been exchanged.

Interoperable health information exchange – the ability of two or more health information systems to exchange and use health-related information.

Logic model – tool for planning, implementing, and evaluating programmatic efforts, by mapping out the theory or rationale that supports what is being done. Logic models typically tie together: long-term problem(s) to be addressed; factors that must be addressed that contribute to the problem(s); strategies and practices, and supporting resources, that can be mobilized to address the factors and the problems; and measurable impacts and outcomes that can be expected to result from implementing the strategies and practices – as these relate to the long-term problem(s).

Medicaid – joint Federal and State program that helps with medical costs for some people with low incomes and limited resources. Medicaid programs vary from state to state, but most health care costs are covered if a person qualifies for both Medicare and Medicaid.

Medicaid Information Technology Architecture (MITA) – framework developed by CMS to help States modernize their Medicaid information systems.

Medicare – Federal health insurance program for persons 65 years of age or older, the disabled, and those with end-stage renal disease.

Medicare Advantage – Medicare program that gives a person more choices among health plans. Everyone who has Medicare Parts A and B is eligible, except those who have End-Stage Renal Disease (unless certain exceptions apply). Medicare Advantage Plans used to be called Medicare + Choice Plans.

Medicare Economic Index (MEI) – index often used in the calculation of the increases in the prevailing charge levels that help to determine allowed charges for physician services. In 1992 and later, this index is considered in connection with the update factor for the physician fee schedule.

Medicare+Choice (M+C) – public or private entity organized and licensed by a State as a risk-bearing entity (with the exception of provider sponsored organization receiving waivers) that is certified by CMS as meeting the M+C contract requirements.

Medicare physician payment update process – annual revisions to the amounts physicians are reimbursed, based on a statutory formula. It utilizes a comparison between target spending for Medicare physicians´ services and actual spending.

Medicare Prescription Drug, Improvement, and Modernization Act (MMA) of 2003 –statute that updated, revised portions of the Medicare program, allowed for increased benefits, and provided seniors with prescription drug benefits and more choices in health care.

Medicare private fee-for-service (PFFS) – type of Medicare Advantage Plan in which a person may go to any Medicare-approved doctor or hospital that accepts the plan´s payment. The insurance plan, rather than the Medicare Program, decides how much it will pay and what the beneficiary pays for the services they get. The patient may pay more or less for Medicare-covered benefits and may receive extra benefits that the original Medicare plan doesn´t cover.

MedPAR – the Medicare Provider Analysis and Review file, a data file containing records for all Medicare beneficiaries who use hospital inpatient services. The records are stripped of most data elements that would permit identification of beneficiaries.

Meta-analysis, meta-evaluation – systematic analysis of a set of existing evaluations of similar programs in order to draw general conclusions, develop support for hypotheses, and/or produce an estimate of overall program effects.

MMA – see Medicare Prescription Drug, Improvement, and Modernization Act of 2003.

MMA benefit stabilization funds – a Medicare+Choice organization can request that an excess amount be withheld and reserved in a stabilization fund. This fund is used to stabilize and prevent undue fluctuations in additional benefits required.

National Health Plan Collaborative – in NIH, a project to bring together major health insurance companies, in partnership with organizations from the public and private sectors, to identify ways to improve the quality of health care for racially and ethnically diverse populations.

Needs assessment – study that measures the nature and extent of the need for a program, either before a new program is developed or during its lifetime.

NIH – National Institutes of Health.

Non-experimental design – only one group receiving the intervention is being observed or studied without the use of a comparison group to control for outside factors.  Thus, such designs generally involve less data collection and are easier to plan and carry out.  They typically involve observing and/or collecting all relevant data–including data on key performance measures–on participants at selected points in time during the project.  Examples of such design include, but are not limited to, case studies, structured interviews, surveys, pre-/post-tests, ethnographic studies, and document reviews. 

Objectives – these statements are derived from the program goals and explain how the program goals will be accomplished. Objectives are well-defined, specific, quantifiable statements of the program's desired results and they should include the target level of accomplishment, thereby further defining goals and providing the means to measure program performance.

Office of Child Support Enforcement (OCSE) – located within ACF; seeks child care payments due from absent parents.

Office of Minority Health (OMH) – located within OPHS, funds health care services and research relating to minorities.

Office of Inspector General – although many federal agencies have an Inspector General, when used in this report, the Office of Inspector General or the Inspector General always refers to the Inspector General of the United States Department of Health and Human Services. Inspectors General have a dual reporting responsibility – their reports are not cleared by the Department in which they work and through which they receive their funding and staffing. But, they submit their reports directly to Congress. This is different from all other components which must clear their work through senior departmental officers before being released to Congress or the public.

OPHS – Office of Public Health and Science.

OSCAR – Online Survey, Certification and Reporting data system, provides staffing data for all U.S. nursing homes that Medicare and/or Medicaid certifies. State survey and certification agencies collect the data, which are part of the annual nursing home certification and recertification process. Each facility completes a standardized form about the facility characteristics, e.g., number of beds, affiliation, etc., resident characteristics, e.g., limitations, chair bound, etc., and staffing levels. State surveyors review the form and enter the data into the OSCAR database. State surveyors also visit each facility and decide whether the facility meets each standard.

Outcome evaluation – used to obtain descriptive data on a program or project and to document (typically) short- and intermediate-term results. Task-focused results are those that describe the output of the activity. Shorter-term results describe the immediate effects of the project on the target audience.  Information from such evaluation can show results such as knowledge and attitude changes, short-term or intermediate behavior shifts, and policies initiated or other institutional changes.

Outcome Measure – measure of an event, occurrence, condition, or result of a program or project that indicates achievement of objectives and goal(s); this type of measure is used to measure the success of a program, project, or system (e.g., the percentage of people who do not get influenza).  

Output Measure – measure of a product, service, or result of a particular activity (e.g., number of people vaccinated with the influenza vaccine, number of personnel trained; number of phone calls processed by the OMH Resource Center); this type of measure provides information about the activity, not the success in achieving the objectives and goals of the program/project.

Part B – Medicare medical insurance that helps pay for doctors' services, outpatient hospital care, durable medical equipment, and some medical services that are not covered by Part A.

Performance measurement – process of designing and implementing quantitative and qualitative measures of program results, including outputs and outcomes.

Population – group of people that may or may not be from the same geographic area, who receive services from public sector or nonprofit organizations.

PPO – see Preferred Provider Organization.

Preferred Provider Organization – plan that (a) has a network of providers that have agreed to a contractually specified reimbursement for covered benefits with the organization offering the plan, (b) provides for reimbursement for all covered benefits regardless of whether the benefits are provided with the network of providers, and (c) is offered by an organization that is not licensed or organized under State law as an HMO.

Probabilistic analysis – a way to define statistical distributions for input parameters (e.g. joint orientation, shear strength, water level), to account for uncertainty in the values of input parameters. When the analysis is computed, this results in a safety factor distribution from which a probability of failure is calculated.

Process/implementation evaluation – examine the tasks and procedures involved in implementing a program or activities, including the administrative and organizational aspects of, and delivery procedures involved in, the efforts.  Such evaluations enable monitoring to ensure feedback during the course of the program or project.

Process measure – measure of the procedures, tasks, or processes involved in implementing program or project activities to produce an output or outcome.

Program – group of individual (grantee) projects, unified by a set of goals, health issues of focus, recommended types of activities, eligible grant recipients, etc.

Project – an individual project (grantee), usually within an overall program, addressing one or more specific target populations or communities, and health issues.

Program components – major clusters of activities in a program that are intended to drive the process of producing outcomes.

Program effectiveness – extent to which the program achieves its intended outcomes.

Program evaluation – systematic process for gathering and interpreting information intended to answer questions about a program.

Program outcomes – what the program appears, through a process of measurement, to have achieved.

Public Health Service Commissioned Corps – see Commissioned Corps.

Public health surveillance – the ongoing, systematic collection, analysis, interpretation, and dissemination of data regarding a health-related event for use in public health action to reduce morbidity and mortality and to improve health.

QALY – see quality-adjusted life-years.

Quality-adjusted life-years – a way of measuring both the quality and the quantity of life lived, as a means of quantifying in benefit of a medical intervention. It is a method of estimating utility that assigns a preference weight to each health state, determines the time spent in each state, and estimates life expectancy as the sum of the products of each preference weight and the time spent in each state.

Quasi-experimental – research designs that do not involve random assignment to program and control groups but do include comparisons that make it easier to sort out the cause and effect linkages that are being tested.

Randomized controlled trials – see experimental design, randomized experiments.

Randomized experiments – research designs that involve randomly assigning units of analysis (usually people) to program and control groups.

Random sample – sample that is selected at random from the population, where each member of the population has an equal or known chance of being selected, which enables the research results to be generalized to the whole population.

Relevance – extent to which the objectives of the program are connected to the assessed needs.

Reliability – extent to which a measurement instrument produces consistent results over repeated applications.

Representative sample – when the characteristics of a sample (demographic characteristics, for example) match those same characteristics for the population, the sample is said to be representative.

Resource and Patient Management System – an integrated data system for management of clinical, business practice and administrative information in Indian Health Care facilities.

RPMS – see Resource and Patient Management System.

SAMHSA – Substance Abuse and Mental Health Services Administration.

SCHIP – State Children´s Health Insurance Program.

Social Indicator – a measure reflecting the status of the population (e.g., age range, income level, education attainment), and contextual influences (e.g., social, economic, ecological, and political influences) known to affect well-being at a particular time or over a period of time.

Social Security Administration (SSA) – Federal agency responsible for administering the Social Security program, which provides economic relief to citizens. The agency also plays important roles in public health, including disability determination and supporting electronic death registration systems.

Sonya Slifka Longitudinal Multiple Sclerosis Study – named for a woman with multiple sclerosis whose family is supporting the study. The study has followed over 2,000 people with MS for five years.

Stakeholders – all persons, agencies and organizations with an investment in the health and well-being of the community and the local public health and welfare systems.

Statistical significance – when the analysis of data results in statistical significance, it means that the result is not likely to have occurred by chance.  It confirms a relationship or difference between variables.

Summative evaluation – look at a combination of measures and conclusions for larger patterns and trends in performance, to assess, in summary, whether the program or project overall did what it was designed to do.  Summative evaluations are primarily retrospective, document evidence, and show results and achievement.  Examples of summative evaluations include outcome and impact evaluations, cost-effectiveness and cost-benefit analyses, and meta-analyses. (see also Formative evaluation)

Sustainable Growth Rate (SGR) – system for establishing goals for the rate of growth in expenditures for physicians' services.

Treatment groups – persons who are provided with a program or some other intervention that is being evaluated.

Triangulation – process of collecting data to answer an evaluation question from a variety of sources and/or using a variety of measurement procedures.

Type 1 (or Type 2 or gestational) diabetes – see Diabetes.

Uniform Data Set (UDS) -- systematic data reporting system recently developed for all OMH-funded activities that organizes data collection and reporting by type of activity conducted. The UDS is an Internet-based system. 

UAF – see Update Adjustment Factor.

Update Adjustment Factor – a payment revision method that includes the sustainable growth rate and squares up the actual Medicare expenditures with targeted Medicare expenditures for the year.

Utility – in economics, utility is a measure of the relative satisfaction from or desirability of consumption of goods.

Utilization – extent to which the program evaluation process and results (findings, conclusions, and recommendations) are deemed by stakeholders to be useful to them.

Validity – extent to which a measuring instrument measures what it is intended to measure.

Variable – an observable characteristic that we expect will be affected by one or more independent variables – in most evaluations, the observed outcomes are dependent variables.

View full report


"PerformanceImprovement2008.pdf" (pdf, 1.29Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®