Conference Report: Increasing Organ Donation and Transplantation: The Challenge of Evaluation. Overview of Evaluation Methods

04/02/1998

While the organ donation community has been very active in implementing diverse approaches for increasing organ donation, there has not been a concerted effort to determine the best methodologies for evaluating these activities. Although many activities have been evaluated at some level, e.g., counting donor card signatures, there is relatively little understanding of how these activities relate to the goal of increasing the number of organs available for transplantation. Well-planned and methodologically sound evaluations, coupled with program timeframes and goals, provide the cornerstone for understanding program impact. Rigorous evaluations of activities designed to increase organ donation will better inform resource allocation among alternative and complementary programs.

There are three essential components of a successful evaluation: 1) an activity to evaluate that is pursuant to the ultimate goal of the program, 2) tested performance indicators, and 3) rigorous evaluation methodology. Exhibit 1 provides an overview of these elements and how they inform and influence each other.

Exhibit 1: Overview of Evaluation Planning
 Exhibit 1: Overview of Evaluation Planning

Despite their diversity in design, costs, performance indicators, and other factors, the aim of evaluation methods in use today is essentially the same, i.e., to assess the effect of an intervention on one group compared to the effect of a different intervention (or no intervention) on a similar group. All evaluations have a control or comparison group, whether explicit or implied. Exhibit 2 depicts a basic framework for considering the methodological rigor of evaluation types, their respective study elements, and examples of organ donation activity evaluations.

The evaluation types in Exhibit 2 are listed in rough order of most to least scientifically rigorous for internal validity, i.e., for accurately representing the causal relationship between an intervention and an outcome in the particular circumstances of a study. This ordering of methods assumes that each study is properly designed and conducted; a poorly conducted large randomized controlled trial (RCT) may yield weaker findings than a well-conducted study that is lower on the design hierarchy. This list is representative; there are other variations of these methodologic designs and some investigators use different terminology for certain methods. However, such a methods hierarchy can help programs to weigh different evaluation options, and implement an evaluation strategy that is both methodologically rigorous and feasible.

 

Exhibit 2: General Strengths and Weaknesses of Evaluation Types
 Exhibit 2: General Strengths and Weaknesses of Evaluation Types

The selection of performance indicators affects the design of an evaluation, the resources required to conduct the study, and the utility of the results. For example, measuring changes in public awareness of organ donation activities may not ultimately provide insight on the impact the activity had on donation rates. Because of the inability to predict who might become a potential organ donator before the occurrence of a traumatic event, any activity targeted at the general public must cast a wide net in order to reach those few people who will become potential organ donors. It is difficult to attribute any measured change in organs retrieved to a given population-based program with acceptable levels of certainty. The evaluation of population-based programs requires careful selection of performance indicators based on the goals and resources of the organization conducting the evaluation.

To overcome limitations in measuring program effectiveness of actual organ retrieval, the organ donation community has used three related sets of performance indicators, each with varying degrees of separation from the actual intended outcome of organ donation. The three types of measures are shown in Exhibit 3 as concentric circles, illustrating their relative proximity to the donation event. Pre-event measures are used to gauge effectiveness in increasing organ donation, e.g., willingness to donate, before an actual donation opportunity arises. Post-event measures are used to measure five crucial steps in organ retrieval after a potential donor situation has occurred, including donor identification rate, referral rate, request rate, consent rate, and retrieval rate. Donation rates are the most direct measure of the success of programs to increase the number of organs available for transplantation. Exhibit 3 depicts the relative sizes of the populations captured in pre-event, post-event, and donation measures relative to the U.S. population.

Exhibit 3: Sample Performance Indicators and Proximity to Donation
Exhibit 3: Sample Performance Indicators and Proximity to Donation

The evaluation methods described above provided a framework for discussion among the invited presenters, expert panel members, and audience during the two-day conference. The descriptions included in this report of programs and findings shared by presenters at the conference are derived from their respective abstracts and presentations, and were not verified by HHS or The Lewin Group.