Performance indicators shape an evaluation, and the choice of performance indicators impacts the resources required to conduct the study and the utility of the study results. For example, measuring changes in public awareness of organ donation activities may not ultimately provide insight on what impact the study activity had on donation rates. Because it is impossible to predict who might become a potential organ donor before the occurrence of a traumatic event, any activity targeted at the general public must cast a wide net in order to reach those few people who will become potential organ donors. The complexity of an evaluation increases because there are a wide variety of reasons that potential donors do not become actual donors. These include not being identified as potential donors, caregivers not asking the families for permission to retrieve organs, families denying consent, and organs incorrectly deemed not transplantable (Gortmaker 1996). It is difficult to measure, with any statistical significance, the effect of a population-based program on the actual number of organs retrieved. The evaluation of population-based programs requires a careful selection of performance indicators based on the goals and resources of the organization conducting the evaluation.
To overcome limitations in measuring program effectiveness on actual organ retrieval, the organ donation community has used three related sets of performance indicators, each with varying
Exhibit 3: Sample Performance Indicators and Proximity to Donation
Source: Lewin, 1998
degrees of separation from the actual intended outcome of organ donation. The three types of measures are shown in Exhibit 3 as concentric circles, illustrating their relative proximity to the donation event. In addition, the US population is used as an example to depict the relative sizes of the populations captured in pre-event, post-event, and donation measures.
- Pre-event measures (represented by the outermost circle) are used to gauge effectiveness in increasing organ donation before an actual donation opportunity arises. The particular measures tend to be specific to the type of activity being performed. Compared to the other types of performance indicators, pre-event measures are most varied in nature, and are the farthest removed from the goal of measuring increased organ donation.
- Post-event measures (the middle circle) are commonly used in the organ donation community to measure the five crucial steps in organ retrieval after a potential donor situation has occurred. Chronologically, and with increasingly close ties to donation itself, these steps measure: 1) donor identification rate, 2) referral rate, 3) request rate, 4) consent rate, and 5) retrieval rate.
- Donation rates (the innermost circle) represent the most direct measure of the success of programs to increase the number of organs made available for donation.
A. Pre-Event Measures
Pre-event measures are most commonly used to evaluate activities that precede the trauma leading to brain death. Though frequently used as a proxy for program effectiveness on donation rates, they are process measures that serve as weak predictors of actual donation rates. A media campaign to increase the number of people who become organ donors when they renew their driver's licenses could be deemed successful if significantly more people signed donor cards than in months past. However, the link between the number of new licenses with organ donation approval and the donation rate is uncertain at best. A large follow-up study would be required to determine whether the process of signing a donor card had an impact on a family’s decision to donate organs. The link between pre-event measures and the primary goal of increasing organ donation is weak.
Many researchers are striving to improve pre-event measures by using these measures to assess the five stages of behavioral change that may lead to organ donation, i.e.: precontemplation, contemplation, preparation, action, and maintenance. These five stages of change were developed by Prochaska et al. (1983), originally in relation to smoking cessation programs. Studies in the areas of smoking cessation, mammography use, and weight loss programs have attempted to: 1) design measures for the stages of change, 2) determine if the stage of change correlates to success in the designated program, and 3) determine if activities can be implemented to improve cycling through the stages of change. These activities are evaluated relative to activities that were not specifically designed to promote cycling through stages of change. In the field of organ donation, these stages of change can be thought of as follows (Rohr, manuscript).
- Precontemplation: the person has not thought about donating organs.
- Contemplation: the person has thought about donating organs.
- Preparation: the person has made phone calls or requested information about organ donation.
- Action: the person has taken action to express his/her wishes about organ donation (e.g., signed a donor card, talked to a lawyer, discussed wishes with family).
- Maintenance: the person may occasionally remind or reinforce statement of wishes.
Multiple studies have indicated that familial consent for organ donation is one of the biggest barriers to donation. Other papers have shown that families are more likely to consent to donation if the deceased individuals have made their wishes known. Applying the stages of change methodology to measure progress towards action may be a useful indicator of outcomes. For example, stages of changes measures can be developed to determine whether a particular intervention moved people to sign donor cards and discuss organ donation with their families.
B. Post-Event Measures: Procurement Process Measures
Another group of broadly applicable measures relates to the various steps of the organ retrieval process: identification, referral, request, consent, and retrieval. Because these steps are more immediate to the act of donation, they are more meaningful, though still intermediate, indicators of the effectiveness of an organ donation activity. Also, because they are common to many different kinds of programs, they allow for more meaningful comparison across programs than pre-event measures. Each of the rates is described in detail below, with comments about potential uses.