This panel of presenters discussed a variety of efforts from within OPOs and hospitals to improve the identification of potential donors and facilitate donation. In contrast to the programs presented in previous panels, these settings offer more immediate links between post-event measures, e.g., referral rate and consent rate, and measures of actual donation rates. However, evaluation of these programs must be designed to determine whether these proximal measures are causally linked at statistically significant levels.
Holly Franz, R.N., from the Partnership for Organ Donation, spoke on "Evaluating Hospital Donation Performance by Monitoring Family Satisfaction with the Donation Process." Arguing that the quality of care received by a potential donor’s family is a significant predictor of its willingness to donate, Ms. Franz presented results of a survey, conducted in cooperation with the Harvard School of Public Health, of families who had been through the request process. A comparison of responses of donor and non-donor families on scales measuring quality of care received, brain death knowledge, and quality of request revealed a significant difference in the experiences of these families on all three scales. She suggested that such instruments would be useful tools in continuous quality improvement of hospital procedures. Further, she described certain methodological weaknesses of the evaluation that the team could correct or adjust for in subsequent studies. These included a significantly lower response rate from non-donor families, partly due to less complete information retained by the OPO for such families and a resulting inability to track them down, and the time elapsed between the donation opportunity and the interview, which may hinder accurate recall.
Phyllis Weber, R.N., the Executive Director of the California Transplant Donor Network presented "Exceptional Requestors: Best Organ Procurement Organization (OPO) Consent Practices." This initiative, being conducted on behalf of the UNOS Council for Organ Availability, seeks to identify exceptional requestors, defined as those with consent rates of more than 75%, and to use surveys to identify their relevant personality traits and practices. The feedback from these surveys will be used to craft a personality profile that can be used to inform the hiring of new requestors, and instructional tools to be used for current requestors and new hires, including vignettes for role-playing. Plans for evaluation include a detailed tracking of individual and institutional consent rates and comparison of rates between those who have undergone best practices training and those who have not. Additional comparisons will be made between participants’ consent rates before and after training, and between new hires and trained "historically employed" staff. Consent rates of cohorts of trainees who participated in the same workshop or instructional module also will be compared. Finally, participants in any workshop or instructional module will be asked for qualitative evaluations of the program.
Aside from selection of appropriate performance measures, the panel raised multiple specific issues for consideration in design of program evaluations, as follows.
- Evaluators must strive for representative samples and otherwise account for selection bias. For example, when studying donor versus non-donor families, panelists suggested that families who chose to respond to the survey might be more satisfied with the quality of care received than families who chose not to respond. This sample bias may have skewed the study results and attempts should be made to reach a truly random sample of non-donor families.
- Evaluators must account for other factors that may confound results. Examples given were accounting for geographic regions and socioeconomic factors. For instance, it might not be appropriate to compare changes in consent rates of hospitals in different parts of the country if there are underlying differences in quality of care or compliance with required-request procedures.
- Another goal in good evaluation is prospective design, which helps to limit selection bias in study participants. For example, the Franz study encountered a low response rate from non-donor families. In addition, other researchers noted a difficulty in tracking down non-donor families for a retrospective study because OPO coordinators do not collect contact information unless a family consents to donate. Rather than requesting a family’s involvement months after the donation event, it might be advantageous to ask families if they are willing to participate in a study prior to consenting to donate. Follow-up information could be collected at the time of the event, and the response rate would likely be higher if families expected to be contacted.
- Careful consideration must be given to sufficient sample sizes in program evaluations. The panel stressed the need for more data in nearly all of the programs presented. For example, in the case of the donor family surveys, the presenters acknowledged the desirability of conducting a larger study that would allow multivariate analysis of the relative importance of quality of care, quality of request, and brain death knowledge on donation decisions.
- Finally, the importance of applying evaluation results to program redesign was noted. Once validated, the Franz survey of families and the results of Weber’s effective requestor study should both provide useful mechanisms for hospitals to improve and continually monitor their performance. Also, a hospital’s key person, as in the Katz study, is in a good position to facilitate a continuous quality improvement process, especially in regard to enforcing protocol compliance.