As in many human services, funding, lack of access to evaluation expertise, and staff attitudes pose barriers to evaluation of PAS programs.
When discussing evaluation, program coordinators identified several difficulties common to other service delivery arenas. These included lack of adequate funding for evaluation, lack of access to evaluation expertise inside or outside the program, concerns about the ethics of experimental designs, and general staff resistance to implementing evaluations.
Funding was the barrier mentioned most frequently by state adoption program managers and PAS coordinators and providers. Evaluation requires substantial resources, whether it is contracted out to an external provider or performed in-house by program staff. Given limited funding, program coordinators frequently place higher priority on meeting service needs than on evaluation. Their belief that "we don't have enough funding for evaluation," might more accurately be stated as, "we don't have enough funding to provide the services that we know are really needed and are convinced to be effective and perform an evaluation." Funding agencies contribute to this situation if they require evaluation without specifying the level at which it is to be done or do not allocate adequate resources for both service delivery and evaluation.
Among the case-study states, the one with the most sophisticated evaluation (and the only one with a specific budget line item for evaluation) allocated approximately 5 percent of its budget to evaluation. This is a rather modest chap151; and almost certainly inadequate chap151; amount, particularly in a new program area for which service delivery models and evaluation methods are not well established.
Evaluation expertise is in part related to funding. Contracting with an external evaluator requires a greater commitment of program funds but provides access to a higher level of expertise. Although program coordinators may have some experience and training in evaluation, it is unlikely to be at the same level as someone whose primary role is evaluation. Program staff with the skills needed to serve adoptive families may have limited qualifications in evaluation design, data collection, or analysis. In addition, in-house program staff may be less likely than external evaluators to implement more rigorous designs because of the tension that they might create among skeptical staff.
Program staff often feel that evaluation activities encroach upon their interactions with families without benefiting the program.
Even if a PAS program is willing to commit the resources to contracting with an external evaluator, finding an evaluator with adequate understanding of adoption issues may be difficult. The field of PAS is young, with neither a large base of published research nor an extensive network of experienced researchers. Program staff may need to invest considerable time in orienting their evaluators to issues that affect the choice of outcome measures, instruments, and timing of data collection. "We learn from them," said one program coordinator, "and have to make sure they learn from us."
PAS program staff identified several concerns about the impact of evaluation on their interactions with families. Some were concerned that the time required to collect evaluation data not only added to their workload but also impinged on their interactions with families. Time spent completing evaluation instruments was seen as encroaching on their opportunities for therapeutic interaction, without necessarily providing any direct benefit to the family. Some staff indicated concern that this would keep families from coming to, or remaining in, PAS.
Program staff were also concerned that evaluation activities introduced a clinical tone to their interaction that was at odds with their efforts to normalize the adoption experience, especially when the instruments used focused on child and family problems. Adding "strength-based" instruments was a commonly suggested strategy, but these are not well developed and can make interviews unacceptably long.
PAS coordinators and providers have yet to be convinced that evaluation can inform their practice.
Case-study interviews revealed few instances in which PAS coordinators or providers identified ways in which evaluation findings had been useful to them or were expected to be. Some limited applications were noted. For example, data were cited to document the volume of services delivered or families' satisfaction with the program. In one program, staff reported having adjusted their training topics and schedule in response to client satisfaction surveys. But there were no reports of evaluation as a source of new and useful input on substantive questions of program design. If evaluation data are not useful to the program's own staff, they are unlikely to be seen as offering much to the larger field. Yet information from evaluations often accrues slowly into a focused message that may not be disclosed until years, or even decades, after the first rigorous evaluations are begun.