[ Main page of Report | Contents of Report ]
The case-study states varied in the amount and kind of evaluation they were conducting with their PAS programs. All states were collecting data with which they could monitor client characteristics, service use, and client satisfaction. These data were not consistently analyzed, however. Outcome evaluations were less commonly done. Several barriers hindered evaluation, including lack of funding, design issues related to the nature of PAS programs, and lack of enthusiasm on the part of PAS program staff.
The organization of evaluation activities varied considerably among case-study states according to the organization of post adoption activities and whether the state used external evaluators. Regional PAS providers in Massachusetts and Oregon’s statewide PAS program compiled program activity data for analysis by external evaluators. These evaluators also analyzed client satisfaction surveys collected by the statewide provider in Oregon and gathered and analyzed additional data in Massachusetts. In Virginia, the statewide PAS coordinator compiled data submitted by regional PAS providers for an annual report to the state. Texas regional PAS providers used client satisfaction data at the local level and submitted activity data to the state for cost reimbursement, but the state did not analyze these data for evaluation due to a lack of funding for that purpose. In Georgia, statewide PAS providers were responsible for evaluating the specific service components that they conduct, with oversight from the state adoption program manager.
| Each PAS program monitored client characteristics and services delivered, but outcome evaluations were less common. |
The types of evaluation activities employed depended on the services being assessed and the expertise available to the PAS program for evaluation design and analysis. Among evaluation activities, needs assessment can be considered formative evaluation designed to guide program design and modify it during the early implementation period. Although states may have been guided by informal reports of adoptive families’ service needs from child welfare and other service providers, only Georgia and Oregon reported having conducted formal needs assessments of adoptive parents as part of their program planning processes. Needs assessments appeared to be of wide interest nationally. Excluding the five case-study states, 23 of 31 states responding to the ILSU survey reported that they were conducting needs assessment or had done so since 1990.
All five states collected the kinds of data normally used for process evaluation: characteristics of clients served and services delivered. Oregon and Massachusetts were far more sophisticated in their collection and use of process evaluation data than were the other three states. Both states developed databases of client contacts and other events and used external evaluation providers to analyze the resulting data. Both of these databases were detailed, including information on the nature, content, and client characteristics of every service delivery event, although Oregon did not record contacts that were handled in a single telephone call. The Massachusetts database was web-based to allow direct access by regional providers. These two states used their event tracking databases to support analyses of training audiences, specific types of services provided, hours of service provided in various service categories, and household composition of client families.
Assessments of client satisfaction also were used in each state, again with varying approaches and levels of rigor. Across the five states, client satisfaction surveys were used for most services, including information and referral, tutoring programs, respite care, family support group, training, and counseling. However, response rates for these surveys were low enough—ranging from 36% to 86%—to raise concerns about the validity of resulting data. Interviewees mentioned using the surveys, even if not aggregated, to inform their sense of how they were doing and what services might be added. Massachusetts also used group interviews and focus groups in its assessment of client satisfaction.
Formal outcome evaluations are conducted less frequently than process and client satisfaction measures. Georgia and Virginia used clinical instruments administered pre- and post-services to assess counseling services. Georgia’s crisis intervention program, which provided intensive case management to families in crisis, administered the Child and Adolescent Functional Assessment Scale at intake, three months, and exit as well as monitoring disruption rates among its clients. Regional PAS providers in Virginia reported using the Achenbach Child Behavior Checklist, Current Feelings About Relationship with Child, and Cline/Helding Adopted and Foster Child Assessment at intake and closing, although not all providers used all of these instruments.
Other forms of outcome assessment were mentioned. Massachusetts evaluators reported using data from their information system to analyze the degree to which goals identified in the course of information and referral calls actually were attained. Virginia planned to monitor the extent to which disruptions occurred among families served by the PAS program. PAS program staff in three states mentioned reviewing their own treatment narratives to assess how families were doing, if they were “looking better.” This informal assessment is worth noting because staff appeared to be implementing it without any direction from above. This suggests that they may have been more comfortable with individual clinical assessment than with program evaluation activities, perhaps reflecting the emphasis of their own professional training or their limited faith in the usefulness of other evaluation activities.
Several barriers to evaluation, or to more extensive evaluation, were identified in the course of interviews. Some of these may have been inherent in the ethos of PAS or in services for children and families. Others were specifically related to program structure.
| Barriers to evaluation included insufficient funding, lack of expertise, and staff concerns. |
Funding was the barrier mentioned most frequently by both state adoption program managers and PAS coordinators/providers. Funding represented either staff time to conduct evaluations or the ability to contract with external evaluators. Although evaluation was among the activities described in either the RFP or the providers’ proposals in each state, it generally was not specified by allocation of specific staff members or budget line items. Massachusetts, the only state with a line item specific to evaluation, spent approximately 5% of its budget on evaluation. Although clearly more extensive than in other states, these efforts represent a relatively small proportion of funds to allocate to evaluation. Interviewees were committed to evaluation, and in some cases their efforts had gone beyond what was required by the state. However, providers noted the burden of evaluation on limited staff time.
Evaluation expertise is strongly related to funding. Contracting with an external evaluator requires a greater commitment of program funds but provides access to a higher level of expertise. Although program coordinators frequently have some experience and training in evaluation, it is unlikely to be at the same level as for someone whose primary role is evaluation. Similarly, local staff generally are chosen according to their service delivery skills and may have limited qualifications in the collection and management of evaluation data.
PAS program staff expressed concerns about evaluation in terms of its impact on their interactions with clients. One PAS provider noted that the worlds of social workers and evaluators are very different, and service providers did not naturally embrace evaluation. Program staff also expressed concern about the amount of client time spent completing assessment instruments, especially if this did not provide any direct benefit to the client. “Families that come to me are under a lot of stress,” said one service provider, “and asking them to fill out something with 140 questions just adds to that.” Program staff feared that adoptive parents would be “put off” by clinical instruments that seemed to focus on child and family problems.
| PAS programs presented specific challenges to evaluation design. |
The very structure of these PAS programs creates inherent challenges for rigorous evaluation design. With the exception of one intensive crisis management intervention that used a defined intervention model, the PAS programs described in this report were intended to be extremely flexible in their delivery, responding to the varied needs of individual families rather than offering a predefined bundle of services for a set period of time. PAS coordinators/
providers considered this tailoring to be a strength of the programs, allowing them to respond to clients’ specific needs in a way that empowers families. Unfortunately, this tailoring was at odds with several assumptions of evaluation design, particularly with respect to outcome evaluation:
The challenges described here are not unique to PAS programs. In HIV-prevention programs, for example (Napp, Gibbs, Jolly, Westover, and Uhl, 2002), limited funding, lack of staff expertise and interest in evaluation, and lack of fit between program models and standard evaluation methods all have been noted as barriers to evaluation. Though few would argue that program models should be radically altered to meet the requirements of rigorous evaluation, several measures could enhance the quality and usefulness of evaluations for PAS programs. A discussion of these measures is the subject of the PAS Evaluation Issues Report.
Main Page of Report | Contents of Report
Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)