Inside the Black Box of Interactions Between Programs and Participants: Re-conceptualizing Subgroups for Fatherhood Program Evaluation. D. Implications for Data Collection


Creating subgroups for use in program impact evaluations requires collecting information on fathers' demographic and psychosocial characteristics, their interpersonal relationships, and their circumstances prior to random assignment (baseline). A key implication is that critical information must be collected by programs during the application and enrollment process, and by evaluators during the study sample intake process. In this section, we highlight key considerations for both programs and evaluators regarding baseline data collection.

Fatherhood program providers need information that allows them to (1) identify fathers who may benefit from program services, and (2) identify which services each father needs. Our study findings suggest that providers may want to assess readiness to change when screening for eligibility for program services to gauge a father's commitment to the program and to behavior change in general. Such a measure would need to be short and simple to administer, and its wording and response options should lend themselves to easily identifying fathers who are especially good candidates-and also perhaps fathers who are especially poor candidates-for the program. Such a readiness measure could be included on a self-assessment filled out by the father, or it could be a rating by program staff based on their informed judgment and years of experience working with fathers whose demeanor and attitudes signal likely program success.

Once enrolled in the program, fathers typically fill out a needs assessment prior to engaging in services to identify service needs and challenges. Our study findings suggest it may be important to identify not only what services the father may need but also what services he wants, based on his goals relating to employment, parenting, co-parenting, and his current or future partner relationships. Baseline needs assessments should also identify the strengths and supports available to him that he can capitalize on in his efforts to change, including barriers that may or may not be addressed with services but that may nevertheless shape the likelihood of success. These may especially be psychosocial in nature, such as counterproductive attitudes and beliefs.

Program providers have few resources and little time to develop measures prior to including them on intake forms. Program evaluators can help the fatherhood field in this regard by developing and pilot-testing items suitable for use on intake forms that are effective at identifying fathers with distinct service needs or fathers who may and may not benefit from program services. These items could tap the theoretically relevant but empirically untested psychosocial variables suggested by this study, or they could be other psychosocial factors not examined in this study but that program providers' perceive as important given their insights and experience working with fathers.

For the purpose of creating subgroups for exploration in impacts analyses, program evaluators have more flexibility. They may have the time to pilot test new and innovative measures, or existing measures that have not typically been used in social service research or with low-income and culturally diverse samples of fathers. Lengthier measures can be used (given that evaluation sample respondents are often compensated for their time completing surveys or interviews), and psychometric analyses can indicate whether briefer versions are equally valid and reliable. Finally, program evaluators can employ more sophisticated methods to create subgroups, such as latent class analysis, cluster analysis, and other methods allowing a constellation of factors to be considered simultaneously. With larger samples to support more rigorous statistical tests, program evaluators can develop and explore the utility of developing profiles of fathers most and least likely to benefit from fatherhood programs. Such findings would not only provide needed research on the effectiveness of fatherhood programs, but it would help fatherhood program providers better target scarce program resources.

View full report


"rpt_insidebb.pdf" (pdf, 1.35Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®