Inside the Black Box of Interactions Between Programs and Participants: Re-conceptualizing Subgroups for Fatherhood Program Evaluation. C. Subgrouping Approaches to Consider

10/12/2012

In addition to selecting the "right" characteristics for creating subgroups, evaluators need to decide how they will assign individuals to subgroups based on these characteristics. These decisions should be informed by the underlying subgroup theory that makes the most sense, given the program or the characteristics of fathers being served. In Section II.B., above, we present three general approaches to defining subgroups:

1.      The single-factor approach is appropriate if an evaluator theorizes that fatherhood program impacts may differ for fathers at higher versus lower levels of risk on a single dimension (such as those using more and less coercive discipline strategies), or for fathers in a qualitatively important circumstance (such as being in an "on again/off again" relationship).

2.    An additive-risk approach is appropriate if an evaluator theorizes that fatherhood program impacts may differ for fathers at higher versus lower levels of cumulative risk across multiple dimensions (such as those lacking employment, conflict resolution skills, and non-coercive discipline strategies).

3.    An interactive approach is appropriate if an evaluator theorizes that fatherhood program impacts may differ depending on the particular constellation of risk and protective factors experienced by fathers (such as those facing personal challenges but also with access to social supports and with effective coping strategies).

Below we draw upon study findings to describe how these approaches may be used to evaluate subgroup impacts in fatherhood programs. We address single-factor and additive-risk approaches together because the overall strategy is the same; only the number of variables used to create the subgroups differs.

1. Single- and Additive-risk Approaches to Defining Subgroups

Whether a single variable or multiple variables are used to define subgroups, both the single- and additive-risk approaches entail dividing a sample into subgroups based on the degree of risk on one or more dimensions. The "compensatory" hypothesis posits that subgroups defined at program entry as "higher risk" may experience the strongest impacts (assuming, of course, that the services received are of sufficient quality and sufficiently different from what equally high-risk control group members would receive). This is consistent with findings from Parents' Fair Share, which was found to increase earnings among fathers lacking a high school diploma but did not affect earnings among high school graduates (Miller and Knox 2001). Findings from the Devaney and Dion 2012 evaluation of FE also support the compensatory hypothesis: Among couples whose relationship quality scores placed them in the bottom half of the distribution at baseline, FE increased relationship stability, relationship quality, the quality of the co-parenting relationship, and the likelihood that fathers financially supported their children. Couples entering the program with higher relationship quality scores did not experience these impacts. In contrast, the "creaming" hypothesis appears to have held for BSF: This program improved relationship quality among couples whose relationship quality scores placed them in the top half of the distribution at baseline but not among their higher-risk counterparts (Wood et al. 2010).

The Goldilocks hypothesis posits that subgroups defined at program entry as moderate risk may experience the strongest impacts (again, assuming a strong treatment-control contrast). For example, fatherhood programs may be effective at improving relationships among couples in distress (moderate risk)-as found in the SHM evaluation (Hsueh et al. 2012)-but not among couples experiencing intimate partner violence or otherwise in a tumultuous relationship (high risk)-as found in the BSF evaluation (Wood et al. 2010).

Which variables are best suited for single- and additive-risk approaches to creating subgroups? Because these approaches focus on risks or needs for services, variables reflecting program outcomes (regarding employment, father involvement, and relationship with child's mother) and variables reflecting content explicitly addressed in the program (such as experiences in the family of origin) may be especially fruitful to explore for creating baseline subgroups using these approaches.

Practically speaking, single- and additive-risk approaches to defining subgroups requires selecting one or more baseline variables and creating a high-risk subgroup reflecting a large number or risks or high level of severity along these dimensions. For example, one could define as high risk or high need those fathers who are considered high risk on employment (for example, currently unemployed, lacking a high school diploma or GED, and little or no employment history), parenting (for example, those endorsing or using harsh discipline practices), and in their partner relationship (for example, coercive and aggressive conflict tactics). Key to this approach is deciding how to define "high risk/need" for each individual variable, and how to combine these variables to produce a "high cumulative risk/need" subgroup. These decisions should be informed by where on the risk/need continuum subgroup impacts are hypothesized to occur.

2. Interactive Approaches to Defining Subgroups

Rather than focusing on the degree or number of risks, the interactive approach to creating subgroups entails dividing a sample into subgroups based on the particular constellation of risk (and potentially) protective factors present at baseline. Some interactive approaches are theory-driven, whereby evaluators specify co-occurring conditions that are expected to shape program impacts. Theory-driven approaches to defining interactive subgroups requires identifying the conditions under which program impacts are most likely, selecting variables reflective of these favorable conditions, then coding and combining these conceptually relevant variables in a way that effectively reflects these conditions. For example, evaluators may create a subgroup comprising individuals who both need and are likely to benefit from program services, hypothesizing that impacts are most likely among those who are committed to change and who are likely to show up at program services. In fact, some fatherhood programs seek to learn this up front in order to enroll only fathers who are "ready, willing, and able" to engage in the program (H. Sullivan, personal communication, May 14, 2012).

Other interactive approaches to defining subgroups are data driven. Service-user typology research and audience segmentation research typically use cluster analysis or latent class analysis to identify naturally occurring subgroups based on individuals' profiles of scores along multiple dimensions. Data-driven approaches to defining subgroups require identifying baseline characteristics hypothesized to influence the likelihood that fathers benefit from program services, then subjecting these variables to a clustering algorithm that combines individuals who are similar along each of these dimensions. This approach is especially fruitful when there is little or no theory or empirical evidence to suggest exactly how these variables should be coded or combined into subgroups because the data effectively identify where the important cut-points are for each subgroup.

Interactive approaches can be used to develop baseline subgroups of individuals with a certain constellation of service needs, in which case, variables reflecting program outcomes or program content are good candidates. In addition, it may be fruitful to explore constellations of factors that could facilitate or hamper participation in fatherhood programs. These could be logistical factors such as transportation and child care, circumstantial factors such as stressful life events and social supports, and psychological factors such as readiness to change and beliefs that change is possible. As noted above, however, fatherhood program evaluations typically do not adopt interactive approaches to creating baseline subgroups for use in impact analyses.

View full report

Preview
Download

"rpt_insidebb.pdf" (pdf, 1.35Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®