Inside the Black Box of Interactions Between Programs and Participants: Re-conceptualizing Subgroups for Fatherhood Program Evaluation. A. Subgroups Examined in Rigorous Fatherhood and Family-strengthening Program Evaluations

10/12/2012

In the Strengthening Families Evidence Review (SFER), researchers at Mathematica conducted a systematic review of research on programs serving low-income fathers (Avellar et al. 2011) and programs serving low-income couples (Avellar et al. 2012). For research examining program impacts, the review examined the strength of the evidence and provided information on outcomes examined and average impacts found. While not focused exclusively on fathers, couples-based family-strengthening programs contain many of the same relationship services-and, increasingly, economic stability services-as fatherhood programs. These programs also target low-income families. Therefore, it could be instructive to consider the subgroup impacts examined in these programs as well.

For the present study, we selected impact studies rated high quality then reviewed the original impact reports to see the extent to which subgroup impacts were examined and, if so, which subgroups were considered and any impacts found. We summarize these findings below-first for high quality impact evaluations of programs serving low-income fathers, then for high quality impact evaluations of programs serving low-income couples.

1. High Quality Impact Evaluations of Employment and Family-strengthening Programs Serving Low-income Fathers

The review of research on programs serving low-income fathers identified 12 impact evaluations that employed designs deemed strong enough to detect program impacts. Of these, eight were rated as meeting high quality standards of evidence (Avellar et al. 2011). In four of the eight studies-evaluations of Filial Therapy (Landreth and Lobaugh 1998), Information and Insights about Infants (Pfannenstiel and Honig 1991), Men As Teachers (Fagan and Stevenson 2002), and the NCP Choices PEER pilot program (Schroeder et al. 2011)-subgroups were not examined, likely due to the small sample sizes. Subgroup impacts were examined in each of the four remaining high quality evaluations, and impacts were found in all four of these studies.

Supporting father involvement. Cowan and colleagues (2009) examined whether impacts of the father-only version of the Supporting Father Involvement program differed for higher- versus lower-income fathers, for married versus cohabiting fathers, for Mexican American versus European American fathers, and for fathers who were satisfied versus those dissatisfied with their couple relationship. No impacts were found in these subgroups. Because the sample sizes were small and most likely did not provide sufficient statistical power to detect effects, it is not clear whether there truly were no impacts for these groups or differences existed but sample sizes were too small to detect them.

Jobs-First GAIN. A traditional mandatory welfare-to-work program, the Jobs-First GAIN (Greater Avenues for Independence) program provided job club, job placement, case management, and an earnings disregard. The Los Angeles Jobs-First GAIN Evaluation examined whether employment and earnings were greater for program fathers than fathers who could receive welfare and food stamps and participate in other work programs obtained on their own (Freedman et al. 2000). They hypothesized that impacts on employment and earnings would be strongest among fathers who could benefit most from participation, defined as those who were the most educationally or economically disadvantaged. The researchers found strongest impacts for those lacking a high school degree or GED at baseline and for the most disadvantaged, defined as high school dropouts who did not recently work for pay and who had received welfare for at least two years.

Wisconsin's child support earnings disregard policy. This policy, enacted in the late 1990s, allowed custodial parents to continue to receive their full Temporary Assistance for Needy Families (TANF) amount even if they received child support. (Standard policy was to subtract the amount of child support paid by noncustodial parents from the custodial parent's TANF check.) The Child Support Demonstration Evaluation examined whether payment of child support was higher for families receiving the full disregard compared to families who received only a partial disregard (Cancian et al. 2008). The researchers hypothesized that the policy would be more effective for fathers with a current child support order (because changes in child support would be most salient to them). They also sought whether the policy was effective for families with no recent welfare experience (on the premise that estimated effects for new cases would better approximate long-term effects of the policy). Cancian and colleagues found that, by the third year, the policy did in fact increase the likelihood of payment and the average amount paid among fathers with a child-support order at study entry and among families with no recent welfare experience.

Parents' Fair Share (PFS). PFS was a mandatory program that sought to increase employment, earnings, child-support payments, and father involvement among low-income non-custodial fathers by providing employment and training, curriculum-based peer support, voluntary mediation with the custodial parent, and short-term modifications of child-support orders (http://familyreview.acf.hhs.gov/ProfileDetails.aspx?prID=153). In their evaluation of PFS, Miller and Knox (2001) explored whether the program was effective for fathers with and without a high school diploma. They found that the program increased earnings among non-graduates but did not affect earnings among high school graduates. The program also increased employment rates only for the most disadvantaged fathers (those without a high school diploma or GED or with little work experience).

2. High Quality Impact Evaluations of Family-strengthening Programs Serving Low-income Couples

A parallel review of family-strengthening programs serving low-income couples found that only four impact evaluations met high quality standards of evidence (Avellar et al. 2012). In one of these four studies-the evaluation of Fatherhood, Relationship, and Marriage Education (FRAME) (Wadsworth et al. 2010)-subgroups were not examined, likely due to the small sample sizes. Subgroup impacts were examined, and found, in each of the three remaining high quality impact evaluations.

Building Strong Families (BSF). The BSF program was developed by a team of researchers under contract to the U.S. DHHS to test the effectiveness of curriculum-based relationship skills education and support, case management, and service referrals provided to low-income unmarried, romantically involved couples who were expecting or recently had a baby (Dion et al. 2006; Dion et al. 2008; Dion et al. 2010). The multi-site evaluation of BSF examined program impacts at 15 months in a number of subgroups defined by socio-demographic characteristics (parental age, race/ethnicity, educational attainment, religiosity), economic circumstances (couple earnings), and psychological distress. Researchers also explored impacts in subgroups defined by attitudes toward marriage and by the nature and quality of the couple relationship at study entry (including whether either member of the couple had a child by a previous partner) (Wood et al. 2010). The strongest and most consistent set of subgroup impacts were found for African-American couples, who experienced improvements on four of the five measures of relationships quality and on a composite relationship index. Wood and colleagues also found positive impacts on relationship quality for couples with less than a high school education and for couples who entered the study with relatively higher levels of relationship quality, and that BSF decreased the likelihood of marriage and continued romantic involvement among couples experiencing multiple partner fertility (that is, at least one member had a child by another partner as of study entry). BSF produced offsetting impacts on the composite relationship index depending on partners' ages: Couples in which at least one partner was younger than 21 experienced positive impacts on the relationship index, whereas couples in which both partners were 21 or older experienced negative impacts on the relationship index. BSF also decreased the likelihood of marriage and continued romantic involvement among these "older" couples.

Family Expectations (FE). FE was the BSF program in Oklahoma City. FE provided information on infant development, self-care, the importance of fathers in the lives of children, and co-parenting, as well as curriculum-based communication and problem-solving skills to low-income married and unmarried couples who were parents of infants or expecting a baby. Devaney and Dion (2010) conducted additional analyses beyond what was conducted for the BSF cross-site evaluation report and likewise found a consistent pattern of positive impacts for African American parents, including fewer maternal depressive symptoms and increases in relationship quality, co-parenting, father involvement, and in the percentage of fathers providing substantial financial support to their child. Favorable impacts were also found for couples with less than a high school education, including increases in relationship quality, partner fidelity, fathers' substantial financial support, mothers' parenting, and reductions in mothers' depressive symptoms and the percentage of mothers reporting physical assault. (These impacts were not found among more educated parents.) Devaney and Dion also found positive impacts for couples entering the program with lower levels of relationship quality, and a few, smaller positive impacts for couples with better quality relationships at baseline.

Supporting Healthy Marriage (SHM). Like BSF, SHM was developed for the express purpose of testing the effectiveness of a research-based set of family-strengthening services. But rather than targeting unmarried couples (as in BSF), SHM served low-income married couples with children. SHM provided relationship and marriage education workshops, complementary activities designed to reinforce workshop material and to allow couples opportunities to interact with other married couples, and family support services including links to community services and one-on-one coaching on topics addressed in the workshops. The multi-site evaluation examined impacts for subgroups defined by race/ethnicity, income (relative to the poverty level), and baseline levels of marital distress. Relatively few subgroup impacts were found. SHM impacts were slightly larger for Hispanic couples and for couples who entered the study at higher levels of marital distress (Hsueh et al. 2012).

3. Summary

When subgroups have been examined in high quality impact evaluations of fatherhood and couples-based family-strengthening programs, variables used for creating subgroups typically involve demographic characteristics such as race/ethnicity, educational attainment, and marital/relationship status. Using these subgrouping variables (or examining impacts among homogeneous subpopulation of fathers within a single study), programs were found to be effective for more economically disadvantaged fathers (Parents' Fair Share), for African American fathers (Men as Teachers), for African American couples (FE), and for Hispanic couples (Supporting Healthy Marriage). All these findings support the compensatory hypothesis of subgroup impacts.

In some cases, variables reflecting baseline levels on outcomes sought for program participants were used as subgrouping variables. Outcomes typically sought in fatherhood programs include improved parenting and co-parenting, more stable and better quality partner relationships, economically viable employment, increased child support, and improved father well-being. Outcome variables used in these high quality impact evaluations for creating subgroups were limited to the nature and/or quality of the partner relationship, and involvement with child support. Using these subgrouping variables, programs were found to be effective for fathers already involved in the child-support system (Wisconsin's Child Support Earnings Disregard Policy), for couples entering the program with lower relationship quality (Supporting Healthy Marriage; FE), and for couples entering the program with higher relationship quality (BSF). There were also a few instances in which programs were found to have unintended negative impacts-specifically, increasing intimate partner violence among couples in "on again/off again" relationships (Baltimore's BSF program), and reducing marriage and romantic involvement among couples experiencing multiple partner fertility and among young couples over the age of 21 (BSF). Overall, there were numerous instances of impacts found in subgroups that did not hold for the sample as a whole, and one example of offsetting impacts in complementary subgroups, illustrating the value of exploring impacts in theoretically meaningful subgroups.

Only two of these high quality impact evaluations examined subgroups defined by psychosocial predictors of fatherhood outcomes: BSF's examination of impacts for couples with more and less favorable attitudes toward marriage (Wood et al. 2010), and the Supporting Father Involvement program's exploration of impacts among fathers who were satisfied versus dissatisfied with their couple relationship (Cowan et al. 2009). None of these high quality impact evaluations examined subgroups defined by psychosocial characteristics reflecting a father's personal history (such as experiences in the family of origin), his personality and identity (such as openness to change and salience of the father role), his values and lifestyle (such as work ethic and criminal activity), the nature and degree of stress (such as financial stress and parenting stress) and his means of coping (such as avoidance or seeking out information), his fatherhood-related knowledge (for example, pertaining to effective discipline), social and peer norms to which he is exposed (such as expectations that he be an involved father), or supports available to him (including informational, emotional, instrumental, and financial support from friends, family, and community institutions). Moreover, none of these high quality impact evaluations examined subgroups defined by psychosocial characteristics conceptualized by behavior change theories as determinants or prerequisites of behavior change-such as, readiness, willingness, and commitment to change. It is therefore unclear whether impacts of these family-strengthening programs for low-income fathers and couples might differ in subgroups defined by these psychosocial predictors of behavior and determinants of behavior change.

Regarding subgrouping approaches used, these high quality impact evaluations typically used single categorical variables, reflecting demographics (such as high school graduation status and race/ethnicity) or family-related status (such as marital status and multiple partner fertility). Wood and colleagues (2010) used multiple variables and an additive approach to create a relationship index prior to categorizing and assigning couples into "high quality" and "low quality" subgroups. Only the Jobs-First GAIN program evaluation (Freedman et al. 2000) adopted a multiple-variable interactive approach to defining subgroups, relying on demographic characteristics reflecting service needs (educational attainment, work history, time on welfare) rather than psychosocial factors predictive of behavior change. None of these impact evaluations used data-driven interactive approaches, such as cluster analysis, to create distinct profiles (subgroups) of individuals.

View full report

Preview
Download

"rpt_insidebb.pdf" (pdf, 1.35Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®