One Destination, Many Roads: Envisioning Universal Measures of Economic Mobility
A virtual series organized by the Office of the Assistant Secretary for Planning and Evaluation (ASPE) and the Institute for Research on Poverty at the University of Wisconsin–Madison
Event Summary
This forum brought together a diverse group of individuals to discuss innovative strategies for measuring and reporting common economic mobility outcomes across federal programs and using integrated data for evidence building purposes. The two day event had 279 attendees including leadership and staff across many federal departments and agencies including Health and Human Services, Labor, Housing and Urban Development, Education, Agriculture, Social Security Administration, Congressional Research Service, Joint Economic Committee, Council of Economic Advisors, and the Office of Budget and Management. In addition, leaders representing various state and local agencies, non-profit agencies, colleges, and universities attended.
The Forum was designed to forward the goals and priorities of the Interagency Council on Economic Mobility, led by HHS. The Council aims to improve federal coordination and focuses on crosscutting issues that cannot be accomplished by a single agency on its own. Innovations and lessons from this event can be applied to federal efforts to learn how to better promote family-sustaining careers and economic well-being. To have a shared vision for outcomes that indicate progress, silos need to be broken down and the focus needs to be on helping people succeed.
Highlights and key takeaways of the event are summarized below. You may also access the event recordings and presenter slides.
Session One – September 9, 2020
Keynote Address: Past, Present, and Future Challenges in Measuring Outcomes for Vulnerable Populations
Speaker: Robert M. George, Senior Research Fellow, Chapin Hall, University of Chicago
- Provided historical perspective on using data for measuring outcomes for vulnerable populations.
- Welfare reform research renewed interest in integrating systems and having comparable data to measure participant outcomes and progress towards economic self-sufficiency.
- Employment and earnings are focal metrics, but multidimensional measures can indicate progress and expand evidence base for what works in human services and workforce development programs.
- The institutionalization of data in silos and fragmented systems is a primary challenge to achieving share outcome measures.
- The TANF Data Collaborative is an example of a multi-level partnership intended to overcome that challenge and spur data use that accelerates evidence building at the federal, state, and local levels.
Presentations and Panel Discussion: State Lessons for Federal Efforts
Moderator: Kosar Jahani, Project Officer, Bill and Melinda Gates Foundation
Panelists:
Christina Church, Two-Generation Program Officer, Maryland Department of Human Services
Tracy Gruber, Senior Advisor for the Utah Intergenerational Poverty Initiative
David Mancuso, Director of the Research and Data Analysis Division, Washington State Department of Social and Health Services
Ki’i Powell, Director of the Office of Economic Security, Colorado Department of Human Services
Discussion Highlights:
- Measure What Matters. Measure outcomes to promote economic stability, not just process or performance metrics. Pay attention to values that drive the program(s), the program goals – those are the outcomes to capture. For programs with similar goals, align outcomes that measure success.
- Start Small. There are no perfect data or measures, so start somewhere and refine over time. Time is of the essence and opportune relevant evidence is more desirable than untimely “perfect” evidence.
- Trust Is Important. Integrating data requires trust and relationship building among the agencies that own program data and buy-in from agency leadership. Producing high-quality analytic work that shows the value of collaboration to produce evidence is one way to build that trust.
- States Need Skilled Analysts. States are rich with data, but limited in their capacity to tap into data resources. Creative initiatives to foster state agency capacity and access to skill analysts is an area where federal leadership could help. For instance, efforts to encourage building partnerships with the academic community would increase analytic capacity.
- A Clear Vision Is Key. Success requires clarity about the purpose of aligning outcome measures across programs that promote economic mobility. The vision for how data will be used must be clear to all stakeholders. Creating that vision may require an examination of the values and philosophies that underlie program design and stated objectives as a means for achieving shared definitions.
Session Two – September 16, 2020
Speaker: Ted McCann, Vice President, American Idea Foundation
- Misconception that Congress does not use evidence or data to inform decision-making about programs. Committees, in particular, spend time reviewing the available evidence.
- Policymaking is driven by values. Evidence clarifies the trade-offs in values (e.g., cost vs. benefits).
- To influence the policymaking process, evidence must: 1) be timely; 2) be trustworthy (most important), 3) be compelling; 4) point in a clear policy direction; and 5) experience a stroke of luck.
- The Families First Act and the First Step Act are examples of evidence-informed policymaking that met the five criteria for “giving evidence a seat at the table in the policymaking process.”
- Lots of potential for evidence-based policymaking, especially on lower profile issues. First need to identify federal nexus between research and local actions and reauthorizations expiring soon.
- Early implementation of the Evidence Act promotes building a culture of evidence through development of learning agendas. Chief Data Officers are extremely important.
Presentations and Panel Discussion: State Lessons for Federal Efforts
Moderator: Nick Hart, President of the Data Foundation and CEO of the Data Coalition
Panelists:
Julia Lane, Provostial Fellow for Innovation Analytics; Professor, NYU Wagner School of Public Policy and Co-Founder of the Coleridge Initiative
Sara Dube, Project Director, Results First Initiative, The Pew Charitable Trusts
David Williams, Director of Policy Outreach, Opportunity Insights
Discussion Highlights:
- Urgency of Now. Current data systems struggle to provide good and timely answers and measures to critical questions. Start with the problem and work backward to develop measures.
- Look to States for Innovation. States, such as Colorado, are developing and using common process and outcomes measures to inform decision-making processes and program adjustments.
- Need Robust Data Analysis Infrastructure. The technical infrastructure is limited and there is a shortage of people trained in how to use and interpret program data. Julie Lane leads an initiative to train federal, state, and local analysts to use administrative data for evidence building. Also critical to find a way to incentivize states to upgrade their information technology infrastructure and use data.
- Theory of Change Drives Measures. Outcomes of interest should be derived from the theory of change that underlies a program or policy. Think about the goals and the metrics that are critical indicators for progress towards achieving those intended outcomes.
- Engage Stakeholders. Federal stakeholder engagement is critical, but data collection efforts often occur at the local and state levels. Create opportunities for stakeholders at all levels to contribute to the discussion (e.g. brown bags, county board of supervisor meetings, listening sessions, requests for information). Provide the “why,” identify benefits, and understand stakeholder needs.
- Actions for Quick Wins. Moving forward, the Council may consider these activities for quick wins.
- Benefit-cost analyses can fill an evidence gap when long-term outcomes are not yet available.
- Outline theories of change for programs to identify points of alignment and opportunities for shared outcomes measures. Ensure that measures are connected to programs goals.
- Do not be afraid to try several things at once and see what sticks
Return to the 2020 Annual Poverty Research and Policy Forum landing page.