Proceedings from a Working Meeting on School Readiness Research: Guiding the Synthesis of Early Childhood Research. Research and Methodological Issues

12/15/2009

Although there has been progress in carrying out randomized controlled trials, and some important questions are being addressed by this research, there are still key design issues that need to be addressed. The randomized controlled evaluations of early childhood interventions reviewed for this meeting are generally designed to address global questions about whether specific intervention programs have impacts on children's development and/or on teachers' instruction compared to a business-as-usual condition.

Compared to what? It is important to note that the comparison of program enhancements to existing early childhood programs, as in recent studies, represents a shift from the earliest randomized studies of early childhood programs (e.g., Perry Preschool, Abecedarian).  In the early studies, an intensive preschool program delivered under ideal conditions (e.g., delivered on a small scale by intervention developers) was compared to a no intervention control group rather than a business-as-usual control group.  Given the different comparison being made, it is not surprising that the earlier programs were found to have larger impacts than the interventions evaluated in the more recent body of work.  A key consideration in synthesizing and drawing conclusions from evaluations of early childhood interventions is the recognition of what comparison is being made.

Testing Intervention Components.  The recent body of randomized studies is designed to provide more rigorous tests of whether the interventions are effective at improving child development or teacher practice.  However, these studies are not designed to address questions about differential impacts of varied intervention components, delivery mechanisms, or professional development approaches.  To address these questions, planned variation studies would be needed, which systematically compare multiple versions of an intervention or training approach.  However, planned variation studies are complex to design and implement.  In addition, they are expensive, since they require substantially larger sample sizes to test multiple treatment conditions.  We either need to increase investments in large studies that allow for the testing of complex models, or lower the bar on what is considered acceptable statistical power and report confidence intervals for effect sizes.

Examining whether Impacts are Sustained.  Given the positive short-term impacts of early childhood interventions that have been found in recent randomized studies, a next step might be to ask about whether initial impacts are sustained and the extent to which subsequent educational experiences mediate and moderate academic and socioemotional outcomes in early elementary school.  However, addressing questions about the longer-term impacts of early childhood interventions presents substantial challenges.  The extent to which subsequent child outcomes can be attributed to the early childhood intervention versus intervening educational experiences cannot easily be isolated.  As students disperse into a wide range of kindergartens and elementary schools, collecting data on the quality of those educational experiences becomes a difficult task.  Furthermore, randomized studies of early childhood interventions are generally not designed with sufficient power to examine how an intervention interacts with subsequent educational experiences, even if there were only a limited range of kindergarten and elementary experiences.  Consensus over the value of conducting longitudinal studies of sustained intervention impacts has not been reached.  Some researchers argue that longitudinal studies of sustained intervention impacts are still warranted despite the methodological challenges and some indications that effects of early childhood interventions dissipate in elementary school.  However, other researchers argue that, given methodological limitations of doing so, such studies result in an attempt to attribute effects on child outcomes in later years (or the lack of effects) to the early childhood intervention without consideration of the role of intervening educational experiences.

Examining Impacts on Subgroups of Students.  Questions also remain about key subgroups within samples. Some arguments have been made for including adequate sample sizes of key subgroups to answer the question, what works for whom. For example, the Early Head Start Research and Evaluation Project found differences between subgroups in the extent to which children benefit from the program (Early Head Start Research and Evaluation Project, 2003). However, some researchers do not view the subgroup question as a critical focus of early childhood evaluation research, especially considering the cost of conducting studies with sufficiently large sample sizes and the practical challenge of developing and implementing a different program model for different groups of children. How to improve early childhood programs so that the average child benefits has been posited as the critical question, rather than that of subgroup variation in impacts.

Limitations of Outcome Measures. Remaining questions about the reliability and validity of measures of child outcomes pose challenges in identifying whether interventions improve child outcomes. There is some indication that more academically-orientated domains may be measured with a higher degree of reliability than socioemotional domains. However, it should be noted that measures of children's cognitive functioning tend to focus on simple aspects of performance rather than thinking or motivation. Research is needed to develop measures of children's thought processes (rather than or in addition to measures of children's mastery of skills), socioemotional development, and approaches to learning. Furthermore, as new measures are developed, it is important that they be normed to ease comparison across studies and provide information about the effectiveness of interventions. Additionally, there is a clear need for more consideration of how best to assess children who are English Language Learners (e.g., not just translating current measures into other languages).

Research limitations. There are also some limitations of what we can learn from randomized studies that are important to note.  Given the nested structure of data in evaluations of early childhood interventions (i.e., children nested within classrooms within centers/schools), large samples of classrooms and/or centers are required to answer main effects questions about the impact of interventions.  The investments required to design studies to answer questions about impacts for subgroups of children or to examine multiple variants of an intervention are an important consideration and can be prohibitive.  Furthermore, randomized evaluation studies are not the best design for exploring some basic research questions that can inform the design of early childhood interventions.

The early childhood field needs more basic research about processes by which children develop (across domains, including math knowledge, cognitive knowledge, language, and their social and emotional functioning). As noted above, research is needed to better understand how foundational skills are learned, how they interrelate, and how they develop over time to support academic success.  That knowledge can form the basis for studies of (1) interactions and behaviors teachers can use to facilitate or accelerate that development, and (2) approaches to training teachers that will help them learn approaches to fostering children's skills and development.

View full report

Preview
Download

"index.pdf" (pdf, 307.32Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"apa.pdf" (pdf, 78.45Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"apb.pdf" (pdf, 401.85Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"apc.pdf" (pdf, 288.78Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"apd.pdf" (pdf, 90.04Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"ape.pdf" (pdf, 190.7Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"apf.pdf" (pdf, 68.04Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®