Proceedings from a Working Meeting on School Readiness Research: Guiding the Synthesis of Early Childhood Research. What Can We Learn from Federally-Funded Research on Early Childhood Mathematics Curricula?

12/15/2009

Although leading professional organizations call for research-based curricula, the meaning of research based is a bit problematic. A restrictive definition might be that the curriculum should derive directly studies that focus on how mathematics should be taught. By this criterion, almost no programs would qualify. The designs of early childhood mathematics curricula are based on research investigating the development of childrens mathematical thinking in the absence of instruction, not from teaching experiments. Thus, a more accurate definition of research-based curricula is one that is inspired by research on young children and attempts to translate the research into an organized program of teaching. The danger with this definition is that it can be over-inclusive. Publishers in particular may claim that their programs meet whatever standards are in place at the moment and, not surprisingly, will advertise that virtually any curriculum for young children is research-based (or developmentally appropriate or whatever the slogan of the day may be). Their goal is sales, not scientific rigor.

Our approach is not to take too seriously the claim of a basis in research. After all, the major question is not whether the program derived from research but whether it is effective. Sometimes, practical applications precede and indeed inspire scientific investigation (Stokes, 1997). A creative curriculum developer may have a hunch, possibly based on some informal exploration, that an activity might work, and indeed it might. The issue is not whether the program is research-based but whether it has been evaluated and is shown to be effective in improving learning outcomes. While to date there have been few rigorous studies examining the effectiveness of mathematics curricula for young children (National Research Council, 2004; D. Clements & Sarama, 2008), the studies that have been conducted indicate that young children from low-income families can indeed benefit from curricula designed specifically to address mathematics learning.

Federally-Funded Cluster Randomized Studies of Mathematics-Specific Curricula: PCER, IES, and NSF Research

Federal dollars have supported the rigorous evaluation of three mathematics-specific early childhood curricula, although the evaluation of each has been supported by a different funding stream. An intervention consisting of the Pre-K Mathematics Curriculum (PreK Math; Klein, Starkey, & Ramirez, 2002) supplemented with the DLM Early Childhood Express Math software (DLM; D. Clements & Sarama, 2003) was evaluated as part of the Institute for Education Sciences (IES) Preschool Curriculum Evaluation Research program (PCER; PCER Consortium, 2008). Development and evaluation of the Building Blocks curriculum (Sarama, 2004; D. Clements & Sarama, 2003, 2008) has been supported by the National Science Foundation. Building Blocks is a designed for use with children as young as three-years-old. The evaluation of the Big Math for Little Kids curriculum (BMLK; Greenes, Ginsburg, & Balfanz, 2004) was supported by a research grant from IES (M. Clements, Lewis, and Ginsburg, 2008). BMLK was developed for use by pre-kindergarten and kindergarten students.[1]

The three curricula share a number of characteristics, including the types of professional development offered to teachers, the contexts in which the curriculum is designed to be taught, and the scope of the curricula. It is important to note that the similarities noted here do not represent precise similarities across the curricula, but rather broad characteristics that they share. The specific representation of each of these characteristics certainly varies across the three curricula, possible in meaningful ways that result in differences in their effectiveness.

Professional development activities were a component of the treatment condition in the rigorous cluster randomized studies used to evaluate each of the curricula. All three of the evaluations included at least one intensive workshop on the curriculum before the beginning of the school year. Each of the interventions was also supported throughout the course of the study with regularly scheduled, periodic professional development sessions for teachers. These ranged from bi-weekly, one-on-one sessions in a teachers classroom to bi-monthly refresher courses in which groups of teachers met to review particular aspects of the curriculum.

Another shared characteristic of the curricula is that all are designed to utilize multiple contexts for teaching mathematics. In terms of at school activities, the three curricula include whole class learning activities and small group activities. The curricula also incorporate information and activities designed to be sent home for parents and children to work on together at home.

A third characteristic shared by the these three curricula is that each was designed to be a comprehensive mathematics curriculum covering multiple important mathematics domains , such as numbers, counting, and operations; shapes (geometry); measurement; and pattern. Its important to note that here we are referring to very broad mathematical domains and that the specific content and emphasis of each curriculum may well vary. The major point is that each curriculum sets out to cover multiple important mathematical domains rather than just number and operations or just shapes. See Table 1 for a comparison of the domains covered (broadly defined) by curricula.

Table 1.
Mathematics Domains Covered by Each Curriculum
Program Ages Number Shape Geometry Measurement Pattern Sorting Sequencing Logic Spatial Data
Big Math for Little Kids PreK & K X X X X   X X  
Building Blocks PreK through Grade 2 X X X X X     X
Pre-K Mathematics PreK X X X X X X    
Childrens School Success PreK not reported
Sophians Curriculum 3 & 4 year olds X X X   X      
Round the Rug Math PreK through Grade 2 X X X X     X X

The PreK Math/DLM, Building Blocks, and BMLK curricula also differ in several possibly important ways. While it is true that the three curricula cover many of the same (broadly defined) mathematics domains, it is certain that the specific topics covered within each domain, the scope of coverage for each topic (in terms of depth and/or breadth), the types of activities and lessons developed to teach each topic, and the ways in which various topics and/or domains are integrated with each other varies across the curricula. Investigating the extent of this variation is beyond the scope of this paper. However, a review of published reports and descriptions of the curricula provide some information about these differences. For example, both PreK Math/DLM and Building Blocks incorporate regular use of computer software, while BMLK does not include a software component.

The findings of the rigorous evaluations of the developmentally appropriate mathematics-specific curricula stand in stark contrast to the findings from the Head Start Impact Study. The Head Start Impact study compared children randomly assigned to attend Head Start to a control group of children who, for the most part, attended some other type of center-based care on a number of cognitive domains. Among four-year-olds, the study found a statistically significant positive impact of Head Start attendance on four of eight language-related cognitive domains, but no difference in early math skills (U.S. Department of Health and Human Services, 2005). Given that two of the mathematics curricula reviewed above were evaluated in Head Start classrooms, it appears that the Head Start Impact Studys lack of significant findings regarding math is due to the dearth of effective early childhood mathematics curricula, not Head Start. In fact, the studys final report points to the need for effective early childhood mathematics curricula and teacher professional development in math education (US DHHS, 2005).

While all three of these curricula have been rigorously evaluated using cluster randomized trials, including variation in the length of the studies and the age of children in the study samples, mathematics outcome measures used in the evaluations, and the types of classrooms settings in which the mathematics curricula were evaluated. As well discuss below, these differences make it difficult to compare the relative effectiveness of the curricula, other than to conclude that all three demonstrate effectiveness in improving childrens understanding of mathematics. The PreK Math/DLM and Building Blocks evaluation studies examined the impact of each curriculum over the course of childrens pre-kindergarten year, and the research took place in a combination of Head Start classrooms and state-funded prekindergarten classrooms. The BMLK evaluation, on the other hand, examined the curriculums impact over the course of childrens pre-kindergarten and kindergarten years among children attending child care centers that are subsidized by the New York City Administration for Child Services and, thus, didnt include either Head Start classrooms or state-funded pre-kindergarten programs.

Another important difference across the studies is that each evaluation utilized a different mathematics assessment as the outcome variable. Both PreK Math/DLM and Building Blocks used assessments developed by the curriculums developer (and evaluators), neither of which is nationally normed. Both sets of authors clearly articulate that their assessment is not overly aligned with the curriculum; they are designed to evaluate childrens understanding of the concepts taught, but do not use the same activities and materials that are part of the curriculum. At the suggestion of IES, BMLK used the mathematics assessment developed for the Early Childhood Longitudinal Study, Birth Cohort (ECLS-B; National Center for Education Statistics) which is nationally normed. Each of the assessment procedures (using versus not using an assessment designed to evaluate a particular curriculum) has its strengths and weaknesses. On the one hand, using an assessment designed to evaluate a particular curriculum is likely to be able to provide a more nuanced understanding about what concepts the curriculum was more (or less) successful at teaching. On the other hand, an assessment that has not been nationally normed will not provide information about how children compare to their peers throughout the country both before and after being taught with the curriculum.

Results from the studies indicate that all three curricula were effective in promoting childrens mathematics learning. The effect sizes for the studies were .43 for BMLK  (compared to control classrooms; M. Clements et al., 2008), .55 for PreK Math/DLM (compared to control classrooms; Klein, Starkey, D. Clements, Sarama, & Iyer, in press), and for Building Blocks the effect sizes were 1.07 (for the comparison with control classrooms) and .47 (for the comparison with classroom using PreK Math/DLM[2]; Clements & Sarama, 2008). See Table 2 for additional details on the evaluation studies. In all three studies, the control classrooms used a variety of curricula, including Creative Curriculum, High/Scope, Montessori, or other local curricula.

Table 2.
Comparison of Curriculum Evaluation Studies
Program Research Funding Study Type Sample Control Condition(s) Measure Effect Size[4]
Big Math for Little Kids IES Research Grant RCT Treatment
  • 16 preK classrooms
  • 10 K classrooms

Control

  • 16 preK classrooms
  • 10 K classrooms
Prevailing curriculum
(e.g., Creative Curriculum, home grown curriculum)
ECLS-B Mathematics .43
Building Blocks (BB) NSF RCT BB
  • 8 classrooms

PMC

  • 7 classrooms

Control

  • 8 classrooms
PreK Math
or
Prevailing curriculum
(e.g., Creative Curriculum, Montesorri, home grown)
Early Mathematics Assessment (EMA) BB vs. PreK Math: .47

BB vs. Control: 1.07

Pre-Kindergarten Mathematics Curriculum with DLM Express Software PCER RCT Treatment
  • 20 classrooms

Control

  • 20 classrooms
Prevailing curriculum
(e.g., Creative Curriculum, Montesorri, High/Scope, home grown)
Child Mathematics Assessment (CMA) .55
Childrens School Success ICSR RCT not reported not reported Woodcock Johnson (WJ), subtest 10 and 18 not reported

We should note that the difference between the BMLK  and control students did not become statistically significant until the second year of the study (the year following pre-kindergarten[3]), while the Building Blocks and PreK Math/DLM studies found statistically significant differences at the end of the pre-kindergarten year. This could be due to several factors. One possibility is that our use of the ECLS-B math assessment (a standardized test that was designed as a general assessment of mathematics learning, and not developed to test a particular mathematics curriculum) resulted in a stricter test of the curriculums effectiveness and, as a result, additional months of exposure to the curriculum were necessary before differences in childrens learning could be detected by this assessment. A second possibility is that more than seven months of exposure to the curriculum were necessary before group differences emerged. In our opinion, the fact that the overall math achievement of children in the study was near the national median score throughout the course of the study rules out a third possibility: that the curriculum is too advanced for preschoolers and not appropriate until children reach kindergarten.

An advantage of using the ECLS-B math assessment in the BMLK evaluation is that we were able to determine the extent to which BMLK helped reduce the achievement gap between children from low-income families and the national average. Specifically, in the fall of pre-kindergarten the average student score on the ECLS-B was at the 48th percentile, but increased to the 56th percentile by the end of the prekindergarten year, and was at the 55th percentile at the end of the kindergarten year.

In summary, evaluations of all three curricula demonstrated that they are effective, with effect sizes ranging from moderate to large. Furthermore, the fact that their effectiveness was demonstrated across a variety of classroom contexts (Head Start, state-funded prekindergarten, and NYC ACS subsidized child care centers) suggests that these comprehensive mathematics curricula are likely to be effective in promoting mathematics-related school readiness among children from low-income families.

A Federally-Funded Cluster Randomized Study of a Comprehensive Curriculum: ISRC research

Among the studies that were funded as part of the Interagency School Readiness Consortium, only the Childrens School Success curriculum (CSS) included a component designed specifically to advance childrens mathematics knowledge. Odom and his colleagues (Leiber et al., 2007; Odom et al., 2007a and 2007b) refer to CSS as an early childhood education model designed to combine science, language, literacy, math, and social skills into a meaningful learning experience. The mathematics component of the curriculum is described as being adapted from D. Clements and Saramas Building Blocks curriculum, but details regarding the extent to which the curriculum was modified are not provided.

Based on research reports (consisting of slides and posters from conference presentations), it is difficult to discern whether or not CSS was effective in promoting more advanced mathematics knowledge among the children attending Head Start centers where it was implemented. Analyses for this study are still underway, and, to date, none of the presentations provide statistical results demonstrating the curriculums effectiveness. However, there are multiple presentations that examine the impact of treatment fidelity and childrens attendance rates on childrens math achievement scores. As would be expected, fidelity of implementation is associated with higher student math scores at the end of the school year.

Other Early Childhood Mathematics Research

In light of the fact that only three early childhood mathematics curricula have been subjected to federally-funded rigorous evaluations, this research review will briefly review additional curricula. They include curricula that were evaluated by non-federal funds and/or by study designs that were not as methodologically rigorous or extensive as those for Building Blocks, PreK Math/DLM, and BMLK.

The National Science foundation has funded recent research on the Round the Rug curriculum (Casey, 2004), which is a supplementary language arts-based curriculum designed to promote childrens understanding of key mathematical concepts including pattern, geometry (shape), measurement, and graphs. The curriculum consists of six books that teachers use to lead lessons that integrate oral story-telling with hands-on mathematics. The impact of one of the lessons (on geometry) has been evaluated in a smaller-scale random assignment study involving six kindergarten teachers (Casey, Erkut, Ceder & Young, 2008). This study found that a lesson taught using both the story-telling and hands-on components promoted greater mastery of the material than the hands-on lesson alone.

The Head Start-University Partnership, a program of the US DHHS Administration for Children and Families, has supported research on a preschool mathematics curriculum that Katherine Sophian developed for use with three- and four-year olds. The curriculum consists of weekly activities that parents and teachers are to complete with children. The emphasis of the program is on measurement with various units and exploring the relationships between shapes rather than identifying features of shapes (i.e., the number of sides or angles). The curriculum has been evaluated in a study of three Head Start centers with children ranging in age from 2.5 years old to 4.6 years. This study found that use of the curriculum had a small positive effect on the math scores of children at the end of the year.

View full report

Preview
Download

"index.pdf" (pdf, 307.32Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"apa.pdf" (pdf, 78.45Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"apb.pdf" (pdf, 401.85Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"apc.pdf" (pdf, 288.78Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"apd.pdf" (pdf, 90.04Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"ape.pdf" (pdf, 190.7Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"apf.pdf" (pdf, 68.04Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®