State-Funded Pre-Kindergarten: What the Evidence Shows. 2. Quasi-experiments show that states can implement successful pre-kindergarten programs

12/01/2003

A well-designed quasi-experiment is an alternative to the randomized experiment when randomly assigning children to different groups may not be practical. In a quasi-experiment, the performance of children who attended the target program is compared to the performance of one or more groups of children who did not attend. The most rigorous quasi-experiments use procedures to ensure that the target group and the comparison groups are as comparable as possible on the range of factors that are known to affect the desired outcomes. For example, level of maternal education is known to affect children’s school readiness; thus procedures should be used to ensure that target and comparison groups have equivalent maternal education levels to rule out the possibility that observed differences in children’s performance are attributable to differences in maternal education. The more rigorous quasi-experiments also measure children’s initial levels of performance prior to the intervention.

a. A recent report of 10 state evaluations shows promising results for state-funded pre-kindergarten.

As of 1998, 10 states had used quasi-experimental designs to evaluate their pre-kindergarten programs. These designs varied in rigor, but each design compared outcomes for children who attended state-funded pre-kindergarten with one or more comparison groups of children that did not attend the program.(28) Not only did study designs vary, but also programs varied in terms of requirements, eligibility and other characteristics. The 10 states include: Florida, Georgia, Kentucky, Maryland, Michigan, New York, South Carolina, Texas, Washington, and the District of Columbia.(29) Most of the evaluations focused on cognition, language and academic achievement, with a few including measures of social, behavioral, and health outcomes. According to recently published technical analyses of these 10 evaluations by Yale University researchers Gilliam and Zigler,(30) state programs produced either consistently positive results or positive results in some areas of competency and no effects in others. For some areas of competency positive results were often found consistently for all cohorts of children studied.(31) Though research designs do not allow for direct comparisons between state-funded pre-kindergarten programs and Head Start, effect sizes for these quasi-experimental studies of state-funded preschool programs were similar to those reported in quasi-experimental studies of Head Start.(32) (See Gilliam & Zigler, 2001, for more detail on the duration of each study, the number of cohorts studied and the proportion of cohorts with positive outcomes, inclusion of pre-tests, composition of comparison groups, sample sizes, measures, and other aspects of the study designs.)

Key findings from quasi-experiments of state-funded pre-kindergarten

Across the 10 states, the child outcomes measured included cognitive and language abilities, reading and mathematics achievement, behavior problems, child health, attendance, grades, special education referrals and placements, parent involvement and grade retention. Not all states measured outcomes in each of these areas.(33) Many tests of children’s abilities were well-known and technically sound standardized measures, while others were relatively unknown “home-grown” measures with little information on technical soundness. On balance, the report finds good evidence that states can develop and implement effective pre-kindergarten programs, with children who attended state-funded pre-kindergarten programs having higher scores in several areas compared to children who did not attend the programs. Results were especially consistent for developmental competence, including cognition and language, at the end of preschool and in kindergarten, as well as for academic achievement, reduced grade retention and school attendance in elementary school.

  • Cognitive and language outcomes were measured in nine states (Florida, Georgia, Kentucky, Maryland, Michigan, New York, South Carolina, Washington, and the District of Columbia). Four of these states (Florida, Kentucky, New York and Washington) measured outcomes at the end of preschool. All four states found consistently across all cohorts studied that children who attended state-funded pre-kindergarten had higher scores on assessments of cognitive and language abilities compared to children who did not attend state-funded pre-kindergarten.

    Six states measured cognitive and language outcomes in kindergarten. Three of the states (Florida, Maryland, Michigan) found that scores were higher for children who had attended pre-kindergarten across all cohorts studied. Three states (Kentucky, New York, and the District of Columbia) showed mixed outcomes, with some of the cohorts of children who attended pre-kindergarten showing significantly higher scores, and some not.

    Three of the states measured cognitive and language outcomes both at the end of preschool and in kindergarten (Florida, Kentucky, New York). Florida found that, for all cohorts studied, children who attended pre-kindergarten had significantly higher scores at both time points. In Kentucky and New York, all cohorts of children who attended pre-kindergarten had higher cognitive and language scores at the end of preschool, with some cohorts having higher scores into kindergarten.

    South Carolina and Georgia measured cognitive and language outcomes only for 1st graders. South Carolina found that children who attended state-funded pre-kindergarten had higher scores on cognitive and language assessments; however, Georgia did not.

    Kentucky followed children until 2nd grade and New York followed children until 3rd grade. In both states and for all of the cohorts of children followed, those children who attended pre-kindergarten had higher scores on cognitive and language assessments at the end of preschool and kindergarten. However, by the time they were in 2nd and 3rd grades, the scores of children who attended pre-kindergarten were similar to the scores of children who did not attend pre-kindergarten.

  • To assess longer-term outcomes, children’s performance on school-administered reading and mathematics achievement tests was examined in seven states (Florida, Georgia, Maryland, New York, South Carolina, Texas, and the District of Columbia). All of the states, with the exception of the District of Columbia, reported that children who attended state-funded pre-kindergarten had higher scores on achievement tests at one or more grades.

    Florida was the only state to administer achievement tests in kindergarten and to administer the tests every year through the 4th grade. For all cohorts studied, children who attended state-funded pre-kindergarten had higher test scores in kindergarten compared to children who did not attend state-funded pre-kindergarten; however, their scores were similar in grades 1 through 4.

    Maryland, Texas, and the District of Columbia tested only 3rd graders: Maryland and Texas found that all cohorts studied had significantly higher scores on achievement tests if they attended pre-kindergarten. Scores were not higher for children who attended pre-kindergarten in the District of Columbia.

    Georgia tested children at 1st grade and New York tested children at 3rd grade: In both states some cohorts of children who attended pre-kindergarten had higher achievement test scores. In South Carolina, children who attended pre-kindergarten had higher scores on achievement tests in the 1st grade, but not in the 2nd and 3rd grades.

  • Behavior problems were assessed in three states. Florida assessed behavior problems only at the 4th grade and found significantly fewer behavior problems for children who attended pre-kindergarten. Kentucky assessed behavior problems every year from the end of preschool through the 4th grade. Some cohorts of children who attended pre-kindergarten had fewer behavior problems at the end of preschool, but not in kindergarten and the later grades. Washington tested for behavior problems in 1st through 3rd grade, but no differences were found between children who attended pre-kindergarten and children who did not.
  • Washington was the only state that measured child health. Health assessments were conducted in 1st through 3rd grade, with no significant outcomes found.
  • All states that evaluated attendance in kindergarten and the early grades found for at least one cohort that school attendance was better for children who participated in state-funded pre-kindergarten compared to children who did not.
  • Six states assessed grade retention beginning at kindergarten, (Florida, Georgia, Maryland, New York, South Carolina and Texas), although data were not collected for every grade in every state. For all cohorts studied, grade retention was significantly lower in at least one of the years from kindergarten through 3rd grade for children who attended pre-kindergarten. However, no state found that grade retention for children who attended state-funded pre-kindergarten was lower every year from kindergarten through 3rd grade, and no state found that grade retention was lower by the time these children were in 4th grade.
  • For the states that measured grades, special education referrals/placements, and parent involvement, there were either no significant outcomes or few consistently positive findings.
  • All but one state gathered data on children’s social, motor, academic, literacy, and self-help skills as well as the cognitive and language skills described earlier. Not every state collected information for each skill area, and across states the information was collected at different times from the end of pre-kindergarten through 4th grade. The pattern of results across all of these skill areas showed that children who attended state-funded pre-kindergarten had higher scores than children who did not attend state-funded pre-kindergarten, and the report concludes that, overall, these score differences were “sizable and robust”.(34) Moreover, “in all state evaluations nontrivial positive outcomes were sustained to kindergarten for at least one cohort.”(35)

Overall, the report finds that these state-funded pre-kindergarten programs had positive outcomes at the end of pre-kindergarten. Longer-term outcomes were evident in kindergarten and first grade, though they were found less consistently.(36) Though some early positive outcomes persisted into elementary school, there was less success in sustaining early positive outcomes through the early grades. This “fade-out” of early outcomes is not unique to studies of state-funded pre-kindergarten. Similar results have been reported in other studies of early childhood interventions, including Head Start. Fade-out effects may be the result of problems with study methods or inadequate instruction in kindergarten and the early grades, or both. The trend has been found consistently across studies that included different types of early childhood interventions, outcome measures, and study designs. This pattern of results strongly suggests that attending high quality schools after pre-kindergarten and perhaps follow-on interventions and services are needed to sustain the immediate positive benefits of pre-kindergarten programs.

Conclusion

Considering the pattern of findings together with their limitations (which are discussed in more detail in Section “c”), it can be said that the positive results for state-funded preschool programs are “encouraging.”(37) The effect sizes and patterns obtained from these studies were similar to those that have been found for other large-scale preschool programs for low-income children, including Head Start.(38) States vary greatly on nearly every component of their pre-kindergarten programs, and “not every state has the will and infrastructure necessary to implement, maintain, and evaluate the success of a comprehensive child and family program such as Head Start.”(39) Yet, the evidence suggests that “states can implement good, comprehensive pre-kindergarten programming, and on an individual basis some states appear to be doing a good job of promoting school readiness.”(40)

b. A more recent quasi-experiment shows that children in Georgia’s pre-kindergarten program make progress in school readiness.

Since the 10-state study review, a more recent quasi-experimental evaluation of Georgia’s pre-kindergarten was released. The primary goal of the study was to examine the effect that the Georgia pre-kindergarten program has on children’s cognitive, language, pre-reading, and pre-mathematics skills, social-emotional development and physical health. The study also examined the progress of children who attended Georgia Head Start and Georgia private preschools.(41)

The Georgia Pre-kindergarten program was established in 1993 and is funded by the state lottery. The program requires a 1:10 teacher-child ratio and a classroom size of 20 or fewer children. The program has school readiness standards, mechanisms for parent involvement, and offers extended services. The program began as a means-tested program, but in 1995 was made available to all four-year-olds whose parents chose to enroll them.

The study examined the progress of a representative sample of all children in the program, and also looked in-depth at a sub-sample of children in Georgia pre-kindergarten with disadvantages similar to those experienced by children in Georgia Head Start. The characteristics of classrooms and teachers were examined for all three Georgia programs (Georgia state pre-kindergarten, Georgia Head Start and private preschool), as well as how these characteristics related to children’s scores on school readiness assessments. The evaluation of the effectiveness of Georgia’s state pre-kindergarten includes a quasi-experimental design that in some ways can be considered more advanced than those used in previous evaluations:

  • The study design takes into account that children’s development is a complex process that is influenced by factors other than preschool, including experiences at home and characteristics of individual children and families. Data were collected on a range of child and family characteristics and used in most of the statistical analyses to control for their influence on children’s outcomes.
  • Child outcomes were collected from a variety of sources that included: 1) widely used and standardized direct assessments of children’s abilities that were administered by trained professionals and allowed comparisons to national norms, 2) direct assessments administered by trained professionals that have been used in other large-scale studies of early childhood programs, including Head Start, and 3) teacher ratings of children’s development and school readiness, and parent surveys.(42)
  • Other technical merits of the analysis included the collection of baseline data for children at the beginning of preschool, allowing changes in children’s abilities to be measured from the beginning of preschool until the beginning of kindergarten; the use of growth-curve analyses to identify the developmental trajectories for children who attended Georgia’s pre-kindergarten program, as well as for children in the two other groups; and the use of technically sound measures of process quality.(43)
  • However, there are still methodological limits to the study and definitive causal statements about the effectiveness of the pre-kindergarten program relative to other programs cannot be made. Differences in children’s outcomes may be due to differences in the effectiveness of the programs or to unmeasured differences in children’s background and entry skills that were not controlled in the analyses.

Findings for the Overall Program

  • On average, children enrolled in Georgia pre-kindergarten came from families with wide ranges of economic resources, parental education, involvement with children and schools, and family structures. Children in Head Start were from homes and families with the greatest disadvantages, and children in private preschools were the least disadvantaged. Some Georgia pre-kindergarten children resembled children in Head Start, while others were similar to children in private schools.
  • Teachers in Georgia pre-kindergarten had significantly higher education levels than teachers in Georgia Head Start or private preschools. Fifty-four percent of Georgia pre-kindergarten teachers had bachelor’s degrees compared to 13% in Georgia Head Start and 20% in private preschools. In Georgia pre-kindergarten, 13% of teachers had less than an associate’s degree compared to 70% in Georgia Head Start and 75% in private preschools. Teachers in Georgia pre-kindergarten as well as Head Start offered classroom experiences that were of significantly better quality than private preschools, and Georgia pre-kindergarten teachers offered these most consistently across classrooms.
  • Process measures of program quality showed that for all three programs — Georgia pre-kindergarten, Head Start and private preschools — higher quality preschool experiences were associated with significantly higher scores on: receptive vocabulary, problem-solving/pre-mathematics, story comprehension and print familiarity, and mastery of basic skills (color knowledge, counting, and number recognition).(44) Letter-word recognition scores were not associated with program quality. These results were maintained after controlling for the influence of child and family characteristics. Because these data are correlational, it cannot be determined whether program quality had a causal influence on children’s test scores.(45)
  • During the preschool year (from fall entry to spring exit), children in Georgia pre-kindergarten made significant gains in: vocabulary, letter-word recognition and expressive language, and mastery of basic skills (knowing colors, counting, and number recognition). Gains were not made in problem-solving/pre-mathematics, story comprehension or print familiarity.

Findings for Disadvantaged Children

The study authors explain that because most children in Georgia have some type of preschool experience, it was not practical to compare children in Georgia’s pre-kindergarten to a group of children without preschool experience in order to assess program effectiveness. Therefore, the evaluation compared children in Georgia’s pre-kindergarten program to children in Georgia Head Start and private preschool to attempt to determine the contribution that Georgia pre-kindergarten makes to children’s progress. However, children in Georgia’s pre-kindergarten program differed significantly from children who attended Head Start.(46) When study groups differ greatly, “self-selection” bias prevents knowing for sure if an intervention caused the observed result or if one intervention produces better outcomes than another.(47) Given differences between the characteristics of children in Georgia pre-kindergarten and Head Start, the evaluation included a sub-study to examine how the progress of the most disadvantaged children attending Georgia pre-kindergarten compared with the progress of a similarly disadvantaged group of children in Head Start. The sub-study used matching procedures and advanced statistical techniques to control as much as possible for self-selection bias.(48)

These procedures resulted in groups that were similar on many, but not all characteristics. Demographics were similar with the exception that disadvantaged children in Georgia pre-kindergarten had mothers with significantly higher levels of education, which is known to impact children’s cognitive performance. The children in the sub-study sample attending pre-kindergarten scored significantly higher on two of six measures (vocabulary and basic skills) upon entering the program than the children attending Head Start.

Sub-study of disadvantaged children: Results for the preschool year(49)

  • Disadvantaged children who attended the Georgia pre-kindergarten appeared to make gains over the preschool year on many aspects of school readiness, including letter-word recognition, story comprehension and print familiarity, receptive vocabulary, and expressive language. Statistical tests have not been conducted to determine if these gains were statistically significant.
  • Disadvantaged children in Georgia pre-kindergarten began the preschool year with significantly higher scores on two of the six measures assessed than those entering Head Start. They scored higher in receptive vocabulary and mastery of basic skills. By the end of the preschool year, the difference remained for vocabulary but not for mastery of basic skills. There were additional differences showing that by the end of the year children in Georgia pre-kindergarten had higher scores on letter-word recognition, story comprehension and print familiarity, and expressive language. There were no differences in problem-solving/pre-mathematics scores. These analyses did not control for differences in child and family characteristics between the Georgia pre-kindergarten and Head Start groups.

Sub-study of disadvantaged children: Results at kindergarten entry

As of kindergarten entry, the pattern of outcomes showed similar results:

  • From preschool entry to kindergarten entry, disadvantaged children in Georgia pre-kindergarten made significant gains in vocabulary, letter-word recognition, story comprehension and print familiarity, problem-solving/pre-mathematics and mastery of basic skills. Compared to children in Georgia pre-kindergarten, gains for children in Head Start were significantly greater for vocabulary. The two groups showed similar gains in all of the other areas.
  • By the beginning of kindergarten, disadvantaged children in Georgia pre-kindergarten scored significantly higher than children in Head Start in vocabulary, letter-word recognition, story comprehension and print familiarity, problem-solving/pre-mathematics and mastery of basic skills. Kindergarten teachers’ ratings of school readiness, academic skills and behavior showed similar differences. Teacher ratings for health and physical well-being were similar for children in Georgia pre-kindergarten and children in Head Start.
  • Score differences on direct assessments and teacher ratings at kindergarten entry could be the result of differences in the effectiveness of programs, differences in child and family characteristics, or differences in children’s experiences over the summer.

In conclusion, based on these results, the evidence suggests that the Georgia pre-kindergarten program is likely to be providing experiences that help disadvantaged children get ready for school.

c. Enhancements are needed in future evaluations of state-funded pre-kindergarten.

This recent study of Georgia pre-kindergarten and the 10-study review by Gilliam and Zigler are the best data available on how state-funded pre-kindergarten programs affect children’s outcomes. However, the amount, type, and quality of the data limit what we can know about the effectiveness of state-funded pre-kindergarten programs.(50) (Most of these limitations also apply to existing studies of Head Start, though a randomized study of Head Start impacts is underway.) To ensure that data from future evaluations are useful, areas that need particular attention include, but are not limited to, the following:

  • Though the authorizing state legislation for many pre-kindergarten programs requires a formal evaluation of program implementation and effectiveness, less than half of state-funded preschool programs have or are currently conducting studies that can adequately test program results. More evaluations of state-funded pre-kindergarten programs are needed to document program results and to identify areas of program strength and areas for program improvement.
  • None of the studies that have been conducted or that are currently underway use randomized designs. More randomized experiments are needed to assess the causal impact of state-funded pre-kindergarten programs. As explained earlier, randomly assigning children to attend state-funded pre-kindergarten is the only way to know for sure if the program, as opposed to other factors, is responsible for any observed results.
  • When randomized experiments are not feasible, quasi-experiments should include well-matched and well-defined control groups that allow for stronger inferences about the effect of programs on child outcomes. Even the most rigorous quasi-experiments cannot completely guard against the possibility that unmeasured differences between children who attended the program and children who did not are responsible either in full or in part for study findings. However, quasi-experiments should, at a minimum, assess children’s abilities at program entry and examine the similarities and differences in the background characteristics of children and families for both the pre-kindergarten and comparison groups that could affect the child outcomes that are measured.
  • To have confidence in the results, the measures used to assess children’s level of performance and longer-term progress should be technically sound and appropriate for the population studied.
  • Statistical analyses should be conducted properly (for example, statistical corrections should be made to correct for significant outcomes that may be obtained by chance due to the large number of significance tests conducted). In addition, the statistical and educational meaningfulness of the sizes of effects should be reported along with results from tests of statistical significance.
  • Better strategies are needed for computing grade retention, special education referrals, and similar variables.
  • Attrition occurs in every long-term study, but it is important to know how children who stayed in the study differed from children who did not and to ensure that the groups of children who remain in the study have the characteristics for which they were originally chosen (such as equivalent income or maternal education levels). Study procedures or statistical analyses should be used to control for attrition patterns that could bias the results, and these approaches should be described in evaluation reports.
  • The procedures for selecting samples often are not reported or are not reported in sufficient detail. Often it is unclear whether the study samples are representative of children attending pre-kindergarten in the state.
  • The studies typically do not break down results for children by their demographic characteristics such as ethnicity, gender and income. More in-depth analyses of study subgroups are needed to determine which children benefit from the program and which do not.

There are several key issues of concern to policy makers that need to be considered when designing future research. Existing studies have design limitations. For example:

  • As with existing Head Start research, research design limitations prevent knowing definitively whether the state-funded pre-kindergarten programs caused the positive effects observed on children’s performance. As mentioned earlier, children must be randomly assigned to the state program or to a comparison group to know for sure that the program caused positive effects, or quasi-experimental studies should have comparison groups that allow for making strong inferences about the causal effects of pre-kindergarten program.
  • Studies of state-funded pre-kindergarten programs cannot directly address whether state-funded pre-kindergarten programs produce effects that are better or worse than Head Start because none of the studies included a sample of Head Start children as a comparison group. In fact, no studies have been designed to rigorously and directly compare the effectiveness of state-funded pre-kindergarten initiatives to the effectiveness of Head Start.
  • Information on the amount of progress children make during the preschool year usually is not available because children typically were not tested before entering pre-kindergarten and again at the end of pre-kindergarten.
  • Studies typically do not address whether the state-funded pre-kindergarten programs narrowed the achievement gap for disadvantaged populations. Reports usually do not compare the performance of disadvantaged children who attended the program with other disadvantaged children who did not attend the program or with a sample of more advantaged children. Children’s performance is rarely compared to national norms.
  • Most states did not measure the full range of knowledge and skills that early childhood programs need to target in order to promote children’s school readiness. This restricted range of measurement limits what can be known about each state’s profile of strengths and weaknesses in promoting school readiness and early school achievement. Because outcome measures varied across studies, strengths and weaknesses cannot be compared across state-funded pre-kindergarten programs.
  • The evaluations did not include implementation studies or process measures to determine how each component of the program was implemented and the degree to which full implementation was achieved. As a result, the studies cannot show which components of the programs contributed to positive outcomes.

In conclusion, existing research on the results of state-funded pre-kindergarten programs has technical limitations that constrain what can be known about the impact of state-funded pre-kindergarten programs on children’s outcomes. In addition, the studies were not designed to answer some critical questions, such as whether state-funded pre-kindergarten programs produce better outcomes than other programs serving similar populations or whether they reduce or eliminate achievement gaps. Taking these caveats into consideration, for some states there is promising evidence that they can implement pre-kindergarten programs that produce positive outcomes in areas that include cognition, language, and academic achievement, with some positive outcomes, such as improved achievement test scores, reduced grade retention and school attendance, in the elementary grades.

View full report

Preview
Download

"paper.pdf" (pdf, 782.83Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®