The Added Value of the School of the 21st Century When Combined With a Statewide Preschool Program

Authors


  • This study was funded by the Winthrop Rockefeller Foundation.

Correspondence concerning this article should be addressed to Misty Ginicola, Southern Connecticut State University, Counseling and School Psychology, 501 Crescent Ave., New Haven, CT 06511. Electronic mail may be sent to ginicolam2@southernct.edu.

Abstract

Although schools have begun to employ multiple programs to reach educational goals, little attention has been paid to the efficacy of combining separate programs. The present study investigates the combination of a school reform model, the School of the 21st Century (21C), and the Arkansas Better Chance (ABC) program, where preschools apply for state funds to provide high quality services to children at risk. To determine if 21C adds anything to the ABC program, the present study compared results from a developmental assessment. The authors compared school-based preschools that had both ABC and 21C services to those programs that offered only ABC services. Results indicated that, at baseline, children within the 21C/ABC programs scored the same or poorer than their ABC-only counterparts. However, at the end of the year, participants at 21C/ABC preschools were significantly ahead of ABC-only programs on all developmental indicators. There were also differences favorable to 21C/ABC preschools between the programs' quality ratings as measured by the Early Childhood Environmental Rating Scale.

Schools face an inordinate number of issues in contemporary society, including academic problems, bullying, low parental involvement, special education needs, and low teacher and staff morale, to name a few. To combat these varied types of problems, multiple programs are often utilized within the school setting. However, with the advent of multiple school-based programs, questions regarding cost-effectiveness and efficacy often take center stage. Are these programs useful in combination? If the programs are broad based and holistic in nature, they may be efficacious alone and in combination with other school-based initiatives.

One such broad-based program is the School of the 21st Century (21C). First conceptualized by Zigler in 1987, 21C is a school reform model that includes school-based child-care and family support programs. The 21C program encourages schools to provide preschool education, quality child-care services, and family support for children from conception to age 12. The primary goal of 21C is to ensure the optimal development of all children through the provision of good quality care and evidence-based programming. Recommended 21C services, referred to as program components, include (a) early preschool child care and education, (b) school-age child care, (c) home visitation programs and parent education services, (d) health and nutrition education and services, (e) information and referral services, and (f) networks and training for preschool child-care providers in the area of the school (Finn-Stevenson & Zigler, 1999). Availability and affordability of these services are crucial, as the goal of 21C is to provide access to all children regardless of family income. In addition, the school and community dictate the nature and implementation of the services needed, so no two 21C sites are alike.

Research on 21C has demonstrated the efficacy of the program on a variety of student, family, and school outcomes. To date, the 21C program has been shown to establish high-quality services, helping children to improve their overall school readiness and to achieve higher math and reading scores than a control group (Desimone, Finn-Stevenson, & Henrich, 1999; Finn-Stevenson, Desimone, & Chung, 1998; Finn-Stevenson & Zigler, 1999). In addition, 21C has been well received by parents, who indicated significantly less stress, less money spent on day care costs, and fewer days missed from work when they enroll children in 21C (McCabe, 1995). In terms of overall school effects, 21C schools have significantly less school vandalism, increased parental involvement, and improved community relations (Deemer, Desimone, & Finn-Stevenson, 1998; Finn-Stevenson & Zigler, 1999). Although this research base has indicated positive findings across school settings, newer program evaluations could continue to support the efficacy of the program.

21C has been implemented in more than 1,300 schools across the country with statewide initiatives located in Connecticut and Kentucky. The most recent statewide 21C implementation has occurred in Arkansas. In 2001, the Winthrop Rockefeller Foundation (WRF) provided grant support to establish 21C as a statewide initiative. The effort, known as the AR21C Initiative, included start-up funds, technical assistance, and training and evaluation for approved schools. As of 2007, 21C has been implemented in 42 school districts across Arkansas in more than 160 sites. In accordance with 21C's guiding principles, Arkansas program implementation reflects the variations in needs and resources among communities.

There are existing school-based resources in Arkansas, as it has been identified as a state in high need; almost 50,000 3- and 4-year-old children in Arkansas live at or below the federal poverty level (Pre-K in Arkansas, 2007). The Arkansas Better Chance (ABC) program, funded by the state, provides early education to these preschool children who are considered at risk (Pre-K in Arkansas, 2007). A preschool program that receives ABC funding must meet national regulations for safety and the quality credential of the National Association for the Education of Young Children (NAEYC), as well as all state licensing requirements and quality approval standards through the Arkansas Department of Human Services, Division of Childcare and Early Education. Every program must also be officially rated through a reliably and valid quality indicator (most commonly the Early Childcare Environmental Rating Scale–Revised Edition [ECERS-R]; Harms, Clifford, & Cryer, 1998). Research on ABC sites indicates that programs maintain high quality (Barnett, Hustedt, Hawkinson, & Robin, 2006). A recent report by Hustedt, Barnett, Jung, and Thomas (2007) also indicated that the ABC program improved children's vocabulary, writing, and math skills. This program does, however, need more extensive program evaluation to reliably infer efficacy.

A significant portion of the ABC sites have implemented preschool child care as part of 21C. As such, they met both the standards of ABC programs and received 21C training and funding. Additionally, these sites offer other components of 21C, thereby ensuring continuity in intervention before, as well as after, preschool years and a comprehensive array of support services for children and families. The importance of continuity of intervention, as opposed to a focus on one age group of children (e.g., preschoolers), is noted as an effective approach more likely to yield positive child outcomes (Brookings Institute, 2007).

The existence of both groups of programs provides a unique opportunity for comparison between schools with ABC and 21C services (21C/ABC) and those sites with ABC alone (ABC-only). Thus, program evaluators can determine the effectiveness of each program alone and in combination. Whereas the centers implementing ABC-only would be of high quality and have parental involvement programming, 21C/ABC sites would have those elements as well as additional support services and characteristics of the 21C model. 21C differs from preschool child-care programs in several ways, most notably in the continuity of support services from the conception of the child through age 12 and the integration of school and community resources (Finn-Stevenson & Zigler, 1999). The hypothesis for the present study was that both programs' participants will evidence improvements across the course of the year, as existing research points to the efficacy of each program. However, it was believed that the 21C program would add to the benefits of ABC and thus would show a larger impact than ABC-only programs.

Method

Participants

Participants were 4-year-old children (= 8,745) enrolled in ABC classrooms in Arkansas. Basic demographic characteristics of the 21C/ABC group (= 2,869) and ABC-only group (= 5,876) are presented in Table 1. Although groups were similar in gender makeup and percentage of parents who paid for services, these groups differed significantly in terms of percentage of minority and English as a Second Language (ESL) students. Specifically, 21C/ABC sites had significantly more minority students and more ESL learners than the ABC-only sites. These demographic differences could tend to work against the hypothesis tested.

Table 1. Demographics of Participants
VariableABC/21CABC-only χ 2 p
Gender

49.7% boys

50.3% girls

50.8% boys

49.2% girls

0.89>.05
Minority students63.3%41.3%376.44<.001
ESL10.5%8.6%9.33<.05
Offered in a public school94.9%53.3%1514.16<.001
Private pay clients2.8%3.2%0.99>.05

Measures and Procedure

All participants' developmental progress was tracked by lead preschool teachers using the Work Sampling System (Meisels, Jablon, et al., 1995), which is required by the Arkansas Department of Human Services, Division of Childcare and Early Education for ABC fund recipients. This assessment is a structured system of checklists that allows teachers to observe, document, and analyze each individual child's strengths and challenges as well as the child's progress toward goals over time. This criterion-referenced observational assessment provides teachers with very specific performance indicators of each construct in order to improve reliability and validity. Checklist data fall into seven domains: (a) personal and social development, (b) language and literacy, (c) mathematical thinking, (d) scientific thinking, (e) social studies, (f) the arts, and (g) physical development and health. Each checklist item reflects the behaviors and activities that a child should present if he or she is at or above the intended developmental level. The teacher enters data into an online program at three different times throughout the school year (beginning, middle, and end). The reports yield two levels of information in regard to these domains. First, the report supplies a mean rating and standard deviation for each time period. Second, each individual is assigned a certain category based on their score: not yet (meaning absent), in process, or proficient. All teachers were blind to the hypothesis of the study. Research has indicated that this test has very high internal consistency reliability (.87 to .94; Meisels, Liaw, Dorfman, & Nelson, 1995). Additionally, the Work Sampling System showed adequate concurrent and predictive validity when compared with the Woodcock Johnson, the McCarthy Scales of Children's Abilities, and the Child Behavior Rating Scale (Meisels, Liaw, et al., 1995).

In addition to Work Sampling assessments, ABC sites provided program quality data using the ECERS-R (Harms et al., 1998) ratings for their program, which is the standard assessment for child-care programs. Child-care quality was measured by an ECERS-R trained outside rater who was also blind to the hypothesis of this study. The ECERS-R is designed to assess group programs for children aged preschool through kindergarten on seven scales: (a) space and furnishings, (b) personal care routines, (c) language–reasoning, (d) activities, (e) interactions, (f) program structure, and (g) parents and staff. The ECERS has 43 items across the seven scales; the ECERS-R requires the rater to observe and rate indoor space, furniture, furnishings, privacy, safety, nap, toileting and diapering, access to books and pictures, verbal communication, types of activities, supervision, staff–child interactions, scheduled activities, free time, provisions for parents and staff interaction, and evaluation, to name a few. The ECERS-R is considered developmentally appropriate and culturally sensitive and is used broadly. It has been validated and is considered reliable, with interrater reliability correlations averaging approximately .90.

Because Arkansas state agencies collect data on the programs, access to the data was restricted to basic demographics, work sampling data, and ECERS-R scores. At the time of the study, Work Sampling data were also available for release only in the aggregate format. Because data were limited to means and standards deviations, a series of t tests was conducted.

Results

For the purposes of this study, we will review differences between the groups in each domain at Period 1 (baseline pretest) and Period 3 (comparison postintervention). Data were analyzed by comparing average scores and proficiency percentages of each group. Response rates on subtests for developmental indicators ranged from 98.3% to 99.7% in the ABC-only group and 98.1% to 99.8% for the 21C/ABC group.

21C/ABC Versus ABC-Only Sites

At pretest, 21C/ABC programs were either on par with or significantly below the outcomes of ABC-only sites, as shown in Table 2 and Figure 1. Although equivalent on other areas of assessment, ABC/21C sites were at a significant disadvantage in the areas of personal and social development, English language and literacy, mathematical thinking, and scientific thinking. These data indicate that 21C/ABC preschools not only contained more children who would be considered at risk, but they also had lower scores on the developmental measures at baseline than those children in the ABC-only group.

Table 2. Baseline and Postintervention Data for 21C/ABC and ABC-Only Sites Comparison
Subscale21C/ABCABC-OnlyPretest tPosttest tPosttest Cohen's d
PretestPosttestPretestPosttest
  1. Note. Standard deviations are in parentheses. The t test is based on the comparison between 21C/ABC and ABC-only at pretest and posttest. Cohen's d was calculated on the posttest data only.

  2. *Significant at the .05 level. **Significant at the .01 level. ***Significant at the .001 level.

Personal & social development 2.19 (0.45)2.83 (0.28)2.21 (0.46)2.79 (0.31)1.92* (df = 8,741)5.23*** (df = 8,738).11
Language & literacy2.04 (0.46)2.77 (0.34)2.08 (0.47)2.74 (0.36)3.76** (df = 8,741)3.72*** (df = 8,741).08
Mathematical thinking1.98 (0.45)2.78 (0.36)2.01 (0.47)2.75 (0.38)2.84** (df = 8,739)3.53*** (df = 8,739).08
Scientific thinking1.97 (0.47)2.79 (0.38)2.01 (0.51)2.73 (0.43)3.52*** (df = 8,722)6.36*** (df = 8,739).14
Social studies2.05 (0.46)2.80 (0.33)2.05 (0.47)2.75 (0.37)0.00 (df = 8,699)6.13*** (df = 8,712).13
The arts2.15 (0.49)2.83 (0.32)2.15 (0.50)2.78 (0.37)0.011 (df = 8,722)6.19*** (df = 8,740).13
Physical development & health2.36 (0.47)2.93 (0.20)2.37 (0.45)2.89 (0.25)0.96 (df = 8,735)7.48*** (df = 8,737).16
Figure 1.

Baseline differences between ABC/21C and ABC-only in work sampling data.

For the principal hypothesis to be supported, the 21C/ABC sites would need to make greater gains during the school year than the ABC-only group, given their lower scores at baseline. The results presented in Table 2 and Figure 2 indicated that not only did the 21C/ABC participants catch up to the ABC-only participants, but also they significantly surpassed them on every measured variable. 21C/ABC participants scored significantly higher in the areas of personal and social development, language and literacy, mathematical thinking, scientific thinking, social studies, the arts, and physical development and health at the < .001 level. As indicated in Table 3, effect sizes for these averaged d = .12, which is a small effect size. This effect size level is on par with other intervention studies and is considered meaningful (Ludwig & Philips, 2007; U.S. Department of Health & Human Services, Administration for Children & Families, 2005). One major confound with this comparison is that 21C sites are generally located in schools, whereas approximately half of ABC-only sites are in schools. Therefore, a second analysis comparing 21C/ABC school sites to ABC-only school sites was also conducted to control for this possible confound.

Table 3. Baseline and Postintervention Data for 21C/ABC and ABC-Only Schools Comparison
Subscale21C/ABC SchoolsABC-Only SchoolsPretest tPosttest tPosttest Cohen's d
PretestPosttestPretestPosttest
  1. Note. Standard deviations are in parentheses. The t test is based on the comparison between 21C/ABC and ABC Schools at pretest and posttest.Cohen's d was calculated on the posttest data only.

  2. *Significant at the .05 level. **Significant at the .01 level. ***Significant at the .001 level.

Personal & social development 2.19 (0.46)2.83 (0.28)2.16 (0.46)2.79 (0.31)2.40* (df = 5,428)4.97*** (df = 5,426).14
Language & literacy2.03 (0.47)2.76 (0.34)2.03 (0.47)2.74 (0.37)0.00 (df = 5,429)2.07* (df = 5,426).06
Mathematical thinking1.97 (0.45)2.79 (0.35)1.99 (0.47)2.76 (0.38)1.62 (df = 5,428)3.02** (df = 5,426).08
Scientific thinking1.96 (0.47)2.80 (0.38)2.00 (0.50)2.75 (0.42)3.02** (df = 5,411)4.58*** (df = 5,426).12
Social studies2.04 (0.47)2.81 (0.32)2.03 (0.45)2.76 (0.36)0.80 (df = 5,412)5.38*** (df = 5,426).15
The arts2.14 (0.50)2.84 (0.31)2.12 (0.49)2.79 (0.36)1.49 (df = 5,412)5.46*** (df = 5,427).15
Physical development & health2.35 (0.47)2.93 (0.19)2.31 (0.44)2.89 (0.25)3.24** (df = 5,427)6.58*** (df = 5,426).18
Figure 2.

Postintervention differences between ABC/21C and ABC-only in work sampling data.

21C/ABC Versus ABC-Only Schools

To account for the possibility that the school setting may provide an advantage, participants in 21C/ABC schools (= 2,578) were compared with ABC-only programs that were based only in schools (= 2,853). As shown in Table 3, the school setting did not explain the original findings in this study. Although 21C/ABC programs were higher on ratings of physical and social development at pretest, all other subscale scores were on par with or lower than ABC-only schools. Just as in the previously mentioned analyses, at posttest, 21C/ABC sites were significantly ahead of the ABC-only schools in all areas.

Early Childcare Environmental Rating Scale–Revised Edition scores in all ABC sites are outstanding, as a minimum of an overall score of 5.0, the threshold for good quality, is required to continue receiving funds.1 However, as shown in Table 4, 21C/ABC schools (= 93) were rated significantly higher than ABC-only programs (= 147) on every ECERS-R subscale and total score. The average effect size was = .36, indicating a small to medium effect.

Table 4. Baseline and Postintervention ECERS Scores for 21C/ABC and ABC-Only Schools Comparison
ECERS    
Subscale21C/ABC SchoolsABC-Only SchoolsPretest tPosttest Cohen's d
  1. Note. Standard deviations are in parentheses.

  2. *Significant at the .05 level. **Significant at the .01 level. ***Significant at the .001 level.

Space & furnishings 5.05 (0.94)4.72 (0.92)2.67** (df = 238).35
Personal care routines4.27 (1.20)3.88 (1.13)2.54** (df = 238).33
Language–reasoning5.87 (0.90)5.47 (1.13)2.88** (df = 238).39
Activities5.58 (0.91)5.31 (1.03)1.91* (df = 238).29
Interactions6.00 (1.07)5.69 (1.22)2.01* (df = 238).27
Program structure5.77 (1.03)5.33 (1.30)2.76** (df = 238).38
Parents and staff6.26 (0.53)6.05 (0.60)2.76** (df = 238).37
Total5.49 (0.65)5.16 (0.75)3.49*** (df = 238).47

Discussion

Participants in both 21C/ABC and ABC-only preschools improved dramatically from the beginning to the end of the school year. The hypothesis that children receiving 21C in addition to ABC services would evidence better outcomes was supported. Despite the fact that children in the 21C/ABC program began the year significantly lower on several measures, these children ended the year significantly above children in the ABC-only group on every subtest of the Work Sampling System. Children in the 21C/ABC group were more likely to be from a minority and ESL background than those in the ABC-only group. 21C sites in Arkansas were chosen for funding if they identified a financial need and a large number of at-risk students. Therefore, it is likely that there is an economic difference between the two populations as well. However, this difference would actually put the 21C/ABC group at a disadvantage in the comparison between groups, which was evidenced in the lower baseline scores.

These results bring attention to the idea that multiple school-based programs can be effective alone or when combined. However, what exactly is it about these two combined programs that can inform school services research on programs that may be efficacious when combined? The focus of the ABC program remains narrow in scope; by increasing the quality of the preschool environment, the hope of the program is to increase school readiness in preschoolers and subsequent academic achievement, especially for children that are considered at risk for school failure. In this regard, the ABC program has been found to be a successful program in improving developmental indicators and kindergarten grades in at-risk children (Hustedt et al., 2007). The 21C program, on the contrary, is much more broad in its scope; although the 21C program also includes a high-quality preschool component, it has several more program components and guiding principles, which go well beyond preschool services to maintain continuity into elementary school. The combination of these two programs allows for a comprehensive service model: a broad-based program that can be tailored to a community (21C) with a program that focuses on creating the highest possible quality preschool environment (ABC) to address conceptually important and empirically supported service elements.

Given the finding that participants from ABC/21C sites had significantly higher scores than those from ABC-only sites, what service elements then did 21C add to the ABC programs in Arkansas? The provision of 21C within an already excellent ABC program was shown to add quality, as measured through an ECERS-R score. The 21C program requires program coordinators to apply the guiding principles to their preschool programs in a consistent and concerted manner. The 21C sites thus included additional support services, which may have been responsible for the increased outcomes: 21C sites regularly include access to health and mental health services, information, and referral services, as well as networking and training for child-care providers in the community.

Additionally, 21C sites focus on parent involvement. It is well known that home environments influence children's school performance. 21C is designed to support parents by providing needed support services, thereby reducing parental stress and improving the home environment.

Another possible explanation is that the training and networking component present within the 21C sites is a positive influence. In Arkansas, the AR21C network facilitates the development of 21C programs and provides a venue for program coordinators and 21C schools' educators to interact with and learn from one another.

There are some limitations to this study. First, comparison groups were not even or matched. Additionally, because we had access only to aggregate data, advanced levels of significance testing and interaction analyses (e.g., interaction between program groups and demographic variables) could not be tested.

However, this study is an important first step in identifying the effectiveness of 21C. The authors are currently working with Arkansas authorities to obtain individual Work Sampling data for a later cohort to repeat our comparisons with data based on individual scores rather than group scores. Thus, future studies may include an analysis of the interaction with important demographic variables (e.g., gender, minority status). However, this investigation is a promising addition to the literature on the effectiveness of both 21C and ABC programs.

  1. 1

    ABC programs with low ECERS-R scores receive focused technical assistance to reach minimum standards.

Ancillary