Eric A. Hurley
Anne Chamberlain
Success for All Foundation

Robert E. Slavin
Johns Hopkins University

Nancy A. Madden
Success for All Foundation

September, 2000

Eric A. Hurley and Anne Chamberlain are research scientists at the Success for All Foundation (SFAF), in Baltimore.  Robert E. Slavin is Co-Director of the Center for Research on the Education of Students Placed at risk at Johns Hopkins University and Chairman of SFAF.  Nancy Madden is President and CEO of SFAF.  This article was written under funding from the Office of Educational Research and Improvement (OERI), U. S. Department of Education (No. R-117-40005). However, any opinions expressed are those of the authors and do not necessarily represent Department of Education positions or policies.

Success for All is a comprehensive reform model for elementary schools, especially Title I schoolwide projects.  Begun in 1987 in one inner-city Baltimore school, Success for All is now in more than 1,800 schools in 49 U.S. states, as well as five other countries.  In Fall, 2000, about one million children will be in schools implementing Success for All.

An important strength of Success for All, and a key to its rapid growth, is the amount and quality of the research done to evaluate the program¹.  Studies in 12 school districts have compared Success for All and matched control schools on individually administered measures of reading, especially the Woodcock, Durrell, and Gray informal reading inventories².  These studies have reported that on average, Success for All schools have exceeded controls on these measures by approximately 50% of a standard deviation, or in grade equivalents approximately 2.5 months in first grade, increasing to 1.1 years by fifth grade.  Effects have been particularly positive for the lowest-achieving students, and as a result, the program has had a profound effect on reducing special education placements and retention³.  In addition to research using student level data and individual-administered measures, several studies have found positive effects of Success for All on a variety of standardized and norm-referenced measures used in state accountability programs⁴.

In all, studies of thousands of elementary school children in dozens of schools and in a variety of settings have been published in the most selective journals in education, and have consistently reported positive impacts of SFA on student achievement.  A few studies have also reported atypical program impacts on student achievement.  The balance of evidence would suggest that such results may be attributable to implementation quality, a variable known to be critical to all school reform efforts, and one that is too often left unconsidered.

Nonetheless, the rigor and variety of evidence indicating positive impacts of SFA on student achievement has established the effectiveness of the SFA program design well beyond the empirical standards typical of education research. One review of research by the American Institutes of Research⁵ on 24 comprehensive reform designs found very few that had ever been compared to a control group. Only two, Success for All and Direct Instruction, met the highest standards for evaluation rigor and outcome. A later compendium by the Thomas Fordham Foundation⁶ came to the same conclusion.

Despite this research, some scholars have questioned the basic effectiveness of Success for All and the evidence which supports it, charging in particular that the majority of the studies were completed by John Hopkins researchers or their colleagues and that "independent" studies have found few benefits beyond the early grades, especially on the group-administered measures for which schools are increasingly held accountable. 

The present study uses data from the Texas Assessment of Academic Skills (TAAS) that are available on the Internet.  This type of outcome data are often reported by program developers, but usually have little scientific validity, since the data presented are for schools selected because they happened to have done well in a given year.  Yet these standardized test scores are the outcomes of greatest importance to educators.  The present study deals with this problem of selection bias by including every school that began  Success for All from 1994-1997, a total of 111 schools, and comparing their gains from the year before SFA to 1998.  By including every Success for All school and every other Texas school, we eliminate selection bias further, objectivity is assured by the fact that any researcher with Internet access can replicate the analysis.  Universal inclusion creates an analysis that is high in generalizability and meaning for policy.

In addition, there is a need to examine the effects of Success for All for students of different ethnicities. Most previous investigations of Success for All have involved schools that are almost entirely African-American or Hispanic, so results for different ethnic groups have not typically been reported. One study that did report data by ethnic group, by Ross, Smith, & Casey⁷, found that while Success for All increased the achievement of both African-American and White students, the gains for African-American students were larger (relative to control groups). At the end of the study there were no ethnic group differences in the Success for All schools, while ethnic group differences remained substantial in control schools. The Ross et al.⁸ study involved only two SFA and two control schools.  However and its analyses of ethnicity by treatment interactions have not been repeated. Positive effects of Success for All have been documented many times for African-American students, for Hispanic students, and for White students⁹, but there have not been large enough samples of each ethnic group to permit conclusive comparisons of relative impacts for each ethnic group. Impacts for African-American and Hispanic students are particularly important, of course, because these groups typically do not score as well as White students on reading measure¹⁰. These differences have profound implications for later school achievement and life success, and are a leading factor in broader social inequities. Convincing evidence that Success for All can significantly help to narrow the reading gap between African-American, Hispanic, and White students, as suggested by Ross et al.¹¹, would represent an extremely important development for education policy and practice. The present study allows for comparisons by ethnic group to illuminate this issue.

This article presents analyses of data from the Texas Assessment of Academic Skills (TAAS) reading measures. It sought to evaluate the program’s outcomes in all of the 111 Texas schools that began the program from 1994-1997. TAAS reading scores are collected in all elementary grades starting in grade 3, so the analyses presented here evaluate the effects of Success for All in the upper-elementary grades. This analysis is by far the largest evaluation of Success for All (in fact, it is the largest evaluation of any comprehensive reform model ever conducted), and it is the first large-scale study to examine results separately by student ethnicity. Further, while the analysis presented could be considered first party, the data were taken from the Texas Education Agency (TEA) web site and are thus available for replication by any interested parties.

What is Success for All?

Success for All is a program designed to ensure the reading success of every child by applying a combination of innovative instructional approaches in grades pre-kindergarten to five or six, one-to-one tutoring for primary-grades children who are struggling in reading, family support programs, and school organization and professional development approaches intended to ensure high-quality implementation of all program elements and replicability in a wide variety of circumstances¹². The program is disseminated by the Success for All Foundation, a non-profit organization that spun off from Johns Hopkins University in 1998. Each element of the program was included because it had a strong base in research, and the program is continually revised based on new developments in research as well as responses to the experiences of schools implementing the program. A hallmark of the program is specificity; well-structured student materials, teacher’s manuals, assessments, and other supports have been developed for every aspect of the program at every grade level. During the spring before program initiation, school staffs are encouraged to learn about Success for All, to read the research, and to visit other schools using it. Ultimately, the entire staff must vote by at least 80% to adopt the program. Also, the school must be able to afford the program which costs (for a school of 500 students) about $75,000-$80,000 in the first year, $30,000 in the second year, and $20,000 i n the third year, plus staff who areusually reallocated from other functions.

A child’s progression through Success for All begins in prekindergarten or kindergarten. The program at this level uses thematic, interdisciplinary units, story telling and retelling, language development activities, phonemic awareness and alphabet activities, and other elements designed to build children’s oral language, background knowledge, concepts of print, and preliteracy skills. From mid-kindergarten into first grade, the program provides a balanced approach to reading instruction that uses a systematic approach to phonics taught in the context of meaningful text, very much in line with the recent blue-ribbon reviews of research on the components of effective early literacy instruction¹³. Children experiencing difficulty with early reading instruction receive daily one-to-one tutoring from certified teachers or well-trained paraprofessionals, to ensure that each of them will get off to a good start in reading.

Beyond the first grade level, teachers use strategies adapted from Cooperative Integrated Reading and Composition, or CIRC¹⁴. In this approach, children work in small, cooperative learning teams on activities focusing on main idea, summarization, vocabulary building, home reading, and creative writing.

Success for All places much emphasis on family support programs, which give the school’s staff strategies for increasing parent involvement and for increasing attendance, improving classroom management and preventing behavior problems, integrating with social and health services, and solving other non-academic problems.

A full-time facilitator, usually an experienced teacher from the school’s own staff, works with all teachers to ensure effective program implementation and reviews schoolwide assessments given every eight weeks to help focus attention on children who are not making adequate progress. Trainers from the Success for All Foundation provide schools with about 24 person-days of on-site training and follow-up during the first implementation year, with continuing training and follow-up in later years. In addition, there is a week-long training for the principals and facilitator before the program begins, and conferences for experienced schools each spring following program initiation.

For schools with many English language learners, Success for All provides two alternatives. Schools with Spanish bilingual or dual language approaches use a Spanish adaptation, Exito para Todos, while those using an English as a second language approach use strategies developed to help English language learners build reading skills and English skills in tandem.

The Texas Evaluation

Texas provided an ideal setting for a large-scale, statewide evaluation of Success for All. First, there are a large number of Success for All schools in the state; until 1999-2000, Texas had the largest number of SFA schools of any state (it was edged by California in that academic year). Second, Texas was one of the first states to make its state assessment and school-level demographic data available on the Internet, making analyses far easier than would have otherwise been the case. Third, Texas used the same assessment, the Texas Assessment of Academic Skills (TAAS), with few changes in the test itself or in the basis used to compute test scores, from 1994-1998, making longitudinal analyses possible. Unfortunately, the Texas Education Agency (TEA) changed test procedures in 1999, requiring inclusion of special education students and significantly changing the rules regarding administration of the English TAAS to limited English proficient children, especially those children who have been taught reading primarily in Spanish. These changes made the 1999 and 2000 data non-comparable with the earlier scores.

One possible problem with the TAAS is that scores on this measure, expressed as the percentage of students meeting a proficiency standard, rose significantly statewide during 1994-1997, and concern has been expressed that there is a ceiling effect for high achievers. This issue is addressed in the analyses described in this article.

The 111 Texas schools involved in this evaluation are almost all Title I schoolwide projects and have very high poverty levels. On average, 85% of the children are designated economically disadvantaged according to state criteria (compared to 45% in the state as a whole). Otherwise, however, the SFA schools are quite diverse. While most are in large cities, especially Houston, San Antonio, Galveston, and El Paso, many are in small towns or rural areas (e.g., Eagle Pass, Harlingen, Morton, and Muleshoe), and some are in inner-suburban districts (such as Aldine near Houston and the North East, Northside, and Southside districts near San Antonio). The schools of interest serve many more minority students than is true in the population of Texas schools. According to state data, students in the Success for All schools were 25% African-American, 62% Hispanic, and 13% White. Corresponding proportions for the state are 14% African-American, 35% Hispanic, and 47% White. Limited English proficient students were 27% of the SFA sample, but only 12% of the state.

Findings

Overall Analyses

For an initial, overall analysis, TAAS Reading scores, obtained from the TEA web site, were averaged for each of the 111 Success for All schools across grades 3, 4, and 5. Each cohort consisted of all schools that began to implement Success for All in the designated academic year (for example, schools in the 1994 cohort have used the program for four years). Gains from the spring before program inception to spring, 1998, were computed for each cohort, and compared to gains in the state as a whole for the same period. The gain scores are graphed in Figure 1.

As shown in Figure 1, gains for each cohort of Success for All schools were greater than gains for the state during the same time period, with both the raw amount of gain and the relative advantage for Success for All schools generally increasing with each additional year of implementation, from a difference of 4.6 percentage points for one-year schools to a difference of 7.7 points for four-year schools. Gain scores were combined across cohorts by subtracting from each SFA school’s gain the gain made in the state over the same period. On average, the Success for All schools gained 5.85 percentage points more than the state, (t (110) = 6.25, p < .001, two tailed). The effect size, the mean difference divided by the standard deviation of school means, was + 0.59.

As noted earlier, the relative gains for Success for All schools may have been affected by a ceiling effect on TAAS. As also noted earlier, Success for All schools are far more impoverished than other Texas schools and contain a much higher proportion of minority students and limited English proficient students. Because of these differences, the gain scores shown in Figure 1 compare dissimilar populations, and should be interpreted cautiously.

To create more comparable samples and to obtain information valuable in its own right, analyses were carried out within ethnic groups. Data by ethnic group (African-American, Hispanic, and White) were also available on the TEA web site. Scores for a given school are reported by TEA if there are at least five students of a given ethnic group in a given grade.

Results for African-American Students

Results for African-American students are shown in Figures 2 and 3. Figure 2 shows gain scores for each of four cohorts (1994, 1995, 1996, and 1997), as in Figure 1. Analysis indicated that African-American students in 66 Success for All schools gained 5.62 percentage points more than those in control schools (t (65) = 2.97, p < .004, two tailed). The effect size was + 0.37. Figure 3 shows the same data in raw percent passing. In this analysis, the African American students are comparable to African-American students in the state as a whole, and there is little evidence of ceiling effects. For example, the 1995 cohort (3 years in SFA) started at 63% passing and increased to 86%, while African-American students in the state as a whole started at 64% and increased to 79%. Only in the 1997 cohort (1 year in SFA) was there a substantial difference at pretest between African-American students in SFA and those in all of Texas.

The net effect of the gains for African-American students in Success for All schools was to substantially narrow the gap with White students. At pretest, African-American students in the 1995 cohort were 24.6 percentage points behind Whites, while at postest (1998) they were 6.5 percentage points behind (while other African Americans remained 13.8 points behind). The closing of the Black-White achievement gap for all Texas students may well be influenced by a ceiling effect for Whites, but the relative advantage for African Americans in SFA schools is unlikely to be affected by this factor.

Results for Hispanic Students

The pattern of results for Hispanic students, the largest ethnic group in Texas Success for All schools, was generally similar to that of African-American students. Figures 4 and 5 summarize the results. Not surprisingly, the greatest gains for Hispanic students in Success for All schools (relative to those in the state as a whole) were for the 1994 cohort (4 years in SFA). In the 1995 cohort, Hispanic students gained somewhat more in the state as a whole. Still, combining across all 95 schools with adequate numbers of Hispanic students, gains for Hispanics in Success for All schools were significantly greater than those for other schools, with a mean difference of 3.35 (t (94) = 2.75, p < .007, two tailed). The effect size was + 0.28. Figure 5 shows the same data in raw percent passing. In this analysis, the Hispanic students are comparable to Hispanic students in the state as a whole, and here again there is little evidence of ceiling effects. Unfortunately, there is no way to do separate analyses for Spanish-dominant students taught in Spanish or English, as these data are mixed in with data for English-dominant students of Hispanic origin. For studies of Success for All that did make these distinctions, see Slavin and Madden¹⁵.

Results for White Students

For White students, the ceiling effect on TAAS scores was most acute. In 1998, 92.7% of White students statewide, an increase from 85.0% in 1994, passed TAAS, meaning that there were many schools that started and ended that period with 100% of their White students passing. Fifty of the SFA schools had enough White students to report data for this subgroup. White students in Success for All schools gained substantially more than other White students in the 1995 cohort (3 years in SFA) and the 1996 cohort (2 years in SFA), but gains were about equal in the larger 1997 and 1994 cohorts. The overall comparison for White students in 50 schools was not statistically significant, but was directionally positive with an effect size of +0.17.

Conclusion

The Texas statewide data reported here show that Success for All schools are significantly and substantially closing the gap in TAAS reading performance between themselves and the far less impoverished schools in the rest of the state. This was particularly true for African-American and Hispanic students, for whom the gap with White students closed significantly more than it did for African-American and Hispanic students in other schools.

The Texas data are important in several ways. First, they provide evidence from widely available test scores showing greater growth on a state accountability measure than was achieved by other schools in the state. The sample of Success for All schools, 111 schools serving more than 60,000 children, was large enough to allow for analyses using school means as the unit of analysis, a very conservative test. Clearly, some schools did better than others in both implementation and outcome; no schools were excluded from the analysis due to poor implementation, or any other reason. At the policy level, it is crucial to know that whatever differences there may have been among schools, the overall effort produced meaningful gains on measures that all Texas schools are trying to affect. These data suffer from all of the random factors that plague all accountability measurement, such as the effects of mobility, differential special education assignments, different amounts of missing data, different test preparation strategies, and different practices regarding testing of limited English proficient students. Yet in a large-scale assessment, these confounding factors balance out, showing a true treatment effect.

Further, the findings of this study show again that the impact of Success for All can be seen on standardized test data in the upper elementary grades (3-5), not just in the early elementary grades where the impacts have historically been best-documented and undisputed by critics.

The results presented here also raise some questions for further research. The outcomes for African-American and Hispanic students largely mirror the positive results seen in earlier studies. However, there have been many fewer studies including significant numbers of White students. Those that have been done¹⁶ have found positive effects of Success for All for White students, or for samples that are majority-White, but a study in Ft. Wayne, Indiana, found (as in the present study) that the effects were greatest for African-Americans, significantly narrowing the achievement gap with Whites¹⁷.

For Hispanics as well, outcomes of Success for All in the present study as well as in earlier studies have been very positive, whether the students have been English-dominant, initially Spanish-dominant and taught in Spanish, or initially Spanish-dominant and taught in English¹⁸.

Clearly, more research is needed to examine different effects of Success for All for different ethnic groups, not only to determine what these differences are but also to understand why they occur. There is no more important policy question in American education than the question of how to eliminate the gap in school performance between African-American and Hispanic students and their White counterparts. This difference appears early, and appears first in reading¹⁹. Research on Success for All holds out the possibility that programs designed to ensure the reading success of all children may be particularly beneficial for African-American and Hispanic children, who have historically been underserved in American schools. This possibility must be explored further.

¹Rebecca Herman, an Educator's Guide to Schoolwide Reform (Washington, DC: American Institute for Researh, 1999).
²Robert E. Slavin and Nancy A. Madden, eds., One Million Children: Success for All (Mahwah, NJ: Erlbaum, 2001).
³Robert E. Slavin, "Neverstreaming: Preventing Learning Disabilities," Educational Leadership, February 1996, pp. 4-7; and Lana J. Smith, Steven M. Ross, and Jason P. Casey, Special Education Analyses for Successfor All in Four Cities (Memphis: Center for Research in Educational Policy).
⁴Nancy A. Madden, Robert E. Slavin, Nancy L Karweit, Larry J. Dolan, and Barbara A. Wasik, "Success for All: Longitudinal effects of a restructuring program for inner-city elementary schools", American Educational Research Journal, vol. 30, 1993, pp. 123-148; John Nunnery, Steven Ross, Lana Smith, Robert Slavin, Phyllis Hunter, and Joseph Stubbs, "An Assessment of Success for All Program Component Configuration Effects on the Reading Achievement of at-risk First Grade Students." Paper presented at the annual meeting of the American Educational Research Association, New York, 1996; Slavin & Madden, op. cit.
⁵Herman, op. cit.
⁶James Traub, Better by Design? A Consumer’s Guide to Schoolwide Reform (Washington, DC: Thomas Fordham Foundation, 1999).
⁷Steven Ross, Lana Smith, and Jason Casey, 1994-1995 Success for All Program in Ft. Wayne, IN: Final Report (Memphis: University of Memphis, Center for Research in Educational Policy, 1995).
⁸Ibid.
⁹Robert E. Slavin and Nancy A. Madden, "Research on Achievement Outcomes of Success for All: A Summary and Response to Critics," Phi Delta Kappan, Sept., 2000, pp. 38-40, 59-66; Robert E. Slavin and Nancy A. Madden, eds., One Million Children: Success for All (Mahwah, NJ: Erlbaum, 2001).
¹⁰Patricia Donahue, Kristin Voelkl, Jay Campbell, and John Mazzeo, NAEP 1998 Reading Report Card for the Nation (Washington, DC: U.S. Department of Education, National Center for Education Statistics, 1999).
¹¹Ross et al., op. cit.
¹²Slavin and Madden, op. cit.
¹³National Reading Panel, Teaching Children to Read: An Evidence-Based Assessment of the Scientific Research Literature on Reading and its Implications for Reading Instruction (Rockville, MD: National Institute of Child Health and Human Development); C.E. Snow, S. M. Burns, and P. Griffin, eds., Preventing Reading Difficulties in Young Children. (Washington, DC: National Academy Press, 1998).
¹⁴Robert J. Stevens, Nancy A. Madden, Robert E. Slavin, and Anna Marie Farnish, "Cooperative Integrated Reading and Composition: Two Field Experiments," Reading Research Quarterly, vol. 22, 1987, pp. 433-454.
¹⁵Robert E. Slavin and Nancy A. Madden, "Effects of Bilingual and English as a Second Language Adaptations of Success for All on the Reading Achievement of Students Acquiring English," Journal of Education for Students Placed at Risk, vol. 4, 1999, pp. 393-416.
¹⁶Steven M. Ross, John Nunnery, and Lana J. Smith, Evaluation of Title I Reading Programs: Amphitheater Public Schools. Year 1: 1995-96 (Memphis: University of Memphis, Center for Research in Educational Policy, 1996); Steven M. Ross, Lana J. Smith, Jason Casey, and Robert E. Slavin, "Increasing the Academic Success of Disadvantaged Children: An Examination of Alternative Early Intervention Programs," American Educational Research Journal, vol. 32, 1995, pp. 773-800.
¹⁷Ross, Smith, and Casey, ibid.
¹⁸Slavin and Madden, 1995, op. cit.
¹⁹Donahue et al., op. cit.