No spin. Here is the Executive Summary of the National Center for Education Statistics (NCES) study comparing charters to regular public schools, with more interesting parts highlighted by me.
Charter schools are a relatively new, but fast-growing, phenomenon in American public education. As such, they merit the attention of all parties interested in the education of the nation’s youth. Accordingly, the National Assessment Governing Board (NAGB), which sets policy for the National Assessment of Educational Progress (NAEP), asked the National Center for Education Statistics (NCES) to conduct a pilot study of charter schools. A special oversample of charter schools, conducted as part of the 2003 fourth-grade NAEP assessments, permitted a comparison of academic achievement for students enrolled in charter schools to that for students enrolled in public noncharter schools. The school sample comprised 150 charter schools and 6,764 public noncharter schools. School participation rates were 100 percent for both charter and public noncharter schools; student participation rates were 92 percent and 94 percent for charter and public noncharter schools, respectively. Initial results employing data from the 2003 NAEP fourth-grade assessments in reading and mathematics were presented in the NCES report America’s Charter Schools: Results From the NAEP 2003 Pilot Study (NCES 2004).
The present report comprises two separate analyses. The first is a “combined analysis” in which hierarchical linear models (HLMs) were employed to examine differences between the two types of schools when multiple student and/or school characteristics were taken into account. The rationale was that if the student populations enrolled in the two types of schools differed systematically with respect to observed background characteristics related to achievement, then those differences would be confounded with straightforward comparisons between school types.
HLMs were a natural choice for this analysis because such models accommodated the nested structure of the data (i.e., students clustered within schools) and facilitated the inclusion of variables describing student and school characteristics. In the combined analysis, the focus is the average difference in school means between the two types of schools in reading and mathematics. (This difference is similar to but not identical with the average difference between the two student populations.) Parallel analyses were carried out for reading and mathematics. In addition, supplementary analyses were conducted to evaluate the sensitivity of the results to various assumptions.
While the first analysis compares charter and public noncharter schools, the second analysis focuses on charter schools only. HLMs were employed to examine the relationship between mean school achievement and various characteristics of charter schools. Many of these characteristics were derived from a specially designed survey responded to by administrative staff in participating charter schools. Statistical significance was determined at the .05 level.
Results From the Combined Analysis
Reading
In the first phase of the combined analysis, all charter schools were compared to all public noncharter schools. The average charter school mean was 5.2 points lower than the average public noncharter school mean. After adjusting for multiple student characteristics, the difference in means was 4.2 points. Both differences were statistically significant. The adjusted difference corresponds to an effect size of 0.11 standard deviations. (Typically, about two-thirds of scale scores fall within one standard deviation of the mean.)
In the second phase, charter schools were classified into two categories based on whether or not they were affiliated with a public school district (PSD). Each category was compared separately with public noncharter schools. On average, the mean scores for charter schools affiliated with a PSD were not significantly different from those of public noncharter schools. However, on average, the means of charter schools not affiliated with a PSD were significantly lower than the means for public noncharter schools, both with and without adjustment. The effect size of the adjusted difference was 0.17 standard deviations.
In the third phase, the comparison between school types was restricted to schools having a central city location and serving a high-minority population, as there has been particular interest in those students who have traditionally not fared well in public schools. For this subset of 61 charter schools, there were no significant differences (for any fitted model) between the average charter school mean and the average public noncharter school mean.
Mathematics
In the first phase of the combined analysis for mathematics, all charter schools were compared to all public noncharter schools. The average charter school mean was 5.8 points lower than the average public noncharter school mean. After adjusting for student characteristics, the difference in means was 4.7 points. Both differences were statistically significant. The adjusted difference corresponds to an effect size of 0.17 standard deviations.
In the second phase, charter schools were classified into two categories based on whether or not they were affiliated with a PSD. Each category was compared separately with public noncharter schools. On average, the mean scores for charter schools affiliated with a PSD were not significantly different from those for public noncharter schools. However, on average, the means of charter schools not affiliated with a PSD were significantly lower than the means for public noncharter schools, both with and without adjustment. The effect size of the adjusted difference was 0.23 standard deviations.
In the third phase, the comparison between school types was restricted to schools having a central city location and also serving a high-minority population. There was a significant difference between the average of all charter school means and the average of public noncharter school means, as well as between charter school means not affiliated with a PSD and public noncharter school means. In both cases, the difference favored public noncharter schools, and the effect size of the adjusted difference was 0.17 standard deviations. However, there were no significant differences between the average of public noncharter school means and the means of charter schools affiliated with a PSD.
Sensitivity analysis
Since most charter schools are located in a relatively small number of jurisdictions, the distribution of charter schools across jurisdictions is not proportional to the distribution of all public schools. It is possible, therefore, that a national comparison between school types could be confounded with average differences in achievement among states. Accordingly, a set of parallel analyses for reading and mathematics was conducted for which the criterion was the difference between the standard student outcome and the mean NAEP score for the state. The results of the second set of analyses were very similar to those from the first set, with the effect size in the second set typically being a little smaller. While there appeared to be some confounding, it was not sufficient to alter the conclusions materially.
NAEP data are derived from a complex survey, and reported NAEP statistics are based on appropriately weighted student data. The HLM results were also based on the use of both student-specific and school-specific weights. Since there is no consensus on how to apply weights in a multilevel regression context (Pfefferman, et al. 1998), HLM analyses were rerun with different combinations of weights. Again, the results were quite similar to those obtained in the primary analysis.
Results From the Charter-School-Only Analysis
In addition to background data about the school, the charter school survey collected information about a number of areas related to school functioning, including policies from which the school had waivers or exemptions, areas in which the school was monitored, entities to which the school was required to report, student population served, and program content. For each area, a number of variables were constructed to represent the responses to the questions. All of these factors, together with student and school background variables, were incorporated in a series of HLMs in order to identify those characteristics that best accounted for differences in mean achievement across charter schools. The variation among school means for reading was nearly twice as large as it was for mathematics. Moreover, the number and nature of characteristics retained differed for reading and mathematics.
Reading
Nearly two-thirds of the variation among all students can be attributed to the variation between students within schools. Differences among schools on student variables (such as gender, race/ethnicity, disability status, status as an English language learner, and eligibility for free/reduced price lunch) accounted for 57 percent of the variance among school means. A reduced set of 10 school characteristics (such as teacher experience, region of the country, areas in which charter schools are monitored, and whether or not a charter school was part of another public school district) accounted for a further 27 percent of the variance. Thus, overall, student and school characteristics accounted for about five-sixths of the variance among school means. Of the 10 school characteristics, 3 were derived from the charter school survey (state monitoring of student achievement, monitoring for compliance with state/federal regulations, and charter school type), and 1 of the 3 (charter school type) was not statistically significant.
Mathematics
Approximately two-thirds of the variance among all students can be attributed to the variation between students within schools. Differences among schools on student variables accounted for 55 percent of the variance among school means. A reduced set of seven school characteristics (such as waivers for certain requirements, areas monitored, and charter granting agency) accounted for a further 11 percent of the variance. Thus, overall, student and school characteristics accounted for about two-thirds of the variance among school means. All seven school characteristics were derived from the charter school survey, and three (waiver for curriculum requirements, waiver for assessment requirements, and state agency granted charter) were statistically significant.
Cautions and Interpretations
There are a number of caveats to bear in mind in interpreting these results. First, the conclusions presented pertain to national estimates. Results based on a census of public schools in a particular jurisdiction may differ. Second, the data are obtained from an observational study rather than a randomized experiment, so the estimated effects should not be interpreted in terms of causal relationships. In particular, charter schools are “schools of choice.” Parents may have been attracted to charter schools because they felt that their children were not well-served by public schools, and these children may have lagged behind their classmates. On the other hand, the parents of these children may be more involved in their children’s schooling and provide greater support and encouragement. Without further information, such as measures of prior achievement, there is no way to determine how patterns of self-selection may have affected the estimates presented. That is, the estimates of the average difference in school means are confounded with average differences in the student populations, which are not adequately captured by the student characteristics employed in the analysis. It is also the case that students currently enrolled in charter schools have spent different amounts of time in one or more such schools. Consequently, the contributions of charter schools to students' learning vary across students both because of the differential effectiveness of the programs and the different amounts of exposure students have had to these programs.
Summary
After adjusting for student characteristics, charter school mean scores in reading and mathematics were lower, on average, than those for public noncharter schools. The size of these differences was smaller in reading than in mathematics.
Charter schools differ from one another in many ways. Some characteristics pertain to all public schools. Other characteristics—such as policies from which the school had waivers or exemptions, areas in which the school was monitored, entities to which the school was required to report, student population served, and program content—pertain only to charter schools. Such characteristics accounted for some of the observed variation in mean school performance.
For example, charter schools differ on whether or not they are affiliated with a public school district. In reading and mathematics, average performance differences between public noncharter schools and charter schools affiliated with a public school district were not statistically significant, while charter schools not affiliated with a public school district scored significantly lower on average than public noncharter schools.
No comments:
Post a Comment