Hoxby Study Gets Kicked After Smackdown

On November 14, we reported on an assessment of Caroline Hoxby's unpublished charter study paid for by Mike Bloomberg. The independent evaluation boiled down to these specific criticisms:

• In measuring the effects of charter schooling on students in grades 4 through 12, the study relies on statistical models that include test scores from the previous year, measured after the admission lotteries take place. Yet because of that timing, those scores could be affected by whether students attend a charter school. As a consequence, the statistical models "destroy the benefits of the randomization" that is a strength of the study's design. (The use of a different model makes the results for students in grades K-3 more credible, he notes.)

• The report's claims regarding the cumulative effects of attending a New York City charter school from kindergarten through eighth grade are based on an inappropriate extrapolation.

• It uses a weaker criterion for statistical significance than is conventionally used in social science research (0.05), referring to p-values of roughly 0.15 as "marginally statistically significant".

• The report describes the variation in charter school effects across schools in a way that may distort the true distribution of effects by omitting many ineffective charter schools from the distribution.

This month's District Administrator has another level of criticism offered by Jonathan Gyurko:

A report issued by the New York City Charter Schools Evaluation Project in September, and since been held up as clear evidence that charter schools are doing a better job than traditional schools, is now facing criticism that its claim of being an “apples to apples” study just isn’t true.

The report, How New York City’s Charter Schools Affect Achievement, examines the performance of students who applied for openings in New York’s charter schools, which are 94 percent filled through random lotteries. When researchers compared the academic performance of those who were “lotteried in” with those who were “lotteried out,” they discovered a higher rate of achievement in the charter group.

Critics claim the report does not take into account the “peer effect,” whereby a child learns not only from teachers but from fellow students. Writing in Edwize, a blog sponsored by New York’s United Federation of Teachers, Jonathan Gyurko says, “Charter schools benefit from the fact that 100 percent of their students hail from motivated families; as a result, a charter student is surrounded by peers who are there by choice—rather than by attendance zone.”

Alexander Hoffman, writing in GothamSchools, an online news source about the New York City public schools, says the report is flawed because, in contrast to a medical study, it has no placebo group. Both groups—students in charter schools and students in traditional schools—know what kind of education they are getting. “I know from my own experience teaching that students who get their choice of schools take a bit more ownership,” Hoffman says. “If they get their second choice, or last choice, or somehow do not get their choice, that’s a big hurdle for their teachers and parents to overcome.”

Led by well-known school choice advocate Caroline Hoxby of Stanford University, the researchers claim that since these two groups were essentially the same—both comprised of students who sought admission to charter schools—they were able to make a comparison in which where students were educated, charter school or traditional school, was the only variable.. . . .