Guest Post: No Excuses for obfuscating abysmal attrition data with complex calculations

By John Thompson

MacArthur Genius grant recipient Roland Fryer did some complex calculations in "Creating 'No Excuses' (Traditional) Public Schools" to estimate student performance gains during the first year of Houston's "Apollo 20." The massive pilot program extends "No Excuses" instruction to neighborhood schools. Fryer, however, omitted a simple statement of its most important data for that type of experiment. In contrast to "No Excuses" charters, low-income students were not going out of their way to volunteer for 21% more hours of instruction and much more rigorous enforcement of academic and behavioral standards. Some of those charters have had attrition rates of 40% or more. So, an evaluation of whether Fryer's model could be scaled up for neighborhood schools requires an estimate of how many students opted out during the program.

Fryer reported that over 7,000 students were enrolled in nine schools, and that all results were based on the sample of students who remained in school long enough to take the spring tests. He did not reveal how many students took those tests. Or if Fryer did, I could not find it.

Fryer found large gains in math for the two grades where Houston could afford more than $2,500 per student for personalized tutoring, but overall results were disappointing. Middle school students largely declined in reading while high school kids increased for a total effect on reading that was statistically insignificant. Results for "double-dozing" math and reading classes also were discouraging.

Fryer acknowledged, however, that even those modest increases are questionable if his statistical models could not account for challenging students opting out. But he was not very transparent in reporting the students' attrition rate. Back when I was in school, we could turn to the tables at the back of an academic paper and discern that sort of essential information. So, I checked out Table Two on page 71 and looked for something like, "N = x." All I found, however, was that Fryer reported 8,693 "observations" on students who began the school year with Apollo and took two spring tests.

So, I checked out the district's interim report, dated January 2011, which said that 7,385 students were in the program. The Houston Chronicle reported in the fall of 2011, only 6,156 enrolled at the beginning of the second year. But still, it was impossible to tell from Fryer's work whether attrition affected his results. The district's interim report, however, said that the students who entered the Apollo program were 86.6% economically disadvantaged. According to Fryer's report, the sample of students who remained until testing was 61% economically disadvantaged. In other words, Fryer started with nine high-poverty schools, but his conclusions seem to be based on a sample that were not far from average poverty for public schools.

Fryer's report is preliminary and it is categorized as a "working paper," but his "back of the envelope" estimates of return on investment are just as perplexing. He worries that the most effective treatment, where each highly-recruited tutor worked intensively with two students, would be difficult to scale up. Even so, Fryer asserted that the return on investment would be 2-1/2 times as great as the returns that James Heckman found for high-quality pre-school. Heckman, however, evaluated gains that were sustained over time, and clearly Fryer did not. Apparently, Fryer's estimate was based on test score increases, which he compared to increases studied by Alan Krueger due to reducing class size. Even if Fryer is a true believer in data-driven instruction, however, it is hard to understand that how anyone could assume that one-year boosts in math scores, after a diet of interim tests every three weeks, as well as benchmark tests, would be life-transforming.

That gets us to back to the key issue. Fryer claims that if all five characteristics of "No Excuses" schools are implemented with fidelity, then the legacy of poverty can be overcome. What else would it take to scale up his model? "Apollo 20" required the removal of 310 teachers and nine principals. The district replaced 100 instructors with Teach for America teachers, and 60 veterans with a history of increasing test scores. Those transfers received two annual $10,000 bonuses. Replacing nine principals required 200 interviews and one third of those who were hired have since been replaced. A national search attracted 1,200 applicants for tutors. Of the 287 who were hired, thirty left. All three groups were subjected to intense training in "No Excuses" values and techniques and were subjected to detailed and expensive oversight.

It is hard to imagine where Houston, or any other district, could find enough of the type of educators that their experiment requires. The key metric, however, was not the attrition of those carefully-selected adults, but that of the students. The big question is how many high-poverty neighborhood school students stuck out the year, and whether test score gains were inflated by the loss If anyone can find that answer in Fryer's work, or explain why he did not volunteer that information so that everyone could evaluate his evidence for themselves, I'm all ears.

Dr. John Thompson was an award-winning historian, lobbyist, and guerilla-gardener who became an award-winning inner city teacher after crack and gangs hit his neighborhood. He blogs at thisweekineducation.com, and huffingtonpost.com, and is writing a book on 18 years of idealistic politics in the classroom and realistic politics outside.