Jay Picks a New Charter School to Fawn Over - but Forgets the Data! (Or, Another Reminder About Why Jay's Newsweek Rankings are Junk)

Earlier this week, Jay Mathews posted about the possibility of a new charter school coming to D.C. The operator of the school, BASIS Schools, runs a number of schools in Arizona and has drawn attention because of their high standards, heavy use of AP classes/tests, and stellar rankings on the US News & World Report and Jay's very own Newsweek list. 

In this post, I'm going to point out a few issues here: 

1. BASIS Tucson, their flagship school, has a serious attrition problem;
2. BASIS Tucson's high attrition means they skyrocket up Jay's Newsweek list; and
3. Jay's concluding remarks about the school and the possibility of others giving this model a try is pretty silly (to put it kindly).
4. [As a bonus, there's also a bit about AZ charters not keeping track of how many F/R lunch kids they serve]

BASIS Tucson attrition: The Arizona Department of Education's website includes a handy page that'll give you statistics on school enrollment. I took this enrollment data from BASIS' Tucson location and put them on one Excel spreadsheet. As you can plainly see, there's TREMENDOUS attrition at this school. The 2009/2010 graduating class, for instance, had 71 6th graders in 2004/2005, but only 24 managed to graduate by the time that cohort of students had made their way to 12th grade. 

[Click to enlarge]

A few caveats here: it's abundantly clear that BASIS Tucson loses a lot of kids between the time they start in 5th grade and the time that cohort of kids makes to 12th grade. It is possible, however, that BASIS Tucson students graduate in 11th grade, which would (unfairly) make their 12 grade classes seem very small. I think that's likely a very minor issue. The other thing to keep in mind is that we don't know if, for instance, the 24 seniors that graduated in 2010 were all part of the class of 71 6th graders during the 2003/4 school year. Students may have entered or exited the school - and the only way to know about that is to use student-level, longitudinal data (which I don't have). 

Gaming Jay's Newsweek Rankings: Jay mentions that Tucson's BASIS charter ranks 6th on his list of "America's Best High Schools." Let me point out how Jay comes up with his "challenge figure," which is the *only* measurement he uses to rank schools (via Newsweek's FAQ):

We take the total number of Advanced Placement, International Baccalaureate or Cambridge (AICE) tests given at a school each year and divide by the number of seniors graduating in May or June.

Pretty simple and straightforward - just straightforward enough for the simpleminded folks that somehow believe a single measure is a sufficient way to measure a school. Schools giving a lot of AP/IB/AICE tests with a small graduating class of seniors is likely to skyrocket up Jay's list. Does it tell us something about a school if they give a heck of a lot of AP/IB/AICE tests and have a small senior class? Yes - it tells us they give a lot of AP/IB/AICE tests and only have a small class of seniors. It really, really doesn't tell us much about the quality of the school. And, of course, the serious attrition at BASIS Tucson means their graduating class of seniors is small - which, once again, makes Jays ranking more than a bit silly.

Silly Conclusion: Here's Jay's concluding paragraph:

It will be interesting to watch. No one has ever planned for D.C. students to work this hard and do this well. If such high standards succeed at BASIS, other schools may have to give them a try.

It might make sense to have other schools give this a try - but only if BASIS was able to keep kids from 5th grade all the way through 12th grade. That's quite clearly not happening. This school is the epitome of the argument that charter schools skim off certain students and discard the rest. Jay gives precisely zero indication of these issues, and his conclusion lacks nuance. 

If anything, this is a boutique model that might work for a select group of kids, but it's likely not something that other schools, especially public schools that are *required to serve all kids*, can emulate. 

Bonus!: Jay asks:

Can such a school survive in the District? Michael Block says 26 percent of BASIS Tucson students are Hispanic or black, and estimates about 20 percent are low-income. Yet the school’s average SAT score last year, 1854, was above the 1852 average at Langley High School in McLean, where only 2 percent of students come from low-income homes. Seventy-two percent of BASIS Tucson’s Hispanic students and 88 percent of its black students had passing scores on AP tests in 2010, compared to passing rates of 42 percent and 27 percent for those ethnicities nationally.

In the era of measurement, data, more measurement and even more data, how is it possible that BASIS can only offer an estimate of the number of low-income students they serve? Well, BASIS doesn't keep track of F/R lunch students because they don't have a meal program (in fact, only about 35% of AZ's charter schools have a lunch program). The 20% figure provided by BASIS Tucson is "based on the number of seniors who needed waivers for SAT/college applications," according to Julia Toews, head of BASIS Tucson (via email).