What the Gates MET Study Really Shows

The final report by the $50 million Gates Foundation Measures of Effective Teaching experiment concludes that their research into value-added models “should give heart to those who have invested considerable effort to develop practices and policies to measure and support effective teaching.” But, it gives no evidence or logic why those policies would work in the real world.

For some reason, Gates researchers claim that studying outcomes of randomly assigned students, from the same school, could determine whether statistical models can distinguish whether some teachers are “truly better” at improving student learning as opposed to being beneficiaries of “better students.”  Had teachers from low-poverty schools been randomly assigned to high-poverty neighborhood schools, that experiment might have produced important findings.  

Why the foundation would bother with an investigation that does not address the issue of effective teachers being fired because they chose to teach in ineffective schools is anyone’s guess. But, Gates admitted, “Our study does not allow us to investigate bias in teacher effectiveness measures arising from sorting between different schools.” 

Neither can I understand why scholars would sign their names to a policy report that does not even reveal the percentage of their study’s sample who are low-income. A year ago, the foundation quietly acknowledged that only 8% of their sample were on special education IEPs. It did not have low-income data for one of the districts it studied. The rest of their sample of students was only 56% low-income. The final report used the missing data from that district as a rational for not providing the low-income rates of its final sample.     

An accompanying paper hinted, however, that the final conclusions were based on a sample of students that is even more unrepresentative. For instance, they randomly assigned middle school teachers to teach four classes with only twenty students per class.

It mentions the demographics of the few high school classes that were analyzed.  Traditionally, the percentages of special education students are higher in high schools, as are the number of truant students with discipline problems.  Less than 8% of the Gates’ high school sample was on IEPs, while they averaged only 11 absences per student.  Incredibly, their sample averaged .15 suspensions per student. During my 19 years in the inner city, I have never had a non-honors class that comes close to resembling the Gates sample.

Buried in the footnotes was an acknowledgment that attrition and transfers further undermined the representativeness of the study.  The randomly assigned students who stuck it out to the end-of-the-year testing were significantly less likely to be on IEPs or to be classified as English Language Learners (ELL).  It admitted, “To the extent that special education students and ELL students tend to be sorted into lower achieving classrooms, this may account for the greater degree of sorting outside of the randomized MET project classrooms.”

The foundation did not cite the research by Audrey Amrein-Beardsley, Edward Haertel, Linda Darling Hammond, and Jesse Rothstein on the ways that value-added is unfair to teachers of classes with high percentages of low-income, special education, and ELL students.

The real problem with the theories of the Gates Foundation, and the rest of the “Billionaires Boys’ Club” is that they ramrodded risky changes in school law without first studying whether those policies could work.  Similarly, the MET paper failed to provide a rationale for staying the course and continuing to attempt value-added evaluations.  Their policy argument came down to one chart - albeit an attractive and multi-colored one.  It showed four bars of correlations with evaluations based on differing percentages of value-added. 

If, for argument’s sake, we accept the Gates’ logic, it does no more than show correlations that apply to the students and teachers that the MET studied.  No effort was made to explain why those patterns would apply to districts that are 80% to 90% low income and where most families have self-sorted by fleeing to charters, magnet school, or low-poverty suburban and/or gentrified schools. 

And, even if someday over the rainbow, economists found a way to control for intense concentrations of poverty, that would only be the beginning of the policy issues that the Gates foundation, and fellow “reformers” like Arne Duncan, failed to consider before imposing their theories on school systems. 

At least for the moment, favored districts like Hillsborough and Memphis are drowning in Gates money for teacher evaluation experiments.  The foundation offers no evidence, however, that their gold-plated systems would be cost effective for the rest of the nation. 

Gates argues that value-added can be done correctly (emphasis mine), and it need not encourage more rote instruction and test prep.  Value-added need not be used improperly, so it need not encourage more educational malpractice, endless litigation, and/or an exodus of teaching talent from schools where it is harder to raise test scores. But, why did the MET not ask whether their theories could work out in such a way in systems with flesh and blood administrators under the gun, during an age of "reform?