The Raj Chetty VAM Balloon Gets Popped

From NEPC:

Contact: 

William J. Mathis, (802) 383-0058, wmathis@sover.net

Moshe Adler, (917) 453-4921, ma820@columbia.edu

URL for this press release: http://tinyurl.com/kpwt6ft

BOULDER, CO (April 10, 2014) – A highly influential but non-peer-reviewed report on teacher impact suffers from a series of errors in methodology and calculations, according to a new review published today.

Professor Moshe Adler reviewed two recent reports released in September 2013 as National Bureau of Economic Research working papers. Dr. Adler’s review for the Think Twice think tank review project is published today by the National Education Policy Center, housed at the University of Colorado Boulder School of Education.

Adler is an economist affiliated with both Columbia’s Urban Planning Department as well as the Harry Van Arsdale Jr. Center for Labor Studies at Empire State College, SUNY, and the author of Economics for the Rest of Us: Debunking the Science That Makes Life Dismal (New Press, 2010).

Adler reviewed Measuring the Impact of Teachers, parts I and II, written by Raj Chetty, John N. Friedman, and Jonah E. Rockoff. Part I is subtitled, Evaluating Bias in Teacher Value-Added Estimates, and Part II, Teacher Value-Added and Student Outcomes in Adulthood. Taken together, the two-part report asserts that students whose teachers have higher value-added scores achieve greater economic success later in life.

The documents (as is standard for NBER working papers) were not peer-reviewed, yet as Adler points out, the research on which they were based has gained extraordinary attention – turning up as references in President Obama’s 2012 State of the Union address, in expert court testimony by the principal author (Chetty), in extensive news coverage, and even as a justification for Chetty’s 2012 MacArthur Foundation “genius” award.

That sort of credibility, Adler suggests in his review, may not be warranted – as demonstrated by a series of problems that he finds with the new two-part report and the research that undergirds it.

The report’s own results reveal that calculating teacher value-added is unreliable, Adler writes. Additionally, the report includes a result that contradicts the central claim; it relies on an erroneous calculation to support a favorable result; and it assumes that the miscalculated result holds across students’ lifetimes – “despite the authors’ own research indicating otherwise,” the reviewer notes.

Finally, Adler explains, the studies relied on by the report as support for its methodology don’t actually provide that support.

“Despite widespread references to this study in policy circles, the shortcomings and shaky extrapolations make this report misleading and unreliable for determining educational policy,” Adler concludes.