KIPP Study Shows Declining Test Performance and No Improvements in "Character"

by Jim Horn

The final piece of Mathematica’s $4 million KIPP study was just published.  It was designed principally to try to gauge the effect of KIPP's continuing expansion on student achievement at elementary, middle, and high school levels. 

One of the prominent findings demonstrates that KIPP’s heyday came about 10 years ago, and KIPP’s continued expansion in the number of schools in recent years (thanks largely to a $50 million handout from USDOE) correlates closely to a decidedly downward trend in KIPP's ability to raise test scores.  The chart below is from p. 31 of Mathematica report.

Middle Schools
The best news for KIPP, Inc. relates to test performance by students in the 43 KIPP middle schools included in the study.  And if we were to read only the key findings summary provided by Mathematica, which obviously provides the basis for reporting by the Washington Post, we could be as misled as reporter Lyndsey Layton by the extent of the gains:

Consistent with prior research, KIPP middle schools have positive, statistically significant, and educationally meaningful impacts on student achievement in math, reading, science, and social studies. Average impacts of middle schools were positive and statistically significant throughout the 10-year period covered by the study, though higher in earlier years than recent years (from Key Findings).

Not surprisingly to anyone who has either worked in the highest performing KIPPs or who has studied test results from these schools, students from the 10 over-subscribed middle schools with lotteries did better than the other middle schools in the study.  

This finding lends credibility to former KIPP teachers who speak of the advantages (parental self-selection and KIPP skimming) of working in a lottery school, as compared to under-subscribed KIPP schools, where teachers have to pound the pavement to recruit wary parents and students from the surrounding community.

If we look closer inside the report, however, we see that the key findings gloss above needs some serious qualifications.  For instance, 23 percent of the middle schools in the Mathematica study had either no statistically-significant impact (8 schools) or negative statistically-significant impact (2 schools) on student achievement:

We also examined the distribution of school-level impacts for our matched-student sample of 37 schools (Figure III.2). Each point plots one KIPP middle school’s math and reading impact estimates two years after students enroll, based on all years covered by our data. Overall, 18 schools have positive and statistically significant impacts in both math and reading and nine schools have a positive and statistically significant impact in either math or reading (but not both). Eight schools have no statistically significant impacts, one school has a negative and statistically significant impact in only one subject, and one school has statistically significant negative impacts in both math and reading (p. 28).

Elementary Schools

We find the same misleading kind of information in Mathematica’s summary of findings for KIPP elementary schools:

KIPP elementary schools have positive, statistically significant, and educationally meaningful impacts on three of four measures of reading and mathematics skills (Key Findings).

Sounds impressive, yes?  When we look inside the report, however, we find only three of the eight schools had significantly positive impacts on student test scores:

The impact of KIPP at the elementary school level varies by school. Among the eight KIPP elementary schools we examine, three have positive and statistically significant impacts on at least one measure of academic achievement (Figure III.1). Two additional schools have marginally significant positive impacts on one outcome (p < 0.10). There are no statistically significant impacts in the other three schools. In these schools, the point estimates for one school are positive and the estimates for two schools are negative in both subjects (p. 25). 

Mathematica's examination of KIPP elementary schools involved only oversubscribed schools with lotteries (8 schools).  By doing so, Mathematica ignored the effects of self-selection by motivated parents, which makes later test results suspect and most likely higher than other KIPP schools, where  teachers had to go out recruiting parents and students from their communities.

With the knowledge, too, among KIPP leaders that they have an oversupply of applicants from which to restock, the oversubscribed KIPP schools have the luxury of “weeding out” low performers, English language learners, special needs children, and other problem students whose lack of performance could damage the KIPP brand.

High Schools
As for high schools, the news for KIPP gets worse, especially for new KIPP high schools.  In locales where KIPP middle school graduates’ parents have a choice to continue to send their children to a KIPP high school, 70 percent enroll their 9^th graders.  I think KIPP would rank that as a C- in retention.  Keep in mind, too, that over half of KIPP middle schoolers who start KIPP in 5^th grade never finish 8^th grade at KIPP.  

We are left to wonder how many KIPP middle school grads would continue to a KIPP high school if they could make that call, rather than their parents. 

We find out, too, from the Mathematica study that KIPP middle school students who do continue on to a KIPP high school should not expect KIPP to have any impact on their achievement, when compared to students attending regular public high schools:

For students continuing from KIPP middle schools, the achievement impacts of KIPP high schools are not statistically significant on average, but these impacts vary by school.

Other findings show that KIPP has a positive testing outcome for high school students who are new to KIPP, even though new students make up only 32 percent of KIPP high school students: 

KIPP high schools have positive, statistically significant, and educationally meaningful impacts on achievement for new entrants to the network (p. xiv).

And yet behind this summary, we find further caveats that are ignored by Mathematica's summary and by the news media.  For instance, 5 of the high schools 12 schools with both math and ELA scores available had either no significant impact or negative impact on new students in KIPP high schools:

As is the case with both KIPP elementary and middle schools, the impact estimates for new entrants vary across the different KIPP high schools in our sample (Figure III.8). Of the 12 schools with both math and ELA impact estimates, two high schools have positive and statistically significant impacts on new entrants in both math and ELA, five schools have a positive and significant impact on new entrants in one subject but not the other, four schools have no significant impacts on new entrants in either subject, and one school has a negative and significant impact on new entrants in one subject (p. 37).

No "Character" Gains
Despite the KIPP's founders insistence that KIPP is focused 49 percent on academics and 51 percent on character, there is no evidence that KIPP's chain gang methods, total compliance, and "No Excuses" brutality does anything to improve any aspect of student behavior, motivation, aspirations, confidence, grit, effort, self-control, or engagement:  

Across grade levels, we generally find no impacts of KIPP schools on measures of students’ motivation, engagement, educational aspirations, or behavior, but positive impacts on the satisfaction of parents with their child’s school. 

At all three grade levels, KIPP did not significantly affect measures of motivation and engagement related to student self-control, academic motivation, academic confidence, grit, school engagement, or effort in school, including student reports of the time spent on homework. Student behavior was measured only at the elementary and middle school levels; we find no evidence that KIPP schools affect behavior, including indices of positive behaviors, undesirable behaviors, peer pressure, illegal activities, parental concerns about their child, frequency of school disciplinary actions (according to the parent), and the extent to which the child is well-adjusted (pp. xxii)

Finally, Mathematica fails to address KIPP skeptics who focus on the selective admissions at KIPP.  In so doing, the research firm opens up questions as to the objectivity of Mathematica’s longitudinal research that KIPP’s benefactors at Atlantic Philanthropies commissioned in 2008 at a cost $4,000,000.  Here is Mathematica's weak attempt at silencing the skeptics:

Skeptics of KIPP argue that these schools rely on selective admission…. However, data on student characteristics provide little evidence that KIPP “creams” or selectively enrolls higher-performing students at the middle school level (Tuttle et al. 2013). The typical KIPP student scored at the 45th percentile within the district—that is, below the district average—in reading and math prior to entering KIPP (pp. 1-2).

The fact that KIPP beginning KIPP students score below the district average tells us nothing about characteristics of KIPP students in comparison with other school children in the same neighborhood.  Large districts like Memphis, for instance, contain many schools in the leafy suburbs and the county countryside, where middle class students attend school.  

If Mathematica had wanted to really find out if the KIPP students in Memphis had self-selecting parents with higher-scoring children, as skeptics insist, they could have compared KIPP student averages with the students in poverty-riddled public schools in and around downtown Memphis. But as the old saying goes, I guess you get what you pay for.

Jim Horn's new book, Work Hard, Be Hard: Journeys Through "No Excuses" Teaching will be published in 2016.