I revised my earlier analysis of RIDE and Brown's Achievement First fiscal impact memo as a post for RI Future:
Even if one accepts all of RIDE and RIIPL’s method and rationale for its fiscal analysis, the bottom line is that that all their calculations are derived from a single, early, clearly idiosyncratic study that overstates the positive achievement effects of AF compared to any other study I have subsequently found.
I've been sneaking peeks at the memo written for RIDE concerning the fiscal impact of the proposed expansion of the Achievement First network in Providence by 2,200 students while wrapping up Fall Forum and getting the next issue of Common Ground together. The memo was prepared by the Rhode Island Innovative Policy Lab (RIIPL) at Brown, which was started with a $4.3 million dollar grant from the Arnold Foundation, so any ed policy nerd can easily guess in what direction this story is going. The memo (that's what they call it, and that's what it is, not a study) is by Noah Kessler, Margarita Machelett, Miraj Shah, Jun Shepard, Justine Hastings, and included in the packet presented to the state Board of Education by the Commissioner last week.
Thanks to the ProJo for presenting some critical responses to the memo today. I was already working on an analysis of the calculations in the memo, and if this issue is hot today, I should probably get a quick (ha!) working version out ASAP. Thus this post. Consider it a draft.
The RIIPL memo is rather obviously a teetering tower of conjecture held up with a few high tension strands of research. While we can and should attack the foundation of this kind of thinking -- that the value of primary and secondary education can be measured in monetary gain or even college attendance -- as well as some of the basic suppositions -- do we really think that it is possible that every teacher in a system of seven schools will perform at one standard deviation above the mean every year? -- what really drives me around the bend is the mistakes and fudges that inevitably turn up in this genre of work with a little digging.
The basic chain of reasoning in the memo is: look at the performance of Achievement First elsewhere, extrapolate those gains over 13 years, run the performance numbers through Chetty, Friedman, & Rockoff's (2014) (CFR) findings to make predictions about future income and college completion, and finally compare that to other alternatives.
Or, as I like to picture it:
Test score growth -> The Chetty-izer -> predicted income growth.
The key piece of evidence here is really Justine S. Hastings (also an author of the AF memo), Christopher A. Neilson and Seth D. Zimmerman's 2012 paper on The Effect of School Choice on Intrinsic Motivation and Academic Outcomes. This is a study of choice in New Haven, with Achievement First's five flagship schools (and only these schools, apparently) representing charter school performance. This is a huge issue in itself, as the flagship schools aren't necessarily representative of the network as a whole across several states. In the district, students are apparently guaranteed a spot in a their neighborhood school (unlike Providence), with the option to apply for a large number of magnet schools:
Currently, half of the schools in the district (20) are magnet schools, and over a third of the students in the district (7,000) apply to them each year through a centralized lottery system. Of nine district high schools, eight are magnet schools open to choice, implying that almost all high school students either participate in school choice themselves or attend school with students who did.
In Providence, we don't really have magnets, other than Classical, but you can choose any district school through a lottery system. I would argue that in Providence's context any over-enrolled district school that you have to win a lottery to attend (Vartan Gregorian, Reservoir Elementary, etc... if anyone has a list, send me a copy) is functionally equivalent to a magnet school as defined in New Haven (remember, it is half the district there, not super exclusive). This is important because it affects the district baseline performance.
OK, let's step back to Chetty, Friedman, & Rockoff for a moment. This has become a very influential report because after some genuinely impressive number-crunching, they were able to show a correlation between, among other things, test scores and future income. Let's just say not everyone is a fan, and it is just one study. Nonetheless, this one of the key threads holding up the RIIPL memo, so it had better be used precisely. So, what data did CFR use in their original study?
We address these two questions using information from two administrative databases. The first is a dataset on test scores and classroom and teacher assignments in grades 3-8 from a large urban school district in the U.S. These data cover more than 2.5 million students and 18 million tests for math and English (reading) spanning 1989-2009. The second is selected data from United States tax records spanning 1996-2010. (page 3, emphasis mine)
CFR only looked at math and reading. We can only draw conclusions from CFR regarding math and reading. Keep that in mind...
The all important math and reading data that the RIIPL memo is going to plug into the Chetty-izer to get future income data comes from the New Haven paper:
We use estimates of AF value-added impact from Hastings, Neilson, and Zimmerman (2012) (HNZ). They find that test scores of children winning the lottery to attend Achievement First increase by 0.346 student level standard deviations.
Feed 0.346 into the Chetty-izer, assume every student at AF will see this effect every year K-12, and they get an increased income at age 28 for each former student of $10,954 - $13,451, extrapolated over a lifetime, that's $268,477 - $330,578.
OK, so, let's look at where this "0.346" number comes from in the New Haven paper. For this sort of thing I use the advanced research technique of searching for the "0.346" in the pdf. Here's where it comes from -- Table 7:
If you look under the third column, "Charter," the "Combined Z-Score" is 0.346. For an extra dollop of confusion, this is also the Z-Score for reading, but that is just a coincidence. Their combined Z-score is the mean of reading, writing and math. The problem is, again, that CFR didn't have writing scores (the word "writing" does not appear in the text at all), and thus we cannot make any CFR-based predictions using writing scores. We're feeding data into the Chetty-izer that it doesn't know how to process.
This is particularly important if you look more closely at the math and writing Z-scores the New Haven study found for AF. Math seems anomalously low and writing seems unreasonably high. It doesn't make any sense and my spidey statistical-noise sense is tingling.
The bottom line is that there is no justification for including writing scores in a CFR-based analysis, and without writing, the combined Z-score for AF goes from 0.346 to 0.127. As a quick recalculation, we can multiply all the subsequent estimates of income gain in the RIIPL by 0.367.
So... let's look back at those projected earnings. For our back of the envelope calculations here, if we assume the income increase goes down linearly with a reduced Z-score, instead of looking at $10,954 - $13,451 at age 28 for a student who had attended AF for 13 years, it would be more like $4,020 - $4,936. That is the difference between including or not including writing scores. Whichever number you believe, if either, there is no way those two numbers should be that far apart.
Another issue illustrated in table 7 is that in math and reading alone, according to Hastings, Neilson and Zimmerman, magnet schools outperform charters (AF). If we assume that over-enrolled Providence district schools are equivalent to New Haven's many magnets, the Chetty-izer would predict that students in the best Providence District Schools are getting a similar lifetime earnings benefit to their projections for Achievement First. This would need to be part of a serious analysis of impact. Not all children in Providence Public Schools are in the lowest performing schools.
There is an additional straightforward mistake in the calculation of the total cohort gains of an additional 2,200 students attending AF schools in Providence. They project the impact of 2,200 students attending AF from K-12, but since AF only proposes a single high school for the three K-8 schools, only 160 of each K-8 cohort of 276 students will get the full "treatment." By my calculations, to correct this their full cohort outcomes should be multiplied by 0.87.
So... my back of the envelope corrections would take the predicted increased lifetime earnings resulting from adding 2,200 seats to Achievement First Providence from RIIPL's $590.6 - $727.3 million down to $188.6 - $232.2 million.
One of the things which is particularly annoying about this economist-written memo about the financial return of a charter school is that they do not do the obvious thing of comparing their projected financial return to the student from this educational improvement to simply chucking money into an index fund. Let's say instead of having a lottery for additional seats at Achievement First, we had a lottery where the winners had $4,347 a year (the amount Providence has to send to AF with each child) deposited in an index fund with a 5% annual return. After 13 years, the student would have about $77,500. Let that sit for 10 years gathering interest (to $126,200), then at 28 start paying out a 40 year annuity of about $7,300. That's a total payout of almost $294,000 (well, if this calculator is accurate). That's nearly in the middle of RIIPL's predicted lifetime benefit of attending AF for 13 years, and basically double my corrected Chetty-ization.
The point is, in financial terms, this is not some kind of extraordinary payout. It is quite ordinary. All the numbers RIIPL throws around -- following the example of CFR's paper -- look quite impressive, but when you start talking about the full cost of educating a child, or a group of children, or cumulative earnings over a lifetime, all the numbers get bigger than you expect very quickly. Also, compound interest is a heck of a thing.
Finally, finally, Commissioner Ken Wagner wrapped up his ProJo op ed supporting the AF expansion by asking:
If one is opposed to the Achievement First proposal, what is the alternate plan for the children of Providence? And how is this plan different - truly different - from what we have already tried over the past 25 years?
I keep this link on auto-dial for occasions like this:
To the current reformers, integration is at best an irrelevance and at worst an excuse to shift attention away from shoddy teaching. But a spate of research says otherwise. The experience of an integrated education made all the difference in the lives of black children — and in the lives of their children as well. These economists’ studies consistently conclude that African-American students who attended integrated schools fared better academically than those left behind in segregated schools. They were more likely to graduate from high school and attend and graduate from college; and, the longer they spent attending integrated schools, the better they did. What’s more, the fear that white children would suffer, voiced by opponents of integration, proved groundless. Between 1970 and 1990, the black-white gap in educational attainment shrank — not because white youngsters did worse but because black youngsters did better.
Not only were they more successful in school, they were more successful in life as well. A 2011 study by the Berkeley public policy professor Rucker C. Johnson concludes that black youths who spent five years in desegregated schools have earned 25 percent more than those who never had that opportunity. Now in their 30s and 40s, they’re also healthier — the equivalent of being seven years younger.
According to the Chetty-izer an education in an integrated school for five years is equivalent to going to Achievement First for twenty-five years, and it is a lot cheaper. All we would need is the will to try something different - truly different - than we've been doing for the last 25 250 years.
For the past few months I've been working part time with Jill Davidson to put on the final Coalition of Essential Schools Fall Forum -- their annual national conference. It is also the end of CES as an independent organization, more or less. It was Jill's conference; I was just assisting her vision. There are a bunch of reasons to gracefully wind up the organization at this point: structurally it has always been too loose and decentralized for the current funding climate, and the expanded reach of many of its principles has come through new organizations like Expeditionary Learning, Big Picture Learning, High Tech High, and Deeper Learning.
So we held the conference last week in Providence at the Omni to give everyone a chance for one last "conversation among friends." I suppose it is not my place to say so, but it went really well. We had between 400 and 500 people, which was enough for our financial requirements and enough to make the venue feel full -- but still small enough for Jill and I and a few other volunteers to give everyone quick and personal service. I could easily keep an eye on the 15 or so session rooms myself. The Omni staff was great; food was tasty; A/V support way beyond what teachers are used to.
The goal was to still have the same working conference for classroom teachers from Coalition schools that Fall Forum has always been, with a strong thread of reflection, nostalgia, and opportunity for closure as a community.
The emotional center of the conference -- at this point you might consider how many conferences you've attended that had an emotional center -- was its grand-matriarchs, Deborah Meier and Nancy Faust Sizer. Debbie's eyesight is failing so I (and I am sure many others) made it my mission to scurry around making her path as smooth as possible. Debbie and Nancy were in the middle of everything all weekend. Nancy Sizer's closing words were emotionally raw but perfect for the moment, and wrapping everything up with some second-line tunes from the Extraordinary Rendition Band worked better than I could have hoped.
From my point of view, even the screw-ups were kind of amazing. I'd sent out invitations for participating authors to do book signings, but the actual implementation was a bit ad hoc. I ducked into the main ballroom to finally get a bite to eat on the first full day of the conference and when I came out, Linda Darling-Hammond, Dennis Littky (of Big Picture), Debbie Meier, and George Wood (president of CES) were all sitting around a little table wedged in between the bookseller and another vendor table, having a grand chat. The only problem is that without a proper sign, etc., even if other people had realized it was supposed to be a book signing, it was a rather intimidating conversation to interrupt for a signature. It was a satisfying moment to stand back and observe, and maybe grab a young person and whisper, "Do you know who those people are!?!".
CES is kind of like an American labor union -- full participation is mostly based on where you happen to work -- and since I've never worked at a CES school, I've never been an active member. But CES and Ted Sizer's work at Brown is what brought me to Providence in the first place, and I have always thought of my educational philosophy first and foremost as a "CES-style progressive," so it was an honor to contribute to its legacy and do something for those who built it over the past three decades.
Erika Sanzi has a post on her personal blog which has also been picked up by Citizen Ed and Fordham celebrating the addition of a middle school gifted program in our neighborhood middle school. Of course, from Sanzi's point of view, this is a foreign, benighted realm into which she rarely, if ever sets foot, except perhaps while slumming for good Cambodian food or visiting a charter school, so the whole thing is a bit annoying, and I left some comments!
It is nice that in a couple years, my child may be able to walk to Roger Williams instead of getting on a bus to Greene or Bishop to get a more challenging middle school experience, but it is not as if those programs exclude students from south Providence. This is similar to the error in your post about Rhode Island’s PARCC scores that forgot that Classical High School has 63% of its students eligible for free and reduced lunch and is majority minority. Clearly low-income and minority students have been challenged and succeeded in Providence Public Schools for a long time.
It is also not accurate to assume that this side of town has not had other innovative, challenging programs in its public schools. They just usually haven’t been called “gifted” programs and had selective enrollment, and they haven’t had much political or public relations muscle. The advent of No Child Left Behind wiped out a whole swath of programs — for example Fortes Elementary was known nationally for both its very early laptop initiative and their in-school local history museum, until NCLB hit, and the same school that was a model yesterday was “failing” today, and promptly gutted.
Of course, middle schools are and pretty much always have been a huge issue all over the city, so creating an additional magnetic island is probably not a bad thing, even if it doesn’t help the rest of the middle school problem much.
Finally I would question what seems to be an implicit assumption that people from other parts of the city won’t be sending their kids down here to the Roger Williams advanced academics program if it becomes successful. It is right off 95!
And...
Also, can we mention the demographics of Nathanael Greene Middle School, the school which has hosted the district middle school gifted program for decades? It has 72% of students eligible for free or reduced lunch and is 92% non-white. The elementary school in that neighborhood — which is NOT the East Side — is 93% free or reduced lunch and 91% non-white. So the entire premise that bringing a gifted program to a low income neighborhood is new for Providence is false.
The fact of the matter is that adding a new, unproven "advanced academics" program to the neighborhood middle school which doesn't exactly have a great track record presents a real dilemma. It would certainly be convenient, and we would prefer to support such an effort through the participation of our children, but at this point who knows if it is going to take, and by the time you get to middle school, the cultural isolation of a neighborhood like this becomes a bit more of a concern. If students in the new Roger Williams program end up being isolated from the rest of the gifted streams of students from the rest of the city, that's a serious risk.
How, exactly, does creating a goal in any way help you accomplish it? What is the point other than to establish an artificial sense of accomplishment by achieving some arbitrarily defined thing?
Apparently when I think that way it is my working class side showing.
Ian Welsh:
I have little patience for all the Brits who are wringing their hands about Labour and parking their votes in the Conservative party. This is a good, non-radical plan that will work. It is a plan of a government that wants to be good to the poor and the young. Corbyn is entirely credible on all of it since he stuck by these principles all thru the Thatcher and Blairite years.
If you’re planning to vote Conservative in the UK, when this is on offer, you’re just an asshole, a “I”ve got mine, fuck you Jack”, or someone who has bought so far into neo-liberal ideology that your political actions make you indistinguishable from an asshole, whether or not you think they “work”. (Especially as all the evidence is that the only work for a minority, presumably a minority which you belong to.)
Brits have something which most of the rest of us don’t in most of the Western world: the opportunity to vote for a government which is not the lesser evil, but which is actually good. If they blow it, as far as I’m concerned, the majority blame will be on Brits, not on Corbyn. This is a character test: do enough Brits still want government which tries to take care of everyone or not?
Remember, the Conservative government, among other policies, cut a program which gave disabled people things like wheelchairs. That resulted, literally, wheelchairs being taken away from cripples. That’s what you’re voting for if you vote Conservative, and yes, you should be judged on that.
And that’s what politicians want! They want kids out of their rooms, away from screens, out there doing shit – basically what skateboarding’s all about.
Also:
This is why architects always photoshop some fucking skateboarder in the 3D plans when they propose them.
And...
Yeah. This for instance is the headquarters of one of the largest banks in northern Europe. I called them up, told them “there’s going to be an event, we’re going to have some music, some drinking and it’s a skateboard event” – “oh we love the skaters, they hang out here all the time, yeah of course”.
