The Truth About Mississippi’s NAEP Gains

July 5, 2023July 6, 2023 Company Updates

Editor’s Note: This blog post is in response to a column appearing in the LA Times titled “How Mississippi gamed its national reading test scores to produce ‘miracle gains.‘” We’re not going to link to it because we don’t believe it is worth your clicks; however, feel free to google.

***

This morning, I was contacted by a long-time national educator reporter for my thoughts about a column that appeared in the LA Times purporting to “debunk” Mississippi’s NAEP gains. What appears below is a slightly edited version of my response. It is based on what I’ve been telling reporters and other education colleagues for over three years and features some very ugly hand-written notes I took on NAEP print-outs in December 2019. I was also interviewed by Bellwether about this topic in 2020 and gave a shortened version of this explanation; you can read that response here. I talk about NAEP scale points in this post, but you should note that not all scale points are equal due to the rules of statistical significance, so you should really look at the notes that show that. The NAEP website also has multiple tools to look at whether a scale point change between two given years is statistically significant, and I would encourage you to look at that as well.

Mississippi’s Actual NAEP Scores

Let’s start this post by exploring Mississippi’s actual NAEP data and how it has changed in the last two decades. This is important because the story of Mississippi’s progress does not begin in 2019, 2015, or even 2013–we’ve actually been on a 20-year journey to “miracle” status.

In 4th grade reading, Mississippi gained 20 scale points between 1992, when the first state NAEP data were released, and 2019, when we first reached the national average. The majority of those gains happened between 2005 and 2019, a time in which the nation was mostly stagnant in its 4th grade reading scores (though it did improve slightly). We had two periods of big gains: 2005-2009 and 2013-2019 (over half of the gains in this one are between 2013-2017). See the ugly print-outs below that you can reproduce on NAEP’s website.

In math, we gained 18 points from 2003 to 2019 (ed. note: there’s a weird 12-point jump between 2000 and 2003, both in Mississippi and nationally, that a NAEP researcher could probably better explain and which this post is not focused on). The math gains were very steady in this 16-year period, with Mississippi improving little by little while the nation stood still until we finally saw Mississippi’s gap-closing jump between 2017 and 2019.

When Did the Literacy-Based Promotion Act Affect 4th Graders?

The Literacy-Based Promotion Act (LBPA) did not pass until spring 2013 and the “gate” (i.e., the requirement that students score a minimum level of proficiency, which originally was a level 2 of 5) did not go into effect for 3rd graders until 2015. This means the first year that kids who experienced the “gate” were in the NAEP sample was 2017, when 15 points of our 20-point gain had already happened. In math, 11 points of the 18-point gain happened between 2003-2015 in slow, steady increments as I note above. What this means is that for good or for ill, the LBPA can only explain 4th grade gains beginning in 2017.

For further context, we should also look at 8th grade data in this same period. Both reading and math improved at 8th grade over the last 20 years despite the national data being mostly flat. Math, again, showed really steady improvement that is actually better than reading, though reading also improved overall and relative to the national numbers. None of this can be explained by the LBPA. The first cohort of 3rd graders affected by the gate would have been 8th graders in 2020(!).

What is the “Bottom 10%” Argument, and Why Is It Unpersuasive?

The LA Times column claims that Mississippi’s NAEP success–specifically our reaching the national average in 4th grade reading–is a sham based on the analysis of a blogger armed with a graph of NAEP 4th grade reading data between 2013 and 2022 and the claim that Mississippi had a “nearly 10%” retention rate in 3rd grade following the LBPA. If you go to the blogger’s analysis, you find that he “added back in” the would-be scores of would-be 4th graders who, but for the LBPA it seems, would have been in the NAEP sample. It’s a little fuzzy exactly how he does this, as his graph only shows a line for the results of “adding back [the] bottom 10%” and said that the result lowers the score average enough that Mississippi had no “real” gains at all between 2013 and 2022.

If you read the above section about what Mississippi’s NAEP data looks like and when the LBPA was in effect, you will immediately notice that the first problem is that the timelines do not align. At best, if the 10% argument has any merit, it only explains the reading gains between 2015 and 2019 (the pandemic makes everything weird for the 2022 data, and since we saw no statistical change from 2019, I’m going to focus on everything up to 2019). The reading gains between 2015 and 2019 were certainly important and statistically significant, but they were not the majority of the improvement, nor were they unusual for the pattern of improvement we have seen in reading over the 20-year period. Also, we had a single point-change (not statistically significant!) between 2015 and 2017, when the first “10%” would have allegedly been disappeared from the sample–why not, if that theory holds? The best argument for the 10% theory is actually the 2017-2019 jump in 4th grade math since, previously, those gains had been so slow… but you have to make a lot more assumptions about reading on the math test/retained students being *even worse* at math (but not those not promoted after 2018 since there was, again, only a single-point, not statistically significant increase in math between 2015 and 2017). If this is beginning to seem a little suspect, well…it just gets more suspect!

I do not know where these esteemed gentlemen got the 10% figure for every year in this period (did they round up from the 2022 overall rate and just assume the rate has been constant?), but the percentage of 3rd graders held back as a result of the LBPA in whole or in part has never been that high. Certainly, the LBPA caused an increase of between 5.74-5.91 percentage points in the retention rate over a base trend of around 3-3.4% in the years immediately prior to the 2014-2015 implementation, but after that first year, the retention rate began to drop back down to pre-LBPA levels. By 2017-2018, the retention rate as a result of the LBPA was no higher than 1.58% and the overall rate was less than 5%.¹ As a result, of the 4th graders taking NAEP in 2019, the person with a blog would only “add back in” 1.58% (or at most “less than 5%,” according to MDE) using the undisclosed “add back” methodology of said blogger.

Because the LBPA caused so few retentions by 2018, Mississippi actually raised the bar in 2018-2019 so that “passing” the gate meant a higher level of proficiency in reading (now a 3 of 5, instead of a 2 of 5). After that, the overall retention rate did reach a high of 9.6% for the 2019 3rd grade cohort (2020’s 4th grade cohort), but those kids weren’t in the 2019 or 2022 4th grade NAEP. The kids in the 2022 4th grade NAEP sample actually did not face the gate, as Mississippi tested in 2021 when they were 3rd graders but those tests did not count for accountability purposes.

I want to pause here to say that I object to this whole construct of the bottom 10% methodology because retained students don’t just disappear such that you need to “add them back in.” They actually eventually get promoted, which means they do show up in 4th grade data, including 4th grade NAEP, just after–hopefully–some remediation. And having better scores after being held back…is that not the point of grade retention?

So Is the Miracle Real or Not? (Spoiler: The Gains Are Real.)

If the mis-NAEPery of the blogger/LA Times guys doesn’t explain anything, what was happening in Mississippi during this time to explain this “miracle” of kids from “[insert insulting adjective from the LA Times piece] Mississippi”? Our people were not getting a lot richer nor did Mississippi outspend other states, and as Mr. LA Times notes, we do not have very generous or progressive social policies. (I would welcome an analysis from a poverty researcher on economic conditions here since 2003 vis a vis the nation.) 

In my humble opinion as a person who might just be an expert on education policy in Mississippi, there’s actually a much less sexy explanation than either a single-policy “miracle” or a sham but one which has the most coherence considering all the information: No Child Left Behind, the nationally aligned Mississippi College and Career Readiness Standards, and a comprehensive approach to improving education since 2012. I believe there is at least one study about the fact that NCLB led to achievement gains nationally, including in Mississippi. We’ll surely see similar studies about the second wave of the standards movement beginning in 2008 at some point in the future. My annotated notes above mark when each of these went into effect as well as other important ed policy changes. Long story short, I think there are very good reasons to believe that the long, upward trajectory of Mississippi in reading and math at BOTH 4th and 8th grade is due to a long-term commitment to increasing standards and aligning testing and accountability systems. The reading work that has become such a sexy national story is great! phenomenal! ground-breaking! and also part of a raft of policies that changed in this time and likely contributed to success. Does that mean that it is not work other states should try to emulate? No, I think it absolutely should be studied and replicated; however, as I’ve repeatedly told anyone who would listen in the last 3 years, you have to look at everything we did in context. The gains are real, and I would welcome any credible researcher do a serious, careful study of our NAEP data and our context to try to explain them better than I have here.

One final note: I don’t generally have the time (or desire) to write these kinds of pieces because I have other, more important things to do for children in Mississippi than entertain half-baked analysis and hit pieces. Mississippi’s education improvements over the last two decades are a bright spot for our state. For those who would discredit them without doing adequate homework, maybe take some time to examine your biases, namely why it is just so impossible for you to believe that kids in Mississippi public schools, over half of whom are kids of color, have reached the national average after literal years of hard work from people who care about them. For everyone else, stay safe out there.

¹ Grace Breazeale on our team pulled the LBPA passage reports for every year of implementation from MDE’s website. These reports show the number of students who passed after the initial test and also after the re-test(s) as well as all the students who qualified for promotion anyway under a good cause exemption. MDE further provides official retention rate data for every grade K-8. She also pulled retention data from prior years from the US Department of Education’s OCR data set as well as MDE’s official enrollment by grade in those years. There is some nuance with these data, so if anyone has questions, we can answer them. You can see our calculations here.