Charter critics in Mississippi (and around the country) like to say that KIPP only has higher achievement than traditional public schools because it is either selective in who it attracts or it systematically gets rid of low-achieving kids or kids with discipline problems. Recently, as charter school legislation has come to a crossroads at the Capitol, I’ve heard these arguments intensify. One likely reason is that the KIPP school in Helena, AR, has become THE example of charter schools everywhere for Mississippi legislators, and critics believe that if they succeed in disparaging that one school or KIPP in general, they will win the argument about charter schools as a whole.
But whether the high achievement of kids at KIPP in Helena, AR, or kids at KIPP as a whole can be explained away by any reason is not a question of who talks loudest. It is a question that has a true or false answer.
If there is interest, I can write other posts about whether KIPP and charter schools as a whole “skim” students. In this series of posts, I am going to focus on attrition, which means the rate at which KIPP students leave KIPP before the school’s highest grade (for example, a KIPP middle school’s highest grade is 8th grade). The question I am going to try to answer is, “Does Attrition at KIPP Explain KIPP Achievement?” Chances are that I might not be able to answer this question if there is not sufficient evidence either way. However, I will at least give us all a better understanding of how close the research is to answering that question. [NOTE: Deputy Director Sanford Johnson worked at the KIPP school in Helena and sat on its board. His wife currently runs the KIPP elementary there. Since he has some personal interest in KIPP, he is not going to be helping me review these studies.]
Rules for Reviewing Research
I want to use this first post to define a few rules for how I will go about using the research to answer this questions:
1) I will only consider the findings of published research. A lot of the ink spilled on this issue in popular media is not actually based on fact. Instead, several sources have printed or re-printed the speculation or personal opinions, without reference to research, of advocates or journalists on both sides of the issue. Even when some of these sources claim to be referencing a study, I will not take their word for it. I will read the original research, report to you my own analysis, and post the original study here for you to do the same.
2) I will weigh the findings of published research in relation to the strength of their methodology. Not all research is equal. No research is perfect, but research can be better or worse. First, there are questions about whether the researcher did enough things right to come to a good conclusion about the data they actually collected. These are concerns about internal validity. A lot of school research has problems with internal validity. Researchers might not adequately control for differences across schools or students or they might have a bad way of measuring achievement, drop out rates, etc. The second concern about research is whether it has external validity–whether the results of a single study focusing on one set of data tells us anything about what the study would have found if the researcher had had access to all the data. For example, if a study with good internal validity that uses data from a single school finds a good result, does that result apply to every school? It may or it may not. As I review the research, I will point out the strengths and weaknesses of the design of the studies in addition to their findings.
3) I will not “cherry-pick” studies. More than anything else, cherry-picking is the bread and butter for advocates on either side of any issue. Avoiding cherry-picking means looking for every published study about KIPP and combing it for references to attrition. I will also take every suggestion from readers as long as it meets the first rule that it is published research (and I will report what the suggestions are and my reasons for reviewing or not reviewing the work).
4) I will take suggestions for other rules. If you think I should include another rule, please email me.
A lot of people make a lot of assumptions when they are arguing about attrition. Several of these assumptions are based on people’s speculations for WHY a child left a school. I am going to warn you now that the research probably won’t tell us WHY individual kids have left a school; instead, we’re going to be looking at the demographic characteristics of students. Demographic data is enough to answer the question of whether attrition explains achievement, but it can’t tell us what causes the patterns. I hope that researchers will start looking at why kids leave any school, whether KIPP, another charter, or a traditional school; please let me know if you know of any research like this.
1) High attrition automatically means that KIPP scores are inflated. What we are really looking for is not how may kids leave but whether kids at one end of the achievement spectrum are leaving at higher rates than other kids and whether those kids are being replaced by kids with different achievement. For example, we should be really concerned if low-achieving kids are systematically leaving and are being replaced by high-achieving kids. If we find instead that high-achieving kids are more likely to leave and be replaced by average or low-achieving kids, then attrition actually works against KIPP achievement. Another scenario might be that kids along the spectrum are leaving at the same rates but that they are being replaced by only high-achieving kids or only low-achieving kids.
2) All attrition is caused by a school’s actions. Kids leave a school for all kinds of reasons. When people think about attrition, they often imagine a child who has left a school because of some action the school has taken (like expulsion). However, a child might leave a school for reasons that have nothing to do with how the school has treated the child: housing insecurity, changes to custody arrangements, illness or death, the child’s social life, where a child’s siblings attend school, or the extra-curricular activities available at the school are all reasons a child might leave a school. High-poverty children tend to be switch schools more often than other children because their parents move around more looking for work or a more affordable place to live. Unless researchers refine the definition of “attrition” to include only those children who leave because of actions taken by the school, attrition rates will include children who leave for reasons not based on school actions.
3) Attrition should be the same for charter school students and comparable traditional public school students. This assumption is a really more of a philosophical one than one of fact v. fiction. Charter schools are schools of choice, which automatically means that people who choose charter schools could choose something different. When choice exists, some people will try something and decide they don’t like it. When choice doesn’t exist, people have to keep what they have even when they don’t like it. Therefore, charter schools may naturally have a higher rate of attrition caused by dislike or dissatisfaction than traditional schools. But how much higher is acceptable if we are only looking at attrition caused by “fit” rather than other mobility? 0%, 5%, 30%? Does it matter to us if it is a parent’s freely made choice that is not skewed by the school trying to influence the decision? This is something that neither I nor the research can answer and so we will leave this question for another day.
Please let me know your thoughts about this post. I plan to start reviewing these studies this weekend. Happy Friday!