Some details on measuring charter performance

Our last post discussed the findings of our recent report, which used a regression model to show that, after controlling for school characteristics, Twin Cities elementary charter schools still lag behind traditional schools in math and reading performance.

Since then, at least one commentator has questioned our regression model, noting that the inclusion of variables for "hours per day" and "days per year" may distort the results, by controlling for the very techniques charters use to produce academic gains.

This is a valid concern. The exact specification of any school performance model is important, and improper inclusion of policy variables into a regression can appear to eliminate important and relevant distinctions between charters and traditional schools. As such, we'd like to take the opportunity to both explain our reasoning, and present differently-specified models that, hopefully, demonstrate the robustness of our findings.

There are three reasons we opted to include instructional time variables in our model:

• First, both measures of instructional time vary significantly in charters and traditional schools alike (though both variables have higher means and significantly greater variance in charters, as can be seen below). The ability to alter instructional time is not an inherent property of charters that would be otherwise captured by the charter dummy variable; if it were, the case for excluding these variables would be substantially stronger. 

• Second, the regression in the most recent report was intended to update our previous work on factors impacting school performance, and those earlier models controlled for instructional time. Showing charters' impact over a span of years is important, because advocates frequently assert that competition in the industry will result in innovation and thus gradually improve academic performance. We cannot test this assertion if we alter our model over time. (For similar reasons, we omitted new data about the percentage of homeless students in a school. The state only recently started collecting this data and thus it was not included in our earlier regressions.) 

• More broadly, we make no claim that all of the independent variables in our model are outside of the control of the schools themselves, as it is difficult for school-level analysis to only incorporate external, "involuntary" factors. Even the archetypal independent variable for a school-level regression - racial and economic demographics - are to some extent the product of school choices and policies, especially in charters. (Indeed, one of the conclusions of our report is that racial and economic segregation are best understood as the consequence of school- or district-level policy decisions, and that we should be wary of applauding heavily-segregated charters for performing well compared to other segregated schools - because there is no rule saying that charters must be heavily segregated.) Instead, our model measures for the effect of innovation, cultural competency, academic focus, and other hard-to-see "X factors" that Minnesota charter advocates typically argue will produce heightened performance.

We certainly do not contest that instructional time, especially the length of the school day, has a notable impact on elementary test score performance. In our regression, in both math and reading, the minutes per day variable has a positive coefficient and is statistically significant beyond the 99 percent confidence interval. Extending the school day appears to be a reliable means of improving academic outcomes, though it should be noted that longer days are not cost-free and thus this practice is resource-limited.

With all that said, we recognize that other researchers might have preferred to omit instructional time from the model. As such, we've re-run our 2014-2015 regression without the instructional time variables. This produces somewhat smaller charter coefficients - but ones that are still negative and statistically significant. You can click to enlarge the full results below.

Concerns were also raised about the inclusion of attendance rate and mobility in the regression, pointing out that there could be a degree of endogeneity associated with these variables. In other words, lower test scores could themselves result in reduced daily attendance and more transfers, rather than the reverse.

Although certainly plausible, we would assert that both attendance and mobility are primarily determined by exogenous factors, such as poverty, familial stability, or housing security. Attendance rate in particular is also likely associated with parental educational participation, and could therefore help control for the selection bias issue, where charter parents are more likely to be involved in a child's education than traditional school parents.

But to fully allay concerns about these variables, we have produced a stripped-down regression that includes only basic demographic student information, plus school size and type. (We have also taken the opportunity to include homelessness in the model, although its impact on school-level performance appears to be small.) Once again, the results show smaller, but still negative and significant, coefficients on the charter dummy.

Ultimately, while there are many ways to specify this model, the top-line findings remain about the same in all of them: in the aggregate, traditional schools have a small but statistically significant edge on charters in academic performance. If there is a case for expanding charters, it must be found elsewhere.

Charter academic performance in Minnesota

The Minnesota School Choice Project is a new initiative aimed at providing a clearer picture of Minnesota's charter schools. Find out more here.

The first question anyone asks about charter schools is how they stack up against traditional schools academically. Charters, which have to recruit their students, often sell themselves to parents and communities as beacons of academic opportunity, and much of the political momentum behind charters is rooted in the impression that they can produce pass rates and test scores far above and beyond those seen in traditional district schools.

But in Minnesota, these claims are false, at least on a sector-wide basis.

Previous work suggests that Minnesota traditional schools outperform charter schools in the aggregate, and our latest report shows that this is still the case.

The report uses a regression model to compare pass rates in math and reading between charter and traditional schools. The findings across a number of years can be seen in the table below, in which negative numbers represent lower scores for charters:

Not only are pass rates lower in charters, the gap in scores is relatively steady for both subjects across the entire timespan.

Charter school advocates have responded to these findings by asserting that this isn't really a fair comparison, because students at charter schools are different than students elsewhere. For instance, in response to coverage of our report in the Pioneer Press, long-time charter supporter Joe Nathan responded that "these comparisons fail to account for the types of students charters often serve, including those who were unsuccessful in traditional schools."

This criticism makes no sense in context. The use of a regression model with student demographics controls for all key student characteristics monitored by the state, such as race, income, or special education or English learner status. If charters tended to enroll more students with a lower propensity for academic achievement, the model would show a smaller gap between charters and traditionals, not a larger one. (And as future posts will demonstrate, many Minnesota charters are considerably less likely to enroll students with special academic needs.)

Now, there are also differences between students that don't show up in state data and thus can't be included in the regression model. For instance, some families are simply more academically engaged than others.

But if anything, omitting those differences is likely to help charters, in the aggregate. That's because families whose children attend a charter school have to seek out the school and enroll their child. This creates selection bias: because a baseline level of educational participation is necessary to enroll at a charter, you would expect charter families to be, on the whole, more academically engaged than non-charter families. While measuring the full impact of this selection bias on Minnesota score gaps is all but impossible, it is likely a substantial consideration.

In other words, if you could fully "account for the types of students charters serve," there are very compelling reasons to think the gap between charter and traditional school performance would be larger and even less favorable to charters.

Introducing the Minnesota School Choice Project

In 1991, Minnesota passed the nation’s first charter school law. Today, despite continual controversy over their academic impact and educational role, there were over two hundred charter schools in the state, with more opening every year.

It's also here in Minnesota that many of the ideas underlying modern charter schools were forged, by civic organizations, policy scholars, and politicians. As an early adopter, the state has often served as a testing ground for political tactics and policy measures related to educational reform. And the Minnesota charter sector is unusually fragmented, representing the range of forms charter schools can take. Our charters run the gamut from remedial institutions for children of color, to online-only schools, to suburban classical academies. Minnesota also continues to experiment with new types of charter regulation, such as by adopting unusual and controversial policies for school accreditation. 

The Institute on Metropolitan Opportunity has produced several previous studies of Minnesota charter schools, focusing on school segregation and performance. These studies reflected a broader national debate about whether charters improved test outcomes. However, as charter schools have expanded nationally, more than their academic accomplishments are coming under scrutiny. Today, the debate over charterization extends far beyond performance alone. Recent reporting and research has raised questions about who these schools serve, how they are funded, how they are regulated, and their role in education politics. 

To contribute to the answering of these wider questions, the Institute on Metropolitan Opportunity is launching a new research initiative. Called The Minnesota School Choice Project, it will take a look at the state’s charter school industry from a variety of perspectives. This research is intended to provide new insight into the effectiveness and impact of charter schools in the state, and serve as a launching ground for a more robust understanding of the role charter schools play in the educational ecosystem, both in Minnesota and nationwide. Project results will be released in six parts, each bringing quantitative and qualitative analysis to specific subjects related to charter schools. These are as follows: 

Part I: Segregation and Performance 

Part II: Special Education and Discipline 

Part III: Screening 

Part IV: Funding and Expenditures 

Part V: Charter Authorizers 

Part VI: The Future of Charter School Politics 

Additional data, news analysis, case studies, and continuing commentary will also be provided on the Institute’s new website and this blog.

The first part, which talks about how and why charters have acted as the leading edge of school segregation in the Twin Cities, is out today. We'll be pulling out its key points to discuss on this blog in the coming days, and you can read the whole thing here.