Some details on measuring charter performance

Our last post discussed the findings of our recent report, which used a regression model to show that, after controlling for school characteristics, Twin Cities elementary charter schools still lag behind traditional schools in math and reading performance.

Since then, at least one commentator has questioned our regression model, noting that the inclusion of variables for "hours per day" and "days per year" may distort the results, by controlling for the very techniques charters use to produce academic gains.

This is a valid concern. The exact specification of any school performance model is important, and improper inclusion of policy variables into a regression can appear to eliminate important and relevant distinctions between charters and traditional schools. As such, we'd like to take the opportunity to both explain our reasoning, and present differently-specified models that, hopefully, demonstrate the robustness of our findings.

There are three reasons we opted to include instructional time variables in our model:

• First, both measures of instructional time vary significantly in charters and traditional schools alike (though both variables have higher means and significantly greater variance in charters, as can be seen below). The ability to alter instructional time is not an inherent property of charters that would be otherwise captured by the charter dummy variable; if it were, the case for excluding these variables would be substantially stronger. 

• Second, the regression in the most recent report was intended to update our previous work on factors impacting school performance, and those earlier models controlled for instructional time. Showing charters' impact over a span of years is important, because advocates frequently assert that competition in the industry will result in innovation and thus gradually improve academic performance. We cannot test this assertion if we alter our model over time. (For similar reasons, we omitted new data about the percentage of homeless students in a school. The state only recently started collecting this data and thus it was not included in our earlier regressions.) 

• More broadly, we make no claim that all of the independent variables in our model are outside of the control of the schools themselves, as it is difficult for school-level analysis to only incorporate external, "involuntary" factors. Even the archetypal independent variable for a school-level regression - racial and economic demographics - are to some extent the product of school choices and policies, especially in charters. (Indeed, one of the conclusions of our report is that racial and economic segregation are best understood as the consequence of school- or district-level policy decisions, and that we should be wary of applauding heavily-segregated charters for performing well compared to other segregated schools - because there is no rule saying that charters must be heavily segregated.) Instead, our model measures for the effect of innovation, cultural competency, academic focus, and other hard-to-see "X factors" that Minnesota charter advocates typically argue will produce heightened performance.

We certainly do not contest that instructional time, especially the length of the school day, has a notable impact on elementary test score performance. In our regression, in both math and reading, the minutes per day variable has a positive coefficient and is statistically significant beyond the 99 percent confidence interval. Extending the school day appears to be a reliable means of improving academic outcomes, though it should be noted that longer days are not cost-free and thus this practice is resource-limited.

With all that said, we recognize that other researchers might have preferred to omit instructional time from the model. As such, we've re-run our 2014-2015 regression without the instructional time variables. This produces somewhat smaller charter coefficients - but ones that are still negative and statistically significant. You can click to enlarge the full results below.

Concerns were also raised about the inclusion of attendance rate and mobility in the regression, pointing out that there could be a degree of endogeneity associated with these variables. In other words, lower test scores could themselves result in reduced daily attendance and more transfers, rather than the reverse.

Although certainly plausible, we would assert that both attendance and mobility are primarily determined by exogenous factors, such as poverty, familial stability, or housing security. Attendance rate in particular is also likely associated with parental educational participation, and could therefore help control for the selection bias issue, where charter parents are more likely to be involved in a child's education than traditional school parents.

But to fully allay concerns about these variables, we have produced a stripped-down regression that includes only basic demographic student information, plus school size and type. (We have also taken the opportunity to include homelessness in the model, although its impact on school-level performance appears to be small.) Once again, the results show smaller, but still negative and significant, coefficients on the charter dummy.

Ultimately, while there are many ways to specify this model, the top-line findings remain about the same in all of them: in the aggregate, traditional schools have a small but statistically significant edge on charters in academic performance. If there is a case for expanding charters, it must be found elsewhere.