Response to “Research shows progressive places, like Minneapolis, have the worst achievement gaps.”

On January 24th the Star Tribune published an opinion piece by Nekima Levy Armstrong, entitled “Research shows progressive places, like Minneapolis, have the worst achievement gaps.” In the piece, Levy Armstrong argues that racial achievement gaps are smaller in politically conservative cities, and that the public should demand progressive cities like Minneapolis to do better at closing those gaps. For empirical support, Levy Armstrong relies on a report by Brightbeam, a project of the school-reform oriented outlet Education Post. The Brightbeam report analyzes student performance gaps between racial groups in the most progressive and conservative large cities in the U.S.

Unquestionably, as in almost all American cities and regions, Minneapolis and the Twin Cities region are home to significant racial disparities, and these disparities are an important problem that policymakers should address. However, there are numerous substantive and methodological issues contained in the Brightbeam report. These problems cast serious doubt on the report’s conclusion that progressive cities are especially likely to produce gaps , or its suggestion that leaders in more conservative cities merit praise for tackling racial gaps.

The Brightbeam report, entitled “The Secret Shame,” is full of rhetorical claims based on results from a variety of unsound research practices. Most importantly, racial segregation is mentioned only once in the report and is not given consideration in the data analysis, an unacceptable omission given the body of research concluding that segregation is a driving factor in producing racial performance gaps.

The report bases its conclusions on analysis of two groups of 12 cities, one putatively consisting of the most progressive cities in the nation, and the other putatively consisting of the most conservative cities. It does not cite data sources or provide much of a methodological account of the research techniques it utilizes. While the report compares two groups containing a total of 24 cities, it fails to conduct analysis of the remainder of the universe of 67 cities described in the report, which show a range political policy preferences.

The report determines the political lean of each city by using surveys of policy preferences of city residents. However, at the outset, this poses serious methodological issues. First, cities and school districts are separate jurisdictions, and the policy preferences of city leaders may not mirror those of its associated district. Indeed, the survey used does not contain any question about schools or school policy, and instead focuses on respondent attitudes towards topics like the federal government.

An even more fundamental issue is that educational policymaking often does not occur at the city or district level. The governmental entities with the greatest control over questions of school policy are typically the state legislature and state department of education. Brightbeam makes no attempt to incorporate any measure of state policy or political lean, meaning that, in effect, the report assumes that city districts have singular and ultimate control over their schools.

But perhaps the most serious concern about the report’s measurement of citywide political opinion is that it assumes that each city is conterminous with its associated district. In reality, cities and school district boundaries often do not always overlap spatially. In fact, about half the cities listed in the survey are associated with school districts that either comprise a fraction of the municipality, or alternatively, include multiple municipalities and unincorporated areas. Importantly, these mismatches are more common in some parts of the country than others (for instance, in the south, districts are often countywide in scope). The report’s “conservative” cities are more likely to have boundary mismatches—6 of the 12 cities used for the report’s analysis of conservative cities have a major discrepancy.

To illustrate, Oklahoma City actually contains nineteen school districts within the city’s boundary, with the city district the most centrally located. The report offers no explanation of how it addressed these scenarios.

The report also appears to err in its use of school performance data. The authors of the report note that the data are derived by states and that the data between states are not strictly comparable to one another. However, unlike other researchers, Brightbeam has not attempted to standardize the results.

The Brightbeam report claims to control for other factors that might explain differences in gaps between progressive and conservative cities. These factors include student population size, and the racial demographics of the district. However, these are a far cry from all explanatory factors, and the report provides few details on how these controls were applied or why they were selected in the first place.

Here again, the Brightbeam analysis is partially limited by the fact that they have selected 12 of the “conservative” and “progressive” cities but do not include the full set of performance results for all the cities for which data exists.

By simply matching Brightbeam’s ‘conservatism’ scores to 2017-18 performance scores provided by The U.S. Department of Education, EdFacts, we are able to display the full range of performance scores in the 67 cities.

This shows that there is a great deal of variation in student performance and conservatism scores across city districts. For instance, chart 1 shows that there are large differences in the percentage of white and black students proficient in reading after sorting by the level of conservatism in the large cities. Clearly, there are wide racial gaps in the data. However, using the full data, it is much less clear how they relate to a city’s relative political conservatism.

Chart 2 shows the same data, but with test performance measured by white-black student racial gap ratios. We find that, generally speaking, as the conservatism score rises in a city, the racial gaps in student performance drops, but that the relationship between these variables is not strong, with a Pearson’s correlation coefficient of -0.32. There are progressive cities with small racial gaps (e.g. Detroit and New Orleans) and conservative cities with large racial gaps (e.g. Oklahoma City), weakening the correlation between ‘conservatism’ and racial gaps.

There is another important dynamic at play, as well. When breaking achievement gap data out into white and black student proficiency scores, we find that most of the difference between the cities’ racial gaps are due to decreasing white test performance scores in conservative-leaning cities, as shown on chart 3. There is a moderate correlation between white student reading performance and ‘conservatism’ in cities (R = -0.35), but practically no correlation at all when it comes to the performance of black students (R= 0.07).

We find quite similar results when comparing the 67 cities student performance in math, and when comparing the results between white and Hispanic students. Table 1 shows that all correlations between student proficiency and conservatism are weak to moderate (-0.28 to -0.36). Table 2 shows that there is virtually no relationship between black or Hispanic student performance and conservatism in cities. We also find this when limiting the data to the 24 cities reported and analyzed by Brightbeam, a finding not noted by the report’s authors.

Needless to say, this is not an overwhelming finding. Indeed, it suggests that conservative cities, far from being places to emulate, are simply providing uniformly mediocre education.

This is not to say that racial gaps in school performance are not a persistent issue, or that Minneapolis does not have large gaps. In fact, Minneapolis public schools have one of the largest white-black student performance gaps and socioeconomic segregation rates of school districts in the U.S., according to the Educational Opportunity Project at Stanford University. The Stanford project addressed this question with a full complement of standardized neighborhood, school and district data covering the entire nation. The researchers at Stanford find that the gaps can be linked to something that the Brightbeam authors neglect entirely: school segregation. The Stanford authors argue that to narrow such gaps, a reduction in segregation is necessary.

Unfortunately, the Brightbeam authors do not make any such policy recommendation. Their report simply suggests there is something magic in the water in conservatives cities, and use their report to laud the leaders of those cities while chiding progressive leaders. They offer no guidance for Minneapolis city leaders in the midst of important education decisions, such as the new Minneapolis Schools District Redesign plan.

We would argue that a more constructive stance is to perhaps disregard the superficial progressive-conservative divide altogether. Educational policy does not fit on a neat left-right spectrum, and some of the more progressive ideas about addressing segregation and racial gaps in schools come from more conservative-leaning areas, like Louisville, Kentucky, Raleigh, North Carolina and San Antonio, Texas. Putting the blame on progressive city leaders and residents also misses the mark, as segregation driven racial gaps are not limited to city boundaries. Instead they cut across city boundaries, metropolitan regions and political geographies. What is truly needed is a more serious and concerted effort by everyone to address them.