ALBUQUERQUE, N.M. — A nonpartisan group of scientists and mathematicians says the state’s new A-F school grading system is too complex for most people to understand, including principals and superintendents.
The group also believes the system adds elements together that aren’t compatible, and that the formula’s sensitivity to small changes results in unreasonable grade changes from one year to the next.
State education chief Hanna Skandera said the system must be complicated to capture all the elements that make a quality school. She also said her department will release a grade calculator application, which will allow schools to see how fluctuations in test scores translate to letter grades.
The group, called the Coalition for Excellence in Science and Math Education, is a nonprofit with several hundred members. It has been active in New Mexico since 1997, and its mission is to “improve science education and science literacy for all citizens,” according to its website.
M. Kim Johnson, past president and an author of the report, said Rep. Rick Miera, D-Albuquerque, asked the group to examine the A-F school grading system and try to replicate it. Johnson said the group has testified before the Legislative Education Study Committee in the past and did so recently to present its findings.
The group said it was not able to replicate the school grades but added that it did not have all of the Public Education Department’s original data.
Those findings were cited by Democrats in the Legislature, who sent out a news release last week expressing “deep concern” over the credibility of the A-F grades. Those quoted in the news release included Sen. Linda Lopez, D-Albuquerque; Sen. Howie Morales, D-Silver City; and Miera.
“The methodology and process of the reform models are severely flawed, with no scientific or educational findings proving their worth,” Lopez said in the release.
The release urged the PED to adopt the group’s recommendations, which include simplifying and seeking an independent review of the formula and providing grading data and software to districts.
Johnson, a retired physicist, said the coalition began its work by looking at the technical manual the PED provided to districts.
“We looked at it, and said ‘This a big complex mess.’ I mean in terms of being able to understand it. It lacked the definition you expect to see in a manual.”
Johnson emphasized that just because his group could not replicate the scores doesn’t mean it can’t be done. But he thinks it does mean the formula is too complex to be very useful.
“We’re not talking about a bunch of schmucks here who haven’t seen this stuff before,” he said. “We think it’s all probably aboveboard, but we don’t think the average person, school principal or superintendent could conceivably follow it.”
Skandera said the calculator application, expected to be released at the end of September, should help make the grades more useful to principals. The application will allow principals to see how test score changes affect their grades.
For example, if a principal wanted to see how the grade would be affected by a two-point increase in the average scores of low-performing students, she could put that in the calculator for an answer.
“I think that will be a real step toward equipping our principals in a more meaningful way,” Skandera said.
She said the formula is complicated, in large part because people around the state wanted a school grade that would capture all the nuances of school quality, and would control for demographic differences such as poverty.
“I think it’s really important that we remember, everyone asked for a complete picture. And that’s exactly what we delivered: a robust view of what’s happening in our schools, with multiple measures,” Skandera said.
She said she has heard educators debate the system.
“I’ve heard lots of educators going back and forth, saying, ‘Until I know how it’s calculated, I can’t improve my letter grade.’ And someone else will say, ‘I know exactly what I need to do.’ … You can create a bogeyman out of this word ‘complication.’ ”
She also said some of the people criticizing the system’s complexity are the same ones who requested nuance. She said Miera, for example, was adamant about including five- and six-year graduation rates in the calculation. She said that makes it more complicated.
The coalition’s report also questioned whether it is mathematically appropriate to add different measures together. Specifically, the A-F grades use simple measures of how many students are scoring at the “proficient” level, which are added to measures of how much students are improving. The “improvement” scores control for demographic factors, like poverty.
The coalition contends it is mathematically inappropriate to add these measures together. According to the report, such addition is “something like adding oranges and cows to derive pickup trucks. The result is not obviously meaningful.”
Johnson said each measure serves a separate purpose, and that it makes sense to calculate them both. But he said adding them together results in a measure that reflects neither growth, nor current standing. Doing so also is part of the reason for dramatic changes in grades.
Those changes have come under scrutiny, as some school grades went from “B” to “F,” and vice versa, between preliminary January grades and final July grades.
“Schools don’t do that in one year. It doesn’t happen that way,” Johnson said. “And so a lot of that is an artifact of adding things together that really don’t make sense to add together.”
Johnson said he does not think the grading system should be dumbed down to a basic level, but does think it could be simplified without losing its meaning.
“That still may require that there’s math in there that’s complex, but make that executable — something that people can put on their computer and run, especially superintendents and principals,” he said.
— This article appeared on page C1 of the Albuquerque Journal