ALBUQUERQUE, N.M. — CenturyLink executives on Wednesday blamed a faulty electronic unit called a “network management card” for its Dec. 27-28 national service outage.
The failed unit, made by California-based Infinera Corp., clogged up CenturyLink’s Internet services with a tidal wave of faulty messaging, state Government Affairs Director Johnny Montoya told the New Mexico Public Regulation Commission during a presentation Wednesday morning. That overloaded the system and caused a two-day outage that impacted people from Massachusetts to Washington state, including tens of thousands of Internet customers in New Mexico and people whose phones are based on voice over Internet protocol, or VoIP.
Verizon users lost phone service during the outage, since that company relies on CenturyLink to manage wireless data traffic in New Mexico.
Emergency 911 calls were affected in places like Albuquerque and Las Cruces because many people didn’t have phone service to make calls and because the lack of Internet service impeded city systems from automatically routing calls that did come through to emergency answering services, Montoya said.
CenturyLink’s corrective response was slowed because the faulty unit impacted the company’s event-management system, which is set up to help technicians rapidly isolate the cause of problems, Montoya said.
“Once the faulty card was identified, we pulled it,” Montoya said. “We then applied filters and reinitialized over 100 network nodes to re-establish customer traffic.”
The faulty unit has since been sent back to Infinera for analysis to determine the root causes of the problem. But CenturyLink accepts its responsibility, Montoya said.
“On behalf of the company, I apologize for the outage,” he said. “It was our fault.”
CenturyLink is also cooperating with the Federal Communications Commission to investigate what happened.
The company has eliminated the possibility of a cyberattack as the culprit, Montoya said.
In the meantime, CenturyLink is installing network filters and monitoring systems to prevent such incidents from re-occurring.
“We’re looking at additional steps to create more fault tolerance in the future,” Montoya said. “…It’s fair to say this was a wake-up call for us and our customers. We’re looking for ways to make our Internet service and all our services more resistant.”
Commissioners and others asked the company to establish more efficient emergency communications in case of future incidents. During the December outage, customers, public officials and others could not communicate with CenturyLink, creating confusion.
In Las Cruces, local managers and technicians thought their own systems had failed, said Doña Ana County Commissioner Shannon Reynolds. Had they known immediately about the problems with CenturyLink, they could have rapidly switched to an analog phone system to manage 911 calls.
“There needs to be more communication upfront,” Reynolds said. “When a problem this massive occurs, I would encourage (CenturyLink) to send out an announcement to every news agency and others to alert the public.”
PRC Chair Theresa Becenti-Aguilar asked the company to establish direct emergency contacts with the PRC and other entities for swift, 24/7 communications in an emergency.