CenturyLink explains end-of-year outage - Albuquerque Journal

CenturyLink explains end-of-year outage

CenturyLink executives on Wednesday blamed a faulty electronic unit called a “network management card” for its Dec. 27-28 national service outage.

The failed unit, made by California-based Infinera Corp., clogged up CenturyLink’s Internet services with a tidal wave of faulty messaging, state Government Affairs Director Johnny Montoya told the New Mexico Public Regulation Commission during a presentation Wednesday morning. That overloaded the system and caused a two-day outage that impacted people from Massachusetts to Washington state, including tens of thousands of Internet customers in New Mexico and people whose phones are based on voice over Internet protocol, or VoIP.

Verizon users lost phone service during the outage, since that company relies on CenturyLink to manage wireless data traffic in New Mexico.

Emergency 911 calls were affected in places like Albuquerque and Las Cruces because many people didn’t have phone service to make calls and because the lack of Internet service impeded city systems from automatically routing calls that did come through to emergency answering services, Montoya said.

CenturyLink’s corrective response was slowed because the faulty unit impacted the company’s event-management system, which is set up to help technicians rapidly isolate the cause of problems, Montoya said.

“Once the faulty card was identified, we pulled it,” Montoya said. “We then applied filters and reinitialized over 100 network nodes to re-establish customer traffic.”

The faulty unit has since been sent back to Infinera for analysis to determine the root causes of the problem. But CenturyLink accepts its responsibility, Montoya said.

“On behalf of the company, I apologize for the outage,” he said. “It was our fault.”

CenturyLink is also cooperating with the Federal Communications Commission to investigate what happened.

The company has eliminated the possibility of a cyberattack as the culprit, Montoya said.

In the meantime, CenturyLink is installing network filters and monitoring systems to prevent such incidents from re-occurring.

“We’re looking at additional steps to create more fault tolerance in the future,” Montoya said. “…It’s fair to say this was a wake-up call for us and our customers. We’re looking for ways to make our Internet service and all our services more resistant.”

Commissioners and others asked the company to establish more efficient emergency communications in case of future incidents. During the December outage, customers, public officials and others could not communicate with CenturyLink, creating confusion.

In Las Cruces, local managers and technicians thought their own systems had failed, said Doña Ana County Commissioner Shannon Reynolds. Had they known immediately about the problems with CenturyLink, they could have rapidly switched to an analog phone system to manage 911 calls.

“There needs to be more communication upfront,” Reynolds said. “When a problem this massive occurs, I would encourage (CenturyLink) to send out an announcement to every news agency and others to alert the public.”

PRC Chair Theresa Becenti-Aguilar asked the company to establish direct emergency contacts with the PRC and other entities for swift, 24/7 communications in an emergency.

Albuquerque Journal and its reporters are committed to telling the stories of our community.

• Do you have a question you want someone to try to answer for you? Do you have a bright spot you want to share?
   We want to hear from you. Please email yourstory@abqjournal.com

Nativo Sponsored Content

taboola desktop


One-on-One: Engineering firm CEO's past trauma helps him lead
ABQnews Seeker
Army veteran helps other find healing ... Army veteran helps other find healing at the Grand Canyon
New Mexico firm's gaming universe inspires thousands
ABQnews Seeker
Ganymede Games' "Xenotheria" appeals to grown-up ... Ganymede Games' "Xenotheria" appeals to grown-up gamers' nostalgia.
J. Crew bids adieu to Albuquerque
ABQnews Seeker
J. Crew first opened its doors ... J. Crew first opened its doors for New Mexicans in 2012, according to earlier reporting, and specialized in men's and women's fashion.
As AI progresses, keep humanity, humility in mind
ABQnews Seeker
When my business partner and I ... When my business partner and I launched our company in the mid-1990s, we debated whether to install ...
NM tech firms awarded state matching grants
ABQnews Seeker
The NM SBIR grants are designed ... The NM SBIR grants are designed to support local science and technology companies in achieving their commercialization goals, according to a news release from ...
New ABQ lifestyle shop aims to offer Old Town ...
ABQnews Seeker
Retailer focuses on providing customers with ... Retailer focuses on providing customers with items ranging from small-batch printed cards to jewelry to beauty products made with New Mexican ingredients
College athlete deals risk tax-free scholarships
ABQnews Seeker
The college football bowl season is ... The college football bowl season is over.
Albuquerque restaurant inspections for Jan. 31, 2022
ABQnews Seeker
Crepes and Waffles, 3211 Coors SW ... Crepes and Waffles, 3211 Coors SW (Jan. 14)
Briefcase: Wealth management firm hires Santa Fe vice president
Robert H. Hoffman has joined Morgan ... Robert H. Hoffman has joined Morgan Stanley Wealth Management's Santa Fe Office as vice president and financial advisor.