So much for fail-safe.
Erie County’s 911 system shut down for nearly four hours Wednesday morning – done in by a malfunctioning air conditioner.
And the failures didn’t stop there. The back-up system didn’t work either, and more than half an hour passed before anyone realized it. County dispatchers headed to a secondary location in Cheektowaga, only to discover that emergency calls that were supposed to be diverted there weren’t coming in.
Now lawmakers want answers about how this could occur after spending tens of millions of dollars in recent years on 911 equipment, maintenance, training and testing.
“You should have had every possible scenario tested, retested and guaranteed,” said Legislator Edward Rath III, chairman of the Legislature’s Public Safety Committee. “Why on earth was that not tested?”
County Executive Mark Poloncarz, who was awakened shortly after 4 a.m. Wednesday, pledged a full review of what happened to keep another failure from happening again.
“I’m not going to sit here and say there’s nothing to worry about because, to tell you the truth, when I get a call from the commissioner of emergency services telling me that the E-911 system went down, that worries me,” Poloncarz said.
County administrators said they didn’t know how many 911 calls were lost during the outage but were grateful the breakdown occurred during off-peak hours. They estimated about 40 calls come in during the time period, based on past call volumes. Some emergency calls were eventually answered, but administrators said that in one test, the phone rang 30 times before a dispatcher answered.
Wednesday morning’s breakdown wasn’t the result of a single system failure, but a failure of multiple systems that ultimately paralyzed the county’s front line public safety response system. It also initially appears to be a failure of Verizon, the vendor responsible for making sure that 911 emergency calls would continue to be answered in the event of a primary system shutdown.
Dan Nevearth, the county’s commissioner of emergency services, said he was awakened at 3:45 a.m. and alerted about the breakdown. After speeding to the Public Safety Campus, he entered the basement room housing the circuits powering the emergency system.
“It was easily in excess of 120 degrees,” he said.
An overnight county building engineer was notified that the air conditioning unit in the circuit room had shut down, but by the time he arrived, the room was already so hot that circuit breakers automatically tripped as a defense mechanism to protect the expensive equipment against damage.
That loss of power shut down every emergency response terminal at the county’s public safety building. It also rendered the dispatchers’ regular phone lines and push-to-talk systems useless. Dispatchers and administrators resorted to using their cell phones to communicate, according to Nevearth and Marlaine Hoffman, deputy director of information services for Central Police Services, who is also interim commissioner for the department.
A backup system was supposed to temporarily divert all incoming 911 calls to police dispatchers in Amherst, Cheektowaga, Hamburg and the Town of Tonawanda. Instead, the Town of Tonawanda was flooded with 911 calls, while the Town of Hamburg received only two, Hoffman said. Amherst and Cheektowaga didn’t get any.
When Central Police Services staff tried to establish a backup call center at the Cheektowaga Fire Training Academy, she said, the system failed to recognize the new network or divert any calls to them.
Ultimately, the air conditioning system that had inexplicably shut off was restarted, Poloncarz said. A specialized electrician then came in and repowered the 911 system. After that, staff began the painstaking process of rebooting all the 911 systems in a particular order, administrators said. Most systems were back online by 7 a.m., 3½ hours after they went down.
Hoffman pointed out that the entire 911 system, which Verizon is responsible for maintaining, is regularly tested to ensure the backup systems work. Before Wednesday, all prior tests had been successful, but an abrupt power outage scenario was never simulated, she said.
Verizon did not respond to requests to comment Wednesday afternoon.
Poloncarz noted that although the 911 system has suffered glitches in the past, they were isolated to certain communities. Never before has the county experienced such a widespread failure. Administrators are now in extended talks with Verizon, as well as Intrado, a subcontracting software provider. The county is also working with Siemens, the vendor that provides the county’s climate control systems, to figure out what happened with the air conditioning system.
Despite major, grant-funded investments in the county’s 911 system, Poloncarz said, the county is ultimately dependent on outside companies like Verizon to make the system work. He also said he’s inquiring whether the county can recoup any money from Verizon for the expenses the county incurred on Wednesday.
“I’m not happy we’re paying lots of money on maintenance agreements, and the systems weren’t working the way we they were supposed to,” he said.
Legislator Rath said he’s scheduling an extended session of the Public Safety Committee on April 7 to gather details on what went wrong and what is being done to fix it.
“I’m not trying to play politics,” he said. “I’m not trying to play the blame game here. What I’m interested in is solutions.”