A week after Erie County’s 911 system unexpectedly shut down for 3½ hours, county legislators listened with growing frustration Thursday to the litany of failures that occurred in the early-morning hours of March 30.
Some emergency services administrators contend that the upgraded emergency system – with its improved mapping, texting and mobile phone tracking technology – has experienced a much higher rate of glitches and failures than the other 911 systems that preceded it. And that may have implications for many other police and public safety organizations that use the same technology.
Unlike older and less-adaptable emergency systems grounded in copper land-line technology and less-sophisticated interfaces, the county’s 4-year-old, multimillion-dollar upgraded system appears far more trouble-prone.
Steve Matisz, a supervisory fire dispatcher for the Town of Amherst, told legislators he was aware of four past breakdowns of the county’s current 911 system, though none so widespread.
“They want to assure us it won’t happen again,” Matisz said later. “I have to consider that it would happen again.”
The 911 “fail-safe” system failed last week because of a series of technological and human errors, including two major software failures, and one guy who may have simply pushed a wrong button.
Here’s what county administrators say they have discovered so far, based on their own research and a report provided by Verizon, which maintains the county’s 911 system:
• An air conditioner broke down in a power supply room in the county’s public safety building. A cooling system circuit board was supposed to activate a second cooling system and issue an emergency alert to county employees after the initial malfunction. It didn’t do either.
• When the room heated to near 100 degrees, a late-shift building engineer responded to an alarm and, administrators contend, “overreacted” in an effort to reset the cooling system. He is believed to have pushed a “kill button” that abruptly cut power to the building, abruptly shutting down dispatch terminals, servers and Internet-powered phone systems, and throwing the Public Safety Campus into darkness. The employee denies pressing the button.
• All 911 calls were supposed to be automatically redirected to police dispatchers in Amherst, the Town of Tonawanda, Cheektowaga and the Town of Hamburg. But a software failure cut off 911 calls to all police departments, except for the Town of Tonawanda, which was deluged with ringing lines.
• Verizon also tried to reroute calls to an emergency dispatch center in the Cheektowaga Fire Training Center, but no calls could be intercepted there because of a software failure. Verizon said this occurred because the emergency dispatch laptops used by the county stored too many old files, which interfered with the software’s function. As a result, the laptops could not communicate with computer servers housed just one room away.
Verizon reported that when the 911 system was compromised for several hours, most calls were eventually answered, though callers likely experienced unacceptably long delays. Marlaine Hoffman, deputy director of information services for Central Police Services, said Verizon reported 180 calls were eventually answered and fewer than 10 legitimate, non-test calls were abandoned or lost.
On Thursday, legislators quizzed county administrators from Information Support Services, Public Works, Emergency Services and Central Police Services to clarify what went wrong, and determine how long it will take to fix the problems. Representatives from Verizon and Siemens, the company responsible for the building’s climate-control system, declined to attend the meeting.
Legislature Majority Leader Joseph Lorigo said no emergency system failure should be blamed on one improperly trained county employee who made a mistake, though the fact that the mistake happened is deeply troubling.
Public Safety Committee Chairman Edward Rath III said he’s scheduling an update on this matter for May 5 and expects more direct answers from Verizon and Siemens, which have contracts with the county.
“If we have to compel them to come in, then that’s what we’re going to do,” he said, referring to possible subpoenas.
Though Verizon initially denied any responsibility for the system malfunction, Emergency Services Commissioner Daniel Neaverth Jr. said he still believes Verizon is ultimately to blame.
“The system, as designed, is supposed to survive a catastrophic attack,” he said.
Clearly, it could not, he said.
Hoffman said the county’s current 911 system is scheduled to be “refreshed” next year. In the meantime, however, more extensive and overnight testing has been conducted on the county’s backup systems. Verizon is also issuing software patches to address software failures that plagued emergency responders last week.
Despite the 911 system’s many advantages, Hoffman acknowledged that more-sophisticated technology can open the door to more unexpected issues.
“It’s wonderful in so many ways, and has more flexibility and functionality than ever before,” she said, “but I guess it is a balance of the level of risk. And again, I want to stress, we tested the hell out of this thing. There are just a series of events than happened that we didn’t anticipate.”