It was 2 a.m. Palm Sunday. Computer screens across Erie County Medical Center flashed white with bright red words: "What happened to your files?"
The ransom demands began with hot pink text.
"Step1: You must send us 1.7 BitCoin for each affected PC OR 24 BitCoins to receive ALL Private Keys for ALL affected PC's."
Hackers had encrypted the hospital's files and wanted the current equivalent of $44,000 to provide a key to unlock them.
By 3:30 a.m., the medical center, while still assessing the damage and risks to private patient information, had shut down all its computer systems as a precaution.
It was a potentially crippling move that forced one of the region's major health care institutions to go low-tech.
Now, after six weeks of around-the-clock efforts to reconstruct its systems, ECMC is closer to normal operations. Officials say no patient data was compromised. But the cyberattack left a lasting impression, magnified by a growing epidemic of computer attacks, including the global ransomware extortion that disabled hundreds of thousands of computers this month.
"What's happening is a form of terrorism like an attack on critical infrastructure," said Thomas Quatroche, president and chief executive officer of the 602-bed hospital and 390-bed long-term-care facility on Grider Street. "It's a call to action to view cybersecurity the way we do law enforcement, to raise the profile of the issue."
The medical center follows a protocol for computer issues and uses regular down times for parts of its system to practice. But no one expected a disruption as long or extensive as this.
ECMC's network would go dark for weeks. But in the hours after the attack, hospital managers had a decision to make: Should they pay the ransom?
The morning of the attack
By 5:30 a.m., the hospital called in cybersecurity consultant GreyCastle from Troy and worked to notify top managers.
"For the first few minutes when I learned what happened, I was in a state of disbelief," said Dr. Jennifer Pugh, associate chief of service for emergency medicine. "Then my reaction changed to anger. This is our Level 1 trauma center. It felt like a direct attack."
Quatroche assembled his management team by 9:30 a.m. to organize a response.
"My first thought was to let people do what they have to do. We needed to identify what was going on and get going using paper," he said.
Many businesses quietly pay ransoms. But one of the first decisions made at ECMC, with advice from GreyCastle and law enforcement authorities, was to refuse to do that.
Among the reasons:
ECMC had access to a tape backup to restore files, as well as HealtheLink, the regional system for sharing health information electronically among hospitals and doctors. The hospital outfitted critical departments, such as the emergency room and intensive care, with borrowed laptops with ad hoc internet access. Through HealtheLink, doctors and nurses could view patient records that existed up to the date of the attack.
Officials also voiced concern that the perpetrators might not provide the key after getting the money. And even with a key to decrypt the system, how could they be certain everything was OK?
"A part of it also was about the integrity of the organization," said Quatroche, acknowledging that the hospital will likely bear a high cost for recovering from the ransomware attack.
He said ECMC increased its cybersecurity insurance coverage in November and, in the context of the small margins generally of hospitals in New York State, remains in a good position financially – with stable patient volumes and a balance sheet about $2 million ahead of expectations for the year as of March.
"Whether to pay or not is a very individual thing," Quatroche said. "If you have no backup, you have no choice."
A week earlier
Ransomware commonly spreads by conning a person to click a link or download an email attachment that looks like a message from a friend or institution, such as a bank requesting verification of a password. Attackers also search the internet for vulnerabilities – systems without updated software security patches, for instance.
This case was different. Officials believe hackers used an automatic program that anti-virus software could not recognize to exploit a hospital web server accessible remotely that should have been configured differently to prevent an incursion. The hackers then applied "brute force" computing – trying millions of character combinations to identify a relatively easy default password to gain entrance into the hospital's system.
Officials believe the hackers randomly accessed the ECMC server about a week before the ransom notes arrived using a variant of ransomware known as SamSam.
Once they had breached the perimeter, it's believed a person then logged in and manually searched files. The intruders then encrypted files in a way that made it more difficult to recover data before they issued the ransom note.
"This attack was in our top 10 percent in terms of sophistication, and the manual intervention with someone poking around was unusual," said Reg Harnish, chief executive officer of GreyCastle Security, the Troy cyber-security consultants hired to assist the hospital.
SamSam, which targets vulnerabilities in servers to infiltrate computer networks, is responsible for other attacks, including a major ransomware incident last year at 10-hospital Medstar Health in Maryland.
Harnish said he does not believe the hackers knew they had hit a large hospital until they searched ECMC files and, after discovering the business of their victim, demanded more money than typical in ransomware attacks.
The decision not to pay the ransom came quickly on April 10. Restoring the system, computer-by-computer, would take weeks.
After the attack
In its response, ECMC turned back to paper charts and face-to-face messaging – easier said than done in any modern hospital that has come to rely on a complex array of integrated computer systems to run every major aspect of the organization, from patient records and communications to bed tracking and image archiving to lab reports and finances.
Quatroche said the hospital managed the crisis with changes that proved bumpy at times and foreign to many staff members too young to have experienced work life before the internet age.
"Our people were tested, and it blew me away. They have been resourceful, and have rallied around each other and the patients," he said. "There also was a silver lining in that we learned that having administrators do rounding through the hospital is something we need to do more of in the future."
According to a timeline provided by ECMC officials, here is how the restoration unfolded in the weeks after the ransom notes first surfaced on Sunday, April 9:
- April 10: The hospital began to obtain laptops, some of them from the Kaleida Health hospital system. The roll-out started with the emergency department and critical care, and included wireless hot spots to access the Internet.
- April 19: The attack affected more than 6,000 computers at the hospital, all of which had to be wiped clean and re-distributed in phases starting on this day, with the emergency department and critical care areas given priority. The computers worked in view-only mode.
- May 5: Doctors could begin to upload their progress notes into the electronic medical record. Nurses in the emergency room could do electronic documentation again.
- May 8-10: Computer-order entry worked again, initially in the emergency department, allowing physicians to communicate with radiology, the lab and other departments. Desktop computers continued to return to their stations.
- May 12: Electronic prescribing came back online.
Harnish praised ECMC's response and characterized its cybersecurity as similar to the average hospital, but noted that just about every business needs to step up efforts to prevent attacks from increasingly resourceful criminals.
"There was nothing negligent or out of the ordinary," Harnish said. "They quickly identified the issue and escalated. That was important. They had done disaster preparedness. There was muscle memory, and people worked well as a team to deal with this instead of finger-pointing."
The identity of the perpetrators, who can easily mask their computer activity, remains unclear, according to ECMC officials. An investigation traced the ransomware to computer connections in such countries as Brazil and the Netherlands, but hasn't determined a point of origin.
What happened at ECMC reflects a global ransomware crisis. On average, more than 4,000 ransomware attacks have occurred daily since Jan. 1, 2016, disrupting hospitals and businesses, a 300 percent increase over the approximately 1,000 attacks per day in 2015, according to the FBI.
In addition to GreyCastle, ECMC has received assistance from an assortment of computing firms, including Microsoft, Cisco, Symantec and Meditech, its electronic medical record vendor. ECMC staff came in on their days off. Information technology personnel from Kaleida Health and the Catholic Health system also helped.
Officials said they expect most systems to be running normally later this week, although work remains to be completed in outpatient clinics.
The focus on the emergency department reflects its importance. More than 80 percent of ECMC's admissions come through the emergency room, and it serves as the region's Level 1 trauma center for adults badly hurt in motor vehicle crashes, industrial accidents and acts of violence.
The emergency department never went on diversion to send patients elsewhere. As in other areas of the hospital, doctors, nurses and other staff found workarounds.
They charted patients on paper, information that will have to be entered eventually into the electronic medical record. They viewed X-rays on old light boxes instead of computer screens. Clerical staff ferried samples and reports back and forth to the lab.
"It was like a blast from the past," said Pugh.
New York State required electronic prescribing in 2016, so some veteran doctors brought in their unused paper prescription pads from home. ECMC rushed an order of stamps for other doctors to use on generic prescription pads.
"One of the key things that got us through this is we have a plan in place and we practiced," Pugh said.
The attack highlights the risks of a connected world.
"This is a people problem, not a technological one," said Harnish, referring to the fact that most incidents arise from inadvertently introducing malicious software into a computer through a scam email. "We want things to be easy and fast. But we also need to develop a culture of security to minimize those risks."
William Pelgrin, co-founder of CyberWA, a security consultant, said organizations must adopt "good cyber hygiene," routinely taking such steps as using strong passwords, backing up data and limiting access to networks.
There is no way to guarantee 100 percent protection, but individuals and businesses can improve the defenses in their control, he said.
"Security is not just for the IT department. We all have a responsibility, and should be held accountable," said Pelgrin, also a co-founder of the Global Cyber Alliance and former president of the Center for Internet Security.
Quatroche said one of the lessons learned is that hospitals like ECMC must alter their thinking about cyber security. Among other steps, the medical center plans to tighten access to the internet.
"Technology is important," Quatroche said. "But there is a tradeoff between convenience and security that requires people to work differently."