Jera-IT’s Three-Pronged Approach to Incident Response and Disaster Recovery

The Akira ransomware attack

One morning, Jera-IT’s help desk started getting many support tickets from a client since several of their servers had gone offline. On diagnosis, the Jera-IT team discovered that the client still had many servers online and could function. Since the team was already discussing server changes and upgrades with that client, the initial evaluation was of a physical host failure.

However, while digging deep into the issue to bring all the servers back online, the Jera-IT team quickly discovered data encryption damage and a text file titled Akira, hinting that it was a cyber incident. “It’s at that point I was called into a conversation with my engineers to support them. I’ve been in the game long enough to know what we were up against and what we had to do,” recollects Clark.

Upon further investigation, the Jera-IT team discovered that almost 90% of the client’s services had been encrypted all the way down to the operating system and virtual machine (VM) level. However, the hackers could not compromise the entire network since Jera-IT had installed some of the client’s servers and domain controllers in other locations.

“We assumed everything on the network was dirty and kicked off a full disaster recovery (DR) plan. We disconnected everything from the internet and had to find other ways even to communicate with each other,” says Clark.

The three-pronged approach

“While tackling this situation, we had three buckets, each coming at the situation from its own angle,” mentions Clark. “First and foremost, our client had to continue its business operations at all costs. Second, an incident response (IR) company was trying to ring-fence everything with a big sticky tape saying, ‘Do not touch.’ Then there was us — and probably the hardest challenge of all — trying to help the client recover and get them back up and running, all while balancing these three buckets and trying to get an answer that would suit everybody,” he adds.

The first thing that Jera-IT did while tackling this situation was to turn to Datto. “We have spent years working with this client, with Datto backups in place, and we were certain that the Datto backups were sound. That was indeed the saving grace for us,” states Clark.

Result: Back up and running

Thanks to Datto’s robust disaster recovery, Jera-IT could recover some of the client’s cloud-based business-critical applications, enabling the client to continue its business operations amid the chaos. Subsequently, the Jera-IT team was able to contain the issue and started bringing servers back online in a controlled manner. They deployed sophisticated tools to understand how the threat actor infiltrated the network and what damage was caused.

Every server was cleaned and protected with passwords, firewalls and other core services before they were brought back online. The servers had an endpoint detection and response (EDR) service added to them that hadn’t been deployed before. In a month’s time, all the client services were back online.

“To be fair, the client never lost a day of production, and one of the main reasons for that is the backups from Datto. If it hadn’t been for that, we’d never have been able to recover the servers in the cloud, and we’d never have been able to restore the client services in the way we did,” asserts Clark.

The key takeaways for MSPs and businesses

According to Clark, this story leaves food for thought for both the MSP and business community.

“Whenever I interact with decision-makers from the SMB world, they often say that they don’t have enough budget to bolster their cybersecurity — but it’s amazing how that budget finds its way to remediate a cyber incident when it hits them,” comments Clark. “At the start of this incident, the client didn’t have an EDR solution in place, and if I was going to speak to them about an EDR solution, I guarantee I would not have gotten the response I expected. However, today, they have a full EDR service implemented. So, it’s interesting why people wait for such incidents to bolster their cybersecurity.”

Clark further notes that such cybersecurity incidents are also a big concern for MSPs. He adds, “While MSPs believe that a business continuity plan or a DR plan would bail them out of such incidents, they’re not prepared for the impact these incidents could have on their business. Most MSPs are running 90% full tilt, and such a cyber incident will take a significant toll on their workforce.”

Clark reminds us that cybersecurity is a continuous journey without a finishing point. “It will be difficult for business owners to hear this, but this conversation around cybersecurity is constantly going to evolve over time,” he concludes.