When systems go down, the technology teams are usually asked what should be restored first, which is natural because they know the environment best. Those teams often end up making risk-based decisions about which systems to prioritize, even though those decisions are, at heart, about the business rather than the technology.
Recovery priorities are business decisions
Those decisions matter because a business rarely needs everything back at once. An organization may need only 20% to 30% of its systems to resume a minimum level of operations. The difficulty lies in identifying which ones, because that depends on knowing which systems support the business services that generate revenue, and that picture is often less complete than leaders expect.
Deciding which services come first is a business judgment and the technology teams’ role is then to set out the options and put the chosen plan into action.
Backups may be less protected than assumed
Once the priorities are clear, the next question is what those systems can be restored from, which for most organizations means their backups. In our experience and that of the recovery specialists we work alongside, most organizations believe their backups are fully protected, though the reality often proves more nuanced.
We often find that backups are linked in some way to the system that decides who can log in to what across the organization, which can give attackers who have taken control of that system a route to the backups. Backups that are believed to be immutable, meaning they are set up so that no one can alter or delete them, may therefore be more exposed than the business realizes.
Organizations also tend to give their strongest protection to the systems they consider most important, such as finance or customer records. Getting those systems running again can depend on many others that support them behind the scenes, and those supporting systems are not always protected to the same standard.
When the backups are in doubt
Even where the backups hold the right information, it is not always clear at first whether they can be used. A copy taken after the attackers gained access may carry their malicious software or the same weaknesses they exploited, so identifying a duplicate from before the compromise takes time. That copy may be weeks or months old, and therefore restoring from it would mean losing whatever has changed since.
If the backups cannot be relied on, the business may need another way to recover its systems. In many incidents the attackers lock the organization’s systems and offer a key to unlock it, which is why leaders may also decide early on whether to make contact with them.
Doing so can buy time and help leaders learn more, such as what the attackers say they have taken and whether anything they offer could restore the data. Contact does not commit the business to negotiating and negotiating does not commit it to paying. Treating these as separate decisions allows leaders to gather what they need without closing off any route back.
Restoring is usually quicker than rebuilding
Once the business knows what it can restore from, the next decision is how to restore it. After an attack there can be a strong pull toward starting again with new equipment, driven by the belief that the existing systems may still be infected and could allow the attack to recur. The outage can also look like a natural moment to update systems the business had planned to replace anyway.
We have often seen this approach take longer than expected. If the core systems the rest of the business runs on are replaced part-way through recovery, older applications built on top of them are often unable to work with the replacements straight away.
The change may then have to be reversed, which can delay restoration, sometimes by weeks. Restoring the existing systems instead of building new ones tends to be quicker, provided the specific weaknesses the attackers used are fixed at the same time. That might mean updating software that had not been patched, changing passwords that may have been stolen or adding an extra check when people sign in. The wider redesign can then follow once the business is running.
Working alongside specialist recovery teams
Whichever approach is chosen, much of the work falls to specialist recovery teams. They operate alongside the people who run the organization’s IT day-to-day, whether that is an internal team, an outside provider, or both. Bringing in specialists means handing control of the organization’s systems to people it may never have worked with, at one of the most difficult moments it will face.
The specialists will put strict controls in place to keep the attackers out, such as blocking connections between systems. These can cut across the way the environment is normally run. IT teams that have not been through a recovery of this kind may question those controls or bypass them, often in an effort to keep the business running.
Each time that happens, the specialists will have to stop and resolve it before restoration can continue, so the delays build up at the point when the business most needs its systems back. Where the specialists’ lead role and the supporting role of the day-to-day IT teams are understood early on, those delays tend to be far fewer.
Deciding when to switch back on
However capable the specialists, the risks they are managing remain the organization’s own. Forensic firms, recovery teams and lawyers can explain the risks and present options, while the decision to accept a level of risk and resume operations rests with the business.
The clearest test of that is deciding when to start switching systems back on. Doing so while the forensic investigation is still under way can feel uncomfortable, because until it is complete the organization cannot be sure how far the attack has spread or whether the attacker could regain access.
That uncertainty is why many assume that nothing should return until the investigation has finished. In many of the recoveries we have been involved in, we have supported the decision to begin a phased return while the investigation continues.
Some systems lend themselves to this, such as phones or video calling run by an outside provider in the cloud, because they sit apart from the organization’s own network. They are much less likely to have been used to gain entry and can often be brought back with simple safeguards.
In practice, that might mean limiting video meetings to invited participants who are admitted one by one and asked to keep their cameras on, so it is clear who is taking part. Essential business activity can also be allowed to resume in a controlled way while the rest of the environment stays locked down. Step by step, the business can regain meaningful function before the investigation has concluded.
None of these choices has a purely technical answer and each is made before the full picture is known. How much disruption, data loss and remaining risk to accept is a judgment for the business.
Recovery tends to go best where it is clear from the start who will make it, which is a question an organization can settle well in advance.
