5 steps to create a security incident response plan

Creating an incident response plan can seem like a daunting task, but there are ways to break the process down into manageable pieces. (Plus: Video on how to create a pandemic disaster recovery plan.)

A man with an umbrella stands waist-deep in water as rain continues to fall.
Gruizza / Stationary Traveller / Getty Images

The only thing worse than getting hit with a cyberattack is getting hit with a cyberattack and not having a strong security incident response plan in place.

Sophisticated Advanced Persistent Threat (APT) attacks are typically aimed at high-value targets like credit card companies, banks, retailers, healthcare facilities and hotel chains that store large volumes of credit card data and other personal information, But no company, no matter the size and no matter the industry, is immune from insider attacks or from random malware, phishing, ransomware and denial-of-service attacks.

[ Security school: Insider Pro subscribers enroll for free ]

The manner in which a company responds to a breach can mean the difference between containing the incident and quickly bouncing back to business as usual; or suffering damage to the company’s reputation that lasts for years.

According to the Ponemon Institute’s 2019 Cost of a Data Breach Report, the average cost of an incident is $3.9 million worldwide and $8.2 million for U.S. businesses. Having an incident response team in place reduces the cost of a breach by $360,000, according to the report.

In addition, companies that complete the entire incident response (IR) lifecycle of detection, containment, eradication, remediation and recovery in less than 200 days save $1.2 million, compared to companies that take longer than 200 days, according to the report.

Organizations also need to keep in mind that only 67 percent of the costs associated with a breach occur during the first year; 22 percent of costs pile up in the second year as companies work to rebuild their reputation, slow down the rate of customer defections and try to attract new customers. And the long-tail effect of a breach is felt in years three and beyond, according to Ponemon.

[ Related: How to protect remote workers from the coronavirus crime wave ] 

Creating an incident response plan can seem like a daunting task, but there are ways to break the process down into manageable pieces. Think about incident response as a complete feedback loop that starts with identifying threats before they cause any harm, quickly stopping the bleeding if a breach does occur, fixing whatever holes in your security defenses allowed the attack to occur and then learning lessons from the incident that can be applied to the company’s ongoing breach prevention and detection activities. In other words, plans need to cover pre-breach, breach and post-breach activities.

1. Build an incident response team

If you’re ever hit with a breach, the last thing you want to be doing is trying to put together an incident response team on the fly. It’s critical that organizations have an incident response team in place that includes your most capable IT security pros, but also includes broad participation across the company.

Support from senior management is important. Having the backing of corporate leadership enables IT execs to recruit the qualified people that you need to make the plan work. And those people need to be trained so they understand their roles and responsibilities.

Depending on the size and global reach of the organization, you might need to create multiple teams, for example, one for North America and one for Asia-Pacific, to accommodate different languages, regulatory and reporting requirements, etc.

In addition to security analysts, you’ll need someone who can handle internal communication between the incident response techies and key stakeholders, including senior leadership, the board of directors, and nontechnical employees. Partners, suppliers and other third-parties involved in the supply chain also need to be immediately informed.

[ Related: The trouble with 2fa ]

Lawyers and auditors need to be in the loop to handle a variety of issues, including regulatory compliance, legal liability and dealing with law enforcement. There has to be a crisis management team on the public relations side of things to try to manage the negative fallout from a breach. Customers need to be notified. And marketing needs to be included at some point to develop a PR strategy aimed at rebuilding customer trust.

The team also has to have a leader, the incident response manager, and it should have someone specifically assigned to capturing documentation. It’s also critical to have clear lines of communication, to have contact information at everyone’s fingertips and to have backups or alternatives in case a key team member is on vacation or not available when disaster strikes.

There should be clear guidelines on what level of detail should be communicated to IT management, to senior management, to affected departments, to affected customers, and to the press.

There are other communication issues that need to be considered: If the attack has just been identified the attacker may still be listening in on internal communications, so it might make sense to stay off of email, messaging or collaboration apps, get everybody in a room and communicate face-to-face. The team might need to travel from corporate headquarters to the site of a breach, so those types of logistics needs to be taken into account.

An incident response team needs to exist not just on paper. Companies need to carve out the time to conduct meaningful fire drills aimed at making sure everyone knows what to do when a real incident occurs. 

[ Related: Don’t let the coronavirus make you a home office security risk ]

2. Create a playbook

Organizations must create a playbook  ̶  a comprehensive, detailed guide for responding to a security incident. Without a playbook, there is no incident response plan.

The playbook should cover preparation, detection, analysis, containment, eradication, recovery and post-incident handling. One key point about the playbook is that it needs to detailed enough so that roles, responsibilities and procedures are clearly spelled out. On the other hand, since no one can predict the type and severity of the breach, the playbook needs to allow for flexibility, for people to have the freedom and authority to make important decisions on the fly, as the situation warrants.

For example, the playbook developed by Carnegie-Mellon University runs 11 pages. The state of California’s Department of Technology incident response plan is four pages and includes a 17-step checklist for team members to follow.

It’s also important that organizations continually update the playbook as circumstances change and threats evolve. The playbook should be refined continuously, based on lessons learned by the incident response team as they respond to threats. Insights from external sources, like industry associations, analysts and peer groups, should also be incorporated into the playbook.

Of course, no amount of practice can prepare an incident response team for the real thing, where incident response staffers might be working 18-hour days, seven days a week, possibly for weeks at a time. But having an incident response playbook that defines roles and responsibilities certainly helps.

[ Related: COVID-19 and tech: New collaboration tools mean new security risks ]

3. Prevent and prepare

Obviously, the best incident response plan is one that prevents breaches in the first place. Organizations should conduct vulnerability assessments and other types of analyses designed to identify and plug security holes. Companies also need to train employees on security best practices in areas like creating strong passwords and not clicking on phishing attempts. The prevention phase should also include making sure the required tools and resources are available in the event of a breach. 

Companies also need to conduct a serious assessment of the business value of data and applications. This step enables security teams to shore up defenses that protect the company’s crown jewels with an eye toward first identifying the types of valuable data that might be attractive to an attacker and making sure extra security measures are in place to protect that data.

One of the most surprising findings in the Ponemon report is that the average time required to identify a data breach is 197 days and the average amount of time required to contain a data breach once it is identified is only 69 days. That means that in many cases attackers are rooting around in your systems for more than six months before they’re discovered.

[ Related: Next-generation endpoint security goes beyond the endpoint ]

How does that happen? Only 40 percent of companies in a recent IDC survey sponsored by Splunk have a broad incident response plan in place, and only 14 percent have an incident response management platform that automates the process.

What ends up happening is that security analysts get buried by an avalanche of alerts that they are unable to accurately triage to determine which are false positives and which pose an immediate threat. Two-thirds of enterprises surveyed by IDC reported that they experience an attack once a week, 30 percent are attacked at least daily, and 10 percent suffer attacks hourly. 

In this environment of constant attack, only 27 percent of companies surveyed said they are coping comfortably with the volume of attacks, while 28 percent said they were struggling and another 5 percent said they were constantly putting out fires. Security teams are overwhelmed by the number of attacks and don’t have the tools, processes and plans in place to effectively manage their incident response activities.

That conclusion is backed up by a research survey from Imperva which found that on average an analyst working in a security operations center (SOC) investigates 20-26 incidents a day. Under this barrage of alerts, security pros end up ignoring some alerts or modifying their policies in order to receive fewer alerts.

A majority of IT professionals (53 percent) in the Imperva survey said that their organization’s SOC’s have struggled to pinpoint which security incidents are critical vs. those that are just noise.

The best way to effectively tune out the background noise is through automation and orchestration of incident response. According to the Ponemon report, automation reduces the average cost of a data breach by $1.55 million, and improves prevention, detection, response and containment of cyberattacks. Through automation, security analysts can become more effective and can take action based on better information. Orchestration can make incident response up to 40 times faster, said Ponemon.

4. Identifiy and contain

When a potential incident is identified, the incident response technical team should immediately spring into action, alert members of the larger, corporate-wide incident response team, start collecting evidence, decide on the type and severity of the incident, and document everything they are doing.

The immediate goal is to contain the incident and prevent further damage from occurring. This can involve a variety of standard security practices, such as isolating a network segment under attack, taking down production servers that have been compromised, or putting any other compromised asset under quarantine.

Longer-term, the goal is to get the affected systems back into production so the company can return to normal operations. This can be a complex task because incident response analysts are probably still trying to figure out how the attacker got in, what they might have gotten away with and whether the attack is ongoing. The next step is to identify the root cause of the attack, eradicate it and then shore up your systems to prevent similar attacks in the future.

In the recovery phase, organizations bring affected production systems back online in a careful, methodical process that involves testing the systems, verifying that they are performing normally and monitoring them to make sure that everything is back to normal.

After recovery comes remediation. For example, if the attacker got in through a point-of-sale (POS) terminal, then the company needs to revisit all of its security policies and procedures around POS. If the attack involved a compromised password, then the organization should re-visit its password policy and consider adopting two-factor authentication.

5. Perform post-incident analysis

After the incident is over, organizations need to take the time to conduct a thorough investigation, identify the cause of the incident, calculate the costs, and draft a strategy to prevent similar incidents in the future.

It may be somewhat of a painful exercise to go back and identify the mistakes that security teams might have made, a configuration error, for example, that led to the breach. Or it might have been an end user clicking on a phishing link. Or an insider attack. Whatever the root cause, gathering information can help companies identify where to focus their attention aimed at preventing breaches in the future. And all of those lessons learned need to be incorporated back into the playbook.