Before you test your incident response plan, you need to define your objectives and scope. What are you trying to achieve with the test? What scenarios will you simulate? Who will participate and what roles will they play? How will you measure and evaluate the results? Having clear and realistic objectives will help you design and execute a meaningful and relevant test.
I suggest to businesses that they work back from where the sensitive data is and build scenarios that can simulate a real-world attack. Incorporate more than IT and technical resources; you will need leaders in the business to contribute during a real breach, so that should also be the case during incident response planning and testing. The objectives should be based on real risks to the business, I suggest looking at your business risk register and using something that would keep the leadership up at night. This also gets the buy-in from leadership when investments are needed and helps them understand how important the investment in cyber security and compliance is. Make sure that the results can be quantified to business impact.
Setting up predetermined goals and parameters is vital to ensure a comprehensive assessment of your incident response plan. What specific results are you targeting to achieve through the testing phase? Which scenarios do you intend to simulate? Equally important is the identification of participants and their respective roles. Furthermore, defining the metrics and criteria for measuring and evaluating the outcomes is crucial. Setting clear and achievable objectives will enhance your ability to design and execute a meaningful and relevant test that serves its intended purpose.
Most of what the readiness of SOC analysts is measured against should be established at the breach [threat] modeling stage. At the beach modeling stage, you make cyber threat-intelligence -driven selection of attack chain types you are concerned about. Then you use MITRE ATT&CK Navigator to pick techniques relevant to APTs and to technologies in your environment. From cybersecurity perspective you should be testing the ability of your security analysts to respond to those techniques. However, back to our dichotomy: NIST 800-61, compliance and legal aspects do make sense for potential sensitive data exposure or even data theft. I suggest having your legal rep to be at your tabletop and other exercises and review| comment on all records.
The essential components highlighted in the Incident Response (IR) plan lay the groundwork for a robust security strategy. In my opinion, setting the incident declaration thresholds judiciously is one of the most critical aspects. This balance ensures that the organization doesn't overburden its operational cycles or take on undue risk. To add, a cross-team collaboration involving security, legal, risk, and business stakeholders is vital. It's not just about responding to incidents but orchestrating a coherent and efficient plan that aligns with the organization's overall objectives and business functions.
Our cyber team is made up of six different departments. When we started the program, we started in the center with the most critical team, then have expanded outward adding additional teams and (therefore) additional objectives. When identifying the objectives, I always ID primary objectives - which are measured and communicated (after the exercise), but I also like to ID secondary objectives with are more subjective and only share those with the oversight group.
Testing your incident response plan can come in different forms, depending on your objectives, resources, and maturity level. Tabletop exercises are a low-cost and low-impact way to validate your plan, identify gaps, and raise awareness. Walkthroughs are more detailed and interactive tests that involve walking through the steps of your plan and performing some actions. Simulations are high-cost and high-impact tests that involve creating realistic and complex scenarios and executing your plan in a live environment; this is useful for assessing capabilities, testing resilience, and improving performance.
It depends on our main objective, we can create excellent cybersecurity incident response plans, but the main objectives need to be defined and also the roles and responsibilities need to be written and disclosed. We can do a tabletop exercise involving the executive board, we can make plans focused on the blue team, we can create a plan based on risk and risk acceptance, but the most important thing is to understand what is most important for your organization, remembering that for each one it can be different, but for all it needs to be well established and with a defined objective.
If we are talking about verifying and validating our ability to respond to cyber breach, we should employ everything within out budget and resource allocation starting from easy to more advanced methods. When you are ready for hands-on exercise, I'd recommend employing an external purple team (can be a red team, if they agree to a purple exercise scenario). Ideally, you should deploy BAS like AttackIQ for continuous assessment. It is also important to test legal and communication side of things which is what IR plan mostly about. Comms with stakeholders, clients, partners. Let's connect cybersecurity breaches and data theft together. Breaches lead to data theft, but breach itself isn't a data theft unless it's successful.
Once you have chosen a testing method, it’s time to prepare and conduct the test. Make sure to communicate the test plan and expectations to your team members and stakeholders, so they understand their roles and responsibilities, as well as the rules and boundaries of the test. Then, execute the test according to the predefined scenarios and objectives. Monitor and document the actions, decisions, and outcomes of the test, making sure to use realistic and challenging scenarios that reflect your current threats and risks. Finally, collect feedback and data from the participants and observers through surveys, interviews, or debriefing sessions to gather their opinions, insights, and suggestions.
Why not consider running mini-tests to see how your team deals with a real-world incident? This is no different to a common practice today where businesses run phishing tests to see if employees have understood their training.
After selecting a testing method, getting ready and carrying out the test is essential. Please ensure the test plan and expectations are communicated to your team members and stakeholders. This will help them understand their roles, responsibilities, as well as the parameters and limitations of the test. Subsequently, proceed with executing the test based on predefined scenarios and objectives. Monitor and meticulously document the test's actions, decisions, and outcomes, ensuring realistic and challenging designs that accurately reflect the existing threats and risks. Lastly, gather feedback and data from participants and observers by conducting surveys, interviews, or debriefing sessions.
Include leadership from across the organization. Legal, Finance, HR, and Customer Support or Success to name a few. Depending on the scope of the incident most will have roles to play. It also helps demystify what is happening during an incident to maintain responders focus with reducing Q&A during an actual incident.
The more tests we conduct the more familiar we become with the players. Because we conduct shorter (90 - 120) min exercises on a quarterly basis I cannot allow the exercise to get too bogged down into minutia or rabbit holes. Before each exercise I remind the players that my job is to keep the exercise flow moving and therefore I might need to interrupt the conversation or inject something to get it back on track. I always try my best to "guess" what the conversation will be like on a certain slide - when my best guess is off - that is when I need to get things back on track. Of course, there are occasions when a side conversation leads to a great finding too - it is a delicate dance to lead!
Once the test is complete, you should review and update your incident response plan based on the feedback and data collected. Analyze the results of the test to identify strengths and weaknesses, compare actual outcomes with expected outcomes, and evaluate your performance. Afterward, make any necessary changes and improvements to your plan, addressing any gaps, issues, or errors discovered during the test. Incorporate any new information, best practices, or lessons learned. Finally, communicate the updated plan to team members and stakeholders, providing them with updated training and documentation to ensure they are aware of the changes and their implications.
It is crucial to thoroughly review and revise your incident response plan at least every six months by considering the feedback and data gathered after the test. Analyze the test results to identify areas of strength and weakness, compare the actual outcomes with the expected ones, and assess your overall performance. Implement any necessary adjustments and enhancements to the plan, addressing any identified gaps, issues, or errors uncovered during the test. This process should involve integrating new information, incorporating best practices, and applying lessons learned. Finally, you can communicate the updated plan to team members and stakeholders.
This is possibly the most important part. There's no point doing tests unless the results will be thoroughly studied for insights on how to refine the organisation's incident response strategies. Organisations must continually iterate their incident response plan as well as their information security processes, using feedback they generate from these tests.
Testing and updating your incident response plan is an ongoing process that should be repeated regularly to ensure its effectiveness. The need for a test or update may be triggered by changes in your organization's structure, policies, or processes; changes in your IT environment; changes in the threat landscape; or scheduled reviews, audits, or exercises. Keeping your plan current is essential for responding effectively to incidents.
The plan must be known, executable, realistic and involve the right people. It must be repeated with different periodicities depending on the size and type of company/industry, it must be adjusted whenever necessary and versioned for auditing purposes, and it must always be communicated to the entire company and especially to the actors who participate in it, whether in the elaboration or execution.
Next month I will be facilitating our 25th cyber exercise. We do quarterly exercises. From my experience - it is easy to say, conduct multiple tests a year, but if there is little value in the exercises - you will lose your audience/stakeholder support fast.
A realistic plan is the only way to know what to do in case of attacks, the idea of the plan is not to avoid an attack, for that we have the tools, controls, processes and technology, but the plan is made so that everyone knows what your role. Besides being important for a quick recovery.
The need for effective cybersecurity incident response practices cannot be overstated. It is vital for organizations and individuals to regularly review and update their incident response plans, remain proactive in detecting and responding to incidents, and allocate sufficient resources and support.
Share schedule of tabletops, results, and action plan with the board of directors. If willing include one board member, head of audit committee preferably as interaction with the internal and external auditors and regulators will likely follow.
Incident response testing is one of the key differentiators between mature organizations interested in mitigating actual risk and immature organizations just checking a box on a compliance report. Your organization will eventually test its plan, one way or another. However, it’s better to test it in controlled conditions on your terms rather than the chaos of an actual incident.