Step 1: Prepare your documentation
You will need to document all your activities, from meeting minutes and decisions down to commands typed into your systems by your incident response team.
For each step, you will need to record, at minimum:
- Identifying information (location, serial no, model no, hostname, MAC address, IP Address)
- Name, title, and phone number of each person who collected or handled evidence during the investigation
- Time and date (including time zone) of each occurrence of evidence handling
- Locations where the evidence was stored
Step 2: Assemble your Team
Get your incident response team together! Where possible use phone communication – your email and chat systems may be compromised and you might tip off an attacker that you are aware of them.
You’ll need a broad set of people:
- Security personnel, including incident responders
- System and network administrators
- Business stakeholders, such as PR
Remember! Apply a need-to-know policy for now – no need to blow it out of proportion just yet.
Hint: If you have Cyber Insurance, notify them of a potential claim.
Step 3: Determine Scope
Working with your team, determine as best you can what devices have been compromised.
Assume the worst – that more of your environment is compromised. Yes, it will increase the scope of the response, but will reduce the chance of an incident recurring.
‘Indicators of compromise’ are unexpected or suspicious behaviour which may mean an incident has occurred. This may include behaviours such as:
- Strange or unexpected system activity
- Alerts from a Network IDS or Antivirus system
- Unscheduled system crashes or server reboots
- Unexplained configuration changes, unusual files, unknown processes, unexpected web-site changes, etc.
- Influx of phishing e-mails, spoofed e-mails, etc.
- Unusual activity in log files, or gaps in or missing logs
- E-mail system showing a large number of bounced/invalid emails
- Large volumes of network traffic to unknown countries and networks
Step 1: Contain to Affected Systems.
A hacker will try to traverse to other systems, so isolate affected systems as soon as possible. The goal here is to prevent the problem from getting worse.
There are some key actions – these may affect incident response, forensic, and legal activities, so make sure you do it right:
- Do unplug the network cable of affected systems
- Do suspend affected VMs (a copy of the RAM is taken, which is important for forensic analysis)
- Do disable wireless connections (in order of preference: at the router, hibernating the laptop, then disconnecting via the computer operating system)
- Do declare an incident, if it appears to be one
- Don’t run an anti-virus scan (this changes timestamps)
- Don’t shut down operating systems
Step 2: Backup affected systems
You want to keep copies of the affected system for forensic purposes.
The best approach is to remove the affected system from the environment, and provide a new system for the user, or build a new server from a clean SOE.
Of course, sometimes this isn’t possible, in which case you should:
- Obtain a brand new disk drive and create a complete bit-for-bit backup; and
- Get another copy on write-once media (CD-R or DVD-R) in the event that you need pursue legal recourse.
Hint: Use the ‘dcfldd’ tool, which is available for Unix and Windows.
3rd party forensic investigators will have disk cloning hardware to perform this task if you don’t have the relevant expertise.
Eradicate the Problem
Step 1: Remove the malicious code
Eradication means removing the problem from affected systems determined through your scoping efforts.
The actual technical actions for eradication may vary considerably.
For attacks on vulnerable systems, cleaning the system and patching the system may be sufficient.
It can be very difficult and time-consuming to verify that systems are in fact secure, and malware has been completely removed. Rootkits in particular need specialist skills to detect.
We recommend rebuilding systems affected by malware, by either:
- Reinstall the operating system from original media or image, and restore data from the last known good backup onto new media;
- Wipe the existing media, reinstall the operating system from original media, and restore data from the last known good backup;
See the eradication tools in the Links below.
Step 2: Apply compensating controls
If you have indeed been breached, it will be best to apply further controls to ensure you are better able to prevent and detect malicious activity next time.
These controls may include:
- Additional logging and monitoring of systems, applications, and databases
- Increased monitoring of infrastructure logs, such as SIEM, firewalls, and IDS/IPS
- Restriction of logical access to databases
- Additional network segmentation
- Restrict access to databases
Step 1: Recover your systems
Once you’ve eradicated the problem, you can recover the affected systems and return them to production.
Remember to check the integrity of your backups before restoring from them. Malware may have been backed up with your system and data files.
There are key actions when you recover your systems:
- Do patch all affected systems
- Do check and remediate the original attack vector
- Don’t re-introduce the vulnerability from your backups
Step 2: Monitor your environment
You need to conduct logging and monitoring of systems and network traffic to verify that the system or environment has been remediated.
- Setup a sniffer on a switches span port to capture all network traffic
- Log all traffic and send to your logging and monitoring solution
- Check for further activity on the network
If you’re satisfied that the attack has been completely eradicated, then you can formally terminate the incident and conduct post-incident activities.
Have a look at our Incident Response Guide (available to subscribers) for supporting information on conducting post-incident activities, and preparing for the next security incident.
Incident Response Flow Chart
What to capture
- Network Diagram
- Internal LAN IP Address(s)
- External WAN IP Address(s)
- Log Files
- IDS / IPS
- Web / Proxy Server
- Other Application Logs as Needed
- PowerShell ▪ SQL
- Active Directory
- Disk Images for Affected Systems
- Forensic Image of Disk
- Volatile Memory Capture o Event Logs
- Specific Application Logs
- Detailed Timeline Prior to Engagement
- Google Rapid Response – https://github.com/google/grr
- GRR, Rekall, plaso (log2timeline), The Sleuth Kit (TSK), libyal, or alternatives like Guidance Encase, AccessData FTK, X-Ways Forensics, Cellebrite, Volatility, Mandiant MIR, etc.