# READINESS ###### Preparing a proactive response to attacks An F-Secure Consulting whitepaper ###### o attacks ----- ### F-SECURE CONSULTING F-Secure Consulting is a research-led cyber security consultancy, partnering with enterprises and early adopters worldwide. We help protect organizations and individuals in a fast-changing world by providing evidence-based recommendations and services. Our evidence is the product of continuous research and development, conducted by consultants whose work has been recognized and utilized by the security community for almost two decades. We are intellectually curious, obsessed with solving the most complex security challenges, and driven to build our clients’ cyber resilience through collaboration and technological innovation. www.f-secure.com/consulting @FSecure_Consult linkedin.com/company/f-secure-consulting ----- ## CONTENTS ###### BACKGROUND PROACTIVE VS REACTIVE RESPONSE INCIDENT READINESS IN ACTION SETTING THE SCENE INCIDENT PHASE 1: INITIAL BREACH INCIDENT PHASE 2: DOMAIN COMPROMISE INCIDENT PHASE 3: ATTACKER CASH-OUT POST-INCIDENT LEARNINGS SUMMARY AND CONCLUSION REFERENCES ###### 01 02 02 03 04 08 12 14 17 18 ----- ## BACKGROUND ###### Recent events have shown that it is possible to effectively respond to a crisis. Those that succeed call on past experience to proactively develop their readiness for inevitable future risks. The measures they take are pre-emptive and continuous. For organizations considering the impacts of a cyber attack, there is much to be learnt from this “ready” mindset. Cyber attacks are a business reality, with 75% of organizations in the UK having experienced one between March 2019 and March 2020[1]. However, the impact of the attack and the damage caused, can be mitigated. Depending on an organization’s incident response (IR) and crisis management efforts, an attack with the potential to disrupt business continuity may have little impact. This can be achieved by anticipating incidents and developing a response around specific threats, their likelihood, and the motives of the attacker. It is never the result of a defensive arsenal of technology on its own. ##### “There are organizations that are​ spending a ton on cyber security and they have very bad risk postures. There are others that are not spending very much but they have very good risk postures. The bottom line is it’s about their level of readiness.”​ – Paul Proctor, VP, Gartner 1 https://www.gov.uk/government/publications/cyber-security-breaches-survey-2020/cyber-security-breaches-survey-2020 ----- ## PROACTIVE VS REACTIVE RESPONSE Readiness—in the context of cyber incidents—can be understood as a state of being and working to continually be prepared for compromise, such that business continuity can be maintained throughout an attack. It is both the measurement of an organization’s current security posture and a set of actions. The latter, when improved, can increase overall organizational resilience. Readiness can be aligned with the “preparation” phase in the IR lifecycle (see fig.1.), influencing all successive phases. It is broad in scope and designed to begin as far in advance of an incident as possible; this preparation phase may take place months or even years before an incident. Although this preparation phase is fundamental to IR as a practice, it is commonly overlooked. PREPARATION IDENTIFICATION CONTAINMENT ERADICATION RECOVERY LEARNINGS Fig. 1. Incident response lifecycle ## INCIDENT READINESS IN ACTION The following walkthrough is based on real IR engagements and contrasts the same attack being deployed against two near identical organizations in the financial sector. It illustrates the success of response with and without prior readiness activity and welcomes the reader to make their own judgement about the value of incident readiness and how they may apply such a mindset in their own organization. ----- ## SETTING THE SCENE ###### ENTERPRISE-A Enterprise-A has a workforce of 500,000 employees, and operations in several geographic regions. One central group security function exists in addition to local security representatives in each regional office. Regions are responsible for their own security function and detection technology; no monitoring of this technology takes place at the central security group level. Security initiatives include penetration testing for applications, infrastructure, and networks. Some policies and processes for IR exist, but are not widely understood due to a lack of training. ###### ENTERPRISE-B Enterprise-B is identical in profile to Enterprise-A, except that it has recently undertaken an incident readiness project including the following elements: - IR plans and playbooks developed at the regional level - First responder training in place for both group and regional staff - Tabletop incident simulation exercises carried out on a regular basis for each region, with group security playing the role of incident management - Review of the central logging solution that collects and aggregates key logs, with an emphasis on logging strategy and policy for priority assets within group and regional offices - A review of high privilege accounts and the management thereof ----- ## INCIDENT PHASE 1: INITIAL BREACH ###### ENTERPRISE-A DAY 1 The third-party developer responsible for developing and maintaining the organization’s money transfer application contacts a representative from head office—suspicious files have appeared on the application servers of two regional offices. These files were not uploaded by the development team, which is the only team with access to the servers. ###### DAY 10 Group security at head office contacts the region where the suspicious files have been identified. It transpires that the office is receiving numerous client complaints regarding small, unrecognized transactions from their accounts. The application owner is currently investigating the matter but is unsure of the cause at present. ###### DAY 21 The developers are instructed by group security to delete the suspicious files and report any further unusual file activity on other regional servers. Group security investigates the unrecognized transactions in the transaction logs and discovers a possible authentication flaw, which allows authentication keys to be reused. The team issues a patch and distributes it to all regions where the application is hosted. ----- ###### ENTERPRISE-B DAY 1 Upon receiving notification from the development team about a suspicious file on the application servers for its money transfer application, a ticket is raised and assigned as a low-risk priority. An analyst from the group security team calls the developer who raised the notification. They ask them to share their screen to enable a two-way discussion about the suspicious nature of the file. The servers are Linux, so the developer logs on via Secure Shell (SSH) and prints the file contents to the shared screen. The analyst takes note of the file contents: ``` #.exe” has been submitted to the malware analysis team for reverse engineering to scrutinize its structure and purpose. Preliminary analysis indicates that some extracted strings have keylogging capability and that the executable is a custom piece of malware **DAY 193** As a result of these findings, group security meets to assess the threat and plan remediation. The consequent actions are informed by both the findings and are in line with its domain compromise playbook: - Exchange administrators from each region are instructed to search for instances of the phishing email in their regional user base. Those regions with a proxy are instructed to search the proxy logs for the phishing URL - A list of hashes for the files identified in the attacker’s tool folder is sent to each regional office. It comes with instruction for the regional security teams to use their anti-virus solution to search for instances of these on their estates - The regional administrator for the Linux environment hosting the money transfer application is instructed to search for instances of logons using the compromised application manager’s credentials. ----- **DAY 198** While no other instances of the phishing email are found, the malware hash search shows hits on six additional servers in the region where the malicious files were first identified. The additional server list includes a domain controller. No anomalous authentication events are identified in the Linux environment. However, it is recognized that limitations in the logging prohibit evidence being collected across the entire period of interest. The administrator increases the log retention policy to 60 days, resets all user credentials, and hardens the firewall rules to ensure that only listed IP addresses can access the Linux servers. A collection script is run on all Linux hosts in the environment, the output of which is shared with the IR provider for analysis. **DAY 201** Following the processes and playbook guidance defined for regional domain compromise, group security obtains executive approval to sever the domain trust between the affected region and head office. The following instructions are sent to the regional security team to eradicate the threat: 1. Restore backups of the affected servers from one month ago[4] 2. Reset all user and administrator passwords via Active Directory (AD) 3. Reset the KRBTGT password as per Microsoft’s best practice guide 4. Identify additional accounts which may have excessive privileges 5. Increase monitoring for all affected assets 6. Monitor for operational impact and report back to group security 7. Schedule a domain hardening assessment with the organization’s cyber security partner By following these processes, Enterprise-B prevents the potential business impact of the attack and successfully contains the incident. It may now evaluate the incident with its IR provider and report back to the business. As part of the lessons learned process defined in each of the organization’s playbooks, the CSIRT conducts a meeting to discuss the information ascertained during the incident. An action plan to address the identified shortcomings is developed during the meeting, and the resulting actions list is assigned to key individuals for completion. 4 The IR provider confirms these are clean, as they predate the servers’ compromise. The executive team know that the backup restoration may cause some business impact and loss of production data. Stakeholders are on standby to deal with any fallout. ----- ## INCIDENT PHASE 3: ATTACKER CASH-OUT ###### ENTERPRISE-A DAY 362 (ATTACKER OBJECTIVE ACHIEVED) Enterprise-A receives an influx of reports from clients who have lost significant sums of money through its money transfer application. These come from multiple regions. The organization’s executive team convenes a crisis management meeting with group security and regional security representatives to investigate and control the situation. An external IR provider is engaged and briefed on the findings so far. The provider requests the following artifacts to support the investigation: - Disk images from the Linux application servers - Transaction logs from the money transfer application - Firewall logs for the Linux environment ###### DAY 368 As most of the application server disk images need to be couriered from the respective regions, limited artifacts are available for analysis. One disk image of a regional application server and some network logs are provided initially. Preliminary findings reveal that malware has been running on the host for over a year. The transaction logs again show that authentication keys have been reused in money transfers, for different accounts, via the API on the application server. The patch issued by group security to prevent the reuse of authentication keys was reversed by the attacker shortly before the bulk of the transactions took place. A timeline of key events from the attack confirms that the provided disk image was not the first in the attack chain. The region where the domain compromise occurred a few months prior is instead determined as the attack’s most likely origin. This conclusion is drawn from the timestamps of the fraudulent transactions and connections on the firewall. Due to the time that has passed since this point, the available historic evidence is limited. ----- **DAY 376** Additional disk images are received for analysis from the regional offices, including the region suspected as the starting point for the attack. **DAY 386** The attacker’s foothold on the Linux servers is confirmed to originate from the corporate Windows environment, with the source of the SSH connection being the application manager’s host. The IR provider requests a disk image of the host, and the laptop is shipped to head office for analysis; the absence of equipment or experience to acquire a disk image prohibits the regional security team to do so independently. This significantly hinders the investigation and causes a substantial delay in the analysis process. No connection exists between the regional application servers. The servers in each region were individually compromised, rather than via lateral movement. Each was compromised via web shells uploaded via the developer’s code maintenance application, followed by the installation of persistent malware. The development team concedes that it could see the web shells being created and wrote a script to delete any files matching the names of said shells. They did not notify the organization, as they believed appropriate mitigation steps were taken by deleting the files. The IR provider identifies variations of the web shells, which were not deleted by the developer’s script. This is the mechanism that allowed the attacker to maintain persistence on the application servers. **DAY 399** The laptop belonging to the application manager is received. It takes the regional team a few hours to find and provide its BitLocker recovery key, after which time the IR provider acquires a disk image and begins analysis. **Day 416** Significant time has passed since the domain compromise, and many laptop artifacts have rolled over or been overwritten. The IR provider highlights that the host was compromised when the user clicked a phishing URL. Custom malware is also running on the host as well as the suspected output from a keylogger. Another investigation is launched to determine if the domain controller is still compromised. ----- ## POST-INCIDENT LEARNINGS Following extensive forensic investigation, remediation of the incident takes several iterations. These cause long periods of operational downtime for Enterprise-A, impacting the business. The investigation finds that the attacker targeted the organization one year prior to their objectives being reached. They maintained persistence on several key assets during this time and systematically gathered data that would later inform and enable their final attack. With the benefit of hindsight, three key detection events are identified. These represent the point at which incident response processes could have been invoked and may ultimately have resulted in a different outcome for Enterprise-A. Enterprise-A’s response approach is illustrated via the following key detection events, the response action taken for each, and the subsequent impact. #### REACTIVE RESPONSE ###### DETECTION EVENT RESPONSE IMPACT **Developer alert:** No investigation; true root Developer instructed to ###### 1 suspicious file on cause not known, servers still delete suspicious files application server vulnerable Review logs, identify Patch stops attackers from **Client alert: strange** authentication being able to transfer funds, ###### 2 transactions on money vulnerability, create and but persistence means they transfer application deploy a patch can disable controls Domain compromise means ###### 3 High risk AV alert on Delete malicious files, that the attacker can gather Windows servers in region reset user credentials important data for over a year to inform final attack Fig. 2. Points of detection, response, and impact within a reactive response approach ----- Following the investigation’s conclusion, critical gaps in Enterprise-A’s readiness posture are identified. At a minimum, the following highlevel readiness measures could reduce the impact of incidents of this nature: - The creation of an IR plan detailing relevant regulations, thresholds, incident severity levels, escalation matrices, and roles and responsibilities to be assumed during a cyber security incident - The development of IR playbooks detailing high-level process flows for different categories of incidents; these would include incident management at the global level and incident categories at the regional level - First responder training for regional representatives to advocate their responsibility for preserving evidence in suitable circumstances. This would also provide them with the skills required to conduct initial analysis from triaged data - Incident response simulations developed and delivered systematically to each region, encouraging buy-in from the regional offices for the incident response plan and playbook We can now consider what a proactive response may have looked like for Enterprise-A had the above elements been in place at the time of the incident: #### PROACTIVE RESPONSE ###### PREPARATION DETECTION EVENT RESPONSE IMPACT **First Responder training:** **Developer alert:** Snapshot virtual server hard **1** staff are aware of how and suspicious file on drives and send for root when to preserve artifacts application server cause analysis **Incident response** **policy detailing roles** Root cause analysis (RCA), **Client alert: strange** and responsibilities, review logs, identify **2** transactions on money business owners, change authentication vulnerability, transfer application management, and patching create and deploy a patch process Capture forensic artifacts, **Playbooks exist for host** **High risk AV alert on** root cause analysis, isolate **3** compromise and domain Windows servers and rebuilt according to and credential theft avoided recovery in one region playbooks Fig. 2. Points of preparation, detection, response, and impact within a proactive response approach ----- Enterprises that prioritize readiness are better equipped to raise the barriers between advanced persistent threats (APTs) and their goals, prior to a compromise. Such threats are systematic and require sufficient resource to succeed. By preserving key data and developing learnings early in the kill-chain, the resource required by the attacker diminishes the value of its persistence. Instead of relying on prevention, and responding reactively when it fails, this approach proactively deters threats by making an organization an uneconomical target. ----- ## SUMMARY AND CONCLUSION An organization’s ability to maintain business continuity through a cyber incident, and recover any losses in the aftermath, is dependent on its governance of the IR lifecycle. This includes the preparation phase where controlled and strategic actions can be taken, based on learnings from tabletop exercises, capability-specific assessments, and real attacks. Examples like the one in this paper show that preparation reaches beyond the deployment, configuration, and testing of tooling. Whilst suitable tooling is essential, it cannot provide incident readiness on its own. Instead, its inclusion alongside strategic management of people and processes—via continually-improved IR plans, playbooks, and low-level procedures—make for an effective response Operational resilience during a real incident is the result of skilled risk management and the technical ability of the CSIRT, in addition to the performance of tooling. For the internal CSIRT managing the frontline during a compromise, proactive response can help circumvent burnout and improve channels of communication with the executive team. Such crossdepartmental collaboration is a step towards security becoming a business priority. Incidents are indeed a business consideration not a pure technical one. The scrutiny of customers, partners, regulators, and the media that results from an incident forms the case to develop security teams equipped to defend against modern threat actors—both constantly evolving and persistent in nature. The value of readiness investment can be represented to senior stakeholders in this context as a route to increased credibility, continuity, and growth. ----- ## REFERENCES **1. Cyber security breaches survey 2020** [https://www.gov.uk/government/publications/cyber-security-breaches-](https://www.gov.uk/government/publications/cyber-security-breaches-survey-2020/cyber-security-breaches-survey-2020) [survey-2020/cyber-security-breaches-survey-2020](https://www.gov.uk/government/publications/cyber-security-breaches-survey-2020/cyber-security-breaches-survey-2020) **2. The F-Secure guide to rainbow teaming: blue team | building resilience** **through response process development and simulation** [https://www.f-secure.com/content/dam/f-secure/en/consulting/our-thinking/](https://www.f-secure.com/content/dam/f-secure/en/consulting/our-thinking/collaterals/digital/F-Secure-Consulting-Blue-Team-paper-2020.pdf) [collaterals/digital/F-Secure-Consulting-Blue-Team-paper-2020.pdf](https://www.f-secure.com/content/dam/f-secure/en/consulting/our-thinking/collaterals/digital/F-Secure-Consulting-Blue-Team-paper-2020.pdf) **3. Mimikatz** [https://github.com/gentilkiwi/mimikatz](https://github.com/gentilkiwi/mimikatz) **4. Enterprise-B is able to restore backups of the affected servers from one** **month prior to day 201 of the attack** The IR provider confirms these are clean, as they predate the servers’ compromise. The executive team know that the backup restoration may cause some business impact and loss of production data. Stakeholders are on standby to deal with any fallout. 1 https://www.gov.uk/government/publications/cyber-security-breaches-survey-2020/cyber-security-breaches-survey-2020 ----- ###### F-SECURE INCIDENT RESPONSE TEAM **US** +1 (917) 341-2116 **UK** +44 (0) 333 311 0014 **Finland** +358 9 4245 0223 **Denmark** +45 89 88 21 10 **South Africa** +27 (10) 500-1921 **Singapore** +65 3159 1795 ###### F-SECURE INCIDENT RESPONSE TEAM www.f-secure.com/consulting **US** +1 (917) 341-2116 **UK** +44 (0) 333 311 0014 **Finland** +358 9 4245 0223 **Denmark** +45 89 88 21 10 **South Africa** +27 (10) 500-1921 **Singapore** +65 3159 1795 www.f-secure.com/consulting -----