###### ICS CYBERSECURITY YEAR IN REVIEW 2020 ----- ###### Contents Introduction..............................................................................................................3 Key Highlights......................................................................................................... 4 In The Headlines.......................................................................................................5 Section One: ICS Threat Landscape........................................................................ 8 Key Updates on Existing Activity Groups.............................................................................. 10 2020’s New Activity Groups........................................................................................................11 STIBNITE..................................................................................................................................... 12 TALONITE................................................................................................................................... 13 KAMACITE..................................................................................................................................14 VANADINITE.............................................................................................................................. 15 Most Common TTPs Across All Industries.............................................................................16 Section Two: ICS Vulnerabilities............................................................................ 17 2020 Vulnerability Details...........................................................................................................19 2020 Vulnerability Severity......................................................................................................... 21 Actionable Guidance Missing in Most 2020 Advisories..................................................... 23 Severity Ratings of Vulnerabilities Remain Error-Prone.................................................... 25 Flaws in TCP/IP Stacks.................................................................................................................27 Vulnerabilities in VPN Appliances Facilitating Remote Work..........................................28 Section Three: Lessons Learned From the Front Lines........................................ 29 Visibility............................................................................................................................................ 31 Segmentation and Connections............................................................................................... 33 Cyber Readiness.............................................................................................................................37 Dragos Red Team......................................................................................................................... 39 Recommendations................................................................................................. 40 Increase OT Network Visibility................................................................................................. 40 Identify and Prioritize Crown Jewels........................................................................................41 Boost Incident Response Capabilities..................................................................................... 42 Validate Network Segmentation.............................................................................................. 43 Secure Credential Management...............................................................................................44 ----- ###### INTRODUCTION The Dragos Year in Review report is an annual analysis of Industrial Control System (ICS)/Operational Technology (OT) focused cyber threats, vulnerabilities, assessments, and incident response insights.[1] The ICS/OT community has long suffered from a lack of public insights into these types of problem areas to have a meaningful discussion on how to address the issues. It is the Dragos team’s goal to share the observations and lessons learned with the industrial community for data-driven analysis and recommendations. In 2020, the industrial community performed amazing feats to keep civilization running under extremely challenging circumstances with the global pandemic. Infrastructure providers kept key services and goods available including electric power, manufactured goods, water, oil and gas, mining, chemical, rail, and transport while many faced hardships globally. As a result of these efforts, organizations shifted in how they conducted business to include an increasingly connected industrial environment. This is a trend that has existed for many years, even while many organizations still believed they had highly segmented or even air-gapped ICS networks. The risk to ICS is not born from an IT and OT convergence, but instead from a convergence of an increasingly ICS-aware and capable threat landscape with the digital transformation and hyperconnectivity of the industrial community. This report captures how some of the community is performing and progressing, and areas of improvement that will be needed to continue to provide safe and reliable operations. ###### INTRODUCTION **1** The terms “ICS” and “OT” will be used interchangeably for the purpose of this report. These terms are used differently in different communities. ----- ###### KEY HIGHLIGHTS Four new threat groups The abuse of valid with the assessed accounts was the motivation of targeting **number one** ICS/OT were discovered, **technique used** accounting for a 36% by named threats. **increase in known** groups. ###### SERVICE ENGAGEMENT VULNERABILITY ADVISORY FINDINGS FINDINGS **43% of ICS** **90% of service** **vulnerability** **engagements included** **advisories** contained a finding around lack errors that would make of visibility across OT it difficult to prioritize networks. mitigations. **64% of advisories** **54% of service** **that had no patch** **engagements included** also had no practical a finding about shared mitigation advice credentials in OT provided by the vendor. systems. **61% of advisories** **88% of service** **that had a patch** did **engagements** not have any alternate included a finding mitigation advice about improper provided by the vendor network segmentation. except for applying the patch, which in many industrial organizations can be difficult or significantly delayed. ----- ###### IN THE HEADLINES ###### IN THE HEADLINES MITRE introduced ATT&CK Ripple20, AMNESIA:33 for ICS in 2020 to codify and communicate the unique threat behaviors, or Tactics, Techniques, Third-party code integration can pose risks to and Procedures (TTPs), that industrial operations. Many vendors do not track ICS adversaries use against OT third-party code libraries, and therefore cannot targets. This independent and accurately inform customers if their products community-sourced framework are impacted. Vendors are beholden to software provides a common lexicon manufacturers to release fixes for vulnerabilities for categorizing ICS-specific that may impact thousands of products. For TTPs to support reporting and example, security researchers disclosed multiple further analysis. Dragos uses vulnerabilities in TCP/IP software libraries called the framework internally and Ripple20[2] and AMNESIA:33[3] that potentially continues to contribute to this impacted many ICS vendors. Example ICS devices program and community resource. impacted include Programmable Logic Controllers The Year in Review report (PLCs), Serial to Ethernet Converters, Protocol leverages information from MITRE Converters, Remote Terminal Units (RTUs), digital ATT&CK about observed activity protective relays, and some managed network in ICS environments to help switches and routers. Most of the devices impacted defenders mitigate threats to their by the vulnerabilities were not accurately identified organizations. and did not have advisories released due to the difficulty in understanding third-party code adoption. Security design flaws in the impacted devices may make this collection of flaws less relevant to adversaries, but the disclosure highlights supply chain risks and complexity. EKANS represents a specific threat to ICS because of its incorporation of potential process and operational disruption features. This ransomware is capable of stopping ICS-related Windows processes before initiating encryption. EKANS activity could produce an unstable or physically disruptive situation by abruptly ending an operationally significant process. The Dragos Intelligence team assesses EKANS is related to a previous strain of ransomware called MEGACORTEX. Throughout 2020, Dragos identified new EKANS activity targeting multiple verticals including electric, oil and gas, medical, pharmaceutical manufacturing, and automotive. Public targets included Fresenius Kabi, a pharmaceutical division of the European company Fresenius Group; global manufacturer Honda; and Italian energy company Enel. ----- ###### IN THE HEADLINES Multiple ICS entities were impacted by a SolarWinds directly, and organizations were massive supply chain compromise first unaware they had it in their environments. revealed in December 2020.[4] Adversaries More commonly though, many integrators, compromised SolarWinds Orion business support contracts, and ICS suppliers software to distribute malware. The use software, including SolarWinds, as adversary had unfettered access for more white-labeled solutions. This means the than 14 months and is thought to now product is in place but under a different have access to other supply chain access name. Suppliers may use the software points throughout the community. themselves on behalf of the client where the Identifying SolarWinds in ICS compromised SolarWinds software was not environments was challenging. To in the end-users ICS network. The software respond appropriately, facilities required may have been present in the supplier or accurate asset lists, software version integrator’s network and used in the end- information, and network monitoring user’s networks across direct connections to identity post-exploitation activity. or maintenance links. SolarWinds often Dragos investigated numerous confirmed has access directly to the control level compromises. The investigations reverted in ICS networks which would allow an to limited host-based analysis and could adversary to not only have access to these only capture days or weeks in analysis. In environments, but direct control of them. an ICS network where not all endpoints Many organizations that did not believe they can have robust host logging and in were impacted were compromised directly compromises where adversaries leverage or accessed from compromised networks the network extensively, it is preferred to due to third-parties. have network traffic analysis and logging. Dragos is aware of at least two global Given the current lack of visibility in ICS Original Equipment Manufacturers industrial networks the assessment of (OEMs) that were using the compromised SolarWinds’ compromise impact is likely SolarWinds software across maintenance to not be fully understood for years. links into ICS networks, including where In some cases plant personnel purchased there was turbine control software.[5] **4** https://www.dragos.com/blog/industry-news/responding-to-solarwinds-compromise-in-industrial-environments/; **[5 ]Dragos attempted to inform both OEMs and eventually leveraged government organizations to ensure that the risk was understood.** ----- ###### IN THE HEADLINES Prior to 2020, OT-targeting adversaries exploited remote services to compromise ICS environments. This technique gained popularity as the COVID-19 pandemic spread globally. Companies began requiring some workers to remain at home in 2020, even those working in industrial environments. Multiple ICS-targeting threat groups historically targeted remote access technologies or logon infrastructure including PARISITE, MAGNALLIUM, ALLANITE, and XENOTIME. This includes compromise of Remote Desktop Protocol (RDP) and Virtual Private Network (VPN) assets. VPN equipment is a common access point for OEMs and integrators to gain access to operations environments. Although many organizations focus on prevention and segmentation, obtaining access through a compromised VPN may allow an adversary into an ICS network with limited detection capabilities. For example, in 2019 the Dragos team reported on the threat group PARISITE noting it was the access operations team for the MAGNALLIUM group and explicitly targeted OT boundary equipment, such as VPNs, for the likely purpose of gaining access to ICS networks. ----- ###### SECTION ONE ICS Threat Landscape ----- ###### INTRODUCTION Cyber risk to industrial sectors has grown and accelerated dramatically, led by ransomware impacting industrial processes, intrusions enabling information gathering and process information theft, and new activity from adversaries targeting ICS. Dragos emphasizes the importance of understanding how adversaries steal information and gain access to better prepare for adversary behavior in the future. Adversaries often build programs and campaigns slowly over time, with later campaigns often being more successful and disruptive due to previous efforts. Some threats tracked by Dragos may proliferate into disruptive and destructive capabilities later, though no such activity is observed at this time. For example, the team would track a threat that was explicitly targeting electric companies with theming toward engineers, engineering projects, or electric operations, though there may be limited or no visibility to confirm if the adversary gained access to ICS networks. The team would not track a threat if it was simply trying to gain access to an electric company. The fundamental assessment of threats tracked by Dragos is that they are explicitly trying to gain access to ICS networks and operations or are successful in achieving access. These types of events, where adversaries gain access to ICS networks but do not have the intention of currently disrupting them, are much more common than is publicly reported. The threats are learning ICS. Although not every compromise will relate to an impact today, many may inform the attacks of the future. Dragos tracks 15 threat Activity Groups, or threat groups,[6] with four of the groups discovered in 2020. Threats are growing at a rate three times faster than they are going dormant.[7] This is likely due to the increased investment made by adversaries in targeting ICS over the last five to 10 years, and whose investment will continue to accelerate the ICS threat environment. ###### INTRODUCTION **6** Organizations track threats in unique ways. At Dragos, analysts utilize the Diamond Model for Intrusion Analysis, which is a common and industry accepted methodology to cluster intrusions into groups representing teams of adversaries that operate in similar ways. The Diamond Model is an open and well-documented framework that allows for transparency in how this analysis is conducted. https://www.activeresponse.org/wp-content/uploads/2013/07/ diamond.pdf **7 Every year feels like the headlines are, “Threats are Getting Worse,” or “Risk** is Increasing.” A useful way to track the evolution of the threat landscape is to determine how many new threat groups are identified compared to how many are going dormant. Groups that are not observed performing new intrusions within a 12-month period are moved to a dormant phase. Typically the growth rate would be less than two groups being created for every group that goes dormant. **MAJOR ICS THREAT TRENDS IN 2020** - ICS THREAT ACTIVITY GROUPS INCREASE SIGNIFICANTLY - PHISHING CONTINUES TO ENABLE ICS INTRUSIONS - REMOTE ACCESS DIRECTLY TO ICS LEVERAGED OFTEN BY THREATS - THE BEGINNING OF RANSOMWARE SPECIFICALLY TARGETING ICS - SUPPLY CHAIN CONCERNS AMPLIFIED ----- ###### Key Updates on Existing Activity Groups Throughout 2020, the 11 Activity Groups identified prior to 2020 remained active against industrial organizations. While already covered in previous Year in Review reports, the following key activities occurred in 2020 that are worth noting: In March, PARISITE leveraged Citrix vulnerability CVE-2019-19781 in intrusions targeting North American electric and oil and gas entities. This was concerning given that PARISITE conducts initial access operations potentially enabling future disruptive operations associated with MAGNALLIUM. This exemplifies the critical need to track any groups interested in ICS through IT networks. In one example, Dragos responded to an incident where PARISITE was identified internal to the organization before MAGNALLIUM could act. Intelligence-informed decisions can and have helped organizations prioritize efforts successfully. In April, new DTrack malware emerged with the ability to communicate with Fujitsu Systemwalker management software utilized in distributed computing and data center management operations. Dragos associated this activity with the energy- targeting group WASSONITE. Interaction with this type of software can significantly impact data center and computational environments resulting in potential ICS or broader operational impacts. **ALLANITE and DYMALLOY remain critical threats to infrastructure operations,** especially in Europe and North America. In May, Dragos observed ALLANITE conducting credential harvesting via watering hole attacks. This was followed by use of captured credentials and built-in system tools to launch intrusions into the German electric system, and potentially water and wastewater sectors. ALLANITE and DYMALLOY continued to target multiple United States (U.S.) industrial entities from September through October 2020. Operations included use of ZeroLogon to further intrusions into victim networks. CHRYSENE showed continued activity with further malware development and used new tools to infiltrate ICS networks in the Middle East for intelligence gathering purposes. ----- ###### 2020’s New Activity Groups STIBNITE TALONITE KAMACITE VANADINITE ----- ###### STIBNITE **STIBNITE specifically targets wind turbine companies** that generate electric power in Azerbaijan. Based on current collection efforts, the activity appears confined exclusively to Azerbaijan. There is ongoing kinetic conflict in the region between Azerbaijan and Armenia due to rights to disputed territory. Historically when there is regional conflict between states, there tends to be targeting of critical infrastructure, including electric operations. There is only a loose correlation between the conflict and STIBNITE operations, and the Dragos team is not making an assessment on who may be responsible for the targeting.[8] Given the specific targeting and the regional conflict, it is a situation and threat group worth watching closely. STIBNITE’s victims share unique technology with wind farms in Ukraine. One possibility for the specific victim targeting is that adversaries targeted the supplier and maintainer for the wind farm itself. The supplier, operator, and maintainer are all based in Ukraine. STIBNITE used shared Command and Control (C2) infrastructure between multiple intrusions in late 2020 and updated its malware capabilities to avoid detection after public reports on its activity were released. STIBNITE uses PoetRAT remote access malware in its intrusion operations to gather information, take screenshots, transfer files, and execute commands on victim systems. STIBNITE gains initial access via credential theft websites spoofing Azerbaijan government organizations and phishing campaigns using variants of malicious Microsoft Office documents. STIBNITE also used information related to the global COVID-19 pandemic for malicious document themes. **8** https://www.wsj.com/articles/armenia-azerbaijan-conflict-11601325097 ###### Target Geography Azerbaijan Victimology Malware PoetRAT ###### Azerbaijan square ICS square Enterprise ###### STIBNITE TTPs from MITRE ATT&CK **INITIAL ACCESS** ###### PoetRAT **T0865** **T0817** Spearphishing Drive-by Compromise **PERSISTENCE** **T0859** **T1050** Valid Accounts New Service **LATERAL MOVEMENT** **T0859** Valid Accounts **COMMAND & CONTROL** **T0869** Standard Application Layer Protocol ----- ###### Target Geography ###### TALONITE **TALONITE‘s operations focus on near exclusive** interest in initial access compromises in the U.S. electric sector. The group uses phishing techniques to deliver either malicious documents or executables. TALONITE uses two custom malware families known as LookBack and FlowCloud for information gathering operations.[9] TALONITE’s phishing campaigns utilize electric and power grid engineering-specific themes and concepts, indicating an intent to gain a foothold within energy sector entities. Such access could facilitate gathering host and identity information, collecting sensitive operational data, or mapping the enterprise environment to identify points of contact with ICS. The identified infrastructure and phishing emails spoofed the National Council of Examiners for Engineering and Surveying (NCEES), North American Electric Reliability Corporation (NERC), the American Society of Civil Engineers (ASCE), and Global Energy Certification (GEC). TALONITE employs malware using legitimate binaries maliciously, or modifies such binaries to include additional functionality. For example, LookBack malware contains persistence mechanisms that add two Windows registry keys to execute legitimate but modified files when the infected user next logs in. FlowCloud launches a renamed copy of the legitimate HTML Help Workshop (hhw.exe) utility from Microsoft. The group uses a combination of owned and compromised network infrastructure. **9** For more public information on these malware families reference Proofpoint’s reporting here: https://www.proofpoint.com/us/threat-insight/post/lookback-malware-targets-united-states-utilities-sectorphishing-attacks ###### North America Victimology ###### or reused through the ###### Appears to be shared square ICS square Enterprise ###### Victimology ###### Infrastructure Appears to be shared or reused through the overlapping LookBack and ###### overlapping LookBack and ###### TALONITE TTPs from MITRE ATT&CK **INITIAL ACCESS** ###### FlowCloud campaigns **T0865** Spearphishing Attachment **EXECUTION** **T1106** Native API **PERSISTENCE** **T1078** Valid Accounts **DISCOVERY** **T1046** Network Scanning Service **LATERAL MOVEMENT** **T1078** Valid Accounts **COLLECTION** **T1113** Screen Capture **T1005** Data from Local System **COMMAND & CONTROL** **T1090** Proxy ----- ###### KAMACITE In 2020, KAMACITE targeted U.S. energy companies by leveraging stolen credentials or brute force logons of remote services to access victim networks. Dragos tracks KAMACITE as a significant threat to industrial operations because of strong overlaps between KAMACITE and the Sandworm group, which enabled a disruptive cyberattack to electric operations in Ukraine. Sandworm is a group tracked by FireEye analysts covering a wide variety of ICS and non-ICS specific targeting. KAMACITE represents the specific, ICS-targeting access enablement component observed by Dragos. KAMACITE also represents a long-running set of related behaviors targeting critical infrastructure and industrial verticals. KAMACITE is a unique group due to new intrusions and evolution in tradecraft distinct from original Sandworm activity.[10] Dragos assesses KAMACITE is an access-enablement team that operates to support other teams conducting disruptive and destructive effects.[11] Previously, Dragos identified ELECTRUM as the group responsible for the Ukraine 2016 electric transmission substation cyberattack. The important distinction is that in Dragos’s analysis, KAMACITE conducted the access operations, enabling ELECTRUM to create and use CRASHOVERRIDE malware to carry out the attack. Dragos determined this based on behavioral differentiations from ELECTRUM activity and earlier access operations demonstrating two distinct groups of activity. In 2020, KAMACITE was observed performing reconnaissance against numerous energy companies in the U.S. Following its reconnaissance, it attempted to take advantage of webmail of those companies and cloudbased logon services including Microsoft Active Directory (AD) and Office 365 services. In addition to this activity, Dragos verified Server Message Block (SMB) (TCP 445) connections from KAMACITE infrastructure to victims that could indicate some successful access attempts. The precise nature of this connectivity is unknown. **10** It is common for the information security community to look at threat group names as names, when in reality they are definitions. Sandworm is tracked by FireEye. While links can be seen to others’ groups, the definition, collection efforts, and analysis of the FireEye analysts are distinct from those at Dragos and proprietary. Dragos decided to start tracking KAMACITE, associated with a link to Sandworm, upon new activity and unique collection emerging in 2020. [11] Dragos has unique visibility and tracks intrusions that are still active as of this past year. Dragos tracks KAMACITE separately f S d ti it D i t tt ti t S d b t i l i i th ICS ti it ###### North America ###### ELECTRUM square ICS square E t i ###### Target Geography North America Europe ###### Europe ###### Victimology ###### Major Incidents BE2, BE3, CRASHOVERRIDE Links SANDWORM, ELECTRUM ###### CRASHOVERRIDE ###### BE2, BE3, ###### KAMACITE TTPs from MITRE ATT&CK ###### SANDWORM, **INITIAL ACCESS** **T1190** **T0866** Exploit Public Facing Application Exploitation of Remote Services **EXECUTION** **T0853** **T0807** Scripting Command-Line Interface **PERSISTENCE** **T1078** Valid Accounts **DISCOVERY** **T0840** **T0846** Network Connection Enumeration Remote System Discovery **LATERAL MOVEMENT** **T1078** **T0866** Valid Accounts Exploitation of Remote Services **COMMAND & CONTROL** **T0885** **T0884** Commonly Used Ports Connection Proxies ----- ###### Target Geography ###### VANADINITE **VANADINITE conducted various initial access** operations targeting industrial entities across the energy, manufacturing, and transportation sectors in North America, Europe, Australia, and Asia. Dragos assesses this group’s activity is focused on information gathering operations, including specifically ICS compromise and data theft. Obtaining documents and intellectual property relating to ICS processes, function, and design could enable this group’s sponsor entity to develop ICS capabilities. VANADINITE’s activity supports long-term strategic advances rather than disruptive or destructive effects. Dragos assesses with low confidence VANADINITE is responsible for the ColdLock ransomware attack that targeted Taiwanese state-owned ICS companies and caused indirect disruption to operations. The objective of this attack — whether financial gain or disruptive messaging — is unclear. Dragos consistently observed a pattern in VANADINITE behavior, including the targeting of recently disclosed “n+1” vulnerabilities in a range of networking and gateway devices impacting remote access services like VPNs. Asset owners and operators should treat vulnerabilities in externalfacing network appliances as a serious issue. Dragos observed multiple entities, including PARISITE, increasingly adopt this methodology. These remote access technologies often directly enable access to ICS networks bypassing enterprise networks and are commonly used by integrators and ICS OEMs. ###### Winnti, LEAD ###### Australia **PERSISTENCE** ###### Victimology ###### North America, Europe, Asia, Australia Links Winnti, LEAD ###### Europe, Asia, ###### VANADINITE TTPs from MITRE ATT&CK **INITIAL ACCESS** ###### North America, **T1190** **T1133** **T1110** **T1555** **T1003** **T1111** Exploit Public Facing Infrastructure External Remote Services Brute Force Credentials from Password Stores Operating System Credential Dumping Two-Factor Authentication Interception **EXECUTION** **T1059** **T1047** Command and Scripting Interpreter Windows Management Instrumentation **T1133** **T1078** External Remote Services Valid Accounts **LATERAL MOVEMENT** **T1078** Valid Accounts square ICS square Enterprise ----- ###### Most Common TTPs Across All Industries The following chart demonstrates how many of the 15 Activity Groups Dragos tracks leverage a specific MITRE ATT&CK for ICS TTP. As shown below, Valid Accounts usage is the most common TTP among the groups. Top Activity Group 5 TTPs 13 12 11 10 9 8 7 6 5 4 3 2 1 T0859 T0865 T0853 T0869 T0862 ----- ###### SECTION TWO ICS Vulnerabilities ----- ###### INTRODUCTION Dragos researchers analyzed 703 ICS/OT vulnerabilities in 2020. For each vulnerability, Dragos independently assesses and confirms – and often corrects – the advisories describing flaws in firmware or software. The vulnerabilities Dragos analyzes are identified by independent security researchers, the Department of Homeland Security’s ICS-Computer Emergency Readiness Team (CERT), vendors, and Dragos analysts. Dragos analyzed 29 percent more vulnerabilities in 2020 than 2019, demonstrating a rise in publicly known flaws in systems supporting industrial operations. Of individually reviewed vulnerabilities, 33 percent contained errors in the Common Vulnerability Scoring System (CVSS) score, potentially impacting patching decisions made by asset owners and operators. Over one-third of vulnerabilities could cause a loss of view and control if exploited by an adversary. Public Proof of Concepts (POCs) were available for 30 vulnerabilities reviewed by Dragos, meaning available resources indicated specifically how an adversary could operationalize the flaw. ###### INTRODUCTION ----- ###### 2020 Vulnerability Details Twenty-three percent of **Advisories Applied to Products** vulnerabilities Dragos analyzed **Bordering the Enterprise** applied to products bordering the enterprise. This can include networking communication equipment, VPNs, data historians, or ###### 23% firewalls commonly deployed in ICS ###### 21% networks. This number is up from 21 percent in 2019. ICS-targeting adversaries, including VANADINITE and PARISITE, have historically leveraged such vulnerabilities for initial access to target environments and pose a risk to industrial operators. These vulnerabilities are of particular interest, as they can provide immediate access to the ICS networks bypassing enterprise security controls. **2019** **2020** ###### Vulnerabilities Deep Within Most vulnerabilities resided deep within the ICS Networks ICS network, meaning they apply to equipment on Levels 0 to 3 of the Purdue Model.[12] This includes engineering workstations, PLCs, ###### 2019 sensors, and industrial controllers. These ###### 2020 vulnerabilities require access to a control system network to exploit, offering some mitigation for organizations provided they implement proper network segmentation. With the increasing connectivity in organizations, this security control is diminishing in value and should be enhanced with efforts such as network monitoring, and where possible, Multi-Factor Authentication (MFA) for remote sessions. **12** https://en.wikipedia.org/wiki/Purdue_Enterprise_Reference_Architecture ----- ###### Impact of Disclosed Flaws 5% square 2019 Loss of View Only square 2020 1% 2% Loss of Control Only 1% 50% Both Loss of View and Loss of Control 36% 43% Neither Loss of View nor Loss of Control 62% The percentage of disclosed flaws that could be used to cause a loss of view and loss of control in ICS systems decreased significantly from 50 percent in 2019 to 36 percent in 2020. This decrease is likely attributed to the increase in identified vulnerabilities bordering the enterprise that do not have direct operational impacts. ----- ###### 2020 Vulnerability Severity In addition to validating the CVSS, Dragos categorizes vulnerabilities based on severity. The following taxonomy is used: Immediate Action A far-reaching threat or vulnerability calling for action broadly across at least one industry. Limited Threat A limited threat, risk, or vulnerability requiring an applicability assessment before taking action. Possible Threat Threat scenarios, research, and vulnerabilities relating to operations but not requiring direct/immediate action. No Action Required Items of interest but likely requiring no action except in unique threat models. Hype A story or vulnerability receiving coverage but not yet worth the attention of operators. ----- In examining individual vulnerabilities, Dragos also categorizes them based on the **“Now, Next, Never” system, a malleable framework developed by CERT/Coordination** Center (CC) to help asset owners and operators identify vulnerabilities and prioritize patching. The framework is not a one-size-fits-all solution for patch management. When combined with consequence-driven threat modeling, it can help OT security practitioners determine when and if to fix flaws in industrial control equipment. The “Now” flaws require immediate action. These flaws include critical vulnerabilities such as perimeter-facing and network exploitable vulnerabilities, and other vulnerabilities that should be addressed as soon as practicable. Limited Threat vulnerabilities fall into the “Next” category. These might be network exploitable but are present deeper in the network and require more work, access, and knowledge for an adversary to exploit or impact OT processes. If an operations network does not have proper segmentation or is accessible from the internet, asset owners and operators should consider the “Next” vulnerabilities a greater risk. In most cases, these vulnerabilities can be mitigated simply by updating firewall rules. It is important that customers conduct a firewall rule audit on a regular basis and justify every allow rule. In 2020, close to two-thirds of vulnerabilities assessed by Dragos were considered “Next,” a Limited Threat. Low vulnerabilities pose a possible threat but rarely require action in vulnerability prioritization. They can be considered “Never” vulnerabilities. It is more beneficial for an organization to monitor its environment for signs of exploitation rather than to take devices and services offline to patch, or take appropriate mitigation measures. Although considered “Never” vulnerabilities, Dragos analysts do not recommend ignoring them entirely if time and resources permit. The reality is patching in ICS is more difficult than in most enterprise IT networks, and the value presented from patching these vulnerabilities is minimal. Asset owners and operators should conduct risk assessments to determine if it is safe to continue operations without addressing the identified vulnerabilities. ###### 7% 58% 35% ###### 7% 63% 30% 2019 2020 square Now – Immediate Action square Next – Limited Threat square Never – Possible Threat/No Action ----- ###### Actionable Guidance Missing in Most 2020 Advisories An increase in the reported number of ICS vulnerabilities overall coincided with an increase in vendors providing patches alongside publicly disclosed flaws. Twenty- two percent of advisories did not have a patch when announced, down from 26 percent year-over-year. Of those, more than two-thirds did not contain practical mitigation advice. Frequently, vendors will not provide advice to asset owners and operators if they are unable to patch the identified vulnerability. Dragos identified 61 percent of advisories contained a patch to fix the vulnerability, but no alternative mitigation if patching was not an option. ###### Advisories with no Patch when Announced ###### Advisories with no Patch when Announced ###### 26% 22% OF THOSE OF THOSE Advisories that Advisories that had no Practical had no Practical 2019 2020 Mitigation Advice Mitigation Advice 76% 64% 74% 78% #### 26+74+ OF THOSE 22+78+ OF THOSE ###### had a patch had a patch Advisories that Advisories that had no Alternate had no Alternate Mitigation Mitigation 55% 61% |26+74+B 6%|Col2| |---|---| |+74+6 %|| ||| ||| |22+78+B 2%|Col2| |---|---| |+78+2 2%|| ||| ||| ###### 26% 2019 ----- ###### Dragos provides customers with insight into managing risks about disclosed ICS vulnerabilities beyond what is included in advisories by the vendor. In 2020, Dragos provided additional mitigation advice for 78 percent of advisories that did not provide this information. Mitigation Advice: Advisories Without Mitigation Advice for which Dragos Provided a Mitigation 80% 70% 78% 77% 2020 2019 60% 50% 40% 30% 20% 10% ----- ###### Severity Ratings of Vulnerabilities Remain Error-Prone In addition to a lack of actionable information from most ICS-related vulnerability advisories in 2020, many advisories and individual vulnerabilities contained errors that could inadvertently mislead practitioners who use CVSS scores to triage for mitigation or patching. ###### By Advisory Advisories with Incorrect Data ###### 43% 2020 ###### By CVE (Error Rate) Individual CVEs Contained Errors ###### 30% 2019 ###### 33% 2020 ###### 19% 2019 ###### The Dragos assessment of 2020 ICS-related vulnerabilities shows that among advisories and vulnerabilities containing errors, most of them understated severity levels. Seventy-three percent of corrected advisories were more severe in the context of an operational environment than the public advisory indicated. ----- ###### By Advisory (Error Rate) — Rate was consistent from 2019 to 2020 Dragos Found to be Dragos Found to be LESS Severe than MORE Severe than Public Advisory Public Advisory 26% 73% 1% Identical Score but Different Exploitation Vector # [7]By CVE [+][1][+][26+][3] ###### CVE where Corrected Severity Score was HIGHER than Reported 9% 24% CVE where Corrected Severity Score was LOWER than Reported ----- ###### Flaws in TCP/IP Stacks Researchers identified several third-party vulnerabilities in the software supply chain that impacted ICS systems. Most notable were Ripple20 and Amnesia:33, commercial names for vulnerabilities in third-party Internet Protocol (IP) stacks. These third-party stacks are used in many embedded products, including some used in industrial products and industrial-supporting Information Technology (IT) systems. Example ICS devices impacted include PLCs, Serial to Ethernet Converters, Protocol Converters, Remote Terminal Units (RTUs), digital protective relays, and some managed network switches and routers. Mass exploitation of these vulnerabilities, like the similar URGENT/11 vulnerabilities reported in 2019, is rare. No organizations have provided evidence of public exploitation. Devices that use third-party protocol stacks are often deeply embedded systems. Exploitation requires understanding the Central Processing Unit (CPU) architecture, memory layout, and hardware connections of the vulnerable device. Developing a working exploit requires a deep understanding of embedded systems. Once an exploit is developed, it may not even function on the same product if, for example, the product undergoes a hardware revision. Fundamentally, these vulnerabilities are not a high risk for the industrial sector. In the industrial space, embedded systems are still most often “insecure by design,” or lacking some security protections. An example of this is the APC Uninterruptible Power Supply (UPS) susceptibility to the Ripple20 vulnerabilities. The researchers of the vulnerabilities developed a working exploit for CVE-2020-11901 against the UPS, which could result in shutting off the UPS. However, the UPS speaks an industrial protocol called BACnet that lacks basic security protections. According to the device instruction manual, and verified by Dragos, a BACnet control point can be used to achieve the same effect using freely available tools. ----- ###### Vulnerabilities in VPN Appliances Facilitating Remote Work Several VPN appliance vulnerabilities were disclosed and fixed in 2020. These include issues in non-industrial appliances that Dragos tracks and models that are specifically marketed toward industrial customers. In 2020, Dragos tracked 13 advisories that include VPN software and hardware commonly used in industrial environments and gateways with optional VPN features. Impacted devices include Palo Alto Networks Global Protect VPN client, Citrix Application Delivery Controller and Netscaler, and ICSspecific VPN services Ewon, Cosy, and Flexy. Enterprise VPN appliances are often used by industrial operators to provide remote access to corporate and operations networks. Some OEMs provide VPN access specifically for their ICS equipment, restricting connections or lateral movement to the rest of the OT network. Many of these products are identified as belonging to utilities via the popular search engine Shodan. Vulnerabilities in these products should be remediated quickly. End users should determine the exposure of VPN appliances. Industrial-specific VPN appliances may provide direct access to process control systems. These devices are best secured by using them in “client” mode where they may connect to a central server. The central server may be hardened to prevent site-to-site communications between field sites, and to monitor for suspicious behavior originating from field sites. ----- ###### SECTION THREE Lessons Learned From The Front Lines ----- ###### INTRODUCTION For the last two decades, the prevailing strategies for safeguarding ICS were focused on protecting the perimeter, preventative security countermeasures, and internal segmentation. In that time, standards, regulations, and best practices codified these methods. Other focuses, such as detection, response, and recovery were included but without equal emphasis or examples. The disparity has left industrial organizations underdeveloped in these core capabilities in an understandable, yet undesirable, situation. In general, the industry lacks visibility into their ICS assets and activities, hampering the overall cyber readiness and ability to understand and manage cyber risk. These themes are confirmed based upon analysis of Dragos service engagements in 2020. The data set includes engagements from many industrial infrastructure sectors including electric, oil and gas, food and agriculture, manufacturing, chemical, transportation, water and wastewater, technology (data center building automation equipment), and mining. THE FOLLOWING DATA IS BASED ON A GROWING SET OF ANNUAL ENGAGEMENTS CONDUCTED BY THE DRAGOS TEAM OF ICS CYBERSECURITY EXPERTS ON SEVERAL SERVICE TYPES, INCLUDING: - ARCHITECTURAL REVIEWS - VULNERABILITY ASSESSMENTS - PENETRATION TESTS - TABLE TOP EXERCISES - INCIDENT RESPONSE (IR) ----- ###### Visibility Over the course of the year, Dragos found that 90 percent of its services customers had limited to no visibility into their ICS environments. While most clients demonstrated a focus on an enhanced asset inventory, this effort is only the foundation for asset visibility. Many customers only monitored the IT to OT boundary without monitoring activity inside the ICS network. Network analysts were blind to critical network traffic. Some collected logs, but few utilized centralized logging to correlate various segments with network traffic analysis. These steps are critical for developing a full picture of what occurred across industrial assets and sites. Extremely Limited / No Visibility into OT Environment 100% 90% 90% 80% 2020 81% 70% 2019 60% 50% 40% 30% 20% 10% ----- ###### Compared to the Dragos 2019 Year in Review report, 90 percent represents a small increase in this statistic. Note that this is not indicative of the same customers failing to improve year-over-year. Lack of visibility, a common finding in architecture reviews, is skewed heavily towards new customers. Architecture reviews conducted with repeat customers are typically not for the same facility either. While not a perfect comparison, the statistic indicates the state of cybersecurity for those customers early in their OT cyber maturity. Dragos’s incident response cases for 2020 provide further support for this lack of visibility because none of them were augmented with any centralized or automated host and network traffic logging. This significantly slows down the incident response process, and in multiple cases, means that the asset owner or operator is not able to get critical questions answered. In at least one case, the impact led to public reporting without an understanding of root cause analysis where cyber activity was heavily suspected, but no evidence was available. IR Cases Facilitated by Automated Logging or Visibility into ICS Network 2019 2020 ----- ###### Segmentation and Connections Although asset owners and operators follow many of the best practices and their applicable regulation, Dragos continues to observe instances of poor segmentation with unexpected or unknown connections from the ICS network. About 88 percent of Dragos services engagements involved significant issues with network segmentation. Examples of observations contributing to this statistic include flat networks, where the only segmentation is the initial firewall between the IT-OT boundary, and unnecessary communication pathways to critical assets within the network. To further illustrate the incidence of poor segmentation, consider that adversaries accessed ICS networks directly from the internet in 100 percent of Dragos’s 2020 incident response cases. These findings are directly related to the previous statistic of 90 percent of organizations that lacked OT visibility. Identifying architecture bypasses and rogue connections and devices is nearly impossible without visibility through network monitoring. ###### Engagements Exhibiting Poor Security Perimeters 71% 2019 ## 71+29+ ###### 88% 2020 ## 88+12+ ----- ###### While the annual comparison shows a 17-point increase in engagements exhibiting poor security perimeters, Dragos does not believe the industry is regressing on this security principle. Dragos attributes the growth to a rise in the number of engagements conducted. However, this statistic does support the assertion that cyber strategies focused on segmentation and prevention are not sufficient. Adversary usage of shared credentials exacerbates the severity of poor security perimeters. Just over half of service engagements found shared credentials between IT and OT networks. For example, an organization may leverage the same credential management on the IT network as it does on the Demilitarized Zone (DMZ) and ICS network. This is another configuration that leads to a weakening of perimeters and may enable an adversary to easily traverse to ICS assets using credentials obtained from IT accounts. For example, 100 percent of incident response cases that confirmed adversary activity involved the adversary leveraging shared credentials for lateral movement. During one case, Dragos analysts responded to an Activity Group that harvested credentials from an IT asset, and then leveraged a vulnerability in a VPN appliance to gain initial access to the ICS environment. In this case, the adversary also used the compromised credentials to move laterally in the ICS network and access the critical assets. ###### Organizations that Lacked Separate IT and OT User Management ###### 54% 54% 2019 2020 ----- ###### External Routable Network Connection to ICS Environments Believed to be Air-Gapped 100% 2019 ###### 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% ###### A recurring theme from the previous Year in Review reports is the disconnect between expectations and reality about air-gapped systems. There was significant improvement in 2020 in this area, with a two-thirds drop in the discovery of external routable network connections to air-gapped ICS environments. During one engagement, a client who believed their network to be air- gapped had two external connections bypassing their architecture and terminating deep within their gas transmission systems. The purpose of the connections was to monitor gas quality and heating values, and ensure exported product met demand and quality specifications. Problematically, communications took place directly between critical controllers used for the industrial process and PLCs used only for data exchange with the remote connection. This connection could have been leveraged by an adversary to gain initial access to the network and move laterally across Level 1 systems within the Purdue model. Some systems in Level 1 had communication pathways through network allowlists to Safety Instrumented Systems (SIS) in the Safety Zone. Access to the Safety Zone may be the goal of an adversary who intends to cause a loss of safety impact, such as XENOTIME. ----- ###### Shared credentials and poor security perimeters allow adversaries to leverage Valid Accounts and gain persistent access to remote ICS. Valid Accounts is the most common TTP used across all Dragos ICS Activity Groups. Analysis of 2020 Dragos incident response cases indicates its effectiveness. ###### IR Cases where Adversary Accessed ICS Network from Internet ###### IR Cases Involving Shared Credentials for Lateral Movement ###### 2019 2020 ###### 66% ###### 100% ###### 2019 2020 ###### 90% ###### 100% ----- ###### Cyber Readiness In 2020, Dragos conducted several tabletop Organizations that did not have exercises and incident response readiness clear cyber incident thresholds workshops. The findings and outcomes of these engagements further illustrate the industry’s deep-seated, over-reliance on prevention and •••••••••• the necessity of strengthening the pillars of •••••••••• a successful ICS cyber strategy; detection, •••••••••• response, and recovery. •••••••••• •••••••••• •••••••••• Dragos observed that 75 percent of clients did •••••••••• not have clearly defined incident declaration ••••• ••••• thresholds or categories of severity within •••••••••• response plans. Detecting threat behaviors and •••••••••• analyzing correlated datasets are generally the 75% first steps an organization takes to recognize an incident. The threshold for declaring an incident depends on many factors including threats, tactics, operational risk requirements, governing laws, and industry regulations. Once incidents were declared, incident managers were often observed by Dragos to have no documented guidance or playbook for how to employ resources or capabilities. Nearly 60 percent of organizations did not Organizations that had have a solidified Incident Response Plan (IRP). a Solidified Incident managers were often left to create IRP Ready 58% tactical response plans in real-time and in the to Test 2020 middle of rapid escalations. In 2020, Dragos observed that 75 percent of 2020 clients did not have clearly defined or documented incident 33% 2019 declaration thresholds or categories of severity available during activation of their incident response plans. ----- ###### In 2020, 62 percent of Dragos client organizations Organizations did not tie a dedicated communications plan to that did not have their cybersecurity incident response activations. Communications This is a continuance of an industry trend from Plans Linked to IRP 2019. Communication plans are critical in enabling Activations incident managers and stakeholders to quickly activate, assemble, and employ resources and •••••••••• capabilities. During an incident, the need to react •••••••••• appropriately is immediate, followed by the need •••••••••• to communicate. Lack of effective communication •••••••••• during a crisis can lead to inadequate resource •••••••••• allocation, compound risks to assets and •••••••••• personnel, and create lingering effects on bottom- •• •••••••• line operations. The periodic testing of IRPs can •••••••••• act as a roadmap for corporate leadership to •••••••••• convey strategic directives and objectives to an incident response team. 100 percent of Dragos ••••••••••62% exercise participants strongly agree that exercises are beneficial to long-term incident response preparation and are a valuable way to identify areas Organizations that had of improvement for industrial cybersecurity IRPs. Difficulty Understanding when to Declare an Incident 83% Dragos 2019 75% 2020 customers who Agree Tabletop Exercises are Beneficial to IRPs ----- ###### Dragos Red Team In penetration tests, Dragos focuses on demonstrating adversary behavior that could have physical impacts on industrial processes. The team utilizes Crown Jewel Analysis to identify high consequence events, the assets that can cause them, and pathways to those assets specific to the client and the process.[13] The team often deploys the same TTPs of specific ICS Activity Groups to provide real-world applicability. ----- |Col1|Col2| |---|---| ||| ||| ||| ###### Recommendations As organizations strategize a path forward, Dragos recommends five key OT cybersecurity initiatives to improve on in 2021 and beyond. These are based on the empirical evidence provided throughout the report. Dragos has included a graphic below each recommendation as a reference for asset owners looking to implement the top five the recommendations of 2020. The recommendations listed on the right are in descending order according to priority. Each recommendation has three or more actions that may help asset owners achieve the desired goal of the recommendation. These actions are numbered with the positive impact on that recommendation. The top five recommendations to enhance the security of an ICS environment are: ###### Increase OT Network Visibility Visibility includes network monitoring, logging, and ### 1 ###### maintaining a Collection Management Framework (CMF). Logging and monitoring of OT systems is essential for detection and incident response in addition to providing actionable data regarding device performance, operation, and reliability. Network traffic analysis should be prioritized based on the considerations of the networks and their risks. **NETWORK** **ASSET** **MONITORING** **INVENTORY** 1 2 ###### HELP ACHIEVE **HOST/NETWORK** **COLLECTION** **INCREASE OT** **TRAFFIC LOG** **MANAGEMENT** **VISIBILITY** **AGGREGATION** **FRAMEWORK** 3 4 ----- ###### Identify and Prioritize Crown Jewels ### 2 ###### Crown Jewels are those assets that exercise control over the components most critical to the safe operation of the industrial process. Examples include HMIs, engineering and operator workstations, gateways, and controllers. Prioritizing the assets that, if compromised, could cause major impacts to the organization is key principle of risk management. Example: At one facility, Instrumentation and Control (I&C) staff could connect into the production network using a VPN. Once connected, they utilized RDP to access their engineering laptop. Their laptops had read/write permissions to any PLC in the Production Network, including the crown jewel PLCs. Additionally, control staff left PLC Run/Remote/Program key switches in remote mode meaning anyone with network access could edit PLC configurations and logic. Lastly, they did not have a mechanism to proactively detect changes to these PLC configurations. In summary, a compromised I&C technician account could modify PLC configurations and operations would not detect it, until an operational issue or periodic maintenance occurred. **ASSET** **NETWORK** **TABLETOP** **INVENTORY** **MONITORING** **EXERCISES** 1 2 3 **COLLECTION** **IDENTIFY AND** **RED/PURPLE** **SEPARATE AD** **MANAGEMENT** **PRIORITIZE CROWN** **TEAM** **INFRASTRUCTURE** **FRAMEWORK** **JEWELS** 4 5 6 ----- |Col1|Col2| |---|---| ||| |Col1|Col2| |---|---| ||| ||| ###### Boost Incident Response Capabilities ### 3 Incident response refers to an organization’s ###### approach for handling cybersecurity incidents. An incident response capability is necessary to minimize loss and restore safe operations. Many organizations have an enterprise IT incident response plan that does not account for, or significantly misunderstands, ICS incident response efforts. **COLLECTION** **ASSET** **MANAGEMENT** **INVENTORY** **FRAMEWORK** 1 2 ###### HELP ACHIEVE **HOST/NETWORK** **TABLETOP** **TRAFFIC LOG** **BOOST IR** **EXERCISES** **AGGREGATION** **CAPABILITIES** 3 4 ----- |4|Col2| |---|---| ||| ###### Validate Network Segmentation ### 4 This includes issues like weak segmentation ###### between IT and OT networks, permissive firewall rulesets, and externally routable network connections. Network segmentation should be continuously monitored to ensure it is not bypassed or negated. Example: A client believed their SIS devices were properly segmented. An external connection was found that could have allowed an adversary to bypass multiple network boundary devices with only a Virtual Local Area Network (VLAN) between the connection and the Safety Zone. **NETWORK** **RED/PURPLE** **ASSET** **MONITORING** **TEAM** **INVENTORY** 1 2 3 **HOST/NETWORK** **SEPARATE AD** **TRAFFIC LOG** **VALIDATE NETWORK** **INFRASTRUCTURE** **AGGREGATION** **SEGMENTATION** 4 5 ----- ###### Secure Credential Management ### 5 This includes accounts shared between IT and ###### OT, default accounts, and vendor accounts. AD shared between the enterprise and ICS networks is one of the most common findings that should be mitigated. Adversaries seek to compromise and leverage valid accounts as a means to access critical industrial systems. During one assessment, Dragos obtained root access to the affected entities by successfully identifying the root password for all the impacted systems related to the client’s HMI. This was accomplished through guessing and using common default passwords. **ASSET** **SEPARATE AD** **INVENTORY** **INFRASTRUCTURE** 1 2 ###### HELP ACHIEVE **RED/PURPLE** **SECURE** **TEAM** **CREDENTIAL** **MANAGEMENT** 3 ----- ###### Dragos is an industrial (OT/ICS/IIoT) cybersecurity company on a mission to safeguard civilization. Dragos is privately held and headquartered in the Washington, D.C. area with regional presence around the world, including Canada, Australia, New Zealand, Europe, and the Middle East. Dragos.com -----