# Monitoring Pulse Connect Secure With Splunk (CISA Emergency Directive 21-03) **[splunk.com/en_us/blog/security/monitoring-pulse-connect-secure-with-splunk-cisa-emergency-directive-21-03.html](https://www.splunk.com/en_us/blog/security/monitoring-pulse-connect-secure-with-splunk-cisa-emergency-directive-21-03.html)** April 21, 2021 By [Dave Herrald April 21, 2021](https://www.splunk.com/en_us/blog/author/dherrald.html) [Contributors: Mick Baccio,](https://www.splunk.com/en_us/blog/author/mbaccio.html) _[James Brodsky, Tamara Chacon, Shannon Davis,](https://www.splunk.com/en_us/blog/author/jbrodsky.html)_ _[Dave Herrald,](https://www.splunk.com/en_us/blog/author/dherrald.html)_ _[Kelly Huang,](https://www.splunk.com/en_us/blog/author/khuang.html)_ _[Ryan Kovar,](https://www.splunk.com/en_us/blog/author/rkovar.html)_ _[Marcus LaFerrerra, Michael Natkin,](https://www.splunk.com/en_us/blog/author/mlaferrera.html)_ _[John Stoner and](https://www.splunk.com/en_us/blog/author/jstoner.html)_ _[Bill Wright](https://www.splunk.com/en_us/blog/author/bwright.html)_ ----- **Update May 4, 2021: Over the last two weeks, there have been several significant** [developments. First and most importantly, Pulse Secure issued an update on May 3](https://kb.pulsesecure.net/articles/Pulse_Secure_Article/SA44784/) addressing multiple vulnerabilities. Splunk recommends that all Pulse Secure users review [and install the update as soon as possible. On April 30, CISA updated Alert (AA21-110A)](https://us-cert.cisa.gov/ncas/alerts/aa21-110a) [with new detections, including the "Impossible Travel" detection and JA3 analysis. We have](https://docs.splunksecurityessentials.com/content-detail/land_speed_privileged/) updated our Splunk-friendly collection of indicators to include the latest from CISA. To immediately see how to find potential vulnerabilities or exploits in your Pulse Connect Secure appliance, skip down to the "Identifying, Monitoring and Hunting with Splunk" section. Otherwise, read on for a quick breakdown of what happened, how to detect it, and MITRE ATT&CK mappings. _https://snappygoat.com/b/29e92459abc8f20ebaa4fffd7921b8f67513db9e_ ## What You Need to Know About the Pulse Connect Secure Attacks Over the past few weeks, there has been increasing chatter regarding adversary groups exploiting multiple vulnerabilities in the Pulse Connect Secure (PCS) virtual private network [(VPN) appliance On April 20 2021 the Mandiant team at FireEye released a blog detailing](https://www.fireeye.com/blog/threat-research/2021/04/suspected-apt-actors-leverage-bypass-techniques-pulse-secure-zero-day.html) ----- their findings from multiple recent incidents involving compromised PCS appliances. This report prompted a flurry of activity from various organizations, including government agencies and security vendors. That same day, DHS Cybersecurity and Infrastructure [Security Agency (CISA) released Alert (AA21-110A) and](https://us-cert.cisa.gov/ncas/alerts/aa21-110a) [Emergency Directive 21-03, the](https://cyber.dhs.gov/ed/21-03/) latter requiring all US Federal agencies to take specific action concerning PCS appliances in their environments. Splunk recommends all US Federal agencies refer to the DHS directive to ensure compliance. According to a [blog post by Pulse Secure, the incidents disclosed this week involve](https://blog.pulsesecure.net/pulse-connect-secure-security-update/) vulnerabilities that were patched in 2019 and 2020, plus a new issue (CVE-2021-22893 [Security Advisory SA44784) discovered this month. The vendor notes that a software update](https://kb.pulsesecure.net/articles/Pulse_Security_Advisories/SA44784) for this new issue will be available in early May. The post contains valuable information on all the vulnerabilities and recommended mitigation measures, and customer support information. Of particular importance is the Pulse Connect Secure Integrity Tool, which allows you to check if essential components of your PCS appliance software have been tampered with. Splunk recommends all PCS customers follow this vendor-published guidance in its entirety. The PCS appliance is a popular VPN solution that offers workers secure access to an organization's internal networks from anywhere in the world. Because VPN appliances play a critical role in securing an organization’s network perimeter and, by design, exposed directly to the Internet, they are often targeted by adversaries. Although the latest news is specific to Pulse Secure products, attacks of this nature are not limited to one vendor. Adversaries commonly return to vulnerabilities in VPN products to gain unauthorized access to organizations. As recently as April 16, 2021, the US National Security Agency released a [cybersecurity advisory warning that older vulnerabilities in at least five different remote](https://www.nsa.gov/News-Features/Feature-Stories/Article-View/Article/2573391/russian-foreign-intelligence-service-exploiting-five-publicly-known-vulnerabili/) access products were being actively exploited. ## Identifying, Monitoring, and Hunting with Splunk Here are some hot-off-the-press searches to help find some of the badness described in the FireEye/Mandiant blog and other sources. If we have coverage for these searches in [Splunk Security Content, we call them out in the MITRE ATT&CK section.](https://github.com/splunk/security_content) Note that proof of concept (POC) exploits are only available for some of the vulnerabilities involved in this attack. The detections below are all derived from a lab environment and informed by the context provided in the FireEye/Mandiant report. If we uncover more information, we will publish updates! ### Ingesting PCS Appliance Data into Splunk Several of the following techniques require logs from your PCS appliances to be ingested into Splunk for analysis. PCS appliances use syslog to log to external systems like Splunk. [Ivanti publishes configuration details in the PCS appliance Admin guide; here is a link to](https://help.ivanti.com/ps/legacy/pcs/9.1rx/9.1r11/ps-pcs-sa-9.1r11.0-admin-guide.pdf) ----- [Admin Guide version 9.1R11 (the latest available at the time of publication). PCS appliances](https://help.ivanti.com/ps/legacy/pcs/9.1rx/9.1r11/ps-pcs-sa-9.1r11.0-admin-guide.pdf) support multiple log export formats ("standard," "WELF," "W3C"). Each of these formats yields slightly different data and results. In 2015, Juniper Networks divested the JunOS Pulse product line to Pulse Secure, LLC, now owned by Ivanti. Early versions of the Splunk Add-on for Juniper included support for the IVE/SA (now PCS) product line, with a sourcetype of "juniper_sslvpn" or "juniper:ive." Since this time, third parties have published their own TAs [on SplunkBase (such as this one) or GitHub (such as](https://splunkbase.splunk.com/app/3852/) [this one) to continue to support the](https://github.com/mnatkin-splunk/TA-PCS) [PCS appliance log data. BOTS V’s RWI scenario made use of WELF-formatted log data from](https://www.splunk.com/en_us/blog/conf-splunklive/bots-v-at-conf20.html) [a Juniper SA utilizing this TA. Note that some examples in this blog use a sourcetype of](https://github.com/mnatkin-splunk/TA-PCS/blob/main/Legacy%20version/Splunk_TA_ive.spl) ‘juniper:ive,’ some use a sourcetype of ‘pulse:connectsecure’, and some use a sourcetype of ‘syslog.’ This emulates the variety of formatting we’ve observed in the wild. You should determine the sourcetype in use in your environment and substitute it in these example searches when necessary. Additionally, we recommend enabling logging for "Unauthenticated Requests" (this setting is disabled by default on PCS appliances). While this may slightly increase the log volume, the [CISA alert demonstrates that it is vital in revealing evidence of tampering and potentially](https://us-cert.cisa.gov/ncas/alerts/aa21-110a) ongoing malevolent activity. ### Version Checking According to this week’s advisories, certain versions of the Pulse Secure appliance software contain critical vulnerabilities. Therefore, it is important to catalog the versions of all Pulse Secure Connect appliances you have on your network so that you can create a mitigation plan. You can use Splunk to do this if you don’t have an alternative method. If you already have a list of IP addresses for your PCS appliances, you can feed that list as a lookup into a simple [Splunk Technical Add-On that we have created, which will sequentially probe each](https://github.com/stressboi/TA-pcsverchecker) appliance in the list and report the PCS version found into Splunk for reporting or alerting. ``` index=main source="checkpcsversions.sh" | stats values(pcs_version) as "PCS Version" by dest_ip | iplocation dest_ip ``` ----- ### CVE-2021-22893 Mitigation Monitoring This technique can be employed to assess your exposure by finding appliances that may still [require mitigation according to the Pulse Secure guidance. We suggest you search through](https://kb.pulsesecure.net/articles/Pulse_Security_Advisories/SA44784) those logs for evidence of the following: Windows File Share Browse Pulse Secure Collaboration Why? Because both of those services are vulnerable, and the temporary "workaround" mitigation that you should have already applied turns both of these functions off. Therefore, **if you are still seeing activity from either function in the logs, and you know that your** **appliance is running a vulnerable Pulse Secure version, you must apply the** **[appropriate workaround via application of the "workaround-2104.xml" file. You can](https://kb.pulsesecure.net/articles/Pulse_Security_Advisories/SA44784)** create a report or alerts based on any appliances still logging file sharing or collaboration activity, and provide the report or alerts to your network team to implement the workaround. Pulse Secure has committed to releasing an update in early May 2021, at which time this workaround can be reverted, thereby restoring these two capabilities if necessary. **Pulse Secure Collaboration Activity** Here is a quick and easy search to determine if Pulse Secure Collaboration is currently being used. You might look at the search and think, what is this reference to "meetings?" Well, ["Secure Meetings" is what the original functionality in the appliance was called. Not all of the](https://docs.pulsesecure.net/WebHelp/Content/PCS/PCS_AdminGuide_8.2/Pulse%20Collaboration%20Overview.htm#:~:text=Pulse%20Collaboration%20(formerly%20Secure%20Meeting,and%20non%2DConnect%20Secure%20users.) meeting fields are displayed here nor are they all required to determine if it is running, but it is meant to show you what can be seen in the events. ``` sourcetype=pulse:connectsecure dest= meeting_id=* | table _time src meeting_name meeting_action meeting_bytes_read meeting_bytes_written meeting_client_version meeting_invitee meeting_server meeting_property meeting_property_initial_value meeting_property_new_value meeting atendee meeting duration ``` ----- ``` | sort + _time ``` **_Note: You should determine the sourcetype in use in your environment and substitute it in_** _these example searches as applicable._ ### Windows File Share Activity As stated, you’re also going to want to track Windows File Share activity. If your PCS appliance is configured to use WebTrends Enhanced Log Format (WELF) formatting for syslog events, this activity will be captured in the proto field. Search your logs for: **_proto=fbr_** for Windows File Share activity And as an alternative to the meeting search above, you can also search: **_proto=meeting_** for Pulse Secure Collaboration activity An example of this, combining both kinds of activity in the same search, is as follows: ``` search sourcetype=juniper:ive proto!=auth | stats values(proto) AS Activity by user ``` ----- **_Note: You should determine the sourcetype in use in your environment and substitute it in_** _these example searches as applicable._ However, if your PCS appliance is configured for "standard" formatting for your syslog, PCS may identify activity based on a message code within the "msg" field. The format is three letters followed by five numbers (e.g. "MTG20035" or "FBR20503"). The examples below extract only the three letter code which indicates the feature being used. As previously noted, FBR denotes Windows File Share activity. Examples of this are as follows: ``` sourcetype="juniper:ive" | rex field=msg "(?^\w{3})" | stats values(product) AS Activity by user ``` ----- In the following example, we’re substituting the activity code in-line (though this could be accomplished with a lookup as well): ``` sourcetype="juniper:ive" | rex field=msg "(?^\w{3})" | eval app=case(match(product,"FBR"),"Windows File Browser",match(product,"JAV"),"Java Embedded App",match(product,"MTG"),"Secure Meeting") | stats values(app) AS Activity by user ``` **_Note: You should determine the sourcetype in use in your environment and substitute it in_** _these example searches as applicable._ Alternatively, if your logging includes URIs, you may search for the following: ``` /dana/fb/smb/wfb.cgi for Windows File Share browsing activity /dana/fb/smb/swsh.cgi for Windows File Share activity ``` ----- ``` /dana na/{meeting_base_URL}/meetingrun.cgi for Pulse Secure Collaboration activity (where {meeting_base_URL} is the administrator-configured base URL for secure meetings) ``` Depending on the logging configuration on the PCS appliance, events such as the ones below are logged when connections are attempted to access files on the Windows servers. Turning to a familiar source of data that you may also be collecting, Windows Security event logs, the search below provides a way to gain insight into files being accessed through a share directly from Windows. This can be useful in case the PCS appliance is not logging this level of fidelity. ``` source="WinEventLog:Security" EventCode=5145 Source_Address= Share_Name="\\\\*\\C$" | table _time ComputerName Security_ID Relative_Target_Name Accesses ``` ----- Similarly, if a share is attempted to be accessed without proper permissions, a Windows EventCode 4625 will trigger. Turning back to the Pulse Secure logs, in the example below, the user "badguy" successfully joins as shown in these events: However, “badguy” tries to connect to a file share and is rejected. The following search sheds light on why he was not able to access the desired resource. In this example with LDAP configured on the Windows device, the "user name does not exist." This search can be used to determine why login events from that originate from the Pulse Server(s) are not authenticating successfully. ``` source="WinEventLog:Security" EventCode=4625 Source_Network_Address= | table _time EventCode hostname user Source_Network_Address Logon_Process ``` Why is all of that important? Because, it enables another potential detection. FireEye noted [in their blog that “the malicious operations of SLOWPULSE can be detected via log](https://www.fireeye.com/blog/threat-research/2021/04/suspected-apt-actors-leverage-bypass-techniques-pulse-secure-zero-day.html) correlation between the authentication servers responsible for LDAP and RADIUS auth and the VPN server. Authentication failures in either LDAP or RADIUS logs with the associated VPN logins showing success would be an anomalous event worthy of flagging.” The following search will return a listing of login events from Pulse that are tied into LDAP; in our case we are using Active Directory for LDAP so our realm is called AD. ``` sourcetype=pulse:connectsecure dest= action=success realm=AD | table _time user | sort + _time ``` ----- From there, we could take those Pulse logins and look for the failed login events, Event Code 4625. That search is above. Timestamps should be very close to one another, within seconds. **_Note: You should determine the sourcetype in use in your environment and substitute it in_** _these example searches as applicable._ ### Indicators of Compromise (IOCs) FireEye/Mandiant published IOCs, including files, hashes, web shells, and sed patterns in their blog posts. [Ivanti and](https://kb.pulsesecure.net/articles/Pulse_Security_Advisories/SA44784/) [CISA published additional IOCs (including additional hashes and](https://us-cert.cisa.gov/ncas/alerts/aa21-110a) affected URIs) in their respective posts. We have consolidated and converted these [indicators into simple CSV format so that you may use them as lookup tables — and posted](https://docs.splunk.com/Documentation/Splunk/8.1.2/Knowledge/Aboutlookupsandfieldactions) to Github [here for review and consumption. But what’s a lookup table, and how does it help](https://github.com/mnatkin-splunk/pulse_connect_secure-splunk-csvs) [with security detection in Splunk? Got you covered there, too.](https://www.splunk.com/en_us/blog/security/lookup-before-you-go-go-hunting.html) ### Webshells Many of the attacks disclosed this week result in the installation of webshells on the PCS [appliance. We felt it valuable to include a reference to this classic Youtube video from James](https://www.youtube.com/watch?v=FEb8KZoEyzI) Bower about how to hunt web shells with Splunk. ## Splunk Enterprise Security ### Know Thyself While we have spent some time explaining this attack and effort needs to be put toward investigating this, it is also important to note that the basics are important. Basic asset management, hopefully via your asset and identity framework, will tell you where your vulnerable systems reside. Running regular vulnerability scans that integrate into Splunk will display which systems are vulnerable and can help you prioritize your patching schedule and better focus your detection efforts. ----- ### Threat Intelligence Framework [You can also leverage the IOCs above (for those that don’t want to scroll up, here they are)](https://github.com/mnatkin-splunk/pulse_connect_secure-splunk-csvs) with [Splunk Enterprise Security. The Threat Intelligence Framework can easily ingest these](https://www.splunk.com/en_us/software/enterprise-security.html) IOCs and Splunk Enterprise Security will let you know if any of them are identified. If you [aren’t sure how to do this, check out our recent blog post that walks you through the process](https://www.splunk.com/en_us/blog/security/smoothing-the-bumps-of-onboarding-threat-indicators-into-splunk-enterprise-security.html) of onboarding IOCs in Splunk Enterprise Security. ## MITRE ATT&CK Reviewing the blog posts from FireEye/Mandiant, we mapped the adversary’s activity to MITRE ATT&CK. Each tactic is then linked to Splunk Content to help you hunt for that information. Be aware; these searches are provided as a way to accelerate your hunting. We [recommend you configure them via the Splunk Security Essentials App. You may need to](https://splunkbase.splunk.com/app/3435/) modify them to work in your environment! Many of these searches are optimized for use with the tstats command. Finally, as more information becomes available, we will update these searches if more ATT&CK TTPs become known. **ATT&CK** **Tactic** **Title** **Splunk Searches** T1003 OS Credential Dumping [Many Detections](https://github.com/splunk/security_content/search?q=T1003) ----- T1016 System Network Configuration Discovery Detect processes used for System Network Configuration Discovery T1021.001 Remote Desktop Protocol [Remote Desktop Network Bruteforce](https://github.com/splunk/security_content/blob/5904eb999d51d86fce9c71c305d4513d2a250276/detections/experimental/network/remote_desktop_network_bruteforce.yml) [Remote Desktop Process Running On System](https://github.com/splunk/security_content/blob/5904eb999d51d86fce9c71c305d4513d2a250276/detections/experimental/endpoint/remote_desktop_process_running_on_system.yml) [Remote Desktop Network Traffic](https://github.com/splunk/security_content/blob/5904eb999d51d86fce9c71c305d4513d2a250276/detections/experimental/network/remote_desktop_network_traffic.yml) T1027 Obfuscated Files or Information T1048 Exfiltration Over Alternative Protocol Malicious PowerShell Process - Encoded Command [Many Detections](https://github.com/splunk/security_content/search?q=T1048) T1053 Scheduled Task/Job [Many Detections](https://github.com/splunk/security_content/search?q=T1053) T1057 Process Discovery Reconnaissance and Access to Processes and Services via Mimikatz modules Reconnaissance and Access to Operating System Elements via PowerSploit modules T1059 Command and Scripting Interpreter T1059.003 Windows Command Shell T1070 Indicator Removal on Host T1070.001 Clear Windows Event Logs [Many Detections](https://github.com/splunk/security_content/search?q=T1059) [Many Detections](https://github.com/splunk/security_content/search?q=T1059.003) [Many Detections](https://github.com/splunk/security_content/search?q=T1070) [Windows Event Log Cleared](https://github.com/splunk/security_content/blob/6212932c3573345c3e7714ae07cb35693d01ae76/detections/endpoint/windows_event_log_cleared.yml) ----- T1071.001 Web Protocols [TOR Traffic](https://github.com/splunk/security_content/blob/5904eb999d51d86fce9c71c305d4513d2a250276/detections/experimental/network/tor_traffic.yml) T1082 System Information Discovery [System Information Discovery Detection](https://github.com/splunk/security_content/blob/5904eb999d51d86fce9c71c305d4513d2a250276/detections/endpoint/system_information_discovery_detection.yml) [Web Servers Executing Suspicious Processes](https://github.com/splunk/security_content/blob/5904eb999d51d86fce9c71c305d4513d2a250276/detections/experimental/application/web_servers_executing_suspicious_processes.yml) T1098 Account Manipulation [Many detections](https://github.com/splunk/security_content/search?q=T1098) T1105 Ingress Tool Transfer CertUtil Download With URLCache and Split Arguments CertUtil Download With VerifyCtl and Split Arguments [BITSAdmin Download File](https://github.com/splunk/security_content/blob/a40dc1283291e9959f38720d7f0efbbdf5fbbdf4/detections/endpoint/bitsadmin_download_file.yml) T1136 Create Account [Many detections](https://github.com/splunk/security_content/search?q=T1136) T1140 Deobfuscate/Decode Files or Information T1190 Exploit Public Facing Application T1505.003 Web Shell [CertUtil With Decode Argument](https://github.com/splunk/security_content/blob/6212932c3573345c3e7714ae07cb35693d01ae76/detections/endpoint/certutil_with_decode_argument.yml) [SQL Injection with Long URLs](https://github.com/splunk/security_content/blob/5904eb999d51d86fce9c71c305d4513d2a250276/detections/experimental/web/sql_injection_with_long_urls.yml) [Supernova Webshell](https://github.com/splunk/security_content/blob/5904eb999d51d86fce9c71c305d4513d2a250276/detections/experimental/web/supernova_webshell.yml) W3WP Spawning Shell [Detect Exchange Web Shell](https://github.com/splunk/security_content/blob/6212932c3573345c3e7714ae07cb35693d01ae76/detections/endpoint/detect_exchange_web_shell.yml) T1518 Software Discovery Reconnaissance and Access to Operating System Elements via PowerSploit modules ----- T1554 Compromise Client Software Binary [Applying Stolen Credentials via Mimikatz modules](https://github.com/splunk/security_content/blob/6212932c3573345c3e7714ae07cb35693d01ae76/detections/endpoint/ssa___applying_stolen_credentials_via_mimikatz_modules.yml) Applying Stolen Credentials via PowerSploit modules T1562 Impair Defenses [Many detections](https://github.com/splunk/security_content/search?q=T1562) T1569.002 Service Execution [Malicious Powershell Executed As A Service](https://github.com/splunk/security_content/blob/a40dc1283291e9959f38720d7f0efbbdf5fbbdf4/detections/endpoint/malicious_powershell_executed_as_a_service.yml) [Create Service In Suspicious File Path](https://github.com/splunk/security_content/blob/dd2393e0570f99ef7d5d8f0d37469212662ca92c/detections/endpoint/create_service_in_suspicious_file_path.yml) [First Time Seen Running Windows Service](https://github.com/splunk/security_content/blob/5904eb999d51d86fce9c71c305d4513d2a250276/detections/experimental/endpoint/first_time_seen_running_windows_service.yml) T1574 Hijack Execution Flow Detect Path Interception By Creation Of program exe Reg exe Manipulating Windows Services Registry Keys Here is the full list of all the MITRE ATT&CK TTP’s that FireEye/Mandiant reported as being associated with malware/groups identified in this attack: ``` T1003,T1016, T1021.001, T1027, T1036.005, T1048, T1049, T1053, T1057, T1059, T1059.003, T1070, T1070.001, T1070.004, T1071.001, T1082, T1098, T1105,T1111,T1133,T1134.001, T1136,T1140,T1190,T1505.003,T1518,T1554,T1556.004,T1592.004, T1562,T1569.002,T1574, T1600 ## Conclusion ``` We know that these are significant vulnerabilities and that customers will want to patch as soon as possible and determine if they were affected by this attack. If you haven’t patched yet (we’ve all been there), hopefully, these searches will provide you the ability to have more visibility into your environment and any malicious activity that you might be experiencing. If they don’t work perfectly, think of them as "SplunkSpiration" :-). As soon as we have more ----- **information, we will update this blog. Need Help? Splunk experts are here for you!** **Customers should reach out to their customer success manager, sales engineer, or** **regional sales manager.** -----