End-to-end Botnet Monitoring //Secureworks/Confidential - Limited External Distribution End-to-end Botnet Monitoring… Botconf 2019 Kevin O’Reilly & Keith Jarvis Counter Threat Unit (CTU) Research Team //Secureworks/Confidential - Limited External Distribution …With Automated Config Extraction and Emulated Network Participation //Secureworks/Confidential - Limited External Distribution Agenda What will we be discussing today? • Two distinct angles: • Sandbox • Emulator More than the sum of their parts? 3 //Secureworks/Confidential - Limited External Distribution 4 Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) 1 Emulator – Sandbox Synergy • Emulator inputs: • C2 domains/IP addresses • RSA public keys • Version numbers • … • Sandbox inputs: • Fresh samples! //Secureworks/Confidential - Limited External Distribution Bots in the Sandbox //Secureworks/Confidential - Limited External Distribution Sandbox Essential Capabilities • Automated Unpacking • Configuration Decoding/Parsing 6 //Secureworks/Confidential - Limited External Distribution CAPE Sandbox 7 Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) 1 “Config And Payload Extraction” • Open Source Project - began 2015 • Derived from spender-sandbox • Itself derived from Cuckoo Sandbox (v1.3) in 2014 • Overlap with Cuckoo today minimal • https://github.com/kevoreilly/CAPE • Community version: https://capesandbox.com //Secureworks/Confidential - Limited External Distribution Parallels With Manual Approach 8 Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) 1 • Dynamic analysis • Victim machine/malware lab • API monitor • Debugger • Dumper • Import reconstructor • Static analysis • Disassembler for unpacked payload • YARA for detection • Decoder or parser for configuration //Secureworks/Confidential - Limited External Distribution The Sharpest Tool in the Sandbox? The Debugger • Powerful tool allowing instruction-level control • Processor (hardware) breakpoints • 4 breakpoints on read/write/execute • Single-step mode • Instruction traces • Trigger Actions: • Manipulate register/flag values • Dump payload or configuration • Set/clear further breakpoints • Set initial breakpoints via Yara signatures or API hooks 9 //Secureworks/Confidential - Limited External Distribution 10 Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) 1 • In-process debugger • Within monitor DLL • Processor (hardware) breakpoints • Debug registers Dr0 – Dr7 • 4 breakpoints (per-thread) • EXCEPTION_SINGLE_STEP • RtlDispatchException hook • SetThreadContext API • NtGet/SetContextThread hook protection Debugger-in-a-DLL //Secureworks/Confidential - Limited External Distribution Debugger demo QakBot Instruction Trace & Anti-VM Bypass 11 //Secureworks/Confidential - Limited External Distribution 12 Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) 1 Family Packages • A whole package devoted to one malware family • Handle specific behaviours • Have to have seen family before • Future versions of family may break package //Secureworks/Confidential - Limited External Distribution Behavioural Packages 13 Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) 1 • Capture payloads for a given behaviour: • Injection into other processes • Extraction of code • Decompression of code • Not dependent on signatures or having seen malware before • Captured payloads can then be detected by Yara signatures • If the extracted/injected payloads contain configuration data: • Yara signature triggers configuration parser //Secureworks/Confidential - Limited External Distribution ‘Compression’ Package 14 • Simplest package • Captures PE payload decompressed by RtlDecompressBuffer function: HOOKDEF(NTSTATUS, WINAPI, RtlDecompressBuffer, CompressionFormat, UncompressedBuffer, UncompressedBufferSize, CompressedBuffer, CompressedBufferSize, FinalUncompressedSize) { NTSTATUS ret = Old_RtlDecompressBuffer(CompressionFormat, UncompressedBuffer, UncompressedBufferSize, CompressedBuffer, CompressedBufferSize, FinalUncompressedSize); if (NT_SUCCESS(ret)) { DoOutputDebugString("RtlDecompressBuffer hook: scanning region 0x%x size 0x%x for PE image(s).\n", UncompressedBuffer, *FinalUncompressedSize); DumpPEsInRange(UncompressedBuffer, *FinalUncompressedSize); } return ret; } //Secureworks/Confidential - Limited External Distribution ‘Injection’ Package 15 Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) 1 • Captures payloads injected into other processes • Uses API hooks • Track newly created processes and threads • Injected directly • WriteProcessMemory, NtWriteVirtualMemory, etc • Process hollowing • NtMapViewOfSection • Transacted hollowing • Dumps prior to execution • NtResumeProcess/NtResumeThread //Secureworks/Confidential - Limited External Distribution ‘Extraction’ Package 16 Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) 1 • Captures payloads ‘extracted’ inside processes • Tracks executable memory regions • Newly allocated • New executable permissions • Uses debugger breakpoints • Write breakpoints • Capture payloads just after they have been written • Execution breakpoints • Capture payloads before they are executed //Secureworks/Confidential - Limited External Distribution Emotet 17 Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) 1 //Secureworks/Confidential - Limited External Distribution Emotet Automated Unpacking 18 Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) 1 AllocationHandler: Adding allocation to tracked region list: 0x003E0000, size: 0x11000. ActivateBreakpoints: TrackedRegion->AllocationBase: 0x003E0000, TrackedRegion->RegionSize: 0x11000, thread 2664 SetDebugRegister: Setting breakpoint 0 hThread=0xdc, Size=0x2, Address=0x003E0000 and Type=0x1. SetThreadBreakpoint: Set bp 0 thread id 2664 type 1 at address 0x003E0000, size 2 with Callback 0x74af7510. ActivateBreakpoints: Set write breakpoint on empty protect address: 0x003E0000 //Secureworks/Confidential - Limited External Distribution Emotet Automated Unpacking 19 Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) 1 CAPEExceptionFilter: breakpoint hit by instruction at 0x75FE9B60 (thread 2664) BaseAddressWriteCallback: Breakpoint 0 at Address 0x003E0000. ContextSetDebugRegister: Setting breakpoint 2 within Context, Size=0x0, Address=0x003E0000 and Type=0x0. BaseAddressWriteCallback: byte written to 0x3e0000: 0x1d. BaseAddressWriteCallback: Exec bp set on tracked region protect address. CAPEExceptionFilter: breakpoint hit by instruction at 0x003E0000 (thread 2664) ShellcodeExecCallback: Breakpoint 2 at Address 0x003E0000 - about to scan region for a PE image (base 0x003E0000, size 0x11000). ScanForDisguisedPE: PE image located at: 0x3e053f DumpImageInCurrentProcess: Attempting to dump 'raw' PE image. DumpPE: PE file in memory dumped successfully //Secureworks/Confidential - Limited External Distribution Emotet Config Extraction 20 Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) 1 rule Emotet { meta: cape_type = "Emotet Payload" strings: $snippet = {33 C0 21 05 ?? ?? ?? ?? A3 ?? ?? ?? ?? 39 05 ?? ?? ?? ?? 74 18 40 A3 ?? ?? ?? ?? 83 3C C5 ?? ?? ?? ?? 00 75} condition: uint16(0) == 0x5A4D and ($snippet) } refc2list = yara_scan(filebuf, '$snippet') c2list_va_offset = int(refc2list['$snippet']) c2_list_va = struct.unpack('i', filebuf[c2list_va_offset+15:c2list_va_offset+19])[0] c2_list_rva = c2_list_va - image_base c2_list_offset = pe.get_offset_from_rva(c2_list_rva) while 1: ip = struct.unpack('ContextRecord->Esp+4*3); DumpAddress = (PVOID)*(DWORD*)((BYTE*)ExceptionInfo->ContextRecord->Esp+4*4); CAPEExceptionFilter: breakpoint hit by instruction at 0x004054AF 0x4054af (05) e86c080000 CALL 0x871 Trace: CALL detected, grabbing size 0x2b and buffer 0x3bcff10 from stack. CAPEExceptionFilter: breakpoint hit by instruction at 0x004054B4 0x4054b4 (03) 83c414 ADD ESP, 0x14 DumpMemory: CAPE output file successfully created: C:\iaDUZcSim\CAPE\2580_70260233749341954122019 Added new CAPE file to list with path: C:\iaDUZcSim\CAPE\2580_70260233749341954122019 Trace: dumped QakBot config from 0x3bcff10. //Secureworks/Confidential - Limited External Distribution The Future… 23 Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) 1 • CAPE v2 just released • Python 3 • A huge thank you to Andriy Brukhovetskyy (@d00m3dr4v3n) – FireEye • https://github.com/kevoreilly/CAPEv2 • (KVM hardended anti-anti-vm: https://github.com/doomedraven/Tools/blob/master/Virtualization/kvm-qemu.sh) • Behavioural packages combined • Enabled by default • Reduce executions to maximum of 2 • Expanded Debugger options • Family-specific “package” all within YARA signature https://github.com/kevoreilly/CAPEv2 https://github.com/doomedraven/Tools/blob/master/Virtualization/kvm-qemu.sh https://github.com/doomedraven/Tools/blob/master/Virtualization/kvm-qemu.sh https://github.com/doomedraven/Tools/blob/master/Virtualization/kvm-qemu.sh https://github.com/doomedraven/Tools/blob/master/Virtualization/kvm-qemu.sh //Secureworks/Confidential - Limited External Distribution Ursnif/ISFB 24 Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) 1 rule Ursnif3 { meta: cape_type = "Ursnif Payload" cape_options = "dll=Debugger.dll,step-out=$crypto32, dumpsize=eax,action0=dumpebx, dumptype0=0x24, base-on-api=RtlAddVectoredExceptionHandler, dump-on-api=RtlAddVectoredExceptionHandler, dump-on-api-type=0x25" strings: $crypto32 = {8B C3 83 EB 01 85 C0 75 0D 0F B6 16 83 C6 01 89 74 24 14 8D 58 07 8B C2 C1 E8 07 83 E0 01 03 D2 85 C0 0F 84 AB 01 00 00 8B C3 83 EB 01 85 C0 89 5C 24 20 75 13 0F B6 16 83 C6 01 BB 07 00 00 00} $golden_ratio = {8B 70 EC 33 70 F8 33 70 08 33 30 83 C0 04 33 F1 81 F6 B9 79 37 9E C1 C6 0B 89 70 08 41 81 F9 84 00 00 00} condition: uint16(0) == 0x5A4D and (all of them) } //Secureworks/Confidential - Limited External Distribution Botnet Emulator Framework //Secureworks/Confidential - Limited External Distribution Why emulation? 26 • Botnet participation allows retrieval of artefacts and context unavailable from samples • Long-term tracking of targeting and operational details • Vetting indicators with high-fidelity • Synergy with config extraction for class of malware with features unavailable in ordinary sandbox detonation //Secureworks/Confidential - Limited External Distribution Architecture 27 //Secureworks/Confidential - Limited External Distribution Data Retention 28 • Less data than you think to store years of botnet interactions • Save everything! verbose logging, HTTP sessions, etc. • Database is under 100 GB • Local EBS volume is 100 GB • S3 storage is 230 GB //Secureworks/Confidential - Limited External Distribution Cost 29 • S • EC2: $30/mo • RDS: $60/mo • S3: $5/mo • EBS: $10/mo • VPS: $30/mo • VPN: $20/mo $255 per month //Secureworks/Confidential - Limited External Distribution OPSEC 30 • Make traffic and behaviour near replica of actual bot • Generate plausible but contrived metadata (e.g., computer and domain names) • Withstand competent non-state investigation • Non-attributable infrastructure unless subject to subpoena • Log timestamps and egress IP addresses for future correlation //Secureworks/Confidential - Limited External Distribution Network Egress 31 • Use a combination of Tor, VPS, and commercial VPNs • Geographic diversity is great for empirically finding geofencing • Simple round robin is largely suitable • VPN microservice allows an emulator to request specific regions/countries and capabilities //Secureworks/Confidential - Limited External Distribution Working with Commercial Providers 32 • Don’t (intentionally) trash someone else’s IP space • VPNs WILL ban you (no refund) • Beware the Internet do-gooder • Few if any allow outbound TCP 25 (necessary for Cutwail) • By product is good intel source of VPN exit nodes //Secureworks/Confidential - Limited External Distribution Action on Data 33 • “ChatOps” notifications to analysts • Email distribution lists (third- parties, LE, working groups) • Intelligence platforms • High fidelity blacklists //Secureworks/Confidential - Limited External Distribution Pitfalls: SOCKS Proxy 34 • OpenSSH SOCKS5 server is unstable during high throughput • Use Dante, open source SOCKS5 server • Protect with iptables and configuration-level filters and authentication //Secureworks/Confidential - Limited External Distribution Pitfalls: Custom HTTP Library 35 • Reasons to not use Requests • Protocol violations ⇉ Uncommon in malware • Absent SOCKS support ⇉ Added in 2016 • Custom header ordering ⇉ Use OrderedDict() • Provide IP address for request ⇉ Patch Requests • Access to endpoint SSL certificate ⇉ Patch Requests //Secureworks/Confidential - Limited External Distribution Pitfalls: Neglect DNS Data 36 • Nature of system allows tracking of DNS resolutions over time • Augment this data with passive DNS (Farsight) • Useful for C2 with flaky DNS but stable hosting • Need to be careful with C2 living in fast-flux systems //Secureworks/Confidential - Limited External Distribution Pitfalls: Neglect SSL Certificates 37 • Save unique certificates as they appear and associate with IP • Flowsynth to generate PCAP from X.509, check coverage • Augment other data sets like Censys, Shodan, or SONAR //Secureworks/Confidential - Limited External Distribution Pitfalls: Dealing with Sinkholes 38 • Regularly retrieve sinkhole list from abuse.ch SinkDB • Detect common sinkhole-related HTTP features • Avoiding sinkholes saves execution times and prevents poisoning other researchers’ data • BitSight won’t track us down and try to sell us a report about our emulator being infected with dangerous malware //Secureworks/Confidential - Limited External Distribution Pitfalls: Data Pruning 39 • Always prune your list of C2 servers • General formula: has the C2 accepted connections in two weeks? Given a valid response in past month? //Secureworks/Confidential - Limited External Distribution PITFALLS: “SKIN DEEP” EMULATION 40 • Register bot, interrogate C2, dispose of bot • Instead, store registered bots and periodically reconnect them to create pool of long-lived “infections” • Good way to get additional payloads to “mature” bots