## End-to-end Botnet Monitoring… Botconf 2019 ###### Kevin O’Reilly & Keith Jarvis Counter Threat Unit (CTU) Research Team ----- ## …With Automated Config Extraction and Emulated Network Participation ----- ##### Agenda ###### What will we be discussing today? • Two distinct angles: • Sandbox • Emulator #### More than the sum of their parts? ----- ##### Emulator – Sandbox Synergy ###### • Emulator inputs: Intelligence Requirements • C2 domains/IP addresses (IR’s) • RSA public keys • Version numbers Priority Intelligence Requirements • … (PIR’s) • Sandbox inputs: Specific Intelligence Requirements (SIR’s) • Fresh samples! ## 1 |Col1|Col2|• Emulator inputs: ence R•e qu Cire 2m de ont ms ains/IP addresses (IR’s) • RSA public keys • Version numbers iority Intelligence Requir• em e…nts (PIR’s) • Sandbox inputs: pecific Intelligence Requir•em enFts resh samples! (SIR’s)|Col4| |---|---|---|---| |• Emulator inputs: Intelligence R•e qu Cire 2m de ont ms ains/IP add (IR’s) • RSA public keys • Version numbers Priority Intelligence Requir• em e…nts (PIR’s) • Sandbox inputs: Specific Intelligence Requir•em enFts resh samples! (SIR’s) 1||• Emulator inputs: ence R•e qu Cire 2m de ont ms ains/IP add (IR’s) • RSA public keys • Version numbers iority Intelligence Requir• em e…nts (PIR’s) • Sandbox inputs: pecific Intelligence Requir•em enFts resh samples! (SIR’s)|| ||• Emulator inputs: Intelligence R•e qu Cire 2m de ont ms ains/IP (IR’s) • RSA public key • Version numb Priority Intelligence Requir• em e…nts (PIR’s) • Sandbox inputs: Specific Intelligence Requir•em enFts resh samples (SIR’s)|• Emulator inputs: ence R•e qu Cire 2m de ont ms ains/IP (IR’s) • RSA public key • Version numb iority Intelligence Requir• em e…nts (PIR’s) • Sandbox inputs: pecific Intelligence Requir•em enFts resh samples (SIR’s)|| ||||| ###### • Emulator inputs: Intelligence Requirements • C2 domains/IP addresses (IR’s) • RSA public keys • Version numbers Priority Intelligence Requirements • … (PIR’s) • Sandbox inputs: Specific Intelligence Requirements (SIR’s) • Fresh samples! ----- ## Bots in the Sandbox ----- ##### Sandbox ###### Essential Capabilities • Automated Unpacking • Configuration Decoding/Parsing ----- ##### CAPE Sandbox ## 1 |Col1|“Config And Pa|Col3| |---|---|---| ||“Config And P|| |• • • • •|Open Source Project - began 2015 Intelligence Requirements (IR’s) Derived from spender-sandbox • Itself derived from Cuckoo Sandbox (v1.3) Priority Intelligence Requirements (PIR’s) Overlap with Cuckoo today minimal Specific Intelligence https://github.com/kevoreilly/CAPE Requirements (SIR’s) Community version: https://capesandbox.com|in 2014| ||1|| ###### Open Source Project - began 2015 Intelligence Requirements (IR’s) Derived from spender-sandbox Itself derived from Cuckoo Sandbox (v1.3) in 2014 Priority Intelligence Requirements (PIR’s) Overlap with Cuckoo today minimal Specific Intelligence ###### https://github.com/kevoreilly/CAPE Requirements (SIR’s) ###### Community version: https://capesandbox.com ----- ##### Parallels With Manual Approach ###### • Dynamic analysis • Victim machine/malware lab Intelligence Requirements • API monitor (IR’s) • Debugger • Dumper Priority Intelligence • Import reconstructor Requirements (PIR’s) • Static analysis • Disassembler for unpacked payload Specific Intelligence Requirements • YARA for detection (SIR’s) • Decoder or parser for configuration ## 1 ###### • Dynamic analysis • Victim machine/malware lab Intelligence Requirements • API monitor (IR’s) • Debugger • Dumper Priority Intelligence • Import reconstructor Requirements (PIR’s) • Static analysis • Disassembler for unpacked payload Specific Intelligence Requirements • YARA for detection (SIR’s) • Decoder or parser for configuration ----- ##### The Sharpest Tool in the Sandbox? ###### The Debugger • Powerful tool allowing instruction-level control • Processor (hardware) breakpoints • 4 breakpoints on read/write/execute • Single-step mode • Instruction traces • Trigger Actions: • Manipulate register/flag values • Dump payload or configuration • Set/clear further breakpoints • Set initial breakpoints via Yara signatures or API hooks ----- ##### Debugger-in-a-DLL ###### In-process debugger • Within monitor DLL Intelligence Requirements (IR’s) Processor (hardware) breakpoints • Debug registers Dr0 – Dr7 • 4 breakpoints (per-thread) Priority Intelligence • EXCEPTION_SINGLE_STEP Requirements (PIR’s) RtlDispatchException hook Specific Intelligence Requirements ###### SetThreadContext API (SIR’s) • NtGet/SetContextThread hook protection ## 1 ###### • Within monitor DLL Intelligence Requirements • Processor (hardware) breakpoints (IR’s) • Debug registers Dr0 – Dr7 • 4 breakpoints (per-thread) Priority Intelligence • EXCEPTION_SINGLE_STEP Requirements (PIR’s) • RtlDispatchException hook Specific Intelligence ###### • SetThreadContext API Requirements (SIR’s) • NtGet/SetContextThread hook protection ----- ##### Debugger demo ###### QakBot Instruction Trace & Anti-VM Bypass ----- ## 1 |Family Packages|Col2| |---|---| |• A whole package devoted to one malware family Intelligence Requirements • Handle specific behaviours (IR’s) • Have to have seen family before Priority Intelligence Requirements (PIR’s) • Future versions of family may break package Specific Intelligence Requirements (SIR’s)|| |• A whole package devoted to one m Intelligence Requirements • Handle specific behaviours (IR’s) • Have to have seen family before Priority Intelligence Requirements (PIR’s) • Future versions of family may brea Specific Intelligence Requirements (SIR’s)|| |1|| ###### A whole package devoted to one malware family Intelligence Requirements (IR’s) • Handle specific behaviours Priority Intelligence Have to have seen family before Requirements (PIR’s) Future versions of family may break package Specific Intelligence Requirements (SIR’s) ----- ## 1 |Behavioural Packages|Col2| |---|---| |• Capture payloads for a given behaviour: • Injection into other processes Intelligence Requirements • Extraction of code (IR’s) • Decompression of code Priority Intelligence • Not dependent on signatures or having seen malware before Requirements (PIR’s) • Captured payloads can then be detected by Yara signatures Specific Intelligence Requirements • If the extracted/injected payloads contain configuration data: (SIR’s) • Yara signature triggers configuration parser 1|| |• Capture payloads for a given behaviour: • Injection into other processes Intelligence Requirements • Extraction of code (IR’s) • Decompression of code Priority Intelligence • Not dependent on signatures or having se Requirements (PIR’s) • Captured payloads can then be detected b Specific Intelligence Requirements • If the extracted/injected payloads contain (SIR’s) • Yara signature triggers configuration p 1|| ||| ###### • Injection into other processes Intelligence Requirements • Extraction of code (IR’s) • Decompression of code Priority Intelligence Not dependent on signatures or having seen malware before Requirements (PIR’s) Captured payloads can then be detected by Yara signatures Specific Intelligence Requirements (SIR’s) ###### If the extracted/injected payloads contain configuration data: • Yara signature triggers configuration parser ----- ##### Compression Package ###### • Simplest package • Captures PE payload decompressed by RtlDecompressBuffer function: ``` HOOKDEF(NTSTATUS, WINAPI, RtlDecompressBuffer, CompressionFormat, UncompressedBuffer, UncompressedBufferSize, CompressedBuffer, CompressedBufferSize, FinalUncompressedSize) { NTSTATUS ret = Old_RtlDecompressBuffer(CompressionFormat, UncompressedBuffer, UncompressedBufferSize, CompressedBuffer, CompressedBufferSize, FinalUncompressedSize); if (NT_SUCCESS(ret)) { DoOutputDebugString("RtlDecompressBuffer hook: scanning region 0x%x size 0x%x for PE image(s).\n", UncompressedBuffer, *FinalUncompressedSize); DumpPEsInRange(UncompressedBuffer, *FinalUncompressedSize); } return ret; } ``` ----- ## 1 |Col1|‘Injection’ Package|Col3| |---|---|---| |||| |• • •|Captures payloads injected into other proces Intelligence Requirements Uses API hooks (IR’s) • Track newly created processes and threa • Injected directly Priority Intelligence • WriteProcessMemory, NtWriteVirtu Requirements • Process hollo(P wIR i’ ns) g • NtMapViewOfSection • Transacted hollowing Specific Intelligence Requirements (SIR’s) Dumps prior to execution • NtResumeProcess/NtResumeThread 1|ses ds alMemory, etc| |||| ###### Intelligence Requirements Uses API hooks (IR’s) • Track newly created processes and threads • Injected directly Priority Intelligence • WriteProcessMemory, NtWriteVirtualMemory, etc Requirements • Process hollowing (PIR’s) • NtMapViewOfSection • Transacted hollowing Specific Intelligence Requirements (SIR’s) ###### Dumps prior to execution • NtResumeProcess/NtResumeThread ----- ## 1 |‘Extraction’ Package|Col2|Col3| |---|---|---| |• Captures payloads ‘extracted’ inside processes Intelligence Requirements • Tracks executable memory regions (IR’s) • Newly allocated • New executable permissions Priority Intelligence Requirements • Uses debugger br(ePaIRk’sp) oints • Write breakpoints • Capture payloads just after they have been written Specific Intelligence • Execut iR oe nqu i bre rm een ats k points (SIR’s) • Capture payloads before they are executed||| ||Captures payloads ‘extracted’ inside processe Intelligence Requirements Tracks executable memory regions (IR’s) • Newly allocated • New executable permissions Priority Intelligence Requirements Uses debugger br(ePaIRk’sp) oints • Write breakpoints • Capture payloads just after they Specific Intelligence • Execut iR oe nqu i bre rm een ats k points (SIR’s) • Capture payloads before they ar|| ||1|| ###### Intelligence Requirements Tracks executable memory regions (IR’s) • Newly allocated • New executable permissions Priority Intelligence Requirements Uses debugger breakpoints (PIR’s) • Write breakpoints • Specific Intelligence ###### • Execution breakpoints Requirements (SIR’s) • Capture payloads before they are executed ----- ##### Emotet ###### Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) ## 1 |Col1|Col2|Col3|Col4| |---|---|---|---| ||||| ||||| ||||| ###### Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) Specific Intelligence Requirements (SIR’s) ----- ##### Emotet Automated Unpacking ###### Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) ``` AllocationHandler: Adding allocation to tracked region list: 0x003E0000, size: 0x11000. ``` Specific Intelligence ``` ActivateBreakpoints: TrackedRegion->AllocationBase: 0x003E0000, TrackedRegion->RegionSize: 0x11000, thread 2664 Requirements ``` (SIR’s) ``` SetDebugRegister: Setting breakpoint 0 hThread=0xdc, Size=0x2, Address=0x003E0000 and Type=0x1. SetThreadBreakpoint: Set bp 0 thread id 2664 type 1 at address 0x003E0000, size 2 with Callback 0x74af7510. ActivateBreakpoints: Set write breakpoint on empty protect address: 0x003E0000 ## 1 ``` |Col1|Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s)|Col3|Col4|Col5| |---|---|---|---|---| |||Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s)||| |||AllocationHandler: Adding allocation to tracked re Specific Intelligence ActivateBreakpoints: RTerqaucirekmedeRnetsg ion->AllocationBase (SIR’s) SetDebugRegister: Setting breakpoint 0 hThread=0xd SetThreadBreakpoint: Set bp 0 thread id 2664 type ActivateBreakpoints: Set write breakpoint on empty|gion l : 0x00 c, Siz 1 at a prote 1|ist: 0x003E0000, size: 0x11000. 3E0000, TrackedRegion->RegionSize: 0x11000, thread 2664 e=0x2, Address=0x003E0000 and Type=0x1. ddress 0x003E0000, size 2 with Callback 0x74af7510. ct address: 0x003E0000| |||||| ###### Intelligence Requirements (IR’s) Priority Intelligence Requirements (PIR’s) ----- ##### Emotet Automated Unpacking ``` CAPEExceptionFilter: breakpoint hit by instruction at 0x75FE9B60 (thread 2664) BaseAddressWriteCallback: Breakpoint 0 at Address 0x003E0000. ContextSetDebugRegister: Setting breakpoint 2 within Context, Size=0x0, Address=0x003E0000 and Type=0x0. BaseAddressWriteCallback: byte written to 0x3e0000: 0x1d. Intelligence Requirements BaseAddressWriteCallback: Exec bp set on tracked region protect address. (IR’s) CAPEExceptionFilter: breakpoint hit by instruction at 0x003E0000 (thread 2664) ShellcodeExecCallback: Breakpoint 2 at Address 0x003E0000 - about to scan region for a PE image (base 0x003E0000, size 0x11000). ###### Priority Intelligence ScanForDisguisedPE: PE image located at: 0x3e053f Requirements DumpImageInCurrentProcess: Attempting to dump 'raw' PE image. (PIR’s) DumpPE: PE file in memory dumped successfully ``` Specific Intelligence Requirements (SIR’s) ## 1 |Col1|Col2|Col3|Col4|Col5| |---|---|---|---|---| |CA Ba Co Ba Ba CA Sh Sc Du|P s n s s P e a m|EExceptionFilter: breakpoint hit by instruction a eAddressWriteCallback: Breakpoint 0 at Address 0x textSetDebugRegister: Setting breakpoint 2 within Intelligence Requirements eAddressWriteCallback: byte written to 0x3e0000: eAddressWriteCallback:( IREx’esc) bp set on tracked reg EExceptionFilter: breakpoint hit by instruction a llcodeExecCallback: Breakpoint 2 at Address 0x003 Priority Intelligence nForDisguisedPE: PE image located at: 0x3e053f Requirements pImageInCurrentProcess: Attempting to dump 'raw' (PIR’s)|t 0x75F 003E000 Contex 0x1d. ion pro t 0x003 E0000 - PE imag|E9B60 (thread 2664) 0. t, Size=0x0, Address=0x003E0000 and Type=0x0. tect address. E0000 (thread 2664) about to scan region for a PE image (base 0x003E0000, size 0x11000). e.| ||mpPE: PE file in memory dumped successfully Specific Intelligence Requirements (SIR’s) 1|pPE: PE file in memory dumped successfully Specific Intelligence Requirements (SIR’s)||| ## 1 Specific Intelligence Requirements (SIR’s) ----- ##### Emotet Config Extraction ``` rule Emotet { meta: cape_type = "Emotet Payload" Intelligence Requirements ``` `strings:` (IR’s) ``` $snippet = {33 C0 21 05 ?? ?? ?? ?? A3 ?? ?? ?? ?? 39 05 ?? ?? ?? ?? 74 18 40 A3 ?? ?? ?? ?? 83 3C C5 ?? ?? ?? ?? 00 75} condition: uint16(0) == 0x5A4D and ($snippet) ###### Priority Intelligence } Requirements (PIR’s) refc2list = yara_scan(filebuf, '$snippet') c2list_va_offset = int(refc2list['$snippet']) c2_list_va = struct.unpack('i', filebuf[c2list_va_offset+15:c2list_va_offset+19])[0] c2_list_rva = c2_list_va - image_base Specific Intelligence ``` Requirements ``` c2_list_offset = pe.get_offset_from_rva(c2_list_rva) ``` (SIR’s) ``` while 1: ip = struct.unpack('ContextRecord->Esp+4*3); Requirements (PIR’s) DumpAddress = (PVOID)*(DWORD*)((BYTE*)ExceptionInfo->ContextRecord->Esp+4*4); CAPEExceptionFilter: breakpoint hit by instruction at 0x004054AF 0x4054af (05) e86c080000 CALL 0x871 Specific Intelligence Trace: CALL detected, grabbing size 0x2b and buffer 0x3bcff10 from stack. Requirements ``` (SIR’s) ``` CAPEExceptionFilter: breakpoint hit by instruction at 0x004054B4 0x4054b4 (03) 83c414 ADD ESP, 0x14 DumpMemory: CAPE output file successfully created: C:\iaDUZcSim\CAPE\2580_70260233749341954122019 Added new CAPE file to list with path: C:\iaDUZcSim\CAPE\2580_70260233749341954122019 Trace: dumped QakBot config from 0x3bcff10. ## 1 ``` |Col1|Col2|Col3| |---|---|---| |• Du Du CA 0x Tr CA 0x Du Ad Tr|Dedicated family package triggered • BreIantkeplloigienntcse: Requirements (IR’s) • Anti-VM bypass • Dump 2 x config region Priority Intelligence Requirements mpSize = (SIZE_T)*(DWORD*)((BYTE*)ExceptionInfo->ContextR (PIR’s) mpAddress = (PVOID)*(DWORD*)((BYTE*)ExceptionInfo->Contex PEExceptionFilter: breakpoint hit by instruction at 0x004 4054af (05) e86c080000 CALL 0x871 Specific Intelligence ace: CALL detected, gRreaqbubirienmge nstsi ze 0x2b and buffer 0x3bcff (SIR’s) PEExceptionFilter: breakpoint hit by instruction at 0x004 4054b4 (03) 83c414 ADD ESP, 0x14 mpMemory: CAPE output file successfully created: C:\1iaDUZ ded new CAPE file to list with path: C:\iaDUZcSim\CAPE\25 ace: dumped QakBot config from 0x3bcff10.|ecord->Esp+4*3); tRecord->Esp+4*4); 054AF 10 from stack. 054B4 cSim\CAPE\2580_70260233749341954122019 80_70260233749341954122019| ----- ##### The Future… ###### • CAPE v2 just released ## 1 |• CAPE v2 just released • Python 3 • A huge thank you to Andriy Brukhove Intelligence Requirements • https://github.com/kevoreilly/CAPEv2 (IR’s) • (KVM hardended anti-anti-vm: https://github.com/doomedraven/To Priority Intelligence Requirements • Behavioural packages combined (PIR’s) • Enabled by default • Reduce executions to maximum of 2 Specific Intelligence Requirements (SIR’s) • Expanded Debugger options • Family-specific “package” all within Y|Col2|Col3|tskyy (@d00m3dr4v3n) – FireEye ols/blob/master/Virtualization/kvm-qemu.sh) ARA signature| |---|---|---|---| ||• Python 3 • A huge thank you to Andriy Bruk Intelligence Requirements • https://github.com/kevoreilly/C (IR’s) • (KVM hardended anti-anti-vm: https://github.com/doomedra Priority Intelligence Requirements • Behavioural packages combined (PIR’s) • Enabled by default • Reduce executions to maximum Specific Intelligence Requirements (SIR’s) • Expanded Debugger options • Family-specific “package” all wit||| |||ven/To|| |||of 2 hin Y|| |1|||| ###### • Intelligence Requirements • https://github.com/kevoreilly/CAPEv2 (IR’s) • (KVM hardended anti-anti-vm: Priority Intelligence Requirements Behavioural packages combined (PIR’s) • Enabled by default • Reduce executions to maximum of 2 Specific Intelligence Requirements (SIR’s) ###### Expanded Debugger options • Family-specific “package” all within YARA signature ----- ## 1 |Col1|Ursnif/ISFB|Col3| |---|---|---| |rule { m dumpt dump- s E0 01 00} 08 41 }|Ursnif3 Intelligence Requirements eta: (IR’s) cape_type = "Ursnif Payload" cape_options = "dll=Debugger.dll,step-out=$cryp ype0=0x24, base-on-api=RtlAddVectoredExceptionHandle on-api-type=0x25" Priority Intelligence trings: Requirements $crypto32 = {8B C3 83 EB 01 85 C0 75 0D 0F B6 16 8 (PIR’s) 03 D2 85 C0 0F 84 AB 01 00 00 8B C3 83 EB 01 85 C0 8 $golden_ratio = {8B 70 EC 33 70 F8 33 70 08 33 30 81 F9 84 00 00 0S0p}e cific Intelligence Requirements condition: (SIR’s) uint16(0) == 0x5A4D and (all of them)|to32, dumpsize=eax,action0=dumpebx, r, dump-on-api=RtlAddVectoredExceptionHandler, 3 C6 01 89 74 24 14 8D 58 07 8B C2 C1 E8 07 83 9 5C 24 20 75 13 0F B6 16 83 C6 01 BB 07 00 00 83 C0 04 33 F1 81 F6 B9 79 37 9E C1 C6 0B 89 70| ||1|| ----- ## Botnet Emulator Framework ----- ###### y • Botnet participation allows retrieval of artefacts and context unavailable from samples • Long-term tracking of targeting and operational details • Vetting indicators with high-fidelity • Synergy with config extraction for class of malware with features unavailable in ordinary sandbox detonation ----- ----- ###### • Less data than you think to store years of botnet interactions • Save everything! verbose logging, HTTP sessions, etc. • Database is under 100 GB • Local EBS volume is 100 GB • S3 storage is 230 GB ----- ###### Cost •• EC2: $30/mo S • RDS: $60/mo • S3: $5/mo • EBS: $10/mo • VPS: $30/mo • VPN: $20/mo # $255 ### per month ----- ###### • Make traffic and behaviour near replica of actual bot • Generate plausible but contrived metadata (e.g., computer and domain names) • Withstand competent non-state investigation • Non-attributable infrastructure unless subject to subpoena • Log timestamps and egress IP addresses for future correlation ----- ###### g • Use a combination of Tor, VPS, and commercial VPNs • Geographic diversity is great for empirically finding geofencing • Simple round robin is largely suitable • VPN microservice allows an emulator to request specific regions/countries and capabilities ----- ###### g • Don’t (intentionally) trash someone else’s IP space • VPNs WILL ban you (no refund) • Beware the Internet do-gooder • Few if any allow outbound TCP 25 (necessary for Cutwail) • By product is good intel source of VPN exit nodes ----- ###### • “ChatOps” notifications to analysts • Email distribution lists (third parties, LE, working groups) • Intelligence platforms • High fidelity blacklists ----- ###### y • OpenSSH SOCKS5 server is unstable during high throughput • Use Dante, open source SOCKS5 server • Protect with iptables and configuration-level filters and authentication ----- ###### y • Reasons to not use Requests • Protocol violations ⇉ Uncommon in malware • Absent SOCKS support ⇉ Added in 2016 • Custom header ordering ⇉ Use OrderedDict() • Provide IP address for request ⇉ Patch Requests • Access to endpoint SSL certificate ⇉ Patch Requests ----- ###### g • Nature of system allows tracking of DNS resolutions over time • Augment this data with passive DNS (Farsight) • Useful for C2 with flaky DNS but stable hosting • Need to be careful with C2 living in fast-flux systems ----- ###### g • Save unique certificates as they appear and associate with IP • Flowsynth to generate PCAP from X.509, check coverage • Augment other data sets like Censys, Shodan, or SONAR ----- ###### g • Regularly retrieve sinkhole list from abuse.ch SinkDB • Detect common sinkhole-related HTTP features • Avoiding sinkholes saves execution times and prevents poisoning other researchers’ data • BitSight won’t track us down and try to sell us a report about our emulator being infected with dangerous malware ----- ###### g • Always prune your list of C2 servers • General formula: has the C2 accepted connections in two weeks? Given a valid response in past month? ----- ###### • Register bot, interrogate C2, dispose of bot • Instead, store registered bots and periodically reconnect them to create pool of long-lived “infections” • Good way to get additional payloads to “mature” bots -----