Developments in IOS Forensics Developments in Cisco IOS Forensics Felix ‘FX’ Lindner BlackHat Briefings Washington DC, Feb. 2008 Agenda � IP Routing Infrastructure and Cisco IOS � Cisco IOS Internals � Debugging and Post Mortem Analysis Today � A New Analysis Approach � Proposal � Features � Challenges � Initial Public Offer � Future Work IP Routing Infrastructure � The Internet and corporate networks almost exclusively run on the Internet Protocol � IP Version 4 is still prevalent protocol � IP Version 6 coming up very slowly � The design of IP requires intelligent nodes in the network to make routing decisions � This is a design principle of the protocol and cannot be changed � “Flat” networks have their own issues IP Infrastructure & Security � All security protocols on top of IP share common design goals: � Guarantee end-to-end integrity (some also confidentiality) of the traffic � Detect modification, replay, injection and holding back of traffic � Inform the upper protocol layers � None of them can recover from attacks rooted in the routing infrastructure � Security protocols cannot influence routing Infrastructure Monoculture � Cisco Systems’ routing platforms form the single largest population of networking equipment today � Equivalently distributed in the Internet core, government and corporate networks � Many different hardware platforms with different CPUs � Large investment sums bound to the equipment � Hard to replace � All run basically the same operating system � Protecting this infrastructure is critical � Therefore, in-depth analysis and diagnostics are of paramount importance Cisco IOS � Cisco® Internetwork Operating System® � Monolithic operating system � Compile-time linked functionality – the 3 dimensional complexity of IOS � Platform dependent code � Feature-set dependent code � Major, Minor and Release version dependent code � Several tens of thousands different IOS images used in today’s networks � Over 10.000 still officially supported Inside Cisco IOS � One large ELF binary � Essentially a large, statically linked UNIX program � Loaded by ROMMON, a kind-of BIOS � Runs directly on the router’s main CPU � If the CPU provides virtual memory and privilege separation (for example Supervisor and User mode on MIPS), it will not be used Inside Cisco IOS � Processes are rather like threads � No virtual memory mapping per process � Run-to-completion, cooperative multitasking � Interrupt driven handling of critical events � System-wide global data structures � Common heap � Very little abstraction around the data structures � No way to force abstraction The IOS Code Security Issue � 12.4(16a) with enterprise base feature set consists of 25.316.780 bytes binary code! � This is a 2600 with PowerPC CPU � Not including 505.900 bytes firmware for E1T1 and initialization � All written in plain C � Sharing the same address space � Sharing the same heap � Sharing the same data structures � Sharing millions of pointers The IOS Code Security Issue � A single mistake in the most unimportant piece of code can influence anything on the system, including kernel, security subsystems and cryptographic code. � Therefore, everything on IOS is a good target for remote code execution exploits in kernel context. Isn’t Cisco aware of that? � Cisco recently started the distribution of the next generation IOS-XR � Commercial QNX microkernel � Real processes (memory protection?) � Concurrent scheduling � Significantly higher hardware requirements � People never use the latest IOS � Production corporate networks usually run on 12.1 or 12.2, which 12.5 is already available � Not even Cisco’s own engineers would recommend the latest IOS release to a customer � That only covers people actively maintaining their network, not everyone running one Just, how often are routers hacked? � Keynote speaker Jerry Dixon mentioned not updated routers as a cause for concern � Do you know how expensive that is? � Old vulnerabilities like the HTTP level 16 bug are still actively scanned for � The router is used as a jump pad for further attacks � TCL backdoors are commonly used � Patched images are not rare � IOS images cost money � People will use images from anywhere � Patching images is not hard � Lawful Interception is its own can of worms � The router’s operator is not supposed to know that LI is performed � Who watches the watchers? And the future? � Ever noticed attackers take on the target with the lowest efforts required and the highest return of invest? � Windows became just a lot harder � UNIXes are hardened, even OS X � Infected PCs leave obvious traces � The question is not: “Will routers become a target?” � The question should be: “Do we want to know when they did?” Summary – Part I � A significant share of the Internet, governmental and corporate networks runs on: � one out of several tens of thousands of builds � of more or less the same code base � in a single process environment ... and we cannot bypass it, even if we could tell that it’s compromised Next question: How can we even tell? Error Handling and Recovery � The software architecture of IOS dictates how exception handling has to be done � Remember, IOS is like a large UNIX process � What happens when a UNIX process segfaults? � Upon an exception, IOS can only restart the entire system � Even on-board, scheduled diagnostic processes can only forcefully crash the system Crash Cause Evidence � Reboot is a clean recovery method � Reboot destroys all volatile evidence of the crash cause � Everything on the router is volatile! � Exception: startup configuration and IOS image � Later IOS releases write an information file called “crashinfo” � Crashinfo contains very little information � Contents depend on what IOS thought was the cause of the crash Runtime Evidence � Crashinfo is only written upon device crashes � Successful attacks don’t cause device crashes � The available methods are: � Show commands � Debug commands � SNMP monitoring � Syslog monitoring Show Commands � IOS offers a plethora of inspection commands known as the “show” commands � Requires access to the command line interface � Geared towards network engineers � Thousands of different options and versions � Almost no access to code � 12.4 even limits memory show commands Debug Commands � “debug” enables in-code debugging output � Debug output has scheduler precedence � Too much debug output halts the router � Not an option in production environments � Enabling the right debug output is an art � Turn on the wrong ones and you see very little � Turn on too many and the router stops working � Commands depend on the IOS version � For debug commands to be useful, you have to know what you are looking for before it happens � Not very useful for security analysis SNMP and Syslog Monitoring � Commonly accepted method for monitoring networking equipment � SNMP depending on the implemented MIB � Geared towards networking functionality � Very little process related information � Syslog is about as useful for security monitoring on IOS as it is on UNIX systems � Both generate continuous network traffic � Both consume system resources on the router � Then again, someone has to read the logs. Summary – Part II � Identifying compromised routers using today’s tools and methods is hard, if not impossible. � There is not enough data to perform any post mortem analysis of router crashes, security related or not. � We cannot distinguish between a functional problem, an attempted attack and a successful attack on infrastructure running IOS. A (not so) New Approach � We need the maximum amount of evidence � A full snapshot of the device is just enough � We don’t need it continuously � We need it on-demand � We need it when the device crashes � We need an independent and solid analysis framework to process the evidence � We need to be able to extend and adjust it Getting the Evidence � Cisco IOS can write complete core dumps � Memory dump of the main memory � Memory dump of the IO memory � Memory dump of the PCI memory (if applicable) � Core dumps are written in two cases � The device crashes � The user issues the “write core” command Core Dump Destinations � IOS supports various destinations � TFTP server (bug!) � FTP server � RCP server � Flash file system (later IOS releases) � Core dumps are enabled by configuration � Configuration commands do not differ between IOS versions � Configuration change has no effect on the router’s operation or performance Core Dump Enabled Infrastructure � Configure all IOS devices to dump core onto one or more centrally located FTP servers � Minimizes required monitoring of devices: A router crashed if you find a core dump on the FTP server � Preserves evidence � Allows crash correlation between different routers � Why wasn’t it used before? � Core dumps were useless, except for Cisco developers and exploit writers. Analyzing Core Dumps Disclaimer: � Any of the following methods can be implemented in whatever your preferred programming language is. � This presentation will be centric to our implementation: Recurity Labs CIR. Core Dump Analyzer Requirements � Must be 100% independent � No Cisco code � No disassembly based analysis � Must gradually recover abstraction � No assumptions about anything � Ability to cope with massively corrupted data � Should not be exploitable itself � Preferably not written in C The Image Blueprint � The IOS image (ELF file) contains all required information about the memory mapping on the router. � The image serves as the memory layout blueprint, to be applied to the core files � Using a known-to-be-good image also allows verification of the code and read-only data segments � Now we can easily and reliably detect runtime patched images Heap Reconstruction � IOS uses one large heap � The IOS heap contains plenty of meta-data for debugging purposes � 40 bytes overhead per heap block in IOS up to 12.3 � 48 bytes overhead per heap block in IOS 12.4 � Reconstructing the entire heap allows extensive integrity and validity checks � Exceeding by far the on-board checks IOS performs during runtime Heap Verification � Full functionality of “CheckHeaps” � Verify the integrity of the allocated and free heap block doubly linked lists � Find holes in addressable heap � Invisible to CheckHeaps � Identify heap overflow footprints � Values not verified by CheckHeaps � Map heap blocks to referencing processes � Identify formerly allocated heap blocks � Catches memory usage peaks from the recent past Process List � Extraction of the IOS Process List � Identify the processes’ stack block � Create individual, per process back-traces � Identify return address overwrites � Obtain the processes’ scheduling state � Obtain the processes’ CPU usage history � Obtain the processes’ CPU context � Almost any post mortem analysis method known can be applied, given the two reconstructed data structures. TCL Backdoor Detection � TCL scripting is available on later Cisco IOS versions � TCL scripts listening on TCP sockets � Well known method � Used to simplify automated administration � Used to silently keep privileged access to routers � Known bug: not terminated when the VTY session ends (fixed) � Simple TCL backdoor scripts published � CIR can extract all TCP script chunks from IOS heap and dump them for further analysis Your Wishes Please ! � Interface tables � Static routing � Router processes � VLAN tables � VPN context � Keys � User � IPv6 tables � ARP tables � Dialer tables � CDP tables � Spanning tree � Access Lists � CEF Tree � User sessions � Listening ports � Callback code for incoming connections � Verification that it is located in .TEXT IOS Packet Forwarding Memory � IOS performs routing either as: � Process switching � Fast switching � Particle systems � Hardware accelerated switching � Except hardware switching, all use IO memory � IO memory is written as separate code dump � By default, about 6% of the router’s memory is dedicated as IO memory � In real world installations, it is common to increase the percentage to speed up forwarding � Hardware switched packets use PCI memory � PCI memory is written as separate core dump IO Memory Buffers � Routing (switching) ring buffers are grouped by packet size � Small � Medium � Big � Huge � Interfaces have their own buffers for locally handled traffic � IOS tries really hard to not copy packets around in memory � New traffic does not automatically erase older traffic in a linear way Traffic Extraction � CIR dumps packets that were process switched by the router from IO memory into a PCAP file � Traffic addressed to and from the router itself � Traffic that was process switching inspected � Access List matching � QoS routed traffic � CIR could dump packets that were forwarded through the router too � Reconstruction of packet fragments possible � Is it desirable? Advanced Traffic Extraction � Writing core to a remote server uses IO memory � Overwrites part of the traffic evidence � CIR can use a GDB link instead of a core dump � Serial GDB protocol allows direct access to router memory via the console � Uses Zynamics GDB debug link � Disconnecting all network interfaces preserves IO and PCI memory contents � Using GDB halts the router � All data is preserved – useful for emergency inspections Traffic Extraction Applications � Identification of attack jump pad routers � 0day identification against systems on segmented network interfaces � If you got the packet, you got the 0day � Spoofing attack backtracking � One hop at the time, obviously � LE detection Challenges � The analysis framework has to handle the complexity of the Cisco IOS landscape � Hardware platforms � Image versions � Any-to-Any relation! � CIR is currently IOS feature set independent � CIR successfully tested against IOS 12.0 – 12.5 � CIR currently supports � Cisco 1700 � Cisco 2600 � Cisco 3600 (upcoming) � Cisco 7200 (upcoming) � Your wishes decide the course. Summary – Part III � Writing core dumps is a viable method for obtaining IOS evidence when it is needed. � The evidence includes forwarded and received packets. � An independent analysis framework can distinguish between bugs and attacks, enabling real forensics on IOS routers. � Recurity Labs’ CIR already reliably identifies many types of attacks and IOS backdoors. � CIR is work-in-progress � CIR’s future depends on the feedback we receive from the community. Initial Public Offer � An analysis framework’s quality is directly related to the amount of cases it has seen � CIR needs a lot more food to grow up � We want to provide it to everyone while constantly developing and improving it � Free Service: http://cir.recurity-labs.com � Processing on our servers � Always using the latest version � Please be gentle, it’s the version � Given enough interest, there will be a professional tool in the future α At the end, it’s all up to you! � We think CIR could be useful � For the networking engineer � For the forensics professional � To finally know the state of our infrastructure � We can think of way too many things � Platforms � Features � Reports � Please help ☺ cir.recurity-labs.com Felix ´FX´ Lindner Head fx@recurity-labs.com Recurity Labs GmbH, Berlin, Germany http://www.recurity-labs.com