Lindner Briefings Developments in Felix ‘FX’BlackHatCisco IOS Forensics Washington DC, Feb. 2008 Agenda ProposalFeaturesChallenges IP Routing Infrastructure and Cisco IOSCisco IOS InternalsDebugging and Post Mortem Analysis TodayA New Analysis ApproachInitial Public OfferFuture Work networks have their own issues IP Version 4 is still prevalent protocolIP Version 6 coming up very slowlyThis is a design principle of the protocol and cannot be changed“Flat”IP Routing InfrastructureThe Internet and corporate networks almost exclusively run on the Internet ProtocolThe design of IP requires intelligent nodes in the network to make routing decisions Guarantee end-to-end integrity (some also confidentiality) of the trafficDetect modification, replay, injection and holding back of trafficInform the upper protocol layersSecurity protocols cannot influence routing All security protocols on top of IP share common design goals:None of them can recover from attacks rooted in the routing infrastructureIP Infrastructure & Security routing platforms form the single Equivalently distributed in the Internet core, government and corporate networksMany different hardware platforms with different CPUsLarge investment sums bound to the equipmentHard to replaceAll run basically the same operating system Cisco Systems’largest population of networking equipment todayProtecting this infrastructure is criticalTherefore, in-depth analysis and diagnostics are of paramount importanceInfrastructure Monoculture IOS images Operating System® Cisco IOS Internetwork Cisco®Monolithic operating systemCompile-time linked functionality – Platform dependent code the 3 dimensional complexity of IOS tens of thousands different Feature-set dependent codeMajor, Minor and Release version dependent codeOver 10.000 still officially supported used in today’s networksSeveral Inside Cisco IOS Loaded by ROMMON, a kind-of BIOSIf the CPU provides virtual memory and privilege separation (for example Supervisor and User mode on MIPS), it will not be used One large ELF binaryEssentially a large, statically linked UNIX programRuns directly on the router’s main CPU Inside Cisco IOS No virtual memory mapping per processInterrupt driven handling of critical eventsCommon heapVery little abstraction around the data structuresNo way to force abstraction Processes are rather like threadsRun-to-completion, cooperative multitaskingSystem-wide global data structures This is a 2600 with PowerPC CPUNot including 505.900 bytes firmware for E1T1 and initialization 12.4(16a) with enterprise base feature set consists of 25.316.780 bytes binary code!All written in plain CSharing the same address spaceSharing the same heapSharing the same data structuresSharing millions of pointers The IOS Code Security Issue on IOS is a good target everything A single mistake in the most unimportant piece of code can influence anything on the system, including kernel, security subsystems and cryptographic code.Therefore, for remote code execution exploits in kernel context. The IOS Code Security Issue Commercial QNX microkernelReal processes (memory protection?)Concurrent schedulingSignificantly higher hardware requirementsProduction corporate networks usually run on 12.1 or 12.2, which 12.5 is already availableNot even Cisco’s own engineers would recommend the latest IOS release to a customerThat only covers people actively maintaining their network, not everyone running one Isn’t Cisco aware of that?Cisco recently started the distribution of the next generation IOS-XRPeople never use the latest IOS ed to know that LI is performed Just, how often are routers hacked? Who watches the watchers?Do you know how expensive that is?The router is used as a jump pad for further attacksIOS images cost moneyPeople will use images from anywherePatching images is not hardThe router’s operator is not suppos Keynote speaker Jerry Dixon mentioned not updated routers as a cause for concernOld vulnerabilities like the HTTP level 16 bug are still actively scanned forTCL backdoors are commonly usedPatched images are not rareLawful Interception is its own can of worms And the future?are hardened, even OS X Windows became just a lot harderUNIXesInfected PCs leave obvious traces Ever noticed attackers take on the target with the lowest efforts required and the highest return of invest? The question is not: “Will routers become a target?”The question should be: “Do we want to know when they did?” Part I How can we even tell? Summary – one out of several tens of thousands of buildsof more or less the same code basein a single process environment A significant share of the Internet, governmental and corporate networks runs on:... and we cannot bypass it, even if we could tell that it’s compromisedNext question: Remember, IOS is like a large UNIX processWhat happens when a UNIX process segfaults?Even on-board, scheduled diagnostic processes can only forcefully crash the system Error Handling and RecoveryThe software architecture of IOS dictates how exception handling has to be doneUpon an exception, IOS can only restart the entire system contains very little information Crash Cause EvidenceEverything on the router is volatile!Exception: startup configuration and IOS imageCrashinfoContents depend on what IOS thought was the cause of the crash Reboot is a clean recovery methodReboot destroys all volatile evidence of the crash causeLater IOS releases write an information file called “crashinfo” is only written upon device crashes monitoring Runtime Evidence Show commandsDebug commandsSNMP monitoringSyslog CrashinfoSuccessful attacks don’t cause device crashesThe available methods are: commands Show Commands Requires access to the command line interface12.4 even limits memory show commands IOS offers a plethora of inspection commands known as the “show”Geared towards network engineersThousands of different options and versionsAlmost no access to code before it happens enables in-code debugging outputDebug Commands Too much debug output halts the routerNot an option in production environments Turn on the wrong ones and you see very littleTurn on too many and the router stops workingCommands depend on the IOS versionNot very useful for security analysis “debug”Debug output has scheduler precedenceEnabling the right debug output is an artFor debug commands to be useful, you have to know what you are looking for Monitoring is about as useful for security monitoring on Geared towards networking functionalityVery little process related information SNMP and SyslogCommonly accepted method for monitoring networking equipmentSNMP depending on the implemented MIBSyslogIOS as it is on UNIX systemsBoth generate continuous network trafficBoth consume system resources on the routerThen again, someone has to read the logs. Part II Summary – Identifying compromised routers using today’s tools and methods is hard, if not impossible.There is not enough data to perform any post mortem analysis of router crashes, security related or not.We cannot distinguish between a functional problem, an attempted attack and a successful attack on infrastructure running IOS. A full snapshot of the device is just enoughWe need it on-demand We need it when the device crashesWe need to be able to extend and adjust itA (not so) New ApproachWe need the maximum amount of evidenceWe don’t need it continuouslyWe need an independent and solid analysis framework to process the evidence command Getting the Evidence Memory dump of the main memoryMemory dump of the IO memoryMemory dump of the PCI memory (if applicable)The device crashesThe user issues the “write core” Cisco IOS can write complete core dumpsCore dumps are written in two cases TFTP server (bug!)FTP serverRCP serverFlash file system (later IOS releases)Configuration commands do not differ between IOS versionsConfiguration change has no effect on the router’s operation or performanceCore Dump Destinations IOS supports various destinationsCore dumps are enabled by configuration Core Dump Enabled Infrastructure Minimizes required monitoring of devices: A router crashed if you find a core dump on the FTP serverPreserves evidenceAllows crash correlation between different routersCore dumps were useless, except for Cisco developers and exploit writers. Configure all IOS devices to dump core onto one or more centrally located FTP serversWhy wasn’t it used before? Analyzing Core Dumps Any of the following methods can be implemented in whatever your preferred programming language is.This presentation will be centric to our implementation: Recurity Labs CIR.Disclaimer: Core Dump Analyzer Requirements No Cisco codeNo disassembly based analysisNo assumptions about anythingAbility to cope with massively corrupted dataPreferably not written in C Must be 100% independentMust gradually recover abstractionShould not be exploitable itself The Image Blueprint The image serves as the memory layout blueprint, to be applied to the core filesNow we can easily and reliably detect runtime patched images The IOS image (ELF file) contains all required information about the memory mapping on the router. Using a known-to-be-good image also allows verification of the code and read-only data segments Heap Reconstruction 40 bytes overhead per heap block in IOS up to 12.348 bytes overhead per heap block in IOS 12.4Exceeding by far the on-board checks IOS performs during runtime IOS uses one large heapThe IOS heap contains plenty of meta-data for debugging purposesReconstructing the entire heap allows extensive integrity and validity checks Heap Verification Verify the integrity of the allocated and free heap block doubly linked listsInvisible to CheckHeapsValues not verified by CheckHeapsCatches memory usage peaks from the recent past Full functionality of “CheckHeaps”Find holes in addressable heapIdentify heap overflow footprintsMap heap blocks to referencing processesIdentify formerly allocated heap blocks stack blockscheduling stateCPU usage historyCPU context Process List Create individual, per process back-tracesIdentify return address overwrites Identify the processes’Obtain the processes’Obtain the processes’Obtain the processes’ Extraction of the IOS Process ListAlmost any post mortem analysis method known can be applied, given the two reconstructed data structures. Well known methodUsed to simplify automated administrationUsed to silently keep privileged access to routersKnown bug: not terminated when the VTY session ends (fixed)Simple TCL backdoor scripts publishedTCL Backdoor Detection TCL scripting is available on later Cisco IOS versionsTCL scripts listening on TCP sockets CIR can extract all TCP script chunks from IOS heap and dump them for further analysis Callback code for incoming connectionsVerification that it is located in .TEXT CDP tablesSpanning treeAccess ListsCEF TreeUser sessionsListening ports Your Wishes Please ! KeysUser Interface tablesStatic routingRouter processesVLAN tablesVPN contextIPv6 tablesARP tablesDialer tables is common to increase the percentage IOS Packet Forwarding Memory In real world installations, it to speed up forwarding Process switchingFast switchingParticle systemsHardware accelerated switchingIO memory is written as separate code dumpBy default, about 6% of the router’s memory is dedicated as IO memoryPCI memory is written as separate core dump IOS performs routing either as:Except hardware switching, all use IO memoryHardware switched packets use PCI memory are grouped by ring buffers IO Memory Buffers SmallMediumBigHuge packet sizeRouting (switching) Interfaces have their own buffers for locally handled trafficIOS tries really hard to not copy packets around in memoryNew traffic does not automatically erase older traffic in a linear way routed trafficTraffic Extraction Access List matchingQoS Traffic addressed to and from the router itselfTraffic that was process switching inspectedReconstruction of packet fragments possibleIs it desirable? CIR dumps packets that were process switched by the router from IO memory into a PCAP fileCIR could dump packets that were forwarded through the router too useful for emergency GDB debug link Overwrites part of the traffic evidenceSerial GDB protocol allows direct access to router memory via the consoleUses ZynamicsUsing GDB halts the router Writing core to a remote server uses IO memoryCIR can use a GDB link instead of a core dumpDisconnecting all network interfaces preserves IO and PCI memory contentsAll data is preserved –inspectionsAdvanced Traffic Extraction If you got the packet, you got the 0dayOne hop at the time, obviously Identification of attack jump pad routers0day identification against systems on segmented network interfacesSpoofing attack backtrackingLE detectionTraffic Extraction Applications 12.5 Challenges Hardware platforms Image versionsAny-to-Any relation!Cisco 1700 Cisco 2600Cisco 3600 (upcoming) Cisco 7200 (upcoming) The analysis framework has to handle the complexity of the Cisco IOS landscape CIR is currently IOS feature set independentCIR successfully tested against IOS 12.0 –CIR currently supports Your wishes decide the course. Part III Summary – The evidence includes forwarded and received packets. Writing core dumps is a viable method for obtaining IOS evidence when it is needed.An independent analysis framework can distinguish between bugs and attacks, enabling real forensics CIR already reliably identifies many future depends on the feedback we receive from the CIR is work-in-progressCIR’scommunity. on IOS routers.Recurity Labs’types of attacks and IOS backdoors. α http://cir.recurity-labs.com Initial Public Offer CIR needs a lot more food to grow upWe want to provide it to everyone while constantly developing and improving itProcessing on our serversAlways using the latest versionPlease be gentle, it’s the version An analysis framework’s quality is directly related to the amount of cases it has seenFree Service: Given enough interest, there will be a professional tool in the future ☺ For the networking engineerFor the forensics professionalTo finally know the state of our infrastructurePlatformsFeaturesReports We think CIR could be usefulWe can think of way too many thingsPlease help At the end, it’s all up to you! Lindner Head fx@recurity-labs.com Felix ´FX´ cir.recurity-labs.com http://www.recurity-labs.com Recurity Labs GmbH, Berlin, Germany