{
	"id": "06cfb32e-cdc4-4b13-a977-d74733674f65",
	"created_at": "2026-05-07T02:42:40.092146Z",
	"updated_at": "2026-05-07T02:44:10.945254Z",
	"deleted_at": null,
	"sha1_hash": "145d5ce46a09352ecfbb51d9baee2945f88d5b2a",
	"title": "Linux Threat Hunting Primer — Part II",
	"llm_title": "",
	"authors": "",
	"file_creation_date": "0001-01-01T00:00:00Z",
	"file_modification_date": "0001-01-01T00:00:00Z",
	"file_size": 1221171,
	"plain_text": "Linux Threat Hunting Primer — Part II\r\nBy VerintCyberSec\r\nPublished: 2020-01-05 · Archived: 2026-05-07 02:22:29 UTC\r\n10 min read\r\nJan 5, 2020\r\nBy Shachar Roitman\r\nIn the previous post “Linux Threat Hunting Primer — Part 1” , we discussed how to start the threat hunting\r\nprocess and reviewed the statistical distribution of the Linux tactics and techniques. We also created lists of\r\ntechniques to search for after performing ROI estimation. Moreover, we began to list the different stages required\r\nin the process of threat hunting. In this post, we will describe and demonstrate a full threat hunting process on one\r\nMITRE ATT\u0026CK technique.\r\nAll of the queries I’m going to show will be in the TPSQL language (Verint Threat Protection System (TPS)\r\nQuery Language). Don’t worry, the language is simple to understand.\r\nTPS agents collect data using various techniques including auditd and custom kernel modules. You can refer to\r\nstrace, ptrace and auditd documentation or output on your system to get additional insight about the fields and\r\nqueries we’re going to use in the post.\r\nCredential Dumping — Example of a full threat hunting process\r\nStage 0: Understand the attack and/or technique you’d like to find\r\nThere are two important questions to ask ourselves at this point :\r\n1. What does the attacker want to accomplish when performing the attack?\r\n2. How is he going to do it?\r\nWhat is credential dumping?\r\n“Credential dumping is the process of obtaining account login and password information, normally in\r\nthe form of a hash or a clear text password, from the operating system and software.” [MITRE\r\nATT\u0026CK definition, T1003]\r\nIn Linux, abusing the /proc directory is one of the most common courses of action for this kind of attack. A\r\ncommon technique is reading the memory of a process from /proc/[pid]/maps and dumping/harvesting passwords\r\nusing root privileges.\r\nhttps://medium.com/verint-cyber-engineering/linux-threat-hunting-primer-part-ii-69484f58ac92\r\nPage 1 of 10\n\nAs a demonstration, we will check if we can find plain text passwords in memory dumps. For the following\r\nexample, I am using Linux Memory Extractor (LiMe):\r\nThe “shachar” user is in the sudo group, so this account can be used for privilege escalation. I dumped the\r\nmemory using LiMe and searched for plain text passwords (using strings and grep commands).\r\nFrom the users point of view:\r\nImage 1: The user wanted to perform an action, which required “root” privileges\r\nFrom the attacker point of view (Memory dump):\r\nImages 2–4: I looked for Shachar’s user login action in the memory and for a password pattern\r\nNow that we verified that some plain-text passwords can be found in memory it’s clear that an attacker can also\r\nleverage this to steal passwords. Next, we’ll look at the different ways this attack can be implemented.\r\nStage 1: Find ways to implement this technique\r\nNow that we understand the technique, we’d like to find the variety of ways to implement it. For example, which\r\nsyscalls, files or process are involved in the different stages of the attack?\r\nIn this stage, we will see malicious tools that implement the attack and how they do it. In addition, we will look\r\nfor the attack footprint on the OS.\r\nReading previous research papers on credential dumping (see: http://www.foo.be/cours/mssi-20072008/davidoff-clearmem-linux.pdf) confirms and reinforces the behavior that we witnessed in the previous stage. The research\r\nindicates that an attacker can see the user’s account information in different processes memory, such as:\r\nBy looking at Gnome Display Manager Process memory dump, we can see the Linux login password in\r\nASCII as well as information from /etc/shadow and /etc/passwd. This includes the login shadow password,\r\nusername, long name, UID, GID, home directory, and shell.\r\nThe thunderbird-bin process memory contains the user’s plain text email password, name, email address,\r\nmail server and related information in ASCII format.\r\nhttps://medium.com/verint-cyber-engineering/linux-threat-hunting-primer-part-ii-69484f58ac92\r\nPage 2 of 10\n\nThe cleartext SSH password was stored as ASCII text within a large block of nulls in the memory image.\r\nStage 2: Find anomalies\r\nUsing the information we gathered from the research on the technique. We’ll now search for the suspicious\r\nbehaviors in the network by focusing on anomalies (Look for techniques using multiple dimensions: parent/child\r\nrelationships, command lines arguments, environment variables, accounts, permissions, memory etc.)\r\nThe following examples will help you define your searches:\r\nSearch for processes that use /proc/\u003cpid\u003e/maps, /etc/passwd, /etc/shadow files or modifications of /etc/login.defs\r\nfile which provides the default configuration information for several user account parameters.\r\nPay attention that useradd, usermod, userdel and groupadd system commands as well as other user management\r\nutilities read the login.defs file. We can study the interaction between these commands and login.defs so that we\r\ncan filter out expected behavior (i.e. false positives) in the next steps of our investigation\r\nLook for users that performed file activity on memory dump files that were created by the OS.\r\nNormally, memory crash files are located in /var/crash, but can also be found in /var/spool or\r\n/var/lib/systemd/coredump\r\nkdump is a kernel crash dumping utility. This utility can be enabled using a systemctl command. We can look for\r\ncommands like:\r\n$ systemctl enable kdump.service\r\n$ systemctl start kdump.service\r\nIn addition, we can check for modifications to coredump sysctl config /etc/sysctl.d/50-coredump.conf. Following\r\nare additional files that can be manipulated for malicious dumping:\r\nhttps://medium.com/verint-cyber-engineering/linux-threat-hunting-primer-part-ii-69484f58ac92\r\nPage 3 of 10\n\n/etc/systemd/coredump.conf\r\n/etc/systemd/coredump.conf.d/*.conf\r\n/run/systemd/coredump.conf.d/*.conf\r\n/usr/lib/systemd/coredump.conf.d/*.conf\r\n/etc/systemd/systemd.conf\r\nWe can also look for the use of memory dumping commands, like:\r\ngcore\r\ncat /proc/\u003cpid\u003e/maps\r\ngdb -pid \u003cpid\u003e Then in the GDB shell: (gdb) dump memory /root/output offset\r\nAnd common credential dumping tools, like:\r\nmimipenguin\r\n3snake\r\nLet’s get a more in-depth look at the these tools, starting with mimipenguin.\r\nmimipenguin — A tool to dump login passwords of Linux users\r\nTo understand mimipenguin process activity we’ll take a look at the process tree. More specifically, we’ll search\r\nfor the gcore utility in the process tree (See below - any gcore image running under bash → sudo) because we\r\nknow mimipinguin uses it to dump memory.\r\nBy analyzing the process tree we can find suspicious parent-child relationships and understand the process\r\nactivity:\r\nImage 5–6: Verint TPS builds a tree for each executed process. We can see in this image that\r\n“mimipenguin” runs under the “sudo” process who is a child of “bash” and creates a “dash” child\r\nhttps://medium.com/verint-cyber-engineering/linux-threat-hunting-primer-part-ii-69484f58ac92\r\nPage 4 of 10\n\nprocess that is the parent of “gnome-keyring-daemon” (service that stores passwords) process. The\r\nprocess tree helps us understand the full activity in the process parent — child dimension.\r\nAfter we have an initial lead from the process tree, we can further investigate the raw data and analyse parameters\r\nsuch as the command line, path and users.\r\nPress enter or click to view image in full size\r\nPress enter or click to view image in full size\r\nImage 7–8: Verint’s TPS presents full command line, image path, user and MD5 for each executed\r\nprocess. We can see in this image the full commands and that the process runs with “sudo”\r\nprivileges under the user “root”. This is important for behavioral analysis and threat hunting.\r\n(Another way to get this full information will be — combining the output from ps, who, uname, ptrace, strace and\r\nfile commands)\r\nGet VerintCyberSec’s stories in your inbox\r\nJoin Medium for free to get updates from this writer.\r\nRemember me for faster sign in\r\nWe will look for gnome-keyring-daemon process that does not run under its normal parent process. To understand\r\nwhat is “normal” we will look at a benign “gnome-keyring” process tree.\r\nImage 9: we will look at the child and parent process of the benign “gnome-keyring-daemon”\r\nprocess from the process tree.\r\nhttps://medium.com/verint-cyber-engineering/linux-threat-hunting-primer-part-ii-69484f58ac92\r\nPage 5 of 10\n\nImage 10: looking for “Xsession” parent process to find “gnome-keyring-daemon” full process tree.\r\nIn conclusion, we will look for gnome-keyring-daemon that does not run under the following tree:\r\ngdm-session-worker → Xsession → gnome-keyring-daemon → gnome-keyring-daemon\r\nNow we’ll turn to look at the other tool mentioned above called “3snake”.\r\n3snake — Dump sshd and sudo credential related strings\r\nBy understanding how “3snake” works, you can learn how to search for it in the data:\r\n3snake reads memory from sshd and sudo system calls that handle password-based authentication\r\nIt doesn’t write to the memory of the traced process\r\n3snake spawns a new process for every sshd and sudo command that it runs\r\nListens for the proc event using netlink sockets to get candidate processes to trace\r\nWhen it detects a running process that uses sshd or sudo, ptrace is attached and traces read and write\r\nsystem calls, extracting strings related to password based authentication\r\nFrom the above analysis, we conclude that 3snake creates multiple threads and processes. We will look for an\r\nexcessive process activity as in the process tree below:\r\nYou should look for this anomaly — multiple processes in 3 generations of the process “3snake” executing on a\r\nsingle endpoint.\r\nhttps://medium.com/verint-cyber-engineering/linux-threat-hunting-primer-part-ii-69484f58ac92\r\nPage 6 of 10\n\nImage 11: Verint’s TPS builds a tree for each executed process. In the above image, we notice that\r\n“3snake” spawns itself multiple times.\r\nStage 3: Filter out “normal” activities\r\nIn this stage, we’ll learn the normal activity of the network. This will help you to reduce the number of results. Be\r\ncareful with your assumptions, so that you don’t filter out too much data or overfit to a specific system. Do not\r\nforget to refactor the query. When you finish, check if there is a technique missing or more “known good” data to\r\nfilter out.\r\nIn our credential-dumping hunt, we can do the following:\r\n1. Identify normal passwd activity, for example, look for all passwd references as a process or as a process\r\ncommand line:\r\nPress enter or click to view image in full size\r\nImage 12: TPS endpoint forensics agent collects process execution information from the Kernel. We\r\ncan search for “passwd” file/system command reference on that data. Another way to get this full\r\ninformation will be — combining the output from “ps”, “who”, “uname”,”ptrace” , “strace” and\r\n“file” commands.\r\nPress enter or click to view image in full size\r\nhttps://medium.com/verint-cyber-engineering/linux-threat-hunting-primer-part-ii-69484f58ac92\r\nPage 7 of 10\n\nImage 13: We can search for “shadow” file\\ system command reference on that data in order to\r\nbaseline benign activity.\r\n3. Check System processes normal activity for relevant commands (like compgen, getent, passwd, useradd,\r\ngroupadd, usermod, chsh, chfn, users, id, groups, last, logname, w, who, whoami, members, groupmod, finger, su,\r\ngpasswd, chgrp)\r\n4. Search whether /etc/shadow and /etc/passwd were copied (which are used to unshadow with ‘John the Ripper’,\r\nan open source tool used for password cracking) by the same user or at the same time.\r\n5. Check system binary activity; Try to specify the suspicious activity by multiple parameters (command line,\r\nprivileges, memory etc.) to avoid whitelisting a full binary that can be poisoned (i.e. replacing system binary with\r\na malicious one).\r\nFor example: If you whitelist “ls” binary activity (by name and process tree only) to avoid result overload, you\r\ncan miss malicious activity in case “ls” was replaced with a malicious file.\r\nStage 4: query, refine and repeat\r\nNow that we understand how an attack looks like and how normal behavior looks like, we will combine both and\r\nuse it for threat hunting.\r\nHunting for memory dumps files\r\nWe’ll look for all dump files that were not created by the abrt process (abrt, atomic bug report tool, is Linux\r\nsystem daemon that reads process memory and may seem suspicious).\r\nQuery example:\r\n(Relying on the dump file name to contain the string “dump” in it is bad, but bear with me for the\r\nsake of this explanation).\r\nTo hunt for processes that creates activity using user credentials, like:\r\nProcesses that perform activities on files that contain passwords and don’t run unser “run-parts” , “getnet”\r\nor “abrt-watch-log” system commands , which are part of the system normal activity.\r\nProcesses with commandline that includes important user control files like shadow, passwd, login.\r\nDont forget to filter the results of this subqueries from normal os behavior (like the “600 shadow.bak” —\r\nspecific command)\r\nLook for dump file creation in the process commandline\r\nhttps://medium.com/verint-cyber-engineering/linux-threat-hunting-primer-part-ii-69484f58ac92\r\nPage 8 of 10\n\nWe’ll look for dumping tools activity, for example:\r\nTo hunt for user credential related file, we’ll search all /etc/passwd, /etc/shadow files activity that is not created by\r\nsystem user control command, for example:\r\nNow, for each query output , analyze the data and try to better understand the system behavior. Refactor the query\r\nusing your conclusions about the behavior. In addition, filter out the “known good” activity that is specific to your\r\nnetwork (e.g. Remove SAP utilities activity)\r\nBeware — filtering out full commands will lower your ability to detect injections or binary poisoning.\r\nSummary \u0026 Conclusions\r\nWe described a threat hunting process which includes four stages:\r\nUnderstanding the attack techniques you’d like to find\r\nConducting research on how attackers implement these technique\r\nSearching the suspicious data in the organization to find anomalies which require further analysis\r\nFiltering out normal activities which look anomalous\r\nRepeating the above process while refining the queries until no anomalies are left or an attack was\r\nidentified\r\nTo exemplify the implementation of this process, we used information from MITRE ATT\u0026CK Matrix as well as\r\nacademic papers which surveyed past attacks against Linux based systems to prioritize a hunting hypothesis. I\r\nfocused on the “credential dumping” technique since it is common, easy to understand and does not require a lot\r\nof research to threat hunt most of its implementations.\r\nThe information provided throughout this blog includes queries against malicious data and examples of known\r\ngood behavior which we can carefully whitelist. Each environment has its own unique anomalies. You need to\r\ncarefully analyze all the anomalies you find and remove those which do not describe real threats to your network.\r\nIt is a tedious and iterative process, but at the end you’ll be able to come to a conclusion about your hunt\r\nhypothesis.\r\nI hope that this post helped you become even more excited about dealing with Linux threat hunting. I think that if\r\nyou take each tactic systematically, you will find it interesting. Look at this experience as a new opportunity to see\r\nthe beauty of Linux Internals.\r\nhttps://medium.com/verint-cyber-engineering/linux-threat-hunting-primer-part-ii-69484f58ac92\r\nPage 9 of 10\n\n—\r\nThanks to Oren Biderman and Michael Gendelman for reviewing this post and providing useful suggestions.\r\nSource: https://medium.com/verint-cyber-engineering/linux-threat-hunting-primer-part-ii-69484f58ac92\r\nhttps://medium.com/verint-cyber-engineering/linux-threat-hunting-primer-part-ii-69484f58ac92\r\nPage 10 of 10",
	"extraction_quality": 1,
	"language": "EN",
	"sources": [
		"Malpedia"
	],
	"origins": [
		"web"
	],
	"references": [
		"https://medium.com/verint-cyber-engineering/linux-threat-hunting-primer-part-ii-69484f58ac92"
	],
	"report_names": [
		"linux-threat-hunting-primer-part-ii-69484f58ac92"
	],
	"threat_actors": [],
	"ts_created_at": 1778121760,
	"ts_updated_at": 1778121850,
	"ts_creation_date": 0,
	"ts_modification_date": 0,
	"files": {
		"pdf": "https://archive.orkl.eu/145d5ce46a09352ecfbb51d9baee2945f88d5b2a.pdf",
		"text": "https://archive.orkl.eu/145d5ce46a09352ecfbb51d9baee2945f88d5b2a.txt",
		"img": "https://archive.orkl.eu/145d5ce46a09352ecfbb51d9baee2945f88d5b2a.jpg"
	}
}