# Linux Threat Hunting: ‘Syslogk’ a kernel rootkit found under development in the wild **[decoded.avast.io/davidalvarez/linux-threat-hunting-syslogk-a-kernel-rootkit-found-under-development-in-the-wild/](https://decoded.avast.io/davidalvarez/linux-threat-hunting-syslogk-a-kernel-rootkit-found-under-development-in-the-wild/)** June 13, 2022 by [David Álvarez and Jan NeduchalJune 13, 202213 min read](https://decoded.avast.io/davidalvarez/linux-threat-hunting-syslogk-a-kernel-rootkit-found-under-development-in-the-wild/) ## Introduction Rootkits are dangerous pieces of malware. Once in place, they are usually really hard to detect. Their code is typically more challenging to write than other malware, so developers resort to code reuse from open source projects. As rootkits are very interesting to analyze, we are always looking out for these kinds of samples in the wild. [Adore-Ng is a relatively old, open-source, well-known kernel rootkit for Linux, which initially](https://github.com/yaoyumeng/adore-ng) targeted kernel 2.x but is currently updated to target kernel 3.x. It enables hiding processes, files, and even the kernel module, making it harder to detect. It also allows authenticated user-mode processes to interact with the rootkit to control it, allowing the attacker to hide many custom malicious artifacts by using a single rootkit. In early 2022, we were analyzing a rootkit mostly based on `Adore-Ng that we found in` the wild, apparently under development. After obtaining the sample, we examined the ``` .modinfo section and noticed it is compiled for a specific kernel version. ``` As you may know, even if it is possible to ‘force load’ the module into the kernel by using the `--force flag of the` [insmod Linux command, this operation can fail if the required](https://man7.org/linux/man-pages/man8/insmod.8.html) symbols are not found in the kernel; this can often lead to a system crash. ``` insmod -f {module} ``` ----- We discovered that the kernel module could be successfully loaded without forcing into a [default Centos 6.10 distribution, as the rootkit we found is compiled for a similar kernel](https://www.linuxvmimages.com/images/centos-6/) version. While looking at the file’s strings, we quickly identified the `PgSD93ql hardcoded file name` in the kernel rootkit to reference the payload. This payload file name is likely used to make it [less obvious for the sysadmin, for instance, it can look like a legitimate PostgreSQL file.](https://www.postgresql.org/) Using this hardcoded file name, we extracted the file hidden by the rootkit. It is a compiled backdoor trojan written in C programming language; Avast’s antivirus engine detects and classifies this file as `ELF:Rekoob – which is widely known as the Rekoobe malware family.` ``` Rekoobe is a piece of code implanted in legitimate servers. In this case it is embedded in ``` a fake SMTP server, which spawns a shell when it receives a specially crafted command. In this post, we refer to this rootkit as `Syslogk rootkit, due to how it ‘reveals’ itself when` specially crafted data is written to the file `/proc/syslogk .` ## Analyzing the Syslogk rootkit The `Syslogk rootkit is heavily based on` `Adore-Ng but incorporates new functionalities` making the user-mode application and the kernel rootkit hard to detect. ## Loading the kernel module To load the rootkit into kernel space, it is necessary to approximately match the kernel version used for compiling; it does not have to be strictly the same. ``` vermagic=2.6.32-696.23.1.el6.x86_64 SMP mod_unload modversions ``` For example, we were able to load the rootkit without any effort in a [Centos 6.10 virtual](https://www.linuxvmimages.com/images/centos-6/) [machine by using the insmod Linux command.](https://linux.die.net/man/8/insmod) After loading it, you will notice that the malicious driver does not appear in the list of loaded [kernel modules when using the lsmod command.](https://linux.die.net/man/8/lsmod) ## Revealing the rootkit The rootkit has a `hide_module function which uses the list_del function of the` [kernel API](https://www.kernel.org/doc/html/v5.8/core-api/kernel-api.html) to remove the module from the linked list of kernel modules. Next, it also accordingly updates its internal `module_hidden flag.` ----- Fortunately, the rootkit has a functionality implemented in the `proc_write function that` exposes an interface in the /proc file system which reveals the rootkit when the value `1 is` written into the file `/proc/syslogk .` [Once the rootkit is revealed, it is possible to remove it from memory using the rmmod Linux](https://linux.die.net/man/8/rmmod) command. The Files section of this post has additional details that will be useful for programmatically uncloaking the rootkit. ## Overview of the Syslogk rootkit features Apart from hiding itself, making itself harder to detect when implanted, `Syslogk can` completely hide the malicious payload by taking the following actions: The `hk_proc_readdir function of the rootkit hides directories containing malicious` files, effectively hiding them from the operating system. The malicious processes are hidden via `hk_getpr – a mix of Adore-Ng functions for` hiding processes. The malicious payload is hidden from tools like `Netstat ; when running, it will not` appear in the list of services. For this purpose, the rootkit uses the function ``` hk_t4_seq_show . ``` The malicious payload is not continuously running. The attacker remotely executes it on demand when a specially crafted TCP packet (details below) is sent to the infected machine, which inspects the traffic by installing a `netfilter hook .` It is also possible for the attacker to remotely stop the payload. This requires using a ``` hardcoded key in the rootkit and knowledge of some fields of the magic packet ``` used for remotely starting the payload. We observed that the `Syslogk rootkit (and Rekoobe payload) perfectly align when used` covertly in conjunction with a fake SMTP server. Consider how stealthy this could be; a [backdoor that does not load until some magic packets are sent to the machine. When](https://www.drkns.net/kernel-who-does-magic/) queried, it appears to be a legitimate service hidden in memory, hidden on disk, remotely ‘magically’ executed, hidden on the network. Even if it is found during a network port scan, it still seems to be a legitimate SMTP server. ----- For compromising the operating system and placing the mentioned hiding functions, `Syslogk uses the already known set_addr_rw and` [set_addr_ro rootkit functions, which](https://github.com/ksaravan910/FileCloakingRootkit/blob/master/rootkit.c#L81) adds or removes writing permissions to the `Page Table Entry` [(PTE) structure.](https://www.kernel.org/doc/gorman/html/understand/understand006.html) After adding writing permissions to the PTE, the rootkit can hook the functions declared in the `hks internal rootkit structure.` PTE Hooks Type of the function Offset Name of the function Original hks+(0x38) * 0 proc_root_readdir Hook hks+(0x38) * 0 + 0x10 hk_proc_readdir Original hks+(0x38) * 1 tcp4_seq_show Hook hks+(0x38) * 1 + 0x10 hk_t4_seq_show Original hks+(0x38) * 2 sys_getpriority Hook hks+(0x38) * 2 + 0x10 hk_getpr The mechanism for placing the hooks consists of identifying the hookable kernel symbols via `/proc/kallsyms as implemented in the` `get_symbol_address function of the rootkit` [(code reused from this repository). After getting the address of the symbol, the](https://github.com/milabs/kmod_hooking/blob/master/module-init.c#L237) `Syslogk` rootkit uses the [udis86 project for hooking the function.](https://github.com/vmt/udis86) ## Understanding the directory hiding mechanism The Virtual File System (VFS) is an abstraction layer that allows for FS-like operation over something that is typically not a traditional FS. As it is the entry point for all the File System queries, it is a good candidate for the rootkits to hook. It is not surprising that the Syslogk rootkit hooks the VFS functions for hiding the Rekoobe payload stored in the file /etc/rc-Zobk0jpi/PgSD93ql . The hook is done by hk_root_readdir which calls to `nw_root_filldir where the` directory filtering takes place. ----- As you can see, any directory containing the substring `-Zobk0jpi will be hidden.` The function `hk_get_vfs opens the root of the file system by using filp_open. This kernel` [function returns a pointer to the structure file, which contains a](https://github.com/torvalds/linux/blob/master/include/linux/fs.h#L956) `file_operations` [structure called f_op that finally stores the](https://github.com/torvalds/linux/blob/master/include/linux/fs.h#L939) [readdir function hooked via](https://man7.org/linux/man-pages/man3/readdir.3.html) `hk_root_readdir .` Of course, this feature is not new at all. You can check the source code of `Adore-Ng and` see [how it is implemented on your own.](https://github.com/yaoyumeng/adore-ng/blob/master/adore-ng.c#L300) ## Understanding the process hiding mechanism In the following screenshot, you can see that the `Syslogk rootkit (code at the right margin` of the screenshot) is prepared for hiding a process called `PgSD93ql . Therefore, the rootkit` seems more straightforward than the original version (see Adore-Ng at the left margin of the screenshot). Furthermore, the process to hide can be selected after authenticating with the rootkit. The `Syslogk rootkit function` `hk_getpr explained above, is a mix of adore_find_task` [and should_be_hidden functions but it uses the same mechanism for hiding processes.](https://github.com/yaoyumeng/adore-ng/blob/master/adore-ng.c#L193) ## Understanding the network traffic hiding mechanism The `Adore-Ng rootkit allows hiding a given set of listening services from Linux programs` like `Netstat . It uses the exported` [proc_net structure to](https://github.com/yaoyumeng/adore-ng/blob/522c80a2dc043c2d523256472becc88c90d66337/adore-ng.c#L662) [change the](https://github.com/yaoyumeng/adore-ng/blob/522c80a2dc043c2d523256472becc88c90d66337/adore-ng.c#L835) [tcp4_seq_show( )](https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_ipv4.c#L2695) handler, which is invoked by the kernel when `Netstat queries for listening connections.` Within the [adore_tcp4_seq_show() function,](https://github.com/yaoyumeng/adore-ng/blob/master/adore-ng.c#L688) [strnstr( ) is used to look in](https://github.com/yaoyumeng/adore-ng/blob/master/adore-ng.c#L697) `seq->buf for a` substring that contains the hexadecimal representation of the port it is trying to hide. If this is found, the string is deleted. ----- In this way, the backdoor will not appear when listing the connections in an infected machine. The following section describes other interesting capabilities of this rootkit. ## Understanding the magic packets Instead of continuously running the payload, it is remotely started or stopped on demand by sending specially crafted network traffic packets. These are known as `magic packets because they have a special format and special` powers. In this implementation, an attacker can trigger actions without having a listening port in the infected machine such that the commands are, in some way, ‘magically’ executed in the system. ### Starting the Rekoobe payload The `magic packet inspected by the` `Syslogk rootkit for starting the` `Rekoobe fake` SMTP server is straightforward. First, it checks whether the packet is a TCP packet and, in that case, it also checks the `source port, which is expected to be` `59318 .` ``` Rekobee will be executed by the rootkit if the magic packet fits the mentioned criteria. ``` ----- Of course, before executing the fake service, the rootkit terminates all existing instances of the program by calling the rootkit function `pkill_clone_0 . This function contains the` hardcoded process name `PgSD93ql ; it only kills the` `Rekoobe process by sending the` ``` KILL signal via send_sig. ``` To execute the command that starts the `Rekoobe fake service in user mode, the rootkit` executes the following command by combining the kernel APIs: [call_usermodehelper_setup,](https://www.kernel.org/doc/htmldocs/kernel-api/API-call-usermodehelper-setup.html) [call_usermodehelper_setfns, and](http://www.hep.by/gnu/kernel/kernel-api/API-call-usermodehelper-setfns.html) [call_usermodehelper_exec.](https://www.kernel.org/doc/htmldocs/kernel-api/API-call-usermodehelper-exec.html) ``` /bin/sh -c /etc/rc-Zobk0jpi/PgSD93ql ``` The Files section of this post demonstrates how to manually craft (using Python) the TCP ``` magic packet for starting the Rekoobe payload. ``` In the next section we describe a more complex form of the `magic packet .` ### Stopping the Rekoobe payload Since the attacker doesn’t want any other person in the network to be able to kill `Rekoobe,` the `magic packet for killing` `Rekoobe must match some fields in the previous` `magic` ``` packet used for starting Rekoobe . Additionally, the packet must satisfy additional ``` requirements – it must contain a key that is hardcoded in the rootkit and located in a variable offset of the `magic packet . The conditions that are checked:` ----- 1. It checks a flag enabled when the rootkit executes `Rekoobe via` `magic packets . It` will only continue if the flag is enabled. 2. It checks the `Reserved field of the TCP header to see that it is` `0x08 .` 3. The `Source Port must be between` `63400 and` `63411 inclusive.` 4. Both the `Destination Port and the` `Source Address, must to be the same that` were used when sending the `magic packet for starting` `Rekoobe .` 5. Finally, it looks for the `hardcoded key . In this case, it is:` `D9sd87JMaij` The offset of the hardcoded key is also set in the packet and not in a hardcoded offset; it is calculated instead. To be more precise, it is set in the `data offset byte (TCP header)` such that after shifting the byte `4 bits to the right and multiplying it by` `4, it points to the` offset of where the `Key is expected to be (as shown in the following screenshot, notice` that the rootkit compares the `Key in reverse order).` In our experiments, we used the value `0x50 for the` `data offset (TCP header) because` after shifting it 4 bits, you get 5 which multiplied by 4 is equal to `20 . Since 20 is precisely` the size of the TCP Header, by using this value, we were able to put the key at the start of the data section of the packet. If you are curious about how we implemented this `magic packet from scratch, then` please see the Files section of this blog post. ## Analyzing Rekoobe When the infected machine receives the appropriate `magic packet, the rootkit starts the` hidden `Rekoobe malware in user mode space.` It looks like an innocent SMTP server, but there is a backdoor command on it that can be executed when handling the `starttls command. In a legitimate service, this command is` sent by the client to the server to advise that it wants to start TLS negotiation. ----- For triggering the `Rekoobe backdoor command (spawning a shell), the attacker must send` the byte `0x03 via TLS, followed by a` `Tag Length Value (TLV) encoded data. Here, the` tag is the symbol `%, the length is specified in four numeric characters, and the value` (notice that the length and value are arbitrary but can not be zero). Additionally, to establish the TLS connection, you will need the certificate embedded in ``` Rekoobe . ``` See the Files section below for the certificate and a Python script we developed to connect with `Rekoobe .` ## The origin of Rekoobe payload and Syslogk rootkit ``` Rekoobe is clearly based on the TinySHell open source project; this is based on ordering ``` observed in character and variables assignment taking place in the same order multiple times. ----- On the other hand, if you take a look at the `Syslogk rootkit, even if it is new, you will` notice that there are also references to `TinySHell dating back to December 13, 2018.` The evidence suggests that the threat actor developed `Rekoobe and` `Syslogk to run` them together. We are pleased to say that our users are protected and hope that this research assists others. ## Conclusions One of the architectural advantages of security software is that it usually has components running in different privilege levels; malware running on less-privileged levels cannot easily interfere with processes running on higher privilege levels, thus allowing more straightforward dealing with malware. On the other hand, kernel rootkits can be hard to detect and remove because these pieces of malware run in a privileged layer. This is why it is essential for system administrators and security companies to be aware of this kind of malware and write protections for their users as soon as possible. ## IoCs Syslogk sample ``` 68facac60ee0ade1aa8f8f2024787244c2584a1a03d10cda83eeaf1258b371f2 ``` ## Rekoobe sample ----- ``` 11edf80f2918da818f3862246206b569d5dcebdc2a7ed791663ca3254ede772d ``` ## Other Rekoobe samples ``` fca2ea3e471a0d612ce50abc8738085f076ad022f70f78c3f8c83d1b2ff7896b ``` ----- ``` 2fea3bc88c8142fa299a4ad9169f8879fc76726c71e4b3e06a04d568086d3470 ``` ## Files Syslogk research tools Rekoobe research tool [rekoobe_backdoor_client.py](https://github.com/avast/ioc/blob/master/SyslogkRootkit/Research%20Tools/rekoobe_backdoor_client.py) [cert.pem](https://github.com/avast/ioc/blob/master/SyslogkRootkit/Research%20Tools/cert.pem) ## IoC repository ----- [The Syslogk and Rekoobe rootkit research tools and IoCs are in our IoC repository.](https://github.com/avast/ioc/tree/master/SyslogkRootkit) [Tagged asanalysis,](https://decoded.avast.io/tag/analysis/) [linux,](https://decoded.avast.io/tag/linux/) [malware,](https://decoded.avast.io/tag/malware/) [rootkit](https://decoded.avast.io/tag/rootkit/) -----