Feuerfest

Photo by Pixabay: https://www.pexels.com/photo/software-engineers-working-on-computers-256219/

During my series about unattended-upgrades (Part 1, Part 2) I noticed that a Debian Mirror I use was unresponsive for 22 days but I had nothing to notify me of this. As this also meant that unattended-upgrades didn't apply any patches I wanted a check for this which in turn will trigger unattended-upgrades when there are outstanding updates.

The problem

The monitoring-plugins provide the check_apt plugin. This is normally used to check for available packages. However, in the default configuration it doesn't execute an apt-get update as this requires root privileges. Personally I think the risk is worth the gain. As adding the -u parameter will execute an apt-get update and therefore check_apt will notify you when apt-get update finishes with a non-zero exit-code.

Take the following problem:

root@host:~# apt-get update
Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
Hit:2 http://debian.tu-bs.de/debian bookworm InRelease
Get:3 http://debian.tu-bs.de/debian bookworm-updates InRelease [55.4 kB]
Reading package lists... Done
E: Release file for http://debian.tu-bs.de/debian/dists/bookworm-updates/InRelease is expired (invalid since 16d 0h 59min 33s). Updates for this repository will not be applied.

A mere check_apt wont notify you of any problems:

root@host:~# /usr/lib/nagios/plugins/check_apt
APT CRITICAL: 12 packages available for upgrade (12 critical updates). |available_upgrades=12;;;0 critical_updates=12;;;0

Making this go undetected.

Executed with the -u parameter however, we are notified of the problem:

root@host:~# /usr/lib/nagios/plugins/check_apt -u
'/usr/bin/apt-get -q update' exited with non-zero status.
APT CRITICAL: 12 packages available for upgrade (12 critical updates).  warnings detected, errors detected.|available_upgrades=12;;;0 critical_updates=12;;;0

The solution

Fixing this via Icinga is however a bit more complicated, as the standard apt CheckCommand from the Icinga Template Library (ITL) doesn't include the -u option and isn't prefixed to use sudo despite root privileges being needed. This can be checked here: https://github.com/Icinga/icinga2/blob/master/itl/command-plugins.conf#L2155 or in your local /usr/share/icinga2/include/command-plugins.conf if you happen to use Icinga.

Normally you utilize Event Commands for that type of scenario: "If service X is in state Y execute event command Z."

The CheckCommand check_apt_update

Therefore I recommend creating your own apt_update CheckCommand and using that.

object CheckCommand "check_apt_update" {
        command = [ "/usr/bin/sudo", PluginDir + "/check_apt" ]
        timeout = 5m

        arguments = {
                "-u" = {
                        description = "Perform an apt-get update"
                }
        }
}

Defining the service and configuring the EventCommand

Then in your service definition add a suitable event_command:

apply Service "apt repositories" to Host {
  import "hourly-service"

  check_command = "check_apt_update"

  enable_event_handler = true
  // Execute unattended-upgrades automatically if service goes critical
  event_command = "execute_unattended_upgrades"
  // For services which should be executed ON the host itself
  command_endpoint = host.vars.agent_endpoint

  assign where host.vars.distribution == "Debian"

}

Creating the EventCommand

And create the EventCommand like this:

object EventCommand "execute_unattended_upgrades" {
  command = [
    "/bin/sh", "-c",
    "[ \"$$1\" = HARD ] || exit 0; case \"$$2\" in WARNING|CRITICAL) exec /usr/bin/sudo /usr/bin/systemd-run --unit=icinga-unattended-upgrade --collect --no-block /usr/bin/unattended-upgrade ;; esac",
    "sh",
    "$service.state_type$",
    "$service.state$"
  ]
}

We need sudo to execute unattended-upgrades as root. And we need to start unattended-upgrades as it's own unita via systemd-run. If the command would be sudo /usr/bin/unattended-upgrades the package upgrades would break every time when icinga related packages are updated.

As with just being started via sudo the unattended-upgrade process would be in the same cgroup as the Icinga process. And when Icinga gets restarted all processes in the same cgroup get terminated, hence also all dpkg processes started by unattended-upgrade.

The /bin/sh -c stuff checks for the state of the service. As Icinga executes event_commands on every state change. And we just want to execute unattended-upgrade when the state is HARD and either WARNING or CRITICAL. Not when it's in SOFT state or OK.

Yes, normally you would define a wrapper script for that, but that would be something I need to define, put into the zone, etc. pp. And why do this, if 2 lines of shellcode achieve the same?

Necessary sudo rights

This requires a sudo config file for the icinga user executing that command. And the commands must be executable without the need for a TTY, hence we end up with the following:

root@host:~# cat /etc/sudoers.d/icinga2
# This line disables the need for a tty for sudo
#  else we will get all kind of "sudo: sorry, you must have a tty to run sudo" errors
Defaults:icinga2 !requiretty

# sudo rights for Icinga2
icinga2  ALL=(root) NOPASSWD: /usr/bin/systemd-run --unit=icinga-unattended-upgrade --collect --no-block /usr/bin/unattended-upgrade
icinga2  ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/check_apt -u

Conclusion

This is now sufficient as I'm notified when something prevents APT from properly updating the package lists. APT itself takes care to validate the various entries inside the Release file and exits with a non-zero exit-code, so there is no need to put that logic inside of check_apt.

Setting up similar checks for other monitoring systems is of course also possible. In general raising an alarm when apt-get update throws an non-zero exit-code is a somewhat foolproof method.