Feuerfest

Just the private blog of a Linux sysadmin

RedHat/Debian packages and Epoch numbers in packages

Another small important detail I learned today about epoch numbers in package names. Or rather: I was forced to understand what I already saw for years but never thought about.

RedHat's package manager yum (and by that extent also dnf and rpm) treat a non-specified epoch number as 0. This confused me today as a specified package couldn't be found. And an dnf info packagename didn't show the Epoch: field. Despite the package name clearly including an epoch version of 0.

Turns out this is document in RedHat Enterprise Linux 9: Advanced Topics - 6.3.1. The Epoch directive and Debian treats and Epoch number of 0 in the exact same way.

RedHat says:

The Epoch directive enables to define weighted dependencies based on version numbers.

If this directive is not listed in the RPM spec file, the Epoch directive is not set at all. This is contrary to common belief that not setting Epoch results in an Epoch of 0. However, the dnf utility treats an unset Epoch as the same as an Epoch of 0 for the purposes of depsolving.

However, listing Epoch in a spec file is usually omitted because in majority of cases introducing an Epoch value skews the expected RPM behavior when comparing versions of packages.

This effectivly means: If the epoch number is listed as None, it is 0. Just like in this example:

user@host:~$ rpm -q --qf "%{EPOCH}:%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n" bind-utils
(none):bind-utils-9.20.21-150700.3.18.1.x86_64

And Debian says pretty much the same in their manual regarding Control files and their fields: 5.6.12 version

epoch
This is a single (generally small) unsigned integer. It may be omitted, in which case zero is assumed.

Epochs can help when the upstream version numbering scheme changes, but they must be used with care. You should not change the epoch, even in experimental, without getting consensus on debian-devel first.

Interesting. I only had to use an Epoch number of 1 for a company internal Debian package once. As someone build a different package with the exact same name (despite the name having nothing to with what his because did) as this person just copy&pasted some build-scripts but apparently didn't change all important values. That was 10+ years ago.

Never stumbled about this particular case with how the 0 is treated.

You never stop learning.

Comments

Installing & configuring ISSO under Debian Trixie

For some reasons ISSO stopped working after I upgraded to Debian Trixie. Steps which weren't necessary when I first wrote my post Integrating Isso into Bludit via Apache2 and WSGI.

Take this post as an addition to the previous one.

Troubleshooting steps for Debian Trixie

ModuleNotFoundError: No module named 'pkg_resources'

This one was solved with an pip install setuptools. Despite being already installed and shown as installed when pip list is executed.

ModuleNotFoundError: No module named '_cffi_backend'

This one still baffles me. pip list shows the cffi module as being installed. However under Debian the package python3-cffi-backend must be installed for me. I think that I maybe hadn't all needed ffi packages installed, so the compiled /opt/your-venv-here/lib/python3.13/site-packages/_cffi_backend.cpython-313-x86_64-linux-gnu.so isn't fully working. Have to investigate.

If these two things are taken care of ISSO works fine under Debian Trixie.

Conclusion

All in all I suspect that I did get those errors as I did an in-place dist-upgrade from bullseye to bookworm to trixie on one afternoon. No fresh re-install as I currently lack the time for this.

Comments

Pi-hole, IPv6 and NTP - How to fix: "No valid NTP replies received, check server and network connectivity"

The following log message would only sporadically be logged on my Pi-hole. Not every hour, and not even every day. Just... sometimes. When the stars aligned... When, 52 years ago, Monday fell on a full moon and a 12th-generation carpenter was born... You get the idea.  😄

The error message was:

"No valid NTP replies received, check server and network connectivity"

Strange. NTP works. Despite Pi-hole sometimes fancy otherwise.

Inspecting the Pi-hole configuration

pihole-FTL returned the following NTP configuration:

user@host:~$ pihole-FTL --config ntp
ntp.ipv4.active = true
ntp.ipv4.address =
ntp.ipv6.active = true
ntp.ipv6.address =
ntp.sync.active = true
ntp.sync.server = 1.de.pool.ntp.org
ntp.sync.interval = 3600
ntp.sync.count = 8
ntp.sync.rtc.set = false
ntp.sync.rtc.device =
ntp.sync.rtc.utc = true

That looked good to me.

It was here that I had my suspicions: Wait, does the NTP Pool Project already offer IPv6? I have never knowingly used public NTP pools with IPv6. In customer networks, NTP servers are usually only reachable via IPv4. I don't have an NTP server in my home network. Sadly, many services are still not IPv6 ready.

Some companies even remove IPv6 support, like DigiCert (a commercial certificate authority!), who removed IPv6 support when they switched to a new CDN provider. This left me speechless. Read https://knowledge.digicert.com/alerts/digicert-certificate-status-ip-address if you want to know more.

NTP & IPv6? Only with pools that start with a 2

A short search for IPv6 support in NTP-Pools and https://www.ntppool.org/en/use.html provided the answer:

Please also note that the system currently only provides IPv6 addresses for a zone in addition to IPv4 addresses if the zone name is prefixed by the number 2, e.g. 2.pool.ntp.org (provided there are any IPv6 NTP servers in the respective zone). Zone names not prefixed by a number, or prefixed with any of 0, 1 or 3, currently provide IPv4 addresses only.

It turns out that the problem lies in my dual-stack setup, since I use IPv4 and IPv6 in parallel. Or rather... It's with the NTP pools. I checked with dig to see if any AAAA records were returned for 1.de.pool.ntp.org. The pool I was using.

dig aaaa 1.de.pool.ntp.org returns no AAAA-Records.

user@host:~$ dig aaaa 1.de.pool.ntp.org

; <<>> DiG 9.18.33-1~deb12u2-Debian <<>> aaaa 1.de.pool.ntp.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43230
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; EDE: 3 (Stale Answer)
;; QUESTION SECTION:
;1.de.pool.ntp.org.             IN      AAAA

;; AUTHORITY SECTION:
pool.ntp.org.           0       IN      SOA     d.ntpns.org. hostmaster.pool.ntp.org. 1749216969 5400 5400 1209600 3600

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Fri Jun 06 16:10:31 CEST 2025
;; MSG SIZE  rcvd: 134

And surely enough a dig aaaa 2.de.pool.ntp.org returns AAAA-Records.

user@host:~$ dig aaaa 2.de.pool.ntp.org

; <<>> DiG 9.18.33-1~deb12u2-Debian <<>> aaaa 2.de.pool.ntp.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47906
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;2.de.pool.ntp.org.             IN      AAAA

;; ANSWER SECTION:
2.de.pool.ntp.org.      130     IN      AAAA    2a0f:85c1:b73:62:123:123:123:123
2.de.pool.ntp.org.      130     IN      AAAA    2a01:239:2a6:d500::1
2.de.pool.ntp.org.      130     IN      AAAA    2606:4700:f1::1
2.de.pool.ntp.org.      130     IN      AAAA    2a01:4f8:141:282::5:1

;; Query time: 656 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Fri Jun 06 16:33:32 CEST 2025
;; MSG SIZE  rcvd: 158

My new Pi-hole configuration

The fix was easy, just configure 2.de.pool.ntp.org instead of 1.de.pool.ntp.org. Done.

user@host:~$ pihole-FTL --config ntp
ntp.ipv4.active = true
ntp.ipv4.address =
ntp.ipv6.active = true
ntp.ipv6.address =
ntp.sync.active = true
ntp.sync.server = 2.de.pool.ntp.org
ntp.sync.interval = 3600
ntp.sync.count = 8
ntp.sync.rtc.set = false
ntp.sync.rtc.device =
ntp.sync.rtc.utc = true

Now my Pi-hole instances aren't running long enough to really verify that the error is gone but I suspect so.

Some weeks later: The error is gone. It didn't re-appear.

Comments

How to fix Pi-hole FTL error: EDE: DNSSEC bogus

If you are instead searching for an explanation of the error code have a look at RFC 8914.

I noticed that the DNS resolution on my secondary Pi-hole instance wasn't working. host wouldn't resolve a single DNS name. As the /etc/resolv.conf included only the DNS servers running on localhost (127.0.0.1 and ::1) DNS resolution didn't work at all. Naturally I started looking at the Pi-hole logfiles.

/var/log/pihole/pihole.log would log this for all domains.

Jun  4 00:02:54 dnsmasq[4323]: query 1.de.pool.ntp.org from 127.0.0.1
Jun  4 00:02:54 dnsmasq[4323]: forwarded 1.de.pool.ntp.org to 127.0.0.1#5335
Jun  4 00:02:54 dnsmasq[4323]: forwarded 1.de.pool.ntp.org to ::1#5335
Jun  4 00:02:54 dnsmasq[4323]: validation 1.de.pool.ntp.org is BOGUS
Jun  4 00:02:54 dnsmasq[4323]: reply error is SERVFAIL (EDE: DNSSEC bogus)

Ok that was a first hint. I checked /var/log/pihole/FTL.log and there would be this message repeated all over again.

2025-06-03 00:02:52.505 CEST [841/T22762] ERROR: Error NTP client: Cannot resolve NTP server address: Try again
2025-06-03 00:02:52.509 CEST [841/T22762] INFO: Local time is too inaccurate, retrying in 600 seconds before launching NTP server

NTP is not the culprit

I checked the local time and it matched the time on the primary Pi-hole instance. Strange. I even opened https://uhr.ptb.de/ which is the official time clock for Germany (yes, per law). And it matched to the second. timedatectl would also print the correct time for both UTC and CEST and state that the system clock is synchronized.

root@host:~# timedatectl
               Local time: Wed 2025-06-04 00:51:07 CEST
           Universal time: Tue 2025-06-03 22:51:07 UTC
                 RTC time: n/a
                Time zone: Europe/Berlin (CEST, +0200)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

What the heck was going on?

Unbound leftovers

I googled "EDE: DNSSEC bogus" dnsmasq and found the solution in https://www.reddit.com/r/pihole/comments/zsrjzn/2_piholes_with_unbound_breaking_dns/.

Turns out I forgot to execute two critical steps.

  1. I didn't delete /etc/unbound/unbound.conf.d/resolvconf_resolvers.conf
  2. I didn't comment out the line starting with unbound_conf= in /etc/resolvconf.conf

Or they came back, when I updated that Raspberry from Debian Bullseye to Bookworm today. Anyway after doing these two steps and restarting Unbound it now works flawlessly.

And I learned which files are not kept in sync by nebula-sync. 😉

Comments

Installing Unbound as recursive DNS server on my PiHole

I run a Pi-hole installation on each of my Raspberry 3 & 4. As I do like to keep my DNS queries as much under my control as I can, I also installed Unbound to serve as recursive DNS server. This way all DNS queries will be handled by my Raspberry Pis.

Pi-hole is already installed using one of the following methods: https://github.com/pi-hole/pi-hole/#one-step-automated-install. If you don't have that done yet, do it first.

There is a good guide at the Pi-hole website which I will basically following.

https://docs.pi-hole.net/guides/dns/unbound/

root@host:~# apt install unbound

Regarding the configuration file I go with the one in the guide. However as I did have some problems in that past I needed to troubleshoot I include the following lines regarding loglevels and verbosity:

root@host:~# head /etc/unbound/unbound.conf.d/pihole.conf
server:
    # If no logfile is specified, syslog is used
    logfile: "/var/log/unbound/unbound.log"
    val-log-level: 2
    # Default is 1
    #verbosity: 4
    verbosity: 1

    interface: 127.0.0.1
    port: 5335
root@host:~# 

You can add that if you want but it's not needed to make Unbound work.

Next the guide tells us to download the root hints. A file maintained by Internic which contains information about the 13 DNS root name servers. Under Debian we don't need to download the named.root file from Internic as shown in the guide. Debian has its own package for that: dns-root-data.

It no only contains information about the 13 DNS root name servers but also the needed DNSSEC keys (also called root trust anchors). And together with unattended-upgrades we even automate updating that. Saving us the creation of a Cronjob or systemd timer.

root@host:~# apt install dns-root-data

In order for Unbound to have a directory and logfile to write into we need to create that:

root@host:~# mkdir -p /var/log/unbound
root@host:~# touch /var/log/unbound/unbound.log
root@host:~# chown unbound /var/log/unbound/unbound.log

As we are running under Debian we now need to tweak the Unbound config a little bit. Else we will get problems with DNSSEC. For this we are deleting a Debian generated file from Unbound and comment out the unbound_conf= line in /etc/resolvconf.conf so that it isn't included anymore.

root@host:~# sed -Ei 's/^unbound_conf=/#unbound_conf=/' /etc/resolvconf.conf
root@host:~# rm /etc/unbound/unbound.conf.d/resolvconf_resolvers.conf

Now all that is left is restarting Unbound.

root@host:~# systemctl restart unbound.service

Testing DNS resolution:

root@host:~# dig pi-hole.net @127.0.0.1 -p 5335

; <<>> DiG 9.18.33-1~deb12u2-Raspbian <<>> pi-hole.net @127.0.0.1 -p 5335
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46191
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;pi-hole.net.                   IN      A

;; ANSWER SECTION:
pi-hole.net.            300     IN      A       3.18.136.52

;; Query time: 169 msec
;; SERVER: 127.0.0.1#5335(127.0.0.1) (UDP)
;; WHEN: Sun May 25 18:21:25 CEST 2025
;; MSG SIZE  rcvd: 56

And to verify & falsify DNSSEC. This request must return an A-Record for dnssec.works.

root@host:~# dig dnssec.works @127.0.0.1 -p 5335

; <<>> DiG 9.18.33-1~deb12u2-Raspbian <<>> dnssec.works @127.0.0.1 -p 5335
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14076
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;dnssec.works.                  IN      A

;; ANSWER SECTION:
dnssec.works.           3600    IN      A       46.23.92.212

;; Query time: 49 msec
;; SERVER: 127.0.0.1#5335(127.0.0.1) (UDP)
;; WHEN: Sun May 25 18:22:52 CEST 2025
;; MSG SIZE  rcvd: 57

This request will not result in an A-Record.

root@host:~# dig fail01.dnssec.works @127.0.0.1 -p 5335

; <<>> DiG 9.18.33-1~deb12u2-Raspbian <<>> fail01.dnssec.works @127.0.0.1 -p 5335
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 1552
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;fail01.dnssec.works.           IN      A

;; Query time: 19 msec
;; SERVER: 127.0.0.1#5335(127.0.0.1) (UDP)
;; WHEN: Sun May 25 18:23:41 CEST 2025
;; MSG SIZE  rcvd: 48

Now all that is left to connect our Pi-hole with Unbound. Logon to your Pi-hole website and navigate to Settings -> DNS. Expand the line Custom DNS servers and enter to IP and Port to our Unbound server. 127.0.0.1#5335 for IPv4 and ::1#5335 for IPv6. If you don't use one of these two just don't add the line. After that hit "Save & Apply" and we are done.

Creating a logrotate config for Unbound

Sadly Unbound still doesn't deliver a logrotate config with its package. Therefore I just copy & paste from my previous article Howto properly split all logfile content based on timestamps - and realizing my own fallacy.

root@host:~# cat /etc/logrotate.d/unbound
/var/log/unbound/unbound.log {
        monthly
        missingok
        rotate 12
        compress
        delaycompress
        notifempty
        sharedscripts
        create 644
        postrotate
                /usr/sbin/unbound-control log_reopen
        endscript
}

Troubleshooting

fail01.dnssec.works timed out

The host fail01.dnssec.works tends to not answer requests sometimes. Others noticed this too. dig will only show the following message:

root@host:~# dig fail01.dnssec.works @127.0.0.1 -p 5335
;; communications error to 127.0.0.1#5335: timed out
;; communications error to 127.0.0.1#5335: timed out
;; communications error to 127.0.0.1#5335: timed out

; <<>> DiG 9.18.33-1~deb12u2-Raspbian <<>> fail01.dnssec.works @127.0.0.1 -p 5335
;; global options: +cmd
;; no servers could be reached

If that is the case, just execute the command again. Usually it will work the second time. Or just wait a few minutes. Sometimes the line ;; communications error to 127.0.0.1#5335: timed out will be printed, but the dig query will work after that nonetheless.

Comments

How to monitor your APT-repositories with Icinga

Photo by Pixabay: https://www.pexels.com/photo/software-engineers-working-on-computers-256219/

During my series about unattended-upgrades (Part 1, Part 2) I noticed that a Debian Mirror I use was unresponsive for 22 days but I had nothing to notify me of this. As this also meant that unattended-upgrades didn't apply any patches I wanted a check for this which in turn will trigger unattended-upgrades when there are outstanding updates.

The problem

The monitoring-plugins provide the check_apt plugin. This is normally used to check for available packages. However, in the default configuration it doesn't execute an apt-get update as this requires root privileges. Personally I think the risk is worth the gain. As adding the -u parameter will execute an apt-get update and therefore check_apt will notify you when apt-get update finishes with a non-zero exit-code.

Take the following problem:

root@host:~# apt-get update
Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
Hit:2 http://debian.tu-bs.de/debian bookworm InRelease
Get:3 http://debian.tu-bs.de/debian bookworm-updates InRelease [55.4 kB]
Reading package lists... Done
E: Release file for http://debian.tu-bs.de/debian/dists/bookworm-updates/InRelease is expired (invalid since 16d 0h 59min 33s). Updates for this repository will not be applied.

A mere check_apt wont notify you of any problems:

root@host:~# /usr/lib/nagios/plugins/check_apt
APT CRITICAL: 12 packages available for upgrade (12 critical updates). |available_upgrades=12;;;0 critical_updates=12;;;0

Making this go undetected.

Executed with the -u parameter however, we are notified of the problem:

root@host:~# /usr/lib/nagios/plugins/check_apt -u
'/usr/bin/apt-get -q update' exited with non-zero status.
APT CRITICAL: 12 packages available for upgrade (12 critical updates).  warnings detected, errors detected.|available_upgrades=12;;;0 critical_updates=12;;;0

The solution

Fixing this via Icinga is however a bit more complicated, as the standard apt CheckCommand from the Icinga Template Library (ITL) doesn't include the -u option and isn't prefixed to use sudo despite root privileges being needed. This can be checked here: https://github.com/Icinga/icinga2/blob/master/itl/command-plugins.conf#L2155 or in your local /usr/share/icinga2/include/command-icinga.conf if you happen to use Icinga.

The root cause is also the number one main problem I have with the check_apt CheckPlugin. check_apt is designed to actually install package updates when check_apt reports outstanding available updates. This however breaks the number one paradigm I have regarding monitoring systems: They should not modify the system on their own. And when they do, they should do it in the same way as it is normally done. check_apt breaks this.

Maybe that person should have read a blog article about unattended-upgrades prior to writting that plugin? 😜

Normally you utilize Event Commands for that type of scenario: "If service X is in state Y execute event command Z."

The CheckCommand check_apt_update

Therefore I recommend creating your own apt_update CheckCommand and using that.

object CheckCommand "check_apt_update" {
        command = [ "/usr/bin/sudo", + PluginDir + "/check_apt" ]

        arguments = {
                "-u" = {
                        description = "Perform an apt-get update"
                }
        }
}

Defining the service and configuring the EventCommand

Then in your service definition add a suitable event_command:

apply Service "apt repositories" to Host {
  import "hourly-service"

  check_command = "check_apt_update"

  enable_event_handler = true
  // Execute unattended-upgrades automatically if service goes critical
  event_command = "execute_unattended_upgrades"
  // For services which should be executed ON the host itself
  command_endpoint = host.vars.agent_endpoint

  assign where host.vars.distribution == "Debian"

}

Creating the EventCommand

And create the EventCommand like this:

object EventCommand "execute_unattended_upgrades" {
  command = "sudo /usr/bin/unattended-upgrades"
}

Necessary sudo rights

This requires a sudo config file for the icinga user executing that command. And the commands must be executable without the need for a TTY, hence we end up with the following:

root@host:~# cat /etc/sudoers.d/icinga2
# This line disables the need for a tty for sudo
#  else we will get all kind of "sudo: a password is required" errors
Defaults:icinga2 !requiretty

# sudo rights for Icinga2
icinga2  ALL=(ALL) NOPASSWD: /usr/bin/unattended-upgrades
icinga2  ALL=(ALL) NOPASSWD: /usr/bin/unattended-upgrade
icinga2  ALL=(ALL) NOPASSWD: /usr/bin/apt-get
icinga2  ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/check_apt

Conclusion

This is now sufficient as I'm notified when something prevents APT from properly updating the package lists. APT itself takes care to validate the various entries inside the Release file and exits with a non-zero exit-code, so there is no need to put that logic inside of check_apt.

Setting up similar checks for other monitoring systems is of course also possible. In general raising an alarm when apt-get update throws an non-zero exit-code is a somewhat foolproof method.

Comments