Feuerfest

I needed a quick throwaway Ubuntu VM for some testing, so I reached for the classic combo: an Ubuntu cloud image, a local cloud-init NoCloud datasource served over HTTP, and a bare qemu-system-x86_64 invocation. Nothing fancy, I've done this a dozen times before - just that the last time was ages ago..

My first attempt looked like this:

root@vm:~# qemu-system-x86_64 -net nic \
    -net user -machine accel=kvm:tcg -m 512 \
    -nographic -hda noble-server-cloudimg-amd64.img \
    -smbios type=1,serial=ds='nocloud;s=http://127.0.0.1:8000/'

And of course, it didn't just work. Last time I did was ages ago and directly on the hardware of my rootserver, not my local Proxmox server. As it turned out, I had two separate problems in this command, and to make things even more interesting: Everything is executed inside a Debian 13 VM, which itself lives on top of Proxmox. A VM inside a VM. So every weirdness I hit had to be checked twice. Once for "Is this a QEMU problem?" and once for "Is this a nested-virtualization problem?".

Preface

I created the directory /root/temp and but the files there according to the cloud-init QEMU tutorial. Then I started a webserver via the http.server Python module.

Have a look at https://docs.cloud-init.io/en/latest/tutorial/qemu.html#define-the-configuration-data-files regarding the files and the content.

Problem 1: Stuck at "Booting from Hard Disk..."

The VM wouldn't get past the SeaBIOS boot message "Booting from Hard Disk...". It became stuck there, forever. Additionally I saw no request to the local webserver on port 8000/tcp.

Turns out this is entirely expected, once you know why. Ubuntu's released Noble Numbat (LTS) cloud image is built for UEFI boot. SeaBIOS, the legacy BIOS QEMU uses by default, has no idea what to do with the GPT-partitioned disk and just... stops. No error, no hint, just sitting at "Booting from Hard Disk...".

The fix is to hand the VM the proper UEFI firmware via the Open Virtual Machine Firmware (OVMF) package instead of relying on SeaBIOS:

apt-get install ovmf

If you find older posts, they mention the /usr/share/OVMF/OVMF_VARS.id file, but on Debian 13 there's a small surprise waiting here too, the package no longer ships the plain OVMF_CODE.fd or OVMF_VARS.fd files I was used to. Instead you get the 4M variants:

root@vm:~# ls /usr/share/OVMF/OVMF_VARS_4M.*
/usr/share/OVMF/OVMF_VARS_4M.fd  /usr/share/OVMF/OVMF_VARS_4M.ms.fd  /usr/share/OVMF/OVMF_VARS_4M.snakeoil.fd

The .ms.fd variant ships with Microsoft's Secure Boot certificates pre-enrolled, and .snakeoil.fd is pre-signed with a test certificate for Secure Boot development. Neither is what I wanted, so plain OVMF_VARS_4M.fd it is, paired with the matching OVMF_CODE_4M.fd. CODE and VARS need to come from the same "generation" (2M vs. 4M), mixing them doesn't end well.

You also don't want to point QEMU directly at the VARS file in /usr/share/OVMF/, copy it locally first, since QEMU will write to it (that's where your EFI variables actually live):

root@vm:~# cp /usr/share/OVMF/OVMF_VARS_4M.fd .

Problem 2: Wrong IP for the webserver

While I was at it, I remembered that the datasource URL is used inside the guest and therefore refers to the guest itself, not my Proxmox VM serving the cloud-init files. With QEMU's user-mode networking -net user, the host is reachable from inside the guest at the fixed address 10.0.2.2. And since I'm serving the metadata over HTTP rather than a local path, nocloud-net is the correct datasource identifier, not nocloud.

Putting it together:

root@vm:~# qemu-system-x86_64 \
  -machine q35,accel=kvm:tcg \
  -m 512 \
  -nographic \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.fd \
  -drive if=pflash,format=raw,file=./OVMF_VARS_4M.fd \
  -drive file=noble-server-cloudimg-amd64.img,if=virtio,format=qcow2 \
  -net nic -net user \
  -smbios type=1,serial=ds='nocloud-net;s=http://10.0.2.2:8000/'

That got me past the boot prompt. Progress.

Problem 3: It boots, but agonizingly slowly

The VM eventually made it into the kernel, but everything crawled. The console was full of entries like this:

[  116.771971] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 32 times, consider switching to WQ_UNBOUND
Starting systemd-udevd version 255.4-1ubuntu8.16
[  148.409180] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 64 times, consider switching to WQ_UNBOUND
[  226.621365] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 128 times, consider switching to WQ_UNBOUND

These particular messages are a symptom, not the disease. What's actually going on: accel=kvm:tcg tells QEMU "Use KVM if you can, otherwise silently fall back to TCG (software emulation)". If, for whatever reason, KVM isn't available inside the nested VM, QEMU just quietly emulates the entire CPU in software which is orders of magnitude slower and explains why a workqueue would get starved for 10ms stretches and longer.

Since this whole setup is a VM inside a VM, the first suspect is always nested virtualization. So, checks on the Proxmox-hosted Debian VM:

root@vm:~# ls -la /dev/kvm
ls: cannot access '/dev/kvm': No such file or directory
root@vm:~# lscpu | grep -i virtualization
Virtualization type:                     full

As the /dev/kvm device isn't present, we now have confirmed that QEMU falls back to software emulation for the CPU.

The solution? Changing the CPU for the VM in Proxmox from x86-64-v2-AES (or whatever is configured) to host. This enables the VM to directly use the CPU of my Proxmox host.

This can be done comfortably in the WebUI of Proxmox. Just power down the VM.

Before:

And after:

After a reboot we have a /dev/kvm device and full support for Intel VT-x virtualisation. This confirms Proxmox is passing VT-x through cleanly to the nested VM. So KVM should be usable in principle, meaning if the inner QEMU VM is still falling back to TCG, it's not because nested virt is broken at the Proxmox level.

root@vm:~# ls -la /dev/kvm
crw-rw---- 1 root kvm 10, 232 Jul 27 00:48 /dev/kvm
root@vm:~# lscpu | grep -i virtualization
Virtualization:                          VT-x
Virtualization type:                     full

To actually find out, instead of trusting the silent kvm:tcg fallback, it's better to force KVM explicitly and let it fail loudly if something's wrong:

root@vm:~# qemu-system-x86_64 \
  -machine q35,accel=kvm \
  -cpu host \
  -m 512 \
  -nographic \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.fd \
  -drive if=pflash,format=raw,file=./OVMF_VARS_4M.fd \
  -drive file=noble-server-cloudimg-amd64.img,if=virtio,format=qcow2 \
  -net nic -net user \
  -smbios type=1,serial=ds='nocloud-net;s=http://10.0.2.2:8000/'

Removing the :tcg fallback means any KVM problem (permissions, missing nested virt on the Proxmox host CPU type, whatever) surfaces immediately as a hard error instead of a mysteriously slow VM. The parameter -cpu host on top makes sure the guest actually sees the hosts CPU features rather than a generic, lowest-common-denominator CPU model.

And the command worked, however, as we are still booting a VM inside a VM expect it to take some time. 😅

My learnings

For anyone hitting the same wall on their own Proxmox nested-VM setup, the checklist is:

1. Is /dev/kvm present on the outer (Proxmox) VM at all?
2. Is nested virtualization actually enabled on the Proxmox host itself?

   • # For Intel-CPUs - prints Y if enabled/available
     root@proxmox:~# cat /sys/module/kvm_intel/parameters/nested
     Y
     # For AMD-CPUs - prints Y if enabled/available
     root@proxmox:~# cat /sys/module/kvm_amd/parameters/nested
     Y

3. Is the outer VM's CPU type set to host in the Proxmox config? So VMX/SVM flags get passed through
   • Check in the Hardware tab of the VM inside Proxmox's WebUI, or: 
   • Check with qm config VM-ID und verify the cpu line says host

     root@proxmox:~# qm config 100
     [...]
     cores: 1
     cpu: host
     [...]

4. Does the inner QEMU call use accel=kvm (not kvm:tcg) plus -cpu host, so a broken KVM path fails loudly instead of silently degrading to software emulation?
5. Is the /dev/kvm device present on the VM where you want to start the QEMU VM?
6. Is CPU virtualization supported?

Nothing here is exotic once you know to look for it, but "silent TCG fallback" combined with "one VM inside another" is a pretty effective way to make a simple performance problem look mysterious for a while.

A small pre-preface for users who followed the "Ultimate Pi-hole Setup" tutorial from the YouTuber WunderTech

If you used the configuration files he provided on his homepage: https://www.wundertech.net/ultimate-pi-hole-setup/ you will experience the exact same problems sooner or later.

The reason is that the VIPs from keepalived are not bound on a separate dummy interface and hence the pihole-FTL process will take them into account when dynamically building the hostname and choosing the "correct IPs".

You HAVE to at least enable dns.reply.host in the /etc/pihole/pihole.toml to mark the static IP used for the server Pi-hole is running on.

Jump to The solution if you are not interested in the details.

Preface

One of the main reasons why I have my homelab is to hone my skills. And today was a day this happened.

From my one of my LAN hosts I wanted to connect to my Raspberry4 (raspi4.lan, IP: 192.168.178.8) via SSH. This host is configured as the secondary/backup instance in keepalived for the DNS VIP (192.168.178.100).

The reality however was different:

user@lanadmin:~$ ssh raspi4.lan
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Please contact your system administrator.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/user/.ssh/known_hosts:10
  remove with:
  ssh-keygen -f '/home/user/.ssh/known_hosts' -R 'raspi4.lan'
Host key for raspi4.lan has changed and you have requested strict checking.
Host key verification failed.
user@lanadmin:~$  ssh-keygen -f '/home/user/.ssh/known_hosts' -R 'raspi4.lan'
# Host raspi4.lan found: line 8
# Host raspi4.lan found: line 9
# Host raspi4.lan found: line 10
/home/user/.ssh/known_hosts updated.
Original contents retained as /home/user/.ssh/known_hosts.old

Granted I don't log on often onto raspi4.lan as everything is automated and monitored and the Pi-hole config is synced via Nebula-Sync from raspi3.lan. So I suspected I didn't purge the entries related to raspi4.lan from my ~/.ssh/known_hosts file after I re-installed that system a while ago.

user@lanadmin:~$ ssh raspi4.lan
The authenticity of host 'raspi4.lan (ULA:ffff)' can't be established.
ED25519 key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:5: raspi3.lan
    ~/.ssh/known_hosts:12: 192.168.178.9
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'raspi4.lan' (ED25519) to the list of known hosts.
Linux raspi3 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64
###############################
## Primary Pi-hole instance! ##
###############################
Last login: Fri Jul 24 21:46:36 2026 from w.x.y.z

user@raspi3:~$

Huh? How did I end up on raspi3.lan, when I clearly entered raspi4.lan as the host to connect to? Something is very wrong.

Sadly I overlooked that the IP SSH provided did list the IPv6 VIP (ending in :ffff) which is plain wrong, but it shouldn't take long for me to discover that..

Overview over the current setup

We have two Raspberry Pi's in this setup. 

Host A: Raspberry 3
IPv4: 192.168.178.9/24
IPv6: ULA:9/64
Hostname: raspi3.lan
Keepalived: Primary

Host B: Raspberry 4
IPv4: 192.168.178.8/24
IPv6: ULA:8/64
Hostname: raspi4.lan
Keepalived: Secondary

The three used VIPs are:
VIPv4: 192.168.178.100/32 (DNS: pihole.lan) and 192.168.178.101/32 unused, just for testing
VIPv6: ULA:ffff/128 (DNS: pihole.lan)

Both Raspberries run a Pi-hole instance with unbound and keepalived. The VIPs are automatically configured on eth0 on the primary instance. raspi3.lan has the primary keepalived role.

It's always DNS™

At first I suspected that I screwed up when I created my local DNS entries, but no the Local DNS records in my Pi-Hole setup are fine. The IPs for the hosts and vip match. The config is synced with nebula-sync from raspi3.lan to raspi4.lan, so no room for typos there. Nevertheless I accessed both WebUIs and checked independently, wouldn't be the first time a sync-mechanism failed. However, everything was in order.

A dig however showed the problem clearly:

root@lanadmin:~# dig -t a raspi3.lan +noall +answer
raspi3.lan.             0       IN      A       192.168.178.9
root@lanadmin:~# dig -t a raspi4.lan +noall +answer
raspi4.lan.             0       IN      A       192.168.178.100

The /etc/resolv.conf of that system is:

root@lanadmin:~# cat /etc/resolv.conf
domain lan
search lan
nameserver 192.168.178.8
nameserver 192.168.178.9

So raspi4.lan is queried first, than raspi3.lan. And I forgot to change the nameserver to the VIP...

/etc/hosts was also fine. Only standard entries for the .8 on raspi4.lan and .9 on raspi3.lan. In short: the local eth0 system IPs with the corresponding FQDN and hostname.

getent hosts however was a bit strange too. It returned the loopback address when each host asked for it's own name:

root@raspi3:~# getent hosts raspi3.lan
::1             raspi3.lan
root@raspi3:~# getent hosts raspi4.lan
192.168.178.8   raspi4.lan

root@raspi4:~# getent hosts raspi4.lan
::1             raspi4.lan
root@raspi4:~# getent hosts raspi3.lan
192.168.178.9   raspi3.lan

Hence I suspected Avahi (mDNS) as it's installed and listed before the dns resolution in /etc/nsswitch.conf:

root@raspi4:~# grep hosts /etc/nsswitch.conf
hosts:          files mdns4_minimal [NOTFOUND=return] dns

But several hard facts speak against this.

1. Avahi only works for entries ending in .local
2. dig doesn't use libnss and therefore doesn't honor mDNS/Avahi at all, but it still showed the wrong IPs

These two facts effectively eliminated Avahi/mDNS as the source of the problem. Avahi was however responsible for returning the loopback address when the host queried for it's own IP. After all mdns4_minimal was listed before dns in /etc/nsswitch.conf. So that works as designed, but doesn't help at all during troubleshooting as it just adds to the confusion..

The hosts don't know themselves...

At my wits end I took a step back and decided to check DNS from a third host against both raspi3.lan (192.168.178.9) and raspi4.lan (192.168.178.8) how both resolve the DNS A-Records of each other.

# Querying for the IPv4 of raspi3.lan
# Against raspi3.lan
root@lanadmin:~# dig @192.168.178.9 raspi3.lan +noall +answer
raspi3.lan.             0       IN      A       192.168.178.101
# Against raspi4.lan
root@lanadmin:~# dig @192.168.178.8 raspi3.lan +noall +answer
raspi3.lan.             0       IN      A       192.168.178.9

# Querying for the IPv4 of raspi4.lan
# Against raspi4.lan
root@lanadmin:~# dig @192.168.178.8 raspi4.lan +noall +answer
raspi4.lan.             0       IN      A       192.168.178.100
# Against raspi3.lan
root@lanadmin:~# dig @192.168.178.9 raspi4.lan +noall +answer
raspi4.lan.             0       IN      A       192.168.178.8

This is looks strange.

Whenever we ask a Raspberry itself for it's own IP we get a wrong result.
Querying 192.168.178.9 (raspi3.lan) to resolve raspi3.lan returns 192.168.178.101.
Querying 192.168.178.8 (raspi4.lan) to resolve raspi4.lan returns 192.168.178.100.

How? Avahi was ruled out. There was no DHCP at play and the static DNS entries are correct.

Something was messing with my setup.

Is it Pi-hole?

I diff'd the /etc/pihole/pihole.toml suspecting I missed something in that, as I knew that the pihole.toml isn't sync by nebula-sync, but there was nothing.

user@lanadmin:~$ diff -u <(ssh 192.168.178.8 sudo cat /etc/pihole/pihole.toml) <(ssh 192.168.178.9 sudo cat /etc/pihole/pihole.toml)
--- /dev/fd/63  2026-07-25 03:44:15.628030101 +0200
+++ /dev/fd/62  2026-07-25 03:44:15.628030101 +0200
@@ -1,7 +1,7 @@
 # Pi-hole configuration file (v6.7)
 # Encoding: UTF-8
 # This file is managed by pihole-FTL
-# Last updated on 2026-07-14 10:31:50 CEST
+# Last updated on 2026-07-24 23:26:54 CEST

 [dns]
   # Upstream DNS Servers to be used by Pi-hole. If this is not set, Pi-hole will not

I searched a bit and stumbled upon a setting regarding FTL: dns.domain and dns.expandHosts.

  [dns.domain]
    # The DNS domain used by your Pi-hole.
    #
    # This DNS domain is purely local. FTL may answer queries from its local cache and
    # configuration but *never* forwards any requests upstream *unless* you have
    # configured a dns.revServer exactly for this domain. In the latter case, all queries
    # for this domain are sent exclusively to this server (including reverse lookups).
    #
    # For DHCP, this has two effects; firstly it causes the DHCP server to return the
[... removed as DHCP is not relevant in this case ...]
    #
    # You can disable setting a domain by setting this option to an empty string.
    #
    # Allowed values are:
    #     Any valid domain
    name = "lan"

So the FTL-Cache will be queried for records ending in .lan, which is fine as it's my local domain and requests for .lan shouldn't leave my home network. Additionally I understood the sentence "but never forwards any requests upstream" as: These requests don't even hit Unbound or dnsmasq.

expandHosts makes sure the FQDN is added to /etc/hosts. Something I already did manually (or the Debian installer).

  # If set, the domain is added to simple names (without a period) in /etc/hosts in the
  # same way as for DHCP-derived names
  #
  # Allowed values are:
  #     true or false
  expandHosts = true ### CHANGED, default = false

From what I read online pihole-FTL builds the FQDN of the local system itself completely independent from settings in /etc/hosts + dns.domain. Could this be a lead?

How do we verify the entry is actually in the cache and contains the wrong IP? Glad I asked myself! The command killall -USR1 pihole-FTL
dumps the cache entries from dnsmasq into /var/log/pihole/pihole.log.

root@raspi4:~# killall -USR1 pihole-FTL
root@raspi4:~# vi /var/log/pihole/pihole.log
Jul 25 02:49:12 dnsmasq[1147]: time 1784940552
Jul 25 02:49:12 dnsmasq[1147]: cache size 10000, 0/141 cache insertions re-used unexpired cache entries.
[...]
Jul 25 02:49:12 dnsmasq[1147]: Host        Address            Flags      Expires      Source
Jul 25 02:49:12 dnsmasq[1147]: ----------- ------------------ ---------- ------------ ------------
Jul 25 02:49:12 dnsmasq[1147]: pihole.lan  192.168.178.100    4FRI   H                /etc/pihole/hosts/custom.list
Jul 25 02:49:12 dnsmasq[1125]: raspi4.lan  192.168.178.8      4FRI   H                /etc/hosts
Jul 25 02:49:12 dnsmasq[1125]: raspi3.lan  192.168.178.9      4FRI   H                /etc/pihole/hosts/custom.list

Well, that only proves my assumption that local static DNS records are NOT honored, if the hostname matches the host on which Pi-hole is running. We can clearly see that the entry for raspi4.lan has a source of /etc/hosts and not /etc/pihole/hosts/custom.list, while the record for raspi3.lan is taken from /etc/pihole/hosts/custom.list.

This proves that some, currently unknown, automatism is at work and goes horribly wrong.

If nothing helps, try rebooting

As I had no real trace of where to look next, I now focused on trying to re-produce the issue. After all, if it was just some quirk of a non-restarted service utilizing some old file - only present in the cache of it's processes file handles.. As raspi4.lan currently was the secondary node for keepalived, it didn't own the VIPs. Hence I stopped the keepalived process on raspi3.lan, forcing a failover to raspi4.lan. After making sure raspi4.lan had the VIPs I rebooted the system.

It gets stranger...

After the reboot, in order to get a bit more insight, I executed several dig queries and watched the log simultaneously. All @ip's are IPs which are currently present on the eth0 interface of the raspi4.lan host.

Those were the commands:

root@lanadmin:~# dig @192.168.178.8 raspi4.lan +noall +answer
raspi4.lan.             0       IN      A       192.168.178.100
root@lanadmin:~# dig @192.168.178.100 raspi4.lan +noall +answer
raspi4.lan.             0       IN      A       192.168.178.100
root@lanadmin:~# dig @192.168.178.101 raspi4.lan +noall +answer
raspi4.lan.             0       IN      A       192.168.178.100

# On raspi4.lan, just to have everything neatly together:
root@raspi4:~# killall -USR1 pihole-FTL

And this showed up in the logfile:

root@raspi4:~# tail -f /var/log/pihole/pihole.log |grep "raspi4.lan"
Jul 25 02:58:17 dnsmasq[1125]: query[A] raspi4.lan from 192.168.178.7
Jul 25 02:58:17 dnsmasq[1125]: Pi-hole hostname raspi4.lan is 192.168.178.100
Jul 25 02:58:17 dnsmasq[1125]: query[A] raspi4.lan from 192.168.178.7
Jul 25 02:58:17 dnsmasq[1125]: Pi-hole hostname raspi4.lan is 192.168.178.100
Jul 25 02:58:17 dnsmasq[1125]: query[A] raspi4.lan from 192.168.178.7
Jul 25 02:58:17 dnsmasq[1125]: Pi-hole hostname raspi4.lan is 192.168.178.100
Jul 25 02:58:36 dnsmasq[1125]: raspi4.lan   192.168.178.8     4FRI   H     /etc/hosts

And here I tilted a bit. Why does the log state "Pi-hole hostname raspi4.lan is 192.168.178.100" but then, just seconds later state that the IP retrieved from /etc/hosts is 192.168.178.8? At least the IP 192.168.178.100 was reliably returned for all queries towards IPs on the eth0 interface of raspi4.lan with the goal to resolve the name raspi4.lan. It wasn't some kind of race-condition nor did it feel like a bug.

And just to be sure, I tried logging into raspi4.lan from lanadmin.lan:

user@lanadmin:~$ ssh raspi4.lan
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
Please contact your system administrator.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/user/.ssh/known_hosts:16
  remove with:
  ssh-keygen -f '/home/user/.ssh/known_hosts' -R 'raspi4.lan'
Host key for raspi4.lan has changed and you have requested strict checking.
Host key verification failed.
user@lanadmin:~$ host raspi4.lan
raspi4.lan has address 192.168.178.100
raspi4.lan has IPv6 address fd6d:f92:17e4:0:464e:6dff:feb5:ffff

Nope, still the same problem. Only that this time I would have actually logged on to the right host, as the IP was currently owned by raspi4.lan. Nonetheless did the hostname still resolve to the wrong IP.

I had no idea why the IP kept changing. Yes, I had provided no specific IP for pihole-FTL to listen on, but this should never lead to such an behaviour. I decided to read through /etc/pihole/pihole.toml, if only to set a specific listener address and do a bit more troubleshooting.

Finally: Enlightenment

Revelation hit me, when I read the [dns.reply.host] block. The describe behaviour matched the observed one perfectly!

    [dns.reply.host]
      # Use a specific IPv4 address for the Pi-hole host? By default, FTL determines the
      # address of the interface a query arrived on and uses this address for replying to A
      # queries with the most suitable address for the requesting client.
      #
      # This setting can be used to use a fixed, rather than the dynamically obtained,
      # address when Pi-hole responds to the following names:
      # - "pi.hole"
      # - "<the device's hostname>"
      # - "pi.hole.<local domain>"
      # - "<the device's hostname>.<local domain>"
      #
      # Allowed values are:
      #     true or false
      force4 = false

      # Custom IPv4 address for the Pi-hole host
      #
      # Allowed values are:
      #     A valid IPv4 address or empty string ("")
      IPv4 = ""

      # Use a specific IPv6 address for the Pi-hole host? See description for the IPv4
      # variant above for further details.
      #
      # Allowed values are:
      #     true or false
      force6 = false

      # Custom IPv6 address for the Pi-hole host
      #
      # Allowed values are:
      #     A valid IPv6 address or empty string ("")
      IPv6 = ""

As it can be clearly seen, static IPs for the local hostname were disabled. This made FTL choose a new "best matching" IP for each received query. Which is such a strange mechanism to implement! Why obscure such things!?

Why design such a ... mechanism?

And then it hit me.. Novice and inexperienced users. Pi-hole is a DNS and Ad-Blocker. Primarily aimed at home users. And those lack knowledge and experience. I'm a frequent reader of subreddits like r/selfhosted or r/HomeServer so I know full well how many users struggle with IPs, interface bindings, file rights, etc. All the basic stuff one learns over time but can be pretty hard for people new to Linux.

I suspect this mechanism was developed to ease the usage of Pi-hole, to just "make it work" no matter what. Alas.. This caused way more trouble for an experienced user this way. And this is why I don't really like that they implemented this mechanism. It's just one of these automatisms which ignore standards and work without following an established process. Effectively hindering novice users to learn "How it is normally done"?

The solution

The fix was rather easy. Just enable dns.reply.host for IPv4 and IPv6 and set the corresponding IPs. Then restart the service. Done. Below is the config for raspi4.lan.

    [dns.reply.host]
      # Use a specific IPv4 address for the Pi-hole host? By default, FTL determines the
      # address of the interface a query arrived on and uses this address for replying to A
      # queries with the most suitable address for the requesting client.
      #
      # This setting can be used to use a fixed, rather than the dynamically obtained,
      # address when Pi-hole responds to the following names:
      # - "pi.hole"
      # - "<the device's hostname>"
      # - "pi.hole.<local domain>"
      # - "<the device's hostname>.<local domain>"
      #
      # Allowed values are:
      #     true or false
      force4 = true ### CHANGED, default = false

      # Custom IPv4 address for the Pi-hole host
      #
      # Allowed values are:
      #     A valid IPv4 address or empty string ("")
      IPv4 = "192.168.178.8" ### CHANGED, default = ""

      # Use a specific IPv6 address for the Pi-hole host? See description for the IPv4
      # variant above for further details.
      #
      # Allowed values are:
      #     true or false
      force6 = true ### CHANGED, default = false

      # Custom IPv6 address for the Pi-hole host
      #
      # Allowed values are:
      #     A valid IPv6 address or empty string ("")
      IPv6 = "fd6d:ULA:8" ### CHANGED, default = ""

Please note that I obscured parts of my ULA IPv6 address.

Why didn't I notice sooner?

Then there is always this question which creeps into ones mind: Why didn't I notice it sooner? Why did it work for so long?

At least in this case the answer is simple: I rarely need to login into these systems.

Other improvements

Dummy network interface for VIPs

Apparently it is also better to create a dummy network device and let keepalived bind the VIPs to that interface. One big advantage is that dummy interfaces don't reply to ARP-Requests at all. Which is a crucial problem in HA setup. And also one point WunderTech didn't mention with one word in his tutorial..

VRRP-Scripts to check DNS service availbility/healthiness

In the current setup keepalived will only switch to another machine if it stops sending out VRRP-Announcements, which usually only happens when a machine fails completely (power cut or really catastrophic failures). If just the pihole-FTL service, Unbound or any other piece of software - apart from keepalived - fails nothing will happen.

For this, keepalived supports the execution of VRRP-scripts. These will be executed every few seconds and are there to check service availbility/healthiness and trigger a failover if the script execution fails or doesn't exit successfully.

I plan to write a blog post about that too. When it is ready, I will link it here.

Another small important detail I learned today about epoch numbers in package names. Or rather: I was forced to understand what I already saw for years but never thought about.

RedHat's package manager yum (and by that extent also dnf and rpm) treat a non-specified epoch number as 0. This confused me today as a specified package couldn't be found. And an dnf info packagename didn't show the Epoch: field. Despite the package name clearly including an epoch version of 0.

Turns out this is document in RedHat Enterprise Linux 9: Advanced Topics - 6.3.1. The Epoch directive and Debian treats and Epoch number of 0 in the exact same way.

RedHat says:

The Epoch directive enables to define weighted dependencies based on version numbers.

If this directive is not listed in the RPM spec file, the Epoch directive is not set at all. This is contrary to common belief that not setting Epoch results in an Epoch of 0. However, the dnf utility treats an unset Epoch as the same as an Epoch of 0 for the purposes of depsolving.

However, listing Epoch in a spec file is usually omitted because in majority of cases introducing an Epoch value skews the expected RPM behavior when comparing versions of packages.

This effectivly means: If the epoch number is listed as None, it is 0. Just like in this example:

user@host:~$ rpm -q --qf "%{EPOCH}:%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n" bind-utils
(none):bind-utils-9.20.21-150700.3.18.1.x86_64

And Debian says pretty much the same in their manual regarding Control files and their fields: 5.6.12 version

epoch
This is a single (generally small) unsigned integer. It may be omitted, in which case zero is assumed.

Epochs can help when the upstream version numbering scheme changes, but they must be used with care. You should not change the epoch, even in experimental, without getting consensus on debian-devel first.

Interesting. I only had to use an Epoch number of 1 for a company internal Debian package once. As someone build a different package with the exact same name (despite the name having nothing to with what his because did) as this person just copy&pasted some build-scripts but apparently didn't change all important values. That was 10+ years ago.

Never stumbled about this particular case with how the 0 is treated.

You never stop learning.

I finally got around setting myself up with a Termix instance (their GitHub). Its a connection manager for various protocols (SSH, RDP, Telnet, etc.) accessible via a web-frontend. Termix it self runs inside a Docker container.

Here is a view of the web-frontend (I resized the window to make it smaller). I generated a new SSH-Key solely for the use connecting from Termix to the configured hosts. Then added the public key to 2 hosts, put them inside a folder for better overview, hit connect and it works.

(Click to enlarge)

It supports the creation of tunnels too and various other options. So far I have only used it with SSH so I can't say much regarding RDP (or Telnet 😂). Having this reachable via HTTPS could be a nice solution in environments where direct SSH (and VPN) is blocked.

The docker compose file

I configured SSL with certificates from my own CA. These are mounted read-only into the container under /certs. This all works without Traefik, Caddy or Nginx for SSL.

services:
  termix:
    image: ghcr.io/lukegus/termix:latest
    container_name: termix
    restart: unless-stopped
    environment:
      - ENABLE_SSL=true
      - SSL_PORT=8443
      - SSL_DOMAIN=host.tld
      - PORT=8080
      - SSL_CERT_PATH=/certs/host.tld.crt
      - SSL_KEY_PATH=/certs/host.tld.key
    ports:
      - "6666:8443"
    volumes:
      - /opt/docker/termix/data:/app/data
      # Mount cert-dir for certificates read-only
      - /opt/docker/certs/:/certs:ro

A welcomed surprise

I was pleasantly surprised to notice that the Termix docker container automatically reported "Healthy" inside my dashboard. Without me ever having defined a proper healthcheck.

Turns out Termix is one of these rare projects who define a healthcheck in the container image itself:

root@host:~# docker inspect termix | grep -A 20 Healthcheck
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "wget -q -O /dev/null http://localhost:30001/health || exit 1"
                ],
                "Interval": 30000000000,
                "Timeout": 10000000000,
                "StartPeriod": 60000000000,
                "Retries": 3
            },

Nice!

I wanted to get rid of the HTTPS-Warning when opening the web-frontend of my DSL-Router. As I still use the vendor-supplied selfsigned certificate there. Hence I used my ca-scripts (GitHub) to generate a certificate for the IP and standard hostname (fritz.box).

Only to get the error:

error 47 at 0 depth lookup: permitted subtree violation
error 192.168.1.1.crt: verification failed

Huh? This is how I used my script. hostcert.sh calls sign.sh to sign the CSR and verifies the signed certificate against the CA-Root certificate.

root@host:~/ca# ./hostcert.sh 192.168.1.1 fritz.box
CN: 192.168.1.1
DNS ANs: fritz.box
IP ANs: 192.168.1.1
Enter to confirm.

writing RSA key
Reading pass from $CAPASS
CA signing: 192.168.1.1.csr -> 192.168.1.1.crt:
Using configuration from ca.config
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'DE'
localityName          :ASN.1 12:'Karlsruhe'
organizationName      :ASN.1 12:'LAN CA host cert'
commonName            :ASN.1 12:'192.168.1.1'
Certificate is to be certified until Mar 14 20:57:03 2027 GMT (365 days)

Write out database with 1 new entries
Database updated
CA verifying: 192.168.1.1.crt <-> CA cert
C=DE, L=Karlsruhe, O=LAN CA host cert, CN=192.168.1.1
error 47 at 0 depth lookup: permitted subtree violation
error 192.168.1.1.crt: verification failed

The offending command is:

root@host:~/ca# openssl verify -CAfile ca.crt fritz.box.crt 
C=DE, L=Karlsruhe, O=LAN CA host cert, CN=fritz.box
error 47 at 0 depth lookup: permitted subtree violation
error fritz.box.crt: verification failed

The root cause is that I forgot that I added an X509v3 Name Constraints. This dictates that all Common Name or SubjectAltNames, have to end in .lan and clearly fritz.box is in violation of that.

root@host:~/ca# openssl x509 -in ca.crt -text | grep "X509v3 Name" -A2
            X509v3 Name Constraints: 
                Permitted:
                  DNS:lan

The solution is to generate it solely for the IP, right?

root@host:~/ca# ./hostcert.sh 192.168.1.1
CA verifying: 192.168.1.1.crt <-> CA cert
C=DE, L=Karlsruhe, O=LAN CA host cert, CN=192.168.1.1
error 47 at 0 depth lookup: permitted subtree violation
error 192.168.1.1.crt: verification failed

Yeah no.. It's wrong too. In the first certificate the IP was also defined. I just thought fritz.box is the offending SAN as it is listed first (my script adds IP SANs after DNS SANs).

Through this I learned that as soon as one name constraint is specified, all SubjectAltNames have to follow the constraints. Constraints of type DNS and IPAddress are checked independently. And 192.168.1.1 doesn't match the Permitted DNS zone of .lan.

The corresponding RFC 5280 sections are:

• https://www.rfc-editor.org/rfc/rfc5280.html#section-4.2.1.10
• https://www.rfc-editor.org/rfc/rfc5280.html#section-6.1.4 scroll down to step (g).

Looks like I have to generate a new CA. Narf! This time however, I will make sure to extract all allowed and denied name constraints from the CA root certificate and check it against the supplied SubjectAltName BEFORE I create or sign the CSR.

TL;DR: The configured unicast_src IP isn't present on any network interface. In my case DHCPv6 was to blame.

I accidentally unplugged the power cable from my RaspberryPi 4 today. Due to this I learned a few things today.

1. First that my home DSL router (a FritzBox) doesn't always honor the preferred IPv4/v6 addresses send in DHCP-Requests
   • /etc/dhcpcd.conf did contain static ip_address=... and static ip6_address=...
2. The FritzBox can't set DHCP reservations for IPv6 addresses - only IPv4 - WHY!?
3. I have to read the keepalived error message while actually using my brain
   • I stumbled across the cannot assign requested address and thought of DHCP and was confused why the hell keepalived does DHCP things (the word requested mislead me)
   • In the following line the reason is written in plain text...  entering FAULT state (src address not configured)
4. Static IP-configuration for servers was, is and will always be the best
5. A mixed static & dynamic IPv6  configuration isn't hard at all once you read a bit about SLAAC
   • I went with DHCP for IPv4/IPv6 first as this is simply the Raspbian default and hey, it worked for some years..
   • I made that into a separate blogpost: Configuring an mixed IPv6 setup - static ULA, dynamic GLA

Long story short, this was the keepalived error I got. The VRRP-Instance immediately went into FAULT state and stayed there.

root@raspi:~# systemctl status keepalived.service
[...]
Feb 05 13:14:22 raspi Keepalived_vrrp[1279]: Delaying startup for 5 seconds
Feb 05 13:14:22 raspi Keepalived[1278]: Startup complete
Feb 05 13:14:22 raspi systemd[1]: Started keepalived.service - Keepalive Daemon (LVS and VRRP).
Feb 05 13:14:22 raspi Keepalived_vrrp[1279]: bind unicast_src fd87:f53:25b4:0:231d:4cbb:bca7:10 failed 99 - Cannot assign requested address
Feb 05 13:14:22 raspi Keepalived_vrrp[1279]: (VI_2): entering FAULT state (src address not configured)
Feb 05 13:14:22 raspi Keepalived_vrrp[1279]: (VI_2) Entering FAULT STATE
Feb 05 13:14:22 raspi Keepalived_vrrp[1279]: VRRP_Group(ALL) Syncing instances to FAULT state

At first I skipped the following line:

Feb 05 13:14:22 raspi Keepalived_vrrp[1279]: (VI_2): entering FAULT state (src address not configured)

Hence I searched a bit and found an older GitHub issue where this problem was explained with VRRP trying to do stuff to fast, while the interface wasn't ready. The solution mentioned in keepalived issue #2237: Keepalived entering fault state on reboot was to set vrrp_startup_delay inside the global_defs section of /etc/keepalived/keepalived.conf. However this was already the present in my case.

Yeah, turns out the configured unicast_src IP wasn't present on any interface. As the FritzBox deemed it fit to assign a random one from the configured DHCP-Range. We can verify this quickly by grep'ing for the IPv6 address.

root@raspi:~ # ip -6 a | grep fd87:f53:25b4:0:231d:4cbb:bca7:10
root@raspi:~ #

The solution

In my case I finally switched to a mixed static and dynamic IPv6 setup. Configuring the local ULA address as a static one, but still receive and apply the router advertisement (RA) to get a global IPv6 so my RaspberryPi can still connect to the Internet.

Then it showed up on the interface.

root@raspi:~ # ip -6 a | grep fd87:f53:25b4:0:231d:4cbb:bca7:10
    inet6 fd87:f53:25b4:0:231d:4cbb:bca7:10/64 scope global
root@raspi:~ #

Another viable solution would of course be to just reboot the RaspberryPi and hope your DHCP-Server now assigns the correct IP. However my FritzBox only allows to set an IPv4 reservation in the DHCP settings. IPv6 addresses can't be used for DHCP reservations at all. So this was no solution for me.

If you want to know how to configured a mixed static and dynamic IPv6 read here: Configuring an mixed IPv6 setup - static ULA, dynamic GLA