The start: Knockoff browser extension

A few weeks ago I wrote a post about the Knockoff browser extension which helps consumers to filter out cheap and bogus brands from the Amazon search results. Now I found two websites which help me even more in a) filtering out/recognising bad brands and b) find good brands that still produce quality products which are worth their price.

Discovering: Worse on Purpose.com

It all started when I noticed that the WMF Clip&Close ice cube box is similar to the one produced by EMSA. And both had the same worse reviews as the plastic doesn't seem to be made for storing in freezes as it breaks easily and constantly. Naturally I was curious why WMF sells such crappy products.

And I found the solution on a webpage called Worse on Purpose. In the article Your Cookware Got Worse On Purpose author Keyana Sapp writes in detail about how family-owned companies are bought by international conglomerates who just seek to own the established brand name known for it's high quality products, to sell their cheap & crappy stuff.

The solution to this curiosity was rather boring. Under the headline Six companies, eighty-one brands we learn that both companies are owned by Groupe SEB (Wikipedia) who bought WMF & EMSA in 2016 and is, according to Wikipedia, the world's largest manufacturer of cookware.

Great, that will definitely not have any impace on the quality of WMF products rights? Oh wait.. There have been numerous factories closed since 2025...

The Brand Legder

Worse on Purpose operates The Brand Ledge where you can search for brands and companies names to get a small overview and can now decided for yourself if you still trust that brand. Takes this example for WMF: https://ledger.worseonpurpose.com/brands/wmf

Neat! And very handy!

BuyItForLife.com, the Chef'n Juicer & WMF

I don't really know what to make out of buyitforlife.com. It's a website seemingly vetting products the website owner has used/is using himself.

He states and the frontpage:

Browse 500+ Buy It For Life products I've personally vetted for quality and longevity. Compare where they're made, warranty, category, and price in seconds.
Source: https://buyitforlife.com/

And on the about page:

This list isn't sponsored or biased. It's just me, someone who likes well-built products, going down a rabbit hole to find things that are actually built to last.
Source: https://buyitforlife.com/about

Why I am experiencing such an uneasy feeling when reading that site?

Well, I found the site today. I was search for a manually operated lemon/citrus squeezer/press. The WMF one looked nice at first for 38,99€ but luckily the  Chef'n PalmZester (22,79€) was displayed right next to it.. And.. Well look for yourself:

First the Chef'n model in yellow, than the WMF one in black. They are exactly the same model, just different colours. And apparently black is 16€ more expensive.. Yeah, are you kidding me!?

     

Amazon reviews on both products state, that it breaks after just 1 year of regular usage. As apparently the plastic can't deal with the stress and breaks at some point. Wow.

What is even more interesting: Chef'n is owned by Lifetime Brands Inc.

And that is a totally different company which as nothing to do with Group SEB who owns WMF. This proves, again, that the brand WMF now stands for nothing. Just a shell used to sell cheap products for a higher price.

Good job capitalism!

BuyItForLife however has this product listed on it's site. Among many other products from companies which Worse on Purpose has on it's Watchlist or where people on Reddit say for years that the products became shittier over the time. I think this is my problem with that site. BuyItForLife is just a single person without any special knowledge. Just recommending products he finds useful. That's fair. Nothing against that, but it isn't as objective as it claims to be. It's just a single persons opinion - marketed as something more...

Nowhere on the site can I read a detailed review about a product. All links point directly to Amazon. No "This is why I recommend that item", nothing. For me, personally, this simply doesn't add up. So no, I'm not going to recommend that site.

In TinyMCE 6.8.1 (the WYSIWYG editor used by Bludit) iframe sandboxing was introduced. This automatically adds the sandbox="" parameter to all inserted <iframe>-tags. This blocks all embedded videos from playing, even when the CSP-Headers are correct.

As I just wanted to write a short blogpost about a YouTube video which discusses why we Germans are able to eat raw pork ("Mett") and suddenly the video wasn't displayed in the editor-view. Browser console showed no problems with CSP-Headers and so I was left to searching..

YouTube gave me the following <iframe> block for embedding the video:

<iframe width="560" height="315"
 src="https://www.youtube-nocookie.com/embed/azdV7EzP0v4?si=sKj7zfe0OkVNxjsi" title="YouTube video player"
 frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
 referrerpolicy="strict-origin-when-cross-origin"
 allowfullscreen>
</iframe>

However, after copy&pasting that into the TinyMCE in Bludit and saving the article, it changed to:

<iframe width="560" height="315"
 src="https://www.youtube-nocookie.com/embed/azdV7EzP0v4?si=sKj7zfe0OkVNxjsi" title="YouTube video player"
 frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
 referrerpolicy="strict-origin-when-cross-origin"
 allowfullscreen="allowfullscreen"
 sandbox="">
</iframe>

And the added sandbox-parameter triggers the following error in the browser console:

GET https://www.youtube-nocookie.com/img/meh7.png NS_BINDING_ABORTED
A resource is blocked by OpaqueResponseBlocking, please check browser console for details.

Of to the search engine I went and luckily the TineMCE folks wrote so in their release notes:

Note: sandbox_iframes: is set to false by default, which is the existing behavior. This is because enabling sandbox_iframes may break existing media embeds such as YouTube, Vimeo, and Codepen, as actions such as scripting and same-origin access are prevented.
Source: https://www.tiny.cloud/docs/tinymce/6/6.8.1-release-notes/#new-sandbox_iframes-option-that-controls-whether-iframe-elements-will-be-added-a-sandbox-attribute-to-mitigate-malicious-intent

Which is why I was able to find it so quickly.

The fix

I fixed it, by adding the YouTube domains to the iframe-exclusion tag inside the File bludit-folder/bl-plugins/tinymce/plugins.php.

1. Add the sandbox_iframes: true, line
2. The exclusions are defined for the three most used YouTube-domains: youtube.com, youtube-nocookie.com, youtu.be

        tinymce.init({
                selector: "#jseditor",
                auto_focus: "jseditor",
[...]
                link_default_target: '_blank',
                sandbox_iframes: true,
                sandbox_iframes_exclusions: [
                        'youtube.com',
                        'youtube-nocookie.com',
                        'youtu.be'
                ]
        });

Now your TinyMCE behaves like before and embedding videos works again.

Everything I ever wanted to know about Mett. Honestly I was just bored when I clicked on that video, thought of it to have a bit of over-promising/misleading title.

But that turned out to be wrong. Oliver Kim or Microbehunter explains in detail why we Germans are able to eat Mett. Which is somewhat of a oddity, given that raw pork can be dangerous and is frowned upon in many cultures and countries.

He explains the bacteria involved and also the system Germany has in place to ensure the meat is healthy. And this was really something new for me. I know that Germany is often praised for it's good infrastructure, however never had I considered uninterrupted cold-chains and strict meat controls to be a part of that. Turns out, Oliver is right, to offer raw pork at any butchery or even bread rolls with Mett as ready-to-eat items in bakeries is quite an achievement.

Source: https://www.youtube.com/watch?v=azdV7EzP0v4

I needed a quick throwaway Ubuntu VM for some testing, so I reached for the classic combo: an Ubuntu cloud image, a local cloud-init NoCloud datasource served over HTTP, and a bare qemu-system-x86_64 invocation. Nothing fancy, I've done this a dozen times before - just that the last time was ages ago..

My first attempt looked like this:

root@vm:~# qemu-system-x86_64 -net nic \
    -net user -machine accel=kvm:tcg -m 512 \
    -nographic -hda noble-server-cloudimg-amd64.img \
    -smbios type=1,serial=ds='nocloud;s=http://127.0.0.1:8000/'

And of course, it didn't just work. Last time I did was ages ago and directly on the hardware of my rootserver, not my local Proxmox server. As it turned out, I had two separate problems in this command, and to make things even more interesting: Everything is executed inside a Debian 13 VM, which itself lives on top of Proxmox. A VM inside a VM. So every weirdness I hit had to be checked twice. Once for "Is this a QEMU problem?" and once for "Is this a nested-virtualization problem?".

Preface

I created the directory /root/temp and but the files there according to the cloud-init QEMU tutorial. Then I started a webserver via the http.server Python module.

Have a look at https://docs.cloud-init.io/en/latest/tutorial/qemu.html#define-the-configuration-data-files regarding the files and the content.

Problem 1: Stuck at "Booting from Hard Disk..."

The VM wouldn't get past the SeaBIOS boot message "Booting from Hard Disk...". It became stuck there, forever. Additionally I saw no request to the local webserver on port 8000/tcp.

Turns out this is entirely expected, once you know why. Ubuntu's released Noble Numbat (LTS) cloud image is built for UEFI boot. SeaBIOS, the legacy BIOS QEMU uses by default, has no idea what to do with the GPT-partitioned disk and just... stops. No error, no hint, just sitting at "Booting from Hard Disk...".

The fix is to hand the VM the proper UEFI firmware via the Open Virtual Machine Firmware (OVMF) package instead of relying on SeaBIOS:

apt-get install ovmf

If you find older posts, they mention the /usr/share/OVMF/OVMF_VARS.id file, but on Debian 13 there's a small surprise waiting here too, the package no longer ships the plain OVMF_CODE.fd or OVMF_VARS.fd files I was used to. Instead you get the 4M variants:

root@vm:~# ls /usr/share/OVMF/OVMF_VARS_4M.*
/usr/share/OVMF/OVMF_VARS_4M.fd  /usr/share/OVMF/OVMF_VARS_4M.ms.fd  /usr/share/OVMF/OVMF_VARS_4M.snakeoil.fd

The .ms.fd variant ships with Microsoft's Secure Boot certificates pre-enrolled, and .snakeoil.fd is pre-signed with a test certificate for Secure Boot development. Neither is what I wanted, so plain OVMF_VARS_4M.fd it is, paired with the matching OVMF_CODE_4M.fd. CODE and VARS need to come from the same "generation" (2M vs. 4M), mixing them doesn't end well.

You also don't want to point QEMU directly at the VARS file in /usr/share/OVMF/, copy it locally first, since QEMU will write to it (that's where your EFI variables actually live):

root@vm:~# cp /usr/share/OVMF/OVMF_VARS_4M.fd .

Problem 2: Wrong IP for the webserver

While I was at it, I remembered that the datasource URL is used inside the guest and therefore refers to the guest itself, not my Proxmox VM serving the cloud-init files. With QEMU's user-mode networking -net user, the host is reachable from inside the guest at the fixed address 10.0.2.2. And since I'm serving the metadata over HTTP rather than a local path, nocloud-net is the correct datasource identifier, not nocloud.

Putting it together:

root@vm:~# qemu-system-x86_64 \
  -machine q35,accel=kvm:tcg \
  -m 512 \
  -nographic \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.fd \
  -drive if=pflash,format=raw,file=./OVMF_VARS_4M.fd \
  -drive file=noble-server-cloudimg-amd64.img,if=virtio,format=qcow2 \
  -net nic -net user \
  -smbios type=1,serial=ds='nocloud-net;s=http://10.0.2.2:8000/'

That got me past the boot prompt. Progress.

Problem 3: It boots, but agonizingly slowly

The VM eventually made it into the kernel, but everything crawled. The console was full of entries like this:

[  116.771971] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 32 times, consider switching to WQ_UNBOUND
Starting systemd-udevd version 255.4-1ubuntu8.16
[  148.409180] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 64 times, consider switching to WQ_UNBOUND
[  226.621365] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 128 times, consider switching to WQ_UNBOUND

These particular messages are a symptom, not the disease. What's actually going on: accel=kvm:tcg tells QEMU "Use KVM if you can, otherwise silently fall back to TCG (software emulation)". If, for whatever reason, KVM isn't available inside the nested VM, QEMU just quietly emulates the entire CPU in software which is orders of magnitude slower and explains why a workqueue would get starved for 10ms stretches and longer.

Since this whole setup is a VM inside a VM, the first suspect is always nested virtualization. So, checks on the Proxmox-hosted Debian VM:

root@vm:~# ls -la /dev/kvm
ls: cannot access '/dev/kvm': No such file or directory
root@vm:~# lscpu | grep -i virtualization
Virtualization type:                     full

As the /dev/kvm device isn't present, we now have confirmed that QEMU falls back to software emulation for the CPU.

The solution? Changing the CPU for the VM in Proxmox from x86-64-v2-AES (or whatever is configured) to host. This enables the VM to directly use the CPU of my Proxmox host.

This can be done comfortably in the WebUI of Proxmox. Just power down the VM.

Before:

And after:

After a reboot we have a /dev/kvm device and full support for Intel VT-x virtualisation. This confirms Proxmox is passing VT-x through cleanly to the nested VM. So KVM should be usable in principle, meaning if the inner QEMU VM is still falling back to TCG, it's not because nested virt is broken at the Proxmox level.

root@vm:~# ls -la /dev/kvm
crw-rw---- 1 root kvm 10, 232 Jul 27 00:48 /dev/kvm
root@vm:~# lscpu | grep -i virtualization
Virtualization:                          VT-x
Virtualization type:                     full

To actually find out, instead of trusting the silent kvm:tcg fallback, it's better to force KVM explicitly and let it fail loudly if something's wrong:

root@vm:~# qemu-system-x86_64 \
  -machine q35,accel=kvm \
  -cpu host \
  -m 512 \
  -nographic \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.fd \
  -drive if=pflash,format=raw,file=./OVMF_VARS_4M.fd \
  -drive file=noble-server-cloudimg-amd64.img,if=virtio,format=qcow2 \
  -net nic -net user \
  -smbios type=1,serial=ds='nocloud-net;s=http://10.0.2.2:8000/'

Removing the :tcg fallback means any KVM problem (permissions, missing nested virt on the Proxmox host CPU type, whatever) surfaces immediately as a hard error instead of a mysteriously slow VM. The parameter -cpu host on top makes sure the guest actually sees the hosts CPU features rather than a generic, lowest-common-denominator CPU model.

And the command worked, however, as we are still booting a VM inside a VM expect it to take some time. 😅

My learnings

For anyone hitting the same wall on their own Proxmox nested-VM setup, the checklist is:

1. Is /dev/kvm present on the outer (Proxmox) VM at all?
2. Is nested virtualization actually enabled on the Proxmox host itself?

   • # For Intel-CPUs - prints Y if enabled/available
     root@proxmox:~# cat /sys/module/kvm_intel/parameters/nested
     Y
     # For AMD-CPUs - prints Y if enabled/available
     root@proxmox:~# cat /sys/module/kvm_amd/parameters/nested
     Y

3. Is the outer VM's CPU type set to host in the Proxmox config? So VMX/SVM flags get passed through
   • Check in the Hardware tab of the VM inside Proxmox's WebUI, or: 
   • Check with qm config VM-ID und verify the cpu line says host

     root@proxmox:~# qm config 100
     [...]
     cores: 1
     cpu: host
     [...]

4. Does the inner QEMU call use accel=kvm (not kvm:tcg) plus -cpu host, so a broken KVM path fails loudly instead of silently degrading to software emulation?
5. Is the /dev/kvm device present on the VM where you want to start the QEMU VM?
6. Is CPU virtualization supported?

Nothing here is exotic once you know to look for it, but "silent TCG fallback" combined with "one VM inside another" is a pretty effective way to make a simple performance problem look mysterious for a while.

A small pre-preface for users who followed the "Ultimate Pi-hole Setup" tutorial from the YouTuber WunderTech

If you used the configuration files he provided on his homepage: https://www.wundertech.net/ultimate-pi-hole-setup/ you will experience the exact same problems sooner or later.

The reason is that the VIPs from keepalived are not bound on a separate dummy interface and hence the pihole-FTL process will take them into account when dynamically building the hostname and choosing the "correct IPs".

You HAVE to at least enable dns.reply.host in the /etc/pihole/pihole.toml to mark the static IP used for the server Pi-hole is running on.

Jump to The solution if you are not interested in the details.

Preface

One of the main reasons why I have my homelab is to hone my skills. And today was a day this happened.

From my one of my LAN hosts I wanted to connect to my Raspberry4 (raspi4.lan, IP: 192.168.178.8) via SSH. This host is configured as the secondary/backup instance in keepalived for the DNS VIP (192.168.178.100).

The reality however was different:

user@lanadmin:~$ ssh raspi4.lan
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Please contact your system administrator.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/user/.ssh/known_hosts:10
  remove with:
  ssh-keygen -f '/home/user/.ssh/known_hosts' -R 'raspi4.lan'
Host key for raspi4.lan has changed and you have requested strict checking.
Host key verification failed.
user@lanadmin:~$  ssh-keygen -f '/home/user/.ssh/known_hosts' -R 'raspi4.lan'
# Host raspi4.lan found: line 8
# Host raspi4.lan found: line 9
# Host raspi4.lan found: line 10
/home/user/.ssh/known_hosts updated.
Original contents retained as /home/user/.ssh/known_hosts.old

Granted I don't log on often onto raspi4.lan as everything is automated and monitored and the Pi-hole config is synced via Nebula-Sync from raspi3.lan. So I suspected I didn't purge the entries related to raspi4.lan from my ~/.ssh/known_hosts file after I re-installed that system a while ago.

user@lanadmin:~$ ssh raspi4.lan
The authenticity of host 'raspi4.lan (ULA:ffff)' can't be established.
ED25519 key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:5: raspi3.lan
    ~/.ssh/known_hosts:12: 192.168.178.9
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'raspi4.lan' (ED25519) to the list of known hosts.
Linux raspi3 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64
###############################
## Primary Pi-hole instance! ##
###############################
Last login: Fri Jul 24 21:46:36 2026 from w.x.y.z
user@raspi3:~$

Huh? How did I end up on raspi3.lan, when I clearly entered raspi4.lan as the host to connect to? Something is very wrong.

Sadly I overlooked that the IP SSH provided did list the IPv6 VIP (ending in :ffff) which is plain wrong, but it shouldn't take long for me to discover that..

Overview over the current setup

We have two Raspberry Pi's in this setup. 

Host A: Raspberry 3
IPv4: 192.168.178.9/24
IPv6: ULA:9/64
Hostname: raspi3.lan
Keepalived: Primary

Host B: Raspberry 4
IPv4: 192.168.178.8/24
IPv6: ULA:8/64
Hostname: raspi4.lan
Keepalived: Secondary

The three used VIPs are:
VIPv4: 192.168.178.100/32 (DNS: pihole.lan) and 192.168.178.101/32 unused, just for testing
VIPv6: ULA:ffff/128 (DNS: pihole.lan)

Both Raspberries run a Pi-hole instance with unbound and keepalived. The VIPs are automatically configured on eth0 on the primary instance. raspi3.lan has the primary keepalived role.

It's always DNS™

At first I suspected that I screwed up when I created my local DNS entries, but no the Local DNS records in my Pi-Hole setup are fine. The IPs for the hosts and vip match. The config is synced with nebula-sync from raspi3.lan to raspi4.lan, so no room for typos there. Nevertheless I accessed both WebUIs and checked independently, wouldn't be the first time a sync-mechanism failed. However, everything was in order.

A dig however showed the problem clearly:

root@lanadmin:~# dig -t a raspi3.lan +noall +answer
raspi3.lan.             0       IN      A       192.168.178.9
root@lanadmin:~# dig -t a raspi4.lan +noall +answer
raspi4.lan.             0       IN      A       192.168.178.100

The /etc/resolv.conf of that system is:

root@lanadmin:~# cat /etc/resolv.conf
domain lan
search lan
nameserver 192.168.178.8
nameserver 192.168.178.9

So raspi4.lan is queried first, than raspi3.lan. And I forgot to change the nameserver to the VIP...

/etc/hosts was also fine. Only standard entries for the .8 on raspi4.lan and .9 on raspi3.lan. In short: the local eth0 system IPs with the corresponding FQDN and hostname.

getent hosts however was a bit strange too. It returned the loopback address when each host asked for it's own name:

root@raspi3:~# getent hosts raspi3.lan
::1             raspi3.lan
root@raspi3:~# getent hosts raspi4.lan
192.168.178.8   raspi4.lan
root@raspi4:~# getent hosts raspi4.lan
::1             raspi4.lan
root@raspi4:~# getent hosts raspi3.lan
192.168.178.9   raspi3.lan

Hence I suspected Avahi (mDNS) as it's installed and listed before the dns resolution in /etc/nsswitch.conf:

root@raspi4:~# grep hosts /etc/nsswitch.conf
hosts:          files mdns4_minimal [NOTFOUND=return] dns

But several hard facts speak against this.

1. Avahi only works for entries ending in .local
2. dig doesn't use libnss and therefore doesn't honor mDNS/Avahi at all, but it still showed the wrong IPs

These two facts effectively eliminated Avahi/mDNS as the source of the problem. Avahi was however responsible for returning the loopback address when the host queried for it's own IP. After all mdns4_minimal was listed before dns in /etc/nsswitch.conf. So that works as designed, but doesn't help at all during troubleshooting as it just adds to the confusion..

The hosts don't know themselves...

At my wits end I took a step back and decided to check DNS from a third host against both raspi3.lan (192.168.178.9) and raspi4.lan (192.168.178.8) how both resolve the DNS A-Records of each other.

# Querying for the IPv4 of raspi3.lan
# Against raspi3.lan
root@lanadmin:~# dig @192.168.178.9 raspi3.lan +noall +answer
raspi3.lan.             0       IN      A       192.168.178.101
# Against raspi4.lan
root@lanadmin:~# dig @192.168.178.8 raspi3.lan +noall +answer
raspi3.lan.             0       IN      A       192.168.178.9
Querying for the IPv4 of raspi4.lan
Against raspi4.lan
root@lanadmin:~# dig @192.168.178.8 raspi4.lan +noall +answer
raspi4.lan.             0       IN      A       192.168.178.100
Against raspi3.lan
root@lanadmin:~# dig @192.168.178.9 raspi4.lan +noall +answer
raspi4.lan.             0       IN      A       192.168.178.8

This is looks strange.

Whenever we ask a Raspberry itself for it's own IP we get a wrong result.
Querying 192.168.178.9 (raspi3.lan) to resolve raspi3.lan returns 192.168.178.101.
Querying 192.168.178.8 (raspi4.lan) to resolve raspi4.lan returns 192.168.178.100.

How? Avahi was ruled out. There was no DHCP at play and the static DNS entries are correct.

Something was messing with my setup.

Is it Pi-hole?

I diff'd the /etc/pihole/pihole.toml suspecting I missed something in that, as I knew that the pihole.toml isn't sync by nebula-sync, but there was nothing.

user@lanadmin:~$ diff -u <(ssh 192.168.178.8 sudo cat /etc/pihole/pihole.toml) <(ssh 192.168.178.9 sudo cat /etc/pihole/pihole.toml)
--- /dev/fd/63  2026-07-25 03:44:15.628030101 +0200
+++ /dev/fd/62  2026-07-25 03:44:15.628030101 +0200
@@ -1,7 +1,7 @@
 # Pi-hole configuration file (v6.7)
 # Encoding: UTF-8
 # This file is managed by pihole-FTL
-# Last updated on 2026-07-14 10:31:50 CEST
+# Last updated on 2026-07-24 23:26:54 CEST
[dns]
Upstream DNS Servers to be used by Pi-hole. If this is not set, Pi-hole will not

I searched a bit and stumbled upon a setting regarding FTL: dns.domain and dns.expandHosts.

  [dns.domain]
    # The DNS domain used by your Pi-hole.
    #
    # This DNS domain is purely local. FTL may answer queries from its local cache and
    # configuration but *never* forwards any requests upstream *unless* you have
    # configured a dns.revServer exactly for this domain. In the latter case, all queries
    # for this domain are sent exclusively to this server (including reverse lookups).
    #
    # For DHCP, this has two effects; firstly it causes the DHCP server to return the
[... removed as DHCP is not relevant in this case ...]
    #
    # You can disable setting a domain by setting this option to an empty string.
    #
    # Allowed values are:
    #     Any valid domain
    name = "lan"

So the FTL-Cache will be queried for records ending in .lan, which is fine as it's my local domain and requests for .lan shouldn't leave my home network. Additionally I understood the sentence "but never forwards any requests upstream" as: These requests don't even hit Unbound or dnsmasq.

expandHosts makes sure the FQDN is added to /etc/hosts. Something I already did manually (or the Debian installer).

  # If set, the domain is added to simple names (without a period) in /etc/hosts in the
  # same way as for DHCP-derived names
  #
  # Allowed values are:
  #     true or false
  expandHosts = true ### CHANGED, default = false

From what I read online pihole-FTL builds the FQDN of the local system itself completely independent from settings in /etc/hosts + dns.domain. Could this be a lead?

How do we verify the entry is actually in the cache and contains the wrong IP? Glad I asked myself! The command killall -USR1 pihole-FTL
dumps the cache entries from dnsmasq into /var/log/pihole/pihole.log.

root@raspi4:~# killall -USR1 pihole-FTL
root@raspi4:~# vi /var/log/pihole/pihole.log
Jul 25 02:49:12 dnsmasq[1147]: time 1784940552
Jul 25 02:49:12 dnsmasq[1147]: cache size 10000, 0/141 cache insertions re-used unexpired cache entries.
[...]
Jul 25 02:49:12 dnsmasq[1147]: Host        Address            Flags      Expires      Source
Jul 25 02:49:12 dnsmasq[1147]: ----------- ------------------ ---------- ------------ ------------
Jul 25 02:49:12 dnsmasq[1147]: pihole.lan  192.168.178.100    4FRI   H                /etc/pihole/hosts/custom.list
Jul 25 02:49:12 dnsmasq[1125]: raspi4.lan  192.168.178.8      4FRI   H                /etc/hosts
Jul 25 02:49:12 dnsmasq[1125]: raspi3.lan  192.168.178.9      4FRI   H                /etc/pihole/hosts/custom.list

Well, that only proves my assumption that local static DNS records are NOT honored, if the hostname matches the host on which Pi-hole is running. We can clearly see that the entry for raspi4.lan has a source of /etc/hosts and not /etc/pihole/hosts/custom.list, while the record for raspi3.lan is taken from /etc/pihole/hosts/custom.list.

This proves that some, currently unknown, automatism is at work and goes horribly wrong.

If nothing helps, try rebooting

As I had no real trace of where to look next, I now focused on trying to re-produce the issue. After all, if it was just some quirk of a non-restarted service utilizing some old file - only present in the cache of it's processes file handles.. As raspi4.lan currently was the secondary node for keepalived, it didn't own the VIPs. Hence I stopped the keepalived process on raspi3.lan, forcing a failover to raspi4.lan. After making sure raspi4.lan had the VIPs I rebooted the system.

It gets stranger...

After the reboot, in order to get a bit more insight, I executed several dig queries and watched the log simultaneously. All @ip's are IPs which are currently present on the eth0 interface of the raspi4.lan host.

Those were the commands:

root@lanadmin:~# dig @192.168.178.8 raspi4.lan +noall +answer
raspi4.lan.             0       IN      A       192.168.178.100
root@lanadmin:~# dig @192.168.178.100 raspi4.lan +noall +answer
raspi4.lan.             0       IN      A       192.168.178.100
root@lanadmin:~# dig @192.168.178.101 raspi4.lan +noall +answer
raspi4.lan.             0       IN      A       192.168.178.100
On raspi4.lan, just to have everything neatly together:
root@raspi4:~# killall -USR1 pihole-FTL

And this showed up in the logfile:

root@raspi4:~# tail -f /var/log/pihole/pihole.log |grep "raspi4.lan"
Jul 25 02:58:17 dnsmasq[1125]: query[A] raspi4.lan from 192.168.178.7
Jul 25 02:58:17 dnsmasq[1125]: Pi-hole hostname raspi4.lan is 192.168.178.100
Jul 25 02:58:17 dnsmasq[1125]: query[A] raspi4.lan from 192.168.178.7
Jul 25 02:58:17 dnsmasq[1125]: Pi-hole hostname raspi4.lan is 192.168.178.100
Jul 25 02:58:17 dnsmasq[1125]: query[A] raspi4.lan from 192.168.178.7
Jul 25 02:58:17 dnsmasq[1125]: Pi-hole hostname raspi4.lan is 192.168.178.100
Jul 25 02:58:36 dnsmasq[1125]: raspi4.lan   192.168.178.8     4FRI   H     /etc/hosts

And here I tilted a bit. Why does the log state "Pi-hole hostname raspi4.lan is 192.168.178.100" but then, just seconds later state that the IP retrieved from /etc/hosts is 192.168.178.8? At least the IP 192.168.178.100 was reliably returned for all queries towards IPs on the eth0 interface of raspi4.lan with the goal to resolve the name raspi4.lan. It wasn't some kind of race-condition nor did it feel like a bug.

And just to be sure, I tried logging into raspi4.lan from lanadmin.lan:

user@lanadmin:~$ ssh raspi4.lan
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
Please contact your system administrator.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/user/.ssh/known_hosts:16
  remove with:
  ssh-keygen -f '/home/user/.ssh/known_hosts' -R 'raspi4.lan'
Host key for raspi4.lan has changed and you have requested strict checking.
Host key verification failed.
user@lanadmin:~$ host raspi4.lan
raspi4.lan has address 192.168.178.100
raspi4.lan has IPv6 address fd6d:ULA:ffff

Nope, still the same problem. Only that this time I would have actually logged on to the right host, as the IP was currently owned by raspi4.lan. Nonetheless did the hostname still resolve to the wrong IP.

I had no idea why the IP kept changing. Yes, I had provided no specific IP for pihole-FTL to listen on, but this should never lead to such an behaviour. I decided to read through /etc/pihole/pihole.toml, if only to set a specific listener address and do a bit more troubleshooting.

Finally: Enlightenment

Revelation hit me, when I read the [dns.reply.host] block. The describe behaviour matched the observed one perfectly!

    [dns.reply.host]
      # Use a specific IPv4 address for the Pi-hole host? By default, FTL determines the
      # address of the interface a query arrived on and uses this address for replying to A
      # queries with the most suitable address for the requesting client.
      #
      # This setting can be used to use a fixed, rather than the dynamically obtained,
      # address when Pi-hole responds to the following names:
      # - "pi.hole"
      # - "<the device's hostname>"
      # - "pi.hole.<local domain>"
      # - "<the device's hostname>.<local domain>"
      #
      # Allowed values are:
      #     true or false
      force4 = false
  # Custom IPv4 address for the Pi-hole host
  #
  # Allowed values are:
  #     A valid IPv4 address or empty string ("")
  IPv4 = ""

  # Use a specific IPv6 address for the Pi-hole host? See description for the IPv4
  # variant above for further details.
  #
  # Allowed values are:
  #     true or false
  force6 = false

  # Custom IPv6 address for the Pi-hole host
  #
  # Allowed values are:
  #     A valid IPv6 address or empty string ("")
  IPv6 = ""</code></pre>
As it can be clearly seen, static IPs for the local hostname were disabled. This made FTL choose a new "best matching" IP for each received query. Which is such a strange mechanism to implement! Why obscure such things!?
Why design such a ... mechanism?
And then it hit me.. Novice and inexperienced users. Pi-hole is a DNS and Ad-Blocker. Primarily aimed at home users. And those lack knowledge and experience. I'm a frequent reader of subreddits like r/selfhosted or r/HomeServer so I know full well how many users struggle with IPs, interface bindings, file rights, etc. All the basic stuff one learns over time but can be pretty hard for people new to Linux.
I suspect this mechanism was developed to ease the usage of Pi-hole, to just "make it work" no matter what. Alas.. This caused way more trouble for an experienced user this way. And this is why I don't really like that they implemented this mechanism. It's just one of these automatisms which ignore standards and work without following an established process. Effectively hindering novice users to learn "How it is normally done"?
The solution
The fix was rather easy. Just enable dns.reply.host for IPv4 and IPv6 and set the corresponding IPs. Then restart the service. Done. Below is the config for raspi4.lan.
    [dns.reply.host]
      # Use a specific IPv4 address for the Pi-hole host? By default, FTL determines the
      # address of the interface a query arrived on and uses this address for replying to A
      # queries with the most suitable address for the requesting client.
      #
      # This setting can be used to use a fixed, rather than the dynamically obtained,
      # address when Pi-hole responds to the following names:
      # - "pi.hole"
      # - "<the device's hostname>"
      # - "pi.hole.<local domain>"
      # - "<the device's hostname>.<local domain>"
      #
      # Allowed values are:
      #     true or false
      force4 = true ### CHANGED, default = false
  # Custom IPv4 address for the Pi-hole host
  #
  # Allowed values are:
  #     A valid IPv4 address or empty string ("")
  IPv4 = "192.168.178.8" ### CHANGED, default = ""

  # Use a specific IPv6 address for the Pi-hole host? See description for the IPv4
  # variant above for further details.
  #
  # Allowed values are:
  #     true or false
  force6 = true ### CHANGED, default = false

  # Custom IPv6 address for the Pi-hole host
  #
  # Allowed values are:
  #     A valid IPv6 address or empty string ("")
  IPv6 = "fd6d:ULA:8" ### CHANGED, default = ""</code></pre>
Please note that I obscured parts of my ULA IPv6 address.
Why didn't I notice sooner?
Then there is always this question which creeps into ones mind: Why didn't I notice it sooner? Why did it work for so long?
At least in this case the answer is simple: I rarely need to login into these systems.
Other improvements
Dummy network interface for VIPs
Apparently it is also better to create a dummy network device and let keepalived bind the VIPs to that interface. One big advantage is that dummy interfaces don't reply to ARP-Requests at all. Which is a crucial problem in HA setup. And also one point WunderTech didn't mention with one word in his tutorial..
VRRP-Scripts to check DNS service availbility/healthiness
In the current setup keepalived will only switch to another machine if it stops sending out VRRP-Announcements, which usually only happens when a machine fails completely (power cut or really catastrophic failures). If just the pihole-FTL service, Unbound or any other piece of software - apart from keepalived - fails nothing will happen.
For this, keepalived supports the execution of VRRP-scripts. These will be executed every few seconds and are there to check service availbility/healthiness and trigger a failover if the script execution fails or doesn't exit successfully.
I plan to write a blog post about that too. When it is ready, I will link it here.