Feuerfest

Just the private blog of a Linux sysadmin

Using nebula-sync to synchronize your Pi-hole instances

As I have 2 running Pi-hole instances I have the problem of keeping them in sync. I do want all local DNS entries to be present on both. Also I do want the same filter lists, exceptions, etc.

Entering: nebula-sync. After Pi-hole released version 6 and made huge changes towards the architecture & API, the often used gravity-sync stopped working and was archived.

As nebula-sync can be run as a docker image the setup is fairly easy.

I just logged into my Portainer instance and spun up a new stack with the following YAML. I disable the gravity run after nebula-sync as this currently breaks the replica. The webserver process is killed and doesn't come back. Hence no connections to port 80 (HTTP) or 443 (HTTPS) are possible and therefore the API can't be reached either.

This is currently investigated/worked on in the following issue: Pi-hole FTL issue #2395: FTL takes 5 minutes to reboot?

---
services:
  nebula-sync:
    image: ghcr.io/lovelaze/nebula-sync:latest
    container_name: nebula-sync
    environment:
    - PRIMARY=http://ip1.ip1.ip1.ip1|password1
    - REPLICAS=http://ip2.ip2.ip2.ip2|password2
    - FULL_SYNC=true
    - RUN_GRAVITY=false   # running Gravity after sync breaks the replica, see: https://github.com/pi-hole/FTL/issues/2395
    - CRON=*/15 * * * *
    - TZ=Europe/Berlin

And if everything works we see the following:

2025-05-26T16:01:24Z INF Starting nebula-sync v0.11.0
2025-05-26T16:01:24Z INF Running sync mode=full replicas=1
2025-05-26T16:01:24Z INF Authenticating clients...
2025-05-26T16:01:25Z INF Syncing teleporters...
2025-05-26T16:01:25Z INF Syncing configs...
2025-05-26T16:01:25Z INF Running gravity...
2025-05-26T16:01:25Z INF Invalidating sessions...
2025-05-26T16:01:25Z INF Sync completed

When I did have RUN_GRAVITY=true in my stack I would always see the first sync succeeding. This run would however kill the webserver - hence the API isn't reachable any more and the nebula-sync container would only log the following error message:

2025-05-26T16:34:29Z INF Starting nebula-sync v0.11.0
2025-05-26T16:34:29Z INF Running sync mode=full replicas=1
2025-05-26T16:34:29Z INF Authenticating clients...
2025-05-26T16:34:31Z INF Syncing teleporters...
2025-05-26T16:34:31Z INF Syncing configs...
2025-05-26T16:34:31Z INF Running gravity...
2025-05-26T16:34:31Z INF Invalidating sessions...
2025-05-26T16:34:31Z INF Sync completed

2025-05-26T16:35:00Z INF Running sync mode=full replicas=1
2025-05-26T16:35:00Z INF Authenticating clients...
2025-05-26T16:35:02Z INF Invalidating sessions...
2025-05-26T16:35:04Z WRN Failed to invalidate session for target: http://ip2.ip2.ip2.ip2
2025-05-26T16:35:04Z ERR Sync failed error="authenticate: http://ip2.ip2.ip2.ip2/api/auth: Post \"http://ip2.ip2.ip2.ip2/api/auth\": dial tcp ip2.ip2.ip2.ip2:80: connect: connection refused"

Comments

Sweet memories

At one of my previous employers we were allowed to create our own groups in the Intranet portal. There I created a group called "Christian's curious cyber content cabaret" and renamed it to "Christian's curious Corona cyber content cabaret" during the pandemic years.

The image below is the header image from that group which - as far as I am informed - still exists today. Years after I left that company. Albeit nobody posts there anymore.

Comments

Installing Unbound as recursive DNS server on my PiHole

I run a Pi-hole installation on each of my Raspberry 3 & 4. As I do like to keep my DNS queries as much under my control as I can, I also installed Unbound to serve as recursive DNS server. This way all DNS queries will be handled by my Raspberry Pis.

Pi-hole is already installed using one of the following methods: https://github.com/pi-hole/pi-hole/#one-step-automated-install. If you don't have that done yet, do it first.

There is a good guide at the Pi-hole website which I will basically following.

https://docs.pi-hole.net/guides/dns/unbound/

root@host:~# apt install unbound

Regarding the configuration file I go with the one in the guide. However as I did have some problems in that past I needed to troubleshoot I include the following lines regarding loglevels and verbosity:

root@host:~# head /etc/unbound/unbound.conf.d/pihole.conf
server:
    # If no logfile is specified, syslog is used
    logfile: "/var/log/unbound/unbound.log"
    val-log-level: 2
    # Default is 1
    #verbosity: 4
    verbosity: 1

    interface: 127.0.0.1
    port: 5335
root@host:~# 

You can add that if you want but it's not needed to make Unbound work.

Next the guide tells us to download the root hints. A file maintained by Internic which contains information about the 13 DNS root name servers. Under Debian we don't need to download the named.root file from Internic as shown in the guide. Debian has its own package for that: dns-root-data.

It no only contains information about the 13 DNS root name servers but also the needed DNSSEC keys (also called root trust anchors). And together with unattended-upgrades we even automate updating that. Saving us the creation of a Cronjob or systemd timer.

root@host:~# apt install dns-root-data

In order for Unbound to have a directory and logfile to write into we need to create that:

root@host:~# mkdir -p /var/log/unbound
root@host:~# touch /var/log/unbound/unbound.log
root@host:~# chown unbound /var/log/unbound/unbound.log

As we are running under Debian we now need to tweak the Unbound config a little bit. Else we will get problems with DNSSEC. For this we are deleting a Debian generated file from Unbound and comment out the unbound_conf= line in /etc/resolvconf.conf so that it isn't included anymore.

root@host:~# sed -Ei 's/^unbound_conf=/#unbound_conf=/' /etc/resolvconf.conf
root@host:~# rm /etc/unbound/unbound.conf.d/resolvconf_resolvers.conf

Now all that is left is restarting Unbound.

root@host:~# systemctl restart unbound.service

Testing DNS resolution:

root@host:~# dig pi-hole.net @127.0.0.1 -p 5335

; <<>> DiG 9.18.33-1~deb12u2-Raspbian <<>> pi-hole.net @127.0.0.1 -p 5335
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46191
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;pi-hole.net.                   IN      A

;; ANSWER SECTION:
pi-hole.net.            300     IN      A       3.18.136.52

;; Query time: 169 msec
;; SERVER: 127.0.0.1#5335(127.0.0.1) (UDP)
;; WHEN: Sun May 25 18:21:25 CEST 2025
;; MSG SIZE  rcvd: 56

And to verify & falsify DNSSEC. This request must return an A-Record for dnssec.works.

root@host:~# dig dnssec.works @127.0.0.1 -p 5335

; <<>> DiG 9.18.33-1~deb12u2-Raspbian <<>> dnssec.works @127.0.0.1 -p 5335
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14076
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;dnssec.works.                  IN      A

;; ANSWER SECTION:
dnssec.works.           3600    IN      A       46.23.92.212

;; Query time: 49 msec
;; SERVER: 127.0.0.1#5335(127.0.0.1) (UDP)
;; WHEN: Sun May 25 18:22:52 CEST 2025
;; MSG SIZE  rcvd: 57

This request will not result in an A-Record.

root@host:~# dig fail01.dnssec.works @127.0.0.1 -p 5335

; <<>> DiG 9.18.33-1~deb12u2-Raspbian <<>> fail01.dnssec.works @127.0.0.1 -p 5335
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 1552
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;fail01.dnssec.works.           IN      A

;; Query time: 19 msec
;; SERVER: 127.0.0.1#5335(127.0.0.1) (UDP)
;; WHEN: Sun May 25 18:23:41 CEST 2025
;; MSG SIZE  rcvd: 48

Now all that is left to connect our Pi-hole with Unbound. Logon to your Pi-hole website and navigate to Settings -> DNS. Expand the line Custom DNS servers and enter to IP and Port to our Unbound server. 127.0.0.1#5335 for IPv4 and ::1#5335 for IPv6. If you don't use one of these two just don't add the line. After that hit "Save & Apply" and we are done.

Creating a logrotate config for Unbound

Sadly Unbound still doesn't deliver a logrotate config with its package. Therefore I just copy & paste from my previous article Howto properly split all logfile content based on timestamps - and realizing my own fallacy.

root@host:~# cat /etc/logrotate.d/unbound
/var/log/unbound/unbound.log {
        monthly
        missingok
        rotate 12
        compress
        delaycompress
        notifempty
        sharedscripts
        create 644
        postrotate
                /usr/sbin/unbound-control log_reopen
        endscript
}

Troubleshooting

fail01.dnssec.works timed out

The host fail01.dnssec.works tends to not answer requests sometimes. Others noticed this too. dig will only show the following message:

root@host:~# dig fail01.dnssec.works @127.0.0.1 -p 5335
;; communications error to 127.0.0.1#5335: timed out
;; communications error to 127.0.0.1#5335: timed out
;; communications error to 127.0.0.1#5335: timed out

; <<>> DiG 9.18.33-1~deb12u2-Raspbian <<>> fail01.dnssec.works @127.0.0.1 -p 5335
;; global options: +cmd
;; no servers could be reached

If that is the case, just execute the command again. Usually it will work the second time. Or just wait a few minutes. Sometimes the line ;; communications error to 127.0.0.1#5335: timed out will be printed, but the dig query will work after that nonetheless.

Comments

Friedrich Merz ist ein rückständiger Clown

Friedrich Merz wettert also gegen die Work-Life-Balance?

In seiner Rede im Bundestag sagte er:

Mit Vier-Tage-Woche und Work-Life-Balance werden wir den Wohlstand dieses Landes nicht erhalten können.

Wow. Wie Rückständig ein Mann ein doch denken kann. Was für ein Clown.

Er musste vermutlich noch nie jahrelange, unbezahlte Carearbeit (Haushalt, Kinder, Angehörige) leisten, oder?
Litt nie unter AD(H)S, Autismus, CFS/ME, COPD, Depressionen, Burnout, Multiple Sclerosis oder anderen Erkrankungen, oder?
Hat nie einfach mal so, weil die Gesellschaft ist, wie sie ist, ständig immer nur Druck bekommen, und "Mist fressen müssen" um irgendwie die Miete zahlen zu können?

Kapiert dieser Mensch eigentlich wovon er da redet?
Wie verdammt privilegiert dieser alte, weiße Mann sein Leben lang gewesen sein muss, damit er überhaupt auf die Idee kommen kann so etwas öffentlich im Bundestag(!!) zu sagen?

Wie entkoppelt von der Realität kann man sein?
Friedrich Merz: Ja.

👿

Quelle u.a.: https://www.tagesspiegel.de/gesellschaft/bundeskanzler-lehnt-vier-tage-woche-ab-work-life-balance-ist-langst-ein-standortfaktor-13691380.html

Comments

Better SQL*Plus output

Oracle's sqlplus tool is horrible to work with as it lacks many basic command line features. No tab-completition, no command history scrolling with Arrow-Up/-Down or the like and additionally the output formatting is near unreadable.

Hence I was delighted when I found out that command set markup csv on will switch to CSV output formatting.

Enable it the following way:

oracle@host:~$ sqlplus user/password @iXXXc1
SQL> set markup csv on
Comments

Working in IT is sometimes like this...

Task: Upgrade a custom application on a customer system

Technical details: This involves executing some script beforehand, than a Puppet Run and some menial afterwork/clean up tasks.

  1. Prepare everything. Shutdown system. Take a Snapshot.
  2. Executing the script: Error! I can't find this dependency!
    • Oh, yeah. That package requiring the dependency isn't needed here. Commenting out that package from the list.
  3. Executing script again. It finishes properly. Fine.
  4. Running Puppet agent.
    • Error after 45seconds: Left over process found. Aborting.
    • Checking Relases Notes: Ah, known bug. ps -ef |process and kill pid is the workaround ok.
    • ps'ing and kill'ing.
  5. Run puppet agent -t again.
  6. It runs for 30min, then aborts. A partition has not enough free space. Narf.
    • Searching for stuff to delete/compress/move.
    • Start Puppet again to verify it works. It does.
  7. Revert system to snapshot as this is policy.
  8. Shutdown. Snapshot. So I have a clean start with all known error sources eliminated.
  9. Puppet agent run.
  10. It runs for 2.5 hours. Then breaks with Execution expired and some Ruby stacktrace.
    • Checking Release Notes again. Nothing. Hmm ok.
    • Reading code to understand the problem. No real insight gained. It SHOULD work.
  11. Reverting to snapshot. Now taking a 2nd snapshot right before the Puppet run.
  12. 3rd Puppet agent run. Same error after 2.5 hours.
    • Oh come on..
  13. Reverting to snapshot.
  14. Installing the RPM which installation triggers the stacktrace manually. It works. Package install takes 45min.
  15. Reverting to snapshot.
  16. Just for the sake of trying I start a 4th Puppet run. Expecting no change.
  17. After 55min of Puppet running I send a status mail to project lead, some other involved people, application developer, etc. that I couldn't update the system in time today and will need help and an additional 1-2 hours tomorrow.
  18. 5min after sending the mail: Hey, it's me! Puppet! I finished. No errors!
  19. I send out a follow-up mail, stating that the biggest part of the update is done. But due to the missed time window I will still need 1-2 hours tomorrow for the afterwork/clean up tasks.

And I still don't where or what the error was! Due to Execution expired I suspect that somewhere deep in the Ruby code something timed out. Maybe something which isn't documented nor written in any logfile. Something which had a maintenance window at exactly the same hours as we had.

Hopefully the developer knows more.

Sometimes IT sucks.😂

Comments