Feuerfest

Part 1 of this series is here: Little helper scripts - Part 1: no-screenlock-during-meeting.ps1 or use the following tag-link: https://admin.brennt.net/tag/littlehelperscripts

This script is no rocket science. Nothing spectacular. But the amount of hours it saved me in various projects is astonishing.

The sad reality

There are far too many companies (even IT-focused companies!) out there who have a very low level of automation. Virtual machines are created by hand - not by some script using an API. Configurations are not deployed via of some Configuration Management software like Puppet/OpenVox/Chef/Ansible or a Runbook automation software like Rundeck - no, they are handcrafted. Bespoke. System administration like it's 1753. With all the implications and drawbacks that brings.

Containerisation? Yeah.. Well.. "A few docker containers here and there but nobody in the company really knows how that stuff works so we leave it alone" is a phrase I have heard more than a few times. Either directly, or reported from colleagues working in other companies.

This means that I have to log on to systems manually and execute commands by hand. Something I can do and do regularly in my home lab. But to do it for dozens or even hundreds of systems? Yeah... No. Sorry, I've got better things to do. And as an external consultant, the client always keeps an eye on my performance metrics. After all, they are paying my employer a lot of money for my services. Sitting there all day and getting paid to copy and paste commands? It doesn't look good on my performance reporting spreadsheet and it doesn't meet my personal standards of what a consultant should be able to deliver.

I'm just a guest

What's more, because of my work as a consultant, I'm just an external contractor. I come in for a few months to solve a problem or help with a task, and then I move on to the next project at another company. That means I can't just do everything the way I want. I can't just go and install software on all the systems, even though I've been given root privileges. I can't just implement Ansible. I have to design my solutions so that they survive and continue to work when I'm gone. Sure, I can introduce dozens of new technologies and whole new technology stacks. I'm sure my employer would love to have the follow-on support contracts for those constructs. But going it alone will seriously damage the customer relationship. Especially with the IT people. After all, they'll be the ones stuck with new technology they don't understand and will have to spend time learning and familiarising themselves with. And I have been a sysadmin long enough to know what they will think of me if I start pulling such stunts.

Of course I can suggest changes. I can push for standardisation and automation. But for most customers, that will make no difference. After all, there are reasons why a company has stopped keeping up with technology. And fixing this takes time and usually involves a complete change of the dominating mindset. Something I cannot achieve as a lone consultant.

Scripting to the rescue!

First I went with the cheap & easy solution of for server in hosta hostb hostc; do ssh user@$server "command --some-parameter bla"; done but I grew tired of writing it all completely anew for each task.

Natively systems are often grouped into categories (webservers, etc.) or perform the same tasks (think of clusters). Hence commands must be executed on the same set of hosts again and again. One of my colleagues already compiled lists of hostnames group by tasks, roles and installed software. As some systems had the same software installed but were just configured to do different tasks with that software.

Through these list I got an idea: Why not feed those into a for or do-while loop and be done?

In the end I added some safety & DNS checks and named the script automation.sh. Later I added the capability to log the output on each host and named the script automation2.sh, which can be viewed below.

Yes, it's just a glorified nesting of if-statements but the amount of time this script saved me is insane. And as it utilizes only basic Posix & Bash commands I've yet to find a system were it can't be executed.

As always: Please check my GitHub for the most recent version as I won't update the script shown in this article.

#!/bin/bash
# vim: set tabstop=2 smarttab shiftwidth=2 softtabstop=2 expandtab foldmethod=syntax :
#
# Small script to automate custom shell command execution
# Current version can be found here:
# https://github.com/ChrLau/scripts/blob/master/automation2.sh

# Bash strict mode
#  read: http://redsymbol.net/articles/unofficial-bash-strict-mode/
set -euo pipefail
IFS=$'\n\t'

# Set pipefail variable
# As we use "ssh command | tee" and tee will always succeed our check for non-zero exit-codes doesn't work
#
# The exit status of a pipeline is the exit status of the last command in the pipeline,
#  unless the pipefail option is enabled (see: The Set Builtin).
# If pipefail is enabled, the pipeline's return status is the value of the last (rightmost)
#  command to exit with a non-zero status, or zero if all commands exit successfully.

VERSION="1.6"
SCRIPT="$(basename "$0")"
SSH="$(command -v ssh)"
TEE="$(command -v tee)"
# Colored output
RED="\e[31m"
GREEN="\e[32m"
ENDCOLOR="\e[0m"

# Test if ssh is present and executeable
if [ ! -x "$SSH" ]; then
  echo "${RED}This script requires ssh to connect to the servers. Exiting.${ENDCOLOR}"
  exit 2;
fi

# Test if tee is present and executeable
if [ ! -x "$TEE" ]; then
  echo "${RED}tee not found.${ENDCOLOR} ${GREEN}Script can still be used,${ENDCOLOR} ${RED}but option -w CAN NOT be used.${ENDCOLOR}"
fi

function HELP {
  echo "$SCRIPT $VERSION: Execute custom shell commands on lists of hosts"
  echo "Usage: $SCRIPT -l /path/to/host.list -c \"command\" [-u <user>] [-a <YES|NO>] [-r] [-s \"options\"] [-w \"/path/to/logfile.log\"]"
  echo ""
  echo "Parameters:"
  echo " -l   Path to the hostlist file, 1 host per line"
  echo " -c   The command to execute. Needs to be in double-quotes. Else getops interprets it as separate arguments"
  echo " -u   (Optional) The user used during SSH-Connection. (Default: \$USER)"
  echo " -a   (Optional) Abort when the ssh-command fails? Use YES or NO (Default: YES)"
  echo " -r   (Optional) When given command will be executed via 'sudo su -c'"
  echo " -s   (Optional) Any SSH parameters you want to specify Needs to be in double-quotes. (Default: empty)"
  echo "                 Example: -s \"-i /home/user/.ssh/id_user\""
  echo " -w   (Optional) Write STDERR and STDOUT to logfile (on the machine where $SCRIPT is executed)"
  echo ""
  echo "No arguments or -h will print this help."
  exit 0;
}

# Print help if no arguments are given
if [ "$#" -eq 0 ]; then
  HELP
fi

# Parse arguments
while getopts ":l:c:u:a:hrs:w:" OPTION; do
  case "$OPTION" in
    l)
      HOSTLIST="${OPTARG}"
      ;;
    c)
      COMMAND="${OPTARG}"
      ;;
    u)
      SSH_USER="${OPTARG}"
      ;;
    a)
      ABORT="${OPTARG}"
      ;;
    r)
      SUDO="YES"
      ;;
    s)
      SSH_PARAMS="${OPTARG}"
      ;;
    w)
      LOGFILE="${OPTARG}"
      ;;
    h)
      HELP
      ;;
    *)
      HELP
      ;;
# Not needed as we use : as starting char in getopts string
#    :)
#      echo "Missing argument"
#      ;;
#    \?)
#      echo "Invalid option"
#      exit 1
#      ;;
  esac
done

# Give usage message and print help if both arguments are empty
if [ -z "$HOSTLIST" ] || [ -z "$COMMAND" ]; then
  echo "You need to specify -l and -c. Exiting."
  exit 1;
fi

# Check if username was provided, if not use $USER environment variable
if [ -z "$SSH_USER" ]; then
  SSH_USER="$USER"
fi

# Check for YES or NO
if [ -z "$ABORT" ]; then
  # If empty, set to YES (default)
  ABORT="YES"
# Check if it's not NO or YES - we want to ensure a definite decision here
elif [ "$ABORT" != "NO" ] && [ "$ABORT" != "YES" ]; then
  echo  "-a accepts either YES or NO (case-sensitive)"
  exit 1
fi

# If variable logfile is not empty
if [ -n "$LOGFILE" ]; then

  # Check if logfile is not present
  if [ ! -e "$LOGFILE" ]; then
    # Check if creating it was unsuccessful
    if ! touch "$LOGFILE"; then
      echo "${RED}Could not create logfile at $LOGFILE. Aborting. Please check permissions.${ENDCOLOR}"
      exit 1
    fi
  # When logfile is present..
  else
    # Check if it's writeable and abort when not
    if [ ! -w "$LOGFILE" ]; then
      echo "${RED}$LOGFILE is NOT writeable. Aborting. Please check permissions.${ENDCOLOR}"
      exit 1
    fi
  fi
fi

# Execute command via sudo or not?
if [ "$SUDO" = "YES" ]; then
  COMMANDPART="sudo su -c '${COMMAND}'"
else
  COMMANDPART="${COMMAND}"
fi

# Check if hostlist is readable
if [ -r "$HOSTLIST" ]; then
  # Check that hostlist is not 0 bytes
  if [ -s "$HOSTLIST" ]; then
  
    while IFS= read -r HOST
    do

      getent hosts "$HOST" &> /dev/null
      
      # getent returns exit code of 2 if a hostname isn't resolving
      # shellcheck disable=SC2181
      if [ "$?" -ne 0 ]; then
        echo -e "${RED}Host: $HOST is not resolving. Typo? Aborting.${ENDCOLOR}"
        exit 2
      fi

      # Log STDERR and STDOUT to $LOGFILE if specified
      if [ -n "$LOGFILE" ]; then
        echo -e "${GREEN}Connecting to $HOST ...${ENDCOLOR}" 2>&1 | tee -a "$LOGFILE"
        ssh -n -o ConnectTimeout=10 "${SSH_PARAMS}" "$SSH_USER"@"$HOST" "${COMMANDPART}" 2>&1 | tee -a "$LOGFILE"

        # Test if ssh-command was successful
        # shellcheck disable=SC2181
        if [ "$?" -ne 0 ]; then
          echo -n -e "${RED}Command was NOT successful on $HOST ... ${ENDCOLOR}" 2>&1 | tee -a "$LOGFILE"

          # Shall we proceed or not?
          if [ "$ABORT" = "YES" ]; then
            echo -n -e "${RED}Aborting.${ENDCOLOR}\n" 2>&1 | tee -a "$LOGFILE"
            exit 1
          else
            echo -n -e "${GREEN}Proceeding, as configured.${ENDCOLOR}\n" 2>&1 | tee -a "$LOGFILE"
          fi
        fi

      else

        echo -e "${GREEN}Connecting to $HOST ...${ENDCOLOR}"
        ssh -n -o ConnectTimeout=10 "${SSH_PARAMS}" "$SSH_USER"@"$HOST" "${COMMANDPART}"

        # Test if ssh-command was successful
        # shellcheck disable=SC2181
        if [ "$?" -ne 0 ]; then
          echo -n -e "${RED}Command was NOT successful on $HOST ... ${ENDCOLOR}"

          # Shall we proceed or not?
          if [ "$ABORT" = "YES" ]; then
            echo -n -e "${RED}Aborting.${ENDCOLOR}\n"
            exit 1
          else
            echo -n -e "${GREEN}Proceeding, as configured.${ENDCOLOR}\n"
          fi
        fi

      fi

    done < "$HOSTLIST"

  else
    echo -e "${RED}Hostlist \"$HOSTLIST\" is empty. Exiting.${ENDCOLOR}"
    exit 1
  fi

else
  echo -e "${RED}Hostlist \"$HOSTLIST\" is not readable. Exiting.${ENDCOLOR}"
  exit 1
fi

Like many IT people, I pay to have my own server for personal projects and self-hosting. As such, I am responsible for securing these systems as they are, of course, connected to the internet and provide services to everyone. Like this blog for example. So I often read about people installing Fail2Ban to "increase the security of their systems".

And every time I read this, I am like this popular meme from the TV series Firefly:

As I don't share this view of Fail2Ban - in fact, I'm against the view that it improves security - but I'll keep quiet, knowing that starting this discussion is simply not helpful. Nor that it is wanted.

For me, Fail2Ban is just a log cleanup tool. Its only benefit is that it will catch repeated login attempts and deny them by adding firewall rules to iptables/nftables to block traffic from the offending IPs. This prevents hundreds or thousands of extra logfile lines about unsuccessful login attempts. So it doesn't improve the security of a system, as it doesn't prevent unauthorised access or strengthen authorisation or authentication methods. No, Fail2Ban - by design - can only act when an IP has been seen enough times to trigger an action from Fail2Ban.

With enough luck on the part of the attacker - or negligence on the part of the operator - a login will still succeed. Fail2Ban won't save you if you allow root to login via SSH with the password "root" or "admin" or "toor".

Granted, even Fail2Ban knows this and they write this prominently on their project's GitHub page:

Though Fail2Ban is able to reduce the rate of incorrect authentication attempts, it cannot eliminate the risk presented by weak authentication. Set up services to use only two factor, or public/private authentication mechanisms if you really want to protect services.

Source: https://github.com/fail2ban/fail2ban

Yet, the number of people I see installing Fail2Ban to "improve SSH security" but refusing to use public/private key authentication is staggering.

I only allow public/private key login for select non-root users specified via AllowUsers. Absolutely no password logins allowed. I've changed the SSH port away from port 22/tcp and I don't run Fail2Ban. As with this setup, there are not that many login attempts anyway. And those that do tend to abort pretty early on when they realise that password authentication is disabled.

Although in all honesty: Thanks to services like https://www.shodan.io/ and others finding out the changed SSH port is not a problem. There are dozens of tools that can detect what is running behind a port and act accordingly. Therefore I do see my fair share of SSH bruteforce attempts. Denying password authentication is the real game changer.

So do yourself a favour: Don't rely on Fail2Ban for SSH security. Rely on the following points instead:

• Keep your system up to date! As this will also remove outdated/broken ciphers and add support for new, more secure ones. All the added & improved SSH security gives you nothing if an attacker can gain root privileges via another vulnerability.
• AllowUsers or AllowGroups: To only specified users to login in via SSH. This is generally preferred over using DenyUsers or DenyGroups as it's generally wiser to specify "what is allowed" as to specify "what is forbidden". As the bad guys are pretty damn good in finding the flaws and holes in the later one.
  • If you don't see what I mean, read this hilarious article about how "virtual Minecraft Denmark" was blown up with minecarts using TNT: https://www.gamespot.com/articles/danish-government-creates-entire-country-in-minecraft-users-promptly-blow-it-up-and-plant-american-flag/1100-6419412/
• DenyUsers or DenyGroups: Based on your groups this may be useful too but I try to avoid using this.
• AuthorizedKeysFile /etc/ssh/authorized_keys/%u: This will place the authorized_keys file for each user in the /etc/ssh/authorized_keys/ directory. This ensures users can't add public keys by themselves. Only root can.
• PermitEmptyPasswords no: Should be self-explaining. Is already a default.
• PasswordAuthentication no and PubkeyAuthentication yes: Disables authentication via password. Enabled authentication via public/private keys.
• AuthenticationMethods publickey: To only offer publickey authentication. Normally there is publickey,password or the like.
• PermitRootLogin no: Create a non-root account and use su. Or install sudo and use that if needed. See also AllowUsers.

It happened again. Someone asked an LLM a benign technical question. "How to check the speed of a hard drive?"

And the LLM answered!

Only the human wasn't clever enough to understand the answer. Nor was it particularly careful in formulating the question. Certain details were left out, critical information was not recognised as such. The human was too greedy to acquire this long-sought knowledge.

But... Is an answer to a thoughtlessly asked question a reliable & helpful one?

The human didn't care. He copy & pasted the command directly into the root shell of his machine.

The sacred machine spirit awakened to life and fulfilled its divine duty.

And... It actually gave the human the answer he was looking for. Only later did the human learn that the machine had given him even more. The machine spirit answered the question, "How quickly can the hard drive overwrite all my cherished memories from 20 years ago that I never backed up?"

TL;DR: r/selfhosted: TIFU by copypasting code from AI. Lost 20 years of memories

People! Make backups! Never, ever work on your only, single, lifetime copy of your data while executing potentially harmful commands. Jesus! Why do so many people fail to grasp this? I don't get it..

And as for AI tools like ChatGPT, DeepSeek & others: Yes, they can be great & useful. But they don't understand. They have no sentience. They can't comprehend. They have no understanding of syntax or semantics. And therefore they can't check if the two match. ChatGPT won't notice that the answer it gives you doesn't match the question. Hell, there are enough people out there who won't notice. YOU have to think for the LLM! YOU have to give the full, complete context. Anything left out will not be considered in the answer.

In addition: Pick a topic you're well-versed in. Do you know how much just plain wrong stuff there is on the internet about that subject? Exactly.

Now ask yourself: Why should it be any different in a field you know nothing about?

All the AI companies have just pirated the whole internet. Copied & absorbed what they could in the vague hope of getting that sweet, sweet venture capital money. Every technically incorrect solution. Every "easy fix" that says "just do a chmod -R 777 * and it works".

And you just copy and paste that into your terminal?

If an LLM gives you an answer you do the following:

1. You ask the LLM to check if the answer is correct. Oh, yes. You will be surprised how often an LLM will correct itself.
   • And we haven't even touched the topic of hallucinations...
2. Then you find the documentation/manpage for that command
3. You read said documentation/manpage(s) until you understand what the code/command does. How it works
4. Now you may execute that command

Sounds like too much work? Yeah.. About that...

I switched to a new blog theme. While Solen was great, I didn't like the mandatory article images. It just makes no sense to search for the 20th "Code lines displayed in some kind of terminal or IDE" image for a blogpost. Also the overall look & feel was a bit too much "early 2000s". I wanted something that looked a bit more refined. More clean.

Browsing through https://themes.bludit.com/ I found the Keep It Simple-Theme. Only that it was last modified in 2020 for a 2.x Bludit version, while we are now at 3.16.2. 

Luckily only a few modifications were needed and I finally took the time to create a separate Git-Repository for my theme modifications. Have a look at if you want to use it too: https://github.com/ChrLau/keep-it-simple

Although it contains some CSS changes, but I kept the original CSS in via comments so it should be rather easy to switch back. The changes mostly affect colours, blockquotes, fonts. Not the general layout, hence incorporating updates from the original StyleShout template should be easy.

Some minor tweaks are still coming, as I still have to check mobile & widescreen support and I want a different font used in blockquotes. The current one merriweather-italic doesn't look good as the vertical alignment is too uneven for my liking. Especially the letter "e" is too high and gives every word with an "e" a somewhat strange look.

But in general I am happy with the current look & feel.

I mean, the W3 Validator still isn't completely happy, but there are some things I can't fix directly and the only option I have is to open a pull request: Bludit: Fix canoncial links in siteHead plugin. Sadly alt-tags for images are also not possible and the corresponding Issue in the Bludit repository is closed since 2022 as "this will be fixed with Bludit 4.x". Meanwhile we are still at Bludit 3.x, a 4.x branch doesn't even exist and development really slowed down and there isn't much activity from the only developer. I seriously hope these are not bad omens..

Also no activity on my Cookie security issue Enhance cookie security by setting samesite attribute and adding __Secure- prefix to sessionname (Bludit issue 1582) and at least one unpatched Stored XSS does exist: Bludit - Stored XSS Vulnerability (Bludit issue 1579).

Let's just hope the best for now...

gethomepage.dev: https://gethomepage.dev/assets/homepage_demo_clip.webp

homepage (GitHub) - the dashboard I use to keep an overview of all the services I run in my LAN released their 1.0 version yesterday on March 14th 2025.

With that they introduced a new parameter to limit which hosts can show the dashboard. I haven't yet read about why this was introduced, but it's fixed quickly.

As Watchtower does all the maintenance work for the containers running on my installation of Portainer I was already greeted with the following error message:

In the logfile for the container running homepage I saw the following error:

[2025-03-15T16:54:13.497Z] error: Host validation failed for: portainer.lan:3000. Hint: Set the HOMEPAGE_ALLOWED_HOSTS environment variable to allow requests from this host / port.

As I don't use a separate IP or hostname for the dashboard and just forward the port 3000/tcp towards the homepage-container I access it using the hostname of my Portainer host. Therefore this message makes sense.

Luckily the documentation for the newly required environment variable is already on their homepage: https://gethomepage.dev/installation/#homepage_allowed_hosts

Armed with this knowledge we can change the stack file (Portainers term for a docker-compose file - not to be confused with the docker swarm command docker stack) and introduce the HOMEPAGE_ALLOWED_HOSTS parameter. I added the IP-address too, in case the DNS servers in my LAN should stop working.

services:
  homepage:
    image: ghcr.io/gethomepage/homepage:latest
    container_name: homepage
    ports:
      - 3000:3000
    environment:
      HOMEPAGE_ALLOWED_HOSTS: 192.168.178.21:3000,portainer.lan:3000
    volumes:
      - /opt/docker/homepage.dev/config:/app/config # Make sure your local config directory exists
      - /var/run/docker.sock:/var/run/docker.sock # (optional) For docker integrations
      - /opt/docker/homepage.dev/icons:/app/public/icons # icons, reference as /icons/picture.png in YAML

After that just hit the "Update the stack" button and it's working again.

Photo by Jonathan Cooper: https://www.pexels.com/photo/animals-head-on-exhibition-9660890/

For years Mastodon had one prominent missing feature. You wouldn't see all replies to a toot (that's the term for "Tweet" in the Mastodon world) because of the federated nature of Mastodon. There were exceptions, but in general it means that you have to open the post on the Mastodon instance it originated from. Only then you would be able to read all comments.

Naturally this feature was sought after for years. One GitHub issue was opened in 2018 (Mastodon #9409: Fetch whole conversation threads).

Now it seems the biggest technical part has been resolved! In Mastodon #32615: Add Fetch All Replies Part 1: Backend user sneaker-the-rat did all the basic work to incorporate that functionality into the backend of Mastodon. And while he mentions that there is still work to do the Mastodon community so far seems happy that this issue is finally getting fixed providing a much smoother experience for everyone.