Feuerfest

Just the private blog of a Linux sysadmin

GitHub is apparently a search engine

Colleague 1 (ironically): "GitHub what's that?"
Colleague 2: "Search engine for OpenAI API-Keys, AWS-Credentials and open S3-Buckets. Also private keys and passwords."

Comments

Anagrams

Colleague: "Did you know? Shitposting is an anagram of Top Insights."

Comments

You want my opinion? Then follow my blog, my Fediverse account. And NOT my LinkedIn profile!

Today I read a post on LinkedIn (in german) who was criticising that too few IT people speak about their opinions, their views, their stances on various topics. Who was sad that so much Know-how and stories from real projects were left untold.

And I must admit that, at first, I didn't understand what he meant. I was all like:

"What is he talking about? I read a dozens or so post each day from (former) colleagues, people I met on various IT or CCC events or whom I just happen to follow which are highly political. Or taking a technical deep dive on some obscure technology (sometimes even from the 1970s) in which they have taken an interest. I don't get his point."

Only then I happen to realize: Did he solely refer to LinkedIn?

And that explained a lot. Regerettably I must admit: Instantly I had severel prejuicides against that person.

This lead to my writing the following two comments (which I translated into english):

    I have to disagree. The IT bubble, at least the one I'm in, is very political. But not on LinkedIn, because that's completely the wrong place for many people. I also had this experience myself when I was researching the status of various alternatives to Google apps for Android mobile phones and noticed how many Russian open source developers have suddenly gone completely silent in the last 3-4 years. Before, the GitHub activity bars were a bright, lush green. Regular posts in the XDA Developers forum, etc. and then, from one day to the next: Nothing. Silence. Completely.

    I wrote about it here on LinkedIn. Made a reference to the Ukraine war and Russian recruitment for their war of aggression. And yes, I also got carried away with a ‘4-letter-word Putin!’. (Note: I meant "Fuck Putin!" which I obviously couldn't write again on LinkedIn.)

    The result? Less than 10 minutes later, the post was set to invisible. I objected twice, but each time I was automatically rejected by the system.

    So sorry, but if you want to see the politically active IT scene: Get out. Get out of the silos of a Silicon Valley tech company. Into the private blogs, into the Fediverse.

    I also have a private blog. I'm generally not very reserved with my private opinions. And fortunately, I've received much more praise and recognition for this than criticism.

    But not everyone has this luxury. Many companies have very strict social media guidelines on how I am allowed to appear as an employee of a company. And there are judgements that say that simply mentioning your employer in your profile means that these rules apply and you have to comply with them, as you are no longer 100% private.

    And some people simply don't want to express their political opinions under their real name. Be it for protection or because they value privacy.

    In addition, many IT professionals have a very well-founded (and justified) aversion to so-called social networks. Their content is too soft. Too auto-moderated. Too undemocratic when it comes to appeals against content offences, TOS, etc.

    Plus the fact that many networks like LinkedIn are perceived as nothing more than business horseshow (which, unfortunately, it often is).

    Yes, LinkedIn is not a good platform if you are interested in IT. For that I recommend the various blogs, Fediverse accounts, some forums (if they are still alive), Newsgroups (if they are still alive) and IRC channels (if they are.. You get the gist..).

    Why?

    1. IT people are technical, they dislike business mumbo-jumbo and attention farming. They want to write a text about something they care about and discuss it with like-minded individuals. LinkedIn is the completely wrong platform for that.
      • Also, technical content on LinkedIn gets far fewer views, interactions, etc. Believe me, I can see the views here in on my blog and for my content on LinkedIn.
      • Well, yeah, because IT people usually don't like automated feeds that they can't configure. It's much more enjoyable to look in your RSS reader and see which one of the blogs you follow has posted something new.
      • Would you write huge technical texts on LinkedIn knowing that your audience is on a completely different platform? You wouldn't. That's why I blog here and just do a post on LinkedIn to redirect a few people from LinkedIn towards my blog.
    2. IT people are generally very prejudiced towards social networks operated by commercial companies. They are well aware of the downsides of Facebook/Meta, Twitter/X, Google Plus, Instagram, TikTok and yes - also LinkedIn. That these networks are about as far away from a democratic platform as you can get. That moderation is largely automated with no chance of "getting a human on the line". Why should we make ourselves dependent on that?
      • LinkedIn doesn't even provide a button to copy the link from a post into your clipboard! All actions are aimed at keeping people on LinkedIn. LinkedIn/Microsoft don't want you to drive people away from LinkedIn! And IT people seriously hate silos and gatekeeping.
    3. Many companies have social media guidelines that make it impossible to engage in meaningful conversations about sensitive topics.
    4. IT people value their privacy. If they have a LinkedIn account, it shows their real name and employer. And maybe they don't want a connection between the two.
    5. Some may see blogging as a hobby that they do in their spare time, so they stay away from LinkedIn.

    And there are maybe more points I forgot to mention..

    Comments

    Little helper scripts - Part 2: automation.sh / automation2.sh

    Part 1 of this series is here: Little helper scripts - Part 1: no-screenlock-during-meeting.ps1 or use the following tag-link: https://admin.brennt.net/tag/littlehelperscripts

    This script is no rocket science. Nothing spectacular. But the amount of hours it saved me in various projects is astonishing.

    The sad reality

    There are far too many companies (even IT-focused companies!) out there who have a very low level of automation. Virtual machines are created by hand - not by some script using an API. Configurations are not deployed via of some Configuration Management software like Puppet/OpenVox/Chef/Ansible or a Runbook automation software like Rundeck - no, they are handcrafted. Bespoke. System administration like it's 1753. With all the implications and drawbacks that brings.

    Containerisation? Yeah.. Well.. "A few docker containers here and there but nobody in the company really knows how that stuff works so we leave it alone" is a phrase I have heard more than a few times. Either directly, or reported from colleagues working in other companies.

    This means that I have to log on to systems manually and execute commands by hand. Something I can do and do regularly in my home lab. But to do it for dozens or even hundreds of systems? Yeah... No. Sorry, I've got better things to do. And as an external consultant, the client always keeps an eye on my performance metrics. After all, they are paying my employer a lot of money for my services. Sitting there all day and getting paid to copy and paste commands? It doesn't look good on my performance reporting spreadsheet and it doesn't meet my personal standards of what a consultant should be able to deliver.

    I'm just a guest

    What's more, because of my work as a consultant, I'm just an external contractor. I come in for a few months to solve a problem or help with a task, and then I move on to the next project at another company. That means I can't just do everything the way I want. I can't just go and install software on all the systems, even though I've been given root privileges. I can't just implement Ansible. I have to design my solutions so that they survive and continue to work when I'm gone. Sure, I can introduce dozens of new technologies and whole new technology stacks. I'm sure my employer would love to have the follow-on support contracts for those constructs. But going it alone will seriously damage the customer relationship. Especially with the IT people. After all, they'll be the ones stuck with new technology they don't understand and will have to spend time learning and familiarising themselves with. And I have been a sysadmin long enough to know what they will think of me if I start pulling such stunts.

    Of course I can suggest changes. I can push for standardisation and automation. But for most customers, that will make no difference. After all, there are reasons why a company has stopped keeping up with technology. And fixing this takes time and usually involves a complete change of the dominating mindset. Something I cannot achieve as a lone consultant.

    Scripting to the rescue!

    First I went with the cheap & easy solution of for server in hosta hostb hostc; do ssh user@$server "command --some-parameter bla"; done but I grew tired of writing it all completely anew for each task.

    Natively systems are often grouped into categories (webservers, etc.) or perform the same tasks (think of clusters). Hence commands must be executed on the same set of hosts again and again. One of my colleagues already compiled lists of hostnames group by tasks, roles and installed software. As some systems had the same software installed but were just configured to do different tasks with that software.

    Through these list I got an idea: Why not feed those into a for or do-while loop and be done?

    In the end I added some safety & DNS checks and named the script automation.sh. Later I added the capability to log the output on each host and named the script automation2.sh, which can be viewed below.

    Yes, it's just a glorified nesting of if-statements but the amount of time this script saved me is insane. And as it utilizes only basic Posix & Bash commands I've yet to find a system were it can't be executed.

    As always: Please check my GitHub for the most recent version as I won't update the script shown in this article.

    #!/bin/bash
    # vim: set tabstop=2 smarttab shiftwidth=2 softtabstop=2 expandtab foldmethod=syntax :
    #
    # Small script to automate custom shell command execution
    # Current version can be found here:
    # https://github.com/ChrLau/scripts/blob/master/automation2.sh
    
    # Bash strict mode
    #  read: http://redsymbol.net/articles/unofficial-bash-strict-mode/
    set -euo pipefail
    IFS=$'\n\t'
    
    # Set pipefail variable
    # As we use "ssh command | tee" and tee will always succeed our check for non-zero exit-codes doesn't work
    #
    # The exit status of a pipeline is the exit status of the last command in the pipeline,
    #  unless the pipefail option is enabled (see: The Set Builtin).
    # If pipefail is enabled, the pipeline's return status is the value of the last (rightmost)
    #  command to exit with a non-zero status, or zero if all commands exit successfully.
    
    VERSION="1.6"
    SCRIPT="$(basename "$0")"
    SSH="$(command -v ssh)"
    TEE="$(command -v tee)"
    # Colored output
    RED="\e[31m"
    GREEN="\e[32m"
    ENDCOLOR="\e[0m"
    
    # Test if ssh is present and executeable
    if [ ! -x "$SSH" ]; then
      echo "${RED}This script requires ssh to connect to the servers. Exiting.${ENDCOLOR}"
      exit 2;
    fi
    
    # Test if tee is present and executeable
    if [ ! -x "$TEE" ]; then
      echo "${RED}tee not found.${ENDCOLOR} ${GREEN}Script can still be used,${ENDCOLOR} ${RED}but option -w CAN NOT be used.${ENDCOLOR}"
    fi
    
    function HELP {
      echo "$SCRIPT $VERSION: Execute custom shell commands on lists of hosts"
      echo "Usage: $SCRIPT -l /path/to/host.list -c \"command\" [-u <user>] [-a <YES|NO>] [-r] [-s \"options\"] [-w \"/path/to/logfile.log\"]"
      echo ""
      echo "Parameters:"
      echo " -l   Path to the hostlist file, 1 host per line"
      echo " -c   The command to execute. Needs to be in double-quotes. Else getops interprets it as separate arguments"
      echo " -u   (Optional) The user used during SSH-Connection. (Default: \$USER)"
      echo " -a   (Optional) Abort when the ssh-command fails? Use YES or NO (Default: YES)"
      echo " -r   (Optional) When given command will be executed via 'sudo su -c'"
      echo " -s   (Optional) Any SSH parameters you want to specify Needs to be in double-quotes. (Default: empty)"
      echo "                 Example: -s \"-i /home/user/.ssh/id_user\""
      echo " -w   (Optional) Write STDERR and STDOUT to logfile (on the machine where $SCRIPT is executed)"
      echo ""
      echo "No arguments or -h will print this help."
      exit 0;
    }
    
    # Print help if no arguments are given
    if [ "$#" -eq 0 ]; then
      HELP
    fi
    
    # Parse arguments
    while getopts ":l:c:u:a:hrs:w:" OPTION; do
      case "$OPTION" in
        l)
          HOSTLIST="${OPTARG}"
          ;;
        c)
          COMMAND="${OPTARG}"
          ;;
        u)
          SSH_USER="${OPTARG}"
          ;;
        a)
          ABORT="${OPTARG}"
          ;;
        r)
          SUDO="YES"
          ;;
        s)
          SSH_PARAMS="${OPTARG}"
          ;;
        w)
          LOGFILE="${OPTARG}"
          ;;
        h)
          HELP
          ;;
        *)
          HELP
          ;;
    # Not needed as we use : as starting char in getopts string
    #    :)
    #      echo "Missing argument"
    #      ;;
    #    \?)
    #      echo "Invalid option"
    #      exit 1
    #      ;;
      esac
    done
    
    # Give usage message and print help if both arguments are empty
    if [ -z "$HOSTLIST" ] || [ -z "$COMMAND" ]; then
      echo "You need to specify -l and -c. Exiting."
      exit 1;
    fi
    
    # Check if username was provided, if not use $USER environment variable
    if [ -z "$SSH_USER" ]; then
      SSH_USER="$USER"
    fi
    
    # Check for YES or NO
    if [ -z "$ABORT" ]; then
      # If empty, set to YES (default)
      ABORT="YES"
    # Check if it's not NO or YES - we want to ensure a definite decision here
    elif [ "$ABORT" != "NO" ] && [ "$ABORT" != "YES" ]; then
      echo  "-a accepts either YES or NO (case-sensitive)"
      exit 1
    fi
    
    # If variable logfile is not empty
    if [ -n "$LOGFILE" ]; then
    
      # Check if logfile is not present
      if [ ! -e "$LOGFILE" ]; then
        # Check if creating it was unsuccessful
        if ! touch "$LOGFILE"; then
          echo "${RED}Could not create logfile at $LOGFILE. Aborting. Please check permissions.${ENDCOLOR}"
          exit 1
        fi
      # When logfile is present..
      else
        # Check if it's writeable and abort when not
        if [ ! -w "$LOGFILE" ]; then
          echo "${RED}$LOGFILE is NOT writeable. Aborting. Please check permissions.${ENDCOLOR}"
          exit 1
        fi
      fi
    fi
    
    # Execute command via sudo or not?
    if [ "$SUDO" = "YES" ]; then
      COMMANDPART="sudo su -c '${COMMAND}'"
    else
      COMMANDPART="${COMMAND}"
    fi
    
    # Check if hostlist is readable
    if [ -r "$HOSTLIST" ]; then
      # Check that hostlist is not 0 bytes
      if [ -s "$HOSTLIST" ]; then
      
        while IFS= read -r HOST
        do
    
          getent hosts "$HOST" &> /dev/null
          
          # getent returns exit code of 2 if a hostname isn't resolving
          # shellcheck disable=SC2181
          if [ "$?" -ne 0 ]; then
            echo -e "${RED}Host: $HOST is not resolving. Typo? Aborting.${ENDCOLOR}"
            exit 2
          fi
    
          # Log STDERR and STDOUT to $LOGFILE if specified
          if [ -n "$LOGFILE" ]; then
            echo -e "${GREEN}Connecting to $HOST ...${ENDCOLOR}" 2>&1 | tee -a "$LOGFILE"
            ssh -n -o ConnectTimeout=10 "${SSH_PARAMS}" "$SSH_USER"@"$HOST" "${COMMANDPART}" 2>&1 | tee -a "$LOGFILE"
    
            # Test if ssh-command was successful
            # shellcheck disable=SC2181
            if [ "$?" -ne 0 ]; then
              echo -n -e "${RED}Command was NOT successful on $HOST ... ${ENDCOLOR}" 2>&1 | tee -a "$LOGFILE"
    
              # Shall we proceed or not?
              if [ "$ABORT" = "YES" ]; then
                echo -n -e "${RED}Aborting.${ENDCOLOR}\n" 2>&1 | tee -a "$LOGFILE"
                exit 1
              else
                echo -n -e "${GREEN}Proceeding, as configured.${ENDCOLOR}\n" 2>&1 | tee -a "$LOGFILE"
              fi
            fi
    
          else
    
            echo -e "${GREEN}Connecting to $HOST ...${ENDCOLOR}"
            ssh -n -o ConnectTimeout=10 "${SSH_PARAMS}" "$SSH_USER"@"$HOST" "${COMMANDPART}"
    
            # Test if ssh-command was successful
            # shellcheck disable=SC2181
            if [ "$?" -ne 0 ]; then
              echo -n -e "${RED}Command was NOT successful on $HOST ... ${ENDCOLOR}"
    
              # Shall we proceed or not?
              if [ "$ABORT" = "YES" ]; then
                echo -n -e "${RED}Aborting.${ENDCOLOR}\n"
                exit 1
              else
                echo -n -e "${GREEN}Proceeding, as configured.${ENDCOLOR}\n"
              fi
            fi
    
          fi
    
        done < "$HOSTLIST"
    
      else
        echo -e "${RED}Hostlist \"$HOSTLIST\" is empty. Exiting.${ENDCOLOR}"
        exit 1
      fi
    
    else
      echo -e "${RED}Hostlist \"$HOSTLIST\" is not readable. Exiting.${ENDCOLOR}"
      exit 1
    fi
    
    Comments