Feuerfest

Just the private blog of a Linux sysadmin

Working in IT is sometimes like this...

Task: Upgrade a custom application on a customer system

Technical details: This involves executing some script beforehand, than a Puppet Run and some menial afterwork/clean up tasks.

  1. Prepare everything. Shutdown system. Take a Snapshot.
  2. Executing the script: Error! I can't find this dependency!
    • Oh, yeah. That package requiring the dependency isn't needed here. Commenting out that package from the list.
  3. Executing script again. It finishes properly. Fine.
  4. Running Puppet agent.
    • Error after 45seconds: Left over process found. Aborting.
    • Checking Relases Notes: Ah, known bug. ps -ef |process and kill pid is the workaround ok.
    • ps'ing and kill'ing.
  5. Run puppet agent -t again.
  6. It runs for 30min, then aborts. A partition has not enough free space. Narf.
    • Searching for stuff to delete/compress/move.
    • Start Puppet again to verify it works. It does.
  7. Revert system to snapshot as this is policy.
  8. Doing 1-6 again.
  9. Puppet agent run.
  10. It runs for 2.5 hours. Then breaks with Execution expired and some Ruby stacktrace.
    • Checking Release Notes again. Nothing. Hmm ok.
    • Reading code to understand the problem. No real insight gained. It SHOULD work.
  11. Reverting to snapshot. Now taking a 2nd snapshot right before the Puppet run.
  12. 3rd Puppet agent run. Same error after 2.5 hours.
    • Oh come on..
  13. Reverting to snapshot.
  14. Installing the RPM which installation triggers the stacktrace manually. It works. Package install takes 45min.
  15. Reverting to snapshot.
  16. Just for the sake of trying I start a 4th Puppet run. Expecting no change.
  17. After 55min of Puppet running I send a status mail to project lead, some other involved people, application developer, etc. that I couldn't update the system in time today and will need help and an additional 1-2 hours tomorrow.
  18. 5min after sending the mail: Hey, it's me! Puppet! I finished. No errors!
  19. I send out a follow-up mail, stating that the biggest part of the update is done. But due to the missed time window I will still need 1-2 hours tomorrow for the afterwork/clean up tasks.

And I still don't where or what the error was! Due to Execution expired I suspect that somewhere deep in the Ruby code something timed out. Maybe something which isn't documented nor written in any logfile. Something which had a maintenance window at exactly the same hours as we had.

Hopefully the developer knows more.

Sometimes IT sucks.😂

Comments

"The IT Crowd" Android easter egg

Holy shit! Didn't know of that easter egg.

Have you ever entered the "new emergency number" from The IT Crowd into your Android caller app?

I advise you to do so now. No, you don't need to press the call button. You don't need to actually establish/make a call. Just enter the complete number, that is:

0118 999 881 999 119 725 3

Cooooooool! 😆 And it even worked with my LineageOS ROM. 😆
(Doesn't work on GrapheneOS phones from what I heard.)

Comments

Opinion: Please don't drag me into your private feuds

That one was a first for me. Someone on Mastodon mentioned me. And at first I was confused. I couldn't remember ever having contact with that person.

Naturally I asked why I was being mentioned. The other toots in that thread didn't make sense to me. Clearly I lacked context.

The person replied with: "Block recommendation for that person/single-user instance."

I groaned immediately. I don't like all this preemptive blocking. Yes, there are scenarios where it can be useful as there are some really nasty & vile Mastodon instances out there. But why should I block that person? Why that person in particular?

I did not know the person who sent me the block recommendation, nor did I know the name of the "blockworthy" person or Mastodon instance. In addition, the admin of the Mastodon instance I use was also mentioned. My best guess was that this person was sending messages to all sorts of administrators of random Mastodon instances. Just the internet being ... the internet.

I replied: "But I have nothing to do with any instance. I'm not an admin anywhere. And why should I block preventively? I actively block when something annoys me. Everything else is a fight against windmills and not worth my time."

And then the following conversation developed:

Internet Person: "Up to you. That's why I also mentioned your instance admin."

Me: "Yes, but why? That took 5 minutes of my time completely unnecessarily. Without you, I would never have found out about the instance. I.. Arg.. Yes, ok. Good. Let's leave it at that. I'm too old and I've been on the internet too long..."

Internet Person: "You were in contact with him, that's why. And whether it costs you 5 minutes of your time is really irrelevant if he simultaneously insults hundreds of others on your instance in a racist, trans- or misogynistic way."

Me: "Ok, THAT is information that I would have liked to have had straight away. Do you still have the link or can you tell me when that was?"

Internet Person: "You were talking about something else, but still. I'm trying to find out from his timeline which major instances have not yet banned him." (Screenshot was attached but isn't shown here.)

It was literally one, ONE reply that person made to a toot of mine about a technical topic. I added several hashtags to that toot and that must have been how that person found my toot. That person doesn't follow me, nor am I. Also I don't know that person. And the conversation happened in February 2025. I had already completely forgotten about it.

I had a brief look at this one-person Mastodon instance and yes, while his profile was private the description that person gave about himself said enough. Also the list of blocked instances he blocked and the reasons he gave for it.

But honestly? I don't fucking care. Hence my answer.

Me: "Thanks for the screenshot! But yes, I'll leave it at that. Since his profile is private, I'll probably never get anything flushed into my feed by accident anyway."

The Internet Person didn't seem happy about this and wished me a good day.

So in the end? What had I gained? Nothing. Absolutely nothing. Someone literally made me waste about 20 minutes of my time just because that person thought it'd be better to block this random internet stranger and drag me into their little skirmish. Sorry, I'm too old. I don't have time for this.

Yes, I know that I am somewhat privileged as a white male, despite falling into several categories that the "blockworthy" person seems to truly despise.

But why do I have to carry his weight? Or that of the internet person? I am dead certain I would never have interacted with, let alone read from, that person again. Ever.

One person replied in a perfectly normal way under one of my Toots. Didn't attack anyone, nothing. Yes, this person may still be an asshole (and to be fair, this guy probably is), but why do I have to waste time on it?

Please don't drag me into your private feuds. Thank you.

Comments

GitHub is apparently a search engine

Colleague 1 (ironically): "GitHub what's that?"
Colleague 2: "Search engine for OpenAI API-Keys, AWS-Credentials and open S3-Buckets. Also private keys and passwords."

Comments