Feuerfest

Just the private blog of a Linux sysadmin

Snap packages and SSL-certificates.. A neverending story

Photo by Magda Ehlers: https://www.pexels.com/photo/a-broken-bicycle-leaning-on-the-wall-5342343/

Alternative title: Why I hate developers who reinvent the wheel - but forget the spokes.

This is more of a rant and link-dump article. I didn't research everything in detail as I generally avoid Snaps. So far they've caused me more troubles than benefits.. But I wanted a place to keep all my arguments and links to certain documentation sites in one place.. So here we go.

Snap is, compared to packaging formats like RPM and APT, relatively new. Maybe this explains why it still has some teething problems. The most annoying one for me is: Snap packages don't use your custom SSL-Certificates, stored in /usr/share/ca-certificates. Which is the default place to store your companies Root-CA certificates. Every browser respects this. There is tooling in every Linux distribution to take care of stored certificates and adding them to the truststore. There is tooling to automatically add these certificates to any Java (or other language) truststore that might reside on your system.

But no, that would be too easy mirroring that behaviour in Snap, right? After all.. Why just reinvent the wheel, when you can have more fun by forgetting the spokes.
Yes, I am aware that centreless/hubless wheels do exist. ;-)

The root cause is often: Snap Confinement (or to be precise: The strict confinement mode). Which means a snap is separated from the system and only has access to directories which are configured at build time of the snap, as stated in Interface Management documentation (see also: https://snapcraft.io/docs/home-interface and https://snapcraft.io/docs/supported-interfaces). And as desirable and good this is. Reality teaches us that for every application you always need a way to modify, to configure it. At least certain aspects. With snap.. Not so much?

In the latest Ubuntu releases Firefox and Chromium have been migrated exclusively to snap. You cannot install a Firefox via apt and have a normal install. You will get a snap package. Which means: Goodbye SSL Client cert authentication, goodbye internal company SSL certificates. Bug 1901586: [snap] CA Certificates from /usr/local/share/ca-certificates are not used has all the details.

Yes, clever people might add: But you can do a bind mount and mount that directory under /home where nearly every Snap application has access. But why do I need to do this? Why does Snap impose that burden on me, the user? And this doesn't fix the SSL issue..

Oh, I should copy them to /etc/ssl/certs? And what about the other troubles this might cause? .. Hello? Nothing? Oh, okay..

And it keeps getting better: If you install the Videoplayer VLC as a snap package and your files are not located under /home.. You can't access them. Snap developers will happily advise you to move your files, change the mountpoint or do a mount --bind under /home. Which can be seen here: Bug 1643706: snap apps need to be able to browse outside of user $HOME dir. for Desktop installs

And this is the point I want to make. It's OK to design a new packaging format for applications. It's okay to add security mechanism like snap confinement. But designing them in way so that each and every user is forced to manage/store/organize files in the way snap dictates? Nope, sorry.

Someone on the Internet wrote that Snap is a great idea for things like phones, PCs in public space (libraries, schools, etc.) or other confined environments where the user doesn't and shouldn't be allowed to configure many aspects. Where a system is designed and offered for only some specific purposes which are more or less static and change seldom.

And I agree. For me the root cause with all my problems regarding Snap is: It isn't the only daemon on my system. It HAS to integrate into the existing system and processes, like update-ca-certificates or even where in the filesystem I store my files. This was all there before Snap existed and now applications which always worked won't because someone thought it is better that way.. No, sorry. Like I said before: It HAS to integrate into the existing system. If it doesn't.. It might still have it use-cases. I'm not arguing against that. But then please let me have a choice! But breaking existing workflows, file structures, etc. that is not acceptable for me. And as the Internet shows, also not for many other users.

The sad part is that the decision to make Firefox only available as a Snap was done by Mozilla & Canoncial. Therefore you can't download the .deb from the Mozilla Webpage or the like. (Luckily there is still a PPA which build Firefox as .deb package offered by volunteers. This means it can potentially vanish if there is no one left doing it. But that's more or less the risk with any piece of software/technology.)

The announcement is on their Discourse: https://discourse.ubuntu.com/t/feature-freeze-exception-seeding-the-official-firefox-snap-in-ubuntu-desktop/24210/1

Oh and Snaps tend to auto-upgrade, even without unattended-upgrades configured. Which can be a problem if you require a specific version. Luckily snap refresh --hold can hold updates for all Snap packages. Or, if you just want it for some specific package/time use snap refresh --hold=72h vlc. This post has more information: Hold your horses, I mean snaps! New feature lets you stop snap updates, for as long as you need (snapcraft.io)

Security considerations

Snap was invented by Canoncial. The Snap-Store is run by Canoncial. Snap packages are slowly replacing more and more .deb-packages in Ubuntu, when you type apt-get install packagename you will automatically get a snap package if one exists.

Or if you try to execute a command which isn't present on your system. Then the command-not-found helper from Ubuntu will recommend you the associated .deb or Snap package.

The problem? Aquasec discovered that many traditional programs are not available as a Snap, but Canoncial still allows you to register Snaps with the exact same name, despite providing an entirely different program. (aquasec.com)

Additionally you can specify aliases or register Snaps for the literal filenames. The example shows how, when you try to execute the command tarquingui, command-not-found recommends to install the tarquin Snap. Which provides the tarquingui program. But they were able to additionally register a Snap with the name tarquingui. What happens? command-not-found now recommends both Snaps.

My personal opinion is that it is a dangerous and dumb oversight to not reserve every APT-package on the Snap store. Preventing the hijacking of well established software packages by potentially malicious third parties. After all it was Canoncial who introduced Snap and it's Canoncial who solely operates the Snap-Store...

How to install Firefox as .deb package (Ubuntu 23.04)

Update: I recently learned of a bug in unattended-upgrades which ignores apt-pinnings if the origin is not listed in the allowed-origins: #2033646 unattended-upgrade ignores apt-pinning to not-allowed origins

To prevent this bug, use the following workaround:

root@host:~# echo 'Unattended-Upgrade::Allowed-Origins:: "LP-PPA-mozillateam:${distro_codename}";' | sudo tee /etc/apt/apt.conf.d/51unattended-upgrades-firefox
  1. Remove the Firefox snap (omit if not present)
    • snap remove firefox
    • If dpkg -l firefox still shows it as installed, execute:
      • apt-get remove --purge firefox
  2. Add the Mozilla PPA
    • add-apt-repository ppa:mozillateam/ppa
      • If the command add-apt-repository is missing, install the package software-properties-common
  3. Pin the Mozilla PPA-Repository with higher priority so packages are installed out of this PPA instead of other repositories where they might be available too
    • echo -e "Package: *\nPin: release o=LP-PPA-mozillateam\nPin-Priority: 1001" | sudo tee /etc/apt/preferences.d/mozilla-firefox
  4. Update repository information
    • apt-get update
  5. Now check that your APT-Package Policies are correct
    • apt-cache policy firefox
      • If set up correctly Candidate: will list the package name which is not a Snap. See the example below.
      • root@ubuntu:/# apt-cache policy firefox
        firefox:
          Installed: (none)
          Candidate: 1:1snap1-0ubuntu3
          Version table:
             1:1snap1-0ubuntu3 500
                500 http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages
             120.0.1+build1-0ubuntu0.23.04.1~mt1 500
                500 https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu lunar/main amd64 Packages
        
        root@ubuntu:/# echo -e "Package: *\nPin: release o=LP-PPA-mozillateam\nPin-Priority: 1001" | tee /etc/apt/preferences.d/mozilla-firefox
        Package: *
        Pin: release o=LP-PPA-mozillateam
        Pin-Priority: 1001
        
        root@ubuntu:/# apt-cache policy firefox
        firefox:
          Installed: (none)
          Candidate: 120.0.1+build1-0ubuntu0.23.04.1~mt1
          Version table:
             1:1snap1-0ubuntu3 500
                500 http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages
             120.0.1+build1-0ubuntu0.23.04.1~mt1 1001
               1001 https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu lunar/main amd64 Packages
  6. Install Firefox. You should see that https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu is being used to download Firefox. If not search for errors.
      • apt-get install firefox
    Comments

    Howto use FreeOTP for Two-Factor-Authentication (2FA) on LinkedIn

    Photo by Pixabay: https://www.pexels.com/photo/black-android-smartphone-on-top-of-white-book-39584/

    Too Long;Didn't Read (TL;DR):

    You can omit the steps listed below. If your 2FA/OTP App allows to specify the secret key, type, algorithm and interval use the following settings for LinkedIn:

    Type: TOTP
    Number of digits: 6
    Algorithm: SHA1
    Interval: 30 seconds

    Original article

    I try to enable Two-Factor-Authentication, or 2FA in short, on any of my accounts that supports it. But: I dislike it, when the 2FA-Codes are sent via Mail or SMS. This is just too insecure as both can be intercepted. And personally I would go so far to say "SMS & Mail isn't a valid & secure second factor." As there are too many reports how scammers and phishers intercept SMS or mails. Yet many companies still default to this. LinkedIn too.

    Therefore I wanted to switch to my Authenticator App of choice: FreeOTP - https://freeotp.github.io/
    The source code is on GitHub: https://github.com/freeotp

    It is completely OpenSource (sponsored by RedHat) and even available in the alternative Android App-Store F-Droid, which only offers Apps which can be build completely from source.

    As naive as I am sometimes I thought it's just the following steps:

    1. Enable 2FA in my LinkedIn profile
    2. Provide password to authenticate
    3. Scan the QR-Code in FreeOTP
    4. Enter the generated code to verify it works
    5. Generate & Save the backup keys in my password manager

    But not so on LinkedIn. They don't display a QR-Code. Well.. To be precise. They did. Before Microsoft bought LinkedIn. After that this changed. Nowadays they only display you the so-called secret key (encoded in Base32) and that's it.
    Then LinkedIn tells you to install the Microsoft Authenticator App, while mentioning, that you can, of course, use any other Authenticator App.

    The problem? The described workflow on what to do with that key only works in the Microsoft Authenticator App.
    Side-Note: Someone told me Google Authenticator should be able to use that code too. But I can't verify this.

    LinkedIn gives you absolutely no additional technical information.

    • No otpauth:// URL
    • No information if TOTP or HOTP must be used
      • Well, to be fair, we can safely assume it's TOTP.
    • Which algorithm must be used?
    • What is the lifetime (interval) of the generated codes?

    Nothing. But this is what I need with FreeOTP. I tried a few combinations, but had no luck.

    So I resorted to the Linux command-line.

    1. Enable 2FA in your account until the secret key is displayed
    2. Install qrencode (or use one of the available Web-Generators for QR-Codes at your own risk)
    3. Build the following string: otpauth://totp/LinkedIn:MyAccount?secret=KEY-YOU-GOT-FROM-LINKEDIN
      • All in one line, no spaces at the end, no enter.
      • You can change "MyAccount" to something more meaningful like your mail address
      • Example: otpauth://totp/LinkedIn:JohnDoe@company.tld?secret=U4NGHXFW6C3CLHWLQEVCBDLM5FQMAQ7E
    4. Paste that string into a textfile.
      • Again, no enter or spaces at the end
    5. Execute: qrencode -r /path/to/file.txt -t png -o /path/to/image.png
      • This will generate a PNG-Image at the location specified by the -o parameter
      • -r is the input file containing the string
    6. Display the QR-Code and scan it with FreeOTP
    7. Verify the code works
    8. Generate your backup keys and save them in your password manager
    9. Profit!

    Some documentation regarding the otpauth:// URL, it's syntax and the parameters you can use is available in the old Google Authenticator repository on GitHub: https://github.com/google/google-authenticator/wiki/Key-Uri-Format
    (Google Authenticator once was OpenSource too, but sadly isn't any more.)

    And while at it, I created the corresponding GitHub Issue for the FreeOTP project #360 [Feature-Request] Allow adding of entries by just specifying label & secret to properly take care of this nuisance. ;-)

    Lessons learned

    FreeOTP assumes the algorithm of SHA1 and an interval of 30 when these parameters are not part of the otpauth-URL. Choosing these works out of the box and I can omit the QR-Code step this way.

    Comments

    Why I don't accept connect/friendship requests from recruiters

    Photo by Andrea Piacquadio: https://www.pexels.com/photo/cheerful-young-woman-screaming-into-megaphone-3761509/

    When you work in IT you are in the privileged situation that people are actively offering you positions. If I wanted, I wouldn't have needed to search for a single one of my jobs. I got plenty of offerings on Xing or LinkedIn, via email or sometimes even through Twitter direct messages or Google Hangouts.

    So, as the question just recently arised from a recruiter on Xing: "Why didn't I accept the request to become connected?"
    Well, short answer: My starting page feed and too much clutter.

    In fact, I did tend to accept those requests years ago. Quickly, this had a rather unpleasant side-effect: My feed was full of job advertisements for various positions in far too many industries. Jobs which were absolutely not relevant for me. For which I did have no skills, no training, no interest, no passion. And.. 99,9% of the time I'm not searching for a job. So why should I be forced to read job ad after job ad and - sorry for the wording - waste my time with it? Especially when it is not a one-time occurrence but a constant stream of non-interesting content.

    Additionally, because of all that clutter, I sometimes didn't notice crucial personal updates from old colleagues & friends. As sadly nowadays it's the standard to just have one single feed where all postings show up. Not sorted into categories or whatever. Therefore I am more or less forced to read every article (and advertisement...) even if I'm not interested in it. Feel free to read my post The problem with social networks - and why I still miss Google+ if you want to know more.

    Thais is the sole reason why I stopped doing that.
    It is really nothing personal. It's just that 99,9% of the time your content is irrelevant to me - as I'm simply not on the lookout for a new challenge. And on top of that: In the short time frames when it is relevant to me, 99% of the content is - again - irrelevant to me - because the jobs don't fit what I'm searching for.

    I like my feed/stream to be about stuff I'm interesting in. I don't like it when I constantly have to read stuff which is neither interesting nor relevant for me.

    If, for example, LinkedIn changes the way their feed works, then my position could change. But until things stay as they are I sadly have to be a bit more rigid in who I accept as a contact.

    Comments

    What does "rc" and ".d" stand for in file-/directory names?

    Photo by Tima Miroshnichenko: https://www.pexels.com/photo/close-up-view-of-system-hacking-in-a-monitor-5380664/

    This was a question which came up recently and while I could have given the answer based on my experience where these two strings commonly occur, I never really researched the initial origin.

    Turns out rc has quite a heritage. Quoting from the Indiana Universities Knowledge Base:

    runcom (as in .cshrc or /etc/rc)

    The rc command derives from the runcom facility from the MIT CTSS system, ca. 1965. From Brian Kernighan and Dennis Ritchie, as told to Vicki Brown:

    "There was a facility that would execute a bunch of commands stored in a file; it was called runcom for "run commands", and the file began to be called "a runcom". rc in Unix is a fossil from that usage."

    Note: The name of the shell from the Plan 9 operating system is also rc.

    And then I learned about the Plan 9 OS of which I've never heard before: https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs

    For .d there is not a real distinctive answer. It seems the common consensus is that it originated from /etc/rc?.d as to have a directory for runcoms and to distinguish it from a normal daemon configuration file - as these are typically stored under /etc. And/or to prevent naming-problems, as files & folders can't share the same name under Unix/Linux the .d as indicator for a directory was added.

    And from there this mechanism was used for other daemons as well. What seems to have started as /etc/rc1.d, /etc/rc2.d, etc. as a way to separate runcoms to be executed on different runlevels, became commonly used today for the configuration of daemons/services.

    Today a something.d directory holds configurations files for the something daemon. The advantage is that you are able to split your configuration in different files. Making it, for example, easier to deploy only certain files to certain hosts via your configuration management tool of choice (Puppet, Ansible, Chef, etc.). Additionally it's easier to spot configuration errors in a relatively small than in one big "contains it all"-file.

    Comments

    Changing the default link target in Bludits TinyMCE

    Photo by Monstera: https://www.pexels.com/photo/woman-holding-book-with-blank-pages-6373293/

    When creating links in TinyMCE the link-plugin is configured to open links in the current window. As this is the default setting. However I don't like this and changed it every time. And when you change something every single time, you should just make it your new default.

    Luckily TinyMCE has good documentation about every plugin. So we learn about the default_link_target parameter we can utilize to achieve exactly this.

    We open the bludit-folder/bl-plugins/tinymce/plugin.php file and search for tinymce.init. Then we add the default_link_target: '_blank' parameter at the end of the list. Don't forget to add a semicolon behind the formerly last parameter.

    In the end it looks like this:

            tinymce.init({
                    selector: "#jseditor",
                    auto_focus: "jseditor",
                    element_format : "html",
                    entity_encoding : "raw",
                    skin: "oxide",
                    schema: "html5",
                    statusbar: false,
                    menubar:false,
                    branding: false,
                    browser_spellcheck: true,
                    pagebreak_separator: PAGE_BREAK,
                    paste_as_text: true,
                    remove_script_host: false,
                    convert_urls: true,
                    relative_urls: false,
                    valid_elements: "*[*]",
                    cache_suffix: "?version=$version",
                    $document_base_url
                    plugins: ["$plugins"],
                    toolbar1: "$toolbar1",
                    toolbar2: "$toolbar2",
                    language: "$lang",
                    content_css: "$content_css",
                    codesample_languages: [$codesampleConfig],
                    default_link_target: '_blank'
            });
    

    And now generated links will per-default open in a new window.

    Comments

    Bludit and syntax highlighting

    Photo by Pixabay: https://www.pexels.com/photo/computer-c-code-276452/

    Bludit offers syntax highlighting via TinyMCE and it's codesample plugin, which you have to enable in the TinyMCE plugin settings first. But it offers only limited support. No Bash, no Puppet, no Perl and so on. Also I wanted line numbering when in code blocks for easier orientation. This is also not included.

    The Prism.js library however offers support for many more languages and also some nice Plugins. For example for line-numbering.

    To make everything work, the following tasks are necessary:

    1. Download & install the Bludit-Prism Plugin from here: https://plugins.bludit.com/plugin/prism
    2. Replace prism.js and prism.css files included in this Plugin with newer ones from https://prismjs.com/
      • As the one included in the Bludit Prism-Plugin only has support for: HTML/XML, JavaScript, CSS, PHP, Ruby, Python, Java, C, C#, C++
    3. Add our new languages to the codesamples config in the TinyMCE plugin config. This will add them in the dropdown-menu when inserting a code-sample block
    4. Edit the plugin.min.js file from the TinyMCE codesample plugin to automatically included the line-numbers plugin into the HTML pre-element.

    For a general overview, and a little bit of background why the Prism plugin was created, see this link: https://forum.bludit.org/viewtopic.php?t=1818

    As I'm somewhat new to Bludit I can't assess if this is the best approach. But for me it works. ;-)

    1. Download & install the Bludit-Prism Plugin from here: https://plugins.bludit.com/plugin/prism
      • Then activate it on the Bludit Plugins page
    2. Download prism.js and prism.css from https://prismjs.com/
      • Include all languages and plugins you need
      • Note: You need to use the JS and CSS-file from the same generation! You can't mix features or versions.
    3. Go to: bludit-folder/bl-plugins/prism and make a backup copy of the original files
      • mv js/prism.js js/prism.js.backup
        mv css/prism.css css/prism.css.backup
        
    4. Download new prism.js and prism.css and put into the correct folders
    5. Go to: Plugins -> TinyMCE -> Codesample languages and add your new languages, in my case:
      • Bash sh|RegEx regex|Puppet puppet|Perl perl|Python python|HTML/XML markup|JavaScript javascript|CSS css|PHP php|Ruby ruby
      • Unrelated side info: This will be written in the file bludit-folder/bl-content/databases/plugins/tinymce/db.php. I was just curious how/where this change is persisted.
    6. Now you will have syntax highlighting. But only in the generated HTML files, not in the TinyMCE code-blocks!
    7. To automatically include the line-numbers plugin from Prism.js we need to include the "line-numbers" class into the pre-element. This is done in the following way:
      • Open: bludit-folder/bl-plugins/tinymce/tinymce/plugins/codesample/plugin.min.js
      • Search for <pre and in the class property add line-numbers. It should now look like this: t.insertContent('<pre id="__new" class="language-'+a+' line-numbers">'+r+"</pre>")
      • A little after that pre you will also find t.dom.setAttrib(e,"class","language-"+a), add the line-numbers class in there too, it should look like this: t.dom.setAttrib(e,"class","line-numbers language-"+a)
      • Again, thanks to this comment on GitHub: https://github.com/tinymce/tinymce/issues/2771#issuecomment-232910444
    8. Keep in mind: As Bludit is a static file generator, your previously generated pages won't get this automatically. Here you will have to edit the pre-element manually.

    If you are interested in some examples, have a look at this page: https://admin.brennt.net/syntax-highlighting-test with line-numbering and without, etc.

    Comments