Feuerfest

I needed a quick throwaway Ubuntu VM for some testing, so I reached for the classic combo: an Ubuntu cloud image, a local cloud-init NoCloud datasource served over HTTP, and a bare qemu-system-x86_64 invocation. Nothing fancy, I've done this a dozen times before - just that the last time was ages ago..

My first attempt looked like this:

root@vm:~# qemu-system-x86_64 -net nic \
    -net user -machine accel=kvm:tcg -m 512 \
    -nographic -hda noble-server-cloudimg-amd64.img \
    -smbios type=1,serial=ds='nocloud;s=http://127.0.0.1:8000/'

And of course, it didn't just work. Last time I did was ages ago and directly on the hardware of my rootserver, not my local Proxmox server. As it turned out, I had two separate problems in this command, and to make things even more interesting: Everything is executed inside a Debian 13 VM, which itself lives on top of Proxmox. A VM inside a VM. So every weirdness I hit had to be checked twice. Once for "Is this a QEMU problem?" and once for "Is this a nested-virtualization problem?".

Preface

I created the directory /root/temp and but the files there according to the cloud-init QEMU tutorial. Then I started a webserver via the http.server Python module.

Have a look at https://docs.cloud-init.io/en/latest/tutorial/qemu.html#define-the-configuration-data-files regarding the files and the content.

Problem 1: Stuck at "Booting from Hard Disk..."

The VM wouldn't get past the SeaBIOS boot message "Booting from Hard Disk...". It became stuck there, forever. Additionally I saw no request to the local webserver on port 8000/tcp.

Turns out this is entirely expected, once you know why. Ubuntu's released Noble Numbat (LTS) cloud image is built for UEFI boot. SeaBIOS, the legacy BIOS QEMU uses by default, has no idea what to do with the GPT-partitioned disk and just... stops. No error, no hint, just sitting at "Booting from Hard Disk...".

The fix is to hand the VM the proper UEFI firmware via the Open Virtual Machine Firmware (OVMF) package instead of relying on SeaBIOS:

apt-get install ovmf

If you find older posts, they mention the /usr/share/OVMF/OVMF_VARS.id file, but on Debian 13 there's a small surprise waiting here too, the package no longer ships the plain OVMF_CODE.fd or OVMF_VARS.fd files I was used to. Instead you get the 4M variants:

root@vm:~# ls /usr/share/OVMF/OVMF_VARS_4M.*
/usr/share/OVMF/OVMF_VARS_4M.fd  /usr/share/OVMF/OVMF_VARS_4M.ms.fd  /usr/share/OVMF/OVMF_VARS_4M.snakeoil.fd

The .ms.fd variant ships with Microsoft's Secure Boot certificates pre-enrolled, and .snakeoil.fd is pre-signed with a test certificate for Secure Boot development. Neither is what I wanted, so plain OVMF_VARS_4M.fd it is, paired with the matching OVMF_CODE_4M.fd. CODE and VARS need to come from the same "generation" (2M vs. 4M), mixing them doesn't end well.

You also don't want to point QEMU directly at the VARS file in /usr/share/OVMF/, copy it locally first, since QEMU will write to it (that's where your EFI variables actually live):

root@vm:~# cp /usr/share/OVMF/OVMF_VARS_4M.fd .

Problem 2: Wrong IP for the webserver

While I was at it, I remembered that the datasource URL is used inside the guest and therefore refers to the guest itself, not my Proxmox VM serving the cloud-init files. With QEMU's user-mode networking -net user, the host is reachable from inside the guest at the fixed address 10.0.2.2. And since I'm serving the metadata over HTTP rather than a local path, nocloud-net is the correct datasource identifier, not nocloud.

Putting it together:

root@vm:~# qemu-system-x86_64 \
  -machine q35,accel=kvm:tcg \
  -m 512 \
  -nographic \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.fd \
  -drive if=pflash,format=raw,file=./OVMF_VARS_4M.fd \
  -drive file=noble-server-cloudimg-amd64.img,if=virtio,format=qcow2 \
  -net nic -net user \
  -smbios type=1,serial=ds='nocloud-net;s=http://10.0.2.2:8000/'

That got me past the boot prompt. Progress.

Problem 3: It boots, but agonizingly slowly

The VM eventually made it into the kernel, but everything crawled. The console was full of entries like this:

[  116.771971] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 32 times, consider switching to WQ_UNBOUND
Starting systemd-udevd version 255.4-1ubuntu8.16
[  148.409180] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 64 times, consider switching to WQ_UNBOUND
[  226.621365] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 128 times, consider switching to WQ_UNBOUND

These particular messages are a symptom, not the disease. What's actually going on: accel=kvm:tcg tells QEMU "Use KVM if you can, otherwise silently fall back to TCG (software emulation)". If, for whatever reason, KVM isn't available inside the nested VM, QEMU just quietly emulates the entire CPU in software which is orders of magnitude slower and explains why a workqueue would get starved for 10ms stretches and longer.

Since this whole setup is a VM inside a VM, the first suspect is always nested virtualization. So, checks on the Proxmox-hosted Debian VM:

root@vm:~# ls -la /dev/kvm
ls: cannot access '/dev/kvm': No such file or directory
root@vm:~# lscpu | grep -i virtualization
Virtualization type:                     full

As the /dev/kvm device isn't present, we now have confirmed that QEMU falls back to software emulation for the CPU.

The solution? Changing the CPU for the VM in Proxmox from x86-64-v2-AES (or whatever is configured) to host. This enables the VM to directly use the CPU of my Proxmox host.

This can be done comfortably in the WebUI of Proxmox. Just power down the VM.

Before:

And after:

After a reboot we have a /dev/kvm device and full support for Intel VT-x virtualisation. This confirms Proxmox is passing VT-x through cleanly to the nested VM. So KVM should be usable in principle, meaning if the inner QEMU VM is still falling back to TCG, it's not because nested virt is broken at the Proxmox level.

root@vm:~# ls -la /dev/kvm
crw-rw---- 1 root kvm 10, 232 Jul 27 00:48 /dev/kvm
root@vm:~# lscpu | grep -i virtualization
Virtualization:                          VT-x
Virtualization type:                     full

To actually find out, instead of trusting the silent kvm:tcg fallback, it's better to force KVM explicitly and let it fail loudly if something's wrong:

root@vm:~# qemu-system-x86_64 \
  -machine q35,accel=kvm \
  -cpu host \
  -m 512 \
  -nographic \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.fd \
  -drive if=pflash,format=raw,file=./OVMF_VARS_4M.fd \
  -drive file=noble-server-cloudimg-amd64.img,if=virtio,format=qcow2 \
  -net nic -net user \
  -smbios type=1,serial=ds='nocloud-net;s=http://10.0.2.2:8000/'

Removing the :tcg fallback means any KVM problem (permissions, missing nested virt on the Proxmox host CPU type, whatever) surfaces immediately as a hard error instead of a mysteriously slow VM. The parameter -cpu host on top makes sure the guest actually sees the hosts CPU features rather than a generic, lowest-common-denominator CPU model.

And the command worked, however, as we are still booting a VM inside a VM expect it to take some time. 😅

My learnings

For anyone hitting the same wall on their own Proxmox nested-VM setup, the checklist is:

1. Is /dev/kvm present on the outer (Proxmox) VM at all?
2. Is nested virtualization actually enabled on the Proxmox host itself?

   • # For Intel-CPUs - prints Y if enabled/available
     root@proxmox:~# cat /sys/module/kvm_intel/parameters/nested
     Y
     # For AMD-CPUs - prints Y if enabled/available
     root@proxmox:~# cat /sys/module/kvm_amd/parameters/nested
     Y

3. Is the outer VM's CPU type set to host in the Proxmox config? So VMX/SVM flags get passed through
   • Check in the Hardware tab of the VM inside Proxmox's WebUI, or: 
   • Check with qm config VM-ID und verify the cpu line says host

     root@proxmox:~# qm config 100
     [...]
     cores: 1
     cpu: host
     [...]

4. Does the inner QEMU call use accel=kvm (not kvm:tcg) plus -cpu host, so a broken KVM path fails loudly instead of silently degrading to software emulation?
5. Is the /dev/kvm device present on the VM where you want to start the QEMU VM?
6. Is CPU virtualization supported?

Nothing here is exotic once you know to look for it, but "silent TCG fallback" combined with "one VM inside another" is a pretty effective way to make a simple performance problem look mysterious for a while.

Dieser Text war ursprünglich ein spontan geschriebener Beitrag auf meinem LinkedIn-Profil. Zudem wurde er in einem emotional sehr engagierten (lies: aufgeregten) Zustand verfasst. Für das Blog habe ich ihn etwas überarbeitet und vor allem die zahlreichen Tippfehler beseitigt.

Die digitale Souveränität ist derzeit in aller Munde. Als Nerd muss ich da schon etwas hochnäsig fragen: "Ach, endlich mal? Wieso nicht schon vor 20 Jahren? Spätestens 9/11 bzw. die daraus resultierenden Gesetze, spätestens aber die Snowden-Enthüllungen hätten ein Weckruf sein müssen."
Na ja, immerhin hat uns die Kanzlerin der Herzen dafür einen memwürdigen Spruch geschenkt. #Neuland

Aber zurück zum Thema. In letzter Zeit habe ich viel gehört. Viel Business-BlaBla und, pardon, viel Schwachsinn.

Entzückt war ich, als ein Kollege von einer Microsoft-Veranstaltung berichtete. Microsoft hat angekündigt, sein Geschäft in Europa nun separat zu führen. Wohl auch mit extra Firmen, die in Europa registriert sind, etc. Das Motto lautet: "Falls es hart auf hart kommt, könnten wir das Europageschäft komplett vom US-Mutterkonzern abspalten."

Klingt toll. Ich habe es aber nicht so richtig geglaubt. So etwas macht ein Tech-Gigant nicht einfach so. Und selbst wenn es rechtlich machbar wäre. Wer betreibt es dann? Aktuell fällt die IT-Branche ja durch Massenentlassungen auf. Es ist schwer vorstellbar, dass Microsoft dann mehrere hundert Leute in Deutschland anstellt, um die Sovereign Cloud und alles Rechtliche drumherum zu managen. Einfach nur, um für den Fall der Fälle die nötige Manpower zu haben. Denn Firmen im jeweiligen Land zu registrieren ist das eine, aber das nötige Know-how vor Ort zu haben, um die Sovereign Cloud dann auch betreiben zu können, das andere.

Microsoft war derweil mit seiner Microsoft Sovereign Cloud am Markt unterwegs und hat erzählt, wie toll alles ist. Ich hingegen habe nur ein handelsübliches Azure gesehen. Dafür jetzt mit blauem Schleifchen dran. 🥰

Und irgendwie gehen alle auf einmal davon aus: "Wenn wir unsere Daten da reintun, dann kommt der böse, fiese Trump nicht dran! Haha!"

Nein, sorry, das gehört eigentlich zusammen. Ja. In der Praxis sind Betrieb und Datenzugriff aber jeweils separat auf Souveränität zu überprüfen. Gerade bei Cloudanbietern.

Dass dem nicht so ist, hat Microsoft höchstpersönlich in Form von Anton Carniaux vor dem französischen Parlament bestätigt.

Auf die Frage, ob er bzw. Microsoft garantieren könne, dass die von Microsoft gehosteten Daten französischer Bürger niemals ohne die Zustimmung der französischen Behörden an ausländische Behörden weitergegeben würden, antwortete er: "Nein, das kann ich nicht garantieren."
Quelle: https://www.senat.fr/rap/r24-830-1/r24-830-11.pdf, Seite 23.

Aber ist es nicht genau das, was alle mit Souveränität erreichen wollen? Unabhängigkeit und Schutz des eigenen Technologiestacks und der Daten vor fremder – und politischer – Einflussnahme?

Was nützt mir also eine Microsoft Sovereign Cloud, wenn dort die exakt gleichen rechtlichen Bedingungen gelten wie bei der "Non-Sovereign Cloud"?

Verstehe ich da etwas nicht? Oder sehe ich das zu engstirnig/idealistisch?

Ich freue mich über Austausch.

Und bitte nicht vergessen: Selbst wenn Microsoft eine vollständig von den USA abgekoppelte "Sovereign EU Cloud" betreibt. Wo alle nötigen Systeme in der EU stehen. Und alle nötigen Mitarbeiter in der EU arbeiten. Dann kommt immer noch ein Großteil der Software aus den USA. Und was nützt eine souveräne Cloud, wenn man keine Sicherheitsupdates mehr erhält?

Insofern sollte man zuerst immer klären: "Wie unabhängig können wir überhaupt sein?" Und "Wie weit wollen wir unabhängig sein?" Von diesem Punkt aus sollten dann die Planungen starten.
Oder ... Einfach Business-BlaBla machen. Ist weniger anstrengend und weitaus kosteneffizienter.

EDIT: Golem hat just heute einen interessanten Artikel dazu veröffentlich wie man im eigenen Unternehmen vorgehen kann um digital Unabhängiger zu werden: Digitale Souveränität: Argumentationshilfe für resiliente IT-Entscheidungen

Photo by Paula: https://www.pexels.com/photo/grey-metal-lockers-is-open-170453/

Recently I got into a discussion about cloud password managers like LastPass, 1Password and the like. People argued that they were great because they offer flexibility, synchronisation, and enable users to automatically choose secure passwords. And while I concurred to these arguments, I still took the position of the opposite side.

I am strongly against any form of online/cloud/managed password managers where I (or an entity I trust) don't control everything in a transparent way. Why?

First let me make my, the what I call, "the conspiracy theory argument": It's the least important argument for me, but is suited just right for paving the way to my main arguments. Most companies are US based. National security letters are a fact. And companies like Lavabit and, presumably, the Open Source Drive/File encryption project Truecrypt had their fair share of experience with them. The US and especially the NSA shows that it has no sane moral limits on what type of data to access, accumulate, and analyse. Would they stop at companies offering password managers? I don't think so. Others take the stance: "Why should they send a NSL to LastPass, if they can access the data they are after directly through others means?" but enough on that point.

My main argument is the following: "If I can't trust them, why should I use them?" A cloud password manager is a blackbox. I put my credentials in, hit save, and that's it. What happens in the background? Is the data stored secure? Are the algorithms used still considered secure? Are there no unencrypted backup copies? How does their security concept look like? Are the servers patched regularly? Am I being informed to re-generate my passwords in case the password-generation algorithm had a flaw and was, for example, tied to the systems date and time (wired.com)? I can't know.

Just search for the name of your cloud password provider and add the lovely word "breach". They were already dozens of it for all online password managers out there. Despite people ironically choosing or recommending them as to be "more secure".

But more secure in comparison to what?

You don't have to use an online/cloud password manager. A locally installed password manager like KeePass works the same way.

"But it doesn't synchronize automatically with all my devices!" - Ah, so it's comfort you are after? Yeah, well that is the common trade-off you have to choose: Comfort, or security. However, you do know KeePass has a build in sync which allows to sync two KeePass database files? I use that and it works fine.

Or I just copy over the file from my Linux workstation onto my Smartphone when I know that I had added no new entry on my smartphone. There are enough tools to allow the accessing of Windows/NFS/whatever file shares from Smartphones. Even via SSH utilizing SCP.

Additionally online password managers with sync-features who do allow to host your own instance do exist! Take for example Bitwarden: https://bitwarden.com/help/install-on-premise-linux/
This would give you the security of hosting it yourself and automatic sync-feature so many people desire. If not made accessable to the internet, but online in your local network you can still sync your smartphone, without opening your crucial application to the biggest security risk in human history: The internet.

Sure, not everybody has the knowledge to selfhost such an application. That's a fair point which Jürgen Geuter aka tante pointed out years ago in his writing "Host your own is cynical". But that's exactly the reason why I make this blogpost about me and my viewpoint. I have the knowledge, and I constantly observe broken promises by the very companies who gave them. And yet I still use KeePass. As I like the simplicity.

That's why I'm on #TeamKeePassUltras. An OpenSource application available for all operating systems out there. A simple file, a key, a passphrase and that's it.