Feuerfest

I needed a quick throwaway Ubuntu VM for some testing, so I reached for the classic combo: an Ubuntu cloud image, a local cloud-init NoCloud datasource served over HTTP, and a bare qemu-system-x86_64 invocation. Nothing fancy, I've done this a dozen times before - just that the last time was ages ago..

My first attempt looked like this:

root@vm:~# qemu-system-x86_64 -net nic -net user -machine accel=kvm:tcg -m 512 -nographic -hda noble-server-cloudimg-amd64.img -smbios type=1,serial=ds='nocloud;s=http://127.0.0.1:8000/'

And of course, it didn't just work. Last time I did was ages ago and directly on the hardware of my rootserver, not my local Proxmox server. As it turned out, I had two separate problems in this command, and to make things even more interesting: Everything is executed inside a Debian 13 VM, which itself lives on top of Proxmox. A VM inside a VM. So every weirdness I hit had to be checked twice. Once for "Is this a QEMU problem?" and once for "Is this a nested-virtualization problem?".

Preface

I created the directory /root/temp and but the files there according to the cloud-init QEMU tutorial. Then I started a webserver via the http.server Python module.

Have a look at https://docs.cloud-init.io/en/latest/tutorial/qemu.html#define-the-configuration-data-files regarding the files and the content.

Problem 1: Stuck at "Booting from Hard Disk..."

The VM wouldn't get past the SeaBIOS boot message "Booting from Hard Disk...". It became stuck there, forever. Additionally I saw no request to the local webserver on port 8000/tcp.

Turns out this is entirely expected, once you know why. Ubuntu's released Noble Numbat (LTS) cloud image is built for UEFI boot. SeaBIOS, the legacy BIOS QEMU uses by default, has no idea what to do with the GPT-partitioned disk and just... stops. No error, no hint, just sitting at "Booting from Hard Disk...".

The fix is to hand the VM the proper UEFI firmware via the Open Virtual Machine Firmware (OVMF) package instead of relying on SeaBIOS:

apt-get install ovmf

If you find older posts, they mention the /usr/share/OVMF/OVMF_VARS.id file, but on Debian 13 there's a small surprise waiting here too, the package no longer ships the plain OVMF_CODE.fd or OVMF_VARS.fd files I was used to. Instead you get the 4M variants:

root@vm:~# ls /usr/share/OVMF/OVMF_VARS_4M.*
/usr/share/OVMF/OVMF_VARS_4M.fd  /usr/share/OVMF/OVMF_VARS_4M.ms.fd  /usr/share/OVMF/OVMF_VARS_4M.snakeoil.fd

The .ms.fd variant ships with Microsoft's Secure Boot certificates pre-enrolled, and .snakeoil.fd is pre-signed with a test certificate for Secure Boot development. Neither is what I wanted, so plain OVMF_VARS_4M.fd it is, paired with the matching OVMF_CODE_4M.fd. CODE and VARS need to come from the same "generation" (2M vs. 4M), mixing them doesn't end well.

You also don't want to point QEMU directly at the VARS file in /usr/share/OVMF/, copy it locally first, since QEMU will write to it (that's where your EFI variables actually live):

root@vm:~# cp /usr/share/OVMF/OVMF_VARS_4M.fd ./OVMF_VARS_4M.fd

Problem 2: Wrong IP for the webserver

While I was at it, I remembered that the datasource URL is used inside the guest and therefore refers to the guest itself, not my Proxmox VM serving the cloud-init files. With QEMU's user-mode networking -net user, the host is reachable from inside the guest at the fixed address 10.0.2.2. And since I'm serving the metadata over HTTP rather than a local path, nocloud-net is the correct datasource identifier, not nocloud.

Putting it together:

root@vm:~# qemu-system-x86_64 \
  -machine q35,accel=kvm:tcg \
  -m 512 \
  -nographic \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.fd \
  -drive if=pflash,format=raw,file=./OVMF_VARS_4M.fd \
  -drive file=noble-server-cloudimg-amd64.img,if=virtio,format=qcow2 \
  -net nic -net user \
  -smbios type=1,serial=ds='nocloud-net;s=http://10.0.2.2:8000/'

That got me past the boot prompt. Progress.

Problem 2: It boots, but agonizingly slowly

The VM eventually made it into the kernel, but everything crawled. The console was full of entries like this:

[  116.771971] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 32 times, consider switching to WQ_UNBOUND
Starting systemd-udevd version 255.4-1ubuntu8.16
[  148.409180] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 64 times, consider switching to WQ_UNBOUND
[  226.621365] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 128 times, consider switching to WQ_UNBOUND

These particular messages are a symptom, not the disease. What's actually going on: accel=kvm:tcg tells QEMU "Use KVM if you can, otherwise silently fall back to TCG (software emulation)". If, for whatever reason, KVM isn't available inside the nested VM, QEMU just quietly emulates the entire CPU in software which is orders of magnitude slower and explains why a workqueue would get starved for 10ms stretches and longer.

Since this whole setup is a VM inside a VM, the first suspect is always nested virtualization. So, checks on the Proxmox-hosted Debian VM:

root@vm:~# ls -la /dev/kvm
ls: cannot access '/dev/kvm': No such file or directory
root@vm:~# lscpu | grep -i virtualization
Virtualization type:                     full

As the /dev/kvm device isn't present, we now have confirmed that QEMU falls back to software emulation for the CPU.

The solution? Changing the CPU for the VM in Proxmox from x86-64-v2-AES (or whatever is configured) to host. This enables the VM to directly use the CPU of my Proxmox host.

This can be done comfortably in the WebUI of Proxmox. Just power down the VM.

Before:

And after:

After a reboot we have a /dev/kvm device and full support for Intel VT-x virtualisation. This confirms Proxmox is passing VT-x through cleanly to the nested VM. So KVM should be usable in principle, meaning if the inner QEMU VM is still falling back to TCG, it's not because nested virt is broken at the Proxmox level.

root@vm:~# ls -la /dev/kvm
crw-rw---- 1 root kvm 10, 232 Jul 27 00:48 /dev/kvm
root@vm:~# lscpu | grep -i virtualization
Virtualization:                          VT-x
Virtualization type:                     full

To actually find out, instead of trusting the silent kvm:tcg fallback, it's better to force KVM explicitly and let it fail loudly if something's wrong:

root@vm:~# qemu-system-x86_64 \
  -machine q35,accel=kvm \
  -cpu host \
  -m 512 \
  -nographic \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.fd \
  -drive if=pflash,format=raw,file=./OVMF_VARS_4M.fd \
  -drive file=noble-server-cloudimg-amd64.img,if=virtio,format=qcow2 \
  -net nic -net user \
  -smbios type=1,serial=ds='nocloud-net;s=http://10.0.2.2:8000/'

Removing the :tcg fallback means any KVM problem (permissions, missing nested virt on the Proxmox host CPU type, whatever) surfaces immediately as a hard error instead of a mysteriously slow VM. The parameter -cpu host on top makes sure the guest actually sees the hosts CPU features rather than a generic, lowest-common-denominator CPU model.

And the command worked, however, as we are still booting a VM inside a VM expect it to take some time. 😅

My learnings

For anyone hitting the same wall on their own Proxmox nested-VM setup, the checklist is:

1. Is /dev/kvm present on the outer (Proxmox) VM at all?
2. Is nested virtualization actually enabled on the Proxmox host itself?

   • # For Intel-CPUs - prints Y if enabled/available
     root@proxmox:~# cat /sys/module/kvm_intel/parameters/nested
     Y
     # For AMD-CPUs - prints Y if enabled/available
     root@proxmox:~# cat /sys/module/kvm_amd/parameters/nested
     Y

3. Is the outer VM's CPU type set to host in the Proxmox config? So VMX/SVM flags get passed through
   • Check with qm config VM-ID und verify the cpu line says host

     root@proxmox:~# qm config 100
     [...]
     cores: 1
     cpu: host
     [...]

4. Does the inner QEMU call use accel=kvm (not kvm:tcg) plus -cpu host, so a broken KVM path fails loudly instead of silently degrading to software emulation?
5. Is the /dev/kvm device present on the VM where you want to start the QEMU VM?
6. Is CPU virtualization supported?

Nothing here is exotic once you know to look for it, but "silent TCG fallback" combined with "one VM inside another" is a pretty effective way to make a simple performance problem look mysterious for a while.