November 26, 2021
With virtualization we can create multiple virtual machines over a single physical computer. The benefits of virtualization are countless, from being able to create virtual representation of different machines, to efficiently use the currently available hardware. Clearly a virtual machine, like any real computer, needs an operating system (OS). In this context it is called a Guest OS, as opposed to the one running on real hardware, called Host OS.
Running graphics applications in a Guest OS can be annoying as they are generally greedy of computing resources, and that can slow you down or give you a bad experience in terms of graphics performance. Being able to accelerate all this by offloading the workload to the hardware can be a great deal. The VirtIO-GPU virtual GPU device comes into play here, allowing a Guest OS to send graphics commands to it through OpenGL or Vulkan. While we are already there with OpenGL, we can not say the same for Vulkan. Well, until now.
As an alternative you could cherry-pick this commit which contains a set of scripts you could use to set up a Docker development environment.
Let us start with a brief description of the projects mentioned in this post:
The following snippets are prefixed by either
(guest) to specify where they should run. Of course, in order to run something in the guest, you should have QEMU and an image already in place.
Venus requires BLOB resources support in QEMU, which in turns requires
/dev/udmabuf. This is not enabled in the default Debian kernel, so make sure your kernel was built with
Please note that you could encounter the following error with kvm on AMD when enabling BLOB support:
error: kvm run failed Bad address.
res-sharing branch from FDO/fahien and compile it with:
(host) git clone -b res-sharing https://gitlab.freedesktop.org/Fahien/virglrenderer.git cd virglrenderer meson build \ -Dprefix=$HOME/.local \ -Dplatforms=egl \ -Dvenus-experimental=true \ -Dminigbm_allocation=false \ -Dbuildtype=debugoptimized ninja -C build install
venus-devfrom FDO/fahien. Then configure and compile it enabling OpenGL, VirGL, GTK (or SDL if you prefer this frontend):
(host) git clone -b venus-dev https://gitlab.freedesktop.org/Fahien/qemu.git cd mesa mkdir build && cd build ../configure \ --prefix=$HOME/.local \ --target-list=x86_64-softmmu \ --enable-kvm \ --disable-werror \ --enable-opengl \ --enable-virglrenderer \ --enable-gtk \ --enable-sdl make -j4 qemu-system-x86_64 && make install
21.1+, configured with
meson -Dvulkan-drivers=virtio-experimental. Please note that to enable Vulkan headless you will need to apply this patch.
vulkan-utils and run
vulkaninfo | grep driver to get some info on the available vulkan drivers.
You will need to provide QEMU an image. Here is an example of how to make one.
(host) ISO=ubuntu-21.04-desktop-amd64.iso wget https://releases.ubuntu.com/21.04/$ISO IMG=ubuntu.qcow2 qemu-img create -f qcow2 $IMG 16G # Start ubuntu installation by booting from CD-ROM. # No need for graphics acceleration at the moment. qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -smp 1 \ -m 4G \ -net nic,model=virtio \ -net user,hostfwd=tcp::2222-:22 \ -hda $IMG \ -display gtk \ -boot d -cdrom $ISO
-d guest_errors will show error messages from the guest.
(host) qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -smp 1 \ -m 4G \ -cpu host \ -net nic,model=virtio \ -net user,hostfwd=tcp::2222-:22 \ -hda $IMG \ -device virtio-vga-gl,context_init=true,blob=true,hostmem=4G \ -vga none \ -initrd /image/rootfs.cpio.gz \ -kernel /kernel/arch/x86_64/boot/bzImage \ -append "root=/dev/sda3 nokaslr" \ -display gtk,gl=on,show-cursor=on \ -usb -device usb-tablet \ -object memory-backend-memfd,id=mem1,size=4G \ -machine memory-backend=mem1 \ -d guest_errors
VirtIO VGA GL (
-device virtio-vga-gl) requires OpenGL support by the current QEMU display, which can be enabled with the following cli option
For some reason, we hit a GTK assertion due to a failure in
gtk_widget_get_realized(). The solution is to run QEMU with
-vga none to avoid having two scanouts, one for VGA and another for virtio-vga-gl.
I made a custom config (x86_64.config) to build VirtIO-GPU and DRM within the kernel with debug info.
(host) git clone --depth 1 -b v5.16-rc1 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git kernel cd kernel ./scripts/kconfig/merge_config.sh arch/x86/configs/x86_64_defconfig x86_64.config make -j12 vmlinux bzImage
Starting Qemu with our custom kernel can be done by setting the current command line options:
-kernel arch/x86_64/boot/bzImage \ -inintrd ramdisk.img \ -append "root=/dev/sda3" \
You can create
mkinitramfs -o ramdisk.img
Make sure VirGL is correctly detected and used by running the following:
(guest) glxinfo -B
If it outputs
llvmpipe instead, build mesa with this configuration:
(guest) git clone -b qemu-venus https://gitlab.freedesktop.org/Fahien/mesa.git cd mesa meson build \ -Dprefix=/usr \ -Ddri3=enabled \ -Dglx=dri \ -Degl=enabled \ -Dgbm=enabled \ -Dgallium-vdpau=disabled \ -Dgallium-vs=disabled \ -Dvalgrind=disabled \ -Dbuildtype=debugoptimized \ -Ddri-drivers= \ -Dgallium-drivers=swrast,virgl \ -Dvulkan-drivers=swrast,virtio-experimental \ -Dvulkan-layers=device-select ninja -C build install
Then compile and and run vkcube to test Venus, making sure to tell mesa the correct Vulkan ICD file name:
(guest) sudo apt install meson build-essential libdrm-dev libgbm-dev libpng-devibwayland-dev libxcb1-dev libvulkan-dev git clone https://github.com/krh/vkcube.git cd vkcube meson build && meson compile -C build VK_ICD_FILENAMES=/usr/shared/vulkan/icd.d/virtio_icd.x86_64.json build/vkcube
At this point the venus driver should be correctly loaded, and debug messages can be enabled by setting the
VN_DEBUG environment variable to one of the following:
-Sstops qemu waiting for gdb
-sstarts a gdb server at
~/.gdbinit should contain this (make sure it does not point directly to
Run gdb and attach to QEMU gdb server:
(host) gdb vmlinux (gdb) target remote :1234 (gdb) hbreak start_kernel (gdb) c
You can use VSCode Debug UI thanks to the Native Debug extension, attaching to gdbserver target
autorun: [ "hbreak kernel_init" ].
Sometimes your only choice is to debug with GDB, therefore here are some useful commands to keep in mind.
||Print the current frame|
||Change to frame number|
||Print out a bunch of lines of code around the current instruction pointer|
||Set a breakpoint|
||Step into a function|
||Step over to the next instruction within the current function|
||Step out of a function|
||Delete all breakpoints|
||Delete breakpoint number|
||Show list of loaded library|
libvulkan.so fails with a segmentation fault, it would be a good idea to build it from source and debug it with gdb. Make sure to checkout a version in line with your
(guest) git clone https://github.com/KhronosGroup/Vulkan-Loader.git vulkan-loader cd vulkan-loader git checkout v1.2.162
While you can enable Vulkan hardware acceleration by checking out the development branches following this guide, there is still further work to do on Virglrenderer and QEMU for proper upstreaming, which might need some time to complete. Virgilrenderer needs to fix resource import/export between OpenGL and Vulkan contexts, and QEMU needs various patches currently under review:
To sum up, if you need assistance with graphics virtualization, we would be happy to help, so please do not hesitate to contact us.
On the other hand, if you are a developer and would be thrilled to work on the open source Linux graphics stack, check out our careers page!
Monado now has initial support for 6DoF ("inside-out") tracking for devices with cameras and an IMU! Three free and open source SLAM/VIO…
When developing an application or a library, it is very common to want to run it without installing it, or to install it into a custom prefix…
An incredible amount has changed in Mesa and in the Vulkan ecosystems since we wrote the first Vulkan driver in Mesa for Intel hardware…
Every file system used in production has tools to try to recover from system crashes. To provide a better infrastructure for those tools,…
The PipeWire project made major strides over the past few years, bringing shiny new features, and paving the way for new possibilities in…
Over the past 18 months, we have been on a roller-coaster ride developing futex2, a new set of system calls. As part of this effort, the…