With virtualization we can create multiple virtual machines over a single physical computer. The benefits of virtualization are countless, from being able to create virtual representation of different machines, to efficiently use the currently available hardware. Clearly a virtual machine, like any real computer, needs an operating system (OS). In this context it is called a Guest OS, as opposed to the one running on real hardware, called Host OS.

Running graphics applications in a Guest OS can be annoying as they are generally greedy of computing resources, and that can slow you down or give you a bad experience in terms of graphics performance. Being able to accelerate all this by offloading the workload to the hardware can be a great deal. The VirtIO-GPU virtual GPU device comes into play here, allowing a Guest OS to send graphics commands to it through OpenGL or Vulkan. While we are already there with OpenGL, we can not say the same for Vulkan. Well, until now.

Jump to a section: Overview | Definitions | Prerequisites | Create an image for QEMU | Running QEMU | Testing Venus | Troubleshooting | Conclusions

Overview

This blog post describes how to enable 3D acceleration of Vulkan applications in QEMU through the Venus experimental Vulkan driver for VirtIO-GPU with a local development environment.

As an alternative you could cherry-pick this commit which contains a set of scripts you could use to set up a Docker development environment.

Definitions

Let us start with a brief description of the projects mentioned in this post:

• QEMU is an open-source machine emulator, and we will use it to run an Ubuntu guest operating system and take advantage of the VirtIO-GPU device available in the virtual machine.
• VirGL is an OpenGL driver for VirtIO-GPU, available in Mesa.
• Venus is an experimental Vulkan driver for VirtIO-GPU, also available in Mesa.
• Virglrenderer is a library that enables hardware acceleration to VM guests, effectively translating commands from the two drivers just mentioned to either OpenGL or Vulkan.

Prerequisites

The following snippets are prefixed by either (host) or (guest) to specify where they should run. Of course, in order to run something in the guest, you should have QEMU and an image already in place.

Host

1. Venus requires BLOB resources support in QEMU, which in turns requires /dev/udmabuf. This is not enabled in the default Debian kernel, so make sure your kernel was built with CONFIG_UDMABUF.

   Please note that you could encounter the following error with kvm on AMD when enabling BLOB support: error: kvm run failed Bad address.

2. Clone virglrenderer res-sharing branch from FDO/fahien and compile it with:

   (host)
   
   git clone -b res-sharing https://gitlab.freedesktop.org/Fahien/virglrenderer.git
   
   cd virglrenderer
   
   meson build \
       -Dprefix=$HOME/.local \
       -Dplatforms=egl \
       -Dvenus-experimental=true \
       -Dminigbm_allocation=false \
       -Dbuildtype=debugoptimized
   
   ninja -C build install
   

3. Clone QEMU branch venus-dev from FDO/fahien. Then configure and compile it enabling OpenGL, VirGL, GTK (or SDL if you prefer this frontend):

   (host)
   
   git clone -b venus-dev https://gitlab.freedesktop.org/Fahien/qemu.git
   
   cd mesa
   
   mkdir build && cd build
   
   ../configure                   \
     --prefix=$HOME/.local        \
     --target-list=x86_64-softmmu \
     --enable-kvm                 \
     --disable-werror             \
     --enable-opengl              \
     --enable-virglrenderer       \
     --enable-gtk                 \
     --enable-sdl
   
   make -j4 qemu-system-x86_64 && make install
   

Guest

• Linux kernel v5.16-rc1+.

• Mesa version 21.1+, configured with meson -Dvulkan-drivers=virtio-experimental.

• Install vulkan-utils and run vulkaninfo | grep driver to get some info on the available vulkan drivers.

• Test vkcube.

Create an image for QEMU

You will need to provide QEMU an image. Here is an example of how to make one.

(host)

ISO=ubuntu-21.04-desktop-amd64.iso
wget https://releases.ubuntu.com/21.04/$ISO

IMG=ubuntu.qcow2
qemu-img create -f qcow2 $IMG 16G

# Start ubuntu installation by booting from CD-ROM.
# No need for graphics acceleration at the moment.
qemu-system-x86_64                  \
    -enable-kvm                     \
    -M q35                          \
    -smp 1                          \
    -m 4G                           \
    -net nic,model=virtio           \
    -net user,hostfwd=tcp::2222-:22 \
    -hda $IMG                       \
    -display gtk                    \
    -boot d -cdrom $ISO

Running QEMU

Running with -d guest_errors will show error messages from the guest.

(host)

qemu-system-x86_64                                               \
    -enable-kvm                                                  \
    -M q35                                                       \
    -smp 1                                                       \
    -m 4G                                                        \
    -cpu host                                                    \
    -net nic,model=virtio                                        \
    -net user,hostfwd=tcp::2222-:22                              \
    -hda $IMG                                                    \
    -device virtio-vga-gl,context_init=true,blob=true,hostmem=4G \
    -vga none                                                    \
    -initrd /image/rootfs.cpio.gz                                \
    -kernel /kernel/arch/x86_64/boot/bzImage                     \
    -append "root=/dev/sda3 nokaslr"                             \
    -display gtk,gl=on,show-cursor=on                            \
    -usb -device usb-tablet                                      \
    -object memory-backend-memfd,id=mem1,size=4G                 \
    -machine memory-backend=mem1                                 \
    -d guest_errors

OpenGL-enabled GTK3 display

VirtIO VGA GL (-device virtio-vga-gl) requires OpenGL support by the current QEMU display, which can be enabled with the following cli option -display gtk,gl=on.

For some reason, we hit a GTK assertion due to a failure in gtk_widget_get_realized(). The solution is to run QEMU with -vga none to avoid having two scanouts, one for VGA and another for virtio-vga-gl.

Build the kernel

I made a custom config (x86_64.config) to build VirtIO-GPU and DRM within the kernel with debug info.

(host)

git clone --depth 1 -b v5.16-rc1 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git kernel

cd kernel

./scripts/kconfig/merge_config.sh arch/x86/configs/x86_64_defconfig x86_64.config

make -j12 vmlinux bzImage

Starting Qemu with our custom kernel can be done by setting the current command line options:

-kernel arch/x86_64/boot/bzImage \
-inintrd ramdisk.img \
-append "root=/dev/sda3" \

You can create ramdisk.img by running mkinitramfs -o ramdisk.img

Testing Venus

Make sure VirGL is correctly detected and used by running the following:

(guest) glxinfo -B

If it outputs llvmpipe instead, build mesa with this configuration:

(guest)

git clone -b qemu-venus https://gitlab.freedesktop.org/Fahien/mesa.git

cd mesa

meson build                                   \
  -Dprefix=/usr                               \
  -Ddri3=enabled                              \
  -Dglx=dri                                   \
  -Degl=enabled                               \
  -Dgbm=enabled                               \
  -Dgallium-vdpau=disabled                    \
  -Dgallium-vs=disabled                       \
  -Dvalgrind=disabled                         \
  -Dbuildtype=debugoptimized                  \
  -Ddri-drivers=[]                            \
  -Dgallium-drivers=swrast,virgl              \
  -Dvulkan-drivers=swrast,virtio-experimental \
  -Dvulkan-layers=device-select

ninja -C build install

Then compile and and run vkcube to test Venus, making sure to tell mesa the correct Vulkan ICD file name:

(guest)

sudo apt install meson build-essential libdrm-dev libgbm-dev libpng-devibwayland-dev libxcb1-dev libvulkan-dev

git clone https://github.com/krh/vkcube.git

cd vkcube

meson build && meson compile -C build

VK_ICD_FILENAMES=/usr/shared/vulkan/icd.d/virtio_icd.x86_64.json build/vkcube

Troubleshooting

At this point the venus driver should be correctly loaded, and debug messages can be enabled by setting the VN_DEBUG environment variable to one of the following: init, result, vtest, or wsi.

Debugging the kernel

1. Run QEMU with arguments -s -S:
   • -S stops qemu waiting for gdb
   • -s starts a gdb server at localhost:1234

2. Your ~/.gdbinit should contain this (make sure it does not point directly to scripts/gdb/vmlinux-gdb.py):

   add-auto-load-safe-path /path/to/linux/vmlinux-gdb.py
   

3. Run gdb and attach to QEMU gdb server:

   (host)
   
   gdb vmlinux
   (gdb) target remote :1234
   (gdb) hbreak start_kernel
   (gdb) c
   

VsCode debugger

You can use VSCode Debug UI thanks to the Native Debug extension, attaching to gdbserver target :1234, and autorun: [ "hbreak kernel_init" ].

GDB

Sometimes your only choice is to debug with GDB, therefore here are some useful commands to keep in mind.

Vulkan loader

If your libvulkan.so fails with a segmentation fault, it would be a good idea to build it from source and debug it with gdb. Make sure to checkout a version in line with your libvulkan-dev.

(guest)

git clone https://github.com/KhronosGroup/Vulkan-Loader.git vulkan-loader
cd vulkan-loader
git checkout v1.2.162

Conclusions

While you can enable Vulkan hardware acceleration by checking out the development branches following this guide, there is still further work to do on Virglrenderer and QEMU for proper upstreaming, which might need some time to complete. Virgilrenderer needs to fix resource import/export between OpenGL and Vulkan contexts, and QEMU needs various patches currently under review:

• virtio-gpu: Shared memory capability
• virtio-gpu: CONTEXT_INIT feature
• virtio-gpu: Support Venus Vulkan driver

To sum up, if you need assistance with graphics virtualization, we would be happy to help, so please do not hesitate to contact us.

On the other hand, if you are a developer and would be thrilled to work on the open source Linux graphics stack, check out our careers page!