Setting up QEMU-KVM for kernel development

Frédéric Dalleau avatar

Posted on 16/01/2017 by Frédéric Dalleau


Before doing linux kernel development, I started by typing make in a kernel tree. After booting, I always had some non working peripherals. So my second step was to use a distribution specific build procedure. For example, the Ubuntu kernel build instructions can be found at https://wiki.ubuntu.com/KernelTeam/GitKernelBuild. It works, and one can easily build a kernel and install it, with all peripheral working. But this method will quickly reach its limitations to write new kernel code. On a decent computer (i7 5600U), the build/test cycle lasts about 30 minutes. It is possible to build only the needed module and insmod/rmmod, but in case of a crash followed by a rebooting, the developer loses its work environment.

The next step is to run the kernel inside a virtual machine.

Booting kernel in a virtual machine

VirtualBox is well known, very user friendly and supports a large amount of different OSes. Installation of a VirtualBox will be under the hour mark. But rebuilding the Ubuntu kernel is still a 30 minute cycle. Additionnaly exchanging files between Virtualbox and the host will involve some kind of networking or file sharing that have to be setup. For kernel development, Virtualbox Guest additions have to be rebuilt often when the kernel is updated.

Qemu is another virtual machine. A complete distro can be installed into it. But it has a very interesting option: -kernel. With that option, QEMU will boot the kernel binary provided as argument. Ubuntu users can try :

$ sudo qemu-system-x86_64 -kernel /boot/vmlinuz-`uname -r`


This will boot your kernel within QEMU, but an error occurs immedialy: There is no filesystem to boot. Also, since /boot is readable only by root, sudo permission is required. This is not needed with a user built kernel.

Adding a rootfs

debootstrap allows to install a debian distribution in a directory. Before going too fast, if your file system is mounted with the nodev option, it won’t be possible to create device nodes. Instead, we will mount a qemu image file on a directory and use debootstrap in the mount point we created as in the following:

qemu-img create $IMG 1g
mkfs.ext2 $IMG
mkdir $DIR
sudo mount -o loop $IMG $DIR
sudo debootstrap --arch amd64 jessie $DIR
sudo umount $DIR
rmdir $DIR


The target rootfs is a matter of taste. For learning purposes, using busybox would be very interesting too. But for development purposes, having all the debian development tools in the rootfs is very useful.

Boot as follow:

$ sudo qemu-system-x86_64 -kernel /boot/vmlinuz-`uname -r`\
                          -hda qemu-image.img\
                          -append "root=/dev/sda single"


To silence the warning about raw format, replace “-hda qemu-image.img” with “-drive file=qemu-image.img,index=0,media=disk,format=raw”

Boot as a single user to change root password and create a user.

Building and booting a kernel

Now it is time to build your own kernel. There exist a make kvmconfig option that tunes an existing configuration and makes it usable from QEMU. However, it will not create a .config file from scratch. So we will start from generic config file and kvmify it. It is still possible to build a dedicated config file that will allow for shorter build time, but that would require several iterations.

git clone --depth=1 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
make x86_64_defconfig
make kvmconfig
make -j 8


Use the resulting file in the command line below. We can drop sudo.

$ qemu-system-x86_64 -kernel arch/x86/boot/bzImage
                     -hda qemu-image.img
                     -append "root=/dev/sda"


Reduce the boot time with kvm

KVM accelerates x86 virtualization in QEMU. It will only accelerate x86 platforms. It is as simple as adding a command line option.

$ qemu-system-x86_64 -kernel bzImage
                     -hda qemu-image.img
                     -append "root=/dev/sda"


Now your debootstrap image boots in less than two seconds. This can be checked in dmesg. Before –enable-kvm, systemd is started 5.9 seconds after boot. After enabling, systemd is started after 1.7 seconds. A more than 3 times shorter boot time. And it just can’t be compared to ubuntu. Note that we don’t have a full user interface up, so we cannot compare apple with peaches.

Connecting into QEMU

Initially, QEMU displays its own screen in a dedicated window. For a terminal use case, this is not really pratical as it gets into the ALT tab list, the keyboard and mouse capture are not suitable for this use either. Copying and pasting also aren’t very practical. It is much more convenient to remove the graphic interface and instruct the kernel to write to ttyS0 that qemu redirect to the terminal in –nographic mode.

$ qemu-system-x86_64 -kernel bzImage
                     -append "root=/dev/sda console=ttyS0"
                     -hda qemu-image.img


Time to leave

Typing halt in qemu will stop the kernel, but the qemu process would continue running on the host and would have to be killed. The proper command to terminate the virtual machine is :

$ shutdown -h now


A second part to this post is planned, stay tuned!


Original post

Comments (17)

  1. Johan:
    Aug 13, 2017 at 12:56 PM

    Thanks for a great post!

    I noticed two things while trying this out. It's not possible to use debootstrap with a qcow2 disk image. It will fail when it tries to create the device nodes, raw works flawlessly though. Secondly, I was not able to boot into single user mode using Debian Stretch. The root account was locked. But this can be worked around by using chroot and setting a root password.

    Reply to this comment

    Reply to this comment

    1. Frédéric Dalleau:
      Aug 21, 2017 at 08:09 AM

      Hello Johan,
      Thanks for feedback, you're right, on stretch chroot must be used after the debootstrap step:

      sudo debootstrap --arch amd64 stretch $DIR

      sudo chroot $DIR


      The raw image can be converted to qcow2 and the resulting file can be used in the command lines instead:

      qemu-img convert -O qcow2 qemu-img.img qemu-img.qcow2

      Best regards,

      Reply to this comment

      Reply to this comment

  2. Dongliang Mu:
    Sep 22, 2017 at 12:37 AM

    I use the shell script you mention at first to generate qemu-stretch.img(stretch) successfully.
    And then I use the following command to boot the kernel 4.12.0-1.
    qemu-system-x86_64 -kernel /boot/vmlinuz-4.12.0-1-amd64 -drive file=qemu-stretch.img,index=0,media=disk,format=raw -append "root=/dev/sda1 single"

    But the kernel panic and shows : "VFS - unable to mount root fs"

    Reply to this comment

    Reply to this comment

    1. Frédéric Dalleau:
      Oct 09, 2017 at 12:30 PM

      Hello Dongliang,
      Assuming you specified the appropriate kernel and disk image location, you could check the parameters passed to kernel command line. The following worked for me (changed /dev/sda1 to /dev/sda).

      qemu-system-x86_64 -kernel arch/x86/boot/bzImage -drive file=../qemu-image.img,index=0,media=disk,format=raw -append "root=/dev/sda single"

      Please note in the previous comment about the changes to add a password to a stretch image.
      Best regards,

      Reply to this comment

      Reply to this comment

      1. Dongliang:
        Oct 10, 2017 at 02:29 PM

        I redo the experiment with the script you provide in the blog, and I build one kernel in my laptop. I try the following command:
        qemu-system-x86_64 -kernel ~/Repos/linux/arch/x86/boot/bzImage -drive file=qemu-image.img,index=0,media=disk,format=raw -append "root=/dev/sda single"

        The error message is :
        VFS: unable to mount root fs on unknown-block(0,0)

        I don't modify anything in the script.

        Reply to this comment

        Reply to this comment

        1. Frédéric Dalleau:
          Oct 11, 2017 at 06:28 AM

          Hello Dongliang,
          The error "VFS - unable to mount root fs" occurs if the image exists but the filesystem could not be mounted.
          Many possibilities exists: the image do not contain a filesystem, or the kernel does not support the selected filesystem.
          According to your blog link, you are using 'sudo mkfs.ext2' and this should not needed. I suggest you investigate in this direction.

          Reply to this comment

          Reply to this comment

          1. Dongliang Mu:
            Oct 11, 2017 at 05:46 PM

            Thanks for your tip. I try to redo the experiment in the Ubuntu 14.04 LTS. It is successful. Amazing!
            In my previous Debian System, "/sbin" is not in the PATH by default. So I directly add "sudo" to change it to higher privilege.
            However, when I use "/sbin/mkfs.ext2" in the script and redo the experiment in my Debian Buster system, it still does not work.
            That's really really wired.

            Reply to this comment

            Reply to this comment

            1. Frédéric Dalleau:
              Oct 12, 2017 at 06:47 AM


              It's a great progress!
              Did you try to copy the working image and kernel on the other machine (with scp) and run the same command line?


              Reply to this comment

              Reply to this comment

              1. Dongliang Mu:
                Oct 12, 2017 at 02:49 PM

                I tried to copy image generated by my Debian Buster system to Ubuntu 14.04 LTS, the image could work. And the system could boot the secure shell with jessie system.
                But when I copied the image generate by Ubuntu 14.04 LTS to Debian Buster, the image that boots successfully in Ubuntu cannot be booted in Debian Buster.
                I checked the version of two systems:
                Debian Buster : qemu 1:2.10.0+dfsg-1 amd64
                Ubuntu 14.04 : qemu 2.0.0+dfsg-2ubuntu1.36 amd64

                Several reasons for those failure:
                - Qemu changes some behaviors from 2.0.0 to 2.10.0;
                - Ubuntu compiles in different way from Debian, (small possibility);

                I will try to install one Ubuntu 16.04 and Debian Stretch and test image there. And at same time send one email to Debian Qemu QA Mailing List.
                They may have some great understanding about this problem.

                Reply to this comment

                Reply to this comment

                1. introom:
                  Nov 17, 2017 at 01:14 PM

                  Hi, dongliang

                  The exact cause of this VFS error depends on your environment.

                  One reason is the misconfiguration of the kernel. Normally "make defconfig" followed by a "make kvmconfig" should do.
                  Another reason is the virtualization platform, if you are using nested virtualization, especially KVM run inside VMWARE , notice that qemu machine type must be no later than 2.6 if you wanna use virtio driver. Be warned, this is a bug.

                  Reply to this comment

                  Reply to this comment

  3. Vakul Garg:
    Dec 27, 2017 at 06:06 AM

    Thanks for this wonderful post.

    The tips provided just jump started me with the kernel version for which I want to develop my kernel module. Earlier, I was struggling to build my kernel development env using user-mode-linux.

    Now, I want to have a shared folder between host and virtual machine. How can I do so?
    It seems that the root file system is mounted read only by linux in virtual machine.
    How can I make it read/write? I may need to install/update packages into root file system.

    Also some addition to this blog to enable networking in virtual machine would be a great value add.

    Reply to this comment

    Reply to this comment

    1. Frédéric Dalleau:
      Jan 26, 2018 at 06:14 PM

      Hi Vakul,
      Thanks for your comment, from the top of my head, I'd say you can remount / read write using that command :
      sudo mount -o rw,remount /
      For networking, check out this article: https://www.collabora.com/news-and-blog/blog/2017/03/13/kernel-debugging-with-qemu-overview-tools-available/


      Reply to this comment

      Reply to this comment

  4. elmazzun:
    Dec 29, 2017 at 02:05 PM

    Nice tutorial!
    I followed all the instructions and I managed to boot the last kernel image, but I can't enter single user mode and reset the password.
    When you say "Boot as a single user to change root password and create a user", I can't enter GRUB or any other particular mode and I'm stuck in the login screen, without being able to proceed.
    How can I boot as a single user?

    Reply to this comment

    Reply to this comment

  5. Sreyas:
    Feb 28, 2018 at 07:40 PM

    Hi Frederic,

    Thanks for the great post.
    I followed the procedure that you've posted here with debootstrap with works fine.

    I am trying to make changes to the ext4 filesystem of the linux kernel. I have configured the ext4 to load as a module with "make menuconfig" but I am still unable to find /lib/modules after booting my kernel with qemu.

    I am not able to find any modules.

    Do let me know if you can help.

    Reply to this comment

    Reply to this comment

  6. Ciro Santilli:
    May 28, 2018 at 11:03 AM

    I have created a fully automated QEMU + Buildroot setup that gets you up on a minimal system in a single command: https://github.com/cirosantilli/linux-kernel-module-cheat

    Reply to this comment

    Reply to this comment

Add a Comment

Allowed tags: <b><i><br>Add a new comment:

Latest Blog Posts

The docker.io Debian package is back to life


Last week, a new version of docker.io, the Docker package provided by Debian, was uploaded to Debian Unstable. Quickly afterwards, the package…

Introducing debos, a versatile images generator


In Debian and derivative systems, there are many ways to build images. The simplest tool of choice is often debootstrap. It works by downloading…

Secure video comes of age


Launched by Haivision in 2017, and freely available on GitHub via the Mozilla Public License 2.0, SRT is an innovative UDP-based protocol…

GStreamer CI support for embedded devices


Embedded devices are a popular deployment target for GStreamer yet they are not tested on the project's Continuous Integration (CI) system.…

Happy 20th, Open Source


In late January 1998, Netscape surprised everyone by releasing the source for Communicator, its web browser, making it readily available…

Four open months at Collabora


At the start of 2018 in January, I joined Collabora, an open source software consultancy, as a Software Engineer Intern with the Multimedia…

Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2018. All rights reserved. Website sitemap.