We're hiring!
*

Setting up QEMU-KVM for kernel development

Frédéric Dalleau avatar

Frédéric Dalleau
January 16, 2017

Share this post:

Reading time:

Introduction

Before doing linux kernel development, I started by typing make in a kernel tree. After booting, I always had some non working peripherals. So my second step was to use a distribution specific build procedure. For example, the Ubuntu kernel build instructions can be found at https://wiki.ubuntu.com/KernelTeam/GitKernelBuild. It works, and one can easily build a kernel and install it, with all peripheral working. But this method will quickly reach its limitations to write new kernel code. On a decent computer (i7 5600U), the build/test cycle lasts about 30 minutes. It is possible to build only the needed module and insmod/rmmod, but in case of a crash followed by a rebooting, the developer loses its work environment.

The next step is to run the kernel inside a virtual machine.

Booting kernel in a virtual machine

VirtualBox is well known, very user friendly and supports a large amount of different OSes. Installation of a VirtualBox will be under the hour mark. But rebuilding the Ubuntu kernel is still a 30 minute cycle. Additionnaly exchanging files between Virtualbox and the host will involve some kind of networking or file sharing that have to be setup. For kernel development, Virtualbox Guest additions have to be rebuilt often when the kernel is updated.

Qemu is another virtual machine. A complete distro can be installed into it. But it has a very interesting option: -kernel. With that option, QEMU will boot the kernel binary provided as argument. Ubuntu users can try :

$ sudo qemu-system-x86_64 -kernel /boot/vmlinuz-`uname -r`

 

This will boot your kernel within QEMU, but an error occurs immedialy: There is no filesystem to boot. Also, since /boot is readable only by root, sudo permission is required. This is not needed with a user built kernel.

Adding a rootfs

debootstrap allows to install a debian distribution in a directory. Before going too fast, if your file system is mounted with the nodev option, it won’t be possible to create device nodes. Instead, we will mount a qemu image file on a directory and use debootstrap in the mount point we created as in the following:

IMG=qemu-image.img
DIR=mount-point.dir
qemu-img create $IMG 1g
mkfs.ext2 $IMG
mkdir $DIR
sudo mount -o loop $IMG $DIR
sudo debootstrap --arch amd64 jessie $DIR
sudo umount $DIR
rmdir $DIR

 

The target rootfs is a matter of taste. For learning purposes, using busybox would be very interesting too. But for development purposes, having all the debian development tools in the rootfs is very useful.

Boot as follow:

$ sudo qemu-system-x86_64 -kernel /boot/vmlinuz-`uname -r`\
                          -hda qemu-image.img\
                          -append "root=/dev/sda single"

 

To silence the warning about raw format, replace “-hda qemu-image.img” with “-drive file=qemu-image.img,index=0,media=disk,format=raw”

Boot as a single user to change root password and create a user.

Building and booting a kernel

Now it is time to build your own kernel. There exist a make KVM configuration target that tunes an existing configuration and makes it usable from QEMU. However, it will not create a .config file from scratch. So we will start from generic config file and kvmify it. It is still possible to build a dedicated config file that will allow for shorter build time, but that would require several iterations.

git clone --depth=1 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
make x86_64_defconfig
# if building an older kernel use 'make kvmconfig" instead if below command fails
make kvm_guest.config
make -j 8

 

Use the resulting file in the command line below. We can drop sudo.

$ qemu-system-x86_64 -kernel arch/x86/boot/bzImage
                     -hda qemu-image.img
                     -append "root=/dev/sda"

 

Reduce the boot time with kvm

KVM accelerates x86 virtualization in QEMU. It will only accelerate x86 platforms. It is as simple as adding a command line option.

$ qemu-system-x86_64 -kernel bzImage
                     -hda qemu-image.img
                     -append "root=/dev/sda"
                     --enable-kvm

 

Now your debootstrap image boots in less than two seconds. This can be checked in dmesg. Before –enable-kvm, systemd is started 5.9 seconds after boot. After enabling, systemd is started after 1.7 seconds. A more than 3 times shorter boot time. And it just can’t be compared to ubuntu. Note that we don’t have a full user interface up, so we cannot compare apple with peaches.

Connecting into QEMU

Initially, QEMU displays its own screen in a dedicated window. For a terminal use case, this is not really pratical as it gets into the ALT tab list, the keyboard and mouse capture are not suitable for this use either. Copying and pasting also aren’t very practical. It is much more convenient to remove the graphic interface and instruct the kernel to write to ttyS0 that qemu redirect to the terminal in –nographic mode.

$ qemu-system-x86_64 -kernel bzImage
                     -append "root=/dev/sda console=ttyS0"
                     -hda qemu-image.img
                     --enable-kvm
                     --nographic

 

Time to leave

Typing halt in qemu will stop the kernel, but the qemu process would continue running on the host and would have to be killed. The proper command to terminate the virtual machine is :

$ shutdown -h now

 

A second part to this post is planned, stay tuned!

 

Original post

Comments (27)

  1. Johan:
    Aug 13, 2017 at 12:56 PM

    Thanks for a great post!

    I noticed two things while trying this out. It's not possible to use debootstrap with a qcow2 disk image. It will fail when it tries to create the device nodes, raw works flawlessly though. Secondly, I was not able to boot into single user mode using Debian Stretch. The root account was locked. But this can be worked around by using chroot and setting a root password.

    Reply to this comment

    Reply to this comment

    1. Frédéric Dalleau:
      Aug 21, 2017 at 08:09 AM

      Hello Johan,
      Thanks for feedback, you're right, on stretch chroot must be used after the debootstrap step:

      sudo debootstrap --arch amd64 stretch $DIR

      sudo chroot $DIR

      passwd


      The raw image can be converted to qcow2 and the resulting file can be used in the command lines instead:

      qemu-img convert -O qcow2 qemu-img.img qemu-img.qcow2

      Best regards,
      Frédéric

      Reply to this comment

      Reply to this comment

  2. Dongliang Mu:
    Sep 22, 2017 at 12:37 AM

    I use the shell script you mention at first to generate qemu-stretch.img(stretch) successfully.
    And then I use the following command to boot the kernel 4.12.0-1.
    qemu-system-x86_64 -kernel /boot/vmlinuz-4.12.0-1-amd64 -drive file=qemu-stretch.img,index=0,media=disk,format=raw -append "root=/dev/sda1 single"

    But the kernel panic and shows : "VFS - unable to mount root fs"

    Reply to this comment

    Reply to this comment

    1. Frédéric Dalleau:
      Oct 09, 2017 at 12:30 PM

      Hello Dongliang,
      Assuming you specified the appropriate kernel and disk image location, you could check the parameters passed to kernel command line. The following worked for me (changed /dev/sda1 to /dev/sda).

      qemu-system-x86_64 -kernel arch/x86/boot/bzImage -drive file=../qemu-image.img,index=0,media=disk,format=raw -append "root=/dev/sda single"


      Please note in the previous comment about the changes to add a password to a stretch image.
      Best regards,
      Frédéric

      Reply to this comment

      Reply to this comment

      1. Dongliang:
        Oct 10, 2017 at 02:29 PM

        I redo the experiment with the script you provide in the blog, and I build one kernel in my laptop. I try the following command:
        qemu-system-x86_64 -kernel ~/Repos/linux/arch/x86/boot/bzImage -drive file=qemu-image.img,index=0,media=disk,format=raw -append "root=/dev/sda single"

        The error message is :
        VFS: unable to mount root fs on unknown-block(0,0)

        I don't modify anything in the script.

        Reply to this comment

        Reply to this comment

        1. Frédéric Dalleau:
          Oct 11, 2017 at 06:28 AM

          Hello Dongliang,
          The error "VFS - unable to mount root fs" occurs if the image exists but the filesystem could not be mounted.
          Many possibilities exists: the image do not contain a filesystem, or the kernel does not support the selected filesystem.
          According to your blog link, you are using 'sudo mkfs.ext2' and this should not needed. I suggest you investigate in this direction.
          Frédéric

          Reply to this comment

          Reply to this comment

          1. Dongliang Mu:
            Oct 11, 2017 at 05:46 PM

            Thanks for your tip. I try to redo the experiment in the Ubuntu 14.04 LTS. It is successful. Amazing!
            In my previous Debian System, "/sbin" is not in the PATH by default. So I directly add "sudo" to change it to higher privilege.
            However, when I use "/sbin/mkfs.ext2" in the script and redo the experiment in my Debian Buster system, it still does not work.
            That's really really wired.

            Reply to this comment

            Reply to this comment

            1. Frédéric Dalleau:
              Oct 12, 2017 at 06:47 AM

              Hello,

              It's a great progress!
              Did you try to copy the working image and kernel on the other machine (with scp) and run the same command line?

              Regards,
              Frédéric

              Reply to this comment

              Reply to this comment

              1. Dongliang Mu:
                Oct 12, 2017 at 02:49 PM

                I tried to copy image generated by my Debian Buster system to Ubuntu 14.04 LTS, the image could work. And the system could boot the secure shell with jessie system.
                But when I copied the image generate by Ubuntu 14.04 LTS to Debian Buster, the image that boots successfully in Ubuntu cannot be booted in Debian Buster.
                I checked the version of two systems:
                Debian Buster : qemu 1:2.10.0+dfsg-1 amd64
                Ubuntu 14.04 : qemu 2.0.0+dfsg-2ubuntu1.36 amd64

                Several reasons for those failure:
                - Qemu changes some behaviors from 2.0.0 to 2.10.0;
                - Ubuntu compiles in different way from Debian, (small possibility);

                I will try to install one Ubuntu 16.04 and Debian Stretch and test image there. And at same time send one email to Debian Qemu QA Mailing List.
                They may have some great understanding about this problem.

                Reply to this comment

                Reply to this comment

                1. introom:
                  Nov 17, 2017 at 01:14 PM

                  Hi, dongliang

                  The exact cause of this VFS error depends on your environment.

                  One reason is the misconfiguration of the kernel. Normally "make defconfig" followed by a "make kvmconfig" should do.
                  Another reason is the virtualization platform, if you are using nested virtualization, especially KVM run inside VMWARE , notice that qemu machine type must be no later than 2.6 if you wanna use virtio driver. Be warned, this is a bug.

                  Reply to this comment

                  Reply to this comment

  3. Vakul Garg:
    Dec 27, 2017 at 06:06 AM

    Thanks for this wonderful post.

    The tips provided just jump started me with the kernel version for which I want to develop my kernel module. Earlier, I was struggling to build my kernel development env using user-mode-linux.

    Now, I want to have a shared folder between host and virtual machine. How can I do so?
    It seems that the root file system is mounted read only by linux in virtual machine.
    How can I make it read/write? I may need to install/update packages into root file system.

    Also some addition to this blog to enable networking in virtual machine would be a great value add.

    Reply to this comment

    Reply to this comment

    1. Frédéric Dalleau:
      Jan 26, 2018 at 06:14 PM

      Hi Vakul,
      Thanks for your comment, from the top of my head, I'd say you can remount / read write using that command :
      ```
      sudo mount -o rw,remount /
      ```
      For networking, check out this article: https://www.collabora.com/news-and-blog/blog/2017/03/13/kernel-debugging-with-qemu-overview-tools-available/

      Regards,
      Frédéric

      Reply to this comment

      Reply to this comment

  4. elmazzun:
    Dec 29, 2017 at 02:05 PM

    Nice tutorial!
    I followed all the instructions and I managed to boot the last kernel image, but I can't enter single user mode and reset the password.
    When you say "Boot as a single user to change root password and create a user", I can't enter GRUB or any other particular mode and I'm stuck in the login screen, without being able to proceed.
    How can I boot as a single user?

    Reply to this comment

    Reply to this comment

  5. Sreyas:
    Feb 28, 2018 at 07:40 PM

    Hi Frederic,

    Thanks for the great post.
    I followed the procedure that you've posted here with debootstrap with works fine.

    I am trying to make changes to the ext4 filesystem of the linux kernel. I have configured the ext4 to load as a module with "make menuconfig" but I am still unable to find /lib/modules after booting my kernel with qemu.

    I am not able to find any modules.

    Do let me know if you can help.

    Reply to this comment

    Reply to this comment

  6. Ciro Santilli:
    May 28, 2018 at 11:03 AM

    I have created a fully automated QEMU + Buildroot setup that gets you up on a minimal system in a single command: https://github.com/cirosantilli/linux-kernel-module-cheat

    Reply to this comment

    Reply to this comment

  7. David:
    Sep 23, 2018 at 02:00 PM

    works nice, however,
    1) I'm getting errors that my file system is read only. any idea how to change that? (when I'm in console inside the qemu).

    2) Is there an easy way to do this with ubuntu sever flavor?

    Reply to this comment

    Reply to this comment

      1. Ciro Santilli:
        Sep 25, 2018 at 08:07 AM

        Ah, I thought that was a reply to me, nevermind last post.

        Reply to this comment

        Reply to this comment

  8. irfan ullah:
    Jul 15, 2019 at 09:43 AM

    Thank you very much for very useful post. I am facing the problem with the log in.
    After booting the kernel with the commands 1, and 2 (given at the end), I have encountered with the problem that I am not able to logged in. For each user name, and password I am getting the same error "log in incorrect".
    I am running Ubuntu 16.04.6 LTS(xenial), and I have tried "sudo chroot $DIR", and "passwd" mentioed at (3), but it didn't work for me.
    Thank you :).

    1). "sudo debootstrap --arch amd64 xenial $DIR"
    2). "qemu-system-x86_64 -kernel /boot/vmlinuz-5.2.0 -append "root=/dev/sda console=ttyS0" -hda qemu-image.img --enable-kvm --nographic",
    3). https://www.collabora.com/news-and-blog/blog/2017/01/16/setting-up-qemu-kvm-for-kernel-development/#qcom182

    Reply to this comment

    Reply to this comment

  9. anon nospam:
    Oct 01, 2020 at 06:02 PM

    WARNING: 'make kvmconfig' will be removed after Linux 5.10
    Please use 'make kvm_guest.config' instead.

    Reply to this comment

    Reply to this comment

  10. Me:
    May 13, 2021 at 02:02 PM

    Please why do I get this error (I read the documentation but I can't decipher it):

    end Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.

    Reply to this comment

    Reply to this comment

    1. Ezequiel Garcia:
      May 14, 2021 at 01:33 PM

      Hello Me,

      Some of us have moved to using virtme, which is a tool wrapping qemu, and solving its complex setup.
      Maybe this post can help you get started:

      https://www.collabora.com/news-and-blog/blog/2018/09/18/virtme-the-kernel-developers-best-friend/

      You can use virtme's github repository issue tracker for other questions, in case you need more help.

      Good luck,
      Ezequiel

      Reply to this comment

      Reply to this comment

    2. Sebastian Reichel:
      May 14, 2021 at 01:53 PM

      Hi Me,

      The error means, that the kernel does not find /sbin/init, which is the first userspace process started by the kernel. That process is expected to start any other required user space process. If it does not exist there is no user space, which is obviously a fatal error. The kernel also suggest, that you can override the path to the binary by adding a kernel parameter, e.g. "init=/usr/local/bin/my-own-init". That's not something you need at this point, though. My guess is, that something went wrong during debootstrap and systemd-sysv has not been installed into your image (which provides /sbin/init in a default Debian system).

      Have fun,

      -- Sebastian

      Reply to this comment

      Reply to this comment

  11. MLK:
    Aug 12, 2021 at 02:34 PM

    If you ever meet this error: while setting root password restart the VM by adding "rw" to the " -append" parameter (just before "single"):
    qemu-system-x86_64 -kernel /boot/vmlinuz-`uname -r` -drive "file=qemu-image.img,index=0,media=disk,format=raw" -append "root=/dev/sda rw single"

    Reply to this comment

    Reply to this comment


Add a Comment






Allowed tags: <b><i><br>Add a new comment:


Search the newsroom

Latest Blog Posts

Automatic regression handling and reporting for the Linux Kernel

14/03/2024

In continuation with our series about Kernel Integration we'll go into more detail about how regression detection, processing, and tracking…

Almost a fully open-source boot chain for Rockchip's RK3588!

21/02/2024

Now included in our Debian images & available via our GitLab, you can build a complete, working BL31 (Boot Loader stage 3.1), and replace…

What's the latest with WirePlumber?

19/02/2024

Back in 2022, after a series of issues were found in its design, I made the call to rework some of WirePlumber's fundamentals in order to…

DRM-CI: A GitLab-CI pipeline for Linux kernel testing

08/02/2024

Continuing our Kernel Integration series, we're excited to introduce DRM-CI, a groundbreaking solution that enables developers to test their…

Persian Rug, Part 4 - The limitations of proxies

23/01/2024

This is the fourth and final part in a series on persian-rug, a Rust crate for interconnected objects. We've touched on the two big limitations:…

How to share code between Vulkan and Gallium

16/01/2024

One of the key high-level challenges of building Mesa drivers these days is figuring out how to best share code between a Vulkan driver…

Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2024. All rights reserved. Privacy Notice. Sitemap.