*

Setting up QEMU-KVM for kernel development

Posted on 16/01/2017 by Frédéric Dalleau

Introduction

Before doing linux kernel development, I started by typing make in a kernel tree. After booting, I always had some non working peripherals. So my second step was to use a distribution specific build procedure. For example, the Ubuntu kernel build instructions can be found at https://wiki.ubuntu.com/KernelTeam/GitKernelBuild. It works, and one can easily build a kernel and install it, with all peripheral working. But this method will quickly reach its limitations to write new kernel code. On a decent computer (i7 5600U), the build/test cycle lasts about 30 minutes. It is possible to build only the needed module and insmod/rmmod, but in case of a crash followed by a rebooting, the developer loses its work environment.

The next step is to run the kernel inside a virtual machine.

Booting kernel in a virtual machine

VirtualBox is well known, very user friendly and supports a large amount of different OSes. Installation of a VirtualBox will be under the hour mark. But rebuilding the Ubuntu kernel is still a 30 minute cycle. Additionnaly exchanging files between Virtualbox and the host will involve some kind of networking or file sharing that have to be setup. For kernel development, Virtualbox Guest additions have to be rebuilt often when the kernel is updated.

Qemu is another virtual machine. A complete distro can be installed into it. But it has a very interesting option: -kernel. With that option, QEMU will boot the kernel binary provided as argument. Ubuntu users can try :

$ sudo qemu-system-x86_64 -kernel /boot/vmlinuz-`uname -r`

 

This will boot your kernel within QEMU, but an error occurs immedialy: There is no filesystem to boot. Also, since /boot is readable only by root, sudo permission is required. This is not needed with a user built kernel.

Adding a rootfs

debootstrap allows to install a debian distribution in a directory. Before going too fast, if your file system is mounted with the nodev option, it won’t be possible to create device nodes. Instead, we will mount a qemu image file on a directory and use debootstrap in the mount point we created as in the following:

IMG=qemu-image.img
DIR=mount-point.dir
qemu-img create $IMG 1g
mkfs.ext2 $IMG
mkdir $DIR
sudo mount -o loop $IMG $DIR
sudo debootstrap --arch amd64 jessie $DIR
sudo umount $DIR
rmdir $DIR

 

The target rootfs is a matter of taste. For learning purposes, using busybox would be very interesting too. But for development purposes, having all the debian development tools in the rootfs is very useful.

Boot as follow:

$ sudo qemu-system-x86_64 -kernel /boot/vmlinuz-`uname -r`\
                          -hda qemu-image.img\
                          -append "root=/dev/sda single"

 

To silence the warning about raw format, replace “-hda qemu-image.img” with “-drive file=qemu-image.img,index=0,media=disk,format=raw”

Boot as a single user to change root password and create a user.

Building and booting a kernel

Now it is time to build your own kernel. There exist a make kvmconfig option that tunes an existing configuration and makes it usable from QEMU. However, it will not create a .config file from scratch. So we will start from generic config file and kvmify it. It is still possible to build a dedicated config file that will allow for shorter build time, but that would require several iterations.

git clone --depth=1 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
make x86_64_defconfig
make kvmconfig
make -j 8

 

Use the resulting file in the command line below. We can drop sudo.

$ qemu-system-x86_64 -kernel arch/x86/boot/bzImage
                     -hda qemu-image.img
                     -append "root=/dev/sda"

 

Reduce the boot time with kvm

KVM accelerates x86 virtualization in QEMU. It will only accelerate x86 platforms. It is as simple as adding a command line option.

$ qemu-system-x86_64 -kernel bzImage
                     -hda qemu-image.img
                     -append "root=/dev/sda"
                     --enable-kvm

 

Now your debootstrap image boots in less than two seconds. This can be checked in dmesg. Before –enable-kvm, systemd is started 5.9 seconds after boot. After enabling, systemd is started after 1.7 seconds. A more than 3 times shorter boot time. And it just can’t be compared to ubuntu. Note that we don’t have a full user interface up, so we cannot compare apple with peaches.

Connecting into QEMU

Initially, QEMU displays its own screen in a dedicated window. For a terminal use case, this is not really pratical as it gets into the ALT tab list, the keyboard and mouse capture are not suitable for this use either. Copying and pasting also aren’t very practical. It is much more convenient to remove the graphic interface and instruct the kernel to write to ttyS0 that qemu redirect to the terminal in –nographic mode.

$ qemu-system-x86_64 -kernel bzImage
                     -append "root=/dev/sda console=ttyS0"
                     -hda qemu-image.img
                     --enable-kvm
                     --nographic

 

Time to leave

Typing halt in qemu will stop the kernel, but the qemu process would continue running on the host and would have to be killed. The proper command to terminate the virtual machine is :

$ shutdown -h now

 

A second part to this post is planned, stay tuned!

 

Original post

Comments (11)

  1. Johan:
    Aug 13, 2017 at 12:56 PM

    Thanks for a great post!

    I noticed two things while trying this out. It's not possible to use debootstrap with a qcow2 disk image. It will fail when it tries to create the device nodes, raw works flawlessly though. Secondly, I was not able to boot into single user mode using Debian Stretch. The root account was locked. But this can be worked around by using chroot and setting a root password.

    Reply to this comment

    Reply to this comment

    1. Frédéric Dalleau:
      Aug 21, 2017 at 08:09 AM

      Hello Johan,
      Thanks for feedback, you're right, on stretch chroot must be used after the debootstrap step:

      sudo debootstrap --arch amd64 stretch $DIR

      sudo chroot $DIR

      passwd


      The raw image can be converted to qcow2 and the resulting file can be used in the command lines instead:

      qemu-img convert -O qcow2 qemu-img.img qemu-img.qcow2

      Best regards,
      Frédéric

      Reply to this comment

      Reply to this comment

  2. Dongliang Mu:
    Sep 22, 2017 at 12:37 AM

    I use the shell script you mention at first to generate qemu-stretch.img(stretch) successfully.
    And then I use the following command to boot the kernel 4.12.0-1.
    qemu-system-x86_64 -kernel /boot/vmlinuz-4.12.0-1-amd64 -drive file=qemu-stretch.img,index=0,media=disk,format=raw -append "root=/dev/sda1 single"

    But the kernel panic and shows : "VFS - unable to mount root fs"

    Reply to this comment

    Reply to this comment

    1. Frédéric Dalleau:
      Oct 09, 2017 at 12:30 PM

      Hello Dongliang,
      Assuming you specified the appropriate kernel and disk image location, you could check the parameters passed to kernel command line. The following worked for me (changed /dev/sda1 to /dev/sda).

      qemu-system-x86_64 -kernel arch/x86/boot/bzImage -drive file=../qemu-image.img,index=0,media=disk,format=raw -append "root=/dev/sda single"


      Please note in the previous comment about the changes to add a password to a stretch image.
      Best regards,
      Frédéric

      Reply to this comment

      Reply to this comment

      1. Dongliang:
        Oct 10, 2017 at 02:29 PM

        I redo the experiment with the script you provide in the blog, and I build one kernel in my laptop. I try the following command:
        qemu-system-x86_64 -kernel ~/Repos/linux/arch/x86/boot/bzImage -drive file=qemu-image.img,index=0,media=disk,format=raw -append "root=/dev/sda single"

        The error message is :
        VFS: unable to mount root fs on unknown-block(0,0)

        I don't modify anything in the script.

        Reply to this comment

        Reply to this comment

        1. Frédéric Dalleau:
          Oct 11, 2017 at 06:28 AM

          Hello Dongliang,
          The error "VFS - unable to mount root fs" occurs if the image exists but the filesystem could not be mounted.
          Many possibilities exists: the image do not contain a filesystem, or the kernel does not support the selected filesystem.
          According to your blog link, you are using 'sudo mkfs.ext2' and this should not needed. I suggest you investigate in this direction.
          Frédéric

          Reply to this comment

          Reply to this comment

          1. Dongliang Mu:
            Oct 11, 2017 at 05:46 PM

            Thanks for your tip. I try to redo the experiment in the Ubuntu 14.04 LTS. It is successful. Amazing!
            In my previous Debian System, "/sbin" is not in the PATH by default. So I directly add "sudo" to change it to higher privilege.
            However, when I use "/sbin/mkfs.ext2" in the script and redo the experiment in my Debian Buster system, it still does not work.
            That's really really wired.

            Reply to this comment

            Reply to this comment

            1. Frédéric Dalleau:
              Oct 12, 2017 at 06:47 AM

              Hello,

              It's a great progress!
              Did you try to copy the working image and kernel on the other machine (with scp) and run the same command line?

              Regards,
              Frédéric

              Reply to this comment

              Reply to this comment

              1. Dongliang Mu:
                Oct 12, 2017 at 02:49 PM

                I tried to copy image generated by my Debian Buster system to Ubuntu 14.04 LTS, the image could work. And the system could boot the secure shell with jessie system.
                But when I copied the image generate by Ubuntu 14.04 LTS to Debian Buster, the image that boots successfully in Ubuntu cannot be booted in Debian Buster.
                I checked the version of two systems:
                Debian Buster : qemu 1:2.10.0+dfsg-1 amd64
                Ubuntu 14.04 : qemu 2.0.0+dfsg-2ubuntu1.36 amd64

                Several reasons for those failure:
                - Qemu changes some behaviors from 2.0.0 to 2.10.0;
                - Ubuntu compiles in different way from Debian, (small possibility);

                I will try to install one Ubuntu 16.04 and Debian Stretch and test image there. And at same time send one email to Debian Qemu QA Mailing List.
                They may have some great understanding about this problem.

                Reply to this comment

                Reply to this comment

                1. introom:
                  Nov 17, 2017 at 01:14 PM

                  Hi, dongliang

                  The exact cause of this VFS error depends on your environment.

                  One reason is the misconfiguration of the kernel. Normally "make defconfig" followed by a "make kvmconfig" should do.
                  Another reason is the virtualization platform, if you are using nested virtualization, especially KVM run inside VMWARE , notice that qemu machine type must be no later than 2.6 if you wanna use virtio driver. Be warned, this is a bug.

                  Reply to this comment

                  Reply to this comment


Add a Comment





Allowed tags: <b><i><br>Add a new comment:


Latest Blog Posts

ipcpipeline: Splitting a GStreamer pipeline into multiple processes

17/11/2017

Earlier this year I worked on a certain GStreamer plugin that is called “ipcpipeline”. This plugin provides elements that make it possible…

Quick hack: Experiments with crosvm

09/11/2017

Running crosvm outside Chromium OS is quite easy, with the only complication being that minijail isn't widely packaged in distros. In these…

Tracing memory leaks in the NFC Digital Protocol stack

06/11/2017

Kmemleak allows you to track possible memory leaks inside the Linux kernel. Basically, it tracks dynamically allocated memory blocks in…

Who knew we still had low-hanging fruit?

17/10/2017

Earlier this month I had the pleasure of attending the Web Engines Hackfest, hosted by Igalia at their offices in A Coruña, and also sponsored…

Performance analysis in Linux (continued)

06/10/2017

In this post, I will show one more example of how easy it is to disrupt performance of a modern CPU, and also run a quick discussion on…

XDC 2017 - Links to recorded presentations (videos)

23/09/2017

Many thanks to Google for recording all the XDC2017 talks. To make them easier to watch, here are direct links to each talk recorded at…

Open Since 2005

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2017. All rights reserved. Website sitemap.