We're hiring!
*

Kernel debugging with QEMU: An overview of tools available

Frédéric Dalleau avatar

Frédéric Dalleau
March 13, 2017

Share this post:

After setting up a virtual machine using debootstrap, let’s do some more advanced work.

Kmemleak

kmemleak can be used to search for memory leaks. Enable it in the kernel and rebuild. Note that we update the “Maximum kmemleak early log entries”. If the value is too low, kmemleak would disable itself at boot time.

        -> Memory Debugging
                [*]Enable kernel memory leak detector.
                (2000) Maximum kmemleak early log entries


After booting this kernel, the memory leaks will be detected periodically. When a leak would be detected, traces will be displayed in dmesg. Leak detection can be triggered manually too.

$ echo scan >> /sys/kernel/debug/kmemleak
$ cat /sys/kernel/debug/kmemleak


A scan has a certain duration, and the detected leaks won’t appear immediately. Even after a manual scan, if no leak is detected, nothing is displayed. To ensure that no leaks where detected, it is better to wait a few tenths of seconds, and also to trigger several manual scans.

Full documentation can be found here: https://www.kernel.org/doc/Documentation/kmemleak.txt

Networking

Using serial link has some limitations as terminal emulation problems sometime occur. A screen or vi session in the guest sometimes get screwed up and need to be restarted. Hence ssh is a better candidate. To enable networking, the option is : -net nic -net user. You can immediatly notice that the interface is not setup.

auto eth0
iface eth0 inet dhcp

 

$ apt-get install openssh-server
$ adduser fredo


To add a redirection from ssh guest port to a host port -net nic -net user,hostfwd=tcp::5555-:22.

$ qemu-system-x86_64 -kernel bzImage
                     -append "root=/dev/sda console=ttyS0 single"
                     -drive file=toto.img,index=0,media=disk,format=raw
                     --enable-kvm --nographic
                     -net nic -net user,hostfwd=tcp::5555-:22


On the host, connect with:

$ ssh -p 5555 fredo@localhost

Bridging a usb device with QEMU

A realworld USB device can be bridged into QEMU. It takes the following option. -usb -usbdevice host:050d:016a. The cryptic last two values are product and vendor ids. They can be retrieved using lsusb on the host.

Booting in GDB

We can go even further in debugging the kernel using GDB. Just add the necessary options. Those can be found in “Kernel hacking”.

        [*] KGDB: kernel debugger  --->
                <*>   KGDB: use kgdb over the serial console
        -> Compile-time checks and compiler options
                [*] Compile the kernel with debug info


We must add the -s option to qemu. It will instruct qemu to create a serial port to control kernel debugging in the guest. This serial port will be backed to a tcp socket that a GDB running on the host can connect to.

Our command line becomes:

$ qemu-system-x86_64 -kernel arch/x86/boot/bzImage
                     -hda toto.img
                     -append "root=/dev/sda" -s


qemu will start as usual and run as if nothing happened.

vmlinux is the uncompressed bzImage and the symbols are included in the binary. This is what we use in GDB:

$ gdb vmlinux


In GDB, attach to the running machine:

(gdb) target remote localhost:1234


Note that the virtual machine has been stopped. From here it is now possible to inspect the backtrace and use typical GDB commands: breakpoint, continue… Try that (and delete quickly):

(gdb) breakpoint spin_lock
(gdb) continue


More kernel debugging options

In kernel hacking, an good amount of debugging tools are available: dynamic_debug: You want to enable dynamic debug.

        -> printk and dmesg options
                [*] Enable dynamic printk() support


ftrace: ftrace can be useful too

        -> Tracers (FTRACE [=y])


I wont tell more here, since excellent lectures already exists: see lwn here and here.

Using several QEMU at same time

Only one QEMU at a time can handle -s. Should you really need to debug two machines at a time, it is possible to create a dedicated serial port per process.

Using the same disk image from two machines at same time will generate undefined results and is highly likely to break the disk image. Just copy the image with a new name to use it. QEMU also provide copy-on-write, which helps to solve that problem by writing changes to the original filesystem in a separate file.

 

Original post

Comments (4)

  1. lixiaoquan:
    Feb 18, 2018 at 08:07 AM

    I meet an error when executing this:

    auto eth0
    iface eth0 inet dhcp

    When I execute them, it reports auto can't be found, it seems "auto / iface"are not included in rootfs which is generated by debootstrap

    Reply to this comment

    Reply to this comment

    1. Vasyl:
      Oct 01, 2018 at 08:49 PM

      This are standard ifupdown scripts configuration. It should be located in /etc/network/interfaces file for example. Note that eth0 should be replaced with actual network interface name on your machine.

      Reply to this comment

      Reply to this comment

  2. pavan:
    Oct 01, 2018 at 04:43 PM

    For using multiple qemu instances, use -gdb tcp:: option.
    for using the same image, we can always pass image in snapshot mode.

    Reply to this comment

    Reply to this comment

  3. Vasyl:
    Oct 01, 2018 at 08:53 PM

    When using gdb remote debugging via qemu gdb server I can not step in kernel code. Every time I try I end up in some interrupt handling code

    (gdb) b vmem_export_single
    Breakpoint 1 at 0xffffffffa0000390: file drivers/dma-buf/vmem-exp.c, line 382.
    (gdb) c
    Continuing.

    Breakpoint 1, vmem_export_single (ctx=0xffff880005886d40, req=0xffffc900003f7e80) at drivers/dma-buf/vmem-exp.c:382
    382 req->vaddr &= PAGE_MASK;
    (gdb) s
    lapic_next_event (delta=, evt=) at arch/x86/kernel/apic/apic.c:462
    462 }
    (gdb) cont
    Continuing.

    Breakpoint 1, vmem_export_single (ctx=0xffff880005886d40, req=0xffffc900003f7e80) at drivers/dma-buf/vmem-exp.c:382
    382 req->vaddr &= PAGE_MASK;
    (gdb) s
    lapic_next_event (delta=, evt=) at arch/x86/kernel/apic/apic.c:462
    462 }
    (gdb)

    Reply to this comment

    Reply to this comment


Add a Comment






Allowed tags: <b><i><br>Add a new comment:


Search the newsroom

Latest Blog Posts

Zink: Fall Update

24/10/2019

I recently went to XDC 2019, where I gave yet another talk about Zink. I kinda forgot to write a blog-post about it, so here’s me trying…

Adding stateless support to vicodec

09/10/2019

Prior to joining Collabora, I took part in Round 17 of the Outreachy internships, to work on the virtual drivers in the media subsystem…

Why HDCP support in Weston is a good thing

03/10/2019

What HDCP is, and why supporting HDCP in Weston is justified in both an economical and technical context.

Virglrenderer and the state of virtualized virtual worlds

28/08/2019

With the release of virglrenderer 0.8.0, getting accelerated OpenGL within a virtual machine (VM) made a big leap forward. Since virglrenderer-0.7.0,…

ROCK Pi and an easy place: Panfrost & Wayland on a Rockchip board

06/08/2019

Ongoing work on the reverse-engineered Panfrost OpenGL ES driver for Arm Mali GPUs has turned the RK3399 SoC into a very attractive platform…

What's new in OpenXR 1.0 & Monado?

02/08/2019

As part of its unwavering commitment to open source and open standards, Collabora is proud to be part of bringing the recently-released…

Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2019. All rights reserved. Website sitemap.