We're hiring!
*

Using syzkaller, part 2: Detecting programming bugs in the Linux kernel

Andre Almeida avatar

Andre Almeida
April 17, 2020

Share this post:

Reading time:

In my previous blog post, we discussed the importance of testing, what is fuzzing, and how the syzkaller fuzzes the kernel in order to find bugs. Now, let’s install the tool and starting using it to improve our code base.

The kernel source will be expected to be found in the $KSRC directory. The syscall descriptions are based on linux-next, so if something fails or triggers a warning that a specific syscall isn’t defined, consider changing to the current linux-next/master branch.

Your kernel should be specifically configured to enhance the performance of syzkaller and to enable it to be properly fuzzed. The following configs are necessary:

CONFIG_KCOV=y
CONFIG_KCOV_INSTRUMENT_ALL=y
CONFIG_KCOV_ENABLE_COMPARISONS=y
CONFIG_DEBUG_FS=y
CONFIG_CONFIGFS_FS=y
CONFIG_SECURITYFS=y

To be able to navigate through the code coverage in the web interface, CONFIG_DEBUG_INFO must also be enabled. To get the most of the power of syzkaller, there are more configuration options that can be enabled, the more the better honorable mention for KASAN. After modifying the configuration, build the kernel again, since we will need a compiled kernel in next steps. Also, it’s required to have the following tools:

  • Go
  • QEMU
  • git
  • OpenSSH
  • A recent C compiler with coverage support (both current GCC and Clang versions should work)

To get the Go toolchain and syzkaller source, run:

go get golang.org/dl/go1.12
go get -u -d github.com/google/syzkaller/...

This doesn’t seem a very canonical way of downloading and installing a package, but since this project is created by Go developers, I believe that their suggestion is currently the best approach. Syzkaller should now be at ~/go/src/github.com/google/syzkaller. Not a very convenient path, but this can be solved with a symlink or the PATH variable. In this example, it will be defined by the following variable:

export SYZPATH=~/go/src/github.com/google/syzkaller

A basic rootfs for the virtual machines can be created using a script provided by syzkaller that requires the debootstrap package. Choose a directory in which to store rootfs images (which is stored as `$IMAGES` below), then run:

export IMAGES=$(pwd)
bash $SYZPATH/tools/create-image.sh

Use -h to explore the other options provided by this tool. In summary, it will create a basic Debian image with the tools that we need and a SSH server to copy and run commands across virtual machines.

Verifying your setup

Just to make sure everything is working fine and avoid some troubleshooting in the future, check if the ssh is properly working. First, boot the machine:

qemu-system-x86_64 \
  -kernel $KSRC/arch/x86/boot/bzImage \
  -append "console=ttyS0 root=/dev/sda debug earlyprintk=serial slub_debug=QUZ"\
  -hda $IMAGES/stretch.img \
  -net user,hostfwd=tcp::10021-:22 -net nic \
  -enable-kvm \
  -nographic \
  -m 2G \
  -smp 2 \
  -pidfile vm.pid \
  2>&1 | tee vm.log

It’s possible to login as root if debugging is necessary.

At the host system, try to access it:

ssh -i $IMAGES/stretch.id_rsa -p 10021 -o "StrictHostKeyChecking no" root@localhost

When done, kill the VM with kill $(cat $IMAGES/vm.pid).

Build and configure syzkaller and its tools

Now we have a working rootfs, we are almost ready. Go ahead and build syzkaller and its tools:

cd $SYZPATH/
make -j8

We need to create a work directory, where the tool will create a database and store results. Having a separate directory is a good way to organize information as it allows us to have different databases for different syscalls or for different configurations. To do that, just create different directories and set the one you want to work with in the configuration file. Any name works, but those starting with workdir* will be ignored by git, as per a rule in .gitignore file.

mkdir workdir

To define the behavior of the tool, create a configuration file somewhere, here called as config.cfg. This example should be enough for an initial run:

{
        "target": "linux/amd64",
        "http": "127.0.0.1:56741",
        "workdir": "$SYZPATH/workdir",
        "kernel_obj": "$KSRC",
        "image": "$IMAGES/stretch.img",
        "sshkey": “$IMAGES/stretch.id_rsa",
        "syzkaller": "$SYZPATH",
        "procs": 8,
        "type": "qemu",
        "vm": {
                "count": 2,
                "kernel": "$KSRC/arch/x86/boot/bzImage",
                "cpu": 2,
                "mem": 2048
        }
}

The contents of the above file are as follows:

  • target: operating system/architecture to be fuzzed
  • http: IP address and port of where syzkaller’s web interface will be exposed
  • workdir: the work directory to be used, as explained above
  • kernel_obj: kernel source directory
  • image: bootstraped distro image
  • sshkey: SSH key to be used to access the VMs
  • syzkaller: path to syzkaller source
  • procs: number of parallel tests inside each VM
  • type: virtual machine hypervisor/device to be used
  • vm: virtual machine configuration:
    • count: number of VM’s to spawn
    • kernel: kernel image to fuzz
    • cpu: number of cores simulate in each VM (just as -smp at QEMU)
    • mem: RAM size of VM (just as -m at QEMU)

By default, all syscall are fuzzed (we will see how to change that later). We can now start the fuzzer, specifying the configuration file:

./bin/syz-manager -config=config.cfg

2020/02/17 14:22:01 loading corpus...
2020/02/17 14:22:01 serving http on http://127.0.0.1:56741
2020/02/17 14:22:01 serving rpc on tcp://[::]:46009
2020/02/17 14:22:01 booting test machines...
2020/02/17 14:22:01 wait for the connection from test machine...
...

Open the URL provided and you should be able to check the web interface. It’s a convenient way to navigate through the features, check coverage reports, crashes and corpus. Notice that the server may seem unresponsive for a few seconds while the tool is performing some heavy operations, so be patient.

This is what the main dashboard looks like. In the first section, you can read the current status fo the tool, like the uptime, number of syscalls enabled, crashes and fuzzer execution statistics. In the second one, you have a list of crashes, how many times it happened. It may also export a report and a C reproducer of the crash. The last section is the log, the same log that the tool outputs in the terminal with live information about the tool progress.

Clicking in corpus, you can see the set of syscalls that produces the current coverage. If a syscall can't increase the coverage, it will not be saved in this database.

The list shows a link for the coverage that this syscall had produce as well as the name of the syscall description. Clicking on the name, you can see the input of this corpus. For instance, this is an exemple of syz_mount_image$ext4:

syz_mount_image$ext4(0x0, &(0x7f0000000040)='./file0\x00', 0x0, 0x0, 0x0, 0x0, 0x0)

Clicking in a coverage report in the corpus page will show the coverage for this particular input. Clicking in the coverage link in the main dashboard will show the global coverage, for the current corpus.

This is the current coverage of the file usercopy.c for the current corpus. An explanation about the coverage colors can be found in the documentation.

And we are done! This is the basic usage of syzkaller. Dig into the documentation for more details and see this source file for more options for the configuration. I found the following ones to be very useful to add at config.cfg file:

  • "enable_syscalls": [ "eventfd", "read$eventfd", ... ]: only those syscalls will be fuzzed. If they depend on another syscall to work (e.g. open()) syzkaller will ask you to enable it. Using read will trigger all definitions for this syscalls, including the eventfd one. Using read$eventfd will trigger just the eventfd one.
  • "disable_syscalls": ["ioctl$int_int", “mmap”, ... ]: all syscalls will be fuzzed, except those defined here.

Those new options may be added anywhere at the config file, with the exception inside the vm curly braces.

Summary

Now you should be ready to starting using syzkaller. Remember that the tool needs some some time to starting showing some results. Join the mailing list to get in touch with developers, see new bugs found by the tool and to report the crashes that you found.

In the final part of this series, we are going to see how to modify syzkaller in order to stress your own changes and improve testing in our development process.

Continue reading (Using syzkaller, part 3: Fuzzing your changes)

Comments (0)


Add a Comment






Allowed tags: <b><i><br>Add a new comment:


Search the newsroom

Latest Blog Posts

Automatic regression handling and reporting for the Linux Kernel

14/03/2024

In continuation with our series about Kernel Integration we'll go into more detail about how regression detection, processing, and tracking…

Almost a fully open-source boot chain for Rockchip's RK3588!

21/02/2024

Now included in our Debian images & available via our GitLab, you can build a complete, working BL31 (Boot Loader stage 3.1), and replace…

What's the latest with WirePlumber?

19/02/2024

Back in 2022, after a series of issues were found in its design, I made the call to rework some of WirePlumber's fundamentals in order to…

DRM-CI: A GitLab-CI pipeline for Linux kernel testing

08/02/2024

Continuing our Kernel Integration series, we're excited to introduce DRM-CI, a groundbreaking solution that enables developers to test their…

Persian Rug, Part 4 - The limitations of proxies

23/01/2024

This is the fourth and final part in a series on persian-rug, a Rust crate for interconnected objects. We've touched on the two big limitations:…

How to share code between Vulkan and Gallium

16/01/2024

One of the key high-level challenges of building Mesa drivers these days is figuring out how to best share code between a Vulkan driver…

Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2024. All rights reserved. Privacy Notice. Sitemap.