We're hiring!
*

Integrating libcamera into PipeWire

Raghavendra Rao avatar

Raghavendra Rao
September 11, 2020

Share this post:

Earlier this year, my colleagues Julian Bouzas and George Kiagiadakis wrote about PipeWire and Wireplumber, the PipeWire session manager. Since then, PipeWire has continued to evolve with the recent integration of libcamera, a library to support complex cameras. In this blog post, I'll explain why libcamera exists, what it does, and how we integrated it in PipeWire.

libcamera

Cameras are complex devices which require heavy hardware image processing. The complexity in the userspace camera application development originates with the evolution of complex hardware. libcamera addresses this problem and offers ease of camera application development. libcamera can be described as a full camera stack library in the userspace to provide the feature that do not belong in the kernel.

The core of libcamera framework exposes kernel driver APIs to userspace while the libcamera application layer abstracts the developer from complex hardware usage. libcamera supports multiple video streams from a single device. In real time use cases such as video conferencing, we may need to preview streams with different resolutions than what we tend to stream over the network. It is natural that we would like to display the live streaming simaultaneously while we capture the video with different resolutions. libcamera offers a perfect solution to fulfill these requirements.

Having said that, what if we'd like to stream a camera device data using different applications simaultaneously? Well, PipeWire answers this question. Besides acting as an interface between hardware devices and applications, PipeWire eases the sharing of hardware devices between applications. Moreover, integration of libcamera into PipeWire brings all the abilities offered by libcamera into PipeWire. With this, PipeWire can become a first class user of modern cameras.

PipeWire-libcamera interworking

As camera application development through v4l becomes more and more complex, Collabora decided to tackle this by integrating libcamera into PipeWire to offer all the evolutions coming from both the frameworks.

The integration work has been merged into PipeWire master.

Give it a try!

Please follow the steps below to try libcamera in PipeWire.

  1. Make sure that the linux kernel version is 5.0.0 or above.
  2. Make sure that the system user is added to video group to access /dev/media*. You can run the command sudo adduser `whoami` video and restart the session.
  3. Make sure that meson build package is installed in your system.
  4. Make sure that sdl2 package is installed in your system.
  5. Make sure that libdrm package is installed in your system.
  6. Follow Getting libcamera to obtain the source and install it. You can also use the commands below to build and install.
    meson --prefix <path/to/install dir> --buildtype=plain build
    cd build
    ninja -C . install
    
  7. Follow Getting PipeWire to obtain the source. Configure with meson and install it. Use the following commands:
    meson --prefix <path/to/install dir> -Dlibcamera=true build
    cd build
    ninja -C . install
    
  8. Now you can test libcamera spa plugin. Run the command "./build/spa/examples/local-libcamera" from the PipeWire directory. You should be able to see the camera captured data being rendered.

We will be blogging more about this integration work and improvements to be made in the future, explaining the motivation and further progress we make. Stay tuned!

Comments (4)

  1. Rogan Dawes:
    Sep 14, 2020 at 05:04 PM

    Is this something that could easily be exported as a remote desktop using VNC? Given the availability of cheap HDMI input dongles that appear as a UVC camera, it would be neat to be able to expose them using a regular "remote desktop" protocol. I guess an input filter would be required to receive the keystrokes and mouse movements, of course. Those could then be emitted using a HID Gadget and ConfigFS or similar.
    I did look at doing this with Xvncserver, but the rawfb support was a bit tricky to figure out, and I never got that working.

    Reply to this comment

    Reply to this comment

    1. Raghavendra:
      Sep 15, 2020 at 06:57 PM

      Hello Rogan Dawes,

      Thank you for your response.

      The idea behind the integration of libcamera was to ease the Camera application development with modern cameras and bring all the capabilities of libcamera to PipeWire.

      It is interesting to note that PipeWire already supports remote desktop sharing with wayland, gnome-remote-desktop and browsers such as firefox, chrome etc. PipeWire acts as interface between the Application and hardware to share the media via "dmabuf" buffers. Applications such as VNC/RDP can connect to PipeWire creating sink nodes to consume the Video frames captured by source nodes such as libcamera node, in this case.

      gnome-remote-desktop daemon enables GNOME to offer remote desktop sharing using VNC with PipeWire. This daemon supports GNOME on X as well as GNOME on Wayland. If you have got ubuntu installed then you may want to recompile mutter with remote desktop option enabled to make it work on ubuntu.

      Thanks,
      Raghavendra

      Reply to this comment

      Reply to this comment

  2. Andrew Zhukov:
    Jan 26, 2021 at 12:57 PM

    Can't get it work with latest libcamera and pipewire. Seems like pipewire uses old libcamera version...

    Reply to this comment

    Reply to this comment

    1. Raghavendra Rao:
      Feb 03, 2021 at 11:48 AM

      Hello Andrew Zhukov,

      Thanks for your message.

      The issue has been fixed and the fix has been merged into pipewire master. You can now check with pipewire master branch,

      Thanks,
      Raghavendra Rao

      Reply to this comment

      Reply to this comment


Add a Comment






Allowed tags: <b><i><br>Add a new comment:


Search the newsroom

Latest Blog Posts

Mainline Linux gains accelerated video decoding for Microchip's SAMA5D4

11/05/2021

The Hantro Video4Linux2 (V4L2) kernel module has gained support for another SoC! The Microchip SAMA5D4 features a single decode unit supporting…

Quick hack: Patching kernel modules using DKMS

05/05/2021

DKMS is a framework that is mostly used to build and install external kernel modules. It can also be used to install a specific patch to…

Build your own application with GTK 4 as a Meson subproject!

29/04/2021

Building GTK 4 as a Meson subproject for your own application is not only useful for Windows builds, but also for many Linux distributions…

Profiling virtualized GPU acceleration with Perfetto

22/04/2021

Recently, we have been using Perfetto to successfully profile Apitrace traces in crosvm through VirGL renderer. We have now added perfetto…

Continuous 3D Hand Pose Tracking using Machine Learning & Monado OpenXR

20/04/2021

As part of a project backed by INVEST-AI, a program managed by IVADO Labs, we have developed a multi-stage neural network-based solution…

An easy to use MTP implementation for your next embedded Linux project

13/04/2021

Did you know you could run a permissively-licensed MTP implementation with minimal dependencies on an embedded device? Here's a step-by-step…

Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2021. All rights reserved. Privacy Notice. Sitemap.

Collabora Limited is registered in England and Wales. Company Registration number: 5513718. Registered office: The Platinum Building, St John's Innovation Park, Cambridge, CB4 0DS, United Kingdom. VAT number: 874 1630 19.