Interviewed by Christian Schaller

Farstream and libnice: An interview with Youness Alaoui

Published on 5th of December 2011

A popular talk at this year's GStreamer Conference was Youness Alaoui talking about his work on GstFilters and Farstream utils, two higher level convenience libraries to make doing VoIP and video conferencing easier and less error prone. So we thought that after the success of the GStreamer 1.0 interview with Wim Taymans, an interview with Youness about his work around Farstream, libnice, GStreamer and Telepathy would be of interest.

The goal of these interviews is to help inform our customers and partners on what is happening with some of the technologies which Collabora offers support and consulting services around and at the same time let everyone get to know some of the talented people working at Collabora a little better.

For those who don't know you, can you tell us a little about yourself?

Youness Alaoui: My name is Youness Alaoui, also known as KaKaRoTo. I'm a Moroccan and I'm living in Canada. I've always been fascinated by computers and I learned to program for my TI-83 calculator back in high school. I eventually followed a Software Engineering degree at the Ecole Polytechnique de Montreal.

I am who I am today thanks to the open source community that allowed me to learn new things everyday and enhance my programming knowledge. I am part of the aMSN project and an active member of the PS3 hacking scene, as one of founder of many open source projects revolving around the jailbreaking of the Playstation 3 system.

I have been working for Collabora since its early days, for the past 4 and a half year's now, and am considered one of the VoIP/Streaming experts, as well as the maintainer of libnice, a NAT traversal library.

How did you originally get involved with open source software development?

Youness Alaoui: I just recently remembered my very first contribution to open source, which was in 2001. I had an ADSL USB modem which didn't have drivers on Linux, and as I was trying to make it work I found that the chipset used was the same as another similar modem which was supported through an open community in France, so I hacked it up to make it work with mine and contributed back my changes, which ended up being useful to Canadians who had a similar model as mine.

What really got me involved with Open Source though was joining the aMSN project in early 2002. That was initiated by my friend Philippe Kalaf (who is now Collabora's founder) who convinced me to join the project. I later became the project leader and was heavily involved in the community. From there, everything unfolded, with me getting hired by Collabora and my various contributions to numerous open source projects.

The biggest project you work on at Collabora is Farstream (formerly Farsight),for those who don't know about Farstream, what does it do?

Youness Alaoui: Farstream is an API and a library that provides GStreamer elements for doing Voice and Video over IP. The most used element is the fsrtpconference bin that provides a full RTP stack and takes care of building the pipeline needed for RTP conferences, while also handling codec discovery and negotiation as well as NAT traversal using libnice.

So for those familiar with high level GStreamer elements such as playbin or encodebin, Farstream is the VoIP equivalent of those?

Youness Alaoui: Kind of, yes, but I'd say it's more like decodebin rather than playbin. The fsrtpconference still needs you to provide it with the source data, and you'd need to listen for 'srcpad_added' signals and connect the sinks to it when you receive data from your peers. But other than that, it pretty much does everything for you, so yes, it would be the equivalent of decodebin+encodebin for VoIP. If you use Farstream-Utils, then you could consider it like playbin since Farstream-Utils takes care of plugging the sources and sinks and building the complete pipeline for you.

You mention Farstream-Utils. Could you tell us a bit about where it fits in?

Youness Alaoui: I talked about Farstream-Utils and also a library called GstFilters at this years GStreamer Conference in Prague. For those who missed the talk, I suggest they view the videos of the GStreamer Conference and specifically the one about GstFilters and Farstream-Utils as it will help them get a complete understanding of the scope and purpose of those libraries. However, a quick overview of those library would be that they are convenience libraries that make it easy to build VoIP applications as well as, in the case of GstFilters, non-VoIP applications that make use of GStreamer. While GStreamer is extremely powerful, it also means that it has an extensive API that is hard to learn and might be intimidating to newcomers. GstFilters tries to solve that by providing a simple API of only two functions which can be used to build very complex pipelines in a minimal amount of effort. In the case of Farstream-Utils, it provides wrappers around Farstream's object and will build the complete GStreamer pipeline for you, allowing you to concentrate on the User Interface rather than on the internal streaming details. With those two libraries combined, you can build a fully featured VoIP application with video effects support, volume and level controls, and even transcoding the remote stream to a file, with just a few hours and a couple hundred lines of code.

What is the status of Farstream development currently and what are your plans going forward?

Youness Alaoui: Farstream is complete and fully functional. It supports conferencing with multiple participants and multiple streams with each participant (multi angle camera views for example). TFRC (TCP-Friendly Rate Control) support was recently added which allows for quality/resolution scaling depending on the available bandwidth, this allows for a superior calling experience. Farstream is being used in various Maemo devices from Nokia, notably the N800, N810, N900 and the recently released Meego phone: the Nokia N9, as well as being used in other platforms and by Empathy on the PC. One of the features that we plan on adding in the future is the support for SIP forking which will allow SIP calls to be dispatched to multiple devices at the same time, with a connection pre-established between all devices, and to enable data transfer with only the device which actually answered the call.

There seems to be a lot of movement in the VoIP codec space and a lot of it seems to be free software. With codecs such as Speex, Silk, CELT and Opus available. What is your opinion on efforts such as Opus and how do you see them impacting the wider audio conferencing technology space?

Youness Alaoui: I don't know much about codecs, I only work with technologies using these codecs, but I think people are starting to understand that there is something to gain by providing open standards. Most companies realize that they are wasting a lot of time and energy reinventing what others have done before them (and kept closed), and I guess they are happy when they encounter an open standard they can use, so they try to give back in the same way. Currently, most VoIP solutions are using open standards like SIP, XMPP, ICE, STUN and TURN. The only missing component is the codecs, so it's important to have open and royalty-free codecs to achieve this task from start to finish.

Video conferencing seems mostly dominated still by proprietary solutions, most notably Skype, but also most chat clients seems to do their own proprietary thing. Google seems to be trying to push video conferencing towards free technologies, with Google Talk being based around XMPP and the plans to push VP8 into that space. What are your thoughts about the developments in that space? Is a future where video conferencing is dominated by free and open technologies, akin to the web and HTML a real viability?

Youness Alaoui: I think that maintaining a proprietary protocol costs a lot of money for these companies. Most of them had developed their own protocol for a long time before XMPP existed, and they are sticking with it. However, the trend is currently shifting, Microsoft dropped their previous proprietary protocol for video conferencing a few year's ago and switched to SIP+ICE (and released documentation on what changes they had to made to these specifications), the Windows Live Messenger service also recently added interoperability with XMPP. Facebook's chat is also using XMPP, and also, the Playstation 3 uses XMPP for their PSN network contact list and chat messaging service. So the technologies are definitely shifting towards open standards and I think the advantage that Skype has is that it just works for everybody and they had a head start when compared to the competition, and when you have all your friends on their network, it's easier to keep using Skype. But now we have alternatives that are as reliable as Skype and soon enough people will realize it's better to switch to something that is more open, where the user actually has control over the technology he's using.

And on a related note, what do you think of WebRTC?

Youness Alaoui: I don't personally like web applications, I prefer to use desktop applications written in C rather than anything else. However, I do find the WebRTC specification very interesting as it opens up a lot of possibilities for web developers. I think its capabilities are also very good since it allows a high level of freedom in how you want to build your streaming pipeline. The fact that the WebRTC specification exists, which represents a recognized and standardized API for VoIP is actually quite an achievement and I'm very happy to see work being done in that area.

Will the upcoming GStreamer 1.0 release have any major impact on Farstream and Telepathy? Are you already looking into porting to the new API?

Youness Alaoui: I don't believe that any major issues would arise from the GStreamer 1.0 release. The users of the library won't have anything to change since the impact on application development will be minimal. Most of the API changes needed for the GStreamer 1.0 release are for plugins only, so the changes will have to happen inside Farstream itself.

Porting to the new API has already started and is actually finished and being actively maintained in parallel and will be released as soon as GStreamer 1.0 is available.

Another of your projects is libnice, the NAT traversal library, what is the state of that library and NAT traveral in general. Is it a solved problem or is there a lot of routers still making VoIP painful?

Youness Alaoui: Libnice is also a completed project and works very reliably. In the recent year, the ICE specification (which libnice uses for NAT traversal) has been standardized as RFC 5245, and the STUN and TURN (Relays) drafts have also received the same treatment by the IETF group. So in terms of the standard, libnice already implements those standards fully. Libnice also has compatibility modes so it can be used with Google Talk's own implementation of the specification, as well as with Microsoft's MSN Messenger, Windows Live Messenger and Microsoft Office Communicator's variants.

As for NAT traversal, it is working with a quite high success rate. The only use case that cannot be handled is if both peers are using a symmetric NAT router, which is not a common scenario. In those cases, a TURN relay can be used as a proxy that allows VoIP to still be successful and both libnice and Farstream have full support for TURN relaying.

One of the recent developments with libnice is the added support for a reliable mode which allows to emulate a TCP stream on top of UDP and allows libnice to be used in contexts other than VoIP, like for doing reliable file transfers in a peer to peer context.

Farstream and Telepathy was developed together and I think for a lot of people they are considered the same, can you please explain how the two libraries work and maybe give examples of usecases where one might want to use Farstream without Telepathy?

Youness Alaoui: Farstream and Telepathy are definitely not the same thing, although it's easy to get confused. Telepathy is an IM and VoIP framework in which multiple components must follow strict specifications in order to work and be compatible with each other. The Telepathy Connection Manager will take care of the signaling protocol and will let the UI know of the contact list, chat messages received and VoIP calls received or sent. Farstream is the framework being used by the UI in order to actually make the VoIP call work by building the pipeline for sending and receiving data from/to the peer.

So for example, the telepathy-sofiasip connection manager will receive a SIP call and will tell the application (UI) about it, then the application will create a new Farstream Conference and will proxy the sessions and streams and remote codecs and candidates to the Farstream object which will start capturing, encoding, decoding and displaying the actual media.

You can find out more about Farstream and Telepathy on the Collabora projects pages.