OBS Studio on Wayland

As of today, I’m happy to announce that all of the pull requests to make OBS Studio able to run as a native Wayland application, and capture monitors and windows on Wayland compositors, landed.

I’ve been blogging sparsely about my quest to make screencasting on Wayland a fluid and seamless experience for about a couple of years now. This required some work throughout the stack: from making Mutter able to hand DMA-BUF buffers to PipeWire; to improving the GTK desktop portal; to creating a plugin for OBS Studio; to fixing bugs in PipeWire; it was a considerable amount of work.

But I think none of it would matter if this feature is not easily accessible to everyone. The built-in screen recorder of GNOME Shell already works, but most importantly, we need to make sure applications are able to capture the screen properly. Sadly our hands are tied when it comes to proprietary apps, there’s just no way to contribute. But free and open source software allows us to do that! Fortunately, not only OBS Studio is distributed under GPL, it also is a pretty popular app with an active community. That’s why, instead of creating a fork or just maintaining a plugin, I decided to go the long hard route of proposing everything to OBS Studio itself.

The Road to Native Wayland

Making OBS Studio work on Wayland was a long road indeed, but fortunately other contributors attempted to do it before I did, and my pull requests were entirely based on their fantastic work. It took some time, but eventually the 3 big pull requests making OBS Studio able to run as a native Wayland application landed.

After that, the next step was teaching OBS Studio how to create textures from DMA-BUF information. I wrote about this in the past, but the tl;dr is that implementing a monitor or window capture using DMA-BUFs means we avoid copying buffers from GPU memory to RAM, which is usually the biggest bottleneck when capturing anything. Exchanging DMA-BUFs is essentially about passing a few ids (integers) around, which is evidently much faster than copying dozens of megabytes of image data per second.

Fortunately for us, this particular feature also landed, introducing a new function gs_texture_create_from_dmabuf() which enables creating a gs_texture_t from DMA-BUF information. Like many other DMA-BUF APIs, it is pretty verbose since it needs to handle multiplanar buffers, but I believe this API is able to handle pretty much anything related to DMA-BUFs. This API is being documented and will freeze and become stable soon, with the release of OBS Studio 27, so make sure to check if and see if there’s anything wrong with it!

This was the last missing piece of the puzzle to implement a Wayland-compatible capture.

PipeWire & Portals to the Rescue!

An interesting byproduct of the development of apps sandboxing mechanisms are portals. Portals are D-Bus interfaces that provide various functionalities at runtime. For example, the Document portal allows applications with limited access to the filesystem to ask the user to select a file; the user is then presented with a file chooser dialog managed by the host system, and after selecting, the application will have access to that specific file only, and nothing else.

The portal that we’re interested in here is the Desktop portal, which provides, among others, a screencasting interface. With this interface, sandboxed applications can ask users to select either a window or a monitor, and a video stream is returned if the user selects something. The insecure nature of X11 allows applications to completely bypass this interface, but naturally it doesn’t work Wayland. At most, Xwayland will give you an incomplete capture of some (or no) applications running through it.

It is important to notice that despite born and hosted under the Flatpak umbrella, portals are mostly independent of Flatpak. It is perfectly possible to use portals outside of a Flatpak sandbox, and even when running it as a Snap or an AppImage. It’s merely a bunch of D-Bus calls after all. Portals are also implemented by important Wayland desktops, such as GNOME, KDE, and wlroots, which should cover the majority of Wayland desktops out there.

Remember that I vaguely mentioned above that the screencast interface returns a video stream? This video stream is actually a PipeWire stream. PipeWire here is responsible for negotiating and exchanging buffers between the video producer (GNOME Shell, Plasma, etc) and the consumer (here, OBS Studio).

These mechanisms (portals, and PipeWire) were the basis of my obs-xdg-portal plugin, which was recently merged into OBS Studio itself as part of the built-in capture plugin! Fortunately, it landed just in time for the release of OBS Studio 27, which means soon everyone will be able to use OBS Studio on Wayland.

Meanwhile at Flatpakland…

While contributing with these Wayland-related features, I sidetracked a bit and did some digging on a Flatpak manifest for OBS Studio.

Thanks to the fantastic work by Bilal Elmoussaoui, there is a GitHub action that allows creating CI workflows that build Flatpaks using flatpak-builder. This allowed proposing a new workflow to OBS Studio’s CI that generates a Flatpak bundle. It is experimental for now, but as we progress towards feature parity between Flatpak and non-Flatpak, it’ll eventually reach a point where we can propose it to be a regular non-experimental workflow.

In addition to that, Flatpak greatly helps me as a development tool, specially when used with GNOME Builder. The Flatpak manifest of OBS Studio is automatically detected and used to build and run it. Running OBS Studio is literally an one-click action:

---

Next Steps

All these Wayland, PipeWire, and portals pull requests are only the first steps to make screencasting on Wayland better than on X11. There’s still a lot to do and fix, and contributions would be more than welcomed.

For a start, the way the screencast interface currently works doesn’t mix well with OBS Studio’s workflow. Each capture pops up a dialog to select a monitor or a window, and that’s not exactly a fantastic experience. If you have complex scenes with many window or screen captures, a swarm of dialogs will pop up. This is clearly not great UX, and improving this would be a good next step. Fortunately, portals are extensible enough to allow implementing a more suitable workflow that e.g. saves and restores the previous sessions.

Since this was tested in a relatively small number of hardware setups and environments, I’m sure we’ll need a round of bugfixes once people start using this code more heavily.

There’s also plenty of room for improvements on the Flatpak front. My long-term goal is to make OBS Studio’s CI publish stable releases to Flathub directly, and unstable releases to Flathub Beta, all automatically with flat-manager. This will require fixing some problems with the Flatpak package, such as the obs-browser plugin not working inside a sandbox (it uses CEF, Chromium Embedded Framework, which apparently doesn’t enjoy the sandbox PID remapping) nor on Wayland (Chromium barely supports Wayland for now).

Of course, I have no authority over what’s going to be accepted by the OBS Studio community, but these goals seem not to be controversial there, and might be great ways to further improve the state of screencasting on Wayland.

I’d like to thank everyone who has been involved in this effort; from the dozens of contributors that tested the Wayland PRs, to the OBS Studio community that’s been reviewing all this code and often helping me, to the rest of the Flatpak and GNOME communities that built the tooling that I’ve been using to improve OBS Studio.