Blog

GNOME 3.32 and other ramblings

GNOME 3.32 was released this week. For all intents and purposes, it is a fantastic release, and I am already using the packages provided by Arch Linux’s gnome-unstable repository. Congratulations for everyone involved (including the Arch team for the super quick packaging!)

I have a few highlights, comments and thoughts I would like to share about this release, and since I own this blog, well, let me do it publicly! 🙂

Fast, Furiously Fast

The most promoted improvement in this release is the improved performance. Having worked or reviewed some these improvements myself, I found it a bit weird that some people were reporting enormous changes on performance. Of course, you should notice that GNOME Shell is smoother, and applications as well (when the compositor reliably sends frame ticks to applications, they also draw on time, and feel smoother as well.)

But people were telling me that these changes were game changing.

There is a grey line between the actual improvements, and people just happy and overly excited about it. And I thought the latter was the case.

But then I installed the non-debug packages from Arch repositories and this is actually a game changer release. I probably got used to using Mutter and GNOME Shell manually compiled with all the debug and development junk, and didn’t really notice how better it became.

Better GNOME Music

Sweet, sweet GNOME Music

One of the applications that I enjoy the most in the GNOME core apps ecosytem is GNOME Music. In the past, I have worked on landing various performance improvements on it. Unfortunately, my contributions ceased last year, but I have been following the development of this pretty little app closely

A lot of effort was put into modernizing GNOME Music, and it is absolutely paying off. It is more stable, better, and I believe it has reached the point where adding new features won’t drive contributors insane.

GNOME Web – a gem

In the past, I have tried making Web my main browser. Unfortunately, that did not work out very well, due to 2 big reasons:

  • WordPress would not work, and as such, I couldn’t write blog posts using Web;
  • Google Drive (and a few other Google websites) would be subtly broken.

Both issues seem to be fixed now! In fact, as you can see from the previous screenshot, I am writing this post from Web. Which makes me super happy.

Even though I cannot use it 100% of the time (mainly due to online banking and Google Meets), I will experiment making it my main browser for a few weeks and see how it goes.

GNOME Terminal + Headebars = 💖

Do I even need to say something?

Hackfests

As I write this, I am getting ready for next week’s Parental Controls & Metered Data Hackfest in London. We will discuss and try to land in GNOME some downstream features available at Endless OS.

I’m also mentally preparing for the Content Apps Hackfest. And GUADEC. It is somewhat hard once you realize you have travel anxiety, and every week before traveling is a psychological war.

Other Thoughts

This was a peculiar release to me.

This is actually the first release where I spent serious time on Mutter and GNOME Shell. As I said in the past, it’s a new passion of mine. Both are complex projects that encompasses many aspects of the user experience, and cleaning the code and improving it has been fantastic so far. As such, it was and still is a challenge to split my time in such a fragmented way (it’s not like I don’t maintain GNOME Settings, GNOME Calendar, and GNOME To Do already.)

Besides that, I am close to finishing moving to a new home! This is an ongoing process, slow and steady, it is becoming something I am growing to love and feel like home.

Advertisements

GNOME Settings: more GNOME, more settings

Before deep diving into the more extensive architectural changes that I’ve been working on GNOME Shell and Mutter, let’s take a moment to highlight the latest changes to GNOME Settings.

Being the (co)maintainer of Settings for a full year now, the development pace has been great so far. I would go as far as to say that the project is healthy and sustainable now. The shared maintainership model that we adopted allows us to decrease the review time, and yet make sure that every single contribution is reviewed by at least one maintainer.

Looking at the numbers, we managed to review 110 merge requests targeting 3.32 (and more if we consider the ones targeting 3.30 as well!). That is almost one merge request reviewed and merged every single day. Considering that this is mostly composed of volunteer work, I am comfortable to say that we found a very efficient maintainership model.

Without further ado, let’s see what happened during this cycle.

New Panels

Applications

Mockups for controlling application settings were around and being discussed for some time now, but it eventually came the day where they were to be implemented. And thanks to the fantastic work by Matthias Clasen, we now have a new Applications panel:

Applications panel
The new Applications panel showing information and settings of a Flatpak-based application.

Flatpak-based applications naturally have more system integration points. There are ongoing ideas being discussed about which other integration points should be, but nothing settled so far.

Applications panel for a non-Flatpak application
Non-Flatpak applications don’t have as many controllable integration points.

There are more immediate improvements that will land before GNOME 3.32 release, but it’s a great addition already.

Sound

Robert Ancell has been working on the Sound panel redesign for some time now, and it’s close to landing. This is how it looks like so far:

Redesigned Sound panel
Redesigned Sound panel with a vertical layout and better organization of the options.

Thanks to the awesome work of Outreachy intern Clarissa Borges, we have user testing results of this new layout. And they look pretty damn good! Overall, the testing clearly shows how much of an improvement the redesigned panel is.

Under the Hood

Display

The Display panel is one of the hardest ones to deal with. Mostly because the almost entirety of the UI is programatically done. This was incredibly annoying, since it is somewhat hard to replace bits of the code with template widget without accidentally changing a lot of code.

Thanks to Benjamin Berg, however, this is not a problem anymore: the Display panel now uses modern best practices and is composed of smaller widgets. A new monitor scale widget is also on the way, although it potentially can be postponed to GNOME 3.34.

Responsive Panels

Purism is a great upstream player in GNOME, and so far demonstrated deep understanding on how upstream communitites work. Naturally, I had the chance to review and eventually land some fantastic working making GNOME Settings responsive:

Responsive GNOME Settings
Responsive GNOME Settings.

More to Come

Thanks to the hard work of these and many other awesome contributors, GNOME Settings is improving the way users can control their systems. But these are not the only improvements that will be part of 3.32, and of course, there is much more being targeted to 3.34!

GNOME Shell and Mutter: better, faster, cleaner

The very first update in the series is about GNOME Shell and Mutter. I’ve been increasingly involved with the development of those two core components of GNOME, and recently this has been the focus of my development time.

Fortunately, Endless allows me to use part of my work time to improve it. Naturally, I prioritize my upstream work considering what will impact Endless OS the most. So far, that lead to a series of very nice improvements to Mutter and GNOME Shell.

GNOME Shell

Most of my work time dedicated to GNOME Shell was oriented to performance and cleanup. At Endless, we have a modified GNOME Shell that constantly needs to be rebased. Since I’m taking care of these rebases now, it makes sense for me to also make myself familiar with the vanilla GNOME Shell codebase.

ShellGenericContainer

I’ll start with the work that makes me the proudest: removing the Shell.GenericContainer class.

First, a bit of history.

There was a time when GJS, the JavaScript engine that GNOME Shell is based on, did not support subclassing GObjects and overriding virtual functions. We could only instantiate GObject-based classes, and subclass them, all thanks to GObject-Introspection, but not override their virtual functions. This made, for example, implementing ClutterContent in JavaScript impossible.

For that reason, GNOME Shell developers created ShellGenericContainer: an actor that sends signals for various virtual functions. Because GJS supports signals, that worked well.

There are a few problems with that approach though:

  • Signals are slow, and should not be used on hot paths like layouting or rendering;
  • Going in and out of JavaScript territory is expensive;
  • It makes the JavaScript code slightly more complicated;

Thanks to the fantastic work by Jasper St. Pierre, GJS now supports overriding virtual functions. And that made Shell.GenericContainer obsolete. So I spent quite some time untangling it from GNOME Shell, and results were positive:

https://gitlab.gnome.org/GNOME/gnome-shell/uploads/b61233544ada29773b87de08e48beeb8/ShellGenericContainer.png
In general, running GNOME Shell without Shell.GenericContainer (blue line) led to more stable framerates compared to the current state (red line).

This is now merged and will be available with GNOME Shell 3.32, to be released on March 2019.

Improvements to the texture cache

After various investigations, another potential improvement that showed up was on StTextureCache. Textures (icons, image files, etc) are cached in GNOME Shell by StTextureCache, and that happened by keeping a ClutterTexture object alive.

That turned out to be a problem.

ClutterTexture is deprecated. Clutter has a new interface for drawing the contents of an actor: ClutterContent. It does not necessarily make the code faster, but it allows different actors to share a single ClutterContent without having to override ClutterActor.paint(). In other words, it is a nice and sane abstraction layer to control what an actor is drawing.

So I went ahead and wiped out ClutterTexture from StTextureCache. Then wiped it out entirely from GNOME Shell.

Unexpectedly, it made a small but noticeable difference! Icons are now slightly faster to load, but the most visible impact was in the startup animation.

Mutter

I did not know how fun and exciting compositors could be. It definitely is a new passion of mine, working on Mutter! So much has happened that it’ll be hard to summarize.

Goodbye, Autotools

During last year’s GUADEC, Jonas Ã…dahl worked on a Meson port of Mutter. After a series of reviews, and a few follow-up fixes, it reached almost complete feature parity with Autotools – the only exception being installed tests.

So I went ahead and added installed tests to the Meson build too.

And also removed Autotools.

Naturally, builds are much faster now. Saving us a few minutes per day.

Wayland vs X11

Another area that was interesting to work on was untangling X11-specific code from Wayland, and vice-versa. There are a handful of developers working on that already, and I had my fair share in better splitting X11 and Wayland code paths in Mutter.

Specifically, I worked on splitting X11-specific code from MetaWindowActor into subclasses. Mutter already handles different surfaces correctly; on X11 sessions, all surfaces are MetaSurfaceActorX11, and under Wayland, MetaSurfaceActorWayland.

MetaWindowActor has now the same split: Wayland windows have a MetaWindowActorWayland associated, while X11 windows have MetaWindowActorX11.

Interestingly, XWayland windows are X11 windows with a Wayland surface. You can check that using GNOME Shell’s Looking Glass:

wayland vs x11.gif
Example of a Xwayland window; it has a MetaSurfaceActorWayland surface, and a MetaWindowActorX11 actor associated.

There’s a lot more happening in this front, but I’ll spare the words for now. You’ll hear more about it in the future (and not necessarily from me).

CPU-side picking

More recently, I’ve been experimenting with the Cogl journal and ironing out a few bugs that are preventing a completely CPU-side picking implementation.

Picking is the process to figure out which element is beneath the cursor. There are two big approaches: geometry-based, and color-based. On games, the latter is the usual approach: each object in the scene is drawn with a plain color, and the final image is read to find out the color beneath a point. Geometry-based picking is what browsers usually do, and it’s basically math around rectangles.

Clutter uses color-based picking, but has a nice feature around that: a journal that tracks drawing operations and, under some conditions, hits an optimized path and does geometry-based picking. This is interesting for Mutter and GNOME Shell because it avoids sending draw operations to the GPU unecessarily when picking, reducing resource usage.

Unfortunately, due to various bugs and implementation details, we do not hit this optimization, causing GPU commands to be issued when they could be avoided.

Figuring out these bugs is what I’ve been experimenting with lately.

 


 

There’s much more that happened, so I will probably do a part 2 of this article soon. But those are big points already, and the post is becoming lengthy.

Many of these experiments and investigations already landed, and will be available with GNOME 3.32. This is all valuable work that is partially sponsored by my employer, Endless, and I’m happy to keep working on it!

Sorry for the silence

It’s been more than 6 months that I do not write in this space. Honestly, that’s the longest period of silence I’ve ever had.

There are various factors that, combined, ended up causing this period of seclusion. Learning about new, massive projects (GNOME Shell, Mutter and GNOME Settings) is one of them. Moving to a new home and marriage is another.

After finishing Masters last year, I have spent the time working with Endless full-time, and dedicating myself to parallel projects, such as Japanese language, music, gardening, and martial arts. It’s being wonderful, but that somehow distracted me from communicating changes to the community.

And for that, my apologies.

I’ll publish a series of blog posts talking about various GNOME-related activities that I’ve been involved since the last update. Hopefully they will be interesting enough.

My Perspective on This Year’s GUADEC

Greetings GNOMEies

This year, I had the pleasure to attend GUADEC at Almeria, Spain. Lots of things happened, and I believe some of them are important to be shared with the greater community.

GUADEC

This year’s GUADEC happened in Almería, Spain. It turns out Almería is a lovely city! Small and safe, locals were friendly and I managed to find pretty good vegan food with my broken Spanish.

I was particularly happy whenever locals noticed my struggle with the language, and helped and taught me some handy words. This alone was worth the entire trip!

Getting there was slightly complicated: there were no direct flights, nor single-connection routes, to there. I ended up having to get a 4 connection route to there, and it was somewhat exhausting. Apparently other people also had troublesome journeys there.

The main accommodation and the main venue could have been closer, but commuting to there was not a problem whatsoever because the GUADEC team schedule a morning bus to there. A well handled situation, I must say — turns out, commuting with other GNOME folks sparked interesting discussions and we had some interesting ideas there. The downside is that, if anyone wanted the GNOME Project to die, we were basically in a single bus 😛

Talks

There were quite a few interesting talks this year. My personal highlights:

BoFs

To me, the BoFs were the best part of this year’s GUADEC. The number of things that happened, the hard talks we’ve made, they all were extremely valuable. I think I made a good selection of BoFs to attend, because the ones I attended were interesting and valuable. Decisions were made, discussions were held, and overall it was productive.

I was particularly involved in five major areas: GNOME Shell & Mutter, GJS, GTK, GNOME Settings, and GNOME To Do.

GNOME Shell & Mutter

A big cleanup was merged during GUADEC. This probably will mean small adaptations in extensions, but I don’t particularly think it’s groundbreaking.

At the second BoF day, me and Jonas Ã…dahl dived into the Remote Desktop on Wayland work to figure out a few bugs we were having. Fortunately, Pipewire devs were present and we figured out some deadlocks into the code. Jonas also gave a small lecture on how the KMS-based renderer of Wayland’s code path works (thanks!), and I feel I’m more educated in that somewhat complex part of the code.

As of today, Carlos Garnacho’s paint volume rework was merged too, after extensive months of testing. It was a high-impact work, and certainly reduces Mutter’s CPU usage on certain situations.

At the very last day, we talked about various ideas for further performance improvements and cleanups on Mutter and GNOME Shell.  I myself am on the last steps of working on one of these ideas, and will write about it later.

As I sidenote, I would like to add that I can only work on that because Endless is sponsoring me to do that. Because

banner-down

Exciting times for GNOME Shell ahead!

GJS

The git master GJS received a bunch of memory optimizations. In my very informal testing, I could measure a systematic 25~33% reduce in the memory usage of every GJS-based application (Maps, Polari and GNOME Shell). However, I can’t guarantee the precisions of these results. They’re just casual observations.

Unfortunately, this rework was making GNOME Shell crash immediately on startup. Philip Chimento tricked me into fixing that issue, and so this happened! I’m very happy with the result, and looks like it’ll be an exciting release for GJS too!

Thanks Philip for helping me deep dive into the code.

GTK

Matthias already wrote an excellent write-up about the GTK BoF, and I won’t duplicate it. Check his blog post if you want to learn more about what was discussed, and what was decided.

GNOME Settings

At last, a dedicate Settings BoF happened at the last day of the conference. It had a surprisingly higher number of attendees than what I was expecting! A few points on our agenda that were addressed:

  • Maintainership: GNOME Settings has a shared maintainership model with different levels of power. We’ll add all the maintainers to the DOAP file so that anyone knows who to ping when opening a merge request against GNOME Settings.
  • GitLab: we want to finish the move to GitLab, so we’ll do like other big modules and triage Bugzilla bugs before moving them to GitLab. With that, the GitLab migration will be over.
  • Offloading Services to Systemd: Iain Lane has been working on starting sessions with systemd, and that means that we’ll be able to drop a bunch of code from GNOME Settings Daemon.
  • Future Plans: we’ve spent a good portion of this cycle cleaning up code. Before the final stable release, we’ll need to do some extensive testing on GNOME Settings. A bit of help from tech enthusiasts would be fantastic!

We should all thank Robert Ancell for proposing and organizing this BoF. It was important to get together and make some decisions for once! Also, thanks Bastien for being present and elucidating our problems with historical context – it certainly wouldn’t be the same without you!

GNOME To Do

Besides these main tracks, me and Tobias could finally sit down and review GNOME To Do’s new layout. Delegating work to who knows best is a good technique:

Tobias' GNOME To Do mockups in my engineering notebook.
Tobias’ GNOME To Do mockups in my engineering notebook.

I was also excited to see GNOME To Do stickers there:

gnome-todo stickers
Sexy GNOME To Do stickers, a courtesy of Jakub

It’s fantastic to see how GNOME To Do is gaining momentum these days. I certainly did not expect it three years ago, when I bootstrapped it as a small app to help me deal with my Google Summer of Code project on Nautilus. It’s just getting out of control.

Epilogue

Even though I was reluctant to go, this GUADEC turned out to be an excellent and productive event. Thanks for all the organizers and volunteers that worked hard on making it happen – you all deserve a drink and a hug!

I was proudly sponsored by the GNOME Foundation.

Sponsored by the GNOME Foundation

Going to GUADEC

Another year, another GUADEC, and here I am crossing oceans to see my fellow GNOMEies. This time, it’s going to be particularly challenging: 32 hours of travel, 4 connections, no vegan meal available. I heard GNOME are resilient folk though, perhaps this is the proving?

I am proudly being sponsored by the GNOME Foundation.

Sponsored by the GNOME Foundation

See y’all there!

Improving the development experience for GNOME Settings

After Bastien and Rui announced that they were stepping down from maintainership of GNOME Settings, I went ahead and volunteered to take care of it. This was not a random act, though;  for quite some time, I’ve been adding and changing code all around. I was pretty involved in moving to the new layout, and with the help of other contributors, implemented the redesigns of various panels. Clearly, I have a technical debt to the app.

Adding to that, assuming maintainership over it also aligns well with the goals of my employer, Endless, and gives a nice upstream/downstream overlap. With that, I can spend a bigger chunk of my work time on upstream tasks. Moreover, it allows us to have a stronger relationship with the GNOME community — after all, it allows me to bridge the insights and knowledge we gain from our users to a wider community.

Turns out, GNOME Settings is a pretty popular app amongst our users. It is one of the top apps that our users run. Developing and improving it became an important part of our fixed set of tasks. Specially now that we’re on the verge of a big release and one exciting new feature (amongst many!) was added into Settings. I’ll write more about it soon, hold your horses! 🙂

Without further ado, let’s start by checking some facts about Settings.

Knowing Settings

  • It has 371 unique dependencies (give or take, depending on the distro);
  • As of today, there are more than 17,500 commits and 500 tags;
  • It has roughly 440,000 lines of code, effectively making it part of the huge apps of GNOME, together with Evolution, GNOME Builder and Mutter.
  • It has 24 panels, the latest one being Thunderbolt;
  • First commit dates back to February 10th 1998, 21:22:12

As you can see, this is really no regular GNOME app. It has an old code base, half a million lines of code, dozens of thousands of commits and went through a lot of contributors. During all this time, it has being serving as the central hub of hardware and modules setting management. This gives us the dimension of this app.

But life is not a bed of roses.

Problems

Of course, during this time, GNOME Settings aged, was rewritten at least three times, and more recently was ported to another build system, Meson. Maintaining an app of this size, with such a colossal amount of knowledge involved, is not easy.

As I learn and find my way around the code base, a few problems were found:

  • You need to have the development packages of everything to build it. This is obvious, but I don’t really enjoy messing up with my host system.
  • It is not trivial to debug Settings. There was only a rudimentary logging system around.
  • There are at least 4 different code styles being used, and none documented.
  • Tests were scattered all around the code base.

With that, the reader hopefully  has enough context to follow the rest of this article: the recent changes that make the development experience of GNOME Settings smoother and more streamlined.

Flatpak, Flakpat, Flaptak, Flatkap, Flapkat!

Flatpak is an application distribution module, and has high adoption from the GNOME community. Turns out, Flatpak also became an excellent development tool, much beyond my initial expectations. It allows me, as a developer, to have a finer control over the build and execution environment, and keeps my system clean and organized. It allows me to install multiple versions of a given app, and they’ll share every common file they have. It’s awesome.

Flatpak is also tightly integrated into GNOME Builder, and developing apps on Builder with Flatpak is a super smooth experience these days. Thanks to Christian Hergert, we have a prime quality IDE!

This led to the creation of a Flatpak manifest for GNOME Settings. Of course, GNOME Settings is not meant to run as a Flatpak by end users. The Flatpak integration is strictly for development purposes.

You'll be warned when running Settings with Flatpak.
You’ll be warned when running Settings with Flatpak.

Spotlights to the blueish “I am a development version” header bar:

Blue headerbars
Fancy header bar!

These details might seem small, but they transform the development experience from a burden to a breeze. One can literally clone GNOME Settings from GNOME Builder and just press the Run button – everything will happen automatically. Besides that, Flatpak integration means I can develop GNOME Settings in a locked host system, such as Endless OS or Fedora Atomic, without having to unlock it and install development packages.

This work was partially done in my work time. Thanks Endless for letting me improve my own workflow 😛

Faster CI

GNOME Settings gained rudimentary CI support just before the 3.28 release. It allows us to have fancy green icons on commits, like these:

CI in action
We’re in a good state, everything is building!

CI is important to make sure everything is always building and working fine. Merge requests are always built before being merged, and having a more automated contribution workflow benefits everyone.

But it was inefficient and bloated.

Thanks to the independent contributor Claudio André, the CI was completely revamped. The build process was shrinked, the build environment was standardized so we can reliably reproduce build failures, and the fat was trimmed. In fact, CI times were cut down from 17 minutes to 3 minutes in average.

Thanks Claudio for your contributions!

Tests, More and Better

Another important aspect of improving the development experience is adding more tests, that validate a larger portion of the code base.

Thanks to Red Hat’s Benjamin Berg, the tests in GNOME Settings were reorganized and improved. As a courtesy, more network tests were also added. These tests are integrated into the CI, and every contribution is tested in a job:

CI in Gitlab
The tests run on every commit, and every merge request.

For obvious reasons, tests are absolutely important to avoid regressions. Unfortunately, though, the tests available in GNOME Settings don’t cover even a tiny fraction of the code base. Writting tests is a great way to start contributing to GNOME Settings!

Thanks Benjamin for reorganizing the tests!

Documentation

In addition to all this mentioned work, I personally have been working on improving the developer documentation available in GNOME Settings. As first steps in that direction, the current code style was documented and contribution guidelines were added.

The goal here is that people unfamiliar with the code base can find answers about code style by their own, reducing the communication overload and making the whole contribution process more efficient. It is also important to clearly document what is the expected behavior of contributors.

Improving the documentation is another wonderful and easy way to contribute.

But That’s not all

Even though these advancements mark a real improvement over the current state, there is a long way ahead. Ideally, we want to automate everything that can be automated, and test everything that should be tested. Settings needs and deserves to have some fresh air and code!

A unmerged feature that Claudio worked on is generating development Flatpak bundles. This is useful to lower the testing barrier, since one wouldn’t need to build Settings AND all the dependencies whatsoever. Just download, double-click the file, install and run. Dead easy.

If you have ideas on how the DX can be improved, feel free to share them! I’d love to hear your comments, suggestions, and ideas.

Acknowledgements

I’d like to thank my employer, Endless, for letting me use part of my work time to develop what I developed. I’d also like to thank Red Hat’s Benjamin Berg for reworking the test suite; independent contributor Claudio André, for making the CI smooth and fast; and specially Bastien Nocera, for helping and advising and reviewing code even after stepping down from the maintainer position.

GNOME Calendar 3.28.2 released

Another quick PSA:

GNOME Calendar 3.28.2 was released yesterday.

If you were having crashes and problems, please upgrade immediately. Quite a few crashers were fixed, and a few polishes went in too. Hopefully the experience of using Calendar will be much more pleasant now.

This new version was available in Flathub just half an hour after yesterday’s official announcement. Flathub is the only distribution mechanism that I officially support – if you rely on distro packages, please ask your distro maintainers to update.

That’s all for now.

(PSA) GLib can now canonicalize file paths

Quick announcement: if you have a relative file path, and want to resolve it against another (absolute) path, GLib can do that now.

An example in C:

g_autofree gchar *path = NULL;
g_autofree gchar *another_path = NULL;

path = g_canonicalize_filename ("../../usr/bin", "/etc/foo");
another_path = g_canonicalize_filename ("../../usr/bin", NULL);

g_print ("%s \n", path); /* Result: "/usr/bin" */
g_print ("%s \n", another_path); /* Passing NULL uses the current working dir */

No I/O is involved. It handles double and triple slashes nicely too.

This is bug 111848. The bug was fixed exactly 15 years and 1 day after initially reported.

We wish you farewell, bug 111848.

(Thanks Philip Withnall for reviewing, and Endless for sponsoring this work. This bug was fixed entirely in my work time.)

The Infamous GNOME Shell Memory Leak

Memory graph

Greetings GNOMErs,

at this point, I think it’s safe to assume that many of you already heard of a memory leak that was plaguing GNOME Shell. Well, as of yesterday, the two GitLab’s MRs that help fixing that issue were merged, and will be available in the next GNOME version. The fixes are being considered for backporting to GNOME 3.28 – after making sure they work as expected and don’t break your computer.

First, I’d like to thank the GJS maintainer, Philip C., for all the hand-holding, the reviews, and the incredibly insightful discussions we had. Secondly, to my employer, Endless, for the support they gave me to fix this issue. And last but not least, to the Ubuntu folks, which made a public call for testing with the changes – this will give us confidence that the fix is working, and that backporting it will be a relatively safe and smooth process.

banner-down
As always, great new features and fixes are a courtesy of Endless

I’m writing this blog post with three goals in mind:

  1. Explain in greater details what is the issue (or at least, what we think it is), the journey to find it, and how it was fixed.
  2. Give more exposure to important extra work from other contributors that absolutely deserve more credits.
  3. Expose a social issue that showed up during this time, and open a discussion about it.

Memory Leak

To me, it all started when I saw GitLab’s ticket #64 passing by in the IRC channels. It was challenging enough, I was curious to dig into GNOME Shell/Mutter/GJS internals, perfect match. Of course, when you’re not familiar with a given codebase, the first step to fixing a bug is being able to reproduce it, so I started to play around with GNOME Shell to see if I could find a reliable way to reproduce it.

Well, I found a way and wrote a very simple observation: running animations (showing and hiding the Overview, switching applications using Alt+Tab, etc) was reliably increasing memory usage. Then a few people came in, and dropped bits of useful information here and there. But at this point, it was still pointing to a wide range of directions, and definitely there was not actionable task there. This is when OMG! Ubuntu first wrote about it.

Carlos Garnacho then came in and wrote a pretty relevant comment with important information. It was specially insightful because he put numbers on the guts of GNOME Shell. His comment was the first real solid step to uncover what was going on.

A week passed, and I experimented different toys tools in order to have a better understanding of memory management inside GNOME Shell. This is the kind of tedious work that nobody talks about, but I learned tons of new stuff, so in the end it was worth the hassle. I even wrote about my crazy experiments, and the results of this long week are documented in a long comment in GNOME/gnome-shell#64. I kept experimenting until I reached heapgraph, an interesting tool that allowed generating the following picture:

Memory graph
Notice the sudden drops of memory at x=42 and x=71

Well, as stated in the comment, GJS’ garbage collect was indeed collecting memory when triggered. Problem is, it wasn’t being triggered at all. That was the leading clue to one of the problems that was going on. One idea came to my mind, then, and I decided to investigate it further.

A Simple Example

Consider that we have a few objects in memory, and they have parent/child relationships:

Example 1
The root object is “1”

Lets suppose that we decided that we don’t need the root object anymore, so we drop a reference to it, and it is marked for garbage collection.

Example 2
The root object is now marked for garbage collection

If we destroy the root object, we would need to destroy the other objects related to it, and go destroying everyone that depended, directly or indirectly, on the root object. Traditionally, JavaScript objects track who they own, so the garbage collector can clean up every other dependent object. Here’s the problem: C objects don’t track who owns them; instead, they only track how many owners they have. This is the traditional reference counting mechanism, and it works fine in C land because C is not garbage collected. To the garbage collector, however, the C objects would look like this:

Example 3
The garbage collector has no means to know the relationships between C objects.

The garbage collector, then, will go there and destroy the root one. This object will be finalized, and the directly dependent objects will be marked for garbage collection.

Example 4
Only the directly dependent objects are marked for the next garbage collection.

But… when will the next GC happen? Who knows! Can be now, can be in 10 minutes, or tomorrow morning! And that was the biggest offender to the memory leak – objects were piling up to be garbage collected, and these objects had child objects that would only be collected after, and so it goes. In other words, this is not really a memory leak – the memory is not being lost. I’d label it as a “misbehavior” instead.

The Solution

While people might think this was somehow solved, the patches that were merged does not fix that in the way it should be fixed. The “solution” is basically throwing a grenade to kill ants. We now queue a garbage collection every time an object is marked for destruction. So every single time an object becomes red, as in the example, we queue a GC. This is, of course, a very aggressive solution.

But it is not all bad. Some early tests shows that this has a small impact on performance – at least, it’s much smaller than what we were expecting. A very convincing explanation is that the higher frequency of GCs is reducing the number of things that are being destroyed each GC. So now we have smaller and more frequent garbage collections.

EDIT: Looks like people need more clarification here, since the comments about it are just plain wrong. I’ll be technical, and precise – if you don’t understand, please do some research. The garbage collector is scheduled every time a GObject wrapped in GJS has its toggle reference gone from >1 to 1. And scheduled here means that a GC is injected into the mainloop as an idle callback, that will be executed when there’s nothing else to be executed in the mainloop. The absolute majority of the time, it means that only one GC will happen, even if hundreds of GObjects are disposed. I’ve spotted in the wild it happening twice. This fix is strictly specific to GObjects wrapped by GJS; all other kinds of memory management, such as strings and whatever else, aren’t affected by this fix. Together with this patch, an accompanying solution landed that reduces the number of objects with a toggle reference.

This obviously needs more testing on a wider ranger of hardwares, specially on lower ends. But, quite honestly, I’m personally sure that this apparently small performance penalty is compensated by the memory management gains.

Other Improvements

While the previous section covered my side of this history, there are a few other contributors that did a great job, and I think it would be unfair with them if their work was not properly highlighted.

Red Hat’s Carlos Garnacho published two merge requests for GJS that, in my testing, substantially improved the smoothness of GNOME Shell. The first one changes the underlying data structure of JS objects, which allows us to stop using an O(n) algorithm and starting an O(1) one. The second one is particularly interesting, and it yields the most noticeable improvements in my computer. Gross, it vastly reduces the number of temporary memory allocations. He also has a number of patches on Mutter and GNOME Shell.

Another prominent contributor regarding performance is Canonical’s Daniel van Vugt, which helped early testing the GJS patches, and is doing some deep surgeries in Mutter to make the rendering smoother.

And for every great contributor, there is a great reviewer too. It would be extremely unfair if those relevant people haven’t had their work valued by the community, so please, take a moment to appreciate their work. They deserve it.

Final Thoughts

At this point, hopefully the cautious reader will have at least a superficial knowledge on the problem, the solution, and other relevant work around the performance topic. Which is good – if I managed to communicate that well enough, by the time you finish reading this blog post, you’ll have more knowledge. And more knowledge is good.

You can stop here if you want nothing more than technical knowldedge.

Still around?

Well, I’d like to raise an interesting discussion about how people reacted to the memory leak news, and reflect upon that. By reading the repercussions of the news, I found it quite intriguing to read comments like these:

weird comment 1

Captura de tela de 2018-04-20 22-51-53

Captura de tela de 2018-04-20 22-52-17

As a regular contributor for the last few years, this kind of comment sound alien to me. These comments sound completely disconnected to the reality of the development process of GNOME. It completely misses the individuality of the people involved. Maybe because we all know each other, but it is just plain impossible to me to paint this whole community as “they”; “GNOME developers”; etc. To a deeper degree, it misses the nuances and the beauty of community-driven development, and each and every individual that make it happen.

To some degree, I think this is a symptom of users being completely disconnected to GNOME development itself.

It almost feels like there’s a wall between the community and the users of what this community produces. Which is weird. We are an open community, with open development, no barriers for new contributors – and yet, there is such a distance between the community of users and the community of developers/designers/outreachers/etc.

Is that a communication problem from our side? How can we bridge this gap? Well, do we want to bridge this gap? Is it healthy to reduce the communication bandwidth in order to increase focus, or would it be better to increase that and deal with the accompanying noise?

I would love to hear your opinions, comments and thoughts on this topic.