The Infamous GNOME Shell Memory Leak

Memory graph

Greetings GNOMErs,

at this point, I think it’s safe to assume that many of you already heard of a memory leak that was plaguing GNOME Shell. Well, as of yesterday, the two GitLab’s MRs that help fixing that issue were merged, and will be available in the next GNOME version. The fixes are being considered for backporting to GNOME 3.28 – after making sure they work as expected and don’t break your computer.

First, I’d like to thank the GJS maintainer, Philip C., for all the hand-holding, the reviews, and the incredibly insightful discussions we had. Secondly, to my employer, Endless, for the support they gave me to fix this issue. And last but not least, to the Ubuntu folks, which made a public call for testing with the changes – this will give us confidence that the fix is working, and that backporting it will be a relatively safe and smooth process.

banner-down
As always, great new features and fixes are a courtesy of Endless

I’m writing this blog post with three goals in mind:

  1. Explain in greater details what is the issue (or at least, what we think it is), the journey to find it, and how it was fixed.
  2. Give more exposure to important extra work from other contributors that absolutely deserve more credits.
  3. Expose a social issue that showed up during this time, and open a discussion about it.

Memory Leak

To me, it all started when I saw GitLab’s ticket #64 passing by in the IRC channels. It was challenging enough, I was curious to dig into GNOME Shell/Mutter/GJS internals, perfect match. Of course, when you’re not familiar with a given codebase, the first step to fixing a bug is being able to reproduce it, so I started to play around with GNOME Shell to see if I could find a reliable way to reproduce it.

Well, I found a way and wrote a very simple observation: running animations (showing and hiding the Overview, switching applications using Alt+Tab, etc) was reliably increasing memory usage. Then a few people came in, and dropped bits of useful information here and there. But at this point, it was still pointing to a wide range of directions, and definitely there was not actionable task there. This is when OMG! Ubuntu first wrote about it.

Carlos Garnacho then came in and wrote a pretty relevant comment with important information. It was specially insightful because he put numbers on the guts of GNOME Shell. His comment was the first real solid step to uncover what was going on.

A week passed, and I experimented different toys tools in order to have a better understanding of memory management inside GNOME Shell. This is the kind of tedious work that nobody talks about, but I learned tons of new stuff, so in the end it was worth the hassle. I even wrote about my crazy experiments, and the results of this long week are documented in a long comment in GNOME/gnome-shell#64. I kept experimenting until I reached heapgraph, an interesting tool that allowed generating the following picture:

Memory graph
Notice the sudden drops of memory at x=42 and x=71

Well, as stated in the comment, GJS’ garbage collect was indeed collecting memory when triggered. Problem is, it wasn’t being triggered at all. That was the leading clue to one of the problems that was going on. One idea came to my mind, then, and I decided to investigate it further.

A Simple Example

Consider that we have a few objects in memory, and they have parent/child relationships:

Example 1
The root object is “1”

Lets suppose that we decided that we don’t need the root object anymore, so we drop a reference to it, and it is marked for garbage collection.

Example 2
The root object is now marked for garbage collection

If we destroy the root object, we would need to destroy the other objects related to it, and go destroying everyone that depended, directly or indirectly, on the root object. Traditionally, JavaScript objects track who they own, so the garbage collector can clean up every other dependent object. Here’s the problem: C objects don’t track who owns them; instead, they only track how many owners they have. This is the traditional reference counting mechanism, and it works fine in C land because C is not garbage collected. To the garbage collector, however, the C objects would look like this:

Example 3
The garbage collector has no means to know the relationships between C objects.

The garbage collector, then, will go there and destroy the root one. This object will be finalized, and the directly dependent objects will be marked for garbage collection.

Example 4
Only the directly dependent objects are marked for the next garbage collection.

But… when will the next GC happen? Who knows! Can be now, can be in 10 minutes, or tomorrow morning! And that was the biggest offender to the memory leak – objects were piling up to be garbage collected, and these objects had child objects that would only be collected after, and so it goes. In other words, this is not really a memory leak – the memory is not being lost. I’d label it as a “misbehavior” instead.

The Solution

While people might think this was somehow solved, the patches that were merged does not fix that in the way it should be fixed. The “solution” is basically throwing a grenade to kill ants. We now queue a garbage collection every time an object is marked for destruction. So every single time an object becomes red, as in the example, we queue a GC. This is, of course, a very aggressive solution.

But it is not all bad. Some early tests shows that this has a small impact on performance – at least, it’s much smaller than what we were expecting. A very convincing explanation is that the higher frequency of GCs is reducing the number of things that are being destroyed each GC. So now we have smaller and more frequent garbage collections.

EDIT: Looks like people need more clarification here, since the comments about it are just plain wrong. I’ll be technical, and precise – if you don’t understand, please do some research. The garbage collector is scheduled every time a GObject wrapped in GJS has its toggle reference gone from >1 to 1. And scheduled here means that a GC is injected into the mainloop as an idle callback, that will be executed when there’s nothing else to be executed in the mainloop. The absolute majority of the time, it means that only one GC will happen, even if hundreds of GObjects are disposed. I’ve spotted in the wild it happening twice. This fix is strictly specific to GObjects wrapped by GJS; all other kinds of memory management, such as strings and whatever else, aren’t affected by this fix. Together with this patch, an accompanying solution landed that reduces the number of objects with a toggle reference.

This obviously needs more testing on a wider ranger of hardwares, specially on lower ends. But, quite honestly, I’m personally sure that this apparently small performance penalty is compensated by the memory management gains.

Other Improvements

While the previous section covered my side of this history, there are a few other contributors that did a great job, and I think it would be unfair with them if their work was not properly highlighted.

Red Hat’s Carlos Garnacho published two merge requests for GJS that, in my testing, substantially improved the smoothness of GNOME Shell. The first one changes the underlying data structure of JS objects, which allows us to stop using an O(n) algorithm and starting an O(1) one. The second one is particularly interesting, and it yields the most noticeable improvements in my computer. Gross, it vastly reduces the number of temporary memory allocations. He also has a number of patches on Mutter and GNOME Shell.

Another prominent contributor regarding performance is Canonical’s Daniel van Vugt, which helped early testing the GJS patches, and is doing some deep surgeries in Mutter to make the rendering smoother.

And for every great contributor, there is a great reviewer too. It would be extremely unfair if those relevant people haven’t had their work valued by the community, so please, take a moment to appreciate their work. They deserve it.

Final Thoughts

At this point, hopefully the cautious reader will have at least a superficial knowledge on the problem, the solution, and other relevant work around the performance topic. Which is good – if I managed to communicate that well enough, by the time you finish reading this blog post, you’ll have more knowledge. And more knowledge is good.

You can stop here if you want nothing more than technical knowldedge.

Still around?

Well, I’d like to raise an interesting discussion about how people reacted to the memory leak news, and reflect upon that. By reading the repercussions of the news, I found it quite intriguing to read comments like these:

weird comment 1

Captura de tela de 2018-04-20 22-51-53

Captura de tela de 2018-04-20 22-52-17

As a regular contributor for the last few years, this kind of comment sound alien to me. These comments sound completely disconnected to the reality of the development process of GNOME. It completely misses the individuality of the people involved. Maybe because we all know each other, but it is just plain impossible to me to paint this whole community as “they”; “GNOME developers”; etc. To a deeper degree, it misses the nuances and the beauty of community-driven development, and each and every individual that make it happen.

To some degree, I think this is a symptom of users being completely disconnected to GNOME development itself.

It almost feels like there’s a wall between the community and the users of what this community produces. Which is weird. We are an open community, with open development, no barriers for new contributors – and yet, there is such a distance between the community of users and the community of developers/designers/outreachers/etc.

Is that a communication problem from our side? How can we bridge this gap? Well, do we want to bridge this gap? Is it healthy to reduce the communication bandwidth in order to increase focus, or would it be better to increase that and deal with the accompanying noise?

I would love to hear your opinions, comments and thoughts on this topic.

Advertisements

123 thoughts on “The Infamous GNOME Shell Memory Leak

  1. no barriers for new contributors

    Are there really none? Here are some of the barriers that I think exist:

    Having to learn to program in the first place. I mean it’s not like the concept behind programming is hard, but so far I haven’t found a PEOPLE-true programming language that just gets out of the way and lets you solve problems, not even Python fits the bill.
    Learning all the supporting tools, especially google: git usability.
    Hours of build time before you are even able to work on core GNOME components (BuildStream even seems to compile GCC from scratch).
    Long iteration times, for example in my experience any change in GNOME Shell requires re-packing sources into C and re-linking.

    Liked by 1 person

    1. I mean it’s not like the concept behind programming is hard,

      The conceptS behind programming are hard.

      but so far I haven’t found a PEOPLE-true programming language that just gets out of the way and lets you solve problems,

      Oh, nice, a language that “just lets you solve problems”.

      Why has humanity spent years devising whole families of languages that are a mix of different tradeoffs when they could have come up with the Perfect Language for everything.

      One that just Lets You Solve Problems.

      Well, actually they have, it’s called Haskell, but people often complain that they really really need to have mutable shit to Solve Problems.

      not even Python fits the bill.

      Python is a problem.

      Like

  2. I know I had no clue what was going on with Gnome with the old bug tracking system. I still have an open bug on there, but it took me quite a while just to figure out where to post it and such. So I think the move to Gitlab will help.
    That said, I didn’t realize there were already fixes coming out with the next version of Gnome, except for the fact that I happened to check planet Gnome and see this blog.

    What would be nice is keeping the Gitlab issue up to date. Hashicorp, for example, does a good job with that if you look at some Terraform issues (I don’t seem to be able to post a link via comment right now).

    Obviously there’s a difference between Hashicorp (with the emphasis on corp) and Gnome as an organization, but for me something simple like that would be helpful.

    Like

  3. First, I would like to thank all of you who helped mitigating this nasty bug.

    Then I would like to ask a question. I didn’t understand how this problem would have to be solved? For now we have a hacky patch which does its work, but if I understood correctly we need a more robust and elegant solution.

    I believe many people thinks GNOME is developed by a skilled team, like the one at Canonical or Red Hat, who has a precise development plan and a precise direction. The move to GitLab will certainly be helpful though.

    Like

    1. It is not every time a deallocation happens. Technically, a GC is queued when a toggle reference goes from >1 to 1. This happens only on wrapped GObjects, and they’re far a few between each other. The GC is injected in the idle loop, and depending on the load, it might take a while to execute. In my testing, it’s actually pretty infrequent – only enough to control the memory of your system.

      Liked by 1 person

  4. I’m not familiar with the code but I think there might be a flaw in the code design if there can be non-used objects left when the GC ends execution. It should run until there’s none. I guess this is obvious.

    There’s some people against Gnome 3 because they didn’t like the UI changes. This fueled animosity. I think some people simply don’t like change and want everything tailored to them. I think Gnome had a problem at first, when there was no extensions ecosystem, now the problem is on them.

    I’m just Gnome, no other involvement. I like it for its simplicity and extensibility. I like to extend my comfort zone. It’s about our mindset.

    Like

    1. I’m not familiar with the code but I think there might be a flaw in the code design if there can be non-used objects left when the GC ends execution. It should run until there’s none. I guess this is obvious.

      There’s some people against Gnome 3 because they didn’t like the UI changes. This fueled animosity. I think some people simply don’t like change and want everything tailored to them. I think Gnome had a problem at first, when there was no extensions ecosystem, now the problem is on them.

      I’m just a Gnome user, no other involvement. I like it for its simplicity and extensibility. I like to extend my comfort zone. It’s about our mindset.

      Like

  5. I think the bug itself was a convenient talking point for people who don’t like Gnome, which seems to be a fashionable viewpoint on communities like /r/linux. The fact that it’s a bug in a Javascript engine played very nicely for people who want to say that Gnome developers aren’t up to something as serious as a desktop (because real programmers use C!).

    I think this is a backlash to certain design decisions Gnome developers have made. Particularly when these decisions are to remove things like files on the desktop, or status icons. Perhaps there could be better communication of why Gnome developers think these kind of controversial changes are worth doing, and how you envisage users adapting?

    Maybe it’s also worth trying to engage more on platforms like reddit. E.g. this article is at the top of /r/linux now, and half the comments are either saying how terrible your fix is, or that the whole idea of using Javascript for a desktop is terrible. I’m sure you’ve got answers to points like these!

    Liked by 1 person

    1. I used to like Gnome 3, but you get tired of it. It’s difficult to find new themes, customize things, etc. There is this absolutly stupid concept that all users should do things as the developers/designers intended and think that’s best for them and the design. I think that as a user, you want to have the option to do things differently, because sometimes it’s just about taste… and efficiency, of course. We all have different minds. You don’t want to waste time looking for and installing extensions and this or that for making things that on other Desktop Enviroments are built-in, and for good reason. From the beginning, i thought that i liked it, but if they continued on that path, Gnome would fall really hard. I was wrong, but that doesn’t change the fact that the potential that Gnome had to make something really good for the linux desktop, has faded away, and now is just a mediocre DE.

      Like

      1. but you get tired of it.

        Well, it’s not a toy, it’s a DE.

        It’s difficult to find new themes, customize things, etc.

        Because themes are a key part of productivity.

        There is this absolutly stupid concept that all users should do things as the developers/designers intended and think that’s best for them and the design.

        That’s a big fat lie. The source is there, if you want to hardcode Comic Sans everywhere you can.

        The rest of us will rather have an environment with some consistency.

        Like

    2. Oh, and just to add to the fire, every now and then they make a new release that breaks compatibility with extensions and other software. It’s just all a big mess…sadly

      Like

    3. I think this is a backlash to certain design decisions Gnome developers have made.

      Of course you’ll see a backslash against anything remotely reasonable from the toxic shitpool that is /r/linux.

      After all, none of those people give a shit about using a computer, they mostly care about Stallman memes and not getting laid.

      Long story short, those people use a Linux distro not because their company doesn’t issue them a Mac and they want a Unix shell, but because they enjoy pain and hate comfort and productivity.

      Particularly when these decisions are to remove things like files on the desktop,

      That’s controversial now?

      or status icons.

      GNOME did not “remove” status icons, they’re still there. GNOME 3.26 removes the legacy tray. Developers have had the time to update their junk but didn’t? Well, their fault.

      Maybe it’s also worth trying to engage more on platforms like reddit.

      I fail to see how it would be a good idea to feed oneself to trolls.

      E.g. this article is at the top of /r/linux now

      Really? Good to know, /r/linux, please die in a fire if you’re reading.

      Liked by 1 person

  6. User/developer disconnection has been going on for years, but I think much is self inflicted. There was a lot of negative feedback to the removal of various customization tools and functionality that was ignored in favor of a unified look and branding that seemed, from the outside, pushed by designers and others who wanted gnome to look the same everywhere regardless if the cost. The tweak tool, many extensions, and having to dig into the dconf editor replaced behavior that used to be easily customizable, the tweak tool itself is no longer an additional item for power users as it’s installed by default by most distributions. Systemd as a potential or real dependency is another controversial area.

    I don’t think Gnome developers are highly trusted by the community, there is a perception that they are going to do what they want regardless of what anyone else (their users) thinks.

    Liked by 1 person

    1. This kind of sums up my own feelings. I remember in around 2006-8 I already knew about this, but it came to bite me when a feature I thought made sense at the time was removed, + a similar thing later on with the reaction to the removal of transparent backgrounds in gnome-terminal.

      Both times, after explanation of why people might use the features, there seemed to be a real attitude of “deal with it”.

      During the second feature, it seemed like the developers were really burned out from users saying they didn’t like things so just didn’t want to listen.

      It’s tricky as there aren’t many developers so of course people that actually do stuff get priority. It’s weird because GNOME does a lot of things to be user-friendly, but the experience of users dealing with some developers can seem user-hostile.

      This is all quite dark – but I’d like to say I’ve been using GNOME since 1997, and still am, the ideals hold up well.

      Like

  7. Just for my two cents, I love gnome and the consistent, beautiful user-friendly experience that it offers. It is the most user-friendly of DEs by far imo. So just want to do thanks for a fantastic DE 🙂

    Liked by 1 person

  8. “We are an open community, with open development, no barriers for new contributors”

    You are nowhere close to open. You don’t listen users opinion (as with the infamous Gnome 2->3 change), you close bug reports and mark them as ‘wont fix’ notoriously. Once I personally volunteered to implement some feature, which was requested several times, and got a response that “I can do that, but there is no way I would merge it” from some very important maintainer™.

    Openness of GNOME is a PR bullshit. This is one of the most closed to outsiders open source project I know.

    Liked by 1 person

    1. You clearly confuse “open” with “we’ll accept anything”. GNOME is open – you can talk to anyone, work on any module and fix any bug. GNOME is also principled – if your code isn’t good, or your feature isn’t desired by some maintainer, it won’t be accepted. If you know that something won’t be accepted, and yet you work on it, good luck.

      I also want to open the discussion about the “listen to users opinion” here. It’ll probably hurt you to hear that, but users are not the holy grail for many open source contributors. They are for companies that profit from users, but very unlikely to be for independent contributors. Most of us don’t profit from our work. You can demand features and bugfixes from open source contributors just as much as you can from any random people on the street.

      Liked by 1 person

      1. Yeah, sorry, but this is wrong headed. Let me tell you why.

        Open source devs ARE getting paid, just not with money. They’re getting paid with user time and attention to their software. To have the sort of QA exposure popular open source projects have would cost the typical proprietary software company hundreds of millions of USD. This is exactly why many corporations use open source software. Only a very few corporations can afford the massive developer AND user review and testing that goes into projects like Linux, FreeBSD, KDE, GTK+, OpenSSH, and the like.

        You can toss all that away if you want. Many open source projects do to their detriment. Or, you can embrace it and listen to users and realize that there’s a symbiosis going on between developer and user. Users know how to use. Developers know how to make. It takes both.

        Like

        1. User time and attention does not pay my bills, my food nor my retirement. I listen to every single user opinion that is thrown at me – I just don’t agree with everything. Contributing to FOSS did not take away my freedom to have opinions.

          See, I owe you and users of my FOSS work nothing except gratitude. Everything I do is open and distributed with the sincere hope that it’ll be useful to someone, but without any guarantees. I do give everyone the rights to read, study, modify and redistribute all of my work, as long as they give everyone the very same rights. That way, if the work I create does not fit your particular purposes, you are free to (i) suggest me a change, (ii) work on that change itself (and if I agree, I’ll take it and credit it to you), or (iii) fork my work and adapt it to fit your purposes.

          Liked by 4 people

          1. First of all, thank you for your hard work. I myself contribute to open source and I feel you a lot.
            Thank you for this wonderful post, it is expressed clearly and for your fix too! Amazing.

            Kind of comments like above just infuriates me. Users are demanding changes to be implemented in the way they want and they think that they’re “paying” developers by using their work.
            No, you don’t? Grow up.

            I think this is partly a problem induced by Silicon Valley. For big companies user count is very important so they try to get users by any means. Users are used to the thought that “I do you a favor by using your software”.
            No, you don’t.

            Another problem is super toxic culture of Reddit/OMGUbuntu. I personally blame OMGUbuntu editors. They made a huge deal of this bug.
            Many people don’t understand how development works. They’re not interested in Free Software. Under every (even unrelated) post there are comments like “why don’t they fix that leak instead”. It is a social problem but I’m not sure that it’s something that GNOME can solve. It’s something that others like OMGUbuntu should explain to people.

            I say, don’t listen to those people. Don’t read comments on those websites. It’s not constructive. These people won’t advocate for your software, won’t submit fixes, they will always demand and complain. GNOME has a mission and I think they’re really successful: DE is super polished and friendly, like nothing else.

            You don’t owe anyone anything. Don’t be a hostage. Do what you think is right.

            Thank you a lot.

            Liked by 1 person

        2. As an open source developer (not Gnome, other projects): no, you are not ‘paying’ in any sense with your attention. I’m happy if I can make something that people want to use, and if you want to help make it better, great. But if we disagree on which way is better, I don’t owe you anything just because you’ve deigned to use my code.

          We’re also aware that, especially for larger projects, the users who we hear from the most are the ones who want to complain about things.

          Like

      2. So, basically, you just confirmed things said in the comment you are replying to.

        It is OK to be closed and doing your own thing, everybody is allowed to do their thing like they want to. The problem we is when you misrepresent it. I would also agree that gnome is a closed community. We can contra-argue that your code being online doesn’t make it an open project. Just the source code is open.

        Unfortunately for the userbase, canonical had decided to use gnome. And the reality is – it is perceived that it will bring good to gnome because they care about users.

        Reputation of gnome, to my understanding, is quite low. People are kinda forced to use it because it is most pushed by the major distros. Definitely it is not in this position of quality and openness.

        Just my two cents

        Like

        1. Your definition of closed community is, well, just yours, since a community when everything is done in the open and you can bring your opinion to litteraly everybody involved is AN OPEN COMMUNITY.
          As Georges said being open does not mean to be ok with everything and everybody, GNOME maintainers and designers follow their own ideas and direction, rightfully so, exactly like any other open source projects.
          Poeple are not forced at all to use gnome, c’mon, there are many alternatives and nobody forcefully install it on your computer, it’s true that many major distributions offers gnome by default, I believe that’s for GNOME merits though, but you’re free to think there’s a cospiracy or something along those lines…

          Like

          1. Yes, actually, me saying people are being forced is quite overstated. More to the point is – it’s being made more available by major players so less familiar people are first exposed to gnome. There is no conspiracy, there is market. Market is dominated and lead by red hat and they are backing gnome. Simple as that.

            They make it more and more difficult for others to use GTK (see Solus and the idea to go to QT; other examples exist as well). How is this in the spirit of open source?

            Don’t get me wrong, there are things i do like about gnome. E.g. the way they remove unneeded clutter (not sarcasm), because they think about what would be a good thing usability wise.
            Sometimes it makes sense, having a strong leadership and vision, no democracy problems like in KDE camp where they have a recepies app, but not an sql app like toad/mssql studio. But this goes waaaay beyond some reasonable limits and some times it is completely wrong. Icons on desktop? Tray icons? Rings a bell? How were these accepted by developers and how by users? Forcing these stupid activities (because it’s where the start menu would be) instead giving a simple use case with one desktop and start menu? Microsoft did the same thing, there was an outlash of users with 8 that they had to reintroduce it in 10.

            Thank god for choices.

            Like

      3. feaneron I respect what you are doing but you are venturing into dangerous territory. “users are not the holy grail for many open source contributors” Everything operates most effective as a market, believe it or not. If you are willing to not listen to the users complaints coming from the people who actually use your software, then your software will not be used by those users. I applaud you for contributing to an open source project and hope that open source is the direction tech is heading, but your attitude toward information that people are giving to you for free, something large companies pay millions of dollars for a year, is appalling. Never talk down on someone because you think you know better. Lessons are learned from everywhere, even from people who can’t code because they are probably more similar to the user of the software you are writing than you are. If you don’t want to hear complaints then don’t work on the project. I have a feeling you truly care about the project, because you are still putting effort into it. You probably have the talent to work a high paying job at a large tech company, but your values are different than theirs and that’s what really brings you to the open source community. We want everyone to have access to great software freely, something the community is grateful for because talented programmers like you spend their time providing that to us. The programmers and users live in a symbiotic manner, without one there does not exist the other and vice versa. Together we can build a user experience and platform that cannot be achieved by Apple, Google, nor Microsoft, but the keyword is together.

        Like

      4. These are your words:

        “…and yet, there is such a distance between the community of users and the community of developers/designers/outreachers/etc.

        Is that a communication problem from our side?”

        Here in this comment you have answered your own question.

        You are the DE that decides how users ought to work, no matter what they say about how they want to work. I have no problem with Gnome being this way, I’ve just used a different DE since the v2 to v3 debacle, but you shouldn’t then be surprised that the “distance” you talk about exists.

        Like

    2. You don’t listen users opinion (as with the infamous Gnome 2->3 change)

      Actually, they did. They chose to ignore it. They were right. Now we have GNOME 3, which is the best DE that is not a Mac.

      Essentially, you guys want this: http://theoatmeal.com/comics/design_hell

      A pool of code monkeys that will implement whatever inconsistent pile of shit you thousand internet idiots can collectively come up with.

      Well, instead the team is pushing some kind of design and will not throw in 40 ways to copypaste. Outrageous, isn’t it?

      Like

  9. All these comments about Gnome being a closed community because every single patch and/or suggestion are not implemented is nonsensical. If every users’ opinion and every developers merge request were accepted, Gnome would be a bloated, buggy, inconsistent mess. It is never possible to please everyone and a good open source project realizes this and acts in accordance with the principals and design philosophies they’ve defined for themselves.

    I’ve been using Ubuntu Gnome many years now, and it is hands down the most reliable and functional desktop environment. I’m estatic it is being used now by vanilla Ubuntu. Do I agree with every design decision? Nope. Thankfully their extension ecosystem allows me to customize my desktop per my tastes.

    Thank you fix forcing this memory management issue and for all the hard work of the contributors.

    Like

    1. Saying that it’s about every single patch/suggestion is a strawman.
      The specifics are wide-ranging but typically take the form of “how did this design ever make it through without someone noticing the obvious flaw” (said design is released and, eventually, removed), or some behavior is changed that, again, obviously shouldn’t have been, and it takes A LONG TIME to get it reverted (in the meantime it gets takes NOTABUG WONTFIX, which just shuts down discussion, and that is the norm because design decisions aren’t up for debate), or, “Z is missing a fundamental feature” (this one pops up in various guises, and is different from system level design changes), and “there’s been a big regression” (this also takes many different forms and includes performance, but also architecture and design decisions that make it essentially impossible for the user to do their work).

      Like

  10. Since you are on the inside you may be unaware of the apparent, and sometimes actual, insularity and intransigence of certain aspects of GNOME development. These tensions have existed for years, and the involved parties have changed, but there’s been pretty consistent theme and it tends to involve a few strongly opinionated folks.
    I will say that things have gotten better in the last few years.

    Like

  11. Γεια σου Γιώργη πατρωτη!
    About all these comments, it’s probably because Gnome turned into a user friendly, robust DE, and non-technical people wait for the community to work like an enterprise company, because you’re competing with such companies (Microsoft, Apple, etc.). It’s like a side-effect of being good at your own job – everyone waits the best from you, no matter the circumstances.

    Like

    1. Most of the code is produced by people being paid to work on GNOME. There’s a massively long tail of volunteers but they aren’t producing anywhere close to a majority of the code.

      Liked by 1 person

          1. I’m not sure if you’re a troll, but I’m definitely detecting an odor. Regardless, I’ll extract the relevant part:
            “people wait for the community to work like an enterprise company, ”

            The community is made up of lots of people but most code is created by relatively few, and those are almost entirely fully employed to work on GNOME.

            Like

          2. There are several gnome developers hired by a bunch of companies (with different priorities and agendas), that’s still a community which cannot be even remotely compared to a company as microsoft, apple or any other structured company.

            Like

  12. This reminds me of reading about pythons GC and how it finds cycles… it’s a long time since I read it, but I recommend reading it, as it seems relevant here.

    Like

  13. I just read the end of your post, about the disconnect between users and developers. Unfortunately this is true, and has been true for a long time. Apart from the obvious thing that a (small) amount of the developers can be quite toxic at times, there has always been a disconnect.

    There are lots of good, and very nice developers – but the way users are treated and seen, certainly seems strange from the outside.

    I love gnome despite this, but as a fan since the 0.9 days, it’s hard to deny there is nothing going on.

    I don’t know – there is also really good stuff going on too, the new developments in nautilus, builder and many other developments. I guess the brain is just wired to notice the exceptions to the good stuff more.

    Like

  14. Why are dependent objects merely marked for destruction and not destroyed immediately? If no object can access them then there shouldn’t be any problems, right?

    Like

  15. Thank you very much for the fix.

    Is there a way that you can fix the gnome copy paste issue when we copy large file/s to external/usb drive. It hangs in the last time and we have to wait till some time then in the background it copies the files.

    Like

  16. Thanks a lot for your work on Gnome and this issue.

    Regarding the disconnect between users I think you only need to look at the Gnome bugzilla and observe for how long and how many users have reported memory issues over the years and have been dismissed when clearly there is a fundamental issue with GJS and memory management. Here is just two links for reference https://bugzilla.gnome.org/show_bug.cgi?id=642652

    I think we would not have any fixes for this issue if it did not become public knowledge, at least for Gnome 3.

    I was personally following the bug report on GitLab for weeks and the real action only started when OMGUbuntu wrote an article about it.

    So as a Gnome user I ask myself: why should I bother reporting bugs when the chance of them being addressed is slim? I mean this fundamental issue was ignored for 7 years.

    Like

    1. I looked at the bug report you linked to and I can’t see anything about reports being dismissed or ignored. All I see is a lot of hard working developers trying to figure out where the leak is and how it can be reproduced. I can’t find a single rude comment there. Can you show a specific example? The simple answer to why it took this long time to fix is that some things are just really complicated, and some bugs are just really, really hard to track down.

      Liked by 1 person

  17. That was a good read, thanks.

    You all fine people hear should not forget that – if you do care about FOSS (free and open source software) – you stand on the same page.

    There is nothing wrong with having a discussion and making arguments pro and contra, if you do it in a supportive and positive way. If someone just does not like it, there are tons of other DEs to choose from.

    But still, the point I’m trying to make: We are the good guys, developers or users, if you care about FOSS, we are on the same page.

    Liked by 1 person

  18. I’ve been commenting very regularly on OMG! Ubuntu! (as Ads20000) for over five years and I think there’s a disconnect because people expect, e.g., for it to be possible to simply divert all GNOME developers onto one problem (that they think is the most important) and for that to directly correlate to a faster outcome for that problem. I also think people are ungrateful for their great free software and, instead of asking how they can help their free software, like to attack and waggle their fingers at it. To be fair, people do use the software in intensive production environments and expect it to work well so the anger is somewhat understandable, but more grace to developers and more of a willingness to help (or patience if they don’t have time to help) I think wouldn’t go amiss. I try my best to counter the (imho) wrong sentiment but I clearly didn’t do well enough this time.

    I often encourage people to report bugs if they come across particular problems. Some of the disconnect may arise because (obviously) we get people to report bugs with ubuntu-bug first, which files bugs in Launchpad for triage, and our Bug Squad is always lacking in numbers, so many bugs and questions related to GNOME get stuck in the Ubuntu community and never reach the GNOME community. There’s many thousands of GNOME related bugs in Launchpad that haven’t been forwarded (I’ve been working through some of the (many) Files ones and forwarding the relevant ones on but it takes time and I don’t have much time, the time I spend doing this is procrastinating on study!)

    Additionally, I don’t think people read Planet GNOME where many design decisions are made. People love to ferociously attack the minimalism of GNOME (the removal of features compared to GNOME 2, particularly in Files) and I doubt many of them have read the justifications for it (I haven’t!) because no-one posts links to the relevant articles explaining various decisions (if there are any)…then people think those slating GNOME are correct in their slating because they’re aren’t challenged and so the anti-GNOME sentiment snowballs (I think that might be what is happening, anyway). It’s been snowballing like that for years, so now there’s a lot of committed hate for GNOME around 😦

    I think Ubuntu are doing well with handling this sort of user outrage – they’re/we’re getting people more involved in the design process (which is messy because there needs to be consistent design but I think there’s something very powerful in harnessing outrage rather than ignoring it, if you can do it right). For example, a constant complaint in Ubuntu is the outdated theme, someone on Didier Roche’s blog suggested a theme contest (like the wallpaper contests that are held before every release) and Didier Roche responded by saying ‘excellent idea … you can contribute this way by leading it’ ( https://didrocks.fr/2017/09/04/ubuntu-gnome-shell-in-artful-day-9/#comment-3501701454 ), which was a risky move but has paid off excellently because tons of volunteered community time has been put into making the ‘Communitheme’ (not quite a theme contest, but the point is that the desire for a new theme was successfully harnessed; so much community activity on it: https://community.ubuntu.com/c/desktop/theme-refresh ).

    In fact, as you can see in the link, this comment came on a blog which was giving all of Didier’s reasoning for all the decisions that Ubuntu made with the new GNOME session. This, in itself, was an excellent move for the community (I don’t think Ubuntu had really used blogs in such a comprehensive fashion before) because whenever anyone objected to the decisions (many on OMG! objected because they wanted a 100% (Fedora-style) vanilla session, some objected because they wanted it to be even more Unity-like) I could just reference Didier’s blog post showing that they had in fact thought it all through and there were very good reasons for the decisions that were made, as well as point out the fact that it’s Didier’s decision to make and complaining doesn’t really help the project (unless it’s a problem or feature request that can be productively acted upon). This post was particularly comprehensive: https://didrocks.fr/2017/08/18/ubuntu-gnome-shell-in-artful-day-5/ Ubuntu also hosted a sprint where they invited community contributors too, which was cool: https://insights.ubuntu.com/2017/08/08/ubuntu-artful-desktop-fit-and-finish-sprint (I’m sure GNOME does that sort of thing though? I don’t know…)

    Hope these thoughts are helpful for GNOME! I’d suggest that GNOME social media shares more blogs on controversial decisions and perhaps promotes the ‘get involved’ stuff more (if it doesn’t already, I’m sure it does), suggesting people to report bugs with GNOME via their distros (maybe GNOME could have a good looking wiki or website page with links to relevant places according to people’s distros). I know OMG! shared the justifications for the removal of desktop icons (via the GitLab Issue) but you still got the usual hate (so much so that the comments are now closed)….

    Liked by 1 person

  19. Thanks a lot for your work. I never participated in the gnome development, but I’m a long year user and like most gnome software quite a lot.

    But I know it’s free of charge, so I will never annoy you with negative comments. Gnome developers can and should decide how the project develops, and if they want they can ask users what they want.

    Don’t let people get control of you who don’t respect you and/or your efforts.

    Liked by 1 person

  20. I first will like to thank you for everything you’ve done in the past and all you’ve did for finding a solution to this issue. This must have costed you a lot of time, painkillers, and understanding from friends and family. Also big thanks for this writeup, it sure helped me get a better understanding. You are great! ❤

    We are an open community, with open development, no barriers for new contributors … I would love to hear your opinions, comments and thoughts on this topic.

    I think Gnome and most other open source projects in general unfortunately still have many barriers for people who don’t code and still want to contribute. For example, I don’t know if I could get someone who is really not interested in code to help translate. When a translator clicks on “get involved” on the Gnome website, s/he is met with so many blue links that s/he believes needs to be carefully read (could there be a more clear path?), then s/he needs to contact a local organizer (is it needed?) and hope for a response and setting up a developer platform (why?). The problem is that mostly everything is resolved around a traditional developer mindset. I have nothing bad to say about developers, without great people like you we would have nothing. Really nothing. You are the most valuable! The main problem however is that for a developer it is perfectly fine setting up a developer environment, it’s a walk in the park. To a developer that’s supereasy. To others, this is a wall you just stir at with a big question mark above your head. A developer will easily climb that wall (that wall is flat for them), a person that is a little interested in code will do what it takes to climb that wall, but everyone else will just break their nails trying or go home – they don’t get that far to ever become a contributor. The last group is the majority of people. This is the reason to why most (even big) projects have so few contributors, it’s also the reason to why most or every contributor is someone who very likely is a developer or at least is interested in code.
    So how do we fix it, how can we get even more people involved? We need to stop thinking as developers for every type of work that ain’t strictly development. For example for translation we stop expecting contributors to work in a developer environment, we instead provide something like Weblate with UI screenshots so that contributors can use a webinterface to easily get involved. We need the correct tools for the correct people, if we don’t do this so many valuable resources will continue to go to waste. If we do this new way of thinking for everything that is not strictly development, I believe developers more easily can focus on what they like and are really good at and with more people involved that are good at other topics open source in general will be a train moving faster and better than ever before.

    Like

    1. Translation is already done pretty much via a web UI. From https://l10n.gnome.org/ you choose your language, then the project you want to help with, download po file, translate what is missing/fix that grammar error/whatever and upload it back.
      The only problem is if you want to actually test it to see if there are long string stretching out the UI etc.

      Like

      1. The download/translate/upload workflow doesn’t sound as easy as it could be. It means people need to choose and install a translation application, and work with files. I know that doesn’t sound like a big deal, but saving a file from one program and opening it in another can take a little bit of mental effort.

        Maybe it would be an interesting project to allow translating entirely within a web interface.

        Like

        1. It should be done, in place, within the app. See a nit, fix a nit.
          If we’ve learned nothing else it’s that a lowering of the entry bar for contribution leads to more contributions.

          Like

      2. I don’t think this really solves it, but sure this is better than some other projects I’ve seen. However you still seems to need to download something and you need to have a tool to work with such as Poedit. With no screenshots of the UI you will have to install the software to get the context of what you are translating. If the version of Poedit (or something else) that your distro provide is old and don’t provide the features you need, you may have to compile Poedit from source. If you are translating unreleased code, you will most likely have to compile it. Weblate lets you provide screenshots of the UI so that the translator always knows in what context the strings are. This also makes it possible to have translators that can translate unreleased code without having to rely on tools or setting up something they don’t understand. And other benefits. If translators prefer using offline tools and work in a more traditional way, Weblate is not hindering it so it will not have any impact on existing translators that may prefer this.

        The same goes for documentation and other topics, the path to help with documentation seems muddy for someone who is not already involved. But again, I don’t think this is specifically a Gnome issue, this is very common problem for most open source projects in general and that’s why we fail to get the majority of interested people involved.

        Like

  21. Thank you for your work on Gnome Shell, and thank you for taking the time to write up a blog post (with diagrams)! As a longtime user of gnome shell who recently has started to play with the extensions mechanism, I find gnome shell and its extensibility invaluable, and I know that its extensibility comes with complexity that can make undertakings like this non-trivial. (Aside: sorry that this is not quite relevant to the discussion proper. I just love gnome-shell!)

    Like

  22. Heh, I remember we ran into the exact same problem in 2013 ;-). I remembee we put a workaround in place for it then but I did not think it was still an issue.

    Truth be told, the problem you identified is an architectural bug and it is basically unfixable (asides from hacks which queue a GC on every object release) with language bindings that use garbage collection as opposed to reference counting.

    The problem I identified back then was that SpiderMonkey’s GC runs when the SM allocator allocates more than 20 megabytes of memory (this is actually a variable threshold that gets bumped up in powers of two depending on how much the last GC freed but as we’ll see that doesn’t matter so much). The kicker is, as you alluded to, that the SM GC has no idea how much memory a bound C object allocated. So you could very well create and load a ton of GdkPixbufs for instance and to the GC it would look like you were creating a few megabytes worth of JSObjects while in the meantime you were allocating gigabytrs worth of images.

    Incidentally this is why cairo objects have a .$release() method.

    An interesting twist on this is that the SM GC dors actually have a way for you to tell it how much memory you’re using on a global basis. I remember doing a hack with GMemVTable to collect and report this info but that’s not the right solution as GMemVTable is for debugging purposes.

    Being more aggressive with the GC is another approach but I am not sure it is the best one long term, since you’d be paying the cost of traversing the object graph pretty much every time a reference gets dropped.

    I imagine the best way to deal with this is some sort of memory statistics singleton that can be used to measure memory pressure. An object could give a hint to the singleton about its expected memory usage. Librariers would have to opt in though. Also, global state.

    Tough problem indeed.

    Liked by 1 person

  23. Thanks that lightens up everything. As a Java dev GC is something you never think of, which is bad since you should to be ready for other languages. Does a manual free() or deallocate exist in JavaScript?

    Regarding those omgubuntu comments… This should not be seen as representative for and is not the open source community. It is just one of the most toxic people in the open source user land. I wasted my time so often by reading those comments. Don’t do it. It is like you put your head into the last circle of hell for a moment and see all the lost souls full of hatred. Keep up the good work. If you want a good example of respectful open source behavior one could look into the new Ubuntu community page. That’s a great place!

    Like

  24. Thanks for all your work. It took me a while to get used to Gnome, but now I use it everywhere.

    Would a “user thrown grenade” be possible? I.e., an app (or something) a user could run that would queue a garbage collection? E.g., I notice that gnome-shell is using more than 20% of my memory and I’d like it to garbage collect old bits from memory. So I run, say gjsgc, and a (recursive?) garbage collection is queued.

    Ideally users wouldn’t need to be concerned about garbage collection, but while waiting for the fixes to distributed, etc., I’d be willing to manually run a program to reclaim some memory. It would be better than logging out/in to reclaim the memory, which is what I do now.

    Best regards.

    Liked by 1 person

    1. … Had to close everything and logout again to recover 5 GBytes of memory from gnome-shell. 😦

      Is there another way to manually tell gnome-shell to clean itself?

      Like

          1. 1) Download the ZIP file from github, then unack it.

            2) Copy the forcegc@email.com into .local/share/gnome-shell/extensions/

            3) Reload the shell: press Alt-F2, type “r” into the box and press Enter.

            4) The extension will show up in the “Extensions” tab in the Tweak tool; by then you just need to activate it.

            HTH

            Like

          1. I’ve had it installed for a while now, and although it does not eliminate the issue it makes it more manageable.

            Hopefully future versions of GNOME will actually fix the problem.

            Like

          2. As I said, this extension is more like a band-aid than anything. It does not solve the problem, it merely makes it manageable.

            I’ve been running it for a while now, and it has helped. Without using it, I’ve seen gnome-shell use up almost 8 GB of RAM after one day of usage. With it enabled, its usage after a whole weekend has not gone above 2 GB. Still not ideal, but it’s a lot better.

            As you said, there are things the garbage collection called from an extension simply can’t reach. Hopefully feaneron’s workaround will make it better.

            I would love to see a real fix to this issue, though. GC seems not to be working properly and forcing it to run all the time really shouldn’t be called a “fix”.

            Like

          3. … and running imports.system.gc() 10,000 times in lookingglass seems to do nothing to reduce gnome-shell’s memory use. It seems lots of memory can’t be touched by the garbage collector.

            Like

  25. The best fix would have been:
    rm all source codes, destroying all hard drives, to be sure it can never be recovered again. Burning all the git servers where any kind of gnome source code has be stored. Safety first! Remove any track from wikipedia, etc.. so that the later generations will never know that something like gnome has ever existed.
    Of course gorgeous C is again at work, being again the root of an epic fail. How much collateral damage has to be done till people start to understand that C should be used very carefully and just in special circumstances. Even in microcontroller programming C++ can be the better option!

    Like

    1. This didn’t really have anything to do with C in particular… It’s an issue with mixing reference counting and a GC; heck, it could’ve probably happened in Rust even.

      Like

  26. “Users” in the sense of people how just uses the software and do not contribute(code, bug reports, art and so on), are not a part of the community. They don’t matter. If you want your voice heard then contribute. Complaining on OMG Ubuntu does not count as contributing. But even if you do contribute you must also realize that it’s not a democracy. At best, ideally it’s a meritocracy. The weight of your voice is based on the quality and the amount of your contributions. In a large project it may take years to build up the reputation to be able to steer a project in the direction you would like it to go.

    The flip side of that is when someone does file constructive and useful bug reports, don’t ignore or dismiss them off hand just because they are new and/or lack technical knowledge. Performance bugs like “The Leak” are pretty black and white. If it affects the person that files the bug it’s real. I’m a dev of a couple GNOME Shell extensions and I had users file a bug against my extension since it triggers the current leak. They were told to so after being dismissed by GNOME devs who blamed my extension. It took me about 2 mins to realize that it was a GC issue. I even went so far as to re-write a new simplified version of the extension in question going to great pains that I did as much as I could to make sure objects were being destroyed as textbook as possible. It did no good. The leak persisted.

    Liked by 1 person

  27. Please help me out here. I’m no expert and I just want to understand what happened. What is GJS’s role in gnome-shell? And do the merges really fix the problem with triggering GC or do they just force GC to compensate for that?

    Like

    1. GNOME Shell is partially written in JavaScript. The problem is that multiple GC passes are necessary to free all the objects. The solution does that, in a rather rustic way at the moment.

      Like

      1. By being “GNOME Shell is partially written in GNOME Shell” I suppose you mean it’s writted in GJS, right? So was it a problem with the implementation of GC? Or with the way it’s “triggered”?

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s