To clarify for casual readers who might not dig into that, it’s not that Fullstaq is refusing to contribute or that they are commercial. They say :
“There will be no paid version. Fullstaq Ruby is fully open source. It is also intended to be a community project where anyone can contribute. There are no monetization plans.”
Thought I would mention that in case the parent’s comment implied something different. It’s more that the core Ruby team are reluctant to.
Maybe I’m missing something, but it seems the only changes here over upstream are jamlloc and/or malloc_trim. I’m not as familiar with malloc_trim, but jemalloc is fairly trivial to setup without recompiling Ruby (using LD_PRELOAD), and there are even buildpacks for environments like Heroku. It seems like more time was spent on the marketing page than on the actual optimizations.
The goal of Fullstaq Ruby is to democratize the fight against Ruby memory bloat. Democratization means that as many people should be able to reap the benefits as possible. It's 2021 now, and expecting users to compile Ruby or Jemalloc from source is no longer realistic. Compiling anything is no longer a non-scary, low-friction thing to do.
It seems like the main value-add of this project is building/packaging pretty much standard Ruby in a consistent way across many Linux distributions, which is something that’s been hard to come by. I would lean into that more than anything else.
I don't know if 2021 is much different than, say, 2012, in that I don't mind compiling something as long as it works? I'm sure there are intricacies with ruby's configuration, specifically. And that you invested time and have a good understanding of it.
But I'm not convinced it's a product and not a service, or a task for someone on staff for larger organisations.
As but one thing: to be a product it needs to have some universal value for a significant chunk of scenarios (possibly with some configuration options). But at that point, it's not clear why it shouldn't be a contribution to the upstream project. Is there, possibly, an "Enterprise Edition" or "Cloud Deployment Pro Plan" in the works?
I'm also (and that's neither here or there but I can't help myself) not entirely sure how democracy got involved here. While the rule of law and a participatory citizenry do tend to lead to obvious benefits, "reaping" evokes a one-way process that is sorta antithetical to the cooperative nature of democracies? Dunno, maybe it's just me. It's also on some buzzword bingo cards, although I can't quite say in what specific context...
> But I'm not convinced it's a product and not a service, or a task for someone on staff for larger organisations.
From my experience with users and the industry at large, the amount of people who are comfortable with compilation and sysadmin-y tasks are rapidly declining, proportion-wise. The industry is heading towards ever-more specialization. Many many backend developers nowadays don't want to think about infrastructure at all, they just want to focus on business logic. There are a huge amount of backend developers who have never seen './configure && make install'.
In an organization with sufficiently advanced human capital, yes there is someone who can take care of that. But the existance of such a person can be taken less and less for granted nowadays even in very large organizations.
There is also the factor of: should we do this work ourselves? Lots and lots of developer tooling nowadays are extremely slick. We've been spoiled. I have been spoiled. I can still write C++ but I don't want to bother with './configure && patch && make install' anymore. The standard nowadays is higher. Why should I spend a day installing a custom-patched Ruby when someone else can do that for me and all I have to do is to add an APT repo? Especially when I always have better things to do?
> it's not clear why it shouldn't be a contribution to the upstream project.
> I'm also (and that's neither here or there but I can't help myself) not entirely sure how democracy got involved here.
Democratization, not democracy. Democratization is about making something available to as many people as possible. Which is a different concept from democracy.
Given that so many people are uncomfortable with compiling, or with LD_PRELOAD, or with anything outside of "bundle install", asking people to "just compile Jemalloc 3, make sure to apply this patch, then modify your systemd init script to include LD_PRELOAD" is too much to ask. It shuts down an entire range of people from benefiting from Jemalloc. With democratization, I seek to combat this.
The parent comment noted that using jemalloc for Ruby without compiling is already trivially easy, so the argument that 'compiling is hard' is irrelevant even if true.
The real argument is that installing jemalloc separately and running Ruby with an environment variable is too hard.
In my comment, as well as in the FAQ, I explain why even using Jemalloc _without_ compiling Ruby has its own caveats.
The Jemalloc version matters a lot. For reasons that are not yet clear, significant memory savings are only achieved with Jemalloc 3, not with Jemalloc 5. Your distribution only ships one Jemalloc version. So likely you need to compile Jemalloc 3 yourself. Here you are already entering compilation land.
But Jemalloc 3 no longer compiles by default on some modern distributions, such as Debian 10. Fullstaq Ruby fixes this by patching Jemalloc for you.
Furthermore, which Ruby binaries are you using? The ones provided by the Linux distribution are perpetually outdated. Another of Fullstaq Ruby's value proposition is that we supply binaries for the latest Ruby version, quickly. We packaged Ruby 3.0 on the same day it came out.
Thanks, these extra details are helpful and seem like the more significant motivations underlying this distribution.
> For reasons that are not yet clear, significant memory savings are only achieved with Jemalloc 3, not with Jemalloc 5.
So the real advantage of this package is that it bundles a 6+ year-old, unsupported version of Jemalloc, because the more recent versions found in current OS distributions don't yield memory savings in practice- for unknown reasons. This doesn't instill very much confidence.
I would be much more excited by efforts to investigate the jemalloc > 3.x changes so Ruby can work optimally with current releases packaged in modern Linux distributions, rather than double-down on a workaround that requires bundling an increasingly-ancient version of the software.
I should also add - as mentioned by the jemalloc author [1], the addition of the time-based purging feature is likely responsible for memory-usage differences between jemalloc 3.x and 5.x, so you can reduce `dirty_decay_ms` and `muzzy_decay_ms` to get 3.x-like memory usage. I have been using this configuration in production since 2018 for significant memory savings in Ruby using jemalloc 5.x.
I think that is an overly cautious take on things. I'll explain why.
First, "for unknown reasons" deserves more nuance. The vague, high-level reason is clear: Jemalloc 3 behaves differently from Jemalloc 5, having different algorithms and data structures. What I mean by unknown is not so much an indication of incomprehensible arcane magic, and that things can collapse at any time.
What I mean is that it's not known in what way the algorithms and data structures are different. Consider that before I did my 2019 research on why Ruby memory bloating occurs[1], Ruby apps suffered from memory bloat "for unknown reasons". That didn't mean that before 2019, all Ruby apps were houses of cards waiting to fall over.
It's like saying "I don't understand why this Linux kernel upgrade made things faster" -- the kernel developers know but they have better things to do than to answer your questions. And the fact that knowledge about a new optimization in the Linux kernel is not widespread, does not mean that that kernel version is unstable.
Nobody truly understands every single detail about all parts of the stack. Yet I can build reliable, high-available web apps just fine without understanding how for example how 5G works and why users on 5G can access my app faster than on 4G.
The differences between Jemalloc 3 and 5 are not explicitly documented anywhere, and to find out requires research. I intend on doing that some time in the future, but not now. Jemalloc 3 is proven to work, it's proven to be stable. The combination of Ruby + Jemalloc is proven to work well, not only because we've had several years of user feedback now, but also because Github has tested this combination for years now even before Fullstaq Ruby.
The pragmatic thing to do is not to prioritize figuring out exactly how Jemalloc 5 works. It's to continue the packaging work to make Ruby + Jemalloc 3 available to the public. Jemalloc 5 can wait.
Aside- thanks for engaging in detailed discussion here and more generally for all of the Ruby-performance contributions you've made over many years, I truly appreciate them.
First, my lack of confidence in depending on Jemalloc 3 in production is not only the 'unknown reasons' underlying such a frozen dependency, but also due to the fact that this particular dependency is over six years old and unmaintained. Not only does this lack more recent security/bug fixes and features, but also makes it more complex to integrate with up-to-date Linux distributions (e.g., your need to maintain custom compilation patches instead of simply depending on the OS's jemalloc package).
> What I mean is that it's not known in what way the algorithms and data structures are different. [...] The differences between Jemalloc 3 and 5 are not explicitly documented anywhere, and to find out requires research.
As I mentioned, the jemalloc developer already highlighted the exact differences back in 2018, and he even provided a MALLOC_CONF environment variable to use that makes memory usage in jemalloc 5 behave like jemalloc 3:
> You could verify this by setting dirty decay and muzzy decay to 0 in the MALLOC_CONF environment variable (i.e. MALLOC_CONF="dirty_decay_ms:0,muzzy_decay_ms:0", unless I've typoed something).
There is also a very readable page on performance tuning in the jemalloc 5 documentation [1].
In your research, have you ever tried running jemalloc 5 with this configuration? I did this back in 2018, tested/verified against my production workload, and have been running Ruby on jemalloc 5 without any issues since.
All that's involved is installing your OS's 'jemalloc' package and setting two environment variables (LD_PRELOAD and MALLOC_CONF). Simple enough and more confidence-inspiring than maintaining a patch against a six-year-old frozen dependency if you ask me.
> The pragmatic thing to do is not to prioritize figuring out exactly how Jemalloc 5 works. It's to continue the packaging work to make Ruby + Jemalloc 3 available to the public. Jemalloc 5 can wait.
I disagree about the relative priorities- I spent a day tuning Jemalloc 5 for my team's Ruby application back in 2018 [2] and it's been a done issue for us since then.
I did try those settings, but was unable to reproduce the same amount of memory savings as Jemalloc 3. But that was quite a while ago, and I need to revisit this some time in the future.
By the way, Fullstaq Ruby does not only provide Jemalloc-patched versions. We also provide an unpatched version (and also a version with only malloc_trim) — in which case Fullstaq Ruby's main value add becomes DEB/RPM packaging only.
Thanks I didn’t know this was possible. I’m currently running Ruby compiled with jemalloc in production on elastic beanstalk with a custom AMI. If I can swap back to their AMI and just use LD_PRELOAD it will make upgrades a lot easier.
At least 10 years ago it was normal or at least in my circles to always compile your own Ruby with all the production settings needed… jemalloc … we just build our own rpms and it’s pretty easy then to roll out upgrades too …
For those wondering why on earth this isn’t simply a contribution to Ruby, the FAQ has a really nice explanation - tldr is that this is a variant optimised for the very specific context of Linux, x86/64, server environments, with pre compiled and packaged binaries, with special memory allocators (jemalloc).
These changes are specific to this context and are unlikely to be accepted into the more generalised Ruby core.
Same team as Phusion Passenger, so excellent background.
I'm not really buying the FAQ explanation, idk this feels too good to be true. Would have loved to see some core team discussion on this if there was one.
Author of Fullstaq Ruby here. There have been discussions. Discussions on jemalloc integration preceded Fullstaq Ruby for years. The bug tracker issue on the malloc_trim patch went stale. All this can be found on the issue tracker.
I also spoke to Matz during EuRuKo 2019. And this is the impression I got.
- Shyouhei Urabe did a marvelous job of resisting pressure from several well-known developers representing multiple high profile ruby companies/projects.
My personal feeling is that while the "Jemalloc FTW" section in http://oldblog.antirez.com/post/everything-about-redis-24.ht... might have been correct back in the jemalloc 3.x days, a lot has changed on both jemalloc and glibc since then. Furthermore, there is also musl in the equation these days.
> Wow, a lot of comments. What Fullstaq Ruby is doing is difficult to add into mainline, so my overall impression is I'm thankful someone's going to the trouble of doing it themselves.
> It's not quite right to say that on Linux with jemalloc it uses less memory. In reality, it would be more correct to say that using a super-old version of jemalloc that for some reason used less memory leads to reduced memory usage (for unknown reasons). So it can't go into mainline.
> That said, I can understand that some people are just happy to see reduced memory usage, so if those people want to keep using the super-old version of jemalloc with reduced memory usage, I have no objection. Go for it, is my feeling.
Many of them are adopting Jemalloc, but they do the Jemalloc integration work themselves because they have the engineering talent and capacity. Fullstaq Ruby is for everybody else: those who can't, don't want to, or can't be bothered to do it themselves. It's for the masses.
I'm sure Ruby would have adopted it if it would have benefitted most users . It's either not benefitting most, is gonna break with future changes (like Guilds/Ractors) or a combination of the two.
Do you believe copy-on-write friendly garbage collection
— which saves ~33% memory, and is essential to Passenger's "smart spawning" methode, Unicork's preload_app mode, Puma's cluster mode — is a thing that benefits most users?
Besides being the author of Fullstaq Ruby, I am also the author of the first copy-on-write-GC-friendliness patch for Ruby. Consider the fact that it took ~5 years for the copy-on-write-friendly-GC feature to find its way back to upstream Ruby.
If you lived in 2008, would you be willing to wait 5 years in order to save 33% memory?
You're not really answering to anything I said though.
Sorry man I get that this is emotional for you, not trying to bring you down. You need to see it through my eyes; I'm not gonna introduce into my company anything that's high risk, what's in it for me? If this goes fine I maybe saved some memory, if this goes wrong I'm gonna look silly (how do I even justify going this route to anyone? it's not like the internet is clear this is the way to go; it's quite controversial or even not the way to go).
I think you're misunderstanding me. I am not selling you anything. I make no money off this. I am not invested in whether you personally choose to move forward or not. My point is different.
Your original statement basically equates quality with authority: you believe a project is qualitatively good if and only if it's officially blessed by an authority you trust. And that belief is fine. Most people either aren't able to, or don't care enough to, verify quality themselves, so we defer to trusted authorities. Nothing wrong with that approach in general.
The thing I'm trying to tell you though, is that in case of MRI, there have been multiple past cases when the lack of official authority blessing did not in fact mean a lack of quality. This isn't hypothetical: there's a proven track record. My own track record is proven as well.
What is in it for you? Well, reduced memory usage, improved performance, faster and lower-hassle installation and security patching!
There is always a risk, yes. But all I'm trying to say is that the equation "lack of authority blessing == lack of quality" is proven not to be true for a Ruby project such as this. Whether you want to take the risk of trying it out, that's entirely up to you.
It's 2021. Installing from source is no longer a low-friction, non-scary, default thing to do or something that people want to bother with. Users now expect binaries and proper package management.
Furthermore, it's not just "dnf install jemalloc-devel". The Jemalloc version matters a lot. For reasons that are not yet clear, significant memory savings are only achieved with Jemalloc 3, not with Jemalloc 5. And your distribution only ships 1 Jemalloc version. So likely you need to compile Jemalloc 3 yourself.
Another caveat is that LD_PRELOADing Jemalloc is more effective than compiling with --with-jemalloc.
Furthermore, installing Ruby is one thing. Keeping it security-patched without having to constantly monitor things, is a whole other can of worms. Here's where Fullstaq Ruby's second value proposition comes in: we supply DEB/RPMs and do the tiny version patching for you, you don't have to go through the whole compilation process and gem migration process every time there's a new Ruby tiny version.
Dont know about Heroku, but Github, Gitlab, Shopify are all using Jemalloc in production. Practically all big Rails shop are using it to the point I thought it was common knowledge in 2021.
There has been 8 - 10 glibc release since then I am not sure if things have improved or not.
Not really common knowledge, where is Shopify talking about jemalloc? I can't find anything. Github there's a tweet confirming it, they never really wrote about what they're doing with jemalloc afaik. I mean wrote as in not a tweet. But yes Eileen confirmed they use it. If people blogged about it more it would have become common knowledge I guess.
Everybody can do it by themselves. Using jemalloc is trivial. Adding the malloc_trim() call to the full GC is trivial. The patches are open.
Why MRI ruby likes to keep unused memory pages around is only, that they don't like to add that compile-time option and probe for jemalloc. It's too trivial to worry
Wow, I'm really excited about this. Running a large rails website, we spent egregious amounts of money getting EC2 instances with extra ram. If they can fulfill their promise of reducing memory usage by 50%, that'll save us tens of thousands of dollars.
You can already use jemalloc on Heroku, it's really simple, so this comment is about releasing memory back to the OS, the other headline feature of Fullstaq.
> Memory bloat is a real problem, even for mid-size Rails apps.
Yes, but the solution isn't releasing the memory back to the OS, that's just papering over the cracks. If you're running (say) 10 Rails processes on a machine with 10 GB RAM, and particular request paths/background jobs whatever pushes the process size to 2 GB RAM size, you can release the memory back to the OS after you're done, but you still have the underlying problem which is the process gets to that size in the first place, and if all the processes hit 2GB at once, now you're 10 GB in debt, either in swap, or OOM errors.
The solution you should be looking for is installing something like Scout (or NewRelic, but IMO Scout is better for memory stuff), looking to see what is bloating the processes, and in 99% of the time fixing it using some the Rails' batch find methods (seriously, it's ActiveRecord objects 99% of the time, bonus points for in-memory CSV reading or write buffers).
Rails applications don't have to balloon in memory.
> If you're running (say) 10 Rails processes on a machine with 10 GB RAM, and particular request paths/background jobs whatever pushes the process size to 2 GB RAM size, you can release the memory back to the OS after you're done, but you still have the underlying problem which is the process gets to that size in the first place, and if all the processes hit 2GB at once, now you're 10 GB in debt, either in swap, or OOM errors.
Fullstaq Ruby author here. I have done research on the memory bloat problem[1], and this statement is not true. With the way memory allocation in glibc's ptmalloc2 works, multithreaded processes can reach bloaty proportions even if they don't do much work. I started investigating this whole issue because I had a trivial Ruby HTTP proxy server: it had no reason to use 1.3 GB of memory, it doesn't do anything besides forwarding small requests. That's when I found that ~1 GB of that 1.3 GB of memory wasn't even due to anything that my app did by itself, but it's just how the memory allocator works.
That's what we see in our app, and I used to see all the time in others (I was a performance and scaling consultant for years). It's trivial to use jemalloc on heroku, so you may as well, and after that it's mostly preventing too much loading of AR objects.
Can you share the method you are using to do it? Not challenging, actually answering. Googling I'm not sure which thing I'm finding is the trivial one you use/recommend.
I recently switched to asdf.
Big time rbenv user, then added nodenv. I also needed multiple yarn versions. As asdf is based off / inspired by rbenv, it was an easy change.
(You don't need to use ruby-install with chruby, you can use whatever method of installing rubies you want. The chruby README even includes instructions for `ruby-build`. But normally whatever method you use, you install to `/opt/rubies` or `~/.rubies` which is where chruby will look. But fullstaq ruby insists on installing to it's own locations. But possibly chruby will be fine with that if you just tell it where that is... but you might need to do it for each ruby you install, which woudl be annoying. You could probably write a little bash shim which just checked the fullstaq install location for which rubies it had, and then just added them each individually?)
Cool project. Not a big fan of the name. Kind of hard to sell to a sourcing team. If they called it “Enterprise Ruby” or something like that the conversation is going to be much easier.
If there are three solutions, you will have to choose which one. I think it's an easier sell to the enterprise, eventhough all three projects are by the same person (passenger, ree, fullstaq)
Author of both projects here. Yes Ruby Enterprise Edition eventually became obsolete because most features found their way back to MRI. But that took a few years. During those years, Ruby Enterprise Edition made a lot of users happy.
When I read fully open source, bells ring in my mind. I adopted Cypress. It advertised open source and community ethos. But when delved deep, it many basic features that were paid. They never advertised it well in documentation. Once bitten, twice shy
What features are you missing? I’ve never paid for cypress purely because it had what I wanted out of the box. Now I’m wondering if I’ve missed something.
That’s fair. We built our own little parallel test runner to handle that runs each file in a different process but I would have paid if the official version supported parallel test runs on a single host.
I think this is interesting, but I'm wondering why there isn't a corresponding project for Python. It seems like people using Python could derive the exact same benefits, so, why doesn't the Python version exist of this project?
...long story short: basically, you can already, but it doesn't really make much difference for most workloads.
Or, if your question is, why doesn't python get optimisation love and become fast like the javascript runtimes?
dunno. I guess we just don't have any big FANG companies pouring money on that, because they don't have a use for a faster python. A faster python does not make numpy, tensorflow or pytorch run faster because those are not written in python.
So I guess, there's a lack of value-for-effort in trying to 'fix' the problem of 'pure' python being slow.
Still, adoption of these solutions seems low (PyPy seems to be a JIT as well is it really a corresponding solution?). If it magically reduces memory by as much as 30%-50% why isn't adoption higher? In other words what are the drawbacks of doing this, there must be cons.
Author of Fullstaq Ruby here. From what I understand by talking to people, the reasons why Fullstaq Ruby doesn't have wider adoption are as follows:
It's not a project officially blessed by Ruby core. Upstream Ruby is actually "good enough" for most. Most people don't feel the pain enough to try something which appears as unofficial. Perhaps their thinking are similar to yours: I'm hesitant because not everbody else is already using this.
I am also not marketing this as hard as I should. I lack free time.
Another reason is because of the beta label. The only reason why the label is there is because I have a specific, ambitious vision on how "final" should look like (i.e. more features), not because it's unstable. Feedback from users regarding stability has been very good so far. I might be shooting myself in the foot marketing-wise by not removing the beta label earlier, but I'd rather do a good job than to hurry.
