Even better than earlyoom is systemd-oomd[0] or oomd[1].

systemd-oomd and oomd use the kernel's PSI[2] information which makes them more efficient and responsive, while earlyoom is just polling.

earlyoom keeps getting suggested, even though we have PSI now, just because people are used to using it and recommending it from back before the kernel had cgroups v2.

[0]: https://www.freedesktop.org/software/systemd/man/latest/syst...

[1]: https://github.com/facebookincubator/oomd

[2]: https://docs.kernel.org/accounting/psi.html

Another option would be to have more memory that required over-engineer and to adjust the oom score per app, adding early kill weight to non critical apps and negative weight to important apps. oom_score_adj is already set to -1000 by OpenSSH for example.

    NSDJUST=$(pgrep -x nsd); echo -en '-378' > /proc/"${NSDJUST}"/oom_score_adj

    vm.overcommit_memory = 0
    vm.overcommit_ratio = 0

    vm.admin_reserve_kbytes = 262144
    vm.user_reserve_kbytes = 262144
    vm.min_free_kbytes = 1024000

Another option is to limit memory per application in cgroups but that requires more explaining than I am putting in an HN comment.

Another useful thing is to never OOM kill in the first place on servers that are only doing things in memory and need not commit anything to disk. So don't do this on a disked database. This is for ephemeral nodes that should self heal. Wait 60 seconds so drac/ilo can capture crash message and then earth shattering kaboom...

    # cattle vs kittens, mooooo...
    kernel.panic = 60
    vm.panic_on_oom = 2

To learn tricks like this what resource do you recommend I read? System administrators handbook? (Still on my TOREAD queue)

Yeah, no way. As soon as you hit swap, _most_ apps are going to have a bad, bad time. This is well known, so much so that all EC2 instances in AWS disable it by default. Sure, they want to sell you more RAM, but it's also just true that swap doesn't work for today's expectations.

Maybe back in the 90s, it was okay to wait 2-3 seconds for a button click, but today we just assume the thing is dead and reboot.

It's always a good idea to have a tiny amount of swap just in case. Like 1GB.

Thanks for this. We resorted to setting ram thresholds in systemd.

Is earlyoom a better solution than that to prevent an erratic process from making an instance unresposnsive?

Some workloads may do better with zswap. Cache is compressed, and pages evicted to disk based swap on an LRU basis.

The case of swap thrashing sounds like a misbehaving program, which can maybe be tamed by oomd.

System responsiveness though needs a complete resource control regime in place, that preserves minimum resources for certain critical processes. This is done with cgroupsv2. By establishing minimum resources, the kernel will limit resources for other processes. Sure, they will suffer. That’s the idea.

Of course swap should be enabled. But oom killer has always allowed access to an otherwise unreachable system. The pause is there so you can impress your junior padawan who rushed to you in a hurry.

What's the performance hit from compressing ram?

Haven't used swap since 2010.

How do you get swap on a VPS?

From memory there aren't many dedicated servers available in the US.

Latency to the US from Europe?

Jacked up prices isn't what is happening here. There is a psychological effect that Heroku and other cloud vendors are (wittingly or unwittingly) the beneficiary of. Customer expectations are anchored in the price they pay when they start using the service, and without deliberate effort, those expectations change in _linear_ fashion. Humans think in linear terms, while actual compute hardware improvements are exponential.

Heroku's pricing has _remained the same_ for at least seven years, while hardware has improved exponentially. So when you look at their pricing and see a scam, what you're actually doing is comparing a 2025 anchor to a mid-2010s price that exists to retain revenue. At the big cloud vendors, they differentiate customers by adding obstacles to unlocking new hardware performance in the form of reservations and updated SKUs. There's deliberate customer action that needs to take place. Heroku doesn't appear to have much competition, so they keep their prices locked and we get to read an article like this whenever a new engineer discovers just how capable modern hardware is.

Heroku is the Vercel of Rails: people will pay a fortune for it simply because it works. This has always been their business model, so it’s not really a new development. There’s little competition since the demand isn’t explosive and the margin is thin, so you end up with stagnation

> It's a shame they don't just license all their software stack at a reasonable price with a similar model like Sidekiq and let you sort out actually decent hardware

We built and open sourced https://canine.sh for exactly that reason. There’s no reason PaaS providers should be charging such a giant markup over already marked up cloud providers.

> $50 for a dyno with 1 GB of ram in 2025 is robbery

AWS isn't much better honestly.. $50/month gets you an m7a.medium which is 1 vCPU (not core) and 4GB of RAM. Yes that's more memory but any wonder why AWS is making money hand-over-fist..

Now I know why the teaching platform I use is trying to kick me off...

Every other time I login to the admin site I get a Heroku error.

I am not sure what's there to license. The hard and expensive part is in the labor to keep everything running. You are paying to make DevSecOps Somebody Else's Problem. You are paying for A Solution. You are not paying for software. There are plenty of Heroku clones mentioned in this thread.

Yeah, I choose railway app for my PaaS hosting for this reason

It's insane how much my local shop charges for an oil change, I can do it much cheaper myself!

It's insane how much a restaurant charges for a decent steak, I can do it much cheaper myself!

...!

Teaching a whole bunch of developers some cloud basics and having a few cloud people around is relatively cheap for quite a while. Plus, having test/staging/prod on similar configurations will help catch mistakes earlier. None of that "localstack runs just fine but it turns out Amazon SES isn't available in region antartica-east-1". Then, eventually, you pay a couple people's wages extra in cloud bills, and leaving the cloud becomes profitable.

Cloud isn't worth it until suddenly it is because you can't deploy your own servers fast enough, and then it's worth it until it exceeds the price of a solid infrastructure team and hardware. There's a curve to how much you're saving by throwing everything in the cloud.

The cloud has made people afraid of linux servers. The markup is essentially just the price business has to pay because of developer insecurity. The irony is that self hosting is relatively simple, and alot of fun. Personally never got the appeal of Heroku, Vercel and similar, because theres nothing better than spinning up a server and setting it up from scratch. Every developer should try it.

Fully replicating prod is helpful. Saves time since deployment is similar and does a better test of what prod will be.

This could be the premise for a fun project based infra learning site.

You get X resources in the cloud and know that a certain request/load profile will run against it. You have to configure things to handle that load, and are scored against other people.

The cloud was a good deal in 2006 when the smallest aws machine was about the size of a ok dev desktop and took over two years of renting to justify buying the physical machine outright.

Today the smallest, and even large, aws machines are a joke, comparable to a mobile phone from 15 years ago to a terrible laptop today, and take about three to six months to in rent as buying the hardware outright.

If you're on the cloud without getting 75% discount you will save money and headcount by doing everything on prem.

also how far you can get with a single machine has changed massively in the past 15 years. 15 years ago a (really beefy) single machine meant 8 cores with 256GB ram and a couple TB of storage. Now a single machine can be 256 cores on 8TB of ram and a PB of storage.

The cloud has made people forget that the internet is decentralized.

The weird thing is the relationship between developer costs and operations costs. For startups that pay salaries, $3000 a month is a pittance!*

* The big caveat: If you don't incur the exact same devops costs that would have happened with a linux instance.

Many tools (containers in particular) have cropped up that have made things like quick, redundant deployment pretty straightforward and cheap.

You put your staging env in the same (kind of) place you put your prod system because you need to replicate your prod environment as faithfully as possible. You also then get to re-use your deployment code.

Cloud often has everyone thinking it's still 2008.

you can literally buy a used dell desktop that matches the spec for hetzner (8 core, 32 gigs of ram) for under 500 USD. Why wouldnt you just do that?

As cloud marches on it continues to seem like a grift.

Just to be aware when you say "Even with all 6 environments and other projects running, the server's resource usage remained low. The average CPU load stayed under 10%, and memory usage sat at just ~14 GB of the available 32 GB."

The load average in htop is actually per CPU core. So if you have 8 CPU cores like in your screenshot, a load average of 0.1 is actually 1.25% (10% / 8) of total CPU capacity - even better :).

Cool blog! I've been having so much success with this type of pattern!

Thanks for sharing. I have an app I'm working on and this seems perfect for it.

what does this service offer over an established tool like Coolify? currently hosting most of my services on a cheap Hetzner VPS so i'm interested what Disco has to offer

Cheers, let me know if you do / hop onto our Discord for any questions.

We know of two similar cases: a bootcamp/dev school in Puerto Rico that lets its students deploy all of their final projects to a single VPS, and a Raspberry Pi that we've set up at the Recurse Center [0] which is used to host (double checking now) ~75 web projects. On a single Pi!

[0] https://www.recurse.com/

Can I ask which hetzner instance you use?

And if you really needed 99.99%, you would be wise to avoid the hyperscalers: see AWS' recent multi-hour long outage.

300 services?? What do they all do?

What in the world?? Surely there must be something more than "generate a HTML page with 500 elements". Any edge cloud hosting lets you do that for free.

I answered elsewhere with the list of dynos, but the short version is that between the list of services that each deployment required, and the size of the database, it truly (and unfortunately) did end up costing $500 per staging.

You're absolutely right! Here are some three points why:

- ...

I'm kidding :-)

Our library is open source, and we're very happy and proud that Idealist is using us to save a bit of cash. Is it marketing if you're proud of your work? :-) Cheers

> But even that's not the issue; the main issue is that the article is basically a marketing piece.

Why is that an issue? Is it forbidden by HN guidelines? Or would you like all marketing to be marked as such? Which articles aren't marketing, one way or another?

It's funny they have this marketing blog post based on competing on price yet don't disclose any of their pricing on their site only a schedule a meeting which is just about the biggest RED FLAG on pricing there is.

heh my first instinct was to go see how they're making money. Guess that's coming soon

This isn't the first time an article is also marketing. Besides, what is wrong with marketing something via a use case article? This is a fairly tame example of it and I found it an interesting and useful read, knowing full well it was also marketing.

6 staging servers: main, dev, and any branches that you want to let other (non tech people) QA.

As for the staging servers, for each deployment, it was a mix of Performance-M dynos, multiple Standard dynos, RabbitMQ, a database large enough, etc. - it adds up quickly.

Finally, Idealist serves ~100k users per day - behind the product is a lot of boring tech that makes it reliable & fast. :-)

From what I read, they're using them as dev environments. Like running many services at once for a single developer to tie into. That's why they wanted multiple ones, one for each dev.

$3000/month = 36k/year

That's more than 1/3 of the cost of a developer there.

That will save you some week of a person's work to set things up and half-a-day every couple of months to keep it running. Rounding way up.

Yes, everyone forget to compute man-days in the cost calculation

Any VPS you fancy that fits the price/performance/location/support you want, then point Coolify/Dokploy/whatever at it.

I did just this using Coolify, Mythic Beasts running Django & Postgres the other month from Google App Engine. Hilariously easy, even with my extremely rusty skills.

It is worth learning to use Docker Swarm. Deployments are as simple as pushing a new container to your registry and running one command. I built a free CLI tool rove.dev that simplifies provisioning and does service diffing.

Either that or use a PaaS that deploys to VMs. Can't make recommendations here but you could start by looking at Semaphore, Dokku, Dokploy.

https://render.com/ is probably the closest, I'm really enjoying using them. Workflow is the same as heroku, but cheaper, no nightly restarts, supports new python versions etc..

Whats wrong with just spinning up a server on hetzner. At most you need to setup nginx and a systemctl service

Oracle has free VPS if your requirements aren't huge. Hobby project etc.

PythonAnywhere: https://www.pythonanywhere.com/

Any place you can get a vps from.

Correct, GitHub is necessary at this point to deploy code.

However, yes, you can ask Disco to fetch an existing Docker image (we use that to self-host RabbitMQ). An example of deploying Meilisearch's image is here [0] with the tutorial here [1].

Do you typically build your Docker images and push them to a registry? Curious to learn more about your deployment process.

[0] https://github.com/letsdiscodev/sample-meilisearch/blob/main...

[1] https://disco.cloud/docs/deployment-guides/meilisearch

Yes, I'm just as curious as you on _why_ does a staging setup needs the same amount of resources as prod.

All of my staging setups are on a ~$15 Hetzner server, with a GitHub Action to `docker compose build && docker compose up -d` remotely, with an Apache service with a wildcard certificate and dynamic host names. We have 3..n staging setups, with each PR spinning up a new staging site just for that PR.

It's been working with us for years, for a team of 10 developers.

Heroku and Vercel don’t ever have any intention of competing on price

They offer convenience

On the other hand for $3k/month you can just hire someone to do it for you (part time at least, but I doubt it's remotely a full-time job).

Fly was supposed to fix Heroku but my bill more than doubled since they changed how they charge for shared CPUs.

https://community.fly.io/t/cpu-quotas-update/23473

I work at Render (render.com); we have over 4 million developers on the platform, and we've migrated many large (and small) Heroku customers over because of our more modern capabilities and scalable pricing.

https://render.com/docs/migrate-from-heroku

You have your range of options - it depends on the size of your team, the kind of apps you're running, etc. The answer can be anything from an "ssh script" to AWS (or K8S), etc.

If you're running something that's too expensive for your taste and can share more information, happy to brainstorm some options.

AWS Elastic Beanstalk gives you more or less the same experience but charges you normal EC2 instance pricing. It's as cheap as PaaS gets.

If you are large enough, you will need an ops team to manage allowing your developers to write terraform and manage AWS costs already.

If you are small enough, you are not going to be truly affected by downtime. If you are just a little bigger, a single hot spare is going to be sufficient.

The place where you get dinged is heavy growth in personnel and bandwidth. You end up needing to solve CPU bound activities quicker because it hurts the whole system. You need to start thinking about sticky round robin load balancing and other fun pieces.

This is where the cloud can allow you to trade money for velocity. Eventually, though, you will need to pay up.

That said, the average SaaS can go a long way with a single server per product.

> I strongly suggest to throw man-days in your cost calculation.

Only if those man-days actually incur a marginal cost. If it's just employees you already have spending their time on things, then it's not worth factoring in because it's a cost you pay regardless.

From having talked to many folks, migrations are psychologically very, very, very very hard.

At least, the "fear" factor (will the new system work? what bugs will it introduce? how much time will I spend, etc.) pushes a lot of folks to accept a very big price differential aka known knowns versus unknowns...

It's understandable really. It's just that once you've migrated, you almost definitely never want to go back :-)

The longest is to adapt your app to a Dockerfile-based deployment, if it isn't already containerized. We have examples for most languages - for Flask, for example, the whole file is 10 lines long [0]

But to provision a new server, as these are "stateless" (per 12 Factor) servers, it's just 1) get a VPS 2) install Docker+Disco using our curl|sh install script 3) authorize github 4) deploy a "project" (what we call an app), setting the env vars.

All in all ~10 minutes for a new machine.

[0] https://github.com/gregsadetsky/example-flask-site/blob/main...

One thing however to note, is that by having a different non-prod and prod environment, it will be possible to test only the application, and not the infra.

Which means, that if they want to test what it will look like running in cloud for prod, they are going to either need a pre-prod environment or go yolo

I bet less time than it takes AWS to recover from a significant event. And I bet it happens less often too.

Ideally these things should go in an Ansible playbook or whatever people are using these days to manage their pets.

> you can easily forgo them and run the stack locally

Not if you're running with external resources of specific type, or want to share the ongoing work with others. Or need to setup 6 different projects with 3 different databases at the same time. It really depends on your setup and way of working. Sometimes you can do local staging easily, sometimes it's going to be a lot of pain.

Although LLM generated, https://deepwiki.com/letsdiscodev/disco-daemon is pretty impressive and has some arch diagrams. But I fully agree, we should have that on the site.

Once everything is installed/running, a very tldr diagram would be:

GitHub (webhook on git push) -> Docker swarm running Caddy -> Disco Daemon REST API which will ask Docker to build the image, and then does a blue-green zero-time deployment swap

But yeah, a clearer/better diagram would be great. Thanks for the push!

From the article:

> Even with all 6 environments and other projects running, the server's resource usage remained low. The average CPU load stayed under 10%, and memory usage sat at just ~14 GB of the available 32 GB.

Every time I've worked somewhere without one, we've wanted it and wasted more developer hours than the cost of having it trying to reproduce issues while working around the differences in the environments.

Netcup is cheaper than Hetzner, but it doesn’t have some of the other features and reviews are mixed.

I think that https://lowendbox.com/ might be a good place to start looking for that

We do have a UI, we're just so behind on the documentation, it's not even funny ha.

If you setup a server with the curl|sh install script on the homepage, you'll get a url at the end that directs you there. And you can use the CLI too of course.

But yeah, thanks for the reminder!

I don't know! I do see a Docker image for Elixir, so I'm pretty sure that would work. But the distributed aspect is harder to answer right now.

How do you typically deploy this?