The way to work around this issue is to provide a presigned S3 url

Have the users upload to s3 directly and then they can either POST you what they uploaded or you can find some other means of correlating the input (eg: files in s3 are prefixed with the request id or something)

I agree this is annoying and maybe I’ve been in AWS ecosystem for too long.

However having an API that accepts an unbounded amount of data is a good recipe for DoS attacks, I suppose the 100MB is outdated as internet has gotten faster but eventually we do need some limit

We became the flagship customer for a division of AWS that was responsible for managing SSL certificates. We were doing vanity URLs and vanity URLs generally require individual SSL certificates for each domain name. We needed thousands and AWS tools for cert management at the time was really only happy with hundreds and they had backlog items to fix it but those were behind a year or two of other work. It took them about three months to get far enough along for our immediate needs. It's surprising the parts of AWS that have not adjusted to outliers that don't seem really to be that exceptional.

I also thought Lambda looked promising at first, but we ultimately abandoned all our Lambda projects and started using containers as needed.

Lambda still requires that you need to update the Node runtime every year or two, while with your own containers, you can decide on your own upgrade schedule.

The hardest problem in computer science is coping a file from one computer to another.

Just to help future readers, there is an ecosystem of "tus" uploaders and endpoints, that chunk uploads, and feature resumeable uploads, that would be ideal for this kind of restriction:

https://tus.io/

> Like the article says, I think serverless has it's place, but I don't think it's for most applications

I feel this way about many of the more popular trends over the last decade or so.

A technology becomes popular because a really large organization uses it to solve problems that only really exist st that scale. Then they talk about how it works and many tend to take that as an indicator that the solution is ideal, regardless of context.

GraphQL, react, Tailwind, NextJS, and many others. They solve specific problems and those may be problems you have. No tool is a universally useful solution though, it takes real experience and knowledge to discern what your problems are and how to best tackle them.

Let me tell you about all the fun I'm having trying to execute my amazon lambda app locally so I can test before deploying...

That’s why Knative (Serverless on Kubernetes) accepts containers. It’s the standard packaging format that lets you lift and shift apps to many different platforms.

But they weren’t building an application. They were building a library that would get integrated in other stateful server applications not running on Cloudflare. The performance benefit comes from running their auth colocated with their customers, not anything else.

Don’t all “serverless” platforms accept docker images? I know Cloud Run does.

If I understand correctly your concern is mostly with “serverless functions” which abstracts away even more.

I've been doing AWS Lambda since it started up over 10 years ago. It solves a lot of problems for me. I don't ever have to worry about load balancing or scaling. I don't have to maintain a server. When it isn't being used, I am not paying for it. I've been running a pretty sophisticated project on Lambda for years, and I pay about $0.00/month for it. Most of the ~$0.45/mo I pay to AWS is in S3.

Lambda code is extremely easy to test locally, if you write it that way. I just run the file locally and it does what it would do in the cloud, there is literally no difference. But of course, YMMV depending on how you approach it.

I created my own build tools for Lambda about a month after Lambda was introduced as a product. It's been working great ever since. The workflow is very simple. When I update a file locally, it simply updates the Lambda function almost instantly. I can then test the Lambda live in the cloud. If I want to run the function locally, I just run it and it behaves the same way it would in the cloud. There's no need to run the function in AWS, if you write the code so it can be run locally. It's really, really easy to do, but I guess some people haven't figured that out yet.

I've never liked containers. It's always been way more opaque than writing Lambdas that can run locally as well as in the cloud.

Yeah; IMO Docker was our last universal improvement to productivity, in 2013, and very little we've invented since then can be said to have had such a wide-ranging positive impact, with such few drawbacks. Some systems are helpful for some companies, but then try to get applied to other companies where they don't make sense and things fall apart or productivity suffers. Cloudflare and others are trying to make v8 isolates a thing, and while they are awesome for some workloads, people want them to be the "next docker", and they aren't.

The model "give me docker image, we put it on internet" is staggeringly powerful. It'll probably still be the most OP way to host applications in 2040.

It’s that, and the fact that precious few people seem to understand fundamentals anymore, which is itself fed by the desire to outsource everything to 3rd parties. You can build an entire stack where the only thing you’ve actually made is the core application, and even that is likely to be influenced if not built by AI.

The industry is creating learned helplessness.

A lot of people don’t know about compilers, bash scripts and libraries.

Excerpt AWS lambda is stupidly cheap!

Helps resumes! No one got to Staff suggesting bash scripts

I don’t know if this is a good rule of thumb, I think it really depends on what you use the dependencies for, how often you need them, etc.

Consider for example a single DB dependency. Should the server be close to the DB or the client? It depends. How often does the client need the server? How often does the server need the DB? Which usecases are expected to be fast and which can be sacrificed as slow? What can be cached in the server? What can be cached in the client? etc etc.

And then of course you can split and do some things on the server and some in the edge…

You are ready for misterio: https://github.com/daitangio/misterio A tiny layer around stareless docker cluster. I created it for my homelab and it gone wild

Docker is much like microservices. Appropriate for a subset of apps and yet touted as being 'the norm' when it shouldn't be.

There are drawbacks to using docker, such as security patching and operational overhead. And if you're blindly putting it into every project, how are you mitigating the risks it introduces?

Worse, the big reason it was useful, managing dependency hell, has largely been solved by making developers default to not installing dependencies globally.

We don't really need Docker anywhere near like we used to, and yet it persists as the default, unassailable.

Of course hosting companies must LOVE it, docker containers must increase their margins by 10% at least!

Someone else down thread has mentioned a tooling fetish, I feel Docker is part of that fetish.

I’ve found this to be true, with one caveat.

Most cloud pain people experience is from a misunderstanding / abuse of solutions architecture and could have been avoided with a more thoughtful design. It tends to be a people problem, not a tool problem.

However, in my experience cloud vendors sell the snot out of their offerings, and the documentation is closer to marketing than truthful technical documentation. Their products’ genuine performance is a closely guarded proprietary secret, and the only way to find out… e.g. whether Lambdas are fast enough for your use case, or whether AWS RDS cross-region replication is good enough for you… is to run your own performance testing.

I’ve been burned enough times by AWS making it difficult to figure out exactly how performant their services are, and I’ve learned to test everything myself for the workloads I’ll be running.

The takeaway isn't that they didn't understand, it's that they are sharing information which you agree is valuable

I would not assume this was a "rookie mistake". I've been here once or twice, and a common story is that engineers don't want to do it a certain way, but management overrules them for some vague hand-wavy reason like, "This way is more modern." Another common story is that you know you're not choosing the most [scalable|robust|performant|whatever] design, but ancillary constraints like time and money push you into a "worse is better" decision.

Or maybe the original implementation team really didn't know what they were doing. But I'd rather give them the benefit of the doubt. Either way, I appreciate them sharing these observations because sharing these kinds of stories is how we collectively get better as a professional community.

My personal experience is that if you want guaranteed anything (quick scaling, latency, CPU, disk or network throughput), your best bet is to manually provision EC2 instances (or use some API that does). Once you give up control hoping to gain performance for free, you usually end up with an unfixable bottleneck.

This is basically criticizing them for admitting to being one of today's 10,000.

https://xkcd.com/1053/

Personally, I appreciate the info and the admission.

> Putting a latency-critical API on a stateless edge runtime

Isn’t this the whole point of serverless edge?

It’s understood to be more complex, with more vendor lockin, and more expensive.

Trade off is that it’s better supported and faster by being on the edge.

Why would anyone bother to learn a proprietary platform for non critical, latency agnostic service?

Agreed. Wondering what sort of discovery or design phase their legacy arch went thru.

But but it's webscale!

But when were serverless systems like lambda and cloud workers "best practices" for low latency apis?

At that point, why should I use serverless at all? If I have to think about the lifetime of the servers running my serverless functions?

Agree, it seems like they decided to use Cloudflare Workers and then fought them every step of the way instead of going back and evaluating if it actually fit the use case properly.

It reminds me of the companies that start building their application using a NoSQL database and then start building their own implementation of SQL on top of it.

In that scenario, how do you keep cold startup as fast as possible?

The nice thing about JS workers is that they can start really fast from cold. If you have low or irregular load, but latency is important, Cloudflare Workers or equivalent is a great solution (as the article says towards the end).

If you really need a full-featured container with AOT compiled code, won't that almost certainly have a longer cold startup time? In that scenario, surely you're better off with a dedicated server to minimise latency (assuming you care about latency). But then you lose the ability to scale down to zero, which is the key advantage of serverless.

Cloudflare has containers now too, and having used AppRunner and Cloud Run, it's much easier to work with. Once they get rid of the container caps and add more flexibility in terms of container resources, I would never go back to the big cloud containers, the price and ease of use of Cloudflare's containers just destroy them.

Indeed.

They get to the bottom of the post and drop:

> Fargate handles scaling for us without the serverless constraints

They dropped workers for containers.

You're saying serverless can have really low latency and fast 24/7?

Isn't serverless at the base the old model, of shared vms, except with a ton of people?

I'm old school I guess, baremetal for days...

I ported my worker project into Django since cloudflare workers wouldn’t allow selection of region for hosting workers which is generally required due to data compliances. This is something all cloud providers provide from day one yet cloudflare made it an enterprise feature.

Also the vendor lock-in doesn’t help with durable objects and D2 instead of simply doing what supabase and others are doing by providing Postgres or standard SQLite as a service.

What do you find so peculiar about it? A lot of people are running Go apps on VPSs.

Next article - why we switched from our own servers to serverless for reliability. A small performance hit was worth it.

TFA states that they’re running on AWS Fargate.

That said, as an example, an m8g.8xlarge gives you 32 vCPU / 128 GiB RAM for about $1000/month in us-east-1 for current on-demand pricing, and that drops to just under $700 if you can do a 1-year RI. I’m guessing this application isn’t super memory-heavy, so you could save even more by switching to the c-family: same vCPU, half the RAM.

Stick two of those behind a load balancer, and you have more compute than a lot of places actually need.

Or, if you have anything resembling PMF, spend $10K or so on a few used servers and put them into some good colo providers. They’ll do hardware replacement for you (for a fee).

They just use two servers and configure a loadbalancer within Cloudflare. Come on. Self-Hosting is no rocket science. You don‘t have to make it seem complicated. People have been doing this decades before AWS invented serverless.

Backup strategy? What do you mean by that?

We did initially but thought cloud flare was a better solution for scalability and latency.

We believed their docs/marketing without doing extensive benchmarks, which is on us.

The appeal was also to use the same typescript stack across everything, which was nice to work with

Have you done new benchmarks since Cloudflare announced their latest round of performance improvements for Workers?

Just curious if this workload also saw some of the same improvements (on a quick read it seems like you could have been hitting the routing problem CF mentions)

Really great writeup. The charts tell the story beautifully, and the latency gains are surely a win for your company and customers. I always wonder about the tradeoffs. Is there a measurable latency difference for your non-colocated customers? What does maintenance look like for your Go servers? I assume that your Cloudflare costs dropped?

Not a question: thanks for the writeup and for the honesty of saying that serverless is not inherently bad, just not the right fit for your usecase!

Unfortunately too many comments here are quick to come to the wrong conclusion, based only on the title. Not a reason to change it though!

Do you have a clearer picture of what use-cases you would use serverless functions for in the future (if any)?

> The serverless approach helped with GTM

Unlikely? They could've just as well deployed their single go binary to a vm from day 1 and it would've been smooth sailing for their use case, while they acquire customers.

The cloudflare workers they chose aren't really suited for latency critical, high throughput APIs they were designing.

I often don't know what to make of DHH. He's a living contradiction. On one hand he will continually rant about how bad the overhead and waste of cloud services is, and on the other hand he will staunchly defend the most inefficient programming language that is regularly used for backend development, as well as defend the enourmous overfetching that active record leads to.

Really I think DHH just likes to tell others what he likes.

Gives him a break from writing out of touch screeds about countries he knows nothing about I guess.