Queues Don't Fix Overload (2014)

22 pointsposted 3 days ago
by locknitpicker

8 Comments

MyHonestOpinon

30 minutes ago

Back in the 90s. I was part of a team working R&D project on flexible manufacturing. One of the central concepts was the use of buffers (also called decouplers) between manufacturing stations. These buffers were the space needed to place not finished products while they waited for the next station. Typically a conveyor belt. The main service is that both stations could run at slightly different rhythm, and they would not be causing problems on the other one. Either starving it or overwhelming it. Queues are pretty similar. You can have a consumer and a producer working and the queue in between prevents them from being blocked by each other. Thus I now see the main role of queues as decouplers.

marcosdumay

10 minutes ago

Hum... The biggest hype on process engineering by the 00s was buffer size reduction. Because those buffers interfere with each other in chaotic ways, and they tend to turn "small problems that blow up soon, with small consequences" into "huge accumulated problem, that blows up hours after it appeared, with business-risking consequences".

Queues are pretty similar.

quentindanjou

23 minutes ago

That's a very good analogy. Queues are not there to solve overload (and they never were), they are there as an *architecture tool* that allows decoupling and *can* (not always) ease the scaling of the queue process (workers).

I think the back-pressure should always be implemented from the very beginning, as it also helps with defining the requirements of what the service should be able to handle

kqr

an hour ago

Once you've read this, pick up Harchol–Balter's book Performance Modeling and Design of Computer Systems: Queueing Theory in Action. It's a really good introduction to the depth of this topic, and you'll come out with superpowers you didn't have before.

hilariously

an hour ago

Even if you just read the first few chapters of this you will not come out unchanged.