abronan
7 hours ago
Hello HN,
I’m Alexandre, I have worked on Docker Swarm and the integration with distributed key value stores (etcd/consul/zookeeper), but also on Docker Swarmkit and the Raft distributed datastore, with the integration of etcd's Raft library.
Last July, I open sourced Mantissa (https://mantissa.io): a fully distributed scheduler and container orchestrator. I did this quietly to keep working on patches and stability. It is still at an experimental stage and will probably stay that way for a while, but could be tested to get a "look and feel" of the project.
Mantissa is using CRDTs[1] and Merkle Search Trees[2] for its data replication strategy (via anti-entropy/gossip). It is a bit of a departure from existing systems that are relying on strong consistency/linearizability (like Kubernetes, Swarm and Nomad): Mantissa focuses on fast convergence and self-stabilization instead. This comes with plenty of benefits (no quorum and Primary/Follower setup, simplified deployment and upgrades, etc.) but also trade-offs (increased storage size, no resource overcommit, temporary divergences in or outside of partition events, etc.) that would be too long to list and detail here.
Mantissa is a particular fit for small clusters, home labs, agent orchestration but could also be used in larger settings (I’d be comfortable to say up to a hundred nodes per cluster view/shard, but I’m making no claims for higher than this, since I don’t yet have access to hardware that would allow me to test and make such a claim). At the core was the idea that it should be trivial to interconnect machines together and create a transparent overlay on top of the operating system, regardless of the scale.
It is sort of a spiritual successor to Docker Swarmkit: going even further with simplification and reducing the deployment and maintenance burden. You'll see that it retains some of its design elements (with the node join flow) while taking inspiration from other systems like Google Omega[3].
Why building Mantissa while Kubernetes and other systems exist? I think it is worthwhile to experiment with new approaches and see how it may impact existing systems. Improving on the UX/DX of schedulers and orchestrators has been a long obsession of mine and Mantissa is a long story, with its initial codebase and architecture being 10 years old. The scope is incredibly large however and working on this alone is quite difficult even with the help of LLMs. AI is a double-edged sword, and I could write a full article around my frustrations with building complex distributed systems with LLMs (they have a hard time “reasoning” correctly about systems relying on convergence for example).
I’m having a lot of fun building this however and I hope this could be useful to some of you, despite its current limitations and experimental status. Feedback is also welcome to help steer the project in a direction where it could be the most useful/impactful to you.
Cheers.
Github: https://github.com/MantissaLabs/mantissa
[1] https://www.lip6.fr/Marc.Shapiro/papers/RR-7687.pdf [2] https://inria.hal.science/hal-02303490/file/paper%20%281%29.... [3] https://research.google/pubs/omega-flexible-scalable-schedul...
verdverm
6 hours ago
I would think eventual consistency would be a big problem for orchestration. I am not seeing how it makes sense, too many edge cases that would make it even harder to reason about fleets, versions, load balancing, and the like - and then make sure what you want is actually what the system has.
I recall a google paper with a tl;dr like "distributed data; central control"
abronan
5 hours ago
It’s a fair concern, and Mantissa is really just moving the complexity around.
One detail is that it relies on Strong Eventual Consistency, which is a slightly stronger model than Eventual Consistency. Replicas that have observed the same updates are guaranteed to converge to the same end state, regardless of the order in which those updates have been received.
That said, convergence alone doesn’t guarantee scheduling correctness. This is why it separates convergence from scheduling coordination. The CRDTs and MSTs provide the replicated state, and the scheduling protocol handles placement decisions and conflicting operations (with atomic slot reservation that is node local). There is a reconciliation loop (owned by a single node that is picked for the service/workload to deploy) that is then responsible for bringing the cluster back to its desired state if it potentially made a wrong call.
And it comes with trade-offs as mentioned in my post. You accept temporary divergences rather than requiring global agreement before every operation. This may or may not be acceptable for certain workloads. This removes the need for a quorum-based and centralized control plane, but it also means that certain invariants and guarantees have to be enforced at the scheduling and workload layer rather than by the datastore itself.
Strong consistency is also not a silver bullet to handle distributed systems. It is relatively hard to get it right and not serve stale values because of the presence of bugs in the protocol or state corruption. It could also bring the availability down and prevent from scheduling any more tasks due to a loss of quorum (which is by design and acceptable in most cases).
Mantissa is just making a different trade-off and accepting a different set of problems to solve: favoring availability and accepting that the desired state could shift temporarily on partition and network blips. If your cluster has good connectivity, then it should be a rare event, for which you should have prepared with some replicas that could handle being moved around. Again, it is not for everybody, and will probably remain a niche tool for users who want a simplified cluster deployment/updates to schedule mostly stateless tasks (even though stateful tasks are supported).
One remaining comment: replicated volumes do use a Raft quorum per volume. There are things that couldn't accept anything weaker than strong consistency, and volumes are one of them. So we are using strong eventual consistency for most of the state replication, but fallback to Raft for things that truly require strong consistency.