huntaub
8 hours ago
Some quick questions that came up in the last post, that I wanted to go ahead and address:
How are you different than existing products like S3 Mountpoint, S3FS, ZeroFS, ObjectiveFS, JuiceFS, and cunoFS?
Archil is designed to be a general-purpose storage system to replace networked block storage like EBS or Hyperdisk with something that scales infinitely, can be shared across multiple instances, and synchronizes to S3. Existing adapters that turn S3 into a file system are either not POSIX-compliant (such as Mountpoint for S3, S3FS, or GoofyFS), do not write data to the S3 bucket in its native format (such as JuiceFS, ObjectiveFS – preventing use of that data directly from S3), or are not designed for a fully-managed one-click set up (such as cunoFS). We have massive respect for folks who build these tools, and we’re excited that the data market is large enough for all of us to see success by picking different tradeoffs.
What regions can I launch an Archil disk in?
We’re live in 3 regions in AWS (us-east-1, us-west-2, and eu-west-1) and 1 region in GCP (us-central1). Today, we’re also able to deploy directly into on-premises environments and smaller GPU clouds. Reach out if you’re interested in an on-premises deployment (hleath [at] archil.com).
Can I mount Archil from a Kubernetes cluster?
Yes! We have a CSI driver that you can use to get ReadWriteOnce and ReadWriteMany volumes into your Kubernetes cluster.
What performance benchmarks can you share?
We generally don’t publish specific performance benchmarks, since they are easy to over-index on and often don’t reflect how real-world applications run on a storage system. In general, Archil disks provide ~1ms latency for hot data and can, by default, scale up to 10 GiB/s and tens of thousands of IOPS. Contact me at hleath [at] archil.com, if you have needs that exceed these numbers.
What happens if your caching layer goes down before a write is synchronized to S3?
Our caching layer is, itself, highly-durable (~5 9s). This means that once a write is accepted into our layer, there are no individual components (such as an instance or an AZ) failure which would cause us to lose data.
What are you planning next for Archil?
By moving away from NFS and using our new, custom protocol, we have a great foundation for the performance work that we’re looking to accomplish in the next 6 months. In the short-term, we plan to launch: one-click Lustre-like scale-out performance (run hundreds of GiB/s of throughput and millions of IOPS without provisioning), the ability to synchronize data from non-object storage sources (such as HuggingFace), and the ability to use multiple data sources on a single disk.
How can I learn more about how the new protocol works?
We’re planning on publishing a bunch more on the protocol in the coming weeks, stay tuned!