I am always scared to put too much trust in new Object Storage services. I love Hetzner and similar but until this new service has been around for a while, I'd only use them for stuff I can afford to lose. From the outside as a consumer of these services you do not know how robust they actually are on the inside.
I remember the data loss from OVH where they put backups in the same building as the primaries and people only found out about this once a fire took out the whole building:
https://blocksandfiles.com/2023/03/23/ovh-cloud-must-pay-dam...
"Cloud 3-2-1" to the rescue...
I take the typical formulation (e.g., [1]), and translate it into:
- Keep 3 copies of your data: production + 2.
- Keep 2 snapshots, stored separately.
- Keep 1 snapshot on-hand (literally in your possession), or with a different provider.
It's great to see more options for "different provider". If I were an early adopter, I would either target this as my on-Hetzner snapshot of my Hetzner-hosted data, and replicate it elsewhere; or I would consider trialing it as a backup to non-Hetzner data. To your point, though, I'd probably wait on the latter until it has gone through some growing pains.
[1] - https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
Years ago there was a fire in OVH who I always trusted as reliable and stable, never needed to worry.. Lost a lot that day. Probably I could have had offsite backup managed myself though. I see no difference here, you can manage your own disaster recovery.
From what I see, this actually might be a great backup/recovery solution for S3 in my case at least.
Maybe if OVH had installed an automatic fire extinguishing system or an electric mains cut-off switch, the firefighters wouldn't have struggled with meter-long electrical arcs and could put out the fire quicker.
Yes, that hosting center was unsafe as hell. All of OVH's early year featured a lot of duct-tape and daring corner-cutting - that is how they were incredibly cheap. But nowadays they are just as mainstream industrial as their competitors and the 2021 fire marked the end of an era... It is sad that it happened as they were already turning the company around to being an established player.
A great writeup of the incident and its context, three years afterwards: https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire...
[deleted]
People seem to have this misconception that "cloud computing" is some kind of magic bullet that guarantees 100% durability. News flash, your data is still being stored on physical hard drives, in some data center. If the building burns down in a fire, of course your data will be gone, the hard drives are hosed, it's not magic. You really have no one to be mad at other than yourself...
> If the building burns down in a fire, of course your data will be gone
From https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu... :
> S3 Standard, [...] redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region. An Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. Availability Zones are physically separated by a meaningful distance, many kilometers, from any other Availability Zone, although all are within 100 km (60 miles) of each other.
Most of the "big" cloud providers aren't as negligent as OVH though. That entire datacenter was just one blunder after the other (wood structures, barely functional fire suppression, no proper power shut down, insane replication strategy that caused most of the replicas to burn in the very same fire in the same building...)
I should mention Tigris[0] here. They're also a new Object Storage service, but they have this two-way replication facility with another S3-compatible service. The primary purpose they built it for is to mirror files from your existing S3 to Tigris as files are requested.
However they also have an option to copy files that are added to Tigris, to S3 automatically [1] (`--shadow-write-through`). I asked their founder if it's okay to use it as an extra redundancy continuously instead of a one-time migration, and they said they have no issues with it.
[0] https://www.tigrisdata.com
[1] https://www.tigrisdata.com/docs/migration/#starting-the-migr...
Backblaze is 3x less expensive and offers more generous free allowances.
I toured a data center in Tornado alley back when leasing cages was pretty common. I asked them about disaster planning regarding getting completely wiped off the map and they sorta scoffed at me. Literally two weeks later a tornado missed them by about a 1/4 mile. Would have loved to be a fly on the wall after that.
another story, a small company I worked for contracted with a small data center. They did things right and ran a tight ship. However something happened like a lightning strike or so other electrical fault and a critical component welded itself into a position they could t get out of. I wish I could remember the details better but they were down multiple days.
S3 durability is pretty much the only thing I consider worth using EC2 for.
S3 egress, on the other hand, is so expensive you can often justify putting a writethrough cache for your entire working set at Hetzner...
No, R2 is not an "S3 cache". It's a replacement for S3 with no egress fees.
One of my VPS was lost in that fire, but since my backups where somewhere else, I didn't lose much.
The trick is not to put all eggs in the same basket and it's easier to do that when you're paying much less than you'd pay on providers like AWS.
You know that if US East suffers a FULL data loss that the recovery would take weeks with the question if it would even be possible. That's what happened to ovh... it wasn't just one building.
> It's not like there is a mirror datacenter just two blocks away
Isn't that exactly what Availability Zones are for? They're physically separate[0] datacenters and each one contains a copy of each S3 object (unless using the explicit single-zone options)
It's also straightforward (although not necessarily that cheap) to replicate S3 objects to another region
[0] https://aws.amazon.com/ec2/faqs/?nc1=h_ls#Platform
If us-east-1 ever suffered a “FULL” data loss, it would be a company-ending event for so many companies that it would practically end society as we know it.
OVH’s failure was a single building. That’s the problem with a lot of server hosters - even Google has their availability zones all co-located in the same building, so a physical event like a fire could take down an entire region. AWS has AZs in physically separate locations, each with 1+ separate DCs.
Is the any GCP documentation on that? Sounds like a far bigger risk going with GCP than AWS if all zones in a region are in the same building.
It just caught those who rely on one provider, which is never a good practice.
rclone to somewhere else at least.
And 3-2-1 rule for backups if you're serious.
I'd probably look at using S3 as a backup and only have the working copy with Hetzner. Not sure how much more expensive that would be, the pricing for the non-standard S3 and glacier does seem to require a bit more work to understand properly.
Couldn't agree with you more. Last spring talked to a senior Hetzner engineer who wasn't too sure about the Object Storage design/implementation and was happy he wasn't involved.
As everyone knows, this is in sharp contrast with serious engineering operations like Amazon or Google, where there are no disagreements on technical designs. Ever.
Did some benchmarks, not too bad.
Benchmark finished! block-size: 4.0 MiB, big-object-size: 1.0 GiB, small-object-size: 128 KiB, small-objects: 100, NumThreads: 6
+--------------------+--------------------+------------------+
| ITEM | VALUE | COST |
+--------------------+--------------------+------------------+
| upload objects | 207.02 MiB/s | 115.93 ms/object |
| download objects | 405.78 MiB/s | 59.14 ms/object |
| put small objects | 278.4 objects/s | 21.55 ms/object |
| get small objects | 498.5 objects/s | 12.04 ms/object |
| list objects | 40977.06 objects/s | 14.64 ms/op |
| head objects | 1995.4 objects/s | 3.01 ms/object |
| delete objects | 312.1 objects/s | 19.22 ms/object |
| change permissions | not support | not support |
| change owner/group | not support | not support |
| update mtime | not support | not support |
+--------------------+--------------------+------------------+
Currently have a project using OVH, which is slower, but than I guess there's currently less load on Hetzner.
Benchmark finished! block-size: 4.0 MiB, big-object-size: 1.0 GiB, small-object-size: 128 KiB, small-objects: 100, NumThreads: 6
+--------------------+-------------------+------------------+
| ITEM | VALUE | COST |
+--------------------+-------------------+------------------+
| upload objects | 80.82 MiB/s | 296.94 ms/object |
| download objects | 274.61 MiB/s | 87.40 ms/object |
| put small objects | 31.6 objects/s | 189.98 ms/object |
| get small objects | 145.1 objects/s | 41.36 ms/object |
| list objects | 6293.39 objects/s | 95.34 ms/op |
| head objects | 177.7 objects/s | 33.76 ms/object |
| delete objects | 136.6 objects/s | 43.93 ms/object |
| change permissions | not support | not support |
| change owner/group | not support | not support |
| update mtime | not support | not support |
+--------------------+-------------------+------------------+
what tool are you using for this wonderful data?
Yes, it's juicefs objbench.
Wonderful tool. Useful for figuring out where some of these cheaper S3 alternatives make their cuts.
missed opportunity to call it JuiceFS' OJBench
AWS: $0.024/GB/month (S3 standard, first 50 TB/month)
GCP: $0.023/GB/month (standard storage, beyond 5 GB free limit)
Azure: $0.019/GB/month (hot, first 50 TB/month)
Cloudflare: $0.015/GB/month (beyond 10 GB free limit)
Scaleway: ~$0.013/GB/month (single-zone)
Backblaze: $0.006/GB/month
Hetzner: ~$5.49/month + $0.0054/GB/month (beyond 1TB free limit)
Egress costs vary though.
Working at a startup that deals in large data (our storage is measured in PB), egress is the only thing thing that matters if we're going to choose a new storage provider. And an automated class management system a la S3 or GCS.
In slightly different terms, Hetzner is roughly 5 €/TB/month of storage plus 1 €/TB of egress (ex. VAT).
In slightly different terms, Hetzner is roughly $0.006/Gb/month , similar to Backblaze
Terrific to see native object storage in Hetzner. I think object storage is become a staple for most providers now.
I can see that Hetzner is starting with WORM capabilities [0]. I wonder if this nascent product is successful they'll consider some of the features that other providers offer such as mutating objects, storage policies, tiering, and conditional writes. I appreciate you've got to start somewhere and this looks terrific for an opening gambit. Kudos Hetzner.
[0] > Object Storage is mainly used to store and share data as it is not possible to edit any data that you uploaded to a Bucket (objects are immutable). So the main purpose of Object Storage is "WORM", which is short for, "Write once, read many [times]".
> mutating objects, storage policies, tiering, and conditional writes
My only hope is that they won't make it more complex than it should be in an effort to match AWS.
Every provider virtually has to mirror AWS. The S3 API is the standard at this point. Nobody will switch if they need to replace all their code.
As someone who has managed many public Ceph clusters at Linode (now Akamai) since 2016 for both block and object storage, I wish the Hetzner engineers good luck!
There are a _lot_ of challenges in keeping the clusters secure, reliable, and performant. Make sure you have systems or tools in place to prevent abuse. Be aware of the little nuances of Ceph, like what time lifecycle policies kick off, or when dynamic bucket resharding will kick in (and block client writes!).
If possible, conduct extensive failure testing in a lab environment under simulated load to see how your clusters will really behave when it eventually happens. Triple check all of your tunables and your pool configuration. Some things like erasure coding profiles are set in stone, and once you have customer data on your clusters, there is no turning back.
One thing I noticed is that some limits seem rather low, I'm not sure if this is for the beta only or by design. Particularly the 1 Gbit/s limit per bucket and the 10 buckets for all projects.
Is it typical to have such per-bucket limits? This means that you might have to take them into account when deciding on how to organize the data to avoid bottlenecks.
Having implemented object storage at a service provider - it's extremely rare for any customer to have 1gbit/sec sustained throughput on an account, much less a bucket.
Those that do likely will already have an actual sales rep and the means to get these restrictions lifted, with a direct line to engineering.
Hetzner certainly markets to a different crowd, which I suspect is why these limits are being put in place ahead of time. It's very difficult to design around noisy neighbor, and will take some battle tested time in production to get right for the "average Joe" style customer who won't engage engineering resources to work with a provider to avoid hot spots and impacting other customers.
I'm more concerned about bursts, not sustained throughput. When handling large files a single client can load at 1 Gbit/s if the other side supports that.
My suspicion was also that they're intentionally targeting a different crowd than S3 and focusing on their niche here.
Definitely agreed. I would be pretty surprised if this was burst capacity though. I have no insight as to how Hetzner implemented their object storage, but in general terms it's pretty difficult to ratelimit burst without adverse effects. Sustained is a lot easier to implement since you are typically having to have multiple load balancers coordinate to achieve such results. Attempting to do this at sub-minute time intervals is pretty tricky at scale.
Presumably it's built on top of their existing VPS and dedicated server boxes, which AFAIK only have 1gbit uplinks, so they'd only be able to exceed that speed when the bucket is split between multiple boxes.
At the Hetzner Summit two a few weeks ago they presented the servers used for this.
I am not sure about the exact specifications anymore, but they are building dedicated servers for this with custom chassis where each server has a ton of drives and I think they each had 40 GBits networking. These are special servers that are not available to customers directly.
Hetzner VPS hosts have 10 GBit links.
Huh you're right, they say their VPSes have 10gig but their dedicated boxes only have 1gig, even the really expensive ones. That seems backwards.
The Hetzner Cloud ARM instances I believe are on 2.5Gb or 1Gb links. I've never been able to get > 1Gbit on the ARM instances, but easy getting it on x86_64.
Not quite correct. Their dedicated servers base configuration have 1gig. But when ordering you can add 10gig as an easy option (or add later).
The 1 gig is too cheap to meter with free traffic. If you choose 10gig you pay for the traffic. Those who need it might use it :-)
We had a fleet of dedicated hetzner boxes w/10 gbit, it’s just an option you get and pay a little extra for. Generally had good performance as well.
Just got a reply from them regarding the bucket amount and permission amount limits and both are in fact beta limits.
No idea about the bandwidth though.
I'm guessing that's because it's in beta, they want to set some low limits and incrementally increase it, is my bet.
A while ago I needed something managed, very cheap and S3 compatible and I went with Contabo Object Storage: https://contabo.com/en/object-storage/
Admittedly I got my money's worth, for about 3 EUR for the very basic plan of 250 GB storage, I get upload speeds of about 6-9 MB/s and download speeds of about 25 MB/s, which is okay for what I'm trying to do. That said, it doesn't seem like there's a way to create additional users with access to only specific buckets not all of them for the same service and the overall offering does feel a bit jank (e.g. when you don't have an active service, you can't log into Contabo at all and while their Object Storage site is nice, the regular VPS one feels functional but dated).
What I'm saying is that when it comes to budget options, Hetzner will probably do great, since they have a good track record and it's not like there are that many other alternatives out there!
Of course, I could have also gone with just self-hosting with MinIO, Garage or SeaweedFS, as long as the VPS that I'd get would also have enough storage. It's nice that there are self-hosted options, too, I almost dread when I see bespoke object storage solutions at work, either developed before S3 was a thing or because the devs just didn't know or had a case of NIH, so I have to look at how a bunch of blobs are passed through servlets and serialized/deserialized, as well as deal with custom metadata and permission mechanisms.
Backblaze seems to be 2x less expensive than Contabo.
Your link returns a 500 right now
They probably used chatgpt to build the product and copied-pasted
Since we're talking object storage, a question for the collective brain: are there any object storage solutions, cloud or on-prem, which support any sort of "operator pushdown"?
By "operator pushdown", i mean any ability to filter or map over the contents of the object on the server side in some way, sending only the results over the network to the client.
For example, say you have a huge CSV file of customer orders in a bucket. You might want to find the timestamp of all the orders which included a particular product. If all you can do is stream the whole file, then you need to do that, just to pick out a few timestamps. But you could imagine a kind of request where you say "only give me lines where the product ID is P01234, and only send the timestamp column". Perhaps you would express that as a pair of regular expressions, or a sed program, or a Lua script, or maybe the server would understand CSV and let you write something a bit like SQL. There are all sorts of ways it could be done. Providing a fully general way might be tricky, but it wouldn't need to be fully general to be useful.
I appreciate that if you want to do this sort of access frequently, you should probably be using a database, not object storage. But it seems like a very useful feature to layer on top of object storage, and one that feels like it should be fairly cheap to execute - the server has to do a small extra amount of computation, but then needs a lto less network bandwidth.
That's been a feature of S3 for quite a long time now, called S3 Select https://docs.aws.amazon.com/AmazonS3/latest/userguide/select...
Despite it being an awesome feature I've been itching to use, I've never actually found a use for it beyond messing around. Most places where S3 Select might make sense seems to be subsumed (for my uses) by Athena. Athena has a rather large amount of conceptual and actual boilerplate to get up and running with, though, S3 Select requires no upfront planning beyond building a fancy query string (or using their SDK wrappers)
Where S3 Select is likely to become fiddly is anywhere multiple files are involved. Athena makes querying large collections of CSVs (etc) straightforward, and handles all the scheduling and results merging for you.
S3 Select is not available anymore for new customers. Athena with columnar file format (eg parquet) in S3 and partitioning with Glue Data catalog is the solution for OP's problem. The cost of this kind of queries is very low because you only pay the actual data consumed/requested. And with the columnar file format Athena only accesses the necessary columns. And the data in the columns is usually compressed so the amount of data is even less.
> Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual.
:(
But you and patrickthebold are spot on in pointing out Athena. I've always thought of it as a database you load via S3, but of course it's equally a tool for querying data in S3.
In the AWS world, Athena does more or less what you are describing: but is is targeted (AFAIU) for large datasets. I'm not sure how it would perform on a single (small) CSV to shave off a few bytes in transit.
Look into duckdb. It's not server side, but It's pretty phenomenal for this use case.
Yeah, on AWS you can set up a Glue Data Catalog and query it with Athena.
Might be a stupid question, but why did Hetzner take off? I feel like I’d barely heard of them before last year, but now they’re the go-to hosting suggestion from many. Are they just a solid host, or do they offer something extra? Their website makes them looks almost generic, so I assume it’s not marketing.
They have been around for a long time. I heard people praising them already in startup events 10+ year ago (I'm not that active in the scene myself any more). They are known for great price/quality ratio for the dedicated servers. There are some gotchas, as you might get random crappy hardware or whatever. The firms I have been working for and which decided to go build their infra mostly with hetzner, I would estimate that they have saved millions in comparison to common cloud services. The staff often complains because managing your own dedicated servers is a bit more work than going with something like AWS, but if you put in the numbers, the amount of money saved in the end can be insane.
I was with linode for a loooong time for my personal VM-in-cloud (like 15 years). But I got an itch to check around to see if there was a cheaper option earlier this year and found that Hetzner would give me literally twice the machine for about two thirds the cost. ($6/mo vs €4~/mo, 1gb nanode vs. CX22 so 1vcpu/1gb to 2vcpu/4gb and 25gb disk to 40gb)
So far I'm happy with the service. Linode wasn't bad per se, Hetzner just offered more for less. Their admin console is a bit more spartan in comparison to Linode's, but that's not a place I spend much time anyway. If Russia glasses Finland and Germany I'd have to find another provider but I'm pretty sure I'd have bigger problems at that point than where to park my wordpress blahg and irc session, lol.
They've existed since 1997 and have always been a solid host with good pricing on both bare metal and later VPS services. They seem to have managed to survive the Cloud onslaught as well by continuing to provide reliable service for good prices.
Perhaps you heard less about them because they're based in Europe and thus less hype sorrounds them?
They became a serious VPS contender amongst DigitalOcean, Linode, and other similar providers. DigitalOcean raised prices while Hetzner gave more powerful server capacity (read double) at lower price points. Hetzner has been increasingly popular in self-hosting communities.
I started using their services since last year for some of the operations, including business, and it's been rock solid. I plan on increasing usage, but will hold on this new object storage offering.
The pricing is much better than other providers, while being a reputable company. Pretty much the best value for non-enterprise workloads.
I've associated them for a while with uber cheap hosting, while also being allegedlly* reasonable (Most of the time) and reliable
* Couldn't tell you, as they flagged my account and asked me for an ID Document I couldn't provide, Maybe I could have sorted it with Customer Service, but instead I just picked a different german hosting
Their accounts department is egregious. Amazing offerings but it's so frustrating.
I had some servers that I had autopay setup for I thought. But the bill went over as unpaid and they shut off the servers. Then I noticed & tried to login to their web account system but my account was locked. Support wanted me to send a wire transfer. I begged and begged, told them how it would cost as much as by debt, and they opened my account for ~36 hours, but after some delay, and I was away vacationing. I sent them the wire transfer almost two months ago now & still nothing from their accounts/billing people, even though the transfer had the account & payment #'s. It's been so frustrating, & so unnecessarily over the top bad & they seem to love making it worse every step. What the hell Hetzner?
I've been with them ~15 years, and they have been solid. Back then I had a dedicated server I was developing an online multiplayer java game on, great fun! until the whole java webstart certificate fiasco. A few years back I was porting that old game over to wasm, and they added the mime type support within a few hours of asking. They are some very knowledgeable engineers, and maybe that is reflected in the simple website design. But I like that simple productive interface, and dread the day the MBAs or marketing teams get their hands on it.
They are cheap. People have more of a focus on cost recently which pushes back against AWS. As far as vibes go, more people seem to be getting back into managing their own machines and self-hosting. I think Linode used to fill that void but for various reasons they relinquished that opportunity.
It's very cheap. They've been around for years. I've been using them for my personal server for the past 2-3 years without problems. The only problems I have seen from them are the control plane API going down (so, like, creating new servers isn't available).
Hetzner is well known and beloved in Europe for a decade or two, so it didn't really start out of nowhere. Maybe you (and most of HN) became aware of them, when they opened their American datacenters in 2021 & 2022, and became a viable option for people outside of Europe.
I don't know about the last year, maybe they ramped up their marketing. But they've been pretty popular in my circle for the past 6-7 years, mostly because price/performance ratio is really favorable. I use it for the past ~3y.
great value and reliability
I got a pretty nice dedicated server with 2x3 TB HDD + 20 TB/month of traffic for just 25€/month. Was great for hosting >1 TB of video content publically.
So, I don't want to sound like a conspiracy theorist, but this announcement has extremely suspicious timing.
You see, I host a cloud storage service on Hetzner. I have 12 PETABYTES of user data stored there. The data is spread over 120 of their SX type storage servers. Last week Hetzner sent me an email saying they were closing my account for an unknown reason. I have repeatedly been asking about the reason of this sudden closure, but they won't tell me. Meanwhile I have a huge problem, because I need to move 12 PB to a different hosting provider, and they only gave me until the end of November. That's an almost impossible deadline for setting up such a large storage cluster. Especially considering that Hetzner's 1 Gbps port speed makes it impossible to transfer the data in less than three weeks.
Don't trust Hetzner, guys. They screwed me over real bad here, and it seems like they will be taking the hardware that I have been renting there for 10 years for themselves now. This is incredibly scummy behaviour.
> extremely suspicious timing.
I've been wondering for well over a decade why Hetzner didn't get its ass in gear and start offering AWS-like services. Instead all they've offered for years is remote control of VMs. Even now, this new S3 offering is very little, very late -- 18 years after Amazon first offered it and took off like a rocket.
The only odd thing about the timing here is what was keeping them.
> Last week Hetzner sent me an email saying they were closing my account for an unknown reason.
Frankly, your service looks a bit suspicious (free and cheap file sharing). You seem to be aware of the potential for abuse - your DMCA/abuse page even says there are a "large number of abuse reports pixeldrain receives every day".
It sucks that Hetzner is closing your account. But, they are known to be pretty conservative and your users have almost certainly violated their T&C many times even if you are making a good faith effort to prevent and respond to abuse.
I have not used Hetzner to serve user facing traffic for many years. Users never connect directly to the storage servers, it's all going through custom caching nodes hosted by a different provider. That shields the hetzner servers from all abuse and also reduces traffic by about 75%. There is no risk for hetzner here.
I concede that my service has been abused a lot. I am working very hard to clean up my act. I have been implementing better content moderation, content scanning and streamlining my dmca handling process. If hetzner has a problem with any of that they should have just contacted me instead of pulling the plug like this. Very unreliable company.
I suspect that what is happening here is that Hetzner received a lot of orders for storage node snapshots and/or evidence preservation from law enforcement (you probably won’t even be notified).
You must have done something really wrong if they don’t even want to resolve this with you. Normally, you would receive an abuse notice from them — we did many times with one of the services I worked on.
I guess your service is somehow connected to other Hetzner servers (not storage), or you are somehow leaking Hetzner IPs. Your servers were probably used to host CSAM or similar content, which is likely why they don’t want to communicate with you. Someone must have reported this to them.
So you're paying them between €15,000 (€124 * 120 servers, hetzners cheapest sx server) and over €30,000 per month, and you really believe they sit in their little office trying to find servers of already paying customers, just to snatch them and host their own S3 service?
This doesn't sound like a conspiracy theory to me; it sounds more like a serious case of megalomania.
That they don't communicate well with you really sucks, and I will read your blog post. But please stay on the sane side of reasoning.
Have you tried calling them? Could it be linked to pixeldrain and some illegal files which were uploaded and they simply don't want to take the risk?
I'm currently on vacation with bad cell reception. I tried calling but it didn't come through. Will try again when I get back.
Hetzner does not host user facing servers for me. It's all behind caching nodes which are hosted by a different company. Hetzner would not know what is hosted on pixeldrain.
Hope you get it sorted and would be interested in a follow up. Double check your contract regarding cancelation periods and maybe get a lawyer if nothing else works to get an extension on your period.
That's terrible. Please consider documenting everything and blogging about this later on.
I will, and when it's done it will definitely appear on this forum. But in the meantime I have to focus on cleaning this mess up.
Thanks for the warning, was just about to start moving there.
As someone who has used Hetzner in one form or another for close to two decades, I frankly find this behaviour shocking.
Besides the normal support channels, try reaching out on their subreddit and also post in the customer forum - to make this more public.
If they don't want your business, fine. But giving you only a couple of weeks to move 12 PB is unreasonable.
>Hetzner's 1 Gbps port speed makes it impossible to transfer the data in less than three weeks
But don't you have that spread over 120 servers?
Each server has a 1 Gbps connection. Some servers are storing more than average, some have 160 TB. That's already 2 weeks at full line rate. But I also need to keep serving user traffic. I can't take the site offline for a month for a data migration, that would completely kill my business.
I'm guessing you're banned because you have such amounts of data/servers and push a substantial amount of bandwidth under the free "1gbps" card for each (even if the cache layer handles most requests directly)
perharps if you moved all servers to 10G and payed for the egress over 20Tb they would reconsider your account?
They still unreliable pulling stuff like this but difficult to find these hardware options off the shelf, ready for order, at this rpice
Linode also has S3 compatible storage now, Backblaze B2 buckets have S3 access too
Wow, I just saw "Hetzner goes Singapore", have been waiting for a decade.
Unfortunately their signature dirt cheap bandwidth isn't so cheap in Singapore, you only get 1TB inclusive instead of 20TB and the overage rates are 7.4x higher than their EU and NA datacenters.
That's a problem but still cheaper than UpCloud. I do find Cloudflare pricing is more attractive for read heavy website and startup plan is useful for us.
IONOS VPS is free egress.
I trust cheap a lot more than free, "free unlimited" anything usually just means there is a limit but you're not allowed to know what it is.
I am trying to register on Hetzner and not able to. I get the account verification email but after verification I am not able to login to the account and also do not receive any Hetzner email with login or account information. Strange. Yeah, I am from from India, so maybe that's why. Anyone else? Who is not able to register?
We've been using Hetzner for years for Pirsch [0] now, and so far we had a great experience. I migrated all of our data from AWS S3 to their new object storage without issues. We only use it for user pictures and small files (white-labeling logos and such).
This is one of the rare cases where AWS is actually cheaper for us. It's probably more worth it if you have a lot of data. The only thing we're missing now is SES, and then we've fully migrated away from AWS :)
[0] https://pirsch.io
The 1000 requests per second per bucket limit feels pretty small.
Oh that’s fun. I can see heztners brand of cheap but decent doing well in this space.
Keen to see what is available to protect public buckets though to prevent huge bills from malicious actors
Cheap is a bit pejorative, I’d call it reasonable or competitive, ie. not exorbitant prices.
They know what they’re doing and they’re doing it well, cheap is more side effect than primary goal.
Cheap is a perfectly fine way to say low in price and you knew exactly what I meant...
I know what you meant, just saying that somebody reading "brand of cheap but decent" may read as "brand of cheap stuff" and for clarification that's not what they represent.
What does this kind of stuff run on the back end? Is it Ceph? Is it something custom like BackBlaze B2?
1k rps per bucket though ...
How does Hetzner compare to DigitalOcean? Lately I have seen a lot of people pushing Hetzner.
I don't understand the pricing. There's a base fee of 0.0081 €/h, which I guess then is translated into 5.00 €/month? Or is that 0.0081 €/h + 5.00 €/month? And that gives me a 1 TB-hour of free usage, and if I use the object storage for the whole month, it's 720 TB-hours of free usage, except if the month has more or less than 30 days (so it's actually between 672 TB-hour and 744 TB-hour of free quota of storage)? And those TB-hours expire at the end of the month, so you might as well store files if you're under a 1 TB?
Amazon S3 pricing looking more and more sane.
Agree that the pricing model is highly unclear (which is usual for cloud services).
It does not tell me if I should count hours in hour-of-the-day, or lapsed time. Also the example tries to demonstrate a case of "you don´t have to pay extra", but then falls silent. Nice, not the info I am looking for.
What about envisioning a customer who asks «what am I going to pay? Specify it right now, right here».
[deleted]
It's not explained very clearly and made more complicated by being charged hourly instead of monthly, but essentially there's a minimum charge of €5 per month, which includes 1TB of storage and about 1TB of bandwidth.
That's not how I read it or what their example says. As I understand it, if you create a bucket and then delete it again within the hour, you only pay for one hour. I think the 5€/month is if you have active buckets the whole month, since it's less than if you actually had to pay for all the hours.
30 * 24 * 0.0081 = 5.83
28 * 24 * 0.0081 = 5.44
That's correct, that's why the hourly billing makes it more complicated. If you don't use cloud storage the entire month you only pay for the time you do use it, and you only get included quota proportional to the amount you pay for. The sum of hourly costs is capped at €5 per month, which is also how it works for their VPSes.
Def needs a calculator if they intend to stay with this model at all IMO.
Yeah we are big fans of Hetzner and this pricing model makes no sense. I'm still trying to wrap my head around it and the strange limits associated with it.
Seems you can only move like 1TB/mo of data though? And a limit of 24TB of storage per account is also quite low.
Where are you seeing these limits? I'm seeing:
Up to 10 TB per object
Up to 1 GBit/s bandwidth per Bucket
Up to 1024 operations/s per Bucket
Up to 100 TB per Bucket
Up to 100,000,000 objects per Bucket
Up to 100 S3 credentials across all projects
Up to 10 Buckets across all projects
$5.29 per TB/mo for storage. Backblaze is at $6, so not revolutionary.
… might be revolutionary for backblaze…
How much debt do they have on their balance sheet?
It is for people using hetzner for stuff.
I run minio on hetzner and wouldn't send stuff across some other network to backblaze.
If you self hosted minio on their SX nodes you could prob get close to half
I was thinking of switching to BlackBlaze as I'm currently using Digital Ocean Spaces -- however I might look at Hetzner again -- not revolutionary but quite competitive.
b2 is not that good
We use it for multiple use cases and all of them require lots of retries and error handling
essentially they're build for backups use case, with lots of spinning rust, any kind of "working data" easily underperforms
backups work fine because not time sensitive and retries handle the problems
1€ / TB egress is extremely attractive, though. Most other "budget" providers charge at least 10 - 20$.
Backblaze egress is free up to 3x the average stored amount, which covers backup recovery at least.
cloudflare r2 has free egress
I know, but I prefer fair pricing over free in situations like this. There are plenty of stories going around of CF forcing users to upgrade to an enterprise plan due to their usage. When there is a price tag, at least I know that won't happen to me (not that my usage would be on CFs radar anyway, it's the principle of it).
Works good. Not so cumbersome options as other providers.
Is this new or just an advertisement?
It must be new - the page says they're in beta since two weeks ago and expect it to take a month or two to hit production.
It's nice to see more competition in this space.
Pretty new. I enrolled in the beta a few weeks back, they will go GA later.
I'm a big Hetzner fanboy, quite sad that pricing is't that competetive...
What is the difference between an "object store" and a file system if it runs at speeds that a basic filesystem can easily match?
Locking semantics? Consistent listing of large numbers of files?
I'm speculating but those are at least the two I can think of that aren't explicitly linked to speed equivalency of a basic filesystem.
If it's on a server on the internet how would anyone know the difference?
i was interested in hetzner and their services, until i tried to sign up and, after they verified my payment card and all went well, i instantly got account suspended email and that was it. and i am glad it happened. i cannot imagine having a tiny problem later on, costing me money, with a company like that.
btw backblaze is the best object storage offer on the planet at this time(i have research all options). second would be wasabi.
I’m a wasabi customer, why is backblaze better as per your research?
Backblaze gives you 3x free egress of your stored data per month. Wasabi gives you only up to 1x. So it's like you have 1 TB of data and you can download 1 TB each month via Wasabi for free, but 3TB with backblaze. Since traffic is the most costly expense when it comes to object storage, this is priceless.
This is highly dependent on the use case. Backblaze gives you 3x free egress, but after you hit that you pay for any additional egress. Wasabi has terms of use not to exceed 1x your storage, but is not in their model to pay egress. As a long-term storage user, you can restore a full system-wide backup without any concern of charges at Wasabi(typically, you're not restoring everything, just recent backups), as long as you are not consistently doing it. Backblaze will get you more egress for sharing data over the public internet, but if you need more, you will get charged.
> typically, you're not restoring everything, just recent backups
Not knowing the pricing difference between the two and assuming they are similar, I would favor Backblaze as it would allow me to exceed the limit if I needed. Based on how you framed it, I would expect that with Wasabi you might hit a hard limit.
https://wasabi.com/glossary/egress-charges
Wasabi doesn't have an egress hard limit and doesn't charge for egress. You get consistent pricing, and if you need to recover everything, it's not an issue at all, and you won't pay for it.
"The reasonable use egress policy indicates that if your monthly downloads (egress) are greater than your active storage volume, then your storage use case is not a good fit for Wasabi’s free egress policy, and we reserve the right to limit or suspend your service."
If you're using Wasabi for normal backup data storage, you shouldn't worry about egress. It is meant to prevent malicious users from uploading data and using up all the egress bandwidth, for example, a 500GB user egressing 5TB with public access or using Wasabi as a dump point to upload in one location and download in another region 1:1 ratio. As your storage goes up, your available consistent monthly egress goes up. It only becomes an issue if you abuse the account by uploading/downloading in a 1:1 ratio on a consistent basis.
What happens is you get an email from support asking if something changed in your use case. If so, they will help troubleshoot it(Think of a CDN scenario, where the CDN gets misconfigured). You also have an egress monitor for suspicious activity in case you aren't normally downloading all your data, and then you see a rise in egress https://docs.wasabi.com/docs/en/whats-new?highlight=egress#e....
problem with wasabi is that their ToS are not concrete with their conditions. backblaze is specific. black on white. no buts of ifs. you know exactly what you pay for. with wasabi, you don't. if you run a business, wasabi is not the way to go. if you run personal things, it is an ok option. but in the end, 1x vs 3x is such a major difference, that there is just no point in discussing it.
Will they delete the entire account if I store the banned word "bitcoin" as an object? (They probably will.)
If you're trying to sensibly engage people in discussion about this then you probably want to provide some context. I personally have no idea what you're referring to.
The context is that Hetzner is well known to delete accounts of users that they suspect of running processes or storing files that have the word "bitcoin" or any other word that Hetzner does not approve of. Hetzner does this without warning too. It doesn't matter if there is no network service, as local processing is sufficient for this action.
> Hetzner is well known
Well - not that well known.
I can find references to less egregious incidents - i.e. they ban running actual crypto software. Do you have any links I could read to find out more about the more extreme end of your claim? i.e. " storing files that have the word "bitcoin" or any other word that Hetzner does not approve of"?
If this is true then it's a reason for anyone to avoid Hetzner as accidentally triggering it would be possible on almost all websites. If you're exaggerating for dramatic effect then that's fine I guess - I'd just like to find out the real state of affairs.
I am not exaggerating it. I stand by what I said. My comment has nothing to do with running actual crypto software, implying that you don't have to be running it to get your services wiped. You also don't have to be using the internet. The ultimate discretion is up to the mood of Hetzner staff and up to when they get around to seeing what you do. If the word "bitcoin" occurs more than randomly in your usage, expect it to be ended without warning. I don't believe it to be an automated process.
they also block external servers on their network that are somehow related to crypto
so if you have some kind of service/api/trading bot/scraping/monitoring hitting a somewhat related to crypto node you may find that you will have no routing from hetzner core router to said server
and support is as friendly as other comments describe :)
> and support is as friendly as other comments describe :)
In my experience, the support is not friendly, but it is no nonsense. Every time I've reached out, they responded quickly, tersely, and took appropriate actions. While I don't personally mind formalities, there's something to be said for their efficacy.