Ask HN: Distributed data centers in our basements

28 pointsposted 6 hours ago
by cmos

Item id: 47587597

46 Comments

Aiolo

4 hours ago

In France, there are at least two companies that are trying (or tried) to commercialize something with a similar idea : domestic radiators that produce heats from embedded computers that are used as cloud infrastucture. - https://www.hestiia.com/en for the end-user market - https://qarnot.com/en that seems to have since pivoted to low-carbon footprint HPC (was mentionned here -- in French -- as doing computer-based heaters : https://www.takagreen.com/solutions/qarnot-radiateur-ordinat... )

Aiolo

4 hours ago

And also, they are a lot of project to redistribute heat from data centers into city heat distribution systems. A data center for Equinix, for example, redistribute the generated heat into SMIREC heat network near Paris. This heat network is used, among other building, to heat an aquatic center that was used during the Olympics for Water Polo, diving and artistic swimming.

https://www.engie-solutions.com/fr/references/chaleur-fatale...

8jef

4 hours ago

That's a great idea. I see at least 2 difficulties emerging: first security, then servicing.

No private or public entity will grant access to valuable proprietary hardware, as unacceptable risks will not only come from building owners, but also from anyone entering premises.

Also, managing remote nodes evenly spreaded across all areas will be costly. Think of armies of techs on the road permanently, with access problem, dogs or pest barriers, and so on.

A way to solve this would be the allocation of a planned space per block everywhere, which would be safely secured - then available and accessible to all utility organizations: electric, isp, water, phone, data, etc. Heat, power, mini data centers, and such could serve all buildings on a block.

Then other problems emerges: having utilities plan and use these together. Would only work if all services belong to the same entity.

A way around, of course, would be for individuals to setup servers they would own, and rent to data brokers, like Holo project once planned for.

deelayman

3 hours ago

There needs to be incentives for people other than the distributed system users to participate as hosts. Risks also need a way to be offloaded cheaply by the hosts.

Risks: Co-mingling your home's ISP with the basement rack seems like a surefire way to get your personal devices blocked if external basement rack users are running a VPN through it and doing heinous stuff. Annoying, maybe solvable with an ISP device reboot. But that particular risk is worse depending on whether the host's jurisdiction allows the assumption of identity based on IP. Risks around general liability. Risks around tax implications when internal revenue folks see the opportunity to collect capital gains tax on your income generating property. So many risks!

The only encounters I've had with companies trying to incentivize this type of setup are Storj and Sia - both pay their host operators in cryptocurrency, which is just another risk IMO. Despite my own involvement with Storj, generating enough income to offset my energy bill by about 25% monthly, the implementation that wins out and gains wide traction has a lot of groundwork to lay for those utility contracts, risks, and incentives.

teekert

7 minutes ago

Where I live this would be nice for about 6 months a year… Can they absorb heat the other 6? …Or, heat the shower water perhaps.

comrade1234

5 hours ago

Does your house have redundant power connections to the grid and a failover generator?

That said, my plex server for my friends is on an ups and I'm on 1Gb fiber and I have better uptime than AWS.

troyvit

5 hours ago

How distributed would it have to be to make up for the lack of redundancy? DDoS attacks work for a reason, so how feasible would it be (if you had massive buy-in) to scale tiny data centers? I honestly don't think that feasible, because you can't get that massive buy-in, but I'm curious what others think.

trollbridge

5 hours ago

For many types of workloads (like AI inference), high availability is not needed for individual racks.

gaws

5 hours ago

> I have better uptime than AWS.

You're not serving tens of millions of people.

bombcar

4 hours ago

You don't know how many friends he has!

BillTthree

6 hours ago

Why don't we all have solar panels on our roof to generate electricity for ourselves?

Why don't we all have small farms on our properties, turning lawns into vegetable producing land for each household?

Why don't we have small datacenters on the property of each business, so the business users and IT folks can keep track of their own servers and data and applications?

kube-system

5 hours ago

> Why don't we have small datacenters on the property of each business, so the business users and IT folks can keep track of their own servers and data and applications?

These are often called server/network closets, and they're pretty common, but the trend has been to move away from it because they are a PITA to manage and it is cheaper and easier to manage at DC scale.

Implement7347

3 hours ago

It might work, but I see the security aspect (physical security) of the data center won't be respected in a normal house basement, unless of it's only intended for personal use, but still the electricity bill might be too high for a house

kube-system

5 hours ago

Many reasons prevent this from being practical for any serious purposes.

1. It depends on what part of the world you are in, but many homes have cooling needs for at least part of the year. The needs to remove excess heat would go up if you are adding more heat -- and it is less efficient to do this at the scale of an individual home than it is at DC scale.

2. Power requirements: While many homelabs have UPS systems -- they lack often lack backup generators, redundant A+B power infrastructure, and don't have the required power density for higher powered servers.

3. Connectivity requirements: most homes don't have access to the connectivity that data centers do.

4. Security requirements: homes simply can't meet the security requirements of most data protection regulations -- things like barriers, access control systems, surveillance, fire protection -- are anywhere from intrusive to completely impossible in a home.

5. Access requirements: homes aren't conducive to a technician responding to an outage at 3am

And those are just the big ones.

WithinReason

5 hours ago

1. Many places in the world don't ever need cooling

2. If servers are distributed then downtime is distributed, you can virtually guarantee that some of the servers over the world will be online so you can get effectively 100% uptime, something that is not possible in a data center

3. To serve tokens you need very little bandwidth, it's just text in and out

4. All of this is down to the HW and the SW itself, not the building. That is, the box that's being deployed.

5. Just switch to a different server until the problem is resolved, in this model there is no urgency. You just need redundancy which you can afford with how much cheaper this would be.

kube-system

5 hours ago

> 1. Many places in the world don't ever need cooling

And data centers also exist in cold places. But if you put 8kw of extra heat in someone's home that previously didn't need cooling, it might need it now.

> 2. If servers are distributed then downtime is distributed, you can virtually guarantee that some of the servers over the world will be online so you can get effectively 100% uptime,

You can! But running more servers with worse uptime is less efficient and requires more capital expense than running fewer servers with better uptime.

> something that is not possible in a data center

This is not only possible, this is how the large clouds are architected. This is what availability zones are for.

> 3. To serve tokens you need very little bandwidth, it's just text in and out

bandwidth is only one of the many connectivity advantages that datacenters provide... and LLMs are a bad choice to run residentially for other reasons, particularly power density

> 4. All of this is down to the HW and the SW itself, not the building.

Absolutely not -- basically all industry data protection standards have physical security standards. At least, any of the ones that matter.

> 5. Just switch to a different server until the problem is resolved, in this model there is no urgency.

That is true, there are data centers without 24/7 access. They tend to struggle to compete, though.

> You just need redundancy which you can afford with how much cheaper this would be.

Is it? Residential power and cooling costs more -- and that's the majority of the cost to colocate servers

WithinReason

5 hours ago

> 1. And data centers also exist in cold places. But if you put 8kw of extra heat in someone's home that previously didn't need cooling, it might need it now.

That's the entire point of being in a cold place that you don't need active cooling. Just open the window.

> 2. But running more servers with worse uptime is less efficient and requires more capital expense than running fewer servers with better uptime.

Even if the cooling is free? Not even free, the cost is negative since it saves heating cost.

> 3. and LLMs are a bad choice to run residentially for other reasons, particularly power density

Can you explain the connection of LLMs to power density? This point makes no sense.

> 4. Absolutely not -- basically all industry data protection standards have physical security standards. At least, any of the ones that matter.

You can lock a box physically

> 5. That is true, there are data centers without 24/7 access. They tend to struggle to compete, though.

Why though if redundancy exists, like you said? Would they still struggle to compete if the cooling cost was effectively negative?

> 6. Is it? Residential power and cooling costs more -- and that's the majority of the cost to colocate servers

You can make cooling cost negative, if that's the majority of the cost, then that's great! And you can also place your servers in residential areas with the cheapest power.

kube-system

4 hours ago

> That's the entire point of being in a cold place that you don't need active cooling. Just open the window.

> Even if the cooling is free? Not even free, the cost is negative since it saves heating cost.

Again, having cold air outside is not unique to residential homes. Locating somewhere cold is a strategy for cooling data centers as well. But it doesn’t make environmental management free. You still need to control humidity and move heat. You can’t just run a server outside. However, it isn't the only concern for hosting a compute workload.

> Can you explain the connection of LLMs to power density? This point makes no sense.

The system requirements for a single server to run a frontier workload is a system that would overwhelm the power requirements of practically all residential electrical systems.

> You can lock a box physically

If only ISO 27000/SOC/NIST SP 1800/PCI DSS/ etc were all that easy lol

> Why though if redundancy exists, like you said? Would they still struggle to compete if the cooling cost was effectively negative?

Because of the additional capital costs associated with buying more servers, the additional operational costs of inconveniencing your employees, and the additional operational costs of powering/housing servers that are down.

> You can make cooling cost negative, if that's the majority of the cost, then that's great!

It isn't, most power in a data center is spent to power compute. Even if you do harness waste energy (which some data centers do), it is at best 100% efficient. Residential heat-pumps already have effective efficiencies better than this.

> And you can also place your servers in residential areas with the cheapest power.

And in those places, industrial rates are typically even lower.

Scale is always cheaper.

brianaker

4 hours ago

If you don't need to manage cooling?

Then you are likely needing to manage humidity.

bognition

5 hours ago

One of the key problems you have to solve is the how do you execute code on an untrusted device. The major cloud providers do a ton of work so you can "trust" the compute you pay for.

Without a truly zero-trust compute platform its going to be difficult to get anyone to trust their workloads to a random compute resource that isn't carefully guarded.

borlox

5 hours ago

We’re currently planning on a datacenter to give its heat to the municipal district heating. That combines your idea to use the heat loss for heating the people’s living space with the needs of a data center like access control, redundancies, interconnections.

GTP

6 hours ago

You would have issues with providing the reliability levels (read: SLA) that we come to expect from data centers. But, if there are enough services that we don't care about if they go down for a few hours, this could be doable. It still relies on the assumption that we got enough services to justify the effort though. It is way more realistic if you set up your own homelab and provide services to your family, under the caveat that they may go offline every now and then.

bognition

6 hours ago

It turns out, if you build it correctly, you can get BETTER reliability SLAs. There's a company https://www.storj.io/ thats been doing this for years.

kube-system

5 hours ago

I doubt it, there are data centers with several decades of 100% uptime.

People often think of the large cloud providers when they think of data centers -- but their data centers are typically mediocre in terms of redundancy and uptime. Their strategy is generally to have less infrastructure redundancy and rely on software failover... e.g. failover to another AZ

seeingnature

an hour ago

At least one company is working on it. Https://leaf.cloud

bilekas

6 hours ago

> It seems like a lot of the blight of data centers is the energy to remove the heat.

Not really the only issue actually, the electricity bill would be astronomical for a household and also have you heard the noise from them ?

Issues with them being distributed range from Data protection to Insurance against damage, connectivity issues. Noble maybe, but it's widely unrealistic.

iamnothere

6 hours ago

A half rack server in a basement isn’t going to consume a lot of power or generate that much noise. I have home servers and they are fine.

Data protection is an issue, but maybe this is something that SGX and family can provide eventually.

A scheme like this makes a lot of sense for distributed redundant backups.

The real problem is bandwidth. Most home users still don’t have decent symmetrical bandwidth. If you could solve this, then home servers could provide a handful of edge services to others in the area. I’m not sure where this makes sense versus local colo though.

kube-system

5 hours ago

I've had a half rack in my home for many years -- if it's half empty, half powered off, and half full of low powered stuff, then it's not going to consumer a lot of power. (e.g. 'only' a few hundred watts) But 20u of 1u virtualization servers filled up -- or a single Nvidia DGX -- would easily overwhelm a normal home electrical system. The kind of workloads in datacenters are not people homelabbing for fun, but people running production workloads.

bilekas

6 hours ago

> A half rack server in a basement isn’t going to consume a lot of power or generate that much noise. I have home servers and they are fine.

I have home servers, designed for home and they are not too bad, and I can turn them off when sleeping for example.. It's very different with a 20U server running and spinning non stop. Not many people will have the soundproofing to simply not hear it at night.

I don't know, I wouldn't see it working, but I'm just one.

aaronax

5 hours ago

A half rack in a rack of last decade is 8kw. A half rack of today's state of the art is 100kw.

A house older than 30 years typically has 100A 120V split phase power which can supply 25000 watts (you wouldn't want to ever fully load it...)

And an 8000 watt space heater will definitely be noisy, and produce too much heat for pretty much any house.

iamnothere

5 hours ago

Part of the problem with current data centers is the density. To make the economics work, you need excessive density, which comes with power, noise, and water requirements.

Smaller servers distributed more widely don’t come with the same requirements. They can’t handle all use cases, but something like a Tinybox can run consumer LLM tasks just fine, a SAN with a small server can provide backup storage or storage for CDNs, etc. No need to turn the house into a full data center.

The key would be to build highly efficient small servers that can work as an appliance. It would need to be very easy to swap them out when one fails.

Again, I’m not sure this has much of a benefit except for providing geographical dispersion. Data centers would still be more cost effective. Maybe it would be helpful for providing local services in small remote areas like islands.

aaronax

2 hours ago

A Bitcoin mining node is the simplest possible way to turn compute into money. Very minimal storage and bandwidth requirements. And yet we still do not see those in houses.

Everything about doing productive computing tasks in houses is more complicated than that! At least it is more profitable, I think?

(I wonder what a rough profit per watt figure is for a datacenter. Very much "it depends" I'm sure.)

kube-system

5 hours ago

The density of data centers provides efficiency gains -- if you take the same workload from a high density nvidia DGX setup in a data center and instead distribute it to Tinyboxes running residentially, you'd have an overall net gain in energy use.

iamnothere

4 hours ago

Yes, along with additional costs for managing and servicing a distributed set of devices and extra redundancy needed for higher expected downtime. But maybe the cost is worth it for some application.

moribvndvs

5 hours ago

You’d have to run a lot of Electron-based apps to go from tepid to hot water.

ishtanbul

5 hours ago

Your energy cost would be very high paying retail. If you actually want to provide services to the internet from your home, your operating cost is not competitive at all with large scale systems. Who would you sell those services to?

kkfx

3 hours ago

What's unrealistic is the widespread state of IT ignorance among the masses and the kleptocracy that profits from it.

Without this widespread ignorance, and with IPv6, a global per host, and without the disappearance and massive price hikes of RAM and storage, we could all have a home server running, for example, a family's personal services:

- contacts (Radicale/Baïkal/Davis/*)

- photos (Immich, PhotoPrism, ...)

- video (Jellyfin and the like, even Stash for those who fancy it)

- files in general (e.g. SyncThing)

- email (fetched via OfflineIMAP and similar, served via Dovecot+webmail for those who want it, etc.)

- federated XMPP/Matrix for family and friends

- ...

And even for the State, a national blockchain for digital identity (NFTs), contracts (e.g. property sales, etc.), and money, with a node for every family and consensus to regulate it, for maximum resilience and reliability, thus also enabling electronic voting.

But well, given the widespread IT ignorance, it's just a pipe dream.

In this model, we would have de facto teleworking, and therefore de-urbanisation, with houses and sheds with solar panels on the roof, batteries, and activities shifted to maximise self-consumption—thereby electrifying without loading the national grid, in a true and substantial transition that would otherwise be unrealistic.

Personally, that's how my house is, national blockchain aside (but with a personal lightning node anyway), and it works beautifully; it would work brilliantly up to ~45° latitude across the EU and slightly less (I think, I haven't checked the PV maps) for North America. Simply doing this would kill off the aforementioned kleptocrats. No cities means:

- no end to private property for the majority

- no dependence on private collective transport in the hands of a tiny few

- no fast tech/fast fashion with very low costs for the vendor but high costs for the customer and nature due to the piles of waste

- no ready-meal deliveries with tons of packaging

A resilient, renewable society (including the built environment) that can evolve but doesn't have the majority enslaved to a tiny few. This is why it isn't happening.

zoklet-enjoyer

5 hours ago

Lots of people have had this idea since the early days of Bitcoin mining. Some have even done it. I recommend looking up how people have set up mining rigs in their homes.

throawayonthe

4 hours ago

why not district heating at that point? less headaches with distributing hardware to a bunch of homes

AndrewKemendo

5 hours ago

I think this is exactly the future for non-giant-corpo internet

The problem is DNS and access to the IP network

So if you can figure out how to build reliable DNS access/approvals with cloudflare etc then it would work

The biggest challenge at the largest scale is political because then you’re gonna be fighting all of the ISP’s and the giant technology companies and ultimately they’re never going to allow for this

Either take it over on their own by offering their own service which people would sign up for

or they’ll just pressure every ISP or certificate authority to not recognize routes that are not going through “allowed” data centers,

most likely you would end up with a series of state bills or even a federal regulation that prevents data routing for public consumption unless it in some kind of “security standard” or whatever bullshit they come up with