Parent is maintainer of rsync.net, in case that wasn't obvious. (There's no contact information in your profile.)

> indefinitely

Which, in practical terms, means "while I or my investors have money". These guys have 70 years worth of data, can you guarantee to keep it for another 70? The real answer to these kinds of problems is managing the data yourself

I highly suggest (if you haven't already done so) you contact Nine PBS, as they (Nine PBS) won't be reading Hacker News.

rsync is good stuff, I've been a happy customer of https://www.rsync.net/products/borg.html for years.

Out of general interest: whats your bus factor?

Epic.

fucking legend

Iron Mountain is typically very stringent on their retention/disposal policies.

Storytime, since I'm no longer under NDAs for the company, over a decade ago somewhere that I worked, a disgruntled employee who decided to quit, made a point to within a week of quitting, take a whole bunch of important physical documents they had (because, FBOW, our email and other data retention was so bad, it was safer to have them as prints) and threw them all in our fancy Iron Mountain 'disposal' bin.

Apparently due to the agreement with them, short of a court order, only specific employees/titles in the company were allowed to be the persons to retrieve them.

Keeping that situation in mind, it's clear that this dance for better or worse is part of their posturing as a retention center with very strict rules to ensure compliance while also avoiding snooping.

They have a judgment against OSS ordering the data to be returned, and affirming ownership.

It still sucks for all involved, and it's a shame there's not a better solution here. It's gotta be so frustrating for the folks at PBS to keep getting hit like this.

> but what ever happened to the 3-2-1 backup rule?

I wasn't 100% sure, but the data backs this up[0]; the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.

- [0] Interestingly, Ngram (https://books.google.com/ngrams/graph?content=%5B3-2-1+backu...) says ~2010 while Trends (https://trends.google.com/explore?q=3-2-1%2520backup%2520rul...) says slow build up from ~2020, guess "everyday" language would sit somewhere around those two different truths.

It's worth noting that PBS affiliates are non-profits that have fairly constrained budgets, especially now that the Corporation for Public Broadcasting has been dissolved due to DOGE.

Non-profit IT has always seemed, to me at least, to be a land of compromises. There's rarely enough money, knowledge, or time to do things "the right way", so you have to do the best you can. If I could only choose one backup to have, it'd be remote on a provider like Iron Mountain. It would seem as if they are storing things on Iron Mountain through an intermediary company, who may or may not have actually paid the bill.

So, y'know, fraud's a possibility here.

I’m not sure this is the only copy of the data.

Sounds like they paid a company to digitise all their old tapes, and store the results. They possibly still have all the tapes (on site) and are just trying to avoid paying the costs to redigitise them.

But, it’s also quite likely they destroyed all original the tapes after digitisation, and this was the only copy.

From the article: It's not clear if the channel has backups of its data, but based on its determination, we suspect not. This is strange, though, especially for a major organization, as one can easily purchase a six-bay NAS on Amazon and populate it with the best HDDs. This also goes against the 3-2-1 backup rule, which says you should keep three copies of your files stored in two different mediums, with one of them running offsite. Hopefully Nine PBS will implement a better backup system once it regains access to its data.

The backups (and servers) seem to still exist, but KETC's vendor is estranged with its own hosting / siting provider, Iron Mountain, which is a going concern.

IM seem to be holding KETC's data hostage, which is Not Cool.

> Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.

This does make sense. I'm guessing the first request some rep just went sure its you're data why not, but in reality, it's OSS's data, and you paid OSS to store it. It's also likely encrypted in someway.

TLDR still, is especially when it's a reasonable amount of data, just have more than 1 backup.

Yeah, we are talking about 3 20TB drives in a closet here.

I'm an independent photographer and I have my 20TB drives behind my monitor, backing up to backblaze with an offsite synology. Keeping this archive up does take time and money, but PBS should have the budget and IT deparment for this.

> broadcasters have of every broadcast they've ever had

It depends - especially if you go back further than 20 years. The Wikipedia article for "Lost television broadcast" goes into detail:

https://en.wikipedia.org/wiki/Lost_television_broadcast

I think for most of broadcasting history video was fairly low quality (according to today’s standards). So I am not totally surprised that this number is low.

> But just 50TB of data? Duplicating that would have been cheap and trivial.

Seriously. I'm just a random nerd and I have a home server with 96TB of raw disk space divided in to two RAIDZ1 pools which cost me around $2500 to build five years ago and has required basically zero maintenance since then.

Hard drive prices are absolutely bonkers right now but even with that in mind if I were to build a modern version of this same machine I could do it for under $10k using brand new hardware and it'd have double the storage

Most broadcasters would consider their ultimate back up as the tape library/vault of their broadcast video tapes. Older stations will have a large collection of analog tapes. Most will only be digitized if someone requests it as capturing their entire library just to have digital files few people will ever want is an expense a station just isn't going to do for the lulz.

It's not PBS, PBS - but an affiliate, Nine PBS (https://www.ninepbs.org/)

From the article it seems like Iron Mountain was doing the actual storage, and OSS was just acting as a middleman and taking some money off the top.

Your assuming it's all digital and can be stored on drives. It might be a large amount of physical film reel, magnetic tape, and tons of mixed legacy formats

50TiB is like $4k consumer SSDs in a shoebox at this point (not actually the recommended solution, but...). And that's massively inflated thanks to the AI boom.

> 50 TiB is so little.

Still, it is 70 years of video from that PBS station.

> William Cravens, the attorney representing Iron Mountain, told the judge his client doesn’t know the format of Nine PBS’ materials that were stored by OSS. He expressed concern about whether Nine PBS’ archival material is lumped together with data from other OSS clients. Iron Mountain wants to avoid potentially corrupting the other data, Cravens added.

That's an entirely reasonable concern and explains why Iron Mountain wants OSS on the hook for fulfilling this order rather than trying to yolo it themselves.

Don't trust Internet Archive as your only backup method though. There has been instances where they delete files without warning with the only explanation being that they don't meet TOS

It's incredibly frustrating. The Internet Archive offers a Vault product, where you pay $2-3/GB one time for perpetual storage into the future of your collections, items, media, etc. This pricing was previously predicated on the cost decline curve of storage. It's gotten a bit more expensive due to the AI bubble inflating the cost of storage, but to your point, the Internet Archive is a superior custodian than some rando tier 2 cloud provider who won a contract from a non profit and will lose all of the data when they go bust.

https://archive-it.org/vault/

(huge fan of Stokes as a fellow archivist, recommend the documentary on her: https://en.wikipedia.org/wiki/Recorder:_The_Marion_Stokes_Pr...)

This good old quote comes to mind:

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

— Andrew S. Tanenbaum

Cram a few 12 TB drives into that sucker and you’ve got all the storage you need for their whole 50 TB archive.

Why should they have to get a court order when the party in custody said they have the data? Unless they don't actually have the data and they're hiding behind the court.

I read it as a statement of just how little data this is by modern standards more than anything else. This company is going out of business and PBS is losing this valuable data, and there's legal drama and multiple corporate entities involved, and if you read all that you might think this is some amazing amount of data that is actually difficult to deal with. Instead it's something I could stroll down to Best Buy right now, plonk down my personal credit card, and walk out with enough stuff to do a credible backup job myself, and we're not even challenging my credit limit or anything.

10 years ago, this was a lot of data. It isn't anymore and it makes it that much more bizarre that they're having that much trouble getting it back. It would be less to just give them access and let them download it themselves and perhaps stick it somewhere else temporarily than to pay lawyers to deal with legal filings.

To put it in vaguely physical terms, what once was a university library is now a good solid bookshelf full of stuff. Legally wrangling over a huge library is one thing, but if it's just a bookshelf, metaphorically, let them come in for an hour and clean it out themselves rather than bringing in the law. Obviously not a perfect metaphor on data size, this is more about how lawsuits are more of a hassle then just giving it to them.

> It's not clear if the channel has backups of its data, but based on its determination, we suspect not. This is strange, though, especially for a major organization, as one can easily purchase a six-bay NAS on [Amazon] and populate it with [the best HDDs].

I think they just put that stuff in so they could shill their affiliate marketing articles rather than out of any actual concern.

Having that be there only backup solution would have been silly and dangerous. Having it be part of their overall backup solution including with something like the provider would have been smarter and better and prevented this entire problem

3 copies, 2 different media, 1 offsite. One backup is none

The point still stands, but in what world can you build a 50TB NAS for $1,000 these days? I want to go there

Technically you're still correct even when you would advocate for both. Every additional copy increases the risk that the data is accessed by someone else.

A NAS is at risk from malware that can erase all of it. 100GB Bluray discs exist and work more easily for archival storage, as does multi-terabyte tape drive storage, both of which can stay disconnected At least two offline copies, combined with error correction codes, are advised. A cloud backup is also advised.

Yeah well you are not holding 90 TB of data based on the decisions of a chain of supervisors.

People like me who no longer pay attention enough after seeing so much upheaval in the cloud space are losing out on ideas of best practice and giving in to whatever works for the time being.

I've kept all photos on iCloud since decades; there's too many to move elsewhere and keep updated, but the local copy on my Mac is regularly backed up to a Time Machine disk.

  > "their vendor could effectively just decide to commit crimes"

  > "their vendor’s vendor could just decide to compound the problem in the face of a court order is wild"

This looks like a similar formality of following the process so that a court legitimises the transfer of data back to Nine PBS.

Statistically a fire is more likely to burn your 1 copy than the data center I bet. Ideally you have both, that survives almost anything outside thermonuclear war

Probably should buy three, and put them in places far enough apart that an earthquake or large fire won't get all of them at once.

This local PBS wasn't playing it's own footage 24/7. The data in question could be just their local segments, an hour or two of local coverage and maybe their fundraisers.

Isn't 50TB just 6 years of videos, in even the best case (1GB/h)?

They weren't recording in 4K last century.

One hour of 1970's-grade NTSC video is around 300MB.

More to the point, it's not like they archived 70 continuous years of video. Many local PBS stations produce an hour or less of video each week. This being a St. Louis station, however, it's probably somewhat more.

Pbs is offering to pay for the recovery, but the company who supposedly has the data basically went non responsive. Honestly this sounds like a mishandled going out of business where the company left with the assets was not prepared to deal with the customers at all.

Because of the next sentence in that paragraph:

> This unidentified vendor provided “hardware, software and cloud-storage services” for storing the public broadcaster’s archival materials and other data.

How do we know the content isn't on BetacamXR or straight film reels ?

Not really, you're never "supposed to" rely on a single company with 3 tapes for your only copy

By my calculation using what I can find of 2019 pricing, a more reasonable solution would have been: Synology RackStation RS1219+ - $1,200 8x Ironwolf Pro 12TB drives - 8x $450 = $3600 so for a usable RAID-6 capacity of 65TB about $4800 + OSS, or double that if they have two locations they could and replicate between them.

I know there would be cheaper hardware options available (I'm partial to older surplus enterprise hardware myself) but it isn't particularly reasonable to expect a small business to choose a more DIY (at least in terms of software configuration) option. You could step down to 10TB drives, but it seems unlikely they would have wanted to risk that not being quite enough.

On top of that, since 2019 is 7 years ago, it is reasonable to expect they would have had a hardware refresh at some point, I would think 5 years would be more reasonable than 3 year for something like this. At that point, the nas would be more ($1300) but the drives would be less (8 * $300), so add either another $3700 (double if both are replaced, but maybe they figured the offsite backup could be stretched even longer than 5 years)

So, rather than $2500, it would probably have been more like $8500 + OSS or $13,300 without OSS. Still that would have meant a proper are archive and a proper backup and they wouldn't be in this mess, so I completely agree with you about what they should have done, but think you were underselling the price a good bit.

That made me feel relief until I realized it's my own local PBS station.

Ah, "Channel Nine PBS", not a cluster of nine pbses

Pretty clear that this is AI copied slop that no one at Toms Hardware bothered editing as it has no clarity that Nine PBS is the PBS affiliate in St Louis, Missouri.

Here is the source article...

https://current.org/2026/08/nine-pbs-sues-iron-mountain-over...

Link updated, thanks!

https://en.wikipedia.org/wiki/Station_identification#United_...

  The United States' Federal Communications Commission (FCC) enforces specific requirements for identification that must be followed by all terrestrial radio and television stations. Stations must, when they sign on, sign off, and as close to the top of each hour as feasibly possible (such as within a "natural break" in programming, like a commercial break), present a visual (television) or aural (radio) station identification that contains, at minimum, the station's callsign, followed by its designated city of license. As a courtesy, top-of-hour identifications may also contain additional information, such as frequencies and a declaration of the station's ownership.[6] Only the name of the licensee, the station's frequency or channel number as stated on its license, and/or network affiliations, may be inserted between the call letters and station location. 

  Technical information is provided. This can include station identification (call sign and city of license), transmitter power, frequency or channel number, translators used, transmitter locations, list of broadcast engineers, and/or studio/transmitter links (STL).

It's actually funny you mentioned the lack of indicating call sign or operating location. I saw something similar done on purpose.

I worked at WCCO in Minneapolis briefly a few years ago. Right around that time, CBS corporate was attempting to standardize all their stations and associated websites. In their mind, that meant making everything "CBS News [Location]" and stripping just about everything else from the public marketing.

WCCO, KPIX in Los Angeles, and a few other stations battled back about this because the local name recognition of the call sign was stronger than CBS would be by itself.

So now, when you bop around the "Local News" section of the CBS Website, some cities have logos with the call letters, and some do not.

In short, when you see a station that that's difficult to identify, try finding out who owns it. Very often the ownership is to blame for one reason or another.

We also have 5?