There was a thread here earlier this month,

> Helldivers 2 devs slash install size from 154GB to 23GB

https://news.ycombinator.com/item?id=46134178

Section of the top comment says,

> It seems bizarre to me that they'd have accepted such a high cost (150GB+ installation size!) without entirely verifying that it was necessary!

and the reply to it has,

> They’re not the ones bearing the cost. Customers are.

I worked in e-commerce SaaS in 2011~ and this was true then but I find it less true these days.

Are you sure that you’re not the driving force behind those metrics; or that you’re not self-selecting for like-minded individuals?

I find it really difficult to convince myself that even large players (Discord) are measuring startup time. Every time I start the thing I’m greeted by a 25s wait and a `RAND()%9` number of updates that each take about 5-10s.

Contrary, every consumer facing product I've worked had no performance metrics tracked. And for enterprise software it was even worse as the end user is not the one who makes a decision to buy and use software.

>>what you mean by software houses

How about Microsoft? Start menu is a slow electron app.

> every consumer facing software product I’ve worked on has tracked things like startup time and latency for common operations as a key metric

Must be nice. In my career, all working on webapps, I've seen a few leaders popping in to ask us to fix a particularly egregious performance issue if the right customers complain, but aside from those finely-targeted and limited-attention-span drives to "improve performance" it seems the answer for the past decade or so is just to assume everyone is on at least a gigabit connection, stick fingers in ears, and just keep adding more node modules. If the developers' disks get full because node_modules got too big, buy a bigger SSD and keep going. (ok that last part is slight hyperbole but I also don't think frontend devs would be deterred from their ravenous appetite for libraries by a full disk).

> I don’t know how many times I’ve heard the repeated quote about how Amazon loses $X million for every Y milliseconds of page loading time, as an example.

This is true for sites that are trying to make sales. You can quantify how much a delay affects closing a sale.

For other apps, it’s less clear. During its high-growth years, MS Office had an abysmally long startup time.

Maybe this was due to MS having a locked-in base of enterprise users. But given that OpenOffice and LibreOffice effectively duplicated long startup times, I don’t think it’s just that.

You also see the Adobe suite (and also tools like GIMP) with some excruciatingly long startup times.

I think it’s very likely that startup times of office apps have very little impact on whether users will buy the software.

Can confirm at least for Firefox. When I worked on it, I've spent literal years shaving seconds from startup, or shutdown, or milliseconds from tab switching.

Everybody likes to hate Telemetry, and yes, it can be abused, but that's how Mozilla (and its competitors) manage to make user's life more comfortable.

Clearly Amazon doesn't care about that sentiment across the board. Plenty of their products are absurdly slow because of their poor engineering.

> every consumer facing software product I’ve worked on has tracked things like startup time and latency for common operations as a key metric

Are they evaluating the shape of that line with the same goal as the stonk score? Time spent by users is an "engagement" metric, right?

The issue here is not tracking, but developing. Like, how do you explain the fact that whole classes of software have gotten worse on those "key metrics"? (and that includes web-selling webpages)

Then why do many software house favor cloud software over on premise?

They often have a recognizable delay to user data input compared to local software

>I don’t know what you mean by software houses, but every consumer facing software product I’ve worked on has tracked things like startup time and latency for common operations as a key metric.

Then respectfully, uh, why is basically all proprietary software slow as ass?

An exception that confirms the rule.

> so I say take as much as you can. Commons would be if it’s owned by nobody

This isn’t what “commons” means in the term ‘tragedy of the commons’, and the obvious end result of your suggestion to take as much as you can is to cause the loss of access.

Anything that is free to use is a commons, regardless of ownership, and when some people use too much, everyone loses access.

Finite digital resources like bandwidth and database sizes within companies are even listed as examples in the Wikipedia article on Tragedy of the Commons. https://en.wikipedia.org/wiki/Tragedy_of_the_commons

Still, because reality doesn't respect boundaries of human-made categories, and because people never define their categories exhaustively, we can safely assume that something almost-but-not-quite like a commons, is subject to an almost-but-not-quite tragedy of the commons.

It has the same effect though. A few bad actors using this “free” thing can end up driving the cost up enough that Microsoft will have to start charging for it.

The jerks get their free things for a while, then it goes away for everyone.

A public park suffers from tragedy of the commons even though it’s managed by the city.

I wouldn't call it "tragedy of the commons" because the very idea was coined as a strawman. As far as I'm concerned, the entire concept is a fallacy, and people should stop perpetuating it.

Right. Microsoft could easily impose a transfer fee if over a certain amount that would allow “normal” OSS development of even popular software to happen without charge while imposing a cost to projects that try to use GitHub like a database.

I doubt anyone is calculating

Remember how GTA5 took 10 minutes to start and nobody cared? Lots of software is like this.

Some Blizzard games download 137 MB file every time you run them and take few minutes to start (and no, this is not due to my computer).

Well, till you choose to host something yourself and it becomes popular

Tragedy of the Microsoft just doesn't sound as nice though

>> The number of companies that have this much respect for the user is vanishingly small.

I think companies shifted to online apps because #1 it solved the copy protection problem. FOSS apps are not in any hurry to become centralized because they dont care about that issue.

Local apps and data are a huge benefit of FOSS and I think every app website should at least mention that.

"Local app. No ads. You own your data."

Software I don’t have to install at all “respects me” the most.

Native software being an optimum is mostly an engineer fantasy that comes from imagining what you can build.

In reality that means having to install software like Meta’s WhatsApp, Zoom, and other crap I’d rather run in a browser tab.

I want very little software running natively on my machine.

Yes because users don't appreciate this enough to pay for the time this takes.

https://news.ycombinator.com/item?id=44843223#44879509

1. If you can price the cost of the externality, you can justify optimizing it.

2. Monopolies and situations with the principal/agent dilemma are less sensitive to such concerns.

If the class takes N minutes and one person arrives 1 minute late, and the rest of the class is waiting for them, it does stack. Every one of those students lost a minute. Far worse than one student losing one minute.

A well designed feature IS considerate of time and attention. Why would I want a game on 20 fps when I could have it on 120? The smoothness of the experience increases my ability to use the experience optimally because I don't have to pay as much attention to it. I'd prefer if my interactions with machines were as smooth as my interactions driving a car down a empty dry highway mid day.

Prehaps not everyone cares but I've played enough Age of Empires 2 to know that there are plenty of people who have felt value gains coming from shaving seconds off this and that to get compound games over time. It's a concept plenty of folks will be familiar with.

I am very interested by something like this but your README is not making it easy to like. Demonstrating with 2-3 sample apps using RDX might have gone a long way.

So how do I start using it if I, for example, want to use it like a decentralized `syncthing`? Can I? If not, what can I use it for?

I am not a mathematician. Most people landing on your repo are not mathematicians either.

We the techies _hate_ marketing with a passion but I as another programmer find myself intrigued by your idea... with zero idea how to even use it and apply it.

Sorry, I am turned off by the CRDT in there. It immediately smells of overengineering to me. Not that I believe git is a better database. But why not just SQL?

Time saved to increased productivity or happiness or whatever is not linear but a step function. Saving one second doesn’t help much, but there is a threshold (depending on the individual) where faster workflows lead to a better experience. It does make a difference whether a task takes a minute or half a second, at least for me.

But there isn't just one company deciding externalizing cost on the rest of us is a great way to boost profit since it costs them very little. Especially for a monopoly like YouTube that can decide that eating up your battery is fine if it saves them a few cents in bandwidth costs.

Not all of those externalizing companies abuse your time but whatever they abuse can be expressed in a $ amount and $ can be converted to a median's person time via median wage. Hell, free time is more valuable than whatever you produce during work.

Say all that boils down to companies collectively stealing 20 minutes of your time each day. 140 minutes each week. 7280 (!) minutes each year, which is 5.05 days, which makes it almost a year over the course of 70 years.

So yeah, don't do what you do and sweettalk the fact that companies externalize costs (private the profits, socialize the losses). They're sucking your blood.

One second is long enough that it can put a user off from using your app though. Take notifications on phones for example. I know several people who would benefit from a habitual use of phone notifications, but they never stick to using them because the process of opening (or switching over to) the notification app and navigating its UI to leave a notification takes too long. Instead they write a physical sticky note, because it has a faster "startup time".

> I have never been convinced by this argument. The aggregate number sounds fantastic but I don't believe that any meaningful work can be done by each user saving 1 second. That 1 second (and more) can simply be taken by me trying to stretch my body out.

I’d see this differently from a user perspective. If the average operations takes one second less, I’d spend a lot of time less waiting for my computer. I’d also have less idle moments where my mind wanders while waiting for some operation to complete too.

Just because one individual second is small, it still adds up.

Even if all you do with it is just stretching, there's a chance it will prevent you pulling a muscle. Or lower your stress and prevent a stroke. Or any number of other beneficial outcomes.

Is it truly naive if most engineer's careers pass and they never meet even one such manager?

For 24 years of career I've met the grand total of _two_ such. Both got fired not even 6 months after I got in the company, too.

Who's naive here?

Explain to me how you self-host a git repo which is accessed millions of time a day from CI jobs pulling packages.

They probably would have experienced issues way sooner, as the self hosted tools don't scale nearly as well.

As long as you don't have any security compliance requirements and/or can afford the cost of self hosting your LLM, sure.

Anyone working in government, banking, or healthcare is still out of luck since the likes of Claude and GPT are (should be) off limits.

I've never been more convinced LLMs are the vanguard of the grift economy now that green accounts are low effort astroturfing on HN.

Microsoft would have long gone out of business if users cared about their time being wasted.

Windows 11 should not be more sluggish than Windows 7.

> Run postgres on a $5 VPS and have everybody accept it as single-point-of-failure

Oh how times have changed. Yes, maybe run two $5 VPSs behind a load balancer for HA so you can patch and then put a CDN in front of it to serve the repository content globally to everyone. Sign the packages cryptographically so you can invite people in your community to become mirrors.

How do people think PyPI, RubyGems, CPAN, Maven Central, or distro Packages work?

Honestly I think the article is a bit ahistorical on this one. ‘go get’ pulls the source code into a local cache so it can build it, not just to fetch the go.mod file. If they were having slow CI builds because they didn’t or couldn’t maintain a filesystem cache, that’s annoying, but not really a fault in the design. Anyway, Go improved the design and added an easy way to do faster, local proxies. Not sure what the critique is here. The Go community hit a pain point and the Go team created an elegant solution for it.

Not only this, but (if I understand the article correctly) at least some of them still use git on the backend.

The problem is that Arch doesn't need to quickly resolve (version -> manifest) for arbitrary versions. With Arch, /var/lib/pacman/sync/core.db has one release of a set of packages. When you install, you just grab whatever's there. Rolling release. pacman -Syu pulls the newest version of that set of packages. If you install sqlite 3.0 and then come back a few years later and "reinstall" all the Arch packages you used to have on a new machine, you'll either (a) use that exact database and pull the same version or (b) pacman -Syu, pull latest package database, and get the newest sqlite (say, 3.5)

There's no concept of installing sqlite 3.0 on a system where sqlite 3.5 is available.

For a language package manager, it's exactly the opposite. I could make a project with every version of sqlite the package manager has ever known about. They all must be resolvable.

If you want to do that resolution quickly (which manifest do I use for sqlite 3.0?), repo-per-package doesn't work without a bunch of machinery that makes it, IMO, not worth it.

Pacman is the best, you'd have to pry Arch from my cold, dead hands. Just different constraints.

For most software, yes. But the value of a package manager is in its adoption. A package manager that doesn’t run up against these problems is probably a failure anyway.

The point is not "design to serve the world". The point is "use the right technology for your problem space".

I'm not OP but I'll guess .... lock files with old versions of libs in. The latest version of a library may be v2 but if most users are locked to v1.267.34 you need all the old versions too.

However a lot of the "data in git repositories" projects I see don't have any such need, and then ...

> Why not just have a post-commit hook render the current HEAD to static files, into something like GitHub Pages?

... is a good plan. Usually they make a nice static website with the data that's easy for humans to read though.

Please share. I’m interested in anything that’s roughly as simple as implementing a centralized registry, is easily inspected by users (preferably with no external tooling), and is very fast.

It’s really important that someone is able to search for the manifest one of their dependencies uses for when stuff doesn’t work out of the box. That should be as simple as possible.

I’m all ears, though! Would love to find something as simple and good as a git registry but decentralized

Sure! It's very raw, though. There's a lot of functionality, and I use it to build all sorts of projects already. But a common thing I do is to write the stupidest possible version of a thing and only do the hard engineering when it becomes untenable. Hence it's not raw as in being new or bare, but it's very raw in that you'll see some really rough stuff in the code.

But, that being said, here's the repo! I added a very basic README for you. It's one command to bootstrap to a self hosting build, so give it a shot if you're interested. My contact is in my profile.

https://github.com/tspader/spn

I've mostly heard FAFO used to describe something obviously stupid.

Building on the same thing people use for code doesn't seem stupid to me, at least initially. You might have to migrate later if you're successful enough, but that's not a sign of bad engineering. It's just building for where you are, not where you expect to be in some distant future

Not at all.

When you fuck around optimizing prematurely, you find out that you're too late and nobody cares.

Oh, well, optimization is always fun, so there's that.

Most of the world does work this way. Problems are solved within certain conditions and for use over a certain time frame. Once those change, the problem gets revisited.

Most software gets to take it to more of an extreme then many engineering fields since there isn't physical danger. Its telling that the counter examples always use the potentially dangerous problems like medicine or nuclear engineering. The software in those fields are more stringent.

Hold up... "lazy engineers" are the problem here? What about a society that insists on shoving the work product of unfunded, volunteer engineers into critical infrastructure because they don't want to pay what it costs to do things the right way? Imagine building a nuclear power plant with an army of volunteer nuclear engineers.

It cannot be the case that software engineers are labelled lazy for not building the at-scale solution to start with, but at the same time everyone wants to use their work, and there are next to no resources for said engineer to actually build the at scale solution.

> the path of least resistance for themselves.

Yeah because they're investing their own personal time and money, so of course they're going to take the path that is of least resistance for them. If society feels that's "unethical", maybe pony up the cash because you all still want to rely on their work product they are giving out for free.

On the other hand, GitHub wants to be the place you choose to build your registry for a new project, and they are clearly on board with the idea given that they help massive projects like Nix packages instead of kicking them off.

As opposed to something like using a flock of free blogger.com blogs to host media for an offsite project.

Fixing problems as they appear is unethical? Ok then.

You realize, there are people who think differently? Some people would argue that if you keep working on problems you don't have but might have, you end up never finishing anything.

It's a matter of striking a balance, and I think you're way on one end of the spectrum. The vast majority of people using Julia aren't building nuclear plants.

What is wrong with you? You berated and name-called open source volunteers because a blog post taught you that package managers using Git are "bad." Let me be clear: a 3 minute read of a blog post offers neither moral superiority nor technical insights that surpass those of actual maintainers.

Contrary to the snap conclusion you drew from the article, there are design trade-offs involved when it comes to package managers using Git. The article's favored solution advocates for databases, which in practice, makes the package repository a centralized black box that compromises package reproducibility. It may solve some problems, but still sucks harder in some ways.

The article is also flat-out wrong regarding Nixpkgs. The primary distribution method for Nixpkgs has always been tarballs, not Git. Although the article has attempted to backpedal [1], it hasn't entirely done so. It's now effectively criticizing collaboration over Git while vaguely suggesting that maybe it’s a GitHub problem. And you think what, that collaboration over Git is "unethical"???

On one side, there are open-source maintainers contributing their time and effort as volunteers. On the other, there are people like you attacking them, labeling them "lazy" and bemoaning that you're "forced" to rely on the results of their free labor, which you deride as "slow, bug-riddled garbage" without any real understanding. I know whose side I'm on.

[1]: https://github.com/andrew/nesbitt.io/commit/8e1c21d96f4e7b3c...

Could you please articulate specifically why that should be concerning?

Right now I don't see the problem because the only criterion for IDs is that they are unique.

If you read the repo README, it just says "generate a uuid". You can use whatever you want as long as it fits the format, it seems.

It’s as random as any other UUID.

This is exactly the right approach. I did this for my package manager.

It relies on a git repo branch for stable. There are yaml definitions of the packages including urls to their repo, dependencies, etc. Preflight scripts. Post install checks. And the big one, the signatures for verification. No binaries, rpms, debs, ar, or zip files.

What’s actually installed lives in a small SQLite database and searching for software does a vector search on each packages yaml description.

Semver included.

This was inspired by brew/portage/dpkg for my hobby os.

This is how WinGet works. It has a small SQLite db it downloads from a hosted url. The DB contains some minimal metadata and a url path to access the full metadata. This way WinGet only has to make API calls for packages it's actually interacting with. As a package manager, it has plenty of problems still, but it's a simple, elegant solution for the git as a DB issue.

I actually find that nixpkgs being a monorepo makes it even better. The code is surprisingly easy to navigate and learn if you've worked in large codebases before. The scaling issues are good problems to have, and git has gotten significantly better at handling large repos than it was a decade ago, when Facebook opted for Mercurial because git couldn't scale to their needs. If anything, it's GitHub issues and PRs that are probably showing its cracks.

For the purposes of the article, git isn't just being used as a database, it's being used as a protocol to replicate the database to the client to allow for offline operation and then keep those distributed copies in sync. And even for that purpose you can do better than git if you know what you're doing, but knowledge of databases alone isn't going to help you (let alone make your engineering more economical than relying on free git hosting).

What's a better option? One that keeps track of history and has a nice review interface?

Sure, but can you back up the expletive with some reason why you think that?

As it is, this comment is just letting out your emotion, not engaging in dialogue.

No, crates.io isn't locked to Github. crates.io uses Github as an identity provider, but there's nothing stopping them from adding more. Furthermore, they've avoided tying themselves to Github in other ways, for example, by resisting all the people just telling them to allow using Github usernames as package namespaces, specifically to prevent them from being locked to Github.

I mean, it's sort of the other way around. Cargo was built to be able to natively understand git-based dependencies, in the sense that you can bypass a crate registry and instead just point it directly at a git repo. That means that Cargo already had to have the ability to clone git repos, and so when it came to decide how to implement the index (which looks pretty similar to a git repo if you squint), choosing to use git required them to add literally no new dependencies and almost no new code.

Let's also keep in mind that the use case mentioned in the OP is specifically about the index, which is just the datastructure that informs the version resolver how to resolve versions. When it came time to replace the git-based index, Cargo didn't replace it with a specialized database, it replaced it with HTTP endpoints (which are probably just backed by an off-the-shelf database). It's not clear what sort of specialized database would be useful to abstract this for other package managers.

The issues with nixpkgs stem from that it is a monorepo for all packages and doubling as an index.

The issues are only fundamental with that architecture. Using a separate repo for each package, like the Arch User Repos, does not have the same problems.

Nixpkgs certainly could be architected like that and submodules would be a graceful migration path. I'm not aware of discussion of this but guess that what's preventing it might be that github.com tooling makes it very painful to manage thousands of repos for a single project.

So I think it can be a lesson not to that using git as a database is bad but that using github.com as a database is. PRs as database transactions is clunky and GitHub Actions isn't really ACID.

Be more specific because I just see a list of workarounds deployed once they had the scale to warrant them, supporting the OP’s claim.

It’s a fair criticism, and this article does serve well as a warning for people to try and avoid this issue from the start.

That costs hosting money no? That might be a bigger problem for someone starting than scalability

A good point!

^^^ Poe's Law may or may not apply to the above comment.

Yeah, I'm implementing a couple of things to make my life easier in the future. I don't use any github APIs and I'm setting up my clients to load the plugin repo URLs from my server so I can change them later if I need to. I want all of the resources my clients need to come from my domain name so I can move it around if I need to.

To be clear though, the rsync trees come from a central Git repo (though it's not hosted on GitHub). And syncing from Git actually makes syncing faster.

Yes, I'm suggesting hosting it on GitHub, leveraging their git lfs support. Just treat it like a binary blob and periodically update with a tagged release.

Feels like there's an opening in the market there. Why can't you expose an SQL server to the public?

Also Stackoverflow exposes a SQL interface so it isn't totally impossible.

Yes but CH is not SQL.

Furthermore, the issues given for nixpkgs are actually demonstrating the success of using git as the database! Those 20k forks are all people maintaining their own version of nixpkgs on Github, right? Each their own independent tree that users can just go ahead and modify for their own whims and purposes, without having to overcome the activation energy of creating their own package repository.

If 83GB (4MB/fork) is "too big" then responsibility for that rests solely on the elective centralization encouraged by Github. I suspect if you could go and total up the cumulative storage used by the nixpkgs source tree distributed on computers spread throughout the world, that is many orders of magnitude larger.

Exactly. Gentoo's main package repo is hosted in Git (but not GitHub, except as a mirror). Now, most users fetch it via rsync, but actually using the Git repo IME makes syncing faster, not slower. Though it does make the initial fetch slower.

Can you rsync a repo from GitHub?

> Git knows how to store [a hash-addressed tree], but git does not know how to transfer it efficiently.

Naïvely, I’d expect shallow clones to be this, so I was quite surprised by a mention of GitHub asking people not to use them. Perhaps Git tries too hard to make a good packfile?..

Meanwhile, what Nixpkgs does (and why “release tarballs” were mentioned as a potential culprit in the discussion linked from TFA) is request a gzipped tarball of a particular commit’s files from a GitHub-specific endpoint over HTTP rather than use the Git protocol. So that’s already more or less what you want, except even the tarball is 46 MB at this point :( Either way, I don’t think the current problems with Nixpkgs actually support TFA’s thesis.

Not in the same sense. An analogy might be: apt is like fetching a git repo in which all the packages are submodules, so lazily fetched. Some of the package managers in the article seem to be using a monorepo for all packages - including the content. Others seem to have different issues - go wasn't including enough information in the top level, so all the submodules had to be fetched anyway. vcpkg was doing something with tree hashes which meant they weren't really addressible.

I consider apt kinds slow. I wish it were much faster.

https://litestream.io/

Go's lock files arrived at around the same time as the proxy, before then you didn't have transitive dependencies pre baked.

Those are great points, and reinforce the concept that there is conflation between Cargo and Git/commits. Commits and Cargo IMO should be separate concepts. Cargo should not be checking my Git history prior to publishing.

it's here https://news.ycombinator.com/item?id=46386211 (and it was last on the front page at the moment)

LOL sorry. You're right. I'll copy paste over there.

Yes agreed. It’s possible to imagine some kind of cached-deltas scheme to get faster/smaller updates, but I suspect the folks who would have to build and maintain that are all on gigabit internet connections and don’t feel the complexity is worth it.

you could, in case you want to make only certain releases publicly available. but then, who wants to do that manual labour? we're talking mainstream here, not specific use cases.

But they were in different domains. Here, we have a strong clash because Rust is positioning itself as secure system and internet language and computer and internet standard are already defined by RFC-s. So, it may be not uncommon, when someone would tell about Rust mechanisms, defined by particular RFC in context of handling particular protocol, defined by... well... RFC too. But not by rust-one.

Not so smart, when we realize, that one of aspects of secure and reliable system is elimination of ambiguities.