Why your social.org files can have millions of lines without performance issues

93 pointsposted 3 months ago
by andros

8 Comments

ChrisArchitect

3 months ago

Context FYI:

Org-social is a decentralized social network that runs on Org Mode

https://news.ycombinator.com/item?id=44889354

pshirshov

3 months ago

Out of curiosity, how popular it is?

frumiousirc

3 months ago

My most recent traversal of the social graph found 46 accessible nodes and 487 edges.

0_____0

3 months ago

So, like ARPANET ca. 1980

andros

3 months ago

It depends on the niche you're in. It's not at the level of X/Twitter or Mastodon. You need a minimum of technical knowledge.

dietr1ch

3 months ago

So, It leverages on HTTP range requests to try fetching only what's needed, but to me it seems that it's something that should be efficiently achievable without having to have a unified document.

In my filesystem I could arrange things to be easy to fetch

ls ~/Org/Social/2025/10/

and then fetch the documents I didn't have.

I feel that all this logic is just to workaround problems that may arise when fetching multiple small files, mostly tied to overhead and artificial data ordering when fetching a set of tiny files. Hasn't HTTP3 fixed most of them already?

mjmas

3 months ago

s/Lines/Millions of Lines/

dang

3 months ago

Fixed. Thanks!