Engineers do not get to make startup mistakes when they build ledgers

590 pointsposted 8 months ago
by fagnerbrack

194 Comments

hn_throwaway_99

8 months ago

Tell that to Synapse customers. Many millions of dollars are missing.

Banks have to follow strict rules to account for where all the money goes. But the way fintechs work, they usually just have one or a couple underlying "FBO" accounts where all the pooled money is held, but then the fintech builds a ledger on top of this (and, as the article points out, to varying levels of engineering competence) to track each individual customer's balance within this big pool of money. In Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts. Lots of folks are assuming fraud but I'm willing to put money that it was just a shitty, buggy ledger.

FWIW, after seeing "how the sausage is made", I would never put money into a fintech depository account. Use a real bank. Fintechs also often put out the fake promise that deposits are FDIC insured, but this only protects you if the underlying bank goes belly up, not if the fintech loses track of your money.

See https://www.forbes.com/sites/zennonkapron/2024/11/08/what-th...

chairmansteve

8 months ago

90 million dollars missing and 250 million dollars frozen. That 250m probably needed by some people to pay rent.

Backed by Andreesen Horowitz who are conducting a scorched earth jihad against all government regulation.

https://finance.yahoo.com/personal-finance/synapse-bankruptc...

abirch

8 months ago

The sad thing is that most people don't learn lessons from history. It took me far too long to start learning lessons from history.

After an asset bubble and collapse people will understand why we have a lot of the regulations from the 1930s.

bboygravity

8 months ago

That's not really a lot compared to what Wallstreet is steeling daily.

ozim

8 months ago

You do understand what you described is basically Bolshevik/French revolution only different times.

Some men with some power using young starry eyed young people to grab much more power from incumbents.

user

8 months ago

[deleted]

mrfox321

8 months ago

At a big co I worked at, the lack of consistency between trading systems caused money to (dis)appear (into)out of thin air.

Prior to one of these hiccups, I hypothesized, given how shitty the codebase was, that they must be tracking this stuff poorly.

This led to an argument with my boss, who assumed things magically worked.

Days later, we received an email announcing an audit one one of these accounting discrepancies.

JPMC proposed using crypto, internally, to consistently manage cash flow.

Not sure if it went anywhere.

HolyLampshade

8 months ago

At all of the exchanges and trading firms I’ve worked with (granted none in crypto) one of the “must haves” has been a reconciliation system out of band of the trading platforms. In practice one of these almost always belongs to the risk group (this is usually dependent on drop copy), but the other is entirely based on pcaps at the point of contact with every counterparty and positions/trades reconstructed from there.

If any discrepancies are found that persist over some time horizon it can be cause to stop all activity.

phire

8 months ago

Fun fact, centralized crypto exchanges don't use crypto internally, it's simply too slow.

As a contractor, I helped do some auditing on one crypto exchange. At least they used a proper double-entry ledger for tracking internal transactions (built on top of an SQL database), so it stayed consistent with itself (though accounts would sometimes go negative, which was a problem).

The main problem is that the internal ledger simply wasn't reconciled with with the dozens of external blockchains, and problems crept in all the time.

naasking

8 months ago

> JPMC proposed using crypto, internally, to consistently manage cash flow.

Yikes, how hard is it to just capture an immutable event log. Way cheaper than running crypto, even if only internally.

hooverd

8 months ago

It's all merkle trees under the hood. I feel like the crypto coin stuff has overshadowed the useful bits.

im3w1l

8 months ago

If its for internal why not just use a normal append only log. x amount transferred from account y to account z. A three column csv oughta do it.

DanielHB

8 months ago

> I hypothesized, given how shitty the codebase was, that they must be tracking this stuff poorly.

That is like half of the plot of Office Space

mlloyd

8 months ago

This sounds like a situation that I know about at the placed identified by name in your comment. It took months to track down the issue.

phonon

8 months ago

Synapse says that it was actually the Bank (Evolve) that made the accounting mistakes, including missing transactions, debits that weren't reported, sending in flight transaction to Mercury while debiting Synapse incorrectly etc.

https://lex.substack.com/p/podcast-what-really-happened-at-s...

hn_throwaway_99

8 months ago

Thanks for posting this, I will definitely listen to it.

While I haven't listened yet, one thing I don't really buy when it comes to blaming Evolve is that it should fundamentally be Synapse's responsibility to do reconciliation. This is what completely baffled me when I first worked with another BaaS company - they weren't doing any reconciliation of their ledgered accounts with the underlying FBO balances at the partner bank! This was insane to me, and it sound likes Synapse didn't do it either.

So even if Evolve did make accounting mistakes and have missing transactions, Synapse should have caught this much earlier by having regular reconciliations and audits.

rawgabbit

8 months ago

Rambling interview. As best as I can tell Synapse said there were technical issues with Evolve the bank.

Meanwhile this article said Synapse withdrew from Evolve the end user funds. Mr. Holmes of Evolve said the bank “transferred all end user funds” to other banks at the request of Synapse, but declined to identify them

https://www.nytimes.com/2024/07/09/business/synapse-bankrupt...

zimbatm

8 months ago

Wise also recently switched their US bank provider from Evolve to Community Federal Savings Bank. Maybe they had similar issues?

abenga

8 months ago

From a cursory look at how it describes itself (BaaS, etc), Evolve is hardly a "bank" in the traditional sense of the word.

hipadev23

8 months ago

Sankaet is full of shit.

hgomersall

8 months ago

I had money disappear from my HSBC account. As in, the double entries didn't match by a small amount (it was a payment between two of my accounts that didn't match up, which I couldn't trivially reconcile in the books). I pursued this for a while but they never even properly acknowledged it let alone fix it.

I had my unfounded suspicion it was some internal subtle theft going on, but incompetence is probably a better explanation.

tossandthrow

8 months ago

If you live in a developed country it should be sufficient to ask them to account for it with a note that a formal complaint will be sent to relevant authorities if nor dealt with in timely manner.

That stuff like this is in order is the foundation of kapital societies and is taken quite seriously.

pera

8 months ago

I had a similar situation with Santander many years ago: it was a small amount and happened when I was closing my account, bank manager couldn't explain it and escalating the problem was a pain - especially because I was about to move to another country and had more urgent things to do.

I wonder how common issues like these are...

lores

8 months ago

You're likely correct it was theft. I was told by a CTO there that topping up accounts with bank money where it has been hacked away was daily routine and cheaper than fixing the systems. Incompetence would not manifest on routine operations like this.

DanielHB

8 months ago

> I had my unfounded suspicion it was some internal subtle theft going on

Had you watched Office Space recently?

TexanFeller

8 months ago

I’m skittish about real banking institutions as well. Vanguard for example outsourced a bunch of their dev work to India a few years ago. Had a friend that worked as a sysadmin for BoA. They were required to keep certain logging for 7 years but he would just delete it anyway when disks were starting to get full.

superzamp

8 months ago

Coincidentally written something about this yesterday [1], but the gist of my take summed up is that the nature of accounting oriented data models doesn’t help when dealing with multiple FBO accounts.

The main problem is that accounting defaults to full fungibility of monetary amounts represented in a ledger, which has the effect of losing track of the precise mapping between assets and liabilities, so you end up in a position where you simply cannot tell precisely to a bank _who_ are the actual customers they owe money to.

[1] https://www.formance.com/blog/engineering/warehousing-promis...

ThinkingGuy

8 months ago

I like Cory Doctorow's saying: "When you hear the term 'fintech,' think 'unlicensed bank.'

chairmansteve

8 months ago

"Fintechs also often put out the fake promise that deposits are FDIC insured, but this only protects you if the underlying bank goes belly up, not if the fintech loses track of your money".

Would you count Wealthfront as a fintech? I was finding their marketing compelling, but this thread makes me think twice.

eweise

8 months ago

In addition to a ledger, fintechs need a reconciliation system to ensure the ledger is correct. Does the card processor audit files match your ledger? Does your ACH and check processing systems match the ledger? What about external money movements at the sponsor bank. Are they recording in the ledger?

ajuc

8 months ago

> In Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts. Lots of folks are assuming fraud but I'm willing to put money that it was just a shitty, buggy ledger.

Bugs are as likely to show more and less money than there really are. But bugs in production will almost always show more :)

e40

8 months ago

I had been debating the merits of using Flourish, but I'm sticking with SNOXX on Schwab. Same rate and I think SNOXX has to be safer, right? Even with the Flourish FDIC guarantee, as others have pointed out, it's only for the underlying back not Flourish itself.

bradleyjg

8 months ago

In Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts. Lots of folks are assuming fraud but I'm willing to put money that it was just a shitty, buggy ledger.

If there was no malfeasance then no money would be gone. The totals would add up, they just wouldn’t no know who was owed what. Since the totals don’t add up, someone got the extra.

throwaway2037

8 months ago

    > Fintechs also often put out the fake promise that deposits are FDIC insured
Does this still happen?

moffkalast

8 months ago

The fact that practically all funding most of the world runs on these days is just a bunch of variables in some shitty program never stops being weird to think about. All it takes to create or destroy trillions is one (or maybe a few) CPU ops.

It really stretches the belief into fiat money to the absolute limit.

PittleyDunkin

8 months ago

> Lots of folks are assuming fraud but I'm willing to put money that it was just a shitty, buggy ledger.

I'm not sure there's much difference. Intent only matters so much.

serbuvlad

8 months ago

> In Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts.

When the banks do this it's called "fractional reserve banking", and they sell it as a good thing. :)

masto

8 months ago

One of the things it took a little time to wrap my head around when I started working at Google was trading off reliability or correctness for scaling.

I had previously built things like billing systems and small-scale OLTP web applications, where the concept of even asking the question of whether there could be any acceptable data loss or a nonzero error rate hadn't occurred to me. It was eye-opening to see, not so much that if you're doing millions of qps, some are going to fail, but more the difference in engineering attitude. Right this instant, there are probably thousands of people who opened their Gmail and it didn't load right or gave them a 500 error. Nobody is chasing down why that happened, because those users will just hit reload and go on with their day. Or from another perspective, if your storage has an impressive 99.99999% durability over a year, when you have two billion customers, 200 people had a really miserable day.

It was a jarring transition from investigating every error in the logs to getting used to everything being a little bit broken all the time and always considering the cost before trying to do something about it.

hedora

8 months ago

Durability levels that poor aren’t state of the art any more.

The rule of thumb I’ve seen at most places (running at similar scale) is to target one data loss, fleet wide, per century.

That usually increases costs by << 10%, but you have to have someone that understands combinatorics design your data placement algorithms.

The copyset paper is a good place to start if you need to understand that stuff.

Scubabear68

8 months ago

This all makes a lot of sense, but I have seen this a lot in the opposite direction. Specifically, people from social media companies with very squishy ideas around failures coming into financial applications. Generally does not go well.

user

8 months ago

[deleted]

shaftway

7 months ago

While I was at Google I was loaned out to the Android team to work on contact synchronizing. There was a problem that I was running into, but it was an extremely rare situation. I pulled aggregated data from production, and it looked like it would affect 0.01% of people. When I presented the solution and mentioned this error rate I was asked how those 200,000 Android users could remediate the situation. They couldn't, their contact sync would just be broken, so I was told to go back to the drawing board.

But yeah, it was the number that was humbling.

There are definitely areas where 4 nines are good enough, but there are just as many areas where the aren't.

figassis

8 months ago

This is the type of thing where it helps to hire the right people from the start. You hire a bunch of leetcode experts, but never stop to ask if, besides conjuring data structures and algos in their minds they actually can build the thing you want to build. If your people know how to build the thing, you don't need to sacrifice growth, it gets built right from the start.

Sometimes you need engineers that have some other type of education, maybe accounting, maybe finance, maybe biology. I always thought that the most important part of my career was understanding every industry I built for, deeply, and knowing experts in those areas so you could ask the really important questions. That is problem solving and engineering. The rest is programming/coding.

manyxcxi

8 months ago

I have never come across a post on HN that was so scarily describing my current day to day and with a comment I agree with so wholeheartedly.

I’ve spent the majority of my career in tech with a finance angle to it, mostly sales and use tax compliance.

What I never fully appreciated was how much those accountants, controllers, and lawyers were rubbing off on me.

I was recently advising a pretty old startup on their ledgering system and was beyond appalled at what it looks like when a bunch of engineers with no finance or accounting background build an accounting system.

We don’t have to find magical accountant engineers either, it will wholly suffice if we sit actual accountants in with our engineering team during the design process.

After my design of their complete overhaul I had a friend who is a CPA completely review my work, we found a few holes in certain scenarios, but by and large we were good.

Money is a difficult engineering problem because with money comes all the human goofery that surrounds it.

kdmtctl

8 months ago

This. Ledgers are a domain knowledge. Anything else should be built on top, including high load optimizations.

slashdave

8 months ago

In some circles, there is the irritating tendency to believe that technology can solve every problem. Experts are eschewed because innovation is valued above all else.

tomgp

8 months ago

This reinforces for me the importance of domain knowledge in engineering leadership. i.e. if you work for a finance company you need to have a decent understanding of finance in order to make the correct technical decisions and tradeoffs, same goes for journalism, same goes for commerce etc. etc. Successful organisations I've worked for have always included domain specific, non-technical, questions to tech team interviews whilst some of the most technically acomplished teams I've worked with have floundered through a lack of domain insight.

sotix

8 months ago

I’m a software engineer and a CPA, so I’ve got the domain knowledge. However, I’m not sure how to find a gig where I can apply that. It seems everywhere I look vastly prefers someone with twice my experience as a software engineer and zero domain knowledge over my years of experience in accounting and fewer years of experience as a software engineer. Any insight on how to utilize it effectively?

collinmcnulty

8 months ago

I agree with your sentiment here, but I think it helps to think of those domain specific questions as technical questions, just from a different technical domain. Finance is technical, mechanical engineering is technical, even sports management and sociology have big technical components too. I think having an expansive view of what technical competence is breeds the humility necessary to work across domains.

spencerchubb

8 months ago

ever since i started working for an insurance company, i realized that understanding the insurance industry is far more difficult than understanding the codebases. if the codebase is acting weird, at least i can step through it with a debugger!

crazygringo

8 months ago

I don't know about that -- I've always understood that to be the role of the product manager, to have all the domain knowledge.

It's the PM's job to work with engineering to ensure that the requirements are correct, and that the built product meets those requirements. And in an agile setting, those conversations and verifications are happening every single sprint, so something can't go for too long without being caught.

If you don't have a PM, then sure I guess your engineering team had better have deep domain knowledge. But otherwise, no -- it's not engineering's responsibility. It's products's responsibility.

gregw2

8 months ago

Pardon me an old story... I never built a double entry accounting system but decades ago I did build a billing system for a internet/telcom startup that grew to a modest 8 figures revenue.

By accident and not knowing any better as a young dev, I ended up building the billing logic from day one, and for better and worse building it in two places in the system (on a consumer-facing billing webpage, and on a separate backed process that generated invoices and charged credit cards.)

It turned out to be remarkably hard to keep them in sync. We were constantly iterating trying to get traction as we burned down our capital, releasing new products and services, new ways of discounting and pricing (per use, per month, first X free, etc), features like masterpayer/subaccounts for corporate accounts, user-assignable cost centers, tax allocation to those cost centers with penny allocation, etc such that new wrinkles and corner cases would keep popping up causing the numbers on my two screens/methods not to match.

Being personally responsible for the billing, I would go over all the invoices by hand for a couple days each month to insure they matched before we charged the credit cards and mailed out printed invoices as a final check to prevent mistakes. There was always/often some new problem I'd find affecting one or a small handful of customers which I would then fix the code before we billed. I never felt good letting go and not doublechecking everything by hand.

I thought about refactoring the billing logic to occur in one place to eliminate these mismatches and my manual crosschecking, but after a lot of thought I realized I wasn't comfortable with a single codebase and liked having two codebases as it helped me catch my own errors. I then just made it easier and easier to run automate and crosschecks between the two. The billing code was a little too gnarly to be proud of, but I was very proud of the outcome in how accurate our billing was, the lack of complaints and many near misses we avoided for many years. I do feel twinges of guilt for the complexity I left my successors but I still don't really regret it.

After that experience, the motivation for double entry bookkeeping has always made a lot of sense to me. I had sort of reinvented it in my own hacky way with double logic billing code to prevent my mistakes from causing problems for my customers...

EdwardDiego

8 months ago

Billing, or anything else involving money, is so easy to get wrong.

The data team I ended up leading at a previous company, had an unfortunate habit of "losing" money - it wasn't real money being lost in transit elsewhere, but records of something we should charge a customer.

Or if the team wasn't losing revenue, it was double charging, etc. etc.

Took us 3 years of hard work to regain the trust of the business leaders.

icapybara

8 months ago

No offense but this sounds like a nightmare. It also sounds like you did a fantastic job achieving accuracy despite the complexity of the system. That’s something to be proud of.

mydogcanpurr

8 months ago

This is also very similar to N-version programming :)

loeber

8 months ago

Good comment, thanks for sharing

Aeolun

8 months ago

No tests either? If you lose track of enough money every transaction that you can make an example of 'Every $5 purchased resulted in $4.98 in the transaction log' I think your problem is far, far bigger than not having double entry bookkeeping.

Who builds a financial system like that an considers it normal? The compensation is one thing, but you'd flee a service like that with all possible haste.

hotsauceror

8 months ago

These guys. They said it themselves. “We could have built it right, but we didn’t.” They chose not to. It was not an accident. They made jokes about “dancing cents”. They did these things because there would never be meaningful consequences for doing them, and they knew it. They move fast, they broke things - money things - and they laughed about it. And now they’re lecturing people as if having willfully made these decisions gives them both moral and technical authority. This is magnificently pompous, startup VC culture nonsense.

riffraff

8 months ago

I still didn't understand how they lost the money. Yes, double entry bookkeeping might have helped diagnose it, but how were they losing it?

tyre

8 months ago

Yeah that was my thought as well. Ledgers have tons of benefits but they’re not going to fix your dancing cents problem. You’re going to have bad number on the ledger.

Sure, maybe that points you to the bugs, but so would writing basic tests.

alexey-salmin

8 months ago

A good design principle is worth a 1000 tests.

watt

8 months ago

I don't understand why the author chooses to bring up the mantra “make it work, make it right, make it fast“ in a negative light, perhaps he misunderstands where the "make it fast" comes in?

to clarify: "make it right" is the second step, and until you make things work correctly ("right"), that's where the work stops, you have to make the system work soundly. the "make it fast", as in, optimize, comes in only after you have got it right, that all correctness and soundness issues are resolved perfectly. then you can start optimizing it (making it run fast).

it has nothing to do with delivery speed. it has nothing to do with working quickly. it's about optimizing only as a last step.

perhaps the author is lamenting the fact that it is possible for something to sort of "work", but to be so far from being "right" that you can't go back and make it right retroactively, that it has to be "right" from the inception, even before it starts barely working?

badcppdev

8 months ago

I think your last sentence is correct. I think author is saying that you can't make it work and THEN make it right with payment systems.

This is my opinion as well and I've been involved in the audit of a fintech system where auditors had to download EVERYTHING into excel spreadsheets and make the numbers balance before they would sign off on the books. That took a lot of time and money I'm guessing made a difference of at least .1 unicorns in the liquidity even that took place 3 years later.

zaphodias

8 months ago

I think author chose the wrong mantra too!

What happens in fast paced startups is that you ship what essentially is a MVP as soon as possible (i.e. you stop at the "make it work" step) because you need to build your customer base, finances, etc.

A better mantra would've been Facebook's "move fast and break things". But, that only works if you can fix the things later. You wouldn't do it if you're building an aircraft for example.

jeremyjh

8 months ago

They said the engineering team followed this mantra, which includes the phrase "make it right". But they didn't. They didn't even bother to find out what they didn't know about fintech before they started building a product.

Given the context in which he used it, I think the misunderstanding you suggest in the first sentence is most likely. Immediately afterward he talks about the time pressure startups face.

user

8 months ago

[deleted]

rkharsan64

8 months ago

Most of the comments here echo what the article is criticising. I can see countless threads of back-and-forth defending single-entry bookkeeping.

Sure, single-entry bookkeeping might be easier and more normalized, but sometimes it is a good idea to just stick with the systems and abstractions that have been developed over centuries.

Just use double-entry bookkeeping unless you definitely need something else. Sure, it might be icky for the programmer inside you, but I think you'll be thankful if you ever need to get actual accountants involved to sort out a mismatch.

On a related note: does anybody know of any good resources for programmers in payments and adjacent fields? Something like an "Accounting for Programmers"?

rambambram

8 months ago

> On a related note: does anybody know of any good resources for programmers in payments and adjacent fields? Something like an "Accounting for Programmers"?

Get a grip on the accounting basics first. I built my own bookkeeping system with double entries and realized the design and the programming was the easy part.

zitterbewegung

8 months ago

If told people that I was using a database system where one and 100th of the data was missing after every 10 transactions would you seriously take my advice as an engineer blog post like these that seem to be calmly focused toward an advertisement of a person’s or a group’s promotion and that it’s just introducing concepts.

bobnamob

8 months ago

For a glimpse at the _real_, human consequences of this kind of slipshod mentality toward money shuffling software see the Post Office scandal[1].

https://en.wikipedia.org/wiki/British_Post_Office_scandal

Anything that moves money should be treated with upmost seriousness and be aware of as many historical mistakes as possible.

amluto

8 months ago

Does anyone have a good explanation for why a ledger database should care about credits and debits and normal balances? This has always seemed more natural to me as a front-end / presentation layer concept. Your ledger entries always sum to zero for each transaction, your income account has a negative balance, and you display it in a sensible manner.

I’m also surprised that this whole article starts by discussing stock trading but has no mention of how to represent stock trades. I assume they are “Sagas” consisting of money moving from the customer to the clearinghouse (or prime broker or PFOF provider or whatever) and shares moving from that provider to the account at which the shares are held. And maybe other associated entries representing fees? It seems to me that this is multi-entry accounting, which is quite common, and that entries don’t actually come in pairs as the article would like us to think.

jasim

8 months ago

That part of the article felt quite wrong to me as well. I've built accounting systems that worked well for a decade, where internally the values were a single amount column in the journal table. If it was a debit, it'd be positive, if a credit, it'd be negative.

In fact, we could call these values yin and yang, for all it mattered.

Also, I'm not able to really follow what he means by "money = assets in the future".

Money is money, but if you wanted to track the intermediate state until the customer gets receipt, you would use an In Transit account (Good In Transit / Service In Transit etc.)

Yet, it doesn't change the fundamental definition of the value in the accounting system. I think the author confuses an engineering concept (sagas, or thunks, or delayed but introspectable/cancellable actions in general) with accounting.

rawgabbit

8 months ago

It is best explained by common scenarios an Italian merchant in the Middle Ages experienced. The basic concept is Assets==Liability (plus Equity). Where positive Assets are entered on the left hand side (debit). And positive Liabilities are entered on the right hand side (credit). In accounting, debit and credit just means left and right.

1. Merchant takes out a loan for $5,000 and receives $5,000 in cash. • Assets (Cash) increase by $5,000 (Debit). • Liabilities (Loan Payable) increase by $5,000 (Credit). • Equity remains unchanged.

2. Merchant buys inventory for $1,000 cash. • Assets (Cash) decrease by $1,000 (Credit). • Assets (Inventory) increase by $1,000 (Debit). • Total assets remain unchanged, and liabilities and equity are unaffected.

3. Merchant sells all inventory for $1,500 cash. • Assets (Cash) increase by $1,500 (Debit). • Assets (Inventory) decrease by $1,000 (Credit) (recording cost of goods sold). • Equity (Retained Earnings) increases by $500 (Credit), representing the profit ($1,500 sales - $1,000 cost).

4. Customer1 deposits $500 in cash for future delivery of goods. • Assets (Cash) increase by $500 (Debit). • Liabilities (Unearned Revenue) increase by $500 (Credit). • Equity remains unchanged.

5. Customer1 transfers half of the future delivery of goods to Customer2. • No changes to assets, liabilities, or equity occur at this point. The merchant’s obligation to deliver goods (reflected as Unearned Revenue) is still $500 but now split between two customers (Customer1 and Customer2). Internal tracking of this obligation may be updated, but the total financial liability remains the same.

BillyTheKing

8 months ago

yes, agree, I think a 'source' 'destination' model is significantly more straight-forward. Just record the 'source' account and the destination account and you essentially end up with a ledger as a directed graph (Martin Kleppmann wrote a great post on it)

I also wrote a super short post on how to model such a system on postgres https://blog.nxos.io/A-simple-double-entry-ledger-with-sourc...

Blockchain actually kinda nails it, that's in essence a source/destination ledger, no 'postings' or similar needed, and from a balance calculation POV has been working pretty well

One reason this model isn't applied in accounting, in my personal view :), is simply historical and the fact that the number 0 didn't exist when accounting principles were created.

Wrote another post on how to model debit/credits on a source/destination ledger here: https://blog.nxos.io/Debit-and-Credits-on-a-Source-Destinati...

It's very straight-forward, you just have to accept that asset accounts have negative balances and present the absolute amount instead of a negative amount in a view.

yobbo

8 months ago

In double-entry, a transaction is the tuple (amount, credit_account, debit_account).

In singly-entry, it is the tuple (amount, account).

jfengel

8 months ago

I've never understood that either. As a computer guy it always struck me as a redundancy -- the kind of redundancy where one thing will always eventually be wrong.

I assumed it has to do with the fact that it was invented for bookkeeping by hand. Accountants assure me that it's even more important with computers, but I've never managed to figure out how that works

neffy

8 months ago

Double entry book keeping implements an error correction and detection algorithm.

mamcx

8 months ago

> This has always seemed more natural to me as a front-end / presentation layer concept.

Consistence is a property of the backend, if that is wrong there is not hope for later

> Your ledger entries always sum to zero for each transaction, your income account has a negative balance, and you display it in a sensible manner.

'sensible manner' is the problem here. The data/money will be diverge with time, and without proper storage of the data it will by impossible to figure out.

The problem here is NOT store 'a transaction'. That with a RDBMs works. Is to store the FLOW of MANY transactions and the divergent ways things works.

Like, your bank is telling you has $100 and your system $120. And your system sum right, but the bank rules.

Or when you do a return and cents are lost in the interchanges and chargebacks.

Or, just wrong data entry, sync, import/export, etc.

---

The way to see this is that `double entry` is a variation of `inmutable data that don't mutate and always track the flow of it' that is golden for business apps.

2mol

8 months ago

Great article. I have an observation to the "engineers should know this and do good engineering" though: I work for a payments company and there is a fundamental career problem with becoming the ledger ninja: It's not enough work, and it's eventually done!

I've seen the following major phases of this work: 1) Build the ledger (correctly), and it will work well for a while. 2) Add convenience code for the callers, assist finance in doing reports/journaling, fix some minor bugs, take care of the operational bits (keep the database up). 3) Reach the scaling limits of your initial approach, but there are some obvious (not trivial) things to do: re-implement the transaction creation directly in the database (10x perf gain), maybe sharding, maybe putting old tx into colder storage, etc.

This is spread out over a while, so I haven't seen it be a full-time job, even at real startup-level (+10% MoM) growth. Even if it was, that's one person, not a whole team. I understand engineers that instead are pulled towards projects where they are in higher demand.

In another comment somebody said ledger systems are trivial when done right and super hard when done wrong - so if you did a good job it kinda looks like you just created 3 tables and some code. That seems thankless, and job searching as this type of specialist is harder than just being a generalist.

scary-size

8 months ago

I suspect this is a "small company" problem which don't sell too many different things. A larger enterprise might have a platform for connecting businesses with businesses and businesses with customers. They might sell services (subscriptions) but also one-time purchases which require different forms of revenue recognition. Those might even be split up across different revenue streams. You end up building sub-ledgers for each one of them because the ERP can't handle the scale. Oh, and you're a public company so make sure everything is SOX compliant and easy to audit. Ah, and you operate on a global scale so do all those things in different currencies for different legal entities.

There's a reason Stripe is as successful as it is. And then there's a world where a company outgrows Stripe.

There are worse career choices ("prompt engineer" LOL) than financial engineering.

jes5199

8 months ago

I’ve built my career on cleaning up should-have-used-a-ledger messes. it’s hard but there’s always another company that needs it, and I get bored staying in one place.

recently I discovered that in a medical billing context the system is way, way weirder than I had seen before. I shipped the three tables, but getting everything into it seems like it might be an endless battle

bob1029

8 months ago

If you are trying to reinvent parts of banking, you could maybe start with what already works as a thought experiment.

Here's an example of a core banking deposit transaction schema that has been extensively battle tested in many small & mid-size US institutions:

https://jackhenry.dev/open-enterprise-api-docs/operational-d...

You may note fields like "Effective Date" & "Affects Balance/Interest", which imply doing this correctly may involve exploring some interesting edge cases. Wouldn't it be cool if you could just cheat and start with an approach that already considers them?

neuroelectron

8 months ago

I don't know what the fuck I just read here. I've worked on ledgers for years with massive codebases with millions of loc and I've never seen an error of "a few cents". Feels like I just read a alibi. Don't develop financial software from scratch.

rstuart4133

8 months ago

That was my reaction too.

It was like reading an engineering team saying their attempt to design a new lightweight mountain bike failed. It turned out saving weight by omitting brakes wasn't a good idea, and their attempt to fix it by riding beside each bike and manually slowing it down wasn't too popular either. Then they have the hubris to follow that up with, "you can avoid the same mistakes by reading what we have to say on bicycle design".

The lessons they can take away have very little to do with engineering or FinTech. I'd file it under doing a bit of research before committing to any major task. Basic accounting principles are centuries old now. They could have learnt about them by buying a high school text book, reading it cover to cover and use doing the exercises. It would have taken them less than a week.

Admittedly that only gives you the "how", not the "why". You realise much, much later that the engineering equivalent of double entry accounting is two systems design by different teams that are constantly comparing their outputs. By the time you've figured that out, it's caught so many errors you realise "holy shit, these systems are really easy to screw up, and are under constant attack by malicious actors".

There is a hidden trap at every step - floating point being imprecise, currency rounding not being reversible, tax calculations being one way, ACID being a hard requirement. I'm being this mob screwed up tax calculations. Floating point throwing out 1 in 100 million transactions was one of the joys they never got to experience.

392

8 months ago

The lesson here is not to never develop from scratch. In fact it may be faster and safer to do so than to vet all of the open source junk that modern ByteByteGo-system-designers think they need. TigerBeetle and Jepsen-proof-engineering are paths to dig down here, along with basic knowledge of computer number storage.

The lesson for me here is that there is no substitute for knowing or caring about what you're doing. Dancing cents don't happen when eng cared about correctness in the first place. I feel like I just read a college freshman explaining how there's no possible way they could have passed their first exam. Better yet he's here to sell us on his thought leadership without ever explaining why he failed the first exam.

ahi

8 months ago

I read the intro and decided I wouldn't learn much from someone willing and eager to admit he eats paint chips.

user

8 months ago

[deleted]

heylook

8 months ago

I've seen it, sighed, fixed it, and ensured the necessary accounting adjustments assigned the loss to the business instead of the customer. Seems that last step, which is the most important one, wasn't followed here, which is the real shame.

slashdave

8 months ago

I had the same thought. I mean, were they actually using ordinary floating-point numbers to represent amounts in their ledger? This sets off so many alarm bells.

fendy3002

8 months ago

It's a suicide to build a finance system or similar without double entry ledger.

Worse, I've worked at where the transaction reference id from vendor is not recorded, it's lost so the past data cannot be reconciled!

blibble

8 months ago

is there no individual accountability regime in the US?

in the UK, as an engineer, if I'd built this I would expect the regulator to come after me personally for not ensuring the system had adequate controls to protect clients money/investments

with a potentially unlimited fine + prison time

masfuerte

8 months ago

Has that ever happened? It's incredibly hard to prosecute directors in the UK for obvious malfeasance. I have never heard of a software engineer being sanctioned for crap code.

mikewarot

8 months ago

>is there no individual accountability regime in the US?

Here in the US, programmers like to call themselves Engineers, forcing everyone else to use the term "Professional Engineer" or "Licensed Engineer" or some other modifier before their title. I hate it, I wish they would stop, but it's not going to happen.

Software here is a wild, wild, West. The motto most live by is "move fast and break things"... even when those things are people's lives.

dylan604

8 months ago

I leave you with the 2008 financial crisis as exhibit A on exactly how nobody gets in trouble for lack of financial accountability

patmorgan23

8 months ago

Not for software that I'm aware of.

For certain regulated professions there is, if a building falls down due to a bad design the professional engineer (PE) that signed and sealed the plans can be held personally liable.

lubujackson

8 months ago

I don't see how a rank and file programmer would ever be personally responsible for their code. You can blame management for forcing untested or known flawed logic, but not some shmoe that pushes an "off by 1" bug while working weekends and late nights with no testing and hard deadlines.

badcppdev

8 months ago

Can you please provide a reference to the regulators coming after an engineer in the UK.

Also you say regulations in the UK have been changed recently.

I'm not aware of regulations that apply to software engineers.

HPsquared

8 months ago

Depends if the particular activity is regulated or not.

arethuza

8 months ago

Are you a professionally qualified engineer?

stavros

8 months ago

It really irks me that the author assumes I know how double-entry accounting works and doesn't mention a single sentence about it. I read half way through the article and couldn't follow it, except that single-entry is bad and double-entry is good.

remram

8 months ago

A double-entry system is one where you can't change the balance of an account, you can only record transfers between accounts. This means it's impossible to forget to update the other side of a transaction, it's a single step. A consequence of that is you can check that the sum of all accounts is always 0.

In practice you have virtual accounts like "cloud expenses" and "customer subscription" that only go up/down over time, to be the counter-party for transactions in/out of your company. So it's not impossible to mess up, but it eliminates a class of mistakes.

antgiant

8 months ago

Could have been clearer but it is there. Here is the relevant section. In short single entry is basically the Account view and Double entry is the full ledger. (Called double because of the hard requirement that all Entries come in pairs.)

> Ledgers are conceptually a data model, represented by three entities: Accounts, Entries and Transactions.

> Most people think of money in terms of what’s theirs, and Accounts are the representation of that point of view. They are the reason why engineers naturally gravitate towards the “simplicity” of single-entry systems. Accounts are both buckets of value, and a particular point of view of how its value changes over time.

> Entries represent the flow of funds between Accounts. Crucially, they are always an exchange of value. Therefore, they always come in pairs: an Entry represents one leg of the exchange.

> The way we ensure that Entries are paired correctly is with Transactions. Ledgers shouldn’t interact with Entries directly, but through the Transaction entity.

foobarian

8 months ago

I've been trying to figure out double-entry accounting for years now and still don't get it. Most explanations are along the lines of "Here is a simple explanation that you are guaranteed to understand: <proceeds to describe the procedure>" which lacks intuition on why this is useful. I suspect I would need to do a gig as an accountant and run into some error conditions that double-entry solves to really grok it.

Edit: no offense but sibling comment is an example :P

jjmarr

8 months ago

> A double-entry system is an accounting method that tracks money at both its source and destination.

There's an entire section on double-entry accounting in the article. The tl;dr is that if you take money out of an account, you need to place money in another account and vice versa. So, you have a table called "accounts receivable" which track money the company is owed. If you actually get paid, you remove money from "accounts receivable" and add money to the "cash" account, instead of just increasing the amount of cash you have.

It makes it much more difficult to lose track of money or have it stolen. In a single-entry system, I could receive money from a customer for services owed and just keep it for myself.

adamcharnock

8 months ago

If you're looking for a pre-built double entry accounting Postgresql schema that provides database-level integrity checks:

https://github.com/adamcharnock/django-hordak

I created and maintain this, along with a couple of others. It is built for Django (so great if you're using Django), but extracting the schema wouldn't be too hard. It also has MySQL support, but the integrity checks are more limited.

(Side note: I'm a freelancer and available!)

jimnotgym

8 months ago

There is a bigger lesson here. When you are creating something new it pays to understand what came before. Do you need to disgard everything?

I gave a boorish lecture to a junior accountant recently...

When you make things up yourself it is just you and Excel. When you use double entry you have 100s of years of history to fall back on. https://en.m.wikipedia.org/wiki/Double-entry_bookkeeping

BlueTemplar

8 months ago

Though there's a weirder life lesson here : a lot (most / all ?) of successes come from founders not having enough experience to know just how difficult their enterprise is.

More experienced founders would take a look at it and immediately nope out of it, and so not achieve success (after a grueling amount of work).

cynusx

8 months ago

Accounting systems are super hard when you do them wrong and kind of trivial when you do it right.

There is no in-between.

Martin fowler wrote quite a bit on the subject and it's a good match for event-sourcing.

jes5199

8 months ago

literally every startup I’ve worked at has had busted accounting system that had to be painfully and expensively rewritten as a proper ledger. We should be teaching “Accounting for Programmers” as a required course in CS programs, or at least spreading memes about how money data must be append-only and immutable or you are being negligent in ways far more serious than most programming errors

probably_wrong

8 months ago

The Software Engineering department next door from my CS department had a mandatory course on accounting. I don't think CS needs accounting, but I did learn two skills that would have helped here: why one should never use floating point for money (which may not apply here, who knows), and how to stick to a formal specification. "Money is neither lost nor created" is the most trivial example one can think for an invariant.

Relatedly, some months ago I asked how to correctly record food in a double-entry bookkeeping system [1] and it triggered a 1300-words, 8-levels-deep discussion. We should remember more often than accounting is a degree by itself.

[1] https://news.ycombinator.com/item?id=39992107

jasim

8 months ago

> Instead of using negative numbers, Accounts have normal balance: normal credit balance literally means that they are normal when its associated entries with type credit have a total amount that outweighs its associated entries with type debit. The reverse is true for normal debit balance.

But that is an interpretation made by the viewer. A customer typically is an asset account, whose balances are in the debit column. But if we somehow owe them money because let's say they paid us an advance, then their balance should be in the credit column. The accounting system need not bother with what the "right" place for each account is.

It is quite practical to have only a simple amount column rather than separate debit/credit columns in a database for journal entries. As long as we follow a consistent pattern in mapping user input (debit = positive, credit = negative) into the underlying tables, and the same when rendering accounting statements back, it would remain consistent and correct.

fendy3002

8 months ago

> It is quite practical to have only a simple amount column rather than separate debit/credit columns in a database for journal entries. As long as we follow a consistent pattern in mapping user input (debit = positive, credit = negative) into the underlying tables, and the same when rendering accounting statements back, it would remain consistent and correct.

Another benefit of Credit / Debit side on double-entry bookkeeping is you need to balance both side in a single transaction. Say if the user account 2003201 is in Credit and it got an addition of 1000 value, a same value need to be added on Debit side. If it's (1) a cash topup, then 1000 value need to be added to Cash account (let's say 1001001) on Debit side. Otherwise if it's a transfer (2) from another user account 203235, then the account need to be Debited 1000 value as well.

It's Asset = Liabilities + Equity, while the left equation is Debit (which increase value when a Debit transaction happen, and the right equation is Credit, which increase when a Credit transaction happen. In (1) case, the cash account increase since it's on Debit account, while in (2) case, the user account decrease because it's a debit transaction on Credit account.

anonzzzies

8 months ago

I saw this so many times. It is rather depressing. Even experienced devs hardly know enough about floats, rounding etc to implement these things, but they do. We were asked to troubleshoot a project for an online insurer ; the devs did two conversions; js and php. So the amount from the frontend ended on the server and got converted and got into the db as an entry. Definitely depressing how frontend numbers are converted and actually this means something to the end result. But the php conversion was also wrong, so it was often cents off. I believe people should never be allowed to have a 'senior developer' job again, but this is really quite commonplace. Most companies don't have enough revenue to notice or care, but given how often we see it, I would say it's actually normal.

grishka

8 months ago

While I've never worked on financial systems myself, I was told many times that monetary amounts should always be stored and manipulated as integer amounts of smallest units of currency (cents). One reason being that integer math is always exact, while floating-point isn't. And when money is involved, you want your math to be exact so everything always adds up every step of the way. So I'm always calling it out when someone is using floats to store money.

naasking

8 months ago

Everyone keeps focusing on double entry book keeping, but that's a ledger that's more suited to manual book keeping. We're in the computer age, people should be using the richer accounting model of REA:

https://en.wikipedia.org/wiki/Resources%2C_Events%2C_Agents

You can see that this model has all of the features discussed in the article, and then some, and REA events map naturally to something like event sourcing. You can project a REA dataset into a double entry ledger, but you often can't go the other way around.

MarkMarine

8 months ago

I read the source you sited but it seems like a foundational part of the model is basically double book accounting:

“At the heart of each REA model there is usually a pair of events, linked by an exchange relationship, typically referred to as the "duality" relation. One of these events usually represents a resource being given away or lost, while the other represents a resource being received or gained.”

This is what I see as the main difference between single book accounting and double book accounting, with REA having some OO things added to the model to more accurately represent business objects when a computer is used. What am I missing about REA that makes it better than double book as implemented in the way this post was talking about implementing it?

ndjdjddjsjj

8 months ago

I feel there is no excuse. Startups cut all sorts of corners but why? Get rich at other people's expense?

The tech founder should know their shit prior to building it so that time and runway and deadlines are no excuse. Really if you are doing fintech you should have employment experience and understand all the operations well.

Otherwise they are no better than say home builders who mismanage and go bankrupt. Or less geneously, they are conmen.

alexey-salmin

8 months ago

> Instead of using negative numbers, Accounts have normal balance: normal credit balance literally means that they are normal when its associated entries with type credit have a total amount that outweighs its associated entries with type debit. The reverse is true for normal debit balance.

I didn't understand this part, can someone give examples of good and bad approaches for both credit and debit accounts?

dataflow

8 months ago

Could someone explain something fundamental to me about the need for double-entry accounting? Why can't your source of truth instead be proper SQL tables following these schemas (using Python tuple/list notation for convenience)?

  transactions: [
    (txID, timestamp, [
      (accountID, delta, otherInfo),
      ...
    ], reason),
    ...
  ]

  accounts: [
    (accountID, routingNumber, accountNumber, ownerID),
    ...
  ]
Crucially, notice that "accounts" don't track balances here, and so I'm asking: why would I need TWO entries per transaction, and why would I need to do my own tracking of the balance of every account, when I can just keep a single entry per transaction in a proper ACID database and then build any view I need on top (such as running balances) with a battle-tested projection mechanism (like a SQL View) so that I still track a single source of truth?

jes5199

8 months ago

what you’ve sketched out already looks like a double-entry bookkeeping system, if you have the constraint that all of the deltas in your array on your transaction sum to zero (so money always _moves_ and is not created or destroyed)

calling it “double” might be misleading - in the ledger systems I’ve worked on, a transaction that has only two entries - a single source and a single destination - is actually rare. In practice, a payment tends to have a source, the intended destination, a secondary destination for platform fees, possibly another destination for taxes.

And much more complicated situations are possible - I currently work in medical billing where there are multiple virtual internal accounts involved in each transaction for different kinds of revenue (including, for example, money we billed for but don’t expect will actually get paid)

so a transaction becomes a bundle of a set of several debits or credits that happen simultaneously and sum to zero. if you have that, you have double-entry bookkeeping, even if you schema puts it all in one table

bvrmn

8 months ago

You transaction includes multiple entries. Your schema is multi-entry by definition :P

It's funny how many commentators here confuse debit/credit with double-entry.

pkd

8 months ago

SQL or not doesn't really matter. Double entry provides redundancy, and redundancy allows for auditing. It's like checksumming to detect errors.

j45

8 months ago

Startups could improve in 2 wasy:

Perhaps popular culture should review the use of nosql databases, and then spending tremendous effort to try and make it into a relational database, while nosql databases can be setup to be eventually consistent.

Money, and accurate accounting don't work too well when the inputs to a calculation of a balance are eventually consistent.

So much work is avoided at times it seems to not learn SQL that it can rival or end up being more work once NOSQL leads you down the path of inevitable relational needs.

In addition to this, maybe it's time to build honeypot ledgers with a prize and reward in it for anyone who can hack or undermine it, similar to vulnerability bounties. Run for long enough, it would determine at least that one side of security mistakes in startups being reduced. Run a sprint, with prizes and let them run like daily deal fantasy and watch things harden.

bsder

8 months ago

Losing cents is because somebody didn't use Decimal for currency. And that's just flat out malfeasance--possibly even regulatory malfeasance.

Double entry is irrelevant, here.

I love the "use crypto" to fix the problem suggestion. LOL!

It's kind of poignant since crypto has given me a fantastic club to beat executives over the head with whenever they want to do stupid handling of money. Since crypto has so many decimal places, you break dumbass floating point handling of money immediately and irretrievably for way more than just "a couple cents". Your CFO starts jumping up and down really excitedly about handling money properly when he has to worry about a rounding error forcing him to compensate a Bitcoin price jump/crash.

phonon

8 months ago

It could be rounding issues as well (roundings do not all cancel out), or having to divide an even amount unevenly (how do you pay $10 in three "equal" installments?) Even with infinite decimal precision there will be special handling involved.

supermatt

8 months ago

The payments industry is rife with nonsense like this.

At a fintech startup I was working with, we built a "shadow ledger" because we couldn't trust a 3rd party PP to give us the correct information, which would otherwise allow double spending on accounts.

We tried 3 different (major!) PPs - they ALL had similar flaws.

cryptica

8 months ago

It's just like it was with pen and paper... Everyone trusts what is written on the financial statement but when they come to withdraw the cash, all at once, suddenly everyone finds out that the gold isn't there.

It's the same with computer systems. The charts show something, but until enough people decide to all withdraw their money or sell their stock at the same time, nobody has any idea that the money or asset simply isn't there or nobody knows just how frothy the valuation is.

Social media and search algorithms are highly optimized to ensure that people don't sell or withdraw stuff at the same time. Modern media directs massive attention towards certain topics as a way to draw attention away from other topics which could collapse the economy.

Also, imagine a bank has a serious bug which causes millions or billions of dollars to disappear every year or creates extra illegitimate dollars. Imagine they only discover this bug after a few years of operation... How likely is it that they will report it to an authority? They didn't notice it for years, why not pretend they didn't notice it for a few MORE years... The incentive to delay the reckoning is an extremely powerful one.

superzamp

8 months ago

Precisely. Designing a transactional system can be solved. Designed a transactional system that properly entangles the bits with the assets they represent is the hard part.

tonyhart7

8 months ago

Hey, great works

I literally build the same system and ask for HN Ask but no one answer it and already using double entry and ledger based on some research and AI advice lol

I implement all the check its mention in your page and quite satisfied using postgress constraint and trigger function to detect abnormality before bad row is inserted

but the problem now is populate the database using fake data because how rigirous my constraint and rule are, I need to simulate real world use case and duplicate it on development machine but this are hard

if you ever read this, can you give some advice please, because simply looping that function is not gonna work because constraint of double entry and ledger

knorker

8 months ago

A related pet peeve of mine is UIs that round off your investment holdings or cash.

Sometimes to 2 digits. Sometimes 3. Or 4. How many significant digits are there even in fractional SPY holdings? You just resort to looking in all the places (statements, transaction history, overview dashboard, …) and going with the one that shows the most digits, and assume that's all of them.

Big and small companies do this. And when I've reported it, they don't care. They don't see the problem.

zerop

8 months ago

Building fintech platforms are not easy, specially for large systems which not only require robustness, scale, but also traceability, daily reconciliations. We even had features to undo and redo (correct and replay) transactions from a time in history.

I did not like finance much, but building a fintech system really teaches a lot, not only from technology perspective but from managing stakeholders, processes, compliance, dealing with all kinds of finance-specific issues.

One should always work in fintech, at some point of time in career.

taneq

8 months ago

Once upon a time I worked for a company that made accounting software. While there I had to explain to the other devs:

1) That being able to escape characters in CSV files is kind of important. (Some customers noticed that a single quote in a transaction description would make the importer silently fail to import the rest of the transactions... :S ), and

2) Why it wasn't good to store all dollar values as doubles. I'm reasonably sure I quoted Superman 3 at some point during this discussion.

michaelcampbell

8 months ago

One of my $previousjob's managed to invent their own kind-of sort-of double entry accounting system, but not quite. So whenever anything went sideways (and we interacted with Stripe, so something always went sideways), it required a lot of manual intervention with changing of amounts in various tables.

I pushed for offsetting entries instead, but was overruled. It was a _nightmare_.

Thankfully, it is not $previousjob.

knorker

8 months ago

If the author is reading this: Please stop hiding the scroll bar. It's user hostile.

I want to see if I have time to finish this article before I have to leave. And now I have to waste time copy-pasting the contents into a text file, just to see where I am.

Edit: Actually, it's invisible only in Firefox. Chrome shows the scrollbar still. So this may be a bug in the author's CSS or something.

pclowes

8 months ago

I think “make it work, make it right, make it fast” is a solid approach.

It appears the startup in question just never even did step one. A financial system that is imprecise does not work.

Additionally that mantra is typically applied at the story/issue level in my experience as in:

1. Get the tests passing 2. Get the code clean 3. Identify scaling bottlenecks and remedy if necessary.

kwakubiney

8 months ago

At $currentJob, we have decided to stick with Fineract for ledger maintenance and that helped us scale without having to deal with the nitty gritty details of accounting. If you don’t have the necessary expertise around you, offload that responsibility to a third party system that is maintained by those who do.

bvrmn

8 months ago

I don't understand what the difference in modeling between:

Entry(account, direction, non-negative amount), direction is debit or credit.

vs

Entry(account, signed amount), + is debit, - is credit (for example).

It's a two way mapping and should be equivalent. Unless debit or credit amount could be negative. But as I understand it's a big NO-NO in accounting.

thecleaner

8 months ago

I guess the lesson here is to not bank with an a16z company. I have been using Wise for years. Very very happy.

yapyap

8 months ago

> It’s just that…it goes without saying that fintech companies should know better

haha, if only.

move fast, break everything if it gives profit

artyom

8 months ago

The number of engineers I've met that realized only too late that, by definition, floats and doubles are absolutely *not good* to deal with money is unbelievable. At this point it should be universal knowledge, but it isn't.

x3n0ph3n3

8 months ago

nit: double entry accounting only goes back to the 13th century, not thousands of years.

analog31

8 months ago

Fibonacci, if I remember correctly.

chris_wot

8 months ago

It took me a while to realise what double-entry bookkeeping actually was, then one day it hit me: it's literally just the flow of money through a system. Understand that, and you understand the cornerstone of accountancy.

intelVISA

8 months ago

if only accountancy 'learned' from OOP and just had one mutable balance for each account :)

afiodorov

8 months ago

Blockchains are examples worth learning from about how to build ledgers. Append-only transactions which start as pending and then eventually confirmed. Balances are cumsums of confirmed transactions.

__MatrixMan__

8 months ago

> Not losing track of money is the bare minimum for fintech companies.

The last fintech I worked for had a joke about how you weren't really an employee until your code had lost some money.

jeffnappi

8 months ago

Feeling vindicated for the double entry transaction system we built at clearvoice.com for our two-sided marketplace, leveraging the fantastic DoubleEntry Ruby Gem from Envato.

charlie0

8 months ago

As an SWE, I've seen how the sausage is made and is the reason why I'm very very careful when using fintechs for anything.

PeterStuer

8 months ago

This was many years ago. I has an injurie that prevented me from driving, so I asked if there was any project on the backburner I could do on my own in 3 weeks without too much interaction (this was long before easy remote work infra).

Sure there was. A financial services company wanted to replace their repayment plan generator that ran on an aging AS/400 with someting running in .Net on Windows.

I dug in an learned all about time value of money, numerical formats and precision, rounding strategies, day counting strategies for incomplete periods (did you know there are many dozens of those) etc.

I made everyting as configurable as I could so we had the option to offer to other financial service clients by just setting up a different profile.

Since I had no interaction with the client before the first presentation meeting, I put in what to me felt like the most plausible config (I had managed projects in the domain before so I was not completely clueless to the mindset).

We had the meeting. I showed a few calculated plans which they compared on the spot to the AS/400 output. I deployed on a test VM for them so they could do more extensive testing. Code was accepted with 0 change requests and put into production shortly thereafter. Don't think they ever changed from the default settings.

elzbardico

8 months ago

An incredibly depressing thing in our profession is how we collectively lack memory.

We routinely re-discover stuff that probably was already solved by a quiet lady writing a CICS transaction in a s-360 system in 1969.

Yeah, dealing with money and especially others people money without double entry bookkeeping is a bad practice.

But this particular problem is the consequence of the choice of using floating point binary math to deal with monetary quantities.

Given the fact that most modern languages have very awkward support for arbitrary precision decimal numbers the most sensible way to deal with money usually boils down to store it as an integer.

user

8 months ago

[deleted]

BlueTemplar

8 months ago

I don't see how these are related (except of course for the errors in the second that could be caught by the first), and hopefully nobody in fintech is so ignorant/dumb to use floating point for money ?

Or are we talking about random scammers as 'fintech' now rather than banks (the fintech of which might indeed be old enough to be still running on COBOL-like systems) ?

morbicer

8 months ago

Hot take: I blame the culture of obsession with CS topics, leetcode, tail recursion, dependent types etc and not enough focus on solid engineering and domain knowledge in areas encountered by 90% of real life jobs.

Starts in the education and perpetrates via hiring, blogosphere and programmer celebrities.

myflash13

8 months ago

Wait, why can’t I just use a Postgres database with the Decimal type and call it a day?

zie

8 months ago

You need to keep state, so that when something goes wrong you can figure out that something went wrong.

PG's rounding can be summed up as: we round to the nearest digit you specify, but how we get there is mostly undefined.

Every bank and organization you do business with(especially if you cross-jurisdictions) will likely round pennies differently. PG and the decimal data type(which is actually the numeric data type) are only part of the solution, as they won't handle rounding pennies correctly for you, but they will store it, if you round properly yourself.

PG also has a money data type, but it also doesn't let you specify the rounding rules, so you have to be careful. Also, money is tied to lc_monetary, which is handled outside of PG's control, so you have to be very careful with it, when moving data across PG instances, or you might be surprised. Also, it doesn't let you specify which currency this money is, which means you have to store that yourself, if you ever have more than one currency to care about.

If you don't care about balancing to the penny with your external organizations(banks, CC processors, etc), you are 100% guaranteed to have a bad time eventually.

p4bl0

8 months ago

Interesting article, thanks for sharing! I believe the things it explains are very important, and not only in fintech. A few years ago, even in the case of trying to keep track of a small organisation spendings and incomes in a spreadsheet I had some of the problems that are mentioned in the article and had to completely rethink and remake the spreadsheet in order to make it actually work.

But apart from the technical points, another interesting thing in the article is its introduction and what it says there about money as a concept. Yes, money is debt, both philosophically and technically it is the right way to think about it. And I believe that's something the crypto-assets industry and enthusiasts as a whole fundamentally gets wrong (mostly because of the libertarian political point of view blockchain tech has been designed with).

mrkeen

8 months ago

I'm not sold on double entry here.

If a new transaction enters the system now, I could follow the advice and record it as two sources of truth. Or I could just record it once.

If I could turn a single transaction into two ledger entries today, I could do the same later if I needed to.

jes5199

8 months ago

the systems I’ve worked on that use your design end up in a state I call “the zoo” - because you never actually have simple transactions like you’re imagining. You end up with one object for credit card payments with stripe fees, another one for purchases with taxes, another one for ACH payouts, all meaning different things

double entry is actually the simplest projection of any possible billing system because anything that moves money can be described as a bundle of credits and debits that happen simultaneously and sum to zero. so you get accounts for taxes collected and fees charged automatically and don’t have to write custom logic for each thing

zie

8 months ago

You are thinking it's duplicating work, but it isn't. You are simply recording the whole transaction in a double entry system. i.e. you know where the money came from and where it went. That's all a double entry system does, records both sides of the transaction.

This means when something goes wrong, you have a chance to figure out what in the world happened.

That's it. If you want to think of it as a state machine, you are storing all the state that happened with the resource you are tracking(usually money). This way when some part of the resource is missing, you can go figure out what happened.

Think about it another way: always record enough of the transaction, that if you were required to sit in a courtroom in front of a jury and explain what happened, you could do so. If you can't do that, you didn't store enough information.

cts1

8 months ago

There's no redundancy. Imagine you have 2 buckets, you reach into bucket A and grab some cash and put it in bucket B. $x dollars leaving bucket A is entry part 1, $x dollars entering bucket B is entry part 2. That's all it is. I honestly don't understand what single entry is by comparison - it sounds like losing an essential attribute of the transaction.

codr7

8 months ago

Keeping entries separate makes it easier to deal with data for specific accounts, the transaction will contain both of them.

matheusmoreira

8 months ago

Learning and using the ledger-cli accounting tool taught me a lot about this. It's incredible how messy seemingly simple things can be and how much trouble a bunch of cents can cause. Seems to be the accounting version of off-by-one errors. It is very tempting to just write them off as losses and forget about them forever.

Rounding in particular is a truly endless source of trouble and has caused me to chase after a lot of cents. Dividing up a large payment into multiple installments is the major cause of rounding in my use case. Life starts to suck the second things fail to be evenly divisible. I created an account to track gains and losses due to rounding and over time and it's adding up to quite the chunk of change.

Hilariously, the payment systems would charge me incorrect rounded up amounts and then they would refund the difference to my credit card at some undefined time in the future. Tracking and correlating all these seemingly random one or two cent transactions has got to be one of the most annoying activities I've ever learned to put up with. Not only do I have to figure out why things aren't quite adding up, I have to patch things up in the future when they fix the mistake.

Why can't these things just charge the correct amounts? For example, imagine splitting up $55.53 into four installments. That's 4x$13.8825. They could charge me 3x$13.88 + 1x$14.89. Instead they round it up to $55.56 and charge me 4x$13.89, then maybe they refund me $0.03 some unknown day in the future. It's like the systems go out of their way to be as annoying as possible. Some systems do this silly single cent refund dance in my credit card statement even though they print the exact same 3xN + 1xN+0.01 solution in the receipt. Makes absolutely no sense to me.

It's getting to the point I'm trying to avoid this nonsense by structuring purchases so the final price is evenly divisible by some common factors. I never liked the .99 cents manipulation trick but I seriously hate it now.

theendisney

8 months ago

Im sure there is a point in the article but ive never seen a dancing cent nor can i imagine one. My numbers are all strings "5" is "5" forever. If one somehow ends up storing it as 4.99 why would the other entry be correct?

that_guy_iain

8 months ago

Your second sentence tells us your first sentence was a lie. You can clearly imagine one which is why you specified which data type you use for money. You know a floating point issue is an issue.

Now let's say your price is "0.00023" per unit and someone uses "213.34" units. Can you imagine it now?

pyrolistical

8 months ago

Btw for those wondering what to do instead is to simply use https://tigerbeetle.com/

jes5199

8 months ago

I found this paragraph

> Prerequisites: TigerBeetle makes use of certain fairly new technologies, such as io_uring or advanced CPU instructions for cryptography. As such, it requires a fairly modern kernel (≥ 5.6) and CPU. While at the moment only Linux is supported for production deployments, TigerBeetle also works on Windows and MacOS.

wha— what? why?? they must be solving some kind of scaling problem that I have never seen

saaaaaam

8 months ago

Thank you for posting this.

I’m trying to get my head around how to build a fairly complex ledger system (for managing the cost schedules in large apartment buildings where everyone might pay a different proportion and groups of apartments contribute towards differing collections of costs) and you’ve just massively accelerated my thinking. And possibly given me an immediate solution.

Have you used tigerbeetle in production?

2mol

8 months ago

I mean, tigerbeetle looks extremely cool (I've watched the project develop since its inception), and I trust them to be rock-solid. But saying "just use this project that is very new and unproven, written in a new and unproven programming language" is just pretty unserious. At least talk about pros, cons, risks, and tradeoffs.

revskill

8 months ago

Fintech is safe heaven for scam.

cromulent

8 months ago

> A double-entry system is an accounting method that tracks money at both its source and destination.

Nope. Double entry bookkeeping means every transaction is recorded in (at least) two accounts.

cromulent

8 months ago

To illustrate the difference (hopefully I get this right): A transaction needs to balance, left vs right.

For example, you receive a sales order for 1000€ of widgets, which cost you 600€. You ship them. You invoice for 1000€.

No money moved (hopefully you do get paid at some point though). However, you need to do some bookkeeping.

On the left side of the ledger (debits): Accounts receivable 1000€. Cost of goods sold 600€.

On the right side of the ledger (credits): Revenue goes up by 1000€. Inventory goes down by 600€.

These completely match. No money has moved, but the books are now up to date and balance.

Any transaction that does not balance should be rejected.

cranium

8 months ago

Double-entry accounting is like strongly typed programming languages.

Yeah it's a real pain to get started because you need to understand the core concepts first then fight to balance the transactions ("compile errors") before you have anything useful. And when your results are wrong, you know that at least it's not because of the basic stuff.

nobodywillobsrv

8 months ago

SV Tech Bros roasting themselves on HN.

Every engineer in London is laughing at this right now.

Wow double entry! Ledgers are hard! Wow. So true.

swyx

8 months ago

> And yet, I used to work for a startup that, on every transaction, simply lost track of a couple of cents. As if they fell from our pockets every time we pulled out our wallets.

is this not the literal plot of Office Space? did you check for a Michael Bolton employee?

yfw

8 months ago

Great post!

Uptrenda

8 months ago

The blockchain industry is basically founded on incompetence. If you go back to early Silk Road and Mtgox you can see some shocking things. Like the founders were posting questions on Stackoverflow about database basics. They then ended up building systems with race conditions (withdrawal code for both systems.)

See, if you're an engineer (from a web background) you might think that using a regular backend language with a relational DB would be fine. But this service is typically optimized for concurrency. That might sound amazing. But what you really want is a simple queue. This ensures everything is applied in order and makes it impossible to accidentally have transactions execute on stale state. Plus, a queue can be executed amazingly fast -- we're talking Nasdaq scales. If you use a standard DB you'll end up doing many horrible broken hacks to simulate what would be trivial engineering had you used the right design from the start.

You've got other things to worry about, too. Financial code needs to use integer math for everything. Floating point and 'decimals' lead to unexpected results. The biggest complexity with running large-scale blockchain services is having to protect 'hot wallets.' Whenever an exchange or payment system is hacked the target is always the funds that sit on the server. When the industry began there were still many ways to protect the security of these hot wallets. The trouble is: it required effort to implement and these protocols weren't widely known. So you would get drive-by exchanges that handled millions of dollars with private keys sitting on servers ready to be stolen...

Today there are many improvements for security. New cryptographic constructs like threshold ECDSA, hardware wallets, hardware key management (like enclaves), smart contract systems (decentralized secret sharing... and even multi-sig can go a long way), and designs that are made to be decentralized that remove the need for a centralized deposit system (still needs some level of centralization when trading to a stable coin but its better than nothing.)

I would say the era where people 'build' ledgers though is kind of over. To me it appears that we're organizing around a super-node structure where all the large 'apps' handle their own changes off-chain (or alternatively based on regular trust.) The bottom layer will still support payments but it will be used less often. With more transaction activity happening on layers above. I think its still important to scale the chain and make it as secure as possible. Bitcoin has a unique focus here on security above everything else. Making it ideal for long-term hedges. For every day stuff I can't think of any chains that have more credible R & D than Ethereum. They seem to even have been doing research on the P2P layer now which traditionally no one has cared about.

thaumasiotes

8 months ago

> Financial code needs to use integer math for everything. Floating point and 'decimals' lead to unexpected results.

Decimals are integers. There's no difference between integer math and decimal math, only in the meaning of the bit pattern afterwards.

exabrial

8 months ago

The whole "move fast and do a shit job" is laughably stupid. It's pure incompetence and egotism and a product of excess salary in the market.

Doing it the right way doesn't take any longer than doing it the shitty way. Successful startups focus on extreme minimalism, not focus on doing the worst possible job.

jprosevear

8 months ago

Except if you don't actually know what to do, or even what concepts to apply. This is where I see many start ups fall down.

lmm

8 months ago

OK but apparently they did get to make the startup mistakes? They built the quick thing that worked well enough, got some customer traction, and then when they had bugs they were able to rework it and continue.

Frankly I'm not even convinced that double-entry is the sole right answer in this space. There are things you need to be able to represent, but the original reasons for doing double-entry (making errors when subtracting figures) no longer apply.

(I've worked at investment banks and fintech startups)

NikolaNovak

8 months ago

Hmm, can you elaborate on the original reasons for double entry and them not applying any.kre? I'm not in that space and double entry always seemed an extremely weird, arbitrary requirement / paradigm. Thanks!

simonvc

8 months ago

Cool post, wish it existed 2 years ago when we started building Pave Bank, or 10 years ago when we started building Monzo.

If you're starting a bank or need a ledger these days (and aren't using a core banking provider that has one), then i usually recommend Tiger Beetle.