If and when a large number of companies blindly turn over their accounts payable workflow to some AI agent system, it'll be very interesting to see the "social engineer the LLM" methods that fraud people use to get money sent to them. Basically the same idea as the ancient "send a fax with a bill for an unsolicited delivery of copier toner to 30,000 businesses" but taken into the modern era.

edit: There's already a number of LLM which are intended for outgoing data loss protection to redact or prevent PII from escaping. Is anyone specifically working on a training set and agent that is specialized in reviewing "is this legit to pay", as a sub-task or filtering step in an AP workflow? I suppose it's a GIGO problem, as it would work best only if you have suppliers enrolled in some kind of existing db, with a specific contracted format for invoices, and correlating with project numbers/cost codes.

My wife (head of accounting for a small business) has been working on automating large parts of her job using AI.

It's not completely reliable and the human cannot be taken out of the loop, but the number of menial tasks she's been able to automate has been really cool. A lot of processing data that arrives in non-standard formats, generating documents based on that data, etc.

She still has to review everything, but her workload is way down, and when her assistant quit she automated away his whole position.

Hahaha non-deterministic accounting probably won’t fly well with the IRS

Hey, the author of the benchmark here.

The benchmark data was prepared in April 2026 (when I was manually doing our VAT return with my co-founder). The invoices were indeed found manually.

Currently we're using a custom "invoice searcher" built on Kimi 2.6 (in our testing several weeks ago it outperformed Opus 4.7; it was just more persistent).

Ultimately, I still verify everything manually after the model is finished fetching invoices for the month -- but it's a great help to have all the invoices already found (usually correctly).

AI helps automate things that didn't already have rigorous formatting and structures available as input... and that's really all it does (99% of the time).

Doesn't matter how many more nines you add, rigorous formatting is still required. In some cases, it has teeth with compliance standards. Those standards cannot be compromised because there are already a lot of other layers contributing inaccuracy. It all adds up.

In most situations, you could just hire a junior dev (or an intern! remember those?) write some CSV scripts and call it a day. Cheaper and auditable too. Those scripts can't change anyway until standards are revised.

I'm still not seeing the benefit outside of solopreneur efforts and shady businesses wanting to launder blame.

> And I would be pretty nervous about asking any of the frontier LLMs to retrieve invoices:

I watched an accountant YouTuber reviewing a new AI-driven personal finance app the other day (I really need to touch grass), and it started out just fine. He had seeded the account with a bunch of his data and was able to ask questions about which categories had the most spend, etc.

About half a dozen questions in, he asked it to calculate a certain segment of his spend (and being an accountant, he had his numbers memorized), and he immediately got back a calculation that he did not expect. So he asked for an itemized response and it hallucinated line items that never appeared in his account data, which he pointed out to viewers. He followed up with the chatbot with "where did line item X come from?" and the bot acknowledged that it wasn't legit. He immediately noped out after that, and who could blame him?

> If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent.

From my understanding, you are the person signing off on the paperwork that is submitted to the IRS.

There is this cache 22 with taxes.

You are responsible, but you outsource it to a accountant. Because you are not knowledgeable about the taxes. But you are expected to be knowledgeable to understand the tax documents that you submit to the IRS. That is why the accountant always ask you to review the documents and sign them like 20 times.

The same applies when you run a company, depending on the country, you need to prove yourself knowledgeable in accounting, before you are allowed to run a company. Normally that is included in a university degree, but if you have a middle school diploma, you need to do a official examen to get that degree.

Whatever you submit for your company, you are again responsible. Even if you hired a accountant.

So while technically, if a accountant makes gross mistakes, the bill will always fall in your lap, because you are expected to understand the reports you submit to the IRS. And catch any errors before doing so.

With the IRS, the burden of proving your innocents is often put you. Its because the good faith argument can be misused easily. That is why the buck stops at you.

So using a LLM or a accountant, really does not matter. Sure, a accountant can go to jail if there has been major issues (its not going to be with one client issue).

But you can lose your house / company, have your life ruined by whatever you submitted.

“A computer can never be held accountable, therefore a computer must never make a management decision.”

– IBM Training Manual, 1979

https://simonwillison.net/2025/Feb/3/a-computer-can-never-be...

1 Just as with traditional non-ai ways of doing things, there is a deterministic layer that can trigger when things are off. This can range from letting the ai know to stopping before a human confirms.

2 Usually, specially for SMBs, nobody goes to prison over accounting errors. That's because SMB owners make mistakes all the time and, because of that, authorities are fairly practiced in understanding how mistakes look, and how fraud looks.

I agree with you on every point but it is interesting to see real world benchmarks like this. Showing the standard benchmarks that all LLMs use is not only boring but at this point likely gamed or even has issues (according to OpenAI) by every LLM.

Sure! In classic accounting firms you'd typically have one accountant doing the books and then the other reviewing the first accountant's work. Given the model's accuracy (and models will only get better), only by automating the "doing" and keeping a human to do the review part we will already shave huge amount of time and costs without risking the compliance part.

The LLM is, after all, trained on a world with a lot of tax fraud in it.

> This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison.

I don't disagree that tax fraud is bad but accuracy really doesn't matter 1/10th what people think it does.

Something to keep in mind in the EU with VAT is that there are approximations, errors and then huge frauds like the infamous "carousel fraud". Sadly the carousel fraud is all too common in the EU: this one gets people really sent to jail. But it's not an error a LLM shall make that could lead to a carousel fraud taking place: it's an elaborate scheme.

Now... If you're self-employed and have to pay the VAT (which may or may not be the case depending of the type of self-employment work you do) or if you're a SME and you've got approximations and errors well it's really not a big deal.

Because the public servants in charge of collecting the VAT and verifying that you filled your stuff correctly... Do make shitload of errors too. And then, depending on the country, these public servants have a special privilege: they can decide an amount of VAT+fine (if there's a fine) that you agree with and if you agree with it, their number is the number.

It's everything but a correct number. It's a number they ended up on that is "close enough" (for some definition of "close"). Sometimes by hammering number after number on a little calculator and then ending up saying:

"We ask you to pay 3500 EUR of additional VAT. If you sign here and accept, the IRS (equivalent of the IRS) won't be able to bother you ever again for that year/years".

This notion that, somehow, there's "one" correct number and that public servants paid something like 2 K EUR net per month (which may be the reason why so many are so prompt to ask for bribes btw) can determine it in a few minutes is ridiculous.

It's simply not happening. Usually the accountant doesn't know the exact number. The SME owner doesn't know the real amount of VAT due. And the public servant(s) in charge of a VAT audit, if any takes place, certainly as zero clue as to what the exact VAT number should be.

The rules are way too complicated, there are way too many special cases, and some things are simply extremely hard to take into account (for example during one audit the person told me he disagree with the 90% of 21% VAT deducted for my ISDN line --yup it was a long time ago-- but then I told him it was during a previous TVA audit, a few years before, that the VAT auditor, seen my job, fixed at 90%).

For some stuff you can opt for a fixed number: say a car, there are rules, depending on the country, where you can, say, get back 75% of 50% of the VAT. But people can also, depending on the country, opt-in for the "real usage": you do 12 387 km during the years, how many were really for work? 6 287 km ? Or may 6 275 km?

What if one day while going to work, you took a detour because you want to drop a gift for your grandma's birthday? How were these kilometers accounted for?

Restaurant bills? What if one of the person wasn't work related (your nephew happened to be in town and you had to take car of him and you knew the clients well so they didn't mind: how does the accountant account for that)?

Gifts to client. You got a discount buying 6 packs of bottle of champagne, but not five (no discount if only five), so you bought a sixth one. But that sixth one you kept for you.

Nobody computes those amounts correctly. Nobody.

And don't get me started on the public servants who'll give, during an audit, an unfair advantage to people from their community (whatever that community may be) or those, literally, banging underneath the table while talking about their next vacations: they're telling you an "arrangement" can be made. Basically you give that one person some money (cash bills of course) and, boom, the 20 K EUR of VAT you had to pay suddenly gets lowered down to 5 K EUR.

You refuse? It'd be too bad if those 20 K became 25 K EUR wouldn't it? Mafia-style.

This is all happening. Don't tell me it's not happening because I know people on both sides of these "trades".

Depending on the EU country, it can even be the norm.

As rational people who like determinism (at least I do), at first glance it looks like a SME's accounting is something square but it really ain't. Or let's put it this way: it there's one number that is the really the amount of VAT the SME has to legally pay, I'd guesstimate 0.01% of the SMEs out there pay, to the cent, the proper amount.

All this to say: the approximations, hallucinations and errors of LLMs in accounting for VAT are probably just on par for the course.

P.S: ah yup... The boss (say major airline company) giving a business card and its PIN code and asks to take good care of the clients: and an employee takes them to strip clubs, they all get drunk like crazy, and explodes the expenses on the business card. Way, way, way more than what is OK to put into expenses to entertain clients. Well guess what: I know for a fact this happens too. And I know for a fact this ain't accounted for properly either.

It’s almost impossible to be in a situation where your taxes end up this wrong you’re accused of fraud. I honestly do not believe my accountant does anything better than AI or AI generated code that does calculations to see what you should be paying.

Info on the founders coming soon -- we're just going public with this.

For slightly out of date founder bios (both Adam and Iva) were also co-founders here:

https://www.biomage.net/our-team

> I don't know how taxes work in Europe, but in the US being "essentially" correct is not good enough for the IRS.

It often is. On multiple years they've found errors in my return, and they just fix it for me and bill/refund the difference (usually a refund).

And obviously, the IRS is not going to quibble over 7 pence, given that you round up/down to the nearest dollar anyway.

The accounting concept of materiality doesn't exist in the US? What about petty cash?

Hey, author of the blog post here.

I was one of the human book-keepers for this benchmark (the preparer; my co-founder verified the VAT submission once ready), and given that at the time of doing this I knew I was eventually going to use this data for evaluating the models, I was super careful. So I guess this is a "good book-keeper". In the previous company our book-keepers made lots of mistakes; some serious enough that we had to restate our company's accounts.

It’s a service that provides an AI bookkeeper so it’s a pretty relevant metric.

You'd care if you had a human bookkeeper and you were considering replacing them with this company's AI bookkeeper.

Note to self: traverseda doesn't have 2factor auth on his email and his LLM seems to have full access. Hmmm

We've got integrations with major UK banks. Curious if you'd like to use a polished product or be more interested in bank-feed-as-an-API type of use case? What banks do you use?

Thanks, that's useful! We did use ChatGPT 5.5 for some time and it did perform pretty well too (of course more expensive than GLM 5.2). We tried Claude 4.7 and 4.8, but we found both models to be "lazy" and very expensive. Claude would always rather prefer the route of saying that the evidence was not found or something is incomplete, rather than put more effort into finding/repairing the particular issue.

I don't know how much a tax lawyer/ account would quote for this in California, but I saved 3000 EUR/ tax year on a German Steuerberater in a similar way.

Why do you think you were audited?

The real test to see if AI is just a rich person thing will be to see how the tax authorities treat it, even for more complex returns.

They can save humans for the really complex edge case stuff but at the end of the day, the tax code is just checkboxes and input forms that get boiled down into Integers, Floats/Doubles and enumerated choices with some Strings for deductions

The problem with LLM's is that they could work correctly for months and years and then do something egregious which will will go unnoticed because of the misplaced trust one develops on a system that "just seems to work." Get flagged for an expensive audit and there go all the savings and then some.

>My accountant hasn't complained yet

They're not going to unless it's obviously and egregiously wrong - the risk on quality of input remains yours. It's the tax version of garbage in, garbage out. They're just guaranteeing the processing step.

Very cool!

We've used the following CLI to do the freeagent upload:

https://github.com/anjor/freeagent-cli

How are you dealing with finding the receipts? Would you like to try a receipt finder that grabs them from your mailbox/ google drive?