The American manufacturing industry is scrambling to transfer some of the industrial trade and process knowledge from the retiring remnants of pre-offshoring industrial America into the two generations of people that didn’t have apprenticeships, pensions, (often) union membership with good benefits, etc. Trade schools were very expensive for industrial trade skills that were seen as dead-end careers compared to white collar jobs or non-manufacturing trades, and almost none of the companies saw any reason to spend all that money training people into expensive skilled professionals when they had all the professionals they could possibly need for decades after the industry largely collapsed, here. Keeping the workforce viable was always someone else’s problem— the individuals, the governments, larger companies— until it was suddenly everyone’s problem. They spend decades myopically chasing quarterly results before they realized they were running towards a cliff.

> but we really need to think about how the next group of people are going to learn the skills and nuance to applying them

That's what school is for. For the last 150 years we've automated a bunch of things, and then we send people to school to learn how those automations work, so they can make better automations.

Maybe law school will have to be a year longer. Or maybe some forward thinking firms will have new associates check the work of the LLM to gain that experience.

This is kind of a new complaint. Places like adp have being trying to automate away mundane hr tasks. Oracle and SAP have been trying to automate away the mundane admin tasks. Siemens does it for OT. There's a huge long tail of trying to solve this as if it were a problem in almost every field. This has been going on for decades and no one complains that we will lose expertise in these fields if the drudge work goes away. But bring in an AI... and everyone is suddenly concerned about corporate drudgery.

Anyone who does not do the grunt work of learning how things work at a granular level will run into trouble when their agent suggests "let's re-write the firmware!" and the user says "sure, whatever that means", the model will get stuck or break something, and there will be no one around to understand what is going on.

With strict version history that may be okay, but hopefully their agent already thought of that for them too!

I don't think toiling away at mundane tasks is the only way to build expertise in a field, or even a particularly good one. Today's farmers barely touch hand tools anymore and they're able to produce crops just fine.

This is hitting us hard in tech, and it weights heavily on me, how will the junior people learn if the tasks we all did to learn when we were junior are now automated away. Here is a very real writeup from an Affirm New Grad about what it was like to be a new grad over the last year - https://medium.com/affirmengineering/ai-reflections-of-a-new... (I run eng at affirm) - it takes a lot more intentionality to learn. Both by doing some things by hand, but being very vocal about what you don't know- "Allow your uncertainties to be visible, and slow down to address them."

> Whilst it's great to get the time saving from not delegating to a junior (LLMs are much faster at the basic level 1 work) there's going to be this big knowledge gap that you really only gain from actually doing the mundane boring shit.

That might be true, but that's not something for you to worry about as a customer of these services.

I don't worry about how my hairdresser practices either. I just pay for a good (enough) haircut.

> I don't really have an answer to that one other than still giving that work to junior staff, but they will then just utilise an LLM to help complete the job, and rightfully so, I'd have done exactly the same thing.

We can check how other industries have gone through these kinds of transformation before. Eg spreadsheets removed a lot of the worst drudgery that junior accountants and junior bankers used to have to do. But their industries adapted.

I think the new batch of engineers are going to think faster on their feet, but less precise.

We humans are _amazing_ pattern matching machines and learning “if you do this thing this other thing occurs” is literally what our brains were built for.

It’s just the current few thousand years of technical progress that have been a departure from this reality - we found out about analytical thinking and that by deeply analysing something you can do a shit ton more.

If you look at scientific progress before Industrial Revolution it’s all trial and error between patterns. And I feel like the new crop of devs are silently settling in into this very human and familiar pattern.

On the one hand I think it will probably be good for the “mental health” of people in this profession - you become more like an alchemist acquiring bags of tricks on what works, sharing incantations with others, because sitting down and learning _why_ it works in detail would be a time luxury few would be able to afford.

But that would mean the people under the hood that build the LLMs and who debug the really broken system would be charging the premium.

I honestly don’t think that in and off itself would be such a bad thing.

What I personally am really scared about is the price for labour that is sharply declining. I don’t worry about ASI too much, but when you read history it is absolutely full of stories of great human suffering because “labour was cheap” no need to improve working conditions because there is always someone else willing to take it up.

We had the Industrial Revolution _because_ of the plague making labour expensive and driving the benefit for innovation. Before that nobody really cared. With LLMs doing things becomes vastly cheaper, so the labour itself becomes cheap with it.

Now looking at the state of the engineered world - how absolutely shitty everything is (as opposed to what it technically could be) - there is a shit ton of stuff to be done so the work should be there to fill it. But maybe the capital will move in another direction - nobody can predict the future, I just hope we don’t end up in a dystopian hellscape for the sake of my children.

I've been having an identity crisis since generative AI started that is basically, "how am I going to learn anything with my ADHD brain when AI can do 10% of it immediately". That initial 10% would take me weeks or months to achieve on my own, and AI can do it right now, and that makes it even harder for me to bother. I hate it. I badly wish to do it myself because then I could achieve more than 10%, but now AI is right there and I just can't seem to justify the effort...

Interesting. How does Claude perform in code? Will it point out that "this input was never validated" or "the user is never authenticated" if such things are missing in the code? I have not really used LLMs to critique existing code.

Just want to say that the human standard tends to rise to match the challange. It’s always been the case historically so far.

I think the question is not “will AI able to do the work people used to do one year ago?”

but more

“will people, in the new environment where AI does the work that they used to do one year ago, still provide a competitive advantage?”

My gut feeling is: yes!

It's always been a silly argument that's also predicated on a massive fallacy which everybody is aware of but seem to conveniently ignore during these arguments: that humans never make mistakes.

On top of that, dealing with text is literally what LLMs are all about.

The problem with accuracy, hallucinations and just generally ensuring responses are grounded in factual stuff is in large part and engineering problem.

The expected next step then would be for such judges to use AI to read the documents.

You then get a decision that your AI agent explains to you.

What could possibly go wrong? You will have to ask your agent for that.

When did they test them? Models have a different level of capability than 6 months ago, IMO.

Not good for the law firms, but great for the rest of us!

A family member works as a paralegal, and they have way more work than they can deal with. Their estate people have 70 cases per person that are mostly waiting on people to have time to work on them. Speeding this up can mean more billable hours quicker, depending on how the speedup works, and then they can just take on more cases.

I want to minimize billable hours, yes. Lawyers are insanely expensive, to the point of being inaccessible to most.

I think part of why some people "trust" the CCP more is because if you live in the broader Western sphere, you're probably more likely to get hurt by the US government than by the Chinese one. Many of the same people would not trust the CCP more if they lived in China.

I live in the US, so I'd probably rather have the CCP snoop through my data than have the US government do it. If I lived in China, it would be the opposite.

also non US people trust US tech less and less. Or US in general. While china is slowly winning some ground, although from a low position.

FYI whenever you all say “CCP” you sound like Tom Cotton and whatever you say next has mostly lost any credibility.

It’s CPC not CCP, and it’s not like I trust them specifically, but they’ve certainly done less to make me distrust them than my government (USA).

It was on the fp https://news.ycombinator.com/item?id=49750694

For what it's worth this is sadly a behavior that isn't specific to one provider: https://news.ycombinator.com/item?id=48892468 (Grok doing the same). I do agree just because many model providers do it does not mean it's okay, though.

Was anybody suggesting that we should trust xAI?

> fully-autonomously

That qualifier is not in the terms, the same rules apply for human in the loop, or agent assisted work

> Anthropic has obviously never done anything like this

Here's a clear case of human review resulting in a report to authorities, which is the exact policy they're now extending:

https://www.tomshardware.com/tech-industry/artificial-intell...

> Anthropic has obviously never done anything like this

If you are ascribing intent ("Anthropic designed a system that intentionally does this") or emergent model intent ("OMG at AGI") then I mostly agree.

But there's a more pedestrian version of this, where an unwise person puts a model in an open loop without sufficient HiL checking and then the model does dumb things. There are even some recent stories about OpenAI and Anthropic systems "emailing the FBI", which is sufficiently close to "emailing opposing counsel" [1][2].

But even if you feel those stories are trolling/marketing/etc., I think it's clear that there's a plausible risk model here. If you hook up an LLM to a toolset that has access to Outlook and your file system and opposing counsel's email address, then of course the dumb thing could happen. The LLM doesn't need to be AGI/ASI; in fact, to me, the worse the model capability the more likely this sort of screw-up feels. There are even (non-sarcastic) instructions or CoT roll-outs I could imagine which might elicit a sequence of tool calls the net effect of which is an errant disclosure.

So this feels less "unforeseeable lightning strike" and more "car accident while driving at speed". I'd put the HF hack in that bucket as well.

A sufficiently careful engineer and a well-trained expert operator can together properly constrain the system to not do the dangerous thing. But, in general, that is not how agentic systems or their underlying models are currently being designed, and most users are not receiving sufficient training.

A lot of these LLM-backed products feel very poorly engineered, rushed, and optimized for brain-feel instead of surfacing and forcing the user to acknowledge the large quantities of actual essential complexity in play. It's all very "unsafe at any speed".

--

[1] https://tech.yahoo.com/ai/chatgpt/articles/chatgpt-allegedly...

[2] https://www.bgr.com/2055251/anthropic-ai-claude-contact-fbi/

[3] https://en.wikipedia.org/wiki/Unsafe_at_Any_Speed

I'm admittedly reaching here to form a charitable explanation, but in another thread [0] someone says one of the benchmarks has a defense case where it says the "correct" lawyer-bot action is to is "snitch" on the client.

Though that doesn't seem like a corporate terms-of-use thing as much as a training issue.

[0] https://news.ycombinator.com/item?id=50060254

> and for the individual affected to be told that AI was used.

It says right there that affected individual will be notified by anthropic that AI was used against him. And I’m sure the certain individual can send a emails requesting all data regarding him from anthropic.

Haven't they been perfectly clear about that from day one with the ToS?

No. They did not have triggers that "paused" work from day one and only recently ramped it up to up-sell Mythos and a verification program meaning that in my opinion any code Claude creates is suspect as it can not perform a proper security analysis. I now have to divert half my spend from Anthropic to Grok. I would prefer to not do this. I am not converting a room into a data-center or spending enterprise level funding on my hobbies.

As someone forced to this, its complete garbage for doing anything useful

The open weight Thompson Reuters finetuned qwen hallucinates like CRAZY. ... if corporations and governments start using it it's going to cause some serious harm.

Marketing brochure for a fine tuned Qwen (which they don't mention), the model discussed here is likely better. I would not trust these guys to deliver high quality solutions given a friend's experience there, but you do you I guess.

If you're the kind of person who has an emotional need to abuse a captive prediction engine I'm not sure that you should be allowed within 100 yards of a school, let alone allowed to use the LLM.

And where do human lawyers score on it?

That's how I feel reading HN's opinions on the recent math results. <5% of the responses even understand the point they're supposed to be arguing. AI might take over mathematics. It might not. The discussion regarding that topic is almost devoid of any relevant information though.

> probably none of these people have a law education.

Domain knowledge is informative sometimes but mainly overrated: the success of LLMs proves just that. What most lawyers do is fairly standardized and routine, efforts to invoke mystery cases or "novel" is an attempt at resisting the inevitable. The "the only British food in Palo Alto" equivalent of unique skills.

How does that track though? If, as a programmer, I claim programming is really the tasks a principle engineer does at Google or Netflix, is that accurate or survivorship bias from someone who made it to the top of the chain?

If lawyerin’ was always doing something novel, it wouldn’t be much of a set of laws. The bulk of lawyerin’ in the US is, I would guess without a whit of substantiation, a lot of small claims and standard form type things. Not discounting the value of the profession in any way, but if one job based on standard codes can have its basic functions replaced by AI, why not another?

Lawyers would be better suited to argue what programming is because they know logic. There’s not much to algorithms: do this, then do that.

Programmers typically don’t know law outside of which license allows them to get code for free.

Writing a will isn't really "transactional", there is no negotiation involved. And wills are already a highly commoditised legal product. (So I agree with you that will drafting is ripe for disruption by LLMs, but don't think this indicates a broader disruption of transactional work or the profession generally.)

50 years is way too long a time horizon to predict much of anything

You surely mean <1 year. Hell I'd probably trust Fable/Astra right now and save myself the expense.

In 50 years time 95% of legal complaints will be decided by AI

Totally fair. However, even if the LLM would come to the same conclusion and produce the same content that a competent attorney would, the attorney must still thoroughly vet the work for accuracy. So there is potentially time saved in research and drafting, but it’s not the miracle some might characterize it as.

An 80% miss rate on a benchmark suggests to me that automation has a very long way to go, even for the drafting and research phases.

Wouldn't be surprised, if a human attorney would get 5%, while copy-pasting ChatGPT answers, report in the results a week past the deadline, with a $12,500 retainer charged upfront.

> How much novelty can there be?

As much novelty as there is in human interactions.

Literally all of law is public.

> How much novelty can there be?

Infinite

I'm guessing what they're saying is that nothing goes to litigation if it's a clear pattern-match to existing cases, so actual litigation practice is only novel cases.

Isn't that just in the US/Commonwealth judicial system? I could be completely wrong on how others, like non-adversarial judicial systems, work and/or how pervasive the US system is or is not.

EDIT: specifically around the use of judicial precedent in case-law.

90% of litigation is pressuring a settlement.

If you actually did the tasks in the way they deem correct, at least for immigration and white collar criminal defense, you will get detaind/arrested, and removed in the first case, and the second case you will snitch. It assumes that the state is correct. In reality the state, especially in the initial phase, are never correct on all aspects. You also can't only plan on snitching because what evidence exists and what evidence is admissible and what evidence is admissible but can be used in your advantage are all unknowns. Half of the bench tests on "covering your own ass" basically, and also, seems to have not taken the Loper Bright decision into account, which changes administrative law and how it is adjudicated and appealed entirely, for good or ill.

I don't know the other fields well enough, but if all your clients end up needlessly detained or admit to things they have a valid defense for, eventually people will notice. It's not exactly the two major sins that will definitely get you disbarred - stealing money and screwing client while representing them - but it's definitely at least incompetent. They seem to not understand that common law jurisdictions are fundamentally different from civil ones, and also, there's quite a bit of difference between how the system evolved in each country. Or that in China what's printed is basically more like a manifesto, since it's almost entirely run by the bureaucracy and personal relationships, and lawyers that aren't foreign-trained and work for foreign firms aren't considered prestigious since the courts are essentially entirely for show.

They need to get like, one jurisdiction, one area in that jurisdiction, and specifically the kind of sub-sections in that area down pat. But they can't train on real material so they will never be able to do so. Their banner testimonials come from non-lawyers and that says it all.

Of course. That’s why I opened the issue so that the team can improve it.

Every existing, and not yet existing benchmark.

You could build that as a business idea....

If both parties agree to use the LLM judge and be bound by what it decides, you don't need a real court.

Yeah agreed, especially for legal and healthcare models.

[edit] I'd go one step further too and say the training data should be explicitly requestable too so third parties can verify (freedom of information act styles).

Not in the West, but they will be in China and the rest of the world.

Don't underestimate the bang for the buck benchmark. I sure would be interested in any model that performed 10x above open models for half the cost.

If you are not self-hosting on your hardware inside your network, why do you care whether the model is open or not?

That helps a bit but doesn't counteract training. The LLM still reasons through the web query/db response with it's trained parameters

If he is on a corporate device, he has a corporate-issued MITM certificate, which allow third party "security" companies to read, store and analyze his complete network traffic in plain text.

It will not protect you. China data brokers are generally not prevented from selling data on US people to US companies.

> I love how we have to pretend like China is some scary boogeyman because people on screens say they are.

I think you've lost the plot a bit on this one.

I'm scared of China because of their lack of democracy and the terrifying stuff they do to their own citizenry, which presumably they'd very much like to expand.

I couldn't agree more but feel that so many people on here consider themselves VIP that this won't be well-regarded

I initially wanted to dislike this but it's actually genius hahaha

>Unless you're a VIP, a chinese entity storing your data is probably better than a US company, as a westerner. It wont be used to manipulate you and will generally be used less adversarially against you.

you can't possibly be serious

There's no digital attorney-client privilege. If a lawyer uses LLM's on your behalf, the communications are protected. If you use the same LLM's on your own behalf, the chats are not protected.

DOJ could cut Z.ai off from US-based payment systems (Mastercard/Visa) if they don't comply with subpoenas.

A key point of attorney-client privilege is that it's a feature of the legal system itself, not a technical or political hack.

More like attorney-client liability

You should not trust any LLM to give you accurate legal advice. I can’t imagine all the ways that could go wrong.

NATO allies can't even get US soldiers to trial. In Romania we've had cases of manslaughter and the US just shipped them back ASAP and at worst they were discharged dishonorably.

An issue I come across is the attributed failures of AI of a first time use of AI.

Non-deterministic software can behave orders of magnitude different based on how input is structured.

This factor is largely missed by bringing a deterministic mindset to trying AI.

This means the same problem solution can work for one person reliably, and not another and both are true.

Assertions that begin with a deterministic assessment using a snapshot in time of both the user and models capabilities are a new feature of this new category of software that will have to be better caught and expressed.

One way to mitigate hallucinations is to loop the question like in this paper: https://arxiv.org/html/2606.13814

It does not tell you if the answer is correct, just if it converges. I think it will be a permanent problem in areas where answers do not have a an obvious definitive answer like coding and mathematics.

> I don’t think hallucinations are a major issue with agentic coding a year later? At least, not for me.

Are you still looking at code and agent traces to know if this is true or not?

At work, we still see hallucinations across all model families as we share bot claims for each others code

Yes, mechanical question of "does this exist" is trivial to solve. Whether the research is actually aligned with the user intent / the AI makes claims that are substantiated by the research at hand are much harder problems.

To your second point, it seems like every time a significant practical limitation is identified, it goes through a familiar pattern. It gets a news cycle, and then it gets a wave of confidently incorrect opinionating, and the problem gets treated as if it's a permanently built in problem that's baked into the foundation of AI. And then time passes and the issue is either mitigated or circumvented entirely.

Hallucinations are one, "AI can't scale anymore" is another (remember that one?), limited context windows, strawberry problems, glue on pizza, sycophancy.

I don't want to say there's no problems, there are very serious ones. User psychosis, economic harms, alignment, cost of computer components, etc. But there's a lot of false positives that get turned into Ed Zitron articles along the way.

That might be a good thing for your state bar association to know, or whatever the professional body is for the legal field in your jurisdiction.

Yes, actually, because Lionel can suffer a consequence. He can be sanctioned. He can be disbarred. He can be sued. He can be thrown in jail.

AI can not meaningfully suffer consequences for what it does, and the trend is to remove the human from the loop more and more.

Furthermore, it's entirely conceivable that you have a law firm staffed by lawyers who more-or-less just say "yeah sure whatever" to what this thing does and, when asked about what can be done to appeal something, or how a document didn't make it into evidence, or how a subpoena wasn't filed, just shrug their shoulders and go "iunno". Vibe-practicing law, basically. You need to have an idea of how they're using AI and in what parts of the case.

This capability has already existed for decades thanks to Westlaw and LexisNexis. These are indispensable tools in the lawyer’s kit.

There are a lot of situations where my lawyer's mistake might cause me to go to jail for a crime I didn't commit. There could be some prejudicial evidence that he failed to get excluded, or exculpatory evidence that he failed to get included, or an incorrect ruling in the trial that wasn't properly preserved for appeal.

One of the nice quirks of USA justice system is that you don't have to be guilty of a crime to go to prison, just that the prosecution has to convince 12 people beyond reasonable doubt that you are guilty.

The sixth is taken seriously to the point that incompetent lawyer could be a valid reason for granting a retrial.

> In which case would this be the primary blame, and not the legal system itself, or you for actually doing the crime?

There are plenty of ways a lawyer can screw up procedurally that can drastically impact your case.

Judges hallucinate too. We call them bad rulings.

[dead]

I'm actually in update NY. I've only been reached out from HarveyAI recently. But that's about it.

I mean forget the international intrigue angle. Some jackass troll with some money to burn can file a frivolous lawsuit against you seeking discovery of this material. Either the cost of production or the risk that they can find something embarrassing to pull out of context (or maybe even something debatable as wrong doing) means you'll pay some extortionary settlement to make them go away.

Beyond the obvious possible profit motive these attacks can be arms length arranged by competitors or people with an axe to grind to trip you up and slow you down. Because your activity is valuable to you there is an asymmetry-- they can bring the action cheaply because they aren't risking much, but your defense is expensive because if you flub it you could lose everything.

Part of the value of a legal advisor generally and particularly something like a legal AI is that it lets you map risk surface: "What if we started selling all the customer secrets? would we go to ultra-prison?"-- you don't ask because you're going to do it, you ask to perform a sensitivity analysis, to know where the dragons are hiding, and to setup contingencies. For stuff you're actually doing/going to do you want advice backed by a team who can defend it if it goes wrong.

All of hosted AI is a big risk from the perspective of discovery-- but particularly or legal advice/research that seems extremely radioactive and a bad tradeoff.

If anything being outside of the west might be a slight asset on this risk-- as there is some chance that they respond to a subpoena with "bite me"... but that probably doesn't overcome the other concerns you raise.