> LLMs don't understand. It's mind-boggling to me that large parts of the tech industry think that.

I think you might be tied to a definition of "understanding" that doesn't really apply.

If you prompt a LLM with ambiguous instructions, it requests you to clarify (i.e., extend prompt to provide more context) and once you do the LLM outputs something that exactly meets the goals of the initial prompt, does it count as understanding?

If it walks like a duck and quacks like a duck, it's a duck,or something so close to a duck that we'd be better off calling it that.

I don't believe the user meant "understand" in the classical biological and philosophical sense, or were otherwise attempting to anthropomorphize the systems. They were speaking from the practical experience of "this thing takes a somewhat ambiguous input with unique constraints and implements the ask more-or-less as intended".

They understand. Anything able to reason about any arbitrary request and form a plan tailored to that request understands well enough to qualify for the verb. The mechanism behind it may feel hollow or fake. But if its responses reliably show understanding, the LLM understands - by any ordinary measure.

Nearly every argument like this has the same fatal flaw, and it's generally not the critique of the AI, but the critique reflected back on to humans.

Humans also don't understand and are frequently faking understanding, which for many tasks is good enough. There are fundamental limits to what humans can do.

The AI of a few months ago before OpenAI's sycophancy was quite impressive, less so now which means it is being artificially stunted so more can be charged later. It means privately it is much better than what is public. I can't say it "understands," but I can say it outclasses many many humans. There are already numbers of tasks based around understanding where I would already choose an LLM over a human.

It's worth looking at bloom's taxonomy (https://en.wikipedia.org/wiki/Bloom%27s_taxonomy): In the 2001 revised edition of Bloom's taxonomy, the levels were renamed and reordered: Remember, Understand, Apply, Analyze, Evaluate, and Create. In my opinion it is at least human competitive for everything but create.

I used to be very bearish on AI, but if you haven't had a "wow" moment when using one, then I don't think you've tried to explore what it can do or tested it's limits with your own special expertise/domain knowledge, or if you have then I'm not sure we're using the same LLMs. Then compare that experience to normal people, not your peer groups. Compare an LLM to people into astrology, crystal healing, or homeopathy and ask which has more "understanding."

meh. I feel this is just a linguistic shortcut, similar to how _trained_ biologists can talk about a species or organism evolving some trait. Of course the organism isn't _really_ evolving with any goal in mind, but that's clear to the speaker and audience. Whether or not LLMs understand (very unlikely), it's clear what we mean by an LLM "understanding": has the context + prior training to make reasonable predictions. But no one wants to write that each time.

They understand tho, it's different than how it's done in our brain but they solve task that would be impossible to do without understanding. I would even say that they can now reason through problems thanks to powerful reasoning models like Gemini 2.5 Pro and o3.

How do you know?

Definition of understanding is based on connecting relations. If there is one thing a llm can do its connecting relations. So I am not sure why you say llms are not understanding.

What is the limit my system will reach?

Thats an interesting word to pick on. Understanding still means something here in a relative sense.

Asking a short question but in a serious way: so what?

I would be annoyed along with you if I thought the post was true.

No employees accomplish anything on their own in the real world, all employees are part of a team. That's why I designed a business strategy and analysis layer (over half the system, in fact), with web tools and connections to all of the insights systems (like mix panel). I built the exact same thing I build at digitalocean but instead of humans I defined them with code, digitalocean runs just fine, so does my LLM system. The whole system I build is self learning, insight gathering and refinement. Competition is for losers, the best teams win via the best insights.

The only investor is me. I build it on my own over a weekend, on my own. I just wanted to confirm it can be done therefore will exist, that is all. Personally, I decided not to peruse it because I am old and lazy and don't want to compete against a16z and sequoia funded adderall filled teenagers.

Thanks!!! I decided not to build it, that space is already too busy, there is a startup with $25MM in stealth, who else is in stealth? On top of that, this method will get stale very very quickly, foundation model businesses are just too hard to work around right now, it's a silly way to do business. My magic is I've build a startup from scratch to over 400 people and watched what they do, it won't be long till that isn't worth much.

Goods and services are a byproduct of business, business is primarily concerned with systems and processes that facilitate value exchange, so my tool, can work with a user, to build a business, not a product or a service. If you bake cupcakes, my tool can get you 100 people at your door, it cannot open the door or provide the cakes.

I would love to know how to do it another way if you have any ideas, I'm sadly not experienced or intelligent enough to think of another way to do it.

Humans modeled time, and morning and night are real.

> I designed something for a business and found I needed 4 major sub-systems (like a real business) - insight/data, cognition, meta cognition and execution, and if you don't define all 4, the system is junk.

Might it be just another realization of Conway's law?

https://en.wikipedia.org/wiki/Conway%27s_law

Might it be possible that the only reason you're assuming a system is junk is just that it doesn't resemble the systems you know and expect? There are so many ways to skin a cat, and certainly no business process represents the optimal process.

For those who don't want to read the article: The previous best was 49 scalar multiplications.

I was not referring to algorithm designing, but to algorithm implementation on a given processor, which consists in things like register allocation, machine instruction selection and static instruction scheduling.

If you cannot predict the running time of an algorithm without running it on the target processor, then by definition the documentation of that processor is incomplete.

For a processor that is completely documented one must be able to run a simulation model that provides the running time for a given program, when the processor is so complex that simpler methods for computing the execution time do not work.

For older NVIDIA GPUs, there exist such simulation models, but they are only partially accurate, because they are based on reverse engineering, without cooperation from the GPU vendor.

While you are right that in general it may be very difficult or quasi-impossible to generate optimal programs, there are also plenty of cases where an optimal program is achievable.

The latter happens when there is one dominant bottleneck for the algorithm, which is determined by the hardware, e.g. the maximum throughput of a certain instruction, like multiplication or memory load. When the implemented program reaches a throughput almost equal with that absolute limit, then one can be certain of its optimality.

>The implication was that the FP32 versions of these kernels have lagged behind the more popular versions.

Help me understand this 'cause I'm a bit slow these days ...

Does that mean optimized FP32 versions of these kernels were already there or not?

I was blown away by some debugging results I got from o3 early on and have been using it heavily since. The early results that caught my attention were from a couple cases where it tracked down some problematic cause through several indirect layers of effects in a way where you'd typically be tediously tracing step-by-step through a debugger. I think whatever's behind this capability has some overlap with really solid work it'll do in abstract system design, particularly in having it think through distant implications of design choices.

In the context of LLMs, what do you mean by "reason"? What does reasoning look like in LLMs and how do you recognize it, and more importantly, how do you invoke it? I haven't had much success in getting LLMs to solve, well, basically any problem that involves logic.

Chain of thought at least introduces some skepticism, but that's not exactly reasoning. It makes me wonder what people refer to when they say "reason".

Very likely. Larger context is significantly beneficial to the LLMs when they can maintain attention, which was part of my point. Imagine being able to hold the word for word text of your required reading book while you are taking a test, while older models were more like a couple chapters worth of text. Two years ago.

SRE / DevOps / coding mostly in the Azure and .NET ecosystems.

The problems I have to solve tend to be the horrible ones that nobody has answers to, anywhere on the Internet, so unsurprisingly the AIs aren't good at it either.

The trick has been to use the AIs for what they are good that, which used to be "nothing" for me at least, but now I can use them productively for certain "spot" tasks.

Random examples:

- Cross-language and cross-platform benchmarking of a bunch of different database clients to see how they stack up. I gave the AI a working example in one language and got it to whip up a series of equivalents with other DB drivers and languages. Sure, it's trivial, but it's way faster than doing it myself!

- Crash dump analysis using WinDbg. I read somwhere that "vibe debugging" of kernel dumps totally works, so when I had an actual crash I gave it a go for laughs. With AI help I managed to extract the name of the specific file that had NTFS corruption and was crashing the server. Deleted the file, restored it from backups, and the server was good to go again!

- If you ever watch the top mechanical engineers on YouTube, they all make their own tools instead of just buying them. Jigs, extenders, unusual sizes, etc... IT work is the same. As a recent example, I got Gemini to make me a code-AST rewriter for a specific issue I wanted to clean up in bulk across a huge code base. Using the Roslyn compiler SDK is a bit fiddly, but it spat out a working tool for me in under an hour. (This is not something you can solve with a script full of regex, it needed a proper parser to handle commented-out blocks and the like.)

Theres zero days in obscure parts of the kernel nobody uses every other day. (It also of course found 100 other things that were not zero days or vulnerabilities, yet professed they were, which is why this trash even on Gemini 9000 Pro keeps spamming security mails)

That's an Ampere chip IIRC so it should have tensor cores

My original question is to understand why it is considered as huge tolerance and what should be considered low tolerance. I am suspecting the paper’s intention is not to compare apples and oranges. They are trying to optimize fp32 baseline by sometime resorting using fp16 as long as the resultant solution’s numerical accuracy is within thr tolerance level. They are going for the “low hanging fruits” type of optimization.

That's an entirely different idea that may or may not work. This is not evidence of that.

Whoops you just did a communism

Generally tiling to a handful of sizes gets you pretty close to "specialized to this size" performance.