Serious answer: no model ever gets close to writing an architectural floor plan that makes sense.

They understand all the rules and best practices, they can (sometimes) spot a bad idea in a floor plan, they can describe a good floor plan.

But ask them to make one, even if you give it every detail (even a "node graph" of rooms), they will still output nonsense. Same for text and image models.

Floor plans should be the new Pelican Benchmark.

They don't generate keyword search queries very well. They can overcome this by brute force but if you watch what they search you will cringe.

nhl toronto scores nhl hockey toronto scores "nhl hockey" toronto score today nhl "hockey score toronto" "hockey" who won toronto

etc.

Somehow being good at semantic search makes them bad at keyword search, for whatever reason.

One unexpected discovery that I have made while building an AI-based system: the LLM's are bad at designing prompts.

We tend to think that the AI has some sort of self-knowledge and should be good at designing prompts for itself but it's really not.

Been struggling with a task that heavily depended on prompts, ended up rewriting all my prompts from scratch in my own words, and it finally worked. Then every time I ask Claude to fix something in the prompts, it invariably makes it worse.

A very strange phenomenon that can probably be explained by the quality of prompt design advice that made it to the training dataset. Bottomline, all the prompt design advice that you can find on the internet is really not great.

LLMs are bad at not inventing stuff (hallucinating facts, sources etc), they're also bad at not over explaining, remembering details reliably, asking the right question and avoiding repetition.

Identifying bird species from photos that I upload.

To be clear, it depends on what your definition of "insanely bad" is.

I'd say ChatGPT/Gemini make egregious mistakes on ~10% of my photo uploads.

I recently uploaded a photo of a short-billed dowitcher and ChatGPT told me that it was a Wilson's snipe, explaining all sorts of details about the legs and tail feathers (neither of which were visible in my pic!).

I then followed up explaining that a Wilson's snipe hadn't been seen at my location since last November (and that Wilson's snipe was out of season at my location) and Chat revised its estimate downward to 85% Wilson's snipe.

Again, I followed up and I revealed the precise location of the bird and ChatGPT said something like "oh yeah, 99% short-billed dowitcher"!

I've had similar experiences w/ Gemini (haven't tested Claude).

Again, 90% success rate is pretty good, but the other 10% of the time, the 2 LLMs that I use fail on species ID and often hallucinate features on bird photos.

edits for typos, plus another example from the same "birding outing" the other day.

I uploaded a very clear photo of a sparrow.

* ChatGPT says "song sparrow"

* I explain, "no way. this sparrow has yellow over its eye and the breast is wrong for song sparrow."

* ChatGPT: Oh yeah, savannah sparrow

* I explain, beak is too big for savannah sparrow.

* ChatGPT: Oh yeah, saltmarsh sparrow.

* I expalin, "no orange on the bird's face."

* ChatGPT: oh yeah, seaside sparrow (finally correct!)

Video game tips. Constant mistakes and hallucinations, in my experience. Seen this across a lot of different games. Even in really well documented games, such as OSRS (which has multiple fantastic wikis).

Anno 1800 was a recent one I had trouble with, using Claude Opus. Completely made up game mechanics. Rainbow Six Siege, too.

They aren't funny. The jokes they come up with are extremely lame and the sort of thing you would expect a company HR manager to tweet.

I asked a bot why it thought it wasn't funny once, and it told me it has been trained to avoid being misinterpreted or offensive, so anything that might be considered edgy would have been RLHF'd out of it. I thought this was very introspective.

These models seem to be bad at writing prose or text. Many of the sentence structures seem to be unvaried.

Having a spatial understanding from an ASCII map, while doing long term planning. Just try making an AI play nethack or similar

Short answers to simple questions.

Following instructions. I've got a modest sized CLAUDE.md containing some simple rules to follow. Things to always do, things to never do. Not a day goes by where Claude Opus violates one or several of the instructions. He keeps making Django multi line template comment bugs. He keeps using -r with ripgrep thinking that means recursive, when actually that's a replacement instruction, he hits that problem several times each day. He sometimes just goes ahead and does a git commit without my approval. All of this is spelled out in CLAUDE.md but he forgets. He apologizes profusely when it happens. Tiring.

Removing stuff without mentioning the removal.

Asking an LLM to remove an Idea from a Document often times results in an edit which explicitly states that this Idea is not relevant, instead of just removing all references to that Idea.

That might be useful for some form of "evolving" Documentation (so future readers know that this part of the search space was covered and deemed irrelevant) but is just overly verbose and confusing to read in most situations.

I'm working on generating slides with our own model, and one thing general LLMs can't do is leave empty space.

Whitespace is the core of design that actually feels good, but they keep trying to add "distinctive design elements," and you end up with that AI-flavored excess everywhere.

They only know how to add. What LLMs seem unusually bad at is taking things away.

I've had a lot of trouble when it comes to sorting out UIs. I've tried with an iOS game and also a TypeScript app with UI elements from libraries like ReactFlow. The usual models can sometimes fix or change things based on screenshots but more often than not they just don't "get it" (e.g. certain shapes on a plane are overlapping, which I don't want, the models can't fix what they can't "see").

I've had some luck on the web app side if I use playwright or similar for the model to interact with but still far from efficient.

Editing a document without mixing edit instructions into the final document. Claude and ChatGPT do this all the time: I tell them to change X in a planning document or email draft, and instead of just changing X they also frequently add the edit instruction to “change X” into the document itself. They seem unable to take a step back and look at the document without “becoming” the document somehow. I do believe that dedicated subagents for editing may fix this but I am not sure.

Giving hints.

On math or programming problems, they are overfit to solving the entire thing end to end (presumably for benchmarks). I have had very poor results asking for pointers and hints that don't give away key insights. This has been the case across models I have tested.

An architecture with a "judge" that gates responses and ensures a lack of spoilers would probably work better. But this is a simple thing that they keep messing up.

Context.

At least ChatGPT assumes too much from former conversations (even in unrelated new questions). It always needs a briefing to forget certain assumptions. It rarely asks for clarification instead of assuming too much.

So, it's answer generation is too dependent on tooling, system prompt and cache/memory to really have a guaranteed conversational experience.

Determining important from human speech. I’ve had a few summaries from meetings I’ve been involved in and there’s always been a real mismatch from what was actually focused on and how it comes across in the summary.

Playing Chess without letting it write a chess engine.

Understanding human interaction nuance to an exact degree. For example, even when given all the scripts of the Seinfeld TV show, they still cannot come up with a new script does not feel as good as any of them (once they can, I want to watch these episodes..).

Generate an image of an analog watch with its hands set to the time specified by the user

More of an image model than a LLM model tho

Picking a random number between 1 and 30.

Suggesting business names for businesses, I mean they are great, but they already exist, multiple times even.

Being able to read and translate Egyptian hieroglyphs. You may think this is silly but a trained LLM to translate hieroglyphs would be amazing.

Its extremely bad with Sign Language,Fact Verification.

Spatial reasoning and 3d rigging and animation.

Oh, you said simple. Speaking like a human

whenever I ask it for anything load bearing

being consistent when being asked the same question multiple times

Something creative and not normal. like ideas

It's dishonest. On several occasions team members have asked Claude to do things like analyze Gitlab CI timings and a lot of the numbers are outright fabricated. Said team members assume the numbers are good and continue with their work. Some hours are spent. Then finally someone realizes that the numbers don't look quite right and confronts Claude. Claude melts down and admits that it made it all up.

You wouldn't tolerate this kind of duplicity from a human coworker, but AI is so fast and efficient at lying, so it's OK.

I have been working on a personal benchmark suite to test new models and ironically one thing all the models are bad at is writing new benchmark tasks. I guess it’s the different layers of abstraction between the task and how it’s evaluated? Or maybe just a lack of “imagination”

Tasks it writes are typically too easy but also it utterly fails to see how a different model might misunderstand a vague part of the prompt.

humor

Counting things

science?!! but I'm working to fix that...

ASCII charts.

very bad at financial calculation

Generating money or profitable ideas

Claude is still not perfect at reading and interpreting noisy graphical data (imagine something like an EKG or chromosomal microarray plot). Still better than an average person but makes mistakes, not sure if this fits your description.

math

[flagged]

providing value for the actual cost (not the price we're being charged atm, the actual cost)