I checked out the development benchmarks for agents, and was suprized to find that they're not really representative of many of my own day-to-day development workflows.

It appears that they're mostly testing the ability to make business tasks autonomous, with ~20% associated to development tasks (ssh here, install this, etc.), but not actual programming.

Seriously, I can't stand these tests of 1000 input tokens generating 1 million output tokens. You can't really convey anything interesting in 100 words, so like almost who cares what it produces? No one is providing value if the extent of their contribution is so small.

Much more interesting is providing a million tokens of meaningful input and getting 1000 tokens out (high level critique of a detailed design doc, finding a subtle bug in a big codebase, etc).

As someone who's done web development for 20+ years, I find the model personalities pretty dang fascinating, especially how they develop (and evolve) design sensibilities.

I'm interested to know how the classic AI "purple preference" emerged (organically?) and if the beige wave came out of specific training efforts to combat it?

To your point on development work (the code itself), I was talking to some friends on the Google Chrome team about any research understanding the model's preferences around framework ergonomics and abilities to properly implement core web standards for given tasks. I think that would be super fascinating.

I'm repeatedly noticing that people working at big ai and tech companies are surprisingly not that... good... at using ai? It's like theyre doing a plausible thing to get something done and calling it a day

The eval world is split into:

1. Long form task based examinations like this that test the ability of the model+harness to remain on task, tool calling, overall effectiveness and taste.

2. More direct 1:1 and qualitative comparisons that you might get with a tool like https://evvl.ai/ - which also uses OpenRouter and does similar one off model comparisons (or lets you use it as a MCP from your dev env to be like: "take the prompt from this loop and try it against these other models")

I've been using code review as my benchmark. Launched a massive parallel Fable/max code review on my lone lisp codebase and recorded all the results and transcripts. Switched to OpenAI and am now running the exact same code review with Sol/max.

It's still a work in progress but preliminary results reveal that Sol is able to reproduce 70%-90% of Fable's performance. This is a very meaningful result for me because code review is what I use AI for.

You touch a point I quickly skimmed in another comment.

Yes, the most valuable benchmarks and evaluations you can write are those that resemble your work.

The evaluations are extremely hard to write and test.

And yes, virtually all benchmarks are E2E one shots, they do not reflect multi turn processes or how most people interact with LLMs.

Which is why every Opus after 4.6 looks better on benchmarks, but is hard to work with interactively.

> Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work?

As a non-designer trying to build something solo, I think value's in the higher-tier models being able to brainstorm and infer a variety of design directions, as well as dissect the "this looks off" comments that would frustrate human designers.

Some of the frontend design skills try to bridge the gap, but the better models perform far better as is, and often better without one of those frontend design skills trying to ram their own workflows in.

Yes, I would be very interested to see follow up prompts that adjusted visual elements as if feedback from the client, then some prompts to change the menu and pricing. Id like to see how well the AI's can maintain their own work.

I've been super impressed with one shot AI images and designs in the past, but have never been able to adjust a design without things going off the rails.

> Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work?

No, you are not wrong. If you now crack the "What is?" in a generalizable way, there is very good money in that.

Without more knowledge about the technical aspect, it could be a good thing that they're all so similar. If you tell humans to go from their kitchen to their bedroom, they all stand up and walk the same way. Nobody decides to crab walk. Maybe you'd have to cripple the model in some way to do that.

On the other hand, if you want something different with LLMs, all it takes is a few more words of creative flair in the prompt.

I trialed Claude for a sorely needed redesign on my website. I was quite impressed with the results, but decided to hang fire on deploying it. Cue my surprise when I just found the same design applied to a coffee shop!

The similarities are beyond coincidence, to the point I'll be scrapping Claude's version of the redesign.

I think this is exactly what we should expect. LLMs are engineered to produce the most likely output given the training data. When you give it a generic design task, you're going to get a distillation of design cliches and tropes from the past decade.

It’s always been interesting to see how similar output is across ostensibly very different models. I remember testing short story writing in the early days and having the models all choose the same niche topic across e.g GPT, Llama, Phi, Claude, etc

Reminds me of the Bootstrap era. I can't count the amount of websites with a centered header navigation, slightly rounded accent color buttons, hero section that was mostly text, and some "fun quirk" in the background, either geometric shapes or squiggles or something.

I think websites have always looked mostly alike. It's sorta always been a thing. Reminds me of "Corporate Memphis" (https://en.wikipedia.org/wiki/Corporate_Memphis)

It's alright though, because some people are OK with middle of the road (Wordpress, Boostrap, Squarespace templates, now AI). And others are willing to either pay a developer to get involved or put in the extra effort to differentiate themselves.

A user, even without much design background, could ask the AI in a follow up prompt to change a few key parameters like overall color and differentiate it more. Even taking a sample image from a design magazine or blog and saying "incorporate some of the design elements here" would probably work great to diverge it.

I found it pretty interesting. To me, the GPT Sol and Luna ones are the nicest, which is not what I was expecting. Font choice goes a long way. The Opus ones are acceptable, but still disappointing. The rest are hideous.

Don't blame LLMs if the zeitgeist for coffee shop websites is sepia tones and a Papyrus-like font.

My university's Principles of WebDev course 10 years ago had a similar assignment and the results all ended up looking like that too.

I was thinking the same thing, but I suspect there was something more to it. Check out this sentence from the page.

>Our default skills also include some UI design guidance, mainly to avoid known gotchas (e.g., the now-dreaded purple AI slop) and get the model to reason about the visual identity appropriate for the user’s ask. But beyond that, each model is free to go build what it thinks we’ll want.

It's also kind of pointless. Why does a coffee shop need a website anyway? Nobody is saying, "man I would love to go to this coffee shop but I can't find their website".

If all you want is "opening hours, the address, a short menu and a photo" there are easier and cheaper ways to do that.

Yeah the difference in token usage across difference models they found in the article was broader than I expected, but I was sort of floored by the variance in token usage for the same prompt (run multiple times) with the same model.

Have you published any of your results? I am quite curious.

The obvious limitation to this approach is that you're limited to doing evals for things where the SOTA models are capable of judging an attempt. Many tasks are not yet amenable to this, especially if you don't have a ground truth to have the SOTA model compare to.

And even for the tasks that are amenable to having LLM judges assess them, there's still a huge benefit in looking at traces yourself and labeling them. It's time-consuming, yes, but you'll learn a lot about the ways an agent fails in your particular domain, it gives you more reliable golden datasets, you have a mechanism to evaluate your judges, etc.

I haven't done this for coding yet (honestly can't really figure out the best approach), but for in-app evals, I built a whole interface to review traces. My app sends up 10k - 30k input tokens on the initial request, between fixed prompt, schema, and dynamic context, so just reviewing it is a huge pain. My interface converts all that raw input data to human-readable objects, highlighting important things and tying together inputs vs outputs for the request. Makes reviewing it, grading it, tagging it, etc. much easier. And then once I've done that for a batch of requests, the passing requests have their inputs and outputs frozen as a golden dataset that I can run other models against. And there the LLM judges are great, because they're very adept at looking at an input prompt, a given model response, and a (human-approved) ground truth, and pointing out any significant differences between the output and the ground truth. Much easier than actually producing the ground truth.

I highly recommend Hamel Husain's writings on evals.

> Building ad-hoc evals is trivial these days.

I highly doubt that you have solved it. Writing proper evals and benchmarks for real-world scenarios is far from trivial.

Benchmarking an agent essentially means freezing, at the very minimum:

- the model

- the model's configuration (e.g. effort, permissions, provider)

- the dataset (e.g. a git repository at a specific sha)

- the code running the agent itself (you can build your own harness, trivial, but you still need to ship it as a single executable, froze in time. benchmarking against closed source runtime like claude code is quite useless, they change too frequently and in ways you cannot directly inspect).

- the tools at agent's disposal. Even a slightly different implementation of tool X (e.g. grep or readfile or sed) has an impact. In general this implies also freezing a very specific container image. In my personal benchmarks I provide a specific list of tools that come with the executable, there's no possibility of interacting with the outside world besides the provided apis, the agent bundles its own tools.

And even then: there's significant noise coming from the LLM providers themselves which noticeably change the models behaviour, I don't know whether this is because they optimize some settings or change the inference over time, etc.

And, last but not least, the output of LLMs is non deterministic.

Also, the LLM as judge presents essentially the same non-deterministic problems, has to be benchmarked itself thoroughly, and writing quality rubrics or "golden answers/outputs" is just difficult. One of the metrics I consistently try to emphasize is to avoid the "shotgun vomit dump" of information. So answers that get right to the point in plain terms avoiding dumps of information filled of jargon on top of the user are rated differently.

In short: its far from trivial to benchmark models on real-world agentic work taken from your personal or professional projects.

And even creating the test cases themselves is hard. No: you cannot take the output of some "sota" and use it as gold standard. This is a very crap approach. It's the sloppiest solution to the problem, in the very sense of slop: plausible, average, lacking any creativity or out of the box thinking, the things that make the real difference in complex software development.

The very point of creating these benchmarks is to find which configuration/model/tools/harness (skills/mcps/documentation/agents.md, etc) works better.

And it only works if you create these benchmarks yourself from genuinely difficult non-trivial work and find a solution that is better by most metrics implementation-wise, albeit you could settle on the implementation solving a series of cases and edge cases.

But if this effect were really that strong, the models should be getting nearly 100% on the common benchmarks. But for many of the benchmarks, even after being public for more than a year, the new models only get 60-80%.