It's trivial to get better score than GPT-4 with 1% of the cost by using my propertiary routing algorithm that routes all requests to Gemini 2.5 Flash. It's called GASP (Gemini Always, Save Pennies)

PPT (price-per-token) is insufficient to compute cost. You will also need to know an average tokens-per-interaction (TPI). They multiply to give you a cost estimate. A .01x PPT is wiped out by 100x TPI.

number of complaints / million tokens?

> How you measure 'performance'

I heard the best way is through valuations

> GPT-4 at $24.7 per million tokens

While technically true why would you want to use it when OpenAI itself provides a bunch of many times cheaper and better models?

(The submitted title was "93% of GPT-4 performance at 1/4 cost: LLM routing with weak bandit feedback")

That's pretty funny. I might need to pilfer it.

> a strong enough notion of question complexity

Aka Wisdom. No, LLMs don't have that. Me neither, I usually have to step in the rabbit holes in order to detect them.

LLMs don't have notions ... they are pattern matchers against a vast database of human text.

This is like asking someone to make you a sandwich and expect them to read your mind to determine what kind of sandwich you want.

LLM is far from the highest AI related cost, so we basically don't care about optimizing LLMs.

Obviously we don't use the super expensive ones like GPT4.5 or so. But we don't really bother with mini models, because GPT4.1 etc.. are cheap enough.

Stuff like speech to text etc.. are still way more expensive, and yes there we do focus on cost optimization. We have no large scale image generation use cases (yet)

In which context? The serious engineering folks are still in exploration phase, costs is mostly not a concern as long as shipping velocity increases. Reselling repackaged tokens is a different beast, no experience here

I almost named my LoRA replacement BEMO, but that felt too cute, so it's just BEM (Bolt-on Expert Modules).

First, I don't think we will ever get to AGI. Not because we won't see huge advances still, but AGI is a moving ambiguous target that we won't get consensus on.

But why does this paper impact your thinking on it? It is about budget and recognizing that different LLMs have different cost structures. It's not really an attempt to improve LLM performance measured absolutely.

LLMs are not on the road to AGI, but there are plenty of dangers associated with them nonetheless.

Is a random paper from Fujitsu Research claiming to be the frontier of anything?

I'm not following this either. You'd think this would be frontier back in 2023

That and LLMs are seemingly plateauing. Earlier this year, it seemed like the big companies were releasing noticeable improvements every other week. People would joke a few weeks is “an eternity” in AI…so what time span are we looking at now?

just because it’s on arxiv doesn’t mean anything

arxiv is essentially a blog under an academic format, popular amongst asian and south asian academic communities

currently you can launder reputation with it, just like “white papers” in the crypto world allowed for capital for some time

this ability will diminish as more people catch on

I'm starting to think that there will not be an 'AGI' moment, we will simply slowly build smarter machines over time until we realize there is 'AGI'. It would be like video calls in the '90s everybody wanted them, now everybody hates them, lmao.