jetrink
2 hours ago
> Tokens become cheaper than tool calls
The author observes that a call to GPT-5.6 Luna is only 4-5 orders of magnitude more expensive than grep, and then predicts that at current rates of progress, calling an LLM will soon be cheaper than a grep. I think this is a good time to invoke Stein's Law: "If something cannot go on forever, it will stop." These efficiency improvements won't continue forever. It's more likely that the per-call cost of high-quality, compiled software like grep will be a lower-bound that LLMs asymptotically approach, rather than a line that they blow past with perpetual exponential progress. (Barring a true breakthrough in something like quantum computing or room-temperature superconductors.)
sanderjd
40 minutes ago
Yeah I bumped on that too. If it's possible to make llms cheaper than current grep, then it is also almost certainly possible to make grep cheaper.
serbuvlad
23 minutes ago
You can burn anything* into an ASIC to make it cheaper per-call.
non-backreferencing grep is not very difficult to implement in an ASIC either. But it's probably not worth it because of how relatively rarely you use it and of the data transfer costs.
LLMs are great candidates for ASIC-burning because they're slow compared even to network speeds and run all the time. The issue is that you don't want to burn a specific model or architecture that then becomes obsolete.
So you've got two possible futures, and both guarantee large price drops: (a) LLMs keep getting better and better and better, so ability/$ keeps rising; or (b) LLMs plateau in ability, in which they will start getting ASIC'd.
scotty79
12 minutes ago
depends on what you are grepping ... greapping a large file might be more expensive one day than generating n-th token with LLM that works fully in hardware
you could make hardware implementation of grep and store the file itself next to it in some ROM but that's not a very useful grep ... while hardware LLM is exactly as useful as software LLM only orders of magnitude faster
FranOntanaya
an hour ago
LLM is spicy memoizing, so it can potentially be faster than a tool call. But people will spend a month tweaking and testing to ensure they have the level of determinism they need, which means it's more expensive, and that they should have used actual memoization in the first place.
gregdeon
an hour ago
Right. And some hardware improvements will speed up both grep and Luna, which won't close the gap.
DoctorOetker
an hour ago
not necessarily, one may be easier to parallelize while the other suffers some serial computation bottleneck.
gunalx
an hour ago
Grep is trivially parallelizable if you care to do it though.
https://iepathos.github.io/ripgrep/performance/#work-stealin...
arijun
an hour ago
It might never beat out grep, but it could beat some more expensive to call tools, similar to how heuristics will often be faster than exact answers. Rust Analyzer can be slow at times, I could see an AI tool taking over a subset of its work.
14u2c
23 minutes ago
> calling an LLM will soon be cheaper than a grep
From a computational standpoint this is obviously nonsense, but from an attentional one I'm not so sure. It may already be more attentionally expensive to use grep in some cases, such the moment you need to remember a non standard arg. And if this applies for performing a simple http operations, then it certainly applies going up the complexity chain.