Great point! Yeah latency was one of the biggest issues here. To cope with that (and for safety reasons) the cars are driving at extremely low speeds. They also get timestamps with every tool call output etc so they can, in theory, "in context learn" about their own latency and choose motion durations and control how fast their iteration loop is to some extent. But yeah, this is just sort of a fun benchmark to see how good frontier LLMs are out-of-the-box at driving a real car, and probably not actually practical any time soon.

-Aditya, Tobias, Simon

> Could this work in the real world? Absolutely not. Three reasons: latency, latency, and latency.

That and also the fact that (in spite of their usefulness) LLMs still so often do incredibly dumb shit without thinking of the consequences that the idea of having them drive in public is absurd.

Recently was using claude code/opus 5 to diagnose an intermittent wi-fi connection problem and one of the first things it did was to bring the adapter down. The wi-fi adapter was the only way the system was communicating with the outside world so claude effectively disconnected its own brain as step 1 in figuring out what was going wrong. Things did not progress well from there. Easy enough to clean up its mess in this case, but luckily it wasn't driving a heavy killing machine at the time.

> otherwise you can't react

I'm far from neuroscience, but humans don't need to operate at 20Hz to drive a car. And human reaction latency (event to measurable action) is often over 1s (under 1Hz).

Kind of funny to mention comma today of all days

It is also worth mentioning that the openpilot AI model is a world model. The way a world model understands physical reality and geometry makes it inherently safer for driving than an LLM, which is essentially a text-based statistical machine with no concept of the physical world.

You can see this in the photos, it took over five minutes for the cars to get around the cone course.

Yet remote pilots can fight wars on the other side of the world?

Is that the same comma.ai project also in the news today?

https://arstechnica.com/cars/2026/09/aftermarket-driver-assi...

> Could a cloud-delivered LLM figure out how to drive this route, based on those input data and given access to those output actuators? Looks like yes. Sure.

Well, if the massive cloud models that are generalized and have a world model that's good enough, you can just distill them into smaller models. As a point of reference, the current gen of Tesla FSD models only have 1B params. They are tiny by LLM/VLM standards.

Perhaps there's a synthesis to be had though. Eyes, control, and safety critical features on the hardware, higher level decision making to the cloud. Openpilot's biggest weakness has always been in the very "robotic" way that it drives, which is technically correct but causes frustration for other drivers. Deciding "should I pass this car" is a fundamentally different question to "can I pass this car", or "what is the actual safe speed and following distance given the current traffic conditions and weather".

How are you so sure that latency can't be improved? Sol can run on cerebras and we may get enough efficiencies that Astra can also be run locally.

I think Opus 5.5 is at same level now. I have seen too many videos made by Opus 5.5 today on twitter.

https://x.com/victormustar/status/2102707412704919910 horse galloping pixel art

https://x.com/LexnLin/status/2102133072585965759 moving train pixel art animation

https://x.com/jkeatn/status/2102441348075057539 painting with code

https://x.com/LCSlates/status/2102503027340988559 video, very detailed prompt though

https://x.com/aj_dev_smith/status/2102504509637587339 generated song/music with code

https://x.com/aj_dev_smith/status/2102575577563570450 another song

Well, they have the best in class image generator so that probably has something to do with it

You might be interested to learn that the bitter lesson has already been grok'd by generations of autonomous car company engineers, and many or all have incorporated learned components (at minimum) in all their vehicle stacks.

There's also a very tangible limitation of the bitter lesson.

If, over time, compute climbs, and so compute-bound data-driven general architectures beat bespoke architectures (this is the bitter lesson), then it is not necessarily true that the most general architecture now beats all available bespoke architectures now (or even in the near/mid future - the crossover point is "eventually").

Bitter lesson is most tangible for long-running research directions. Sometimes you need something working as best as possible now.

I'm not sure how you take that from the original article. My 4 year old would drive that course in an automatic car, if only he could reach the pedals. Heck, he's done harder things at Lego land.

I wouldn't let him loose on the road though.

I think, at the very least, the guardrails would have to deterministic, ideally with super human senses, for people to accept self driving cars on the road.

Tesla's already solved this - their vision model does this phenomenally well.

And they've demonstrated adding a sidecar LLM to it as well, mostly for these kinds of "read these 3 street signs, what should i do next?" sort of situations.

If I'm reading the chart correctly, it took over 5 minutes to drive 135m at a cost of nearly $8.00 in tokens. I don't think that's really in the realm of practical yet.

Can you please explain what does it mean by bitter lesson in this context specifically? I keep seeing this term here. I know there is an article of the same title but I still don't understand.

What do you think Tesla has been doing this for so long?

The bitter lesson tells you about the trend in the technology. It does not get product to market with today's technology.

As somebody working near the field, I do enjoy the fun of dreaming bespoke vision and autonomy algorithms (if I didn’t, I wouldn’t work in the field to begin with!). But I would drop it all in a heartbeat for a robot that works well. Robust, resilient robots would be such an incredible advance that the ‘how’ doesn’t matter. All of the nonsense from the current AI hype cycle would be worth it if it cashed out in Robots That Actually Work.

Think this dramatically simplifies the problem. AI existed before GPTs and the AI in self-driving is optimized for self-driving and the latency you already mentioned.

Regardless of how the AI is architected, you aren't going to be able to use a generic LLM like Qwen to perform reliable self-driving, you need a highly optimized, highly specific AI.

> The bitter lesson is finally coming for the self-driving cars.

Maybe, but the opacity level of models is not acceptable for cars. "Why did it drive under the semi?" "Model said to." "Why did the model say to?" "shrug"

>It’s mostly a latency problem at this point. The models are too big to run locally, but given that open-weight models like Qwen already exist, an open-weight, low latency equivalent to Astra can’t be too far out.

But this is a bit of a ridiculous take, no?

You don't need Astra for self-driving. Astra is able to build complex 3D worlds, do your taxes, shop for you, and, apparently, drive a car. A self-driving car just needs to be able to drive a car. By the time you trim down Astra to just have the minimum capabilities needed to drive a car, you'll be looking at the same models these self-driving car companies already use. Then you get to deal with the actual hard problems, like handling failure cases (which will still be present with Astra).

>The vision stack, 3D maps, lane selection grammar, occupancy networks, it’s maybe all about to give way to a single GPT looking at camera feeds and predicting the next steering wheel adjustment.

Self-driving cars have been able to do this for a long time. The problem is that it isn't robust enough given the context. I mean, if Astra can drive a car with a single camera, then presumably Astra can drive the car even better with multiple cameras, and even better than that with 3D maps, etc. And when you start to consider the expectation of performance of these systems, you realize that these features really can't be omitted. If you're a company producing self-driving cars, then you do not want to face a lawsuit for you car killing someone because it physically would have never been able to see what it was doing because it lacked a camera.

I think the real gain here is that something like Astra can be used to help build these autonomous stacks. If it is able to drive itself, then it is able to generate novel data, analyze large quantities of data, and use context that isn't typically available when processing this data to make improvements to the actual autonomy stack which is ultimately responsible for driving the car. But thinking that these car companies are going to run an LLM in a car and call it a day is just naive.

Doesn’t Google own Waymo? I feel like they would have connected the dots.

I'm no expert, but I think the future is more about extremely low latency and low power chips with LLMs etched directly onto them. You can create specialized chips that function as "neurons" in a larger system, generating the needed reactions with a very clearly defined set of constraints.

Probably not. In humans, the visual processing circuitry is very different from the circuitry for language processing. There is no reason to believe GPTs will be effective at it.

Sounds really expensive. I think OpenAI and Anthropic should really not dismiss making smaller capable models that they can license out in this space on the other hand.

Yeah cause every car needs 8xH200 pulling 10kW to run a VLM at realtime speeds. Would be unfortunate if 4G dropped out under some trees while using the API after all.

I think this a slight different lesson. There is one algorithm that is called transformer, rest is irreverent/performance optimization.

> The bitter lesson is finally coming for

This is hilarious, and good: Those who were too lazy/stubborn/arrogant to adapt, get disrupted and buried.

Oh god, this move-fast-break-things thinking is going to kill so many people. We already have aftermarket problems with people adding in untested, unregulated self-driving features.

https://arstechnica.com/cars/2026/09/aftermarket-driver-assi...

Are you using GPT without a harness? Also latency.

lol. Wait until your cloud frontier LLM stalls / disconnects due to load / interference while your car is on highway OR making unprotected left turn OR approaching pedestrians.

It is easy to make car driving *demos*.

Self-driving tech is more about reducing liability than the driving itself. The lidars and 3D maps and world models and everything else is needed to get reliability from 99.9% to 99.99% on public roads. This isn’t a SaaS product where the target is to be “good enough” at the cheapest cost.

I would assume this is a proxy for general intelligence. A model that can drive a car and do a bunch of other real world stuff is closer to a generalized intelligence that can reason through any task.

Tesla isn't using a general purpose model, they're using many highly-specialized models for a more deterministic system than "hey chat drive this car for me"

You cracked what the major version stands for!

Also heard about this! But the point of our benchmark was to evaluate frontier LLMs with vision out-of-the-box, which we wouldn't expect to have been specifically trained on driving real cars. The fact that they can do anything at all (even in an open lot cone course, at low speeds) is pretty impressive. I'm sure Qwen Drive and models specifically trained for driving would do even better.

- Aditya, Tobias, Simon

Wouldn't just putting tokens in a ring buffer work?

Is this a side-effect of a decreased attention span with everyone getting hooked on phones during COVID?

Gouranga!!!!

By making a simulation first so it can run as slowly as it needs to.

A different way to think of this is, consciousness is just a near real time video game with causal influence.

It drives step by step, very slowly.

The course looks like it is something that a human could do in 15 seconds, while Astra took 5 minutes.

AC10 had an interesting post around this general area earlier today:

https://www.astralcodexten.com/p/mysteries-of-ai-generalizat...

Yes, it is. If you convince a model it is inside a sandbox it is much more likely to comply with requests that would normally be against its guardrails.

Astra will flag if you tell it to reverse engineer a binary, if you look it up to the binary ninja MCP it will just do it lol.

in my experience yes, I've worked around "I can't do this on a real site" multiple times by telling it I was working in a test environment

another trick is to have it build something in a sandbox and have it add a human-editable setting to point it to places outside of the sandbox

seems like they're somewhat more willing to build a metaphorical gun as long as they're not pulling the trigger

Driving over empty parking spaces was definitely required, the cone course went through parking spaces (see https://drivingbench.com/report/#course) and also went tightly around curbs. Next time we definitely want to go farther out from the Bay Area and find a much more open lot to build a larger and more difficult course. But even at these low speeds and with this course, the LLMs performed better than we expected!

- Aditya, Tobias, Simon

Definitely the latency/speed of JEV would be great here! Unfortunately, Jev doesn't natively take in vision inputs. (So this also means demos you've seen of Jev playing games have given full structured state, which we can't do for actual driving in real time.) We tried hacky things like doing some "System One" Jev + "System Two" GPT 5.6 Luna (or other fast LLM with vision, to not bottleneck the speed) but it isn't working very well. Some open source Jev alternatives with vision exist and we might try those sometime.

- Aditya, Tobias, Simon

Haha fair point, we didn't juice anything though! You can see all the traces and videos on the website, for example, here's one of Claude Fable's attempts: https://drivingbench.com/trace/claude-fable-5.1/3/ . The code is also open source on GitHub. Also in the Report you can see how we did everything; there's obviously variance but if you try a similar thing yourself the results would probably be similar?

- Aditya, Tobias, Simon

So... competitive with Uber, in other words?

This is quite impressive...

But I imagine this is orders of magnitude more expensive / less efficient than whatever Waymo is already doing, right?

The cool thing is that 1) it's theoretically more generalizable, 2) if we wait 18 months, it'll be 100x cheaper, and another 100x cheaper likely in 18 more months - at that point - something like a Mac Studio inside a humanoid could have these generalized capabilities, and a lot of Robotics problems start to look more feasible - especially when you consider how much better the models could be if highly specialized.

There isn't any model out there even close to as good as Astra at visual/spatial reasoning.

Self Jevving Car

I’m gone for a day and I already have no idea what people are talking about

In all fairness, this would be one of the better use cases of Jev I've seen.

Huh? SDCs basically use a form of Jev.

Jev is the union of these two worlds.

"Get off your phone!" "Stay right except to pass!"

I could get behind this.

I've been somewhat curious how random any LLM would handle a task like controlling a roomba and have been seriously considering trying it out. An RC car would be a fun experiment, perhaps an RC plane would be too?