Openpilot – Operating system for robotics

241 pointsposted 10 months ago
by punnerud

79 Comments

jmacd

10 months ago

I have a Dodge Ram. Last night I had a 400km drive to do after a very long day. I wasn't exhausted, but I certainly felt like I did not want to drive an extended period of time.

I have a Comma 3x in the truck and felt way more confident, alert and comfortable for the entire drive. OpenPilot/Sunnypilot/Frogpilot are not FSD, but they are hands off driving assistance. The 2020 Ram performs incredibly well. The latest driving models are very smooth as well, no ping-ponging and they handle passing and traffic extremely well.

A legacy car maker would be smart to acquire Comma if its for sale. They would be extremely close to a viable assisted driving capability with it.

dham

10 months ago

I used Open Pilot for ~4 years. According to connect I have 8,000 miles on Comma 2, 20,000 on Comma 3 and 2,000 on Comma 3x. I recently sold my Rav 4 and went to a Tesla. Open Pilot is actually better in a lot of ways than default Tesla auto pilot, especially because it doesn't do crazy fantom braking on freeway. Open Pilot is also way ahead of pretty much every lane assist / adaptive cruise control systems.

Obviously, FSD is way ahead of e2e open pilot with navigation, but since Open Pilot can apply very little torque to the wheel, it can't do anything gnarly. I actually trust Open pilot more at this point but I guess I just need more time with FSD. Some of that is because longitude was Toyota controlled until I used the e2e longitude model more.

Even on "chill" mode, FSD will make random quick lane changes to turn only lanes to try to get around traffic. This is 12.5.2. Even so FSD can get me from point A to B with no interventions 98% of the time.

TSP00N3

10 months ago

There should be an option in FSD to have it not pass on the right and to change what speed difference it will wait to pass for (for example only pass when the car in front is [5] mph slower than what I want to go). These are separate options to look into than chill mode, and could also fall short to Comma, but thought I’d share in case you didn’t know they were there.

pj_mukh

10 months ago

Mind boggling to me that a non-ping pongy lane keeping is not standard in cars. Is it standard in luxury cars? Seems like an obvious thing to add/upsell.

hasperdi

10 months ago

Non ping-ponging lane following assist is already available in many cars including KIA and Hyundai models. They're very conservative and disengage very easily. I think it's by design to minimise their legal accountability

residentraspber

10 months ago

My 2019 Audi S5 was excellent at this. It would ping pong at most once then auto-correct itself to be perfectly centered in the lane.

It did some weird things like if the car in front of you was driving a bit too far to the left/right of a lane, it would copy them. Other than that it was nearly perfect, though. Never had it take an exit by accident, etc.

Their tuning on when to accelerate/brake and make it smooth needed a fair bit of work, but I found that switching the drive mode from Dynamic (Sport) to Comfort changed the eagerness of the system and smoothed things out.

MetaWhirledPeas

10 months ago

> A legacy car maker would be smart to acquire Comma if its for sale

My impression is that the Comma guys were never in this to sell their business

mdaniel

10 months ago

Also, unless I am misunderstanding the situation, since the code is MIT they don't need to acquire Comma to take advantage of the situation. I'd strongly suspect they all want to roll their own implementation for liability reasons, not strictly technical ones

grepexdev

10 months ago

I see that the Dodge Ram is not listed as a compatible vehicle. Could you explain how you managed to make it work?

Edit. I'm dumb. It's listed under "Ram" not "Dodge"

bilsbie

10 months ago

I can’t wrap my head around the fact that 275 car models include all the actuators needed for self driving driving and there’s some kind of port third party software can hook into.

kube-system

10 months ago

Level 2 driving assistance is commonplace on many new vehicles, often as a standard feature. They are just significantly more conservative in their functionality compared to the level 2 offerings from Tesla, and marketed as safety features rather than "self driving" features.

It's important to note that nothing we're talking about here is actually "self driving" per SAE standards. Openpilot, Tesla's Autopilot/FSD, Honda Sensing, Toyota Safety Sense, Hyundai SmartSense, etc are all level 2 driving assistance features.

This turns level 2 driver assistance features into ... nicer level 2 driver assistance features.

mavhc

10 months ago

Really need some decimal points in the SAE list. Fully self driving but a human has to watch, and lane keeping are both level 2

dham

10 months ago

Exactly, Toyota, Honda, Hyundai / Kia, etc have systems, they're just nowhere near Open pilot.

flessner

10 months ago

I don't know if there's a physical port in most cars, but it uses the CAN bus which has been around since the 1980s.

Also, most cars that have distance assist and lane keeping probably have the required hardware to control speed and steering to some extent.

Nevertheless, it's still impressive that so many cars are supported... and that it can be retrofitted like this at all!

fkyoureadthedoc

10 months ago

They also lie about models that are supported and won't assist you when you run into that. Had to return one myself. Found no evidence anywhere that my model/year was ever actually supported and anyone was using it either.

gotts

10 months ago

I suspect Openpilot sends packages to CAN bus to do steering but what makes the physical wheel rotates? I'm confused. Or when a human makes a correction to auto steering, perhaps there could be some kind of full duplex synchronization mechanism

michaelmior

10 months ago

I beleive in the US OBDII ports are required in all vehicles 1996 and later.

user

10 months ago

[deleted]

bdavbdav

10 months ago

I suspect it may be even more than that theoretically. A lot of VAG cars based on the same platform are missing.

RIMR

10 months ago

Somebody should make an adapter so I can plug a PS2 Controller into it.

colesantiago

10 months ago

I am not surprised that comma is still around.

Minimal VC funding, less than 100 employees, not outrageously increasing headcount each month, profitable and sells a product with good margins.

Not many startups do this anymore, they are just chasing funding every 3 months using OpenAI’s API, comma has their own models before the AI hype.

solarkraft

10 months ago

It’s cool to see that it’s possible to innovate sustainably, in a niche. So refreshing to see something stick around rather than become a bubble with a billion dollar valuation that either takes over the world and enshittifies or implodes and takes the product with it.

Make business chill.

That said: The larger they get, the more regulator attention they’ll attract. If some government entity wanted to, they could probably easily kill them.

bks

10 months ago

I chose the Hyundai Ioniq 5 as my current car specifically because it’s compatible with OpenPilot. It’s been a total game-changer for my driving experience. Just like their tagline says, “make driving chill,” and for me, it truly delivers on that promise.

sofixa

10 months ago

This reminds me of Waymo's approach to self-driving cars. Paraphrasing, but basically they found that progressively adding self-driving to help human drivers is bad, because it leads to the humans becoming complacent and not paying enough attention. Therefore they decided on an all or nothing approach, where their cars would be only and entirely self-driven.

typewithrhythm

10 months ago

This always seemed like a bit of bull from waymo. It's not an easy problem to work with existing manufacturers to give a better and or cheaper solution... Especially when there are established competitors with efficient verification and validation processes (that every manufacturer requires).

They decided it wasn't worth explaining that their techniques don't generalise to a driver assist. It would not be good or cheap enough to be worth developing the compliance and integration frameworks.

simondotau

10 months ago

That statement from Waymo always struck me as deeply uninsightful because it was really just a more complicated way of saying “self driving systems need to be good enough to drive the car on their own” which isn’t just obvious, but tautological.

user

10 months ago

[deleted]

zrt1019

10 months ago

I'm confused:

"THIS IS ALPHA QUALITY SOFTWARE FOR RESEARCH PURPOSES ONLY. THIS IS NOT A PRODUCT. YOU ARE RESPONSIBLE FOR COMPLYING WITH LOCAL LAWS AND REGULATIONS. NO WARRANTY EXPRESSED OR IMPLIED."

Where can this be used? In a private parking lot?

rogerrogerr

10 months ago

The driver takes liability, of course, and this can be used wherever the driver deems it safe and useful.

SkyPuncher

10 months ago

They’re just trying to scare away people who thing they can chuck this on their car and suddenly have a self-driving robot that they don’t have to pay attention to.

torlok

10 months ago

You pay a thousand bones for a device you have to babysit. What's confusing about that?

thatgerhard

10 months ago

Is it like an app you install on the car or is it a custom integration?

rvnx

10 months ago

It's a dashcam that you put on the windshield with 2 cameras pointing forward and one inward (filming the driver).

sfblah

10 months ago

There's an element of using these systems that people don't usually discuss. One of the forks, Sunnypilot, enables a mode where you control the pedals but don't have to do the steering. What I've found is you do pay attention in this mode, and since you're controlling the brakes you can easily avoid most issues that you would get with self driving. But, not having to have your hands on the steering wheel makes the experience a lot more pleasant. Also, with traditional lane keeping systems, the so-called "longitudinal" control (accelerator and brakes) is where in my experience the system makes the most mistakes.

I think this mode is something car manufacturers should enable in general. I actually suspect it's significantly safer than completely hands- and feet-free driving modes, and you get most of the benefit of lane-keeping assist.

pinkmuffinere

10 months ago

This is super cool, but:

1. Is this street legal? If so, how?

2. They discuss functional safety, and lots of testing, which is great. But I’d want to see some data on the test results — maybe this exists and I just didn’t see it?

3. It makes me uncomfortable that the anecdotal videos are easily findable, but bulk data/statistics are not. Anecdotes can easily be cherry picked. I get that they’re necessary for marketing, but I don’t feel they’re sufficient given the product’s purpose

ranon

10 months ago

This is a modern ADAS system, but a lot more stable. The driver is always liable for what their vehicle does. It's not claiming to be FSD and it's very apparent when the system will need additional inputs from the driver.

I have 15k miles on it. Was able to retrofit a friend's 2015 car as well with a bit of additional hardware, and he likes it. He also has FSD on a model3. But OP or FSD, driver always has to pay attention and add their inputs.

drivingmenuts

10 months ago

If I was in law enforcement, I’d be rubbing my hands in glee to get ahold of that saved video.

xipix

10 months ago

I'd be even more gleeful if I was in car insurance. "Openpilot's not covered, your insurance is invalidated."

punnerud

10 months ago

Most likely not legal to invalidate in EU. There is laws that’s say that ev everything you can do manually, you are allowed to automate. Any rules against that is null and void.

The “horse winning race” case is a known one where they go into this.

renewiltord

10 months ago

Geico will cover you. You can disclose ahead of time that you have an aftermarket ADAS if you want. If it drives you off the road, it will be as if you drove off the road and you will be declared at-fault, of course.

AyyEye

10 months ago

Insurance is a non-issue with openpilot.

siliconc0w

10 months ago

Been following Openpilot for awhile, would totally use it if I had a supported car. Though these days most cars come with 'pretty good' ADAS, even 'hands free' in some situations, so I wonder how much it's "worth it" to DIY compared to factory default.

ranon

10 months ago

I do think that within the next 5-10 years most cars will be able to hands free highway as reliably as OP. However, cars keep getting more and more expensive. One can buy a 2015-2024 used car for much cheaper and get some very good highway cruising out of it. That's what I did and am very happy about it.

ModernMech

10 months ago

I wonder why the website doesn't say "operating system" but instead calls it an "advanced driver assistance system"

niteshpant

10 months ago

for cars that aren't compatible, can you install additional hardware to make them compatible?

FullGarden_S

10 months ago

for a second, I though this was the ROS alternative I've been forever waiting for smh

rsp1984

10 months ago

Without taking away anything from the substance or achievement of this release, I find phrases like "openpilot is an operating system for robotics." always quite fishy.

No, it's not an OS for robotics. You can't do actual robotics stuff with it, like drive actuators to control limbs or grippers, do motion control or SLAM or perception or any of the usual robotics stack.

Their website correctly says openpilot is an open source advanced driver assistance system that works on 275+ car models of Toyota, Hyundai, Honda, and many other brands. Should've stuck to that.

Thinking about it some more, it's probably just another engagement baiting strategy to get attention and I'm their gullible puppet. Well played.

modeless

10 months ago

George Hotz says: "we developed a proper successor to ROS. openpilot has serialization (with capnp) and IPC (with zmq + a custom zero copy msgq). It uses a constellation of processes to coordinate to drive a car."[1] And Comma sells a robot that runs Openpilot: https://comma.ai/shop/body

> You can't do actual robotics stuff with it, like drive actuators to control limbs or grippers, do motion control or SLAM or perception or any of the usual robotics stack.

A lot of the "usual robotics stack" is not going to be relevant for the new wave of consumer robotics that is coming soon. It will be enabled by end-to-end machine learning and stuff like traditional SLAM methods will not be a part of that. The Bitter Lesson[2] is coming for robotics.

[1] https://x.com/__tinygrad__/status/1834792473081815259

[2] For those unfamiliar: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

ModernMech

10 months ago

I enjoy Hotz as a hacker, but I'm really allergic to this kind of oversold language. "[W]e developed a proper successor to ROS" is a past tense statement, as if they've already done this thing. In reality, at best they have presented a roadmap for a thing that could approximate ROS one day.

GabeIsko

10 months ago

In the robotics community, the stuff coming out of George Hotz has always been considered a kludgy mess, and unsuitable for serious work. Dude is a talented hacker, but the idea that this will replace ROS is kind of a joke.

moffkalast

10 months ago

The point of the bitter lesson is "leverage compute as best you can" not "use DNNs everywhere just because". Oftentimes your available compute is still a crappy ARM machine with no real parallel compute where the best DNN you can run is still not large nor fast enough to even be viable, much less a good fit.

And well some classical algorithms like A* are mathematically optimal. You literally cannot train a more efficient DNN if your problem needs grid search. It will just waste more compute for the same result.

Besides, the nav stack is not really the point of ROS. It's the standardization. Standard IPC, types, messages, package building, deployment, etc. Interoperability where you can grab literally any sensor or actuator known to man and a driver will already exist and output/require the data in the exact format you need/have, standard visualizers and controllers to plug into the mix and debug. This is something we'll need as long as new hardware keeps getting built even if the rest of your process is end to end. It doesn't have to be the best, it just needs to work and it needs to be widely used for the concept to make sense.

rsp1984

10 months ago

With due respect, this has to be one of the most ignorant takes on robotics I have read in a while. Yes, you can always slap serialization and ZMQ on your framework. That doesn't make it an OS.

And no, the usual robotics stack is not going away anytime soon. Maybe develop some actual useful robots before posting like an expert on robotics topics.

conradev

10 months ago

I believe the idea is that openpilot replaces the usual robotics stack with an end to end neural net.

While I agree operating system is usually a marketing term, it does feel correct in this case as it is the operating system for the Comma Three, which can operate cars but also this thing: https://www.comma.ai/shop/body

metal_am

10 months ago

I definitely thought it was a ROS clone based on that first line.

notum

10 months ago

ROS doesn't need a clone, it needs a successor.

Took the bait as well.

punnerud

10 months ago

Isn’t the software for training end-to-end NN to be used in automation? Just a first version that it’s used for cars, and they have been using it for their own robot.

So the claim still stands?

rkagerer

10 months ago

The docs (https://docs.comma.ai/) begin with a more honest - and useful - description:

openpilot is an open source driver assistance system.

tamimio

10 months ago

Yeah came to say the same, I thought a new big player is in the market. It looks great nonetheless.

user

10 months ago

[deleted]

akgrd

10 months ago

This seems to be a mix of C++ and Python, including a script called "realtime.py" (oxymoron?). So am I now exposed to other people using Python on the roads to operate heavy machinery?

traverseda

10 months ago

Well NIST says don't use C++ either: https://www.nist.gov/itl/ssd/software-quality-group/safer-la...

So what, you want everything written in RUST on a linux kernel with hard real-time patches? It uses machine vision anyway, which has no hard guarantees at all. The software it uses to detect lanes or cars is probabilistic by it's very nature.

Python does pretty good at soft real time if you manage your own event loop and disable the garbage collector, and you're a lot less likely to get "crash the entire stack" style memory allocation bugs. Sure, GO or RUST would be better, I think CPP could be worse if handled inexpertly.

thisismyswamp

10 months ago

exception handling in GO makes it unusable IMO

tmarkman

10 months ago

Python has segfault issues, surprising exceptions and version incompatibility.

I've been using Linux/BSD for over a decade now. No C or C++ application has ever crashed, I cannot say the same about Python applications. Outright segfaults are rare but happen. Rogue exceptions are much more common and could basically have the same detrimental effect on a self-driving system as a segfault. And let's not talk about logic bugs due to version incompatibility and the obsessive rewriting of those who took control over CPython.

AyyEye

10 months ago

The interface between the openpilot and the car is a standalone device (the panda) that provides and enforces the safety model. All code is written in C to automotive safety standards including ISO26262, ISO11270, ISO15622, and MISRA-C. 100% line coverage for all safety unt tests.

They also run pretty extensive tests (regression, unit, hardware/software-in-the-loop, mutation, and vehicle specific) on every commit and have actual hardware devices continually running real routes looking for regressions.

https://github.com/commaai/openpilot?tab=readme-ov-file#safe...

https://github.com/commaai/panda?tab=readme-ov-file#code-rig...

bobsomers

10 months ago

Just so we're all clear here, there is a lot of gobbly-gook in this answer which is either off target or irrelevant.

> a standalone device (the panda) that provides and enforces the safety model

What the actual safety model is that is being enforced is far more important here. The safety model could be "there is no safety guarantee whatsoever" and this sentence would still be true.

> All code is written in C to automotive safety standards including ISO26262, ISO11270, ISO15622, and MISRA-C.

26262 says practically nothing about software, what you really want is 21448. And 11270 and 15622 are super low targets for the amount of control authority available here.

MISRA-C is mostly a waste of time when it comes to safety. It gives software developers the warm blanket of having a checklist they can tick items off of, but does little to prevent unsafe systems from being built. Programmers have gotten pretty good about at least using tests and other analysis tools to make sure they're not doing the wildly stupid things that MISRA tries to prevent.

> 100% line coverage for all safety unt tests

100% like coverage is also rather trivial to achieve and doesn't say much. Branch coverage would be better, but being able to make some claims about state space coverage with exposure numbers would be what I'm expecting here.

enragedcacti

10 months ago

> So am I now exposed to other people using Python on the roads to operate heavy machinery?

yes in the sense that python is running the ML models and deciding what the vehicle should do, but it is heavily bounded in what it can do by the safety model which is implemented in bare-metal MISRA C running on the microcontroller that interfaces between openpilot and the CAN bus (panda). It enforces things like accel/braking limits and steering rate limits along with consistency checks, heartbeats, vehicle status checks, etc.

Level 2 self driving is already only a best effort system so if python caused an issue it would just fall back to the safety model on the panda and ultimately the driver to operate the vehicle safely.

jedberg

10 months ago

Sort of. It operates after your vehicle safety systems, so yes, python is controlling the car to an extent, but only within the limits of the built in safety systems.

KeplerBoy

10 months ago

Is this legal in the US?

user

10 months ago

[deleted]

guyfromfargo

10 months ago

I’m surprised to see so many people disliking Open Pilot on HackerNews. I have one of these, and it’s a total game changer on long trips. I drove from Texas to California using my Comma 3 and I didn’t have to overtake it a single time on the interstate.

Sure you have to actively be alert your entire drive, but it’s still significantly better than actually doing the work of driving.

1shooner

10 months ago

I've only driven a driver-assist system once and kind of hated it, but I've always assumed that the mental work of remaining attentive as a passive participant would actually be more mentally taxing than just driving. It seems like you're inevitably going to lose attention.

user

10 months ago

[deleted]

torlok

10 months ago

I always wondered who would pay $1000 for a device you have to babysit, just so that you can become an unpaid beta tester and data drone for SV millionaires. I guess all you have to do is tell people they're a part of a revolution?

392

10 months ago

Or want your car to drive itself. I only drive long distance and my Comma has completely changed how often I visit family and improved my safety coming home from exhausting weekends.