> Seems wild that they can code and deploy an update of a critical-seeming component in the field like that. I thought such a firmware update would need extensive QA in the lab, along with root causing why this bug slipped through their processes anyway.

These cars are in a perpetual Q&A state, they are getting modified between races. That's true for the "big teams", small teams don't have the resources, they're just "happy to be there". That's why it's possible to see stories like Brawn GP[^1]. That's what makes the sport fascinating, because the overtakes are not as much as one would hope for a ~2-3 hours race. But every little thing counts. In the last race 1st and 2nd had ~103ms diff or something like that for example, which is something that didn't happen for 15-20 years (maybe more).

[^1]: https://www.youtube.com/watch?v=jd3miTpvUmI

It seems they simply disabled a bunch of power limits. Probably just settings, rather than an actual code update.

F1 are effectively prototype cars.

Every GP is a slightly different car running bleeding edge tech and code.

I'm wondering how good the simulators are. At the start of the season, they were still learning the systems trying to figure out why they were losing power in the straights. So depending on the driver, issues may or may not become apparent. That's like trying to debug why an email can only be sent to a recipient less than 500 miles away.

It was probably just "config" changes or changing some hard-coded constants. So relatively straightforward.

This reminds me of a story how Deep Space 1 was fixed from 100 million miles-away. [1]

[1] https://gigamonkeys.com/book/lather-rinse-repeat-a-tour-of-t... (the last paragraph, search for "remote debugging" in the page)

The problem was that the ECUs went into limp mode due to multiple factors triggered by the drivers driving slowly (slick tires, wet surface, new PU ... ) and the ECUs are programmed to go into limp mode to protect the ICE, since it had no battery anymore and no turbo boost. The update was simply disabling that failsafe.

It was very unlikely there was actual coding involved, as even the C++ build and test toolchain would run for ages (in my automotive days it would run for 45 minutes and that was just build, link and static analysis). In this case it's probably just a config change.

I have no idea if modern-day SV’s “tests pass… idfk, it’s a web app… LGTM I guess” coding ethos was the cause of this particular bug. However, it’s a great, relatively benign demonstration of why it has absolutely zero fucking place controlling extremely dangerous hardware shooting a human around a race track surrounded by people.

I believe they likely used software they already had or just changed a software. As from what I read, it removed a certain system and limitions in the software which is used to ensure teams aren't cheating (traction control), which was triggering in error due to the strange conditions of the race. So all they were doing is disabling this system

They didn't, it just needed a restart. From the first reports the first car's software genuinely glitched out because of the new track/rain/etc and had to be restarted, the other cars got anti-traction control triggered because going this slow for this long unspools the turbo, which got so bad the detection software made a false positive.

I think those engineers had some memories of their FSAE / FSUK days during that patch.

It is big difference if a "one in a million" bug is deployed to two vehicles or a million vehicles.

When things are broken in prod and customers screaming you bypass QA to get the fix live

They rolled out a prepared safe defaults version with lots of checks, including the crashing one, removed.

I betcha that they are also tweaking the parameters of the cars while the cars are racing.

Disabling code is usually safer than enabling, seems they were able to pinpoint this rather quickly.

It was probably just a json with settings.

it has to be parameters only

The firmware situation is mixed. The ECU hardware is a standard TAG700 from McLaren. Basically these are software-defined ECUs where you can do whatever. However, FIA has some rules on what you're allowed to modify and has some "compulsory" software in there for monitoring and competition regulatory reasons. It appears there was very limited testing of this module in wet conditions which seems to have affected a lot of vehicles. (I work in Motorsports)

The FIA has this software to stop teams from creating a form of traction control (which has been banned for a good while now)

They all run the same ECU and the same firmware on it, yeah. Teams can control some parameters though. It's to stop any kind of traction control.

The teams use a single homologated ECU to prevent driver aids. I would assume it doesn’t have a downlink on-track to prevent tampering, but presumably with umbilical cord connected in the garage during red flag they can make changes, which the FIA probably sent over. Who knows.

Yeah I was wondering this also. I had no idea they had some shared blob (or maybe they have source code access) that runs in all of their cars. Would love to read more about how this process works if anyone knows!

Wonder if they all get given a thumb drive at the start of the weekend with the latest patch. Release notes saying "library updates and bug fixes, now go drive 200mph around a track! Have fun!"

TLDR: this wet weather mode stuff is FIA mandated because it is basically a rule enforcement around traction control. The loss of traction was related to a weird interaction in low-revving engines, turbos not kicking in, and the rest of the systems trying to do something that anti-traction control stuff would prevent.

During the Sky commentary, Anthony Davidson explained the specific electro-mechanical interactions based on what teams were telling him.

The FIA absolutely do need the ability to enforce regulations, but perhaps they should be monitoring for traction control and then stewarding that, rather than managing parts of the ECU firmware...

I like how this world class engineer's answer to Hamilton is basically "have you tried turning it off an on again?"

They may have just rolled back to a previous version.

Still waiting for 6.1 sol to patch a UI bug as am reading this

Is 6.1 Sol that bad? Or just slow?

Yes. And wasn’t this the first wet race of the season as well?

It's not Ilmore. It's Motion Applied (formerly McLaren Applied Technologies). However the software is provided by FIA, and the hardware design is also mandated by FIA.

Things have been changed.

That’s not correct.

The standardized ecu was introduced as a cost savings and fairness measure in 2008. The contract was won by McLaren Applied (which was part of the same McLaren that is the current reigning champion but is now spun off and owned by private equity).

The teams basically get an sdk and are largely responsible for the software the cars run at pretty much everything above the hardware layer of abstraction.

As a weird piece of history, McLaren applied was not the original winner of the contract for the first standardized ecu in 2008. When it was announced, in 2006, Microsoft was.

The issue is of why deaths happen isn't due to gas, its because F1 is simulatenously "build the fastest car that can hold incredible speeds in corners", and "we are going to put restrictions on when you can use technology to make racing fair"

Look at Moto GP. The crash speeds are much slower, because they happen during cornering, and because bikes don't have that much aero downforce even with all the power ranger wings on them now, the riders have to brake a lot more.

Instead of trying to push hybrid tech, which will never be that good for racing, Id like to see F1 focus more on actual ICE engine refinement, because ICE isn't going anywhere even in the age of evs. Something like no front/rear wings, no pitstops, ICE only, standard ECU for power delivery, no ABS, but you can run AWD with electronic suspension and torque vector as much as you want.

Last fuel related death was in 1986. The incident in 2020 with Grosjean's fireball was a 67g impact. If the car were to only have one big EV battery, which it almost is in current regulations, then in such a crash would very likely get punctured and be even riskier fire. So to answer the question, no.

At end of the day, it is a motorsport and it will always have its risks.

If you are that worrid about safety, don't ever watch the Isle of Man TT bike racing.

https://www.youtube.com/watch?v=31RZ5wU-Fg0

There are typically at least two deaths every year. The last time there were no fatailties was in 1982.

https://en.wikipedia.org/wiki/List_of_Isle_of_Man_TT_Mountai...

Those are two seperate issues. Safety issues are solved by making the cars/driver gear safer.

In the first race of the season, his team-mate was taken out before the race started because his car gave him more power than he was expecting all of a sudden. Similar things have happened multiple times this year.

> "It's just horrible. I don't know, it just proves that we shouldn't have any of that stuff, the electronic side," Norris said

It seems he doesn't want ANY electronics, which is frankly ridiculous. The steering wheel has looked like the one from a flighter jet for decades now. There is electronics in everything, even the helmets.

This F1 is shit, I've lost interest in it. When the race winner is not determined by the pilot skills, or the team, or the car itself, but by some software, battery power, excessive ruling, lawyers, etc. it's to me not at an interest to be seen.

To me F1 shall go back, remove all the crappy rules made in the last 20 years, return to pure gasoline V10 engines, no turbo, no hybrid, minimal electronic, no closed park, no limit in consumption of fuel, tires, etc, return to refueling during the race, and let the better pilot with the best car and the best team win.

These F1 is too sophisticated, at this point to me makes very little sense even racing, just race the car in a simulator at this point, everything is decided by the electronics and algorithms anyway, seeing talented pilots like Max Verstappen or Lewis Hamilton struggle to race because the electronics of the car doesn't let them show their potential, because they have to "save battery" or "save fuel" or "save tires", doesn't make a good show to me.

Your comment clearly indicates that you have absolutely no idea what you are trying to talk about. Norris's comment is perfectly clear for anyone who has any inkling of what is going on currently in F1. Why even bother making such a comment when it is a complete waste of bytes?

I wouldn't be surprised if many of the motorsport software engineers you mention come from mechatronic/electronic/aero backgrounds. Don't get me wrong, I've seen an analog electronic engineer perform software wonders; 10,000+ lines of bash, custom C macro libraries, and a PHP runtime older than I am have taught me that programming can be closer to art than science. But relying on that cursed bundle of chaos felt deeply violating and unsafe lol.

Nowadays, I mentally place any engineer without production software experience into a 'research' box. Any resulting code may break the laws of physics, and should be tested as such ;).

Edit: It's also a minor crime to look down on automotive software engineering! The stakes of mass-manufacturing have forged some seriously awesome software. Top-of-mind for me is CAN support in Linux and safety-critical contributions going into ROS2.

Similar HFT and quants, they are elite in a very specialized niche: aerodynamics and CFD. The rest is extremely outdated.

Keep in mind that when James Volwes become the team principal of Williams in 2023, he was horrified to realize Williams operated everything in Excel spreadsheet, and to track the progress of how parts are made, they emailed Excel spreadsheet to each other (and remember, we are talking about a car with 100k parts). And the first thing he did was to onboard Atlassian / Jira and redesigned the process of communication / car production from the basics.

Preach; I worked with a couple guys who left the F1 software industry and my experience matches yours. They wrote JavaScript like it was old school C, didn’t know version control etc, and refused to learn anything different.

Ah, yes, because the software on road cars is so much better.

The amount of times I had to turn off and on again my dad's Skoda because the screen was stuck on "loading user profile".

I looked it up yesterday and they have the ECU in the loop for the driving simulators (the ones the drivers practice with and that are used for checking mechanical changes). But these use empirical models, they can't simulate everything. You would have to get a lot right in the models and top of that know where to test edge cases and then simulate all inputs correctly for those. You can surely input yesterday's weather conditions but whatever model is there might not predict an output that goes to the next model and so on..

I'm sure there's also all kinds of other HIL testing but it comes down to thinking about all conditions and having models and tests that are realistic enough for those conditions. It's impossible to have perfect coverage.

It's probably kind of lucky that enough cars were impacted so that the issue was revealed instead of having a just a few cars knocked out of the race.

With something safety-critical like an ECU, I'd be surprised if they left formal proof out of their software design and testing. This would make unexpected hardware behaviour much more likely imo. I guess I'd get around it by setting up automated hardware-in-the-loop simulators for testing every possible environment and input. Surely the cost would be worth it for F1 as a whole.

At a startup I worked at, we jerry-rigged Pi 5's with commercial of-the-shelf environmental sim software to achieve this. Very cost effective, lol.

Lets all vent in one more place. How this kind of regs get forced is sad. All the casual fans want something fun to watch. All the serious fans want NA engines. The drivers hate these "power units".

Not going to happen. Most of the race tracks on the F1 calendar have noise limits nowadays. Yes, the classics - like Monza - included.

It looks like it will be mild-hybrid V8s with turbos from 2030: https://www.motorsport.com/f1/news/first-details-of-2030-f1-...

NASCAR still uses classic carbureted V8s. I personally prefer that sound over the much higher-pitched F1 engines.

It's a turbo that muffles the sound (and resulting reduced revs because high RPM was restricted due to cost reasons).

[flagged]

Last full analog car must’ve been in the 50s

isnt the regen neutered due to a veto from the bigger teams with the fear of audi. since they have well ran and tested r&d on regen for wec and some of the other E series they run?

For anyone not aware, this was the Bahrain Grand Prix held in Malaysia. The FIA allowed Bahrain to host their Grand Prix at a circuit which isn’t normally used for F1 any more in Malaysia because of the geopolitical situation in countries around Bahrain.

Therefore this was the Bahrain F1, it just happened to take place in Malaysia.

I'm amazed that there's real life fans of George Russell. Kidding. Kind of ;)

I like that George seems to be in a better headspace. He was like that posh brit talking others down with a know it all attitude, but since the summerbreak he has been way more relaxed and less antagonistic.

When you step on the loud paddle but computer says "No" although there is no reason to say "No", that's too much technology.

If you cannot drive in wet conditions without computer intervention because your electric engine applies too much torque for mortal humans to control, that's too much technology.

F1 is a constructors championship, and the constructors are interested in developing electric power trains. Nobody wants to spend 100 million to tweak the last 0.1% performance out of a turbo-charged V8.

My understanding is that the switch to hybrid was to get rid of needing to refuel during a race, since that is dangerous. The 50/50 split is new this year, but they're already working on fixing that for next year. But even then, they had to reduced the number of laps because otherwise they would need to re-engineer the entire car to allow more fuel.

Because electric engines are more performant than gas ones so you can in theory have better driving perfomrance? I.e., high, flat and predictable torque curve, re-use braking energy (so it adds up to ICE), extremely high energy density of the engine unit.

It’s not virtue signaling. It’s research. Pushing the limits of hybrids has downstream benefits for consumer cars, presumably.

I agree the current design is pretty dim-witted. But I think the high-level goals are sensible. It’s certainly more than just appealing to the green crowd, because that obviously is misplaced for this audience.

Let’s all just watch go karts and ignore what F1 always was because we think electricity is political now.

I have a chuckle every time the Williams is on the screen with its Claude sponsorship. Why don't they simply ask Claude to make their car fast, are they stupid?

My wife is tired of this joke.

F1 teams have never fully owned the car, you’ll notice they all look pretty similar. The FIA has a lot of rules around the construction of the cars

Gotta love how people assume software patches have never been prepared in an emergency before LLMs. Ever heard of Apollo 14?

I’ve really enjoyed this year.

What do you think mechanics do with old school cars?

They do. There are two parallel championships. The drivers and constructors championships.

That’s why when Hadjar was off with an injury the Red Bull team got a replacement driver (Liam Lawson from Racing Bulls), because they still want to get as many points for that car in the constructors championship as possible :]