bri3d
3 days ago
This is quite cool since past efforts in this direction have usually relied on crutches like outside-in imaging and positioning.
A few details I picked up:
* The drones are a spec drone across the league. It's a fairly large-footprint FPV racing drone (it's a 5" propped drone, but it's very stretched out and quite heavy) with both a Betaflight flight controller and a Jetson Orin NX onboard. Teams were only allowed an IMU and a single forward camera.
* It's unclear to me whether the teams were allowed to bypass the typical Betaflight flight controller which is present on the drone and use direct IMU input and ESC commands from the Jetson, or whether they were sending and receiving commands from the flight controller and relying on its onboard rate stabilization PID loop.
DCL is kind of a weird drone racing league since it's made for TV; it's mostly simulator based with, more recently, only few real events a year. The spec DCL drone isn't very capable compared to the more open-specification drones in racing leagues like MultiGP, in large part to keep the events more spectator friendly. This probably makes it more amenable to AI, which is an interesting side effect.
generalizations
3 days ago
From near the bottom:
> One of the core new elements of the drone’s AI is the use of a deep neural network that doesn’t send control commands to a traditional human controller, but directly to the motors.
bri3d
3 days ago
I saw that too - I'm assuming it means they're indeed using the DNN for stabilization. This has been done several times over the years, but generally with results which only rival PID and don't surpass it, so that's quite interesting. What's odd is that the physical architecture of the drone doesn't really make sense for this, so there must be some tweaks beyond the "spec" model. Hopefully some papers come soon instead of press releases.
sorenjan
2 days ago
They reference ESA's research in "Guidance and Control Nets", and when looking at ESA's page for their "Advanced Concepts Team" [0] they in turn reference ETH Zürich's research in RL for drone control. Specifically [1] this paper from 2023: "Champion-level drone racing using deep reinforcement learning" [2]. They use a 2x128 MLP for the control policy.
[0] https://www.esa.int/gsp/ACT/
[1] https://www.esa.int/gsp/ACT/projects/rl_vs_imitation_learnin...
koolala
3 days ago
This is crazy, its dexterity and range of motion could potentially exceed all human modeled systems.
HenryBemis
3 days ago
I assume that they shave off milliseconds by doing so, and a gyroscope (or similar) sends back the position/angle of the drone. And like this does it bypass the 'limited' onboard computer and instead uses a much better/faster computer?
pjc50
2 days ago
Reports downthread suggest that the NN is running directly on the drone, in the form of a Jetson. Which would give much better latency and quality of video.
itishappy
3 days ago
There's a few more details in the press release from the league itself. Sounds like they were really trying to put these things through their paces.
> The course design pushed the boundaries of perception-based autonomy—featuring wide gate spacing, irregular lighting, and minimal visual markers. The use of rolling shutter cameras further heightened the difficulty, testing each team’s ability to deliver fast, stable performance under demanding conditions
https://a2rl.io/press-release/9/artificial-intelligence-triu...
pacetest
3 days ago
It used a small RL trained network running on the flight controllers MCU directly that controlled the motors given state (position, orientation ...) inputs. The Jetson handled vision processing.
NegativeLatency
3 days ago
I imagine the slower speed is a closer fit to combat drones (which have a payload and sometimes a fiber optic cable)? Also watching MultiGP they sorta move/accelerate too fast for me to fully appreciate the maneuvering.
Feels kinda similar to the innovation around manned aircraft about 100 years ago when we went from toy/observation platform to killing machine in only a couple of decades. With the ardupilot news today, it was hard to not watch this and imagine the applications to a combat environment.
close04
3 days ago
> which have a payload and sometimes a fiber optic cable
The optic cable is for the human pilot. An AI piloted drone doesn't need it.
akie
2 days ago
If we are ok with AI drones autonomously choosing bombing targets, then you're right.
athrowaway3z
2 days ago
I suspect the first big use case and subsequent rewrite of the battlefield will come from AI drones targeting enemy drones.
IshKebab
2 days ago
I think Ukraine would be ok with AI drones launched next to a Russian airfield autonomously choosing targets...
I expect Russia will be ok with it in any situation.
jiggawatts
2 days ago
Obviously the Russians wouldn't be one hundred percent okay with drones autonomously choosing targets in or near Russian airfields...
genewitch
a day ago
They'd be fine with the drones taking out all of their 3 and 5 engine bombers though.
lupusreal
a day ago
It's been done before, specifically in the naval context.
closewith
2 days ago
Although even autonomous combat/ISTAR drones may require fibre spools for BDA, ISTAR, etc.
Aurornis
3 days ago
> I imagine the slower speed is a closer fit to combat drones
A lot of comments are trying to draw connections to combat drones, but drone racing like this has been a hobby thing for a long time. The capabilities of the drones are set to have an even playing field, not to match combat drones or anything.
These aren't meant to have any parallels to combat drones, drones that fly long distances, or drones that carry payloads.
It's really just a special-purpose hobby thing for flying through a series of gates very quickly. Flight time measured in a couple minutes, no provisions for carrying weight.
david-gpu
3 days ago
We all understand that. People are simply observing that there an obvious path from this technology demonstrator to something similar in the battlefield.
jack1243star
2 days ago
> We all understand that.
I'm afraid not. RC/FPV is already a niche hobby, and media coverage is universally negative. No wonder laypeople mostly think of kamikaze drones when they see something like this.
tekla
2 days ago
You mean the stuff that anyone knew was possible 10 years ago, but was waiting for the tech to become much cheaper?
tough
2 days ago
Yeah, now its just cheaper enough so its happening
wepple
2 days ago
> The spec DCL drone isn't very capable compared to the more open-specification drones in racing leagues like MultiGP
Yeah, I’m sure this is a great milestone but it isn’t notable until AI is beating MCK[1] who would be the “Lee Sodol of FPV”
durandal1
2 days ago
Also, this track looks nothing like a competitive drone race track, the obstacles are easier and it seems designed to cater to the autonomous drones.
bri3d
2 days ago
It’s DCL, they’re all kind of poor like this IMO. Certainly nothing like MultiGP
user
3 days ago
walrus01
2 days ago
> which is present on the drone and use direct IMU input and ESC commands from the Jetson
Does the jetson board even have the appropriate UARTs on it to talk directly to the ESCs? Your typical hobby grade 5" class size ESC (either 4-in-1 or discrete) cannot talk to two different controllers at the same time. If it's already wired to the four UART outputs from a 30x30 size flight controller (such as something STM32H7 based running betaflight), the ESC cannot be in communication with any other device.
bri3d
a day ago
Right, that was my point about the physical layout not really being conducive to NN-driven stabilization, on my cursory read.
Digging in, you could probably bitbang DShot on Orin GPIOs. It would be really sketchy since it's presumably running Linux on out-of-order cores with giant cache, so there's nothing close to realtime going on, but it's so damn fast that the theoretical problem probably doesn't matter and I bet it would work fine.
Orin also has "safety island" for ASIL applications which has lockstep Cortex-R52s. This would be ideal to run an integrated flight controller RTOS but it's supposedly fused off on consumer boards, lame. There's also a Cortex-R5 called "SPE" that could probably run a flight control RTOS as well.
Regardless of all of this theory craft, I was either totally wrong or accidentally right in two ways, depending on how you want to look at it: according to a sibling post, the STM32 board actually runs the NN-backed stabilization loop rather than a PID one! I've seen this done in research before but never in a meaningfully successful way, so I'm even more impressed now - as far as I can tell, it's _two_ novel solves, an actually working PID-or-better NN-based stabilization system on STM32 _and_ inside-out vision based flight planning for racing on the Nvidia board.
user
7 hours ago
pacetest
7 hours ago
Its not just a stabilization/rate control loop but the NN does both guidance and control. It is an MLP that takes state + upcoming gate positions/waypoints and controls the motors directly. I.e. there is no explicit path planning going on but rather the NN outputs motor commands that hopefully help the drone traverse the given waypoints in minimum time. I think this is quite cool because running a similar Optimal Control MPC setup is really not at all feasible on a normal STM32H7 Flight controller, especially at 1kHz.
bri3d
an hour ago
Oh, I see how this works now! That's _incredibly_ clever, not at _all_ what I would have guessed, and quite innovative. Thank you so much for taking the time to post these explanations.