freedomben
3 hours ago
I worked on the control systems for Predators and Reapers back in the mid and late 00s, and the inefficiencies around process were enormous. Safety is extremely important, so you expect some slowness as a result, but it got pretty extreme. I remember one time having to do 6 weeks of testing around a one-line code change because a "helpful" dev fixed a small bug that had no practical impact. Yet because it changed the release build hash, we had to go through a full acceptance test. As you can imagine that incentivized only fixing important bugs, and even those we had to consider whether it was worth it or not. As a result there were a hole pile of bugs that we (and customers) ended up just living with.
On a separate note, I'm curious as to whether AI is making an inroads in that space. I would imagine very minimal, if at all, but very curious.
ACCount37
43 minutes ago
In Ukraine, the first place that was bombed was the red tape factory.
The drone industry was allowed to basically "do whatever as long as it works", consequences be damned. So they use civilian motors, batteries and SoCs, sketchy firmware with zero code inspection, and more. Does it work perfectly? No. It works well enough.
I wonder if anyone is going to learn a lesson about overregulation.
I'm not sure if "AI for red tape mitigation" is a thing, but "AI for killer drones" sure is. I suspect that "killer drones are insufficiently smart" is easier to fix with AI than "too much red tape". Because the amount of red tape, if unopposed, will expand to consume any capacity of dealing with it, AI or not.
setgree
16 minutes ago
I'm not an expert but I think this is an old lesson in warfare, that guerillas can triumph over larger adversaries by being more exploratory/iterative and less rules-bound. Tolstoy tells this story in the second half of War and Peace. Likewise with Iraqi militants wreaking havoc with IEDs. People repelling an invader have every incentive to move fast.
philangist
7 minutes ago
I've just read that section of War and Peace and was blown away by the descriptions of guerrilla tactics as well as Tolstoy's way of capturing the state of mind of the Russian POWs and their ever-shifting relationships with their captors.
As an aside, the word Guerrilla (little war) was coined during Napoleon's occupation of Spain to describe the resistance effort by locals and peasants against the French army.
avianlyric
9 minutes ago
> I wonder if anyone is going to learn a lesson about overregulation.
Seems unlikely. Regulation and Health & Safety are both societal luxuries, which only happen once societies are stable and prosperous enough to start valuing human life beyond its ability to perform labour.
The moment the bombs start dropping, the time for luxuries also stops, and the value of human life drops to value a person can produce defending their society. There isn’t the money or resources for anything more than that.
The US (most developed democracies) places an extremely high value on the lives of soldiers, because dead soldiers in foreign wars does terrible things to politicians in power. Paying 1000X more for the same tech as Ukraine to minimise the number of service members killed using it, is a pretty small price to pay.
godwinson__4-8
6 minutes ago
"It works well enough" is a significant understatement. I think it would be more accurate (especially given the perceptions at the outset of the war) to say that it has worked significantly better than anyone expected. Ukrainian ingenuity is single handedly driving conversations about the "future of warfare" in capitals from Brussels to DC to Beijing.
> I wonder if anyone is going to learn a lesson about overregulation.
This also misses the point imo. A simpler answer is "necessity is the mother of invention". There is value in a regime for peacetime. One is also a fool if they do not recognize needs change drastically in wartime. Two things can be true. The United States, like nearly all sensible nations, has almost always understood this and acted accordingly. On the other hand, nations that govern themselves as if they were on a perpetual war path are usually far less desirable societies. The idea that we need to speed rush "AI for killer drones" because otherwise we will find ourselves on the wrong end of an existential invasion are nonsensical. Americans would be far better off if our leaders and our people stopped acting like every potential conflict was existential.
There is no Russia on our borders. The only thing American adventures overseas have accomplished in the last two decades is making our country weaker.
lokar
23 minutes ago
I don’t know the details of that situation, but I have been on the other side of that debate.
People say “it’s a one line change” (once they argued it was a 1 bit change!). But lacking a fully controlled and hermetic build system with its own exhaustive test suite you can’t be sure about the relationship between the source and the binary. And that continues to every step to get the binary into production (updating existing devices, etc).
ACCount37
16 minutes ago
Welcome to Tradeoff Town.
Sure, your ultra paranoid checking of everything might catch an extremely rare bug caused by something like interactions between a benign code change and a build system. But is it worth slowing down the development process by that much?
Is it worth missing out on an entire generation of technology, like what happened with US and the shift from 00s drone warfare and 20s drone warfare?
Usually not.
justsomehnguy
7 minutes ago
And the consequence of this is a total lack of the consequences when something goes wrong. Complete with a PR cover-up all over the world.
pjc50
3 hours ago
cf the other thread: https://news.ycombinator.com/item?id=48845442 ; Ukraine has a hugely inventive and effective drone industry because it has to work. If it doesn't succeed, there is no Ukraine, and everyone involved in making the drones is dead, fled, in a POW camp, or sucked into the internal Russian displacement system away from their family.
By comparison, if the US products fail, there's no real negative effect on the mainland United States.
scottyah
10 minutes ago
They can pop out defects and if things go wrong and there's friendly fire or civilian loss, it's chalked up to scrappy efforts in war. The USA does not get the same amount of leeway, saving our people is a top priority and the media harps on any mistake.
27183
an hour ago
> if the US products fail, there's no real negative effect on the mainland United States
It's even worse than that. Schedules slipping and cost overruns are good things for the manufacturer, because they can charge more on top of their initial contract. Cost-plus ftw.
Retric
21 minutes ago
Cost plus isn’t nearly as common as it used to be.
But you still run into similar issues regardless of the contract structure. Try and build a rail network without anyone in government wanting something different for 20 years.
barrenko
an hour ago
To add that Ukraine was also USSR's drone research center; not to detract from what you are saying.
rayiner
an hour ago
Yeah, their tech development in drones is really impressive. An invading army has a way of focusing the mind and bureaucracy.
letitgo12345
an hour ago
Well there's long term impact but yea that doesn't create enough political pressure to make process efficient
Jtsummers
an hour ago
> As you can imagine that incentivized only fixing important bugs, and even those we had to consider whether it was worth it or not.
Or you're batching your releases into larger builds because you know it'll take 6 weeks to test regardless. This increases the duration of each development iteration because you have 100 things you want to do and you could do that in, say, 4x13 week efforts, but with the added 6 weeks between iterations (and possibly more after it leaves your shop) that takes a one year effort and turns it into about 1.5. So the program office decides you should do one big release each year, which also ups the risk because a lot of testing that would catch bugs isn't done until the end in that big 6-week test effort. Oops, now your 1 year + 6 week effort just got turned into 1 year + 6 week + (unknown rework time) + 6 weeks. Probably 2 years.
supriyo-biswas
an hour ago
My last job was like this, a full round of QAs manually testing, along with writing up a grandiose release document, stakeholder approvals, and whatnot, for something that should just take two days of development work, and an insistence on putting configuration values into the database, supposedly because it's "safer" than deploying a new configuration file.
legitster
2 hours ago
This is one of the reasons weapons and technology development overall explodes during wartime. Desperation is the cure for risk aversion.
It's also a reason to be skeptical of a military spending a bunch of money developing technology during peacetime. In reality the expensive stuff they went into the war with is always going to be less effective than the cheap stuff they came out with.
throwaway894345
26 minutes ago
Maybe if you're in a war that actually threatens your country. In the US, the Republican Party wages wars (such as the Iran War) pretty exclusively to facilitate borrowing public money and dumping it into the pockets of the rich. What looks like waste to the taxpayer is a feature to Republicans and their paymasters (including Israel, in the case of the Iran War).
giardini
3 minutes ago
a "hole pile" of ...?!
I'm surprised that someone who uses such a phrase was working on classified hardware in the "mid and late 00s".
pixl97
3 hours ago
Life critical software that gets visibility by congress tends to be a very bureaucratic process. Your boss doesn't want your commit being the one that causes a worldwide diplomatic issue.
I assume that smaller/cheaper drones avoid a lot of this because the stakes aren't near as high and quite a bit of the development occurs in private industry first.
matwood
2 hours ago
> visibility by congress tends to be a very bureaucratic process
See also SpaceX vs. NASA. No way would NASA have been allowed to blow up as many rockets as SpaceX did to finally get to their working solution.
throwaway894345
23 minutes ago
Yeah, the anti-regulation people when NASA experiments: "look at all these failures! Cut NASA funding and give public funds to the guy who purchases elections!"
The same people when SpaceX blows up a bunch of rockets: "wow, look at the innovation, they move so fast! Cut NASA funding and give public funds to the guy who purchases elections!"
ACCount37
19 minutes ago
NASA's failures as of late are less "dramatic explosions" and more "delays", "cost overruns" and "lack of ambition so severe it borders on criminal".
The last time NASA caught any serious flak was what, the Starliner shitshow? And that was just splash damage from Boeing getting dunked on by everyone at once.
quickthrowman
an hour ago
I would guess that DoD procurement rules have more to do with it than Congress, but perhaps Congress defined DoD procurement rules.
Milspec is expensive and process heavy, see what a B52 replacement trash can costs, for just one example.
dragonwriter
an hour ago
DoD procurement rules are largely the outcome of Congress trying to prevent executive-industry corruption through mandated process.
dboreham
33 minutes ago
But aren't the politicians also corrupt? (or at least most of them) One therefore assumes that any action by congress must be corrupt. This appears borne out by the evidence over the past few decades.
CoastalCoder
an hour ago
If you think Milspec is expensive, you should see the cost of not having a Milspec supply chain while still being risk-averse.
throwaway27448
an hour ago
We could always be less risk averse. We still seem to kill civilians at high rates and our own soldiers signed up to die. Gold-plated weapons aren't much good against peer powers anyway; it's production volume that wins those wars.
hvb2
3 hours ago
Why would those fixes not be batched up? So fix 20% of those and do one round of testing?
bad_haircut72
2 hours ago
The obvious answer is that the more bugs you batch up, the higher the chances the next build fails - this is why CI became a thing, small iterative changes are safer and lead to greater throughput
dataflow
2 hours ago
CI doesn't mean doing all the tests all the time though. The expensive tests still wait until there's a major reason to run them. I had the same question as the parent and I still don't quite see why this can't work.
aidenn0
an hour ago
I didn't work on this project, but I've been involved with similar ones.
There is a process for getting a change into version control. Each change needs to have a (virtual) paper trail: motivation, risk analysis, sign-offs &c.
If you can't get something into VC quickly, you can't really do CI.
The obvious solution would be to have an integration branch that doesn't need the process to get in, do CI testing on that branch and then make the process for merging to the real branch.
I've never seen this done personally, but I have been told some places do it, and then you end up with "Change X, which got approved had a dependency on Change Y that didn't get approved and we didn't realize it until now because Change Y was put in the integration branch before Change X"
Manuel_D
an hour ago
True, but if each run of CI takes 6 weeks then you're going to vastly hamper development.
HarHarVeryFunny
an hour ago
In contrast I read that Ukraine is approving 4+ new weapons systems PER DAY !!
Even when it comes to more expensive things like cruise missiles it seems the planning has to be that some high percentage of them may be shot down (and much higher for slower moving drones), so you really want them cheap and in high volume, with reliability somewhat of a secondary concern.
pc86
an hour ago
This only makes sense if you have to test each fix in complete isolation which seems silly even for government employees and contractor body shops. You can't batch 80 real bug fixes and 20 "silly bug with no practical impact" fixes together?
cm2012
2 hours ago
This is why Ukraine is making equivalent tech now for 1/10th the price. It's great to see.
yonaguska
2 hours ago
All the drone footage is actually pretty horrific to see.
ndiddy
an hour ago
Yeah FPV drones have massively increased the cost of invading a country. It's basically a return to WWI style trench warfare, except no man's land is the size of the range of the drones (20+ KM).
red-iron-pine
an hour ago
the gray zone is ~200km so long as you have ISR drones able to see it
artillery, missiles, and long range drones are in the mix too. AI enabled spotting makes ISR detection rapid and effective.
some kubernetes container spots a random pixel that means hidden vehicles and a HIMARS strike is dispatched ASAP
floatrock
an hour ago
It's always been pretty horrific. We just didn't have bodycam livestreams invented during trench warfare times.
LgWoodenBadger
an hour ago
Hopefully making the real horrors of war visible to the public/world will make war less frequent. Alas...
scottyah
2 minutes ago
Humans just adapt and normalize things too quickly, so long term seeing more violence will just desensitize people and make it more acceptable. Just look at the difference between protected kids in suburbs vs kids growing up in bad areas.
disgruntledphd2
an hour ago
People have been saying this since Vietnam (and actually now that I think about a little more, this may actually have happened).
CoastalCoder
an hour ago
Most of us who build weapons of war live with some awful dilemmas. One of them is:
We want to perform our work skillfully, effectively, and professionally. But we never want our tools to actually be needed.
(Another is that we can't effectively create a shield without the risk of it being used as a sword.)
ponector
an hour ago
A month ago Sternenko had a crowdfunding campaign for the secret drones. Recently he showed the fruits of that campaign: new drones have better connection and now you can watch kills in 4k!
Also recent advances in battery tech brought increased energy density: the same drones which had range 20km now have 40km.
pjc50
an hour ago
Yes, but it's not necessarily more horrific than the unseen effect of the 155mm shells. It's just that those don't come with killcams.
(The killcam is a WW2 invention, starting with linking cine cameras to the machine guns of fighter aircraft)
meindnoch
an hour ago
Depends on who's on the receiving end.
kilroy123
an hour ago
I agree. Nightmare fuel.
rayiner
an hour ago
Invaders getting blown up from the sky. What’s not to love?
LgWoodenBadger
an hour ago
Surely the dev wasn't able to merge that one-line code fix causing 6 weeks of testing without any other eyes on it and without someone else's PR-like approval...right?
general1465
2 hours ago
The bureaucratic development process sounds like Autosar in automotive. I am not surprised that newcomers from USA and Chinese auto companies are able to completely dominate in software because Autosar based development has been like giving a birth to a hedgehog. Slow and painful.
willmadden
43 minutes ago
They should be deploying half baked prototypes, testing them in the field, and iterating incrementally improved designs as fast as possible, not trying to design the perfect product to sell to the military. That's how you lose a war. The DOD procurement process needs to be deleted and replaced with hundreds of startups with a hardline fail fast philosophy.
stickfigure
an hour ago
"Software saved the aerospace industry. Every other way of adding cost to an aircraft also adds weight."
mmooss
2 hours ago
> one time having to do 6 weeks of testing around a one-line code change because a "helpful" dev fixed a small bug that had no practical impact
Roll back the change? Also, fix the approval process - no way that should have been approved.
Generally speaking that is risk management, an unavoidable engineering tradeoff. In lower stakes situations, for example a critical application or server for a small office, we let low-impact bugs accumulate: Imposing risks, and therefore eventual costs, to avoid minor workarounds and low-impact bugs is poor engineering and risk management.
Engineering and all risk management includes tradeoffs. It's easy to criticize the downside of the tradeoff - the same people criticize the reverse decision when the server (or drone) crashes - when someone is not responsible for both sides of it, when they are not accountable for their words when the outcome occurs.
That's speaking generally. It's also poor risk management to be overly safe. I don't know about the parents' situation. But drone crashes (risking humans), mission failure, $50 million losses, and associated downtime (including delays) and labor costs, seem like high costs that are worth some pain to avoid.
varispeed
2 hours ago
If safety is extremely important, why there were bugs in the first place? Surely these should have been caught before code would get into main, no?
tehjoker
3 hours ago
I don't really understand how any of this contributes to "defense". Sounds like "offense" to me. Just patrolling the skies over non-white countries and launching missiles at weddings. The reason the Pentagon invests so heavily in this kind of technology is our wars are so indefensible, they can't convince Americans to sacrifice blood in any quantity for other people's natural resources.
ivell
2 hours ago
Drones can be used for both offense and defense. For example the usage of drones in Ukraine can be classified as defense.
SanjayMehta
26 minutes ago
The Ukraine used drones to kill students in Starobilsk, a place they claim as their own. The place, not the children.
Some defense.
You support Banderaites in such a defense.
peyton
2 hours ago
They are white [1].
[1]: https://www.documentcloud.org/documents/21211671-1997-revisi...
graemep
2 hours ago
They certainly look white to me.
Are Spanish people white or Hispanic according to those definitions?
'Original peoples' is an interesting phrase. Neanderthals? Beaker people?
A single category for everyone from Pakistanis to Japanese is weird.