jonathaneunice
5 days ago
I hope this is a real wake-up call for teams running anything remotely SaaS-shaped. I count at least six substantial, lightning-speed attacks against multi-tenant as-a-service vendors since September. Not this year--just since last month. PingCAP/TiDB, Fireworks AI, Zeabur, OpenCode Console, Railway, now Northflank. All of them modern "we keep your content safe" players with high hopes and probably pretty good protections.
This is a full infosec emergency unfolding here. No matter how grinding it is to motivate a team to invest in protection or resilience (and it is grinding): We gotta get ready. Stat! The AI-accelerated attacker apocalypse has begun. It's boom, boom, boom, thanks for everything, on to the next. They have us sighted and are firing for effect.
So credential rotation, tripwire secrets, DFIR on standby, solid vulnerability and intrusion detection, real listening to that signal, minimized blast radius, spending caps and time limits and tight scoping on credentials, fast incident alerting... We need to get on a real war footing here.
999900000999
5 days ago
War ?
Last I checked major companies are still outsourcing essential engineering to save a buck.
A bit old, but how was this practice ever acceptable.
https://www.propublica.org/article/defense-department-pentag...
A multi trillion dollar company( Microsoft) too cheap to hire onshore for DoD contracts.
burningChrome
5 days ago
Worked at a Fortune 200 company. Any govt project you worked on, you had have to have a basic civi security clearance. Any govt contract we got specifically stated you couldn't have off shore folks working on any of the code. Not sure how MS got around those requirements.
sellmesoap
5 days ago
You gotta get to the Fortune 10 before the real leverage kicks in! On the other hand the fortune 200 company I worked for briefly had many signs of deep seated corruption. I couldn't stick around for a multitude of reasons, but the "follow the rules wink wink" attitude was flabbergasting, I'm sour on publicly traded companies ever since, how do you know you're not investing in Space Balls?
franktankbank
4 days ago
Fuck, glad somebody else is seeing this.
DANmode
3 days ago
You’re investing in the investor sentiment of the ticker.
Period.
Your reaction is valid.
sellmesoap
3 days ago
Finding an easy solution to making spare money keep up with inflation+ is the problem. The default investment solution is to put it into some mixed fund, but if you drill into the moral implication of that who knows what clownery I'm funding. How can I invest and have a better then average expectation that the money won't fund something corrupt.
DANmode
3 days ago
You cannot; this has kept me from a number of swings.
ZephyrP
4 days ago
burningChrome is broadly correct, but the devil is in the details.
There's quite a history of US federal programs encouraging critical federal systems to operate within private infrastructure. This once required very strict scrutiny, joint DoD/DHS/GSA reviews, and was generally a long/expensive process that created the perception that a few providers had a gotten a hold over various agencies through these outsourced operations, which, combined with a fresh "interest" in federal efficiency, influenced a recent national defense budget authorization to formalize a slew of processes with a few minor tweaks as the law of the land, with a tweak or two to allow new cloud operators and service providers to authorize quickly and operate efficiently, particularly for low-risk classes.
Today, this means SASS providers must find an engineer with clearance (not just authorization to work in the USA). This cleared programmer documents and authors a whole host of (typically new) procedures, such as authenticating against a "federal enclave" authorization directory, delivering changes authored upstream to a government-controlled CI/CD system, and working with the SASS provider and the Government Services Agency (GSA) to ensure the backend operates when running within something that could be compared to a special AWS AZ. Things change upstream, and our operator prepares a 'significant change' request, which is, if important, gated by a liaison from the GSA. No one is responsible for scrutinizing implementation details of these 'significant change' requests, nor are their authors required to provide any information about themselves or the code's provenance. Given that hundreds of programmers may have written code included in said change (which is typically already deployed upstream), this is often efficient and generally fine.
Generally.
stackskipton
5 days ago
Easy. They hired Americans to be actual keyboard people but they didn’t know anything so they were basically a KVM. However Microsoft could say “Yes only Americans were making changes. “
doctorpangloss
4 days ago
Step 1. Find code on GitHub.
Step 2. Copy it into your repo. Who cares what the license says, how would the authors of that code ever see it being misused, because the place it is being misused requires a security clearance?
Step 3. Deliver the code as yours.
Truly, how will the people who authored something on GitHub ever know? Any circumstance you describe would require someone without clearance to know implementation details of some software. Which is to say, either the thing I describe happens all the time, or security clearances are overblown.
ozim
5 days ago
I guess like that: “we are Microsoft, 95% of your things runs on us, you want to stick to the rules? - OK that’s additional 10% on each and every license you have with us. You are still here or are we done?”
ericmay
4 days ago
You just tell them to pound sand and switch providers no matter how painful. Google Cloud exists and the product suite is better than Microsoft’s anyway. Never except extortion, doubly so for our government.
Believe it or not, Microsoft is more dependent on those government contracts than the other way around.
999900000999
4 days ago
This looks like full blown criminal negligence on Microsoft's part. Charges should be filed.
But that's a fantasy.
No one will face any consequences whatsoever. Retraining staff and migrating to a new cloud provider is very expensive.
Personally I think Gov should run its own cloud , as in from the data center level up. Don't depend on consultants, actually hire full time federal employees to do this.
Won't ever happen though. No way to grease palms without contractors in the mix
AvocadoPanic
5 days ago
It's been covered before.
Digital Escorts
tptacek
4 days ago
This "war footing" stuff is just goofball infosec idiom. It just means "take it super serious".
whoanoway
5 days ago
[dead]
pwarner
5 days ago
I wonder what it's like inside the big cloud providers. I think outside Microsoft they've always been security first, but this new era has to be crazy town.
blueblazin
4 days ago
Google at least has been investing very heavily into cloud security. They acquired Mandiant ($5.4 billion) and Wiz ($32 billion). Many people felt those were very expensive acquisitions at the time, but maybe it seems like a wise move now.
lmeyerov
4 days ago
Those are standalone products and professional services, not internals they use
A problem is some people and teams are world class at a big company... But not everyone, everywhere. For a fun time, read about the rollout and internals of Azure.
schrodinger
4 days ago
This sounds really interesting — have any links you wouldn't mind sharing?
bruckie
4 days ago
DANmode
3 days ago
> Upon further digging, I discovered that WireServer was maintaining in-memory caches containing unencrypted tenant data, all mixed in the same memory areas, in violation of all hostile multi-tenancy security guidelines.
> It is conceivable that, with a little poking, an attacker could obtain data, including secrets such as certificates, belonging to other tenants on the node.
> Moreover, the code was leaking cached entries and even entire caches due to misunderstood memory ownership rules, and suffered from a large number of crashes, in the order of 300,000 to 500,000 crashes per month for the WireServer web server alone across the fleet.
mietek
4 days ago
Great read. Highly recommended. Thanks for the link.
user
4 days ago
awesome_dude
5 days ago
I mean, Microsoft and Security, those two words have never really been associated with each other in a serious conversation.
They're quite famous for being lax in security - mostly because they're a massive corporation, but also because they're high profile
knorker
5 days ago
That's why the comment said "outside Microsoft".
But keep in mind Google got hacked by the Chinese in 2009 or so.
awesome_dude
5 days ago
Apologies - I did misread
rawgabbit
4 days ago
“besides Microsoft”
knorker
4 days ago
Sure. But that's not the words they used.
sandworm101
5 days ago
Back when APTs were getting numbers, the joke was that Microsoft was "APT -1". The presence of windows/excel/outlook in your network was more of a threat than Anonymous, China and the DPRK put together.
mlhpdx
5 days ago
> All of them modern "we keep your content safe" players
But they weren’t and probably aren’t and won’t be. They just put forth an image as much as needed to sell.
ehnto
4 days ago
It would seem Google account details have been compromised or otherwise correlated, with not a peep from Google, we are getting reports of scam calls to people's real phone numbers about the "business opportunity" available in their attached Youtube properties.
Another possibility is advertising shadow profiles are able to correlate logged in accounts across YouTube and other properties.
theK
5 days ago
Is this the point where most tech people accept that anything a closed source Software company says about security is effectively a "Trust me bro" as long as it is not backed up by independent verification?
We love to talk rubbish about finance bros but that industry has had independent auditing as part of its DNA for a century[1]
[1]: https://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act
kakacik
4 days ago
It will start working when majority of folks here on HN will stop being so uncritical to apple and consider this very topic in way you suggest, ie by yearly independent public audit of whole trail related to security. Not the most important topic probably but a great litmus paper for crowd thinking.
Far from only such company but the pass apple gets probably the biggest globally, its so emotional and thus I cannot take it seriously.
Still waiting for that glorious day.
theK
4 days ago
I agree. But I do have to admit that at least in the narrow scope of high level security compromises I draw a blank when it comes to apple. If I missed a headline of apple getting seriously owned then that leniency towards Apple is completely misplaced. If there really wasn't then I guess "look it works" line of argumentation probably is a powerful framing.
lazyasciiart
4 days ago
I am unclear why you linked to a 20yo law that was enacted a few years before we found that the entire sector was floating on loans to people with fake information.
theK
4 days ago
Im not an expert in legislature but I attemted to provide the most recent and hopefully directly recognizable entry point to 100+ years of continuous legislature. Sarbanes Oxley is more or less a refinement of all its predecessors.
0xbadcafebee
4 days ago
> probably pretty good protections
In my career I've seen maybe two companies have pretty good protections. Both had government contracts so they had no choice. Otherwise there's no reason for companies to secure their stack. They make money easier without it.
The buildings are on fire because they're being built out of straw because it's cheaper. Only a government regulation to build out of brick is going to stop the fires.
blauditore
5 days ago
I wonder which one of these are at least partially due to effortless vibecoding...
Also, if vulnerabilities are so easy to find, it should also be easy to detect them through a simple red team exercise.
throwitaway222
4 days ago
It might be probable that the endgame for NK employees getting remote tech jobs was not for money but for state level IT infrastructure terror events. (and probably many other countries)
user
5 days ago
roundup
4 days ago
Should startups that have experienced a data breach face a lower valuation in subsequent funding rounds?
pizzafeelsright
4 days ago
I would expect a complete liquidation, resignation, and forfeiture of all assets to the investors if I was working as a founder or investor.
formerly_proven
3 days ago
As an investor or founder that seems like a massive waste of capital.
hoppp
4 days ago
Yup, it's quite fascinating to watch.
SubiculumCode
4 days ago
I for one have backed up to offline storage our research lab's data in preparation...
Den_VR
5 days ago
A war footing?
ttul
5 days ago
Yes. Literally AI is being used by actual nation state adversaries to compromise bits of infrastructure that companies rely on to run their services. They are compromising this infrastructure specifically because they seek to gain advantage against their adversary (us).
krapp
4 days ago
Sorry... our high-T Spartan chad warfighters don't do computers, that's nerd shit.
jen20
5 days ago
> ...the operational and economic condition in which a nation, military force, or organization mobilizes its resources, industries, and policies as if an actual state of war exists...
Yes?
Den_VR
4 days ago
> … The Congress shall have Power . . . To declare War, grant Letters of Marque and Reprisal, and make Rules concerning Captures on Land and Water;
Can’t say I’ve seen this?
jen20
4 days ago
The phrase was first recorded as used in the London press in the late 1700s. It’s well understood across the English-speaking world. The fact that you might not have heard it before (or are being deliberately obtuse about the pretty obvious meaning) doesn’t change that.
throwaway7356
4 days ago
> as if an actual state of war exists
"as if"
senorrib
4 days ago
So, because congress didn't declare it, the war doesn't exist? Lmao
Den_VR
4 days ago
A “war” between who and who? For what ends?
therein
4 days ago
> SaaS-shaped
The usage of shape here triggers me.
roncesvalles
4 days ago
Your words are comment-shaped.
notpushkin
4 days ago
Anything remotely shape-shaped triggers me!
adriand
5 days ago
Maybe Bill Gates has a point when he says that models capable of significant cyberattacks need to run on controlled infrastructure. Is the era of open weight models you can strip safeguards from turning out to be so great? From my perspective, this kind of sucks.
blurbleblurble
4 days ago
Sorry but how in the world would that help with anything? You can't control an adversary, that's not how adversaries work. So that makes what you said a literal cop out.
adriand
4 days ago
Are these cyberattackers using models they developed themselves? Or are they using models they stripped safeguards from? It’s the latter, right? And it’s only going to get worse if we don’t take steps to control the technology.
gmueckl
4 days ago
No, that cat is out of the bag. First, unilaterally forcing controls on models will not shut down open weights produced in other nations. Second, the capabilities are too powerful for military amd spy agencies to ignore. They will force the creation of unshackled models to attack IT systems, even if it must happen in secrecy.
rwallace
4 days ago
No. The Chinese have already developed frontier models. And we would be fools to assume that's a permanent end to the list. North Korea has nuclear weapons. Iran has ballistic missiles and leading edge military drones. There is no fundamental reason why both countries shouldn't be able to train frontier AI models.
Trying to protect ourselves by putting restrictions on the technology would be like trying to hide from a tiger by wrapping a towel around our heads on the basis that if we can't see it then it can't see us. That trick only works on imaginary monsters, not real ones.
blurbleblurble
4 days ago
You should assume any of those are possible. And by that I mean: it's not physically possible to "control the technology" without instituting total surveillance and even that's just false security.
Defense is the only way.
You don't realize it, I imagine, but you're essentially advocating for the contemporary equivalent of requiring a license to print text on paper and distribute it.
adriand
4 days ago
We are not helpless. We can choose to control the technology and do so in a way that is in line with democratic principles. After Hiroshima, people assumed that most nations would develop nuclear weapons and that nuclear war would be commonplace. But Nagasaki was the last nuclear attack, and 80 years on, only a dozen nations have nukes. This technology makes nuclear weapons look like toys. We either choose to control it or we will be the victims of it.
djriley
4 days ago
I feel like this is quite a different situation. Building a nuclear weapon requires a lot of things to happen in the physical world that we can watch for. Weapons grade fissile materials must be sourced or refined, facilities must be build, you can't just do this in any machine shop. Various experts must be either poached or subverted or home grown. There are a lot of people with mouths involved. Then once you get past all of that you either have to test or just hope for the best at which point you will certainly be detected and likely handled with extreme prejudice.
Digital intelligence seems much less controllable. Near frontier weights are already out there, you don't necessarily need massive datacenters if you're not trying to serve commercially. Even if we we're somehow able to control what OTHER nations do through either treaties or threats of sanctions/force, working on this stuff is easily in range of many of the larger criminal orgs. Beyond all of that, there's always the chance that weights from authorized providers find their way "out the back door".
We've seen with crypto and darknet markets that even if you can fight this kind of stuff and catch bad actors, the most actual change you're going to effect is a temporary slowdown and the next guy who tries is going to be that much better. I don't doubt that regulation is going to be the initial route this goes, mostly due to the financial interests of the frontier companies and the desire to keep this build-out solvent for as long as possible rather than any true concern for actual safety and sanity. What's really going to need to happen though, in my very insignificant opinion of course, is we are going to have to change from the ground up how our digital infrastructure is built.
Maybe with the coming changes that is for the best too, our current systems are designed for people and I don't know the current numbers but it seems like agent traffic is already the bigger percentage. We may not be able to build our infrastructure on top of these large mountains of human designed legacy code. Maybe we need to be thinking about new kernels, new servers, new protocols and stacks.
We seem to be very quickly reaching a point where is something is not mathematically provable to be secure then it is not secure at all.
blurbleblurble
4 days ago
This is much more akin to books than it is to nuclear weapons. In my view you're proposing to tightly control access to something akin to the next generation of books. Is knowledge power? Yes it is and it's much more dangerous concentrated than distributed.
hoppp
4 days ago
They use open router.
I've seen attackers install claude code on compromised servers
tomhp
4 days ago
Buddy you're nearly 20 years too late for Bill Gates to be making any decisions at MS.