Buttons840
4 days ago
I'd feel a lot better if I knew some red team security researchers were going hard trying to break into critical national infrastructure, just to ensure it's not too easy, but probably this isn't happening because it would be a felony to even try. The bad guys are free to try though.
All this will tend to get blamed on AI, but AI didn't put these vulnerabilities into place. AI is just a tool that can be used to discover vulnerabilities, and the only question we have to answer is whether or not that tool should be available to regular people.
Some will try to frame the question as being about whether we should allow AI aid in cyber-security, but that's not the question. This technology isn't going away. The bad guys will have it. Nations will have it. Again, the question is, can regular people also continue to have it?
parable
4 days ago
As someone who has helped to secure critical national infrastructure several years ago, I can say that it was a mess to disclose the vulnerabilities I identified, and in some cases it took years to patch them. Some of these vulnerabilities were RCEs and mass PII leaks. It's simply not worth the risk for most people - who aren't in the USA - to bother doing this sort of security research. However, if you are in the USA, there are provisions in the CFAA that enable good-faith security research to take place (https://www.justice.gov/archives/opa/pr/department-justice-a...), so to the best of my understanding, you're able to do good-faith research without being charged or prosecuted, and you should take advantage of this if you have the skills or models to do so. I am not based in the USA, and there are no laws in my country that guarantee me zero jail time for simply trying to do the right thing, so I am not incentivized to continue finding vulnerabilities in my country's infrastructure.
Edit: If other countries don't catch up and include similar provisions in their laws, they'll simply fall behind and put their citizens at risk. I'm sure every country has individuals willing to help that currently feel unsafe - look at the large amount of bug bounty hunters worldwide who, I'm sure, wouldn't mind practicing their skills by securing their country's infrastructure.
Buttons840
4 days ago
There was a guy in Missouri who pressed F12 (view source) and saw that the state website contained the SSN of all teacher--just right there in plain text in the HTML.
He reported it responsibly and then the governor personally started attacking him and threatening him with a felony.
I checked your link and I see that is a 2022 DoJ policy. How much peace of mind do you think a DoJ policy (not a law) from 2022 gives me?
I read as far as the first sentence: "The policy for the first time directs that good-faith security research should not be charged."
Soooo... the underlying message here is that good-faith security research is technically against the law, but their policy is not to press charges. They clearly consider good-faith security research to be technically against the law, because it wouldn't make sense to say "for the first time, our policy is to NOT charge people who follow the law"--that makes no sense. What they're saying is "we could charge good-faith security researches, but let's not as a matter of policy".
Reading further you see the policy is about NOT prosecuting people who do something like use an alias as their account name. That same paragraph says their focus should be on people who access systems without permission.
And to be clear, I am advocating for red teams that operate without permission. The bad guys are attacking our systems without permission, and I think the good guys should be protected when they do the same for a good purpose.
chias
4 days ago
IIRC they were part of a base64 encoded blob. IIRC base64 decoding was also referred to as an "advanced hacking technique". But your point very much stands.
mooreds
2 days ago
> There was a guy in Missouri who pressed F12 (view source) and saw that the state website contained the SSN of all teacher--just right there in plain text in the HTML.
Here's more detail if anyone wants it: https://www.malwarebytes.com/blog/news/2022/02/journalist-wo...
watersb
3 days ago
> I'd feel a lot better if I knew some red team security researchers were going hard trying to break into critical national infrastructure
Decades ago, I met someone at a conference who was red-teaming a NATO country's infrastructure: power plants, water treatment etc.
Said that things had improved dramatically since 1990s nethack days.
But the team had a secret weapon: a young woman in a short skirt with a laptop computer, pretending to be a systems software vendor. The Weapon had been able to compromise every site, had never failed. Never.
mike_d
4 days ago
> I'd feel a lot better if I knew some red team security researchers were going hard trying to break into critical national infrastructure
They are. Most critical infrastructure providers worldwide have in house or contract red teams. Every western government has an agency dedicated to cyber defense and they all have red team programs.
embedding-shape
4 days ago
I can't remember if it was the FBI or NSA or the exact details, but in the beginning of this year, they did runs a bunch of commands against compromised routers to "secure" them, which effectively would count as "hacking routers" were they not whitehackers. I think there was some huge Russian MITM campaign they disrupted by effectively hacking people's routers.
cpa
4 days ago
It is absolutely happening, paid by your taxpayer dollars (if you’re American).
Most major countries have this kind of programs. Those could probably be bigger and they will always miss obvious stuff but they exist, at least for CNI.
AdamN
4 days ago
I'm sure this is happening but I also wonder if ironically those systems are hardened by adversaries. I could imagine nation state actors, once they're in, hardening the systems themselves so they can keep their presence and exclude others.
Using that access is the kind of thing that is only done when there's a need and in the case of really critical stuff that can be years in the future - or never.
One could even imagine hyper-critical exploits being shared back to the host. For instance if Russian agents got access to critical nuclear systems (or vice versa), I could imagine leadership warning the other side and getting out of the system entirely in order to maintain the status quo at the expense of pretenders.
rlt
4 days ago
IMHO our best hope is some benevolent grey hat or government organization unleashes a well-aligned AI botnet to patch everything it can find before the black hats do.
aslilac
4 days ago
imagine if we just turned off 90% of the non-critical internet for two weeks, hardened up in an absolute mad dash where everyone has access to top tier cyber models, and then afterwards we go back to business as usual. it’d be a disaster short-term but long-term would it be worse than the years of slow drip we’re about to go through? idk! I kinda think it’d be interesting to find out!
AlecSchueler
4 days ago
Imagine if everyone had just stayed at home for a week in 2020.
jtolmar
4 days ago
It'd take more than a week, but even now, if the whole world coordinated a simultaneous quarantine, we'd eliminate covid, the flu, the common cold, and a thousand nameless respiratory cruds.
mmwelt
4 days ago
Everyone would be very difficult. You'd still need some people to make sure essential services are still functioning, and that still involves quite a large number of people.
fyredge
4 days ago
That was my first thought as well. With COVID, we recognised that WFH was viable for white collar work. Perhaps a global 2 week cleanse will allow everyone to re-evaluate life priorities outside of the internet.
I see hope in what happened during the blackout in Spain/Portugal.
mentalgear
4 days ago
what happened there ?
fyredge
4 days ago
Sorry about that, you can see this for reference: https://www.bbc.com/news/articles/c70zg3e5ej2o
But essentially, with the power down, people came out as a community and life went on without technology. Granted, the power came back within a day, but I have confidence that they would have found a way.
okeuro49
4 days ago
Don't you think it is odd that the BBC led with this feature, for what would have been a critical failure of infrastructure for a Western nation?
Losing electricity causes significant disruption, emergency generators have to turn on for hospitals, the transport system stops.
The BBC is not what it once was, so assume it is no better than any other outlet in wanting to create a narrative.
The actual reality on the ground for many people would have been far from the whimsical event the BBC portrays.
fyredge
4 days ago
I was not under the impression that it was a feature. I just googled the blackout and copied the first link to provide context. Could be a Spanish blog for all I care.
Emergency blackouts are a disruption for sure and I believe people were injured in that event, but the lesson I took from it is that nothing is more reliable than community. Technology comes and goes, but we will always have each other.
AlecSchueler
4 days ago
I know several people who were there at the time—both Spaniards and tourists—and I've only heard positive experiences.
> emergency generators have to turn on for hospitals
So, they still have power.
> the transport system stops.
And people have to accept being where they are or find other ways of getting around.
bertylicious
4 days ago
I think they are referring to the reports of people suddenly meeting on the streets and hanging out together back then.
fodkodrasz
4 days ago
Nothing, I guess, and that is a good thing, I guess: the world didn't even by a day (few days?) without electricity.
AspireOne
4 days ago
Regular people are often bad guys too, you know. If the reality was "bad guys will have it. Should regular people too?" The entire core dilemma would be entirely eliminated lol.
thatsabadlook
4 days ago
AI is actually a big reason why there are so many vulnerabilities over the past year and a half. People don't even know what their code bases look like anymore. It's really bad so sayeth my friends doing consulting.
A lot of companies cyber security posture is asking Claude to "find all security issues make no mistakes".
lxgr
4 days ago
That's certainly one reason, but frontier LLMs are also really good at finding existing vulnerabilities in completely human-authored codebases.
dannyw
4 days ago
That’s not backed by data at all. We’re discovering more existing vulnerabilities.
thatsabadlook
4 days ago
It's both.
synarchefriend
4 days ago
I would expect code written by the latest frontier models with a "find all security issues make no mistakes" pass to be more secure than average human code written in the pre-AI era (not a high bar).
thatsabadlook
4 days ago
A lot of people expect that. Apparently it's not working out so well in reality. My buddy who is big in this said something to the effect of, when a human codebase has human understanding it may have issues but it's strengthened by ai. When an ai codebase has little human understanding it's not strengthened by people very much. So his take was the most secure code bases he is seeing right now are human ones which use ai tools to help read and review rather then write. My guess is that's also cultural and industry specific. He is niched into businesses that actually care and usually doesn't take work from cowboy outfits.
cowlevel
4 days ago
I would expect it depends on the language. Less unchecked memcpy, more forgetting to check the user is an admin.
lelanthran
4 days ago
> I would expect code written by the latest frontier models with a "find all security issues make no mistakes" pass to be more secure than average human code written in the pre-AI era (not a high bar).
"If you write code as cleverly as you can, you are, by definition, too stupid to debug it."
Do you disagree with this? Because if you don't, then the rate of bugs/RCEs in a completely AI-written-software world will be exactly the same as the rate of bugs in a completely human-written-software world.
A combination of human-authorship + AI-review, with absolutely no AI-written code is going to have the lowest rate of bugs, because there is always one entity who understands the code (because they wrote it) and one entity who is smarter than the author who can find bugs.
The AI-generated code I see from SOTA models is absolute spaghetti, and they start running into problems spotting bugs after even a small number of refinements/changes to the software (say, 50).
The problem with all this spaghetti is that they are quick to write something that even they can't easily read. Even mediocre but experienced humans are not so eager to write something that they can't easily debug.
IOW, for any non-trivial problem, with AI-generated code you are guaranteed to get spaghetti that it cannot understand. With humans, it's only a probability that that will happen.
ModernMech
4 days ago
Well the AI machines don't have to rely on cleverness to find vulnerabilities, so maybe they're not limited by Kernighan's Law. Does throwing virtually unlimited sufficiently clever monkeys at spaghetti code find these vulnerabilities better than humans? Can you prove they can't?
lelanthran
4 days ago
At the point you starts asking to prove a negative, like "Can you prove that there's no God?", you should have the self-awareness to realise that no argument will convince you, because you are proceeding on the power of belief and belief alone.
If you don't have the self-awareness to realise that you are asking the same question as "Can you prove there's no God?", then I'm afraid no proof will ever be sufficient for you.
ModernMech
4 days ago
We're not talking about God, we're talking about a machine, here on Earth, we can poke and prod and explore and test. It's perfectly within the realm of logic to prove it can't do things. Can you prove the AI machine cannot solve the halting problem for arbitrary programs? Yes, that's a negative you can prove, so your objection about proving negatives needs qualification.
Asking you to justify a limitation you asserted doesn't mean I cannot be convinced. You're asserting Kernighan's law holds for AI, I'm saying there are other ways to debug than applications of cleverness. Does throwing more attempts at that spaghetti find vulnerabilities even when the thing that wrote it can't easily understand it? I'm not saying AI must be able to do it because you can't prove it can't. I'm asking what supports your claim that it can't.
Melatonic
4 days ago
Don't worry - Siri is on it - that's actually why they've been so lacklustre. Just too busy hacking for the good guys