melvinroest
4 days ago
Wow, this announcement is good content marketing.
Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't.
It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities.
> We then ran AISLE's autonomous AI system against curl.
They don't even mention what models the use under the hood. It wouldn't surprise me if they are from Anthropic and OpenAI.
tux3
4 days ago
The homepage says something about AI guided fuzzing based on libfuzzer or AFL. Looks like they have the LLMs identify a bunch of interesting functions to test, generate some test harnesses, and then sort through the fuzzer findings at a high level, which sounds like a pretty good idea.
melvinroest
4 days ago
Thanks for figuring that out. Sort of sounds like AI programming programs to find vulnerabilities, of which fuzzing is one of the proven techniques to do it.
bitwize
4 days ago
Back when I first heard of it, the HuggingFace hack reminded me about how you don't have to be particularly smart to find vulns, just aggressive in looking for them. AFL on its own "learned" how to construct well-formed JPEGs and PDFs by fuzzing decoders for these formats, back in the pre-LLM dark ages.
Of course the HuggingFace hack proved to be much more than that...
meander_water
3 days ago
It's an effective approach. Google project zero started doing this in 2024
https://security.googleblog.com/2024/11/leveling-up-fuzzing-...
bch
4 days ago
Also sounds incredibly compute intensive.
matherial
4 days ago
Setting a swarm of agents loose for hours to look for software vulnerabilities is far more compute-expensive than fuzzing. The industry has never thrown this kind of compute resources at pure fuzzing, in part because you can't get much VC money for that.
rcxdude
4 days ago
You also quickly get into diminishing returns with fuzzing. Generally a bug is either going to be found relatively quickly with a given fuzzing approach or it's going to be nearly impossible to find. You're usually better coming up with more intelligent fuzzing approaches than you are just dumping compute into it.
lukeschlather
4 days ago
This sounds like a swarm of agents with particular prompting that happens to guide the LLMs toward doing a lot of fuzzing, so it's not either/or; you're getting all the compute requirements of both.
zamadatix
4 days ago
Fuzzing or having the LLM sort through where might be most useful to fuzz & sorting the results? Neither seem particularly compute intensive to me, fuzzing is a pretty standard step and having the LLM read through to find the most interesting areas to fuzz sounds a lot more efficient than leaving the whole task to the LLM.
syou1024
4 days ago
[dead]
whizzter
4 days ago
Their system can run with various models, they go into more details in this article.
https://aisle.com/blog/system-over-model-zero-day-discovery-...
wky
4 days ago
It wouldn’t surprise me if AISLE uses many different providers’ models, and what’s holding back OpenAI and Anthropic is only using first-party models. Just because OpenAI and Anthropic have arguably the strongest models overall doesn’t mean their models are the strongest at finding any given class of vulnerability or lead to follow.
alephnerd
3 days ago
> Wow, this announcement is good content marketing.
Why do you think companies hire PMMs?
1970-01-01
4 days ago
>All six are rated Low severity
This says it all. Nothing important was missing. This is marketing hype.
drdrd
4 days ago
> what models the use under the hood
Presumably their own, wouldn’t they?
melvinroest
4 days ago
You mean their own trained models, or do you think it's an open source model that they fine-tuned? If they use their own, I'd guess it's the latter.
catlifeonmars
4 days ago
Maybe the model doesn’t matter, maybe you just need something minimally intelligent to seed the fuzzer, generate a test case, and rinse and repeat when the fuzzer gets stuck.
vorticalbox
4 days ago
Default to gpt 5.4 nano
https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...
vorticalbox
4 days ago
It defaults to gpt5.4 nano
https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...
bryanlarsen
4 days ago
A repo named "nano-analyzer" unsurprisingly uses gpt5.4 nano. I doubt their "pay them money" version uses nano.
grumpy-swe-9000
4 days ago
I am pretty sure that the nano-analyzer is just a limited open source demo of their "System over Model" thesis from https://aisle.com/blog/system-over-model-zero-day-discovery-..., not the main product.