jnwatson
7 hours ago
The problem is that "cybersecurity" isn't some special task that only your security team does.
In the project I maintain, I find bugs and fix bugs. Some of those bugs might result in an LPE. I generate a regression test, then I fix the bug.
The problem is that generating a regression test for that type of bug is technically a PoC. I can almost never get Fable to create one. Sometimes Opus 5 punts as well. Same with Sol and Luna.
That is, unless I socially engineer the model. I can't talk about security. I make sure they don't read the file call cve_test.c (literal regression tests for CVEs). I have to hide part of my project from the models for them to work.
Anthropic and OpenAI are driving me to use other models.
csswizardry
5 hours ago
I’m a web performance engineer — I help clients find and fix site speed issues — but recently I spotted what I thought might be a security/privacy issue. Security not being my specialism, I asked Fable to help me triage and, if necessary, raise the issue with my client.
It refused. It’s so so so adjacent to the work we’d already been doing, but the moment I asked it to help me understand what I thought I’d found, it left me high and dry!
SparkyMcUnicorn
5 hours ago
I run into this several times per week.
Sometimes the context where Fable drops to Opus 4.8 is enough for Opus 4.8 to do a first pass on everything, then I start a new session and Fable will happily refactor/improve it.
Super annoying.
igetspam
5 hours ago
Every one of their models has become absolutely useless, for validation of bugs and the remediations. Unless I’m doing straight forward dev work, I’ve turned to alternate models and harnesses I’ve started building on my own. The frontier models have apparently become so good at security that they can’t be bothered to discuss it with laymen…
sscaryterry
7 hours ago
Agree 100%. This is just another level of obscurity. Security through obscurity... Its annoying, very annoying.
MeetingsBrowser
6 hours ago
How does security through obscurity apply here?
sscaryterry
6 hours ago
If you've worked with Codex/CC, you'd have seen it degrade from Fable to Opus, or just not done what you've asked it for.
There are techniques and ways around it. For example, I've found that disabling auto mode in CC sometimes helps (anecdotal, YMMV).
This is all for "security". You can always make it do what you want, it just gets really, really laborious.