Yeah I personally don’t understand the point of AI code review tools all that much, as AI is already generating the code as well. All of these AI code review tools create so much noise, yet don’t catch the really important things.

They do catch important things but it’s really contextual. You can’t grab a model slap it on top and say code review . Hence a dedicated review tool is almost dead . Code review should be part of your pipeline and consume test results from the original task , open spec etc . If you do not have that code review will not help if you do , what is the point of task rabbit just slap <your harness in the sandbox> review against <goal>

It does add lot of noise after a point you start ignoring the suggestions and findings.

Code generated these days with fable and sol are near perfect. What issues they might have is logical errors.

throw his garbage out, the time and effort taken to review that is magnitudes more than what it took to prompt it.

have him start with an overall design doc if his change is 15k, it's definitely worth a design doc.

and then have his contributions reviewed in pieces of 200-300 LoC PRs.

any other solution is trading stability and system knowledge, that's 15k LoC no one is truly familiar with, even if you do try to review it

We try to avoid reviewing AI-generated code and built our own testing framework and platform to make that possible.

Our principle is that our tests should give us enough confidence to not have to look at the code (which ends up being true for most changes we make). The core thing that makes this possible is that we run our entire code and infra (including fakes for external dependencies) in isolated, forkable environments and write tests against that, so they are as E2E as can possibly be.

The problem then shifts from reviewing code to reviewing tests and that's why we built our own platform. We have a UI that can diff tests, so we know what changed, and a visual way to inspect what the tests actually did. A test could drive a browser like a user would, and in our UI we get a replay of that browser interaction to look at. The browser is talking to a real version of our backend, and the tests can perform assertions against the database and fakes and really anything in our system.

The answer to this is gonna vary wildly depending on what kind of codebase it is.

A large, mature codebase that predates LLM’s and for which changes need a high level of scrutiny regardless of who made them (think llvm, WebKit, important foundational software), you’re going to want humans in the loop as much as ever… I think reviewing LLM output is the most important thing a human can provide.

But for vibe coded apps where you can just one-shot another one if anything goes wrong? Just vibe the reviews too, who cares. Let the robots review the robots.

Be careful with doing AI review if your codebase is in the former category. Or your codebase will quickly turn into the latter. Complete with “you can just one-shot another one”, because if nobody understands the code any more, there’s not much lost by just you (or your competitors, etc) replacing it wholesale with an AI-written alternative.

I struggle with this a lot. 2 years ago we had a half dozen PR’s a day with a lot of careful review, and now there’s more like 30 of them per day and most people are just rubber stamping them after the AI reviews it. I’m still fighting the good fight trying to review every line of the PR’s I have time to look at, but that constitutes maybe 10% of them. Not only am I barely making a dent, but it’s awkward when I post nitpicks like “this function should go in this module”, etc, the author usually looks at me funny like “why are you even reading this”. Our codebase is gradually becoming more and more vibe coded, and it’s depressing me.

Invest in having a good test suite that validates the functionality introduced by that code. Also AI can review code in an adversarial way and apply those fixes (that ideally will keep the previous tests you did on green)

Code review is soon to be an outmoded concept, (un)fortunately. You have to design orthogonal code (e.g. independent modules in a modular monolith, or microservices) and soak test using canaries.

I have AI confirm the logic works as expected, but review for system design.

Often in both web/backend I’ve found AI to produce overly duplicative code, or have aspects that could be hard to maintain. Generally less due to the AI, and more because of the prompt itself.

That and even if you’re going to AI slop it up, I’d still demand it be broken up into 1-2k LOC chunks or per meaningful “thing”. This also lets us gradually ramp the change to confirm it actually works earlier on

Generally speaking 'the Fable/Astra built GTA 6' videos are a new phenomenon, so it's clear these models have new capabilities and people will need new ways of interacting with them if they want to leverage these imo.

Astra and Sol are the same price when you factor in token efficiency

That likely won’t change if other competitors don’t take the lead at some point. If companies are willing to pay top dollar for the best models AND they get to extract as much money from Chinese labs distilling Astra/Fable it makes no sense to lower prices. Obviously not great for everyday users who don’t have unlimited money.

That should be expected based on the scaling laws that we expect; larger models are more intelligent and cost more. Now it's very unfortunately that they don't publish the size of their models.

Roughly how we price (high skilled) human labor.

Interesting comment because it is true that Astra is costlier for the same intelligence tasks as Sol.

But this is not the same for Fable at all.

Same experience here, but I have some strange feeling.

Sol I'm used to working a month ago doesn't feel the same I'm using today, slower and less accurate. My gut feeling is that they quantize previous models to prioritize new ones and, who nows, make the new one look better.

Up to July I was using mostly anthropic models and the feeling was the same, so much so that I was able to predict every model release 1 or 2 days before public announcements.

Looks like a shameless OpenAI plug

It's a bigger model, of course it's slower.

fwiw I found Astra to be faster than Sol w/ both on medium reasoning for simple agentic coding

difficult to compare though because for more open ended, complex tasks Sol might miss something that Astra notices and then Sol might yield a cheaper but worse outcome

I heavily A/B tested Opus vs Sol for two weeks, giving Claude Code and Codex the same tasks and comparing the results. In my experience Sol is much closer to Opus than to Fable, with Opus often beating Sol. The only area where Sol is better is code reviews, which these benchmarks confirm. I wish they included Fable.