AI should be used for code review but not in CI.

You should already have 2+ developers looking at most PRs. And these developers should absolutely use AI. The PR author should use AI.

But what you should not do is pipe the AI output directly into the PR and tell the PR author to deal with it. That's adding noise to the PR review process. Everything it says is something the PR author needs to validate as relevant, helpful, etc. A human needs to do that before confronting the author with it.

You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?

IMHO, Codex with Astra/Sol and Claude with Fable/Opus are all any professional programmer should be using in Sep 2026, if they can afford it.

These models are still terrible compared to what we'd actually wish for, but they're the best available.

If you can get away with using the $200/mo subscriptions, it's really not even a money thing for most professionals.

Almost all of my work is now plan, generate, review, plan, generate, review, commit, push.

I'm using Claude or Codex (or both), and they're doing all of the testing "inline" rather than through a CI action, etc.

I only use chinese models for code reviews because you can actually tell them to take an adversarial stance and actively look for security issues without risking refusals. GLM-5.3 has been great for this, although it can be slow on larger PRs.

$0.10 extra per pr review is nothing. What software company is willing to accept worse reviews and less bugs found to save 10 cents?

> The pull requests are public, and the prompts, raw model outputs, judge verdicts, bug-class labels, repeat runs and scoring scripts are committed with this article.

Okay, but where are they? This quote says that PRs are public. Does that mean that everything else is private and we can’t actually reproduce those results?

This, and the fact that the article doesn't mention reasoning levels, sounds fishy to me.

I found Luna and even 5.4-mini to be quite good at code review provided a few things:

1. Run it in multiple cycles, only on the diff, and only emit a few findings at a time.

2. Give it a memory so each cycle, it knows the previous finding to check if it's been fixed.

3. Give it access to canonical docs that encode your human reviewer heuristics. I exposed these as tool calls so they could be tracked via telemetry.

4. Run multiple reviewers, each with a tight focus. Security, performance, structural, database, etc. Each a separate prompt and persona. Additionally, we had file activation filters so the FE React reviewer didn't activate on BE only changes.

Luna and 5.4-mini with no reasoning were exceptionally fast and almost always found issues with code produced by Opus and Fable.

Default prompts for the curious (these are templates deployed by default, but customizable).

Performance: https://github.com/zeeq-ai/zeeq-app/blob/main/src/backend/Ze...

Structural: https://github.com/zeeq-ai/zeeq-app/blob/main/src/backend/Ze...

(Keep in mind each agent also has tools to access and reference external docs.)

They state Luna is good enough, but its accuracy of findings is 74% whereas Astra is 96%. Dealing with false positives is expensive.

I am finding AI doing its own reviews as part of the process to be the key to productivity. I do subagent (fresh context reviews) at multiple stages with well-specified review criteria. It is really expensive to do with OpenAI or Claude API billing. Deepseek or the discounted monthly plans from OpenAI or Claude can be discounted similar to the 28x they state for Luna compared to Astra and you maintain much higher quality.

False positives have a real cost, especially if AI is reading a review. Consider if you have GPT-6 Astra looking at a review and finding a bunch of false positives it burns tokens to figure out.

I was using Copilot Code Review pretty religiously for a while, as I get access for free (the $10 plan) due to my Open Source work, but it recently introduced a monster of a misfeature that caused a massive increase in complexity over time, while I wasn't paying close enough attention to it. Every subsequent model saw that change and the explanation for it in the changelog and assumed it was policy rather than a model being brain-damaged, so it became a fractal of fuckery I had to untangle with a good model and some close human supervision. It was an administrative tool that directly edits configuration files for a service. Copilot code review decided it needed to be an overlay service that applied only the configuration created by our config UI, fully independent of the system service. And, subsequent "bugs" that other LLMs "fixed" were just applying more and more bandaids to that bad decision (attaching the services together so restarting one would restart the other after, etc., making sure there were no conflicts across the files, warnings when one rule conflicted with another, etc.). Because modifying an additional service is simply not what the tool was designed for, it seemed to be really buggy, so there were lots of "fixes". It took me too long to realize the fundamental failure point.

That's my way of saying, I am hesitant to trust a stupid model to do code review because I become complacent and when it suggests a small change that seems reasonable (the LLMs are very good at sounding reasonable, far better at sounding reasonable than being reasonable, in fact), I might not notice that it just did a stupid until much later, when it becomes a big pile of stupids.

My fault for trusting it, of course. But, my eyes glaze over when I read AI prose, whether it's code review or anything else. It's hard to catch one incorrect behavior in a batch of several reasonable suggestions.

This article is missing an incredibly important detail: what is the harness doing?

I get remarkably good results using any recent OpenAI model using the codex-rs harness pointing at a built checkout of the PR. The models use the available tools (i.e. the shell) to understand the repo. I get some false positives and some false negatives, but I don’t believe for a second that I would get comparable performance using a dramatically less capable harness. (Also, the models read the short AGENTS.md for some context as to what’s going on. The prompt I use is about one sentence. I don’t bother with the built in review tool.)

For the mj adversarial review we use a configurable coordinator (I recommend Fable) with six subagents (I recommend Luna) for specific specialties like code complexity. Narrowing down what a smaller model like Luna needs to look for / care about helps them do better work; then the larger model synthesizes and fills in any gaps.

https://blog.brokk.ai/mjolnir-automated-cross-vendor-adversa...

It would have been extremeley nice if the original article had also included Sol and Terra in the testing.

(Also would have been nice if they included the equivalent Anthropic models for comparison as well, but it's not quite as relevant.)

Use cheap models if your code is low value.

I select my review model based on change complexity. If I'm confident that the change is localized (given that I always maintain an up to date and complete mental model of the application), I'll use Luna/Terra. For deeper changes, I use Sol/Astra. Sol is so good for code reviews that I'll only engage Astra in the very riskiest changes.

i think gpt 5.6 luna-max in particular is excellent for non long term agentic use

code reviews, unit tests, docs, whatever you dont want more expensive models working

i think the new muse contributor model is enticing too if you are not using it for private/sensitive stuff

Any model is good enough. Even tiny local models can provide some value and their false positive rate is still relatively low and warrants a proper reply.

In my experience, Luna is not able to find non-trivial impacts from the changes at hand.

I've seen this a few times on relatively simple changes on complex codebases.

Company that sells code review says the cheaper cocaine is good for entry level users, but you need the powerful cocaine for advanced users.

Would you like me to find a herbal formula against cocaine hangover?

For code reviews I believe yea luna is mostly good but for finding security issues something more well versed with it might be more helpful like astra. This also depends on the scale and the impact of the application or the code where it will be used too, I guess

It would be great if all these benchmarks would also provide a slop metric. The main issue I see is not finding bugs but reporting “stupid” bugs and and if the recommendations are followed you end up overengineering the wrong things basically producing AI slop. GPT is prime candidate for this. Fable as well but less than GPT

We're using an even cheaper model (K2.7 right now) and our PRs cost more to review (still < $1). I imagine we have a lot more going on in terms of review, even after moving the context gathering and comment posting responsibilities to scripts that run before/after the agent. It's more of a team, so we also end up with a dozen or so sessions from the subagents used.

We currently think it worth it. The review catches things humans and two paid options miss. It's definitely a wall of text and burnout fodder, the next step is an agent/skill that will make the changes after we humans comment on the comments, because the comment wall is not sustainable.

Definetly. Code Review is no rocket science, but security is another topic.

I have been writing an internal code review tool that is a bit maximalist. I created subagents for many internal domains and technologies we manage, with prompts focused on best practices, common problems, owasp guidelines, etc), a separate tier of wider band subagents (design, rollout, security/privacy), and a final agent at the top orchestrating and combining. I also use adversarial validator passes against all findings.

Right now, Fable 5.1 delivers incredible reviews. Opus 5 delivers good reviews. These agents are finding really impressive issues that humans just don't have the attention span to track down. My reviewer has a very impressive signal to noise ratio at this point, after half a year of iterating and improving. (I use a lot of Opus high, Opus medium for less critical tickets/domains, Fable 5.1 high for critical domains and all of the issue validators, and even Fable 5.1 xhigh for my design agent, whose job it is to think about the project at a high level and provide the kind of high level tech design review that AI notoriously can't do well)

I'm testing Astra so I don't have strong opinions yet. I've also done extensive testing of the same skill and subagent pattern in opencode/omp using GLM 5.3, Kimi K3 max, Deepseek V4 pro, Deepseek V4.1 flash, Qwen 3.8 2.4T max, and others.

My experience is that open weights models find between 1/4 to 1/2 of what Fable/Opus stack can find, and often miss the most critical issues. I work where privacy isn't just good behavior, it's enforced by law, and the Fable/Opus stack has found privacy leaks that the openweights stacks don't find.

You can imagine that paying for these Claude runs isn't cheap, each one can eat 25-33% of my 5 hour limit. I am quite desperate for openweights models to be competitive, but at the end of the day, the biggest limit here isn't the price difference between GLM 5.3 max (my current best-in-class choice for open weights, offering Kimi k3 performance for like half the price), it's the cost to the business for shipping lower quality.

Can't wait to dig in more with Astra, I just haven't iterated much on my skill port to codex yet.

One criticsm I have for the article, that is important for my own work, is not simply comparing "bugs found" because these agents can find endless reams of lows and nitpicks that are just ~worthless hardening. I'd be much more interested to see how many critical/high/medium's each test found, not "overall bug count". I also think review is about A LOT more than "finding bugs"...

They seem to conclude Luna is a better value, but their analysis is dumb. They just break it down to $/bug found.

However, Luna missed 23 bugs that Astra found, and identified 24 bugs that weren't really bugs. That's horrible. Astra had 96% precision.

The cost to care about here isn't just how much it costs to run the code review, or the cost per true-positive. It's the cost of dealing with this system. A code review system that is right about 2/3 sucks, and one that misses another 1/3 of the bugs is also a lot worse. The Astra code review quoted here would become the foundation of how the team works, the Luna version is at best helpful to find some stuff but does not dramatically increase your confidence. It also will force humans or better AI's to have to run down a lot of false positives, and that is treated as free here.

Actual conclusion: The cost for Astra is low in absolute terms compared to the cost of bugs and human attention, and the added value is far far more than the added cost.

Let's assume for a second that the intelligence is there and these models really are great. Would it make sense to hire CTO of some big-tech company to do code reviews for your startup? Feels kind of like overkill to me. Code review is not about more intelligence. To me, it's about more cultural context. And all things equal, what difference would the model make, at anything above sonnet5 medium level?