I've been using https://github.com/choplin/code-review.nvim, which looks like a similar UI, but in the NeoVim interface. `<leader>rc` to comment on a line/selection, then `<leader>ry` to yank all comments into the clipboard to be pasted into a chat.

It leaves the comments as markdown files in ./.code-review, so I also have my `/review` agent set to output in the same format, so an LLM can be reviewing the same code I am, I can edit or dismiss the LLM's reviews, then send the whole thing back to the first agent to fix.

I wonder if a custom UI that auto-layouts different files in a 2D grid and connects them to each other in a graph in an intuitive way would help a lot with review velocity.

As in, if you have a large screen, a particularly-trained/prompted AI can organize the code changes in a "flowchart" with floating windows you can easily follow

Maybe in this UI, each code piece also comes with a summary from an agent that has already auto-reviewed the whole PR and creates a basic summary (instructed to be neutral but surface issues if it finds any).

Same experience here - I built a similar tool, for reviewing both plans and code - https://crit.md (shameless plug), browser based as opposed to TUI.

Having said that, I don't review the code until going through a few iterations of reviews from Claude. Each round it does find some "obvious" issue, so as long as I'm not close to maxing my subscription for the week I let it run an audit -> validator checks the claims -> fix issues before I get to it.

I was thinking for years about doing something like this. Thank you for linking this. Would be nice if it allowed to "reject" or mark a change to fix later, but honestly when it would need to be linked to some tracking tool and it would be overkill.

The video actually convinced me that this might be an interesting tool. I'm going to try it myself for a small one-shot project and see how well it performs.

TUI-based reviews on it's own are already interesting. I had never considered it, I guess.

that's a good addition to fresh editor (also tui) and both rust

Meanwhile here's what it looks like - just ran it on my own other repo: https://github.com/adamjgmiller/worktreehq/pull/145#issuecom...

Quite a huge one, but here's one :) https://github.com/Vija02/TheOpenPresenter/pull/170

No not at all! I haven't used it to review itself actually, for the most part, because adamsreview is mostly english language, not much code.

Here's a comment from adamsreview, but even this was 3 weeks ago, and I've worked on it a lot since then: https://github.com/cdinnison/ray-finance/pull/8#issuecomment...

I'll try to find a good public PR to review sometime soon so I can share that and add it to the Readme. This is really good feedback. I should have had something like this ready before posting to Show HN.

Here's an example: https://github.com/adamjgmiller/worktreehq/pull/145#issuecom...

On small PRs (small features / changes ~hundreds to thousands of lines), I'd say around 500,000 total tokens.

On large PRs (new feature sets for apps ~10,000-30,000 lines), around 2-3 million total tokens.

By the way, I should have mentioned in my original post, adamsreview counts tokens used by sub-agents across the stages, and tells you at the end of each stage the total used so far.

That’s probably more work than the entire repo itself. Would need to be something like SWE-bench with and without “adamsreview”.

You’re right though, but evals are actually fairly tricky to write and maintain.

That's a great idea. I had trouble finding anything like this, a benchmark made for (AI) code reviewers.

I had expected to find something like an eval harness available on GitHub, but couldn't find it.

Any suggestions? Or maybe we/I/someone should build something like this?

I suppose one challenge is that if it's going to be publicly available, it would also be easy to cheat, but still seems it would be useful if people agreed it's a good benchmark and could easily re-test tools themselves.

I was thinking about building a GitHub repo made for evaluating Code Reviews. Something like a complex app (or perhaps a few branches with different options), and then PRs on each branch with varying types and degrees of bugs for a Code Review to find.

I suppose this would not be a 'real' benchmark because it would be public and so you couldn't necessarily trust scores people share about how their own tool did, but it would at least allow anyone to try out code review tools on their own and report relative effectiveness and characteristics.

I'll post again if I end up finding or building something like that. I couldn't find anything when I looked previously.

I'll also keep in mind your question as I continue testing this, because you are right that it would be useful to be able to describe what is different, not just the magnitude of bugs found.

Yes, being comprehensive, so early or blatant cheapo findings do not distract from other ones. That's important for base results. Splitting in both file and task is (currently) important.

Additionally, we run in a loop until it stops finding things, and as part of that, do test amplification when it does find any. We regularly see 3-8 rounds yielding valid results.

IMO half the value is customization to your repo, so copying these and specializing to your repo is super quick and pays off almost immediately . How to find style guides, how to run tests, what dimensions of correctness to look for, etc.

We do a similar look here: https://github.com/graphistry/pygraphistry/blob/master/agent...

This kind of thing makes me question how important Mythos is for security bug finding - doing a High effort loop with a frontier model in code reviews until convergence has already outperformed human review for us . (Doesn't replace, but does find things we miss, and catches many we do see earlier).

Hey thanks for the comment and the question.

I would say my workflow for any meaningful amount of work is (all in Claude Code):

- PRD: I discuss and brainstorm with Claude Code using something like the Grill Me skill https://github.com/mattpocock/skills/tree/main/skills/produc... but that I've modified a bit for my own style, until I have a good PRD (what the goals / design decisions are for what I'm building)

--- I run this PRD through multiple AI reviews (sometimes ChatGPT Pro for really important PRDs, because it seems to have some of the best critical feedback)

--- I read the PRD myself in detail before finalizing.

- PLAN: I have Claude Code develop the plan for implementing the PRD. Again, I have this reviewed several times by CC and sometimes by other tools for effectiveness, consistency with the PRD, consistency with the codebase, and internal consisenty.

- EXECUTE: I have an orchestration command I made that has CC execute the PLAN and use a build journal, using sub-agents whenever possible to save context, so that it can operate for up to several hours.

- QUICK REVIEWS: I have these commands /review-fix-loop /quick-dual-review which loops around running a Claude+Codex sub agents review and then fixing anything critical (deferring items needing human judgment)

- CODE REVIEWS: This is when I run between one and several of the adamsreview reviews, starting with /review --ensemble, then /walkthrough, then /fix; until I am satisfied.

Would it be useful if I packaged all this stuff into a GH repo to share with you and others?

Well, it doesn’t give you an opinion on whether you should merge or not. It gives you a list of issues along with details about those issues, such as fix hints, and whether you need human attention before fixing.

It’s all given to you in a structured file, in your chat, and as a nicely formatted PR comment.

Still up to the human to decide whether and when to merge based on the output!

Definitely!

adamsreview is mostly english language - since it is instructing your CC agents. While there are a number of python scripts and JSON storage, you - and your agent - would easily be able to add your own rules to this. It also respects your Claude.md and one of the lenses it already uses is checking for Claude.md compliance etc.

This has been the gradual progression of the software world for the past half century, so it’s apt to use LLMs to fight LLMs and call it progress.

I wish I was kidding…

You've touched on the core issue... prompt based orchestration is non-idempotent. Same input, different orchestration paths every run. I've had better experience enforcing tool restrictions in discrete phases with deterministic code over orchestrating via LLM. Prompt level instructions get ignored, it's much harder to ignore a protocol-level tool restriction. It's simple and yet oddly VERY effective.

That's awesome to hear and I'd love to see it when you're ready.

I actually think having something like adamsreview orchestrated by deterministic code - instead of simply having AI agents use deterministic code occasionally as this app does - could be even better!

The problem I ran into is that if you build a deterministic app that happens to use LLMs instead of the other way around, I don't think there's any way to get it to use your Claude Code subscription credits. It has to use API. And something like adamsreview would end up being so expensive if not subsidized by Anthropic along with the rest of our CC usage.

Curious to hear about your experience.

It burns a lot of tokens, that's for sure.

Friction - maybe? Depending on what you mean.

But it's extremely useful and effective compared to everything else out there that I've tried, if you're looking for an AI code review. Let me know if you try it - or find anything else that might work too without the bazillion prompts :)

Do you mean that, even though the tooling keeps getting better, people aren't putting the effort into using it?

You forgot “to use Claude to write a HN post to promote..”

s/Claude/Intern/g