Haha, all the software devs who hate writing documentation are naturally finding their preexisting beliefs reinforced when the LLM is able to discern intent without docs. An LLM can be spooky impressive at reading minimized or obfuscated code, for example.

But this article argues that LLMs do better when the context is smaller — when it can understand the totality of the task with as little context as possible. And so having correct API-level docs is greatly advantageous. Anecdotally, this rings true to me — when the local context is good and clear, the LLM writes code matching my intent even when my prompt is sloppy and poorly specified.

Rejoice! The LLM will write the docs for you, relieving you of most of the work.

However without intervention, it will do too much and record absurdly verbose docs (similar to how an LLM will relentlessly refactor your code until you instruct it to move in minimal, incremental changesets). You will still need to edit down what the LLM generates.

In general, as LLMs get smarter, “less is more” becomes increasingly true. You’re polluting their context with dozens or hundreds of instructions, all of which the LLM tries to satisfy. However,

    You don’t need documentation […or…] memory

    The code IS the documentation

Trivial example: LLMs struggled with QA on our app. They didn’t know how to find the seeded test accounts. They would create new ones and do it wrong, or find the seeded test accounts but not know their passwords because they were encrypted, so they’d change the passwords but not tell the other agents. Shitloads of tokens burned. It was a no-brainer to just give them the credentials in a skill that gets loaded when they do QA.

You could say that’s an environment issue, not a code issue. But as far as actual code IME at a minimum we have to tell the agents our general repo structure and architecture patterns.

The days of “you are the world’s greatest Python programmer, write good code” or whatever are over (if that style of prompting even worked in the first place) and as I said less is more. But, still….

Code as documentation works better when the code is declarative or a DSL. These capture intent and promises (as in promise theory) better.

When it is not, it has to be reasoned out and does not work well for documentation.

Other things that code and tests alone do not capture well:

- promises (as in Promise Theory) made to other parties. Claude already has PT in its training data.

- Constraints-inducing-properites, as in Roy Fielding / Christopher Alexander. While tests, and property testing can capture properties, there is no formal connection to the constraints that induces fhem. By constraints, I am not talking about business requirements and business value — those are better understood through Promise Theory. I am talking about things like at-least-once delivery or total ordering (from append-only constraint). Claude already has Fielding’s dissertation and Alexander’s works and ideas in its training data.

- grammars, as in pattern panguages (not just patterns) a la Alexander / Fielding are also not captured in code alone. These tell both humans ans AI how to extend a pattern, and how to identify anti-patterns (when they violate a constraint-inducing-property)

- LLMs are trained with many different worldviews and bounded contexts at the same time, and is very capable of translating across it. However, these need to be soelled out, otherwise it would talk in whatever it infers

Specifications written for the exact way components are wited together run into that stale doc problem. Although it takes much more human attention and token burn to describe things in terms of pattern language and promise theory, it becomes easier over time. The actual implementation plan tends to fall out more cleanly when all those other stuff are at least considered. This is where I have been spending most of my time.

> The code IS the documentation

I liked this advice when humans wrote code. Though even then I'd urge people to write meaningful commit messages that capture the "why" of what they did, so no one tramples their intent by mistake.

But not sure it works in an age where most code is LLM-generated. Especially if that code is not even reviewed by humans (irresponsible or not, it's happening), and commit messages are also generated by AI. I think something is needed to separate "what did the human operator intend" from what the agent went and built.

I do agree that this gets way overengineered. My approach has been more or less what you stopped doing though - committing all our timestamped "plan/implementation docs" and "investigation docs" that document what the user wanted + empirical findings, and making all prior session transcripts searchable. It's seemed mostly helpful? For whatever reason I haven't run into many staleness problems so far.

Code typically documents the "what" and "how", not the "why".

Why something exists, and how it connects to the outside world may be documented in comments, but more often than not it isn't.

It’s a weird thing isn’t it, the urge to save these artifacts? The worst is when the llm refers to the decision and design in code comments. In my opinion there’s one thing that is worth documenting; tricky architecture or implementation details that are some how counterintuitive to what would have normally been done. But again this can be documented in the code and tests.

I don't understand. Code doesn't capture the context in which decisions were taken: why is code the way it is? What is important? What is not? How can agents make correct decisions without knowing context that cannot be inferred from code?

I've come to the same conclusion: don't say in markdown what you meant to say in code - spend the time to make the code more clear. The code is always the source of truth, and stale docs (they all get stale) are a constant drag on the LLMs pattern matching capabilities. If you want the LLM to follow certain patterns, you have to make your code base exemplify those patterns, not write about them.

It's frequent for SWEs to make blanket statements with considering the vast space of issues other people face that they don't have experience with or awareness of. Anyway, I'm not going to tell you what you do or don't need, only what worked and didn't for me.

In my codebase it is difficult to get agreement on comments and documentation so rather than rely on it I adapted. One of the first things I did when I succumbed to agentic development was to point codex at the code and ask it to generate a high level description of where important files, such as our public API, reside, what the hierarchy is, what the code does, etc. In my case, this level of documentation is fairly static if I avoid implementation details. So now I have a handful of agent files in my tree and it seems to save quite a few tokens and improve my results. I frequently have other devs ask me how I get such good results when doing agentic reviews of their changes(always my first step now before I start my human review). I also include instructions in the agents files instructing the agent to maintain the agent files if any relevant changes are made. It seems to work quite well for me.

I think for many projects you're right. One niche where I disagree: reverse-engineering. You'll need to carry context on things cannot be derived from code immediately eg: past experiments, findings, decomposed schematics/firmware etc. Without this, models are prone to re-doing the same work continuously - which sometimes involves "expensive" (in terms of wall time) simulations.

Idk I just can't agree. Code is the what but it doesn't tell you the why. There are so many times where at first glance the code seems suboptimal or bad or wrong, and it's only when you learn of some constraint somewhere else that it begins to make sense.

All code is written under constraints, and most constraints live outside the code.

I partially agree, way too many people are cargo culting overly complex AI workflows with little empirical data. My framing is a bit different though, I consider code the spec and actually keep a decent amount of docs for higher level concepts. So far this is working well for me across Claud and Codex.

Code can get much larger than the documentation that summarizes it. It also doesn't cover intent or rationale. Even if you inexplicable don't want documentation, at the very least use something like gitnexus to map out your code because relying on code alone isn't good enough

Versioned documentation is inspectable, reviewable, and easier to correct than opaque recalled snippets. Documentation should describe current truth; append-only events can preserve history.

Yup. Examples examples examples. All of the descriptive stuff is just nonsense that confuses the point. Makes perfect sense when you remember that these things are not intelligent, but truly just autocomplete on steroids.

>> You don't need documentation or the 3rd party memory systems. The code IS the documentation.

We have heard this nonsense from the "we don't need to write comments, code should be self-documenting" types for decades. It was wrong in that context, and it is wrong in this one.

Code tells you how a system works. It does not tell you why it works that way. That is what memory is for. It exists so that your AI does not keep undoing past decisions when it writes or refactors code.

I went through the same cycle as well.

I think it's going to be an incredibly common, maybe universal cycle people will go through working with AI until they realize it doesn't work long term.

What about big projects, where much of the code is not written yet?

[flagged]

[flagged]

[flagged]

What I wish more people would be talking about is that RAG should be considered harmful.

When you have knowledge distributed in markdown files; finding them puts the path/filename into context as well as some indication of document size. (If its on line 1200 or line 20). This is extremely valuable for picking what ought to be focused on next.

RAG on the other hand creates the hardest challenge for these models. It instead puts 5 ideas with the highest similarity into the context in full.

Its the difference between having to remember a set of numbers when in a crowd that's talking about stuff, and having to remember them when the crowd is shouting out random numbers. The similarity in the task makes things harder. SoTA models work despite this, but its extra-gambling while you're already gambling.

I think their solution is still sub-optimal. Ideally a secondary agent would pre-process the prompt, select relevant information from the catalogue, then pass that on to the primary agent.

This way the primary agent only has relevant information in their context to make decisions and take actions.

Context management is still under valued imo.

> A markdown "brain" has the same problem. How does the "agent" know which documents are relevant before it starts?

I don't know, but I have a pretty standard (I think?) setup, and Claude manages to find every relevant file every time. But I've also only used Claude for greenfield projects, where "documentation is primary, and code flows from documentation" is the philosophy.

I have CLAUDE.md describe all the types of documentation files and the directory structure. And then Claude is pretty aggressive (automatically) about always inserting cross-references everywhere. So a feature description will reference the ADR's that it implements, the ADR's say what feature implements them. A code file will make reference to the "implementation design" document that describes the motivation behind which iOS elements were chosen, how the animation is defined in a particular way that doesn't break another animation, and so forth. So I've really never run into a situation where Claude failed to read a file it should have. I've been pleasantly surprised.

I would say that the one really big thing I've had to learn is to teach Claude both in CLAUDE.md and in the header of every top-level design document, that keeping documentation current and in sync is paramount. Because its default seems to be to keep history and append, e.g. by default it will take a section of a document and mark it "[DEPRECATED]" and add the new version below. So my instructions are pretty clear in having it always be aggressive in maintaining current state only, always replace rather than append. And if there's anything we want to save from the previous approach (e.g. we did it X way previously and it failed because Y), then just add that as a new short note in the new current-state text, possibly with a pointer to a commit or tag or something.

So this seems to solve both recall and staleness in my projects at least.

The only thing I still haven't found a solution for is numbering. Claude is always giving everything numbers, like F23 for feature 23. But I'm always changing the order of things, inserting new things, deleting things, so I wind up with a sequence of development work that goes in order like "Phase 9", "Phase 9b", "Phase 9e", "Phase 11", "Phase 12". I'm halfway ready to abandon numbers entirely and just start giving things names from noun collections instead, so every feature is named after an animal, every ADR is named after a kitchen implement, or something. Or just four-digit hex codes chosen at random. Curious if anyone else has found what works.

Just reading the title my initial thought was “this is just a strange semantic argument”. But I’ve made those assumptions in the past and been pleasantly surprised.

Not today. IMHO Documentation is just a form of structured memory and it’s all just context. Getting that context right is a hard problem and there’s a lot of different ways to skin that cat.

Second paragraph of this comment seems obviously written by AI.

[flagged]

There is a command in oh-my-pi called "/omfg <problem>". You explain what is wrong with agent's response, and it writes a hook to make sure that the problem doesn't happen again. It then re-runs your previous prompt to make sure that hook is triggered, and if not, it rewrites the hook to make your previous prompt trigger the hook. Then each next agent's response is checked by the hook, and if it is triggered, the agent receives feedback on what's wrong and what must be done differently.

> If I say "use jq instead of writing a python script to parse json" it should never write adhoc python scripts to parse json.

I think there is a deeper problem emerging from this sort of behavior. Even when we bother to create agent skills with there own scripts that call tools like jq a specific way to achieve a goal, AI coding assistants and agents still go way out of their way to generate ad-hoc scripts to do the most absurdly stupid tasks such as parsing output in structured language, and even remove whitespaces from a markdown file. This means AI coding assistants and coding agents treat agent skills as mere suggestions of using a alternative option that more often than not the choose to ignore.

This has a very dangerous implication: your average user is trained to develop a pavlovian reflex to authorize agents to just execute their ad-hoc scripting code with our own permissions and credentials in our systems, which includes the ability to call anything over the internet.

I think it's kind of cute the way it writes Python scripts, but I've never seen it do that when the relevant native tool is on the PATH. It's like the most competent ever intern, on speed. No tool to convert SVG to PNG? No problem, I'll write a Python program to do that!

Treat it like any other software system: rules that must not be violated are enforced by static type-checking or a trusted runtime monitor. There’s no other option.

Claude code setup seems to be using jq all the time when I use it but I did what is written here:

https://gist.github.com/cynthiateeters/6868ca26c059a3106cd93...

Obviously, AI is way more comfortable with using adhoc Python scripts, which are used for everything, than it is with using jq, a niche CLI tool.

You could try https://github.com/ioni-dev/mati the constant changes in reasoning effort in consumer models can break things and its more dangerous for devs that relay to much on agents.

I’m the author of the project.

It's often better to just find ways to embrace what it tries to do naturally. Otherwise you're fighting the weights and hidden prompts

[flagged]

[flagged]

[dead]

I've found that most of LLM generated docs are diluted and unfocused. Spending 3 paragraphs on some quirk of a library, then 2 simple examples of a command.

I write README and docs by hand, thus only important stuff goes in there. If it is not worth my attention to write down, it's not worth writing down.

Smaller context, easier to consume.

I've got basically a loop of docs, test (TDD) and code. Starting with and IMPLEMENT-<plan name>.md. I ask to revise as TDD, then loop.

Once qwen3.8-flash-next showed up, it cañ go "forever" with dynamic context pruning.

It's fascinating for local coding.

> It doesn't appear to have a concept of time or reconciliation (both issues we're having at work). A memory might be relevant for a week or a month, but it might no longer matter after that ie "we're migrating systems so keep <thing> in mind"--that doesn't matter after the migration is complete.

It doesn't even understand when its current context isn't...

See the whole issue of refining a concept, removing a few points or notions, and it will write a paragraph why a point that isn't in the text anymore when no version a human will read ever had that point in it.

[flagged]

[flagged]

[flagged]

Deterministic feedback is precisely how frontier models are trained. It’s called RLVR. You let the agent run on a problem and then calculate a deterministic score of how well it did. Repeat 1000x times and you can “brute force” a good solution. (Which includes all thinking traces and you add it to your training data.) And then a Chinese model can copy your advance for 1000x less compute. Which is why US labs call this not learning, but a distillation “attack”. It’s an attack on the business model.

This looks pretty cool but I'd want to be able to setup a bunch of my own project specific code smells, so it's not just a few generic rules. Is that the approach you've taken?

Then for example you could write your own hook and convert existing code smell documentation which agents ignore into a format that works with the hook.

The readme reads like AI slop. Why would one believe the tools would prevent AI slop?

I suspect strategies like this might be even more powerful with less intelligent agents. Could a 40B model outperform a 400B model with good feedback and instructions?

This sounds interesting but (from the homepage too) I don't understand how it's different than having it run any other linter?

It sounds interesting, but this looks too much like a marketing website (huge text everywhere) and not enough like a documentation website, so it was hard for me to see how it might work.

I've done that too, called it Decisions. Was happy for a while, but then the agent started to make decisions longer and longer, putting "x, not y" in there. It started to write them without asking. Suddenly in code review it would say "violates D-236". Some rule it made up that I never approved. Then I made very clear that it should never touch those files. Still does. Now it's just context bloat.

Timely comment! I'm learning harness design and was introduced to the concept of writing guiding principles. I look forward to trying it. (Great site. I agree with another comment. It does look commercial. Had I not been committed to looking for ways people define principles and rules, I might have skipped it. Glad I didn't.)

Do you have any resources that guided your work?

I wrote a cli tool the agent is instructed to call. The tool collects answers to a few questions from the agent, and responds with a steering command. Sometimes it responds with more questions, and then shows the command.

The questions are designed specifically to identify the state the agent is in, in order to assign its steering. Changing this tool changes how the agent works. You just need to make the use of this tool a necessity so it does not forget to call it.

Unlike principles and memories, questions are more openended and tend sometimes to trigger the right mentality in the agent even before it gets to the policy itself. In my opinion agents get lost in local work losing the big picture, my questions jog the big picture back into attention.

1. call ssp tool, no arg, it just reads features from the repo, like most other tools -> ssp locates state and sends 4-5 questions

2. agent responds the first batch -> state is further narrowed down -> ssp sends a second batch of questions

3. agent responds again to the interview -> state gets finally pinned down -> agent gets the steering assigned

4. a log is created of this ssp session, and reflection on the log used to refine the questions and steerings in the ssp tool (name comes from State-Space Policy)

> Something I started doing recently was writing out principles instead of memories.

It seems you tried to reinvent instruction files. What do you think is the difference between your approach and standard tools such as instruction files, AGENTS.md, and even skills?

I wrote a small utility to record the decisions you made, and the paths you didn't take:

https://github.com/skorokithakis/gnosis

This just does a pass at the end, rather than consume context all the time. I've been using it for months and it works great.

Most of these things are doing just like you said, taking up context.

The code is the documentation. There's no need for most of this stuff.

[flagged]

Do agents actually follow it though? Everything I tried that was not just simple instructions had the same failure mode : ADR, mnemonic, hindsight, code-review-graph, etc were simply ignored by agents after a while. They always fall back to their baseline: grep and Python.

No human reads those md anymore

This is literally how memory works.

And then a random model/reasoning setting has a stroke and pollutes it all with ai slop.

[dead]

I don't trust agents writing specs without human approval.

I've been bitten by agent-written ADRs. Agents carelessly add extrapolated details and speculated nice-to-haves that I never asked for, and this becomes a source of bloat that keeps coming back like a boomerang.

There is a hierarchical distillation of understanding in human processes that agentic processes do not yet execute.

Plug here for a python app I wrote, no dependencies. It uses a skill to maintain an index, links, archival, and works with a separate opinionated dev playbook nicely. The best part is now the docs/specs folder maintains itself. (and I can spawn a new session with all the relevant context every time)

https://github.com/ArtRichards/docs-cli

and

https://artrichards.github.io/agent-playbook-suite/blog/

Okay, so using superpowers. Have you actually looked at the superpowers? They're garbage. That shit was maybe helpful a year ago. It's now just cluttering up space.

I think a lot of people are stuck on patterns that "used to work" (and I use that loosely because I really feel like LLMs are getting work done IN SPITE of using all these additional skills/plugins/etc). Go ahead, delete all that shit and try building without. Yeah, sometimes you may have to hand hold the LLM, but that hand holding is a LOT less than doing whatever this is.

This all just sounds like a rube goldberg machine you've made.

This is a problem when humans are responsible for keeping the docs up-to-date. But if you're doing development with agents, it's the agents' job to keep it up-to-date. And as long as your AGENTS.md instructions are clear about the process for this, I find that it just happens automatically. It takes a few tries to get the instructions right, but then documentation becoming fragmented or out-of-date just stops being a thing. Of course, this relies on all team members using the agent.

"Code is the documentation" doesn't solve the problem in my experience. Because what happens is that you still need a lot of "why" comments in the code, and then these go stale, so you still have the same problem you have to solve. And so I find that markdown documentation is a lot easier to organize and review in a structured hierarchical way in one place, than code comments sprinkled across the repo.

But then, how do you keep the theory in your own mind enough to steer the agents to extend the program? I've been asking around, and different people seem to have different ways, colored by the way they work.

[dead]

Consider tomatoes.

Legally and culinarily, they're vegetables. Botanically they're fruit.

Tomatoes are members of the nightshare family which includes tobacco, potato, and chili peppers.

They are used in popular recipes like Mexican salsa and Italian pasta sauce. Pasta sauce commonly uses the San Marzano variety. Here's a <picture> of San Marzano tomatoes from our Italy trip.

Rhett likes tomatoes in all dishes. Link only likes tomatoes if they're blended in a dish like pasta sauce or tomato soup. Frank is allergic and can't have tomatoes.

Tomato prices are up due to a bad <year> yield in <country>.

---

This is all memory. Where does it go?

I like your approach of keeping things a simple and light-weight as possible, and asking Claude to set up its own environment works remarkably well. It even does a great job of configuring OpenCode if you ask it to ;-)

That said - I personally ended up saving my plans and status in my second brain app via MCP. Having that available to me on the road when I'm in the car brainstorming with Claude over voice is a big win.

Yeah… this makes good points about why memory systems don’t work. Then tells you the solution is… a memory system.

I get what you're saying but it seems presumptions to compare a graph database to how our brains work. And also that LLMs <-> (the way they do memory) is the right analog to humans <-> memory.

I take the article's point more directly. It's just a straighter line to have clear communication through documentation than to fuddle around with the perfect memory setup.

Yeah, I think monorepos are pretty great. Sometimes not possible, but for those situations you just need to have a way for the agent to get access to other codebases (gitlab mcp being what I rely on at work).

I don't think you need more documentation or these memory systems. The code IS the documentation. Lots of this stuff is unnecessary. It's a bunch of people doing their special rain dances and then when it happens to rain they say "I did that"

I have memory turned off for everything, context management is the fundamental technique in working with LLMs, and I absolutely do not want random additional things being added to that context without my knowledge because it results in nonsense like using Claude to do work on the industrial IoT SaaS product I work on professionally and being told I could just use the Home Assistant instance which manages my flat to deal with that problem instead.

Might as will try adding the link again anyway. This version hasn’t yet been reintegrated back into the parent project.

https://github.com/Clumps4Linux/SEP/blob/main/SEP-1.1.1.txt

What if the June conversation is outdated and is no longer applicable by July? Do you have a mechanism for dropping older, subsumed propositions in your database?

Github Copilot had the idea of attaching memory to files, and if the file hash changes the memory is automatically dropped (not sure if they still do it). This means they are overly eager to drop stuff (even if the file change is just cosmetic), but at least they don't accumulate outdated cruft too much. (a memory can still be outdated if it was invalidated by a change in another file though)

What years? Agentic systems, and memory alongside them are roughly only year old.

You’re confusing, I think, memory system with llm finetuning. Completely different concepts.

But reading the doc if it's maintained llmwiki style is much cheaper than reading the code on a large codebase.

I don’t understand your point. You say that tools are the only things that are required. But then you say that the quality of tools don’t matter.

I think what you are trying to say is that “give the LLMs access to information and it can figure out how to use it. Don’t get fancy about how to structure the data”.

If that’s the case it has largely turned out to be true. RAGs have fallen out of fashion.

Where does that put AGENTS.md though? Is it worth spending time to structure it or just dump it.

What do you mean dedicated per type? Like a python-flavored read tool for .py files?

The markdown files, documentation, decision files, (3rd party)memory systems, etc. are all a waste of time and context.

Every single project at work that has documentation/ADRs/etc ends up stale as fuck and it pollutes context more than it helps. Even the agent generated docs. Hell, especially the agent generated docs. It's like the LLMs can't just DELETE something, they always amend. So now there's stale details in there about something we abandoned polluting context.

The code is the documentation, none of this stuff is needed. I'm seeing WAY too many devs/engineers go down the rabbit hole of trying to build these tools and systems but not actually getting meaningful work done.

I've been doing my best to remove and delete all the documentation. I usually keep a super high level AGENTS.md/CLAUDE.md file and that's it. The llm/agent can figure it out. You might argue I waste 5 minutes and a dollar or two for the agent to "figure it out" each time, but I'm pretty certain people are wasting FAR more time and money building these things out and maintaining them in each of their projects (or NOT maintaining them and they're actively BAD for their projects).

You don't. But as was ever the case before AI most humans don't know to or want to have documentation written. So the tools have workarounds to try to reduce the number of "AI sucks because I told it X and it did Y" posts.

Because of old outdated info that gets left behind, causing confusion for future agents.

I just don't get the fear at all. I too have 10 to 50 agents working 24/7, from 5+ different providers. They talk to each other directly using tmux send keys. What is the exact frightening thing here? It is all tokens following tokens.

How do you coordinate their work? Do you just have one orchestrator/mayor that you talk to, and it coordinates the rest, and the rest talk amongst themselves? How do you ensure they're doing the right thing or efficiently?

How, if at all, do you keep the architecture or a working theory of the code in your head?

You don't. When you see it saying it stored something "in memory" you can just ask it to document it in the product documentation somewhere, and in future please keep doing that. Of course the note to keep doing that needs to go somewhere, which is what "memory" is for.

> You have the whole session history right there.

But that's one session. Isn't memory for… the next session?

It's just an optimization. You might not pay it to re-read everything. So it sticks some stuff in a notebook that it always reads. Like inspector Poirot.

> Shifting the focus from bloated contextual memory to structured, verifiable documentation and state boundaries is precisely the right systems-level trade-off.

Isn't memory just documentation that agents create and update on the fly to fill in the documentation hole that your average user leaves open?

There was a time when the need to provide context and signal was a key topic in LLM and AI-assisted coding. People talked about MCPs AGENTS.md and README.md and agent skills and even comments, descriptive tests, and naming conventions. Supposedly the theory was that if you provide context, agents wouldn't misbehave so much. Then plan mode and spec-kit approaches stepped in with approaches aimed at providing that context when starting sessions, because users never bothered with docs or MCPs or AGENTS.md or anything. But then plan mode and spec-kit approaches were considered too laborious and requiring too much from users. Then, because users systematically failed to provide context, this task was finally given to agents. And things just worked.

You should ask yourself why in software engineering circles documentation is seen more as a problem than a solution to any problem.