Yep - this has worked well for me too. I do it a little differently:

I have a /review-sessions command & a "parse-sessions" skill that tells Claude how to parse the session logs from ~/.claude/projects/, then it classifies the issues and proposes new skills, changes to CLAUDE.md, etc. based on what common issues it saw.

I've tried something similar to DISCOVERIES.md (a structured "knowledge base" of assumptions that were proven wrong, things that were tried, etc.) but haven't had luck keeping this from getting filled with obvious things (that the code itself describes) or slightly-incorrect things, or just too large in general.

When you get stuck in a loop it's best to remove all code back to a point it didn't have problems. If you continue debugging in that hammering failure loop you get TONS of random future bugs.

Nothing about this feels inflexible to me at the moment - I'm evolving the way I use these tools on a daily basis, constantly discovering new tricks that work.

Just this morning I found out that I can tell Claude Code how to use my shot-scraper CLI tool to debug JavaScript and it will start doing exactly that:

  you can run javascript against the page using:
  shot-scraper javascript /tmp/output.html \
  'document.body.innerHTML.slice(0, 100)'
  - try that

We don't need automation for that, we "achieve" that through our processes already. Specifically, software creation processes of large teams with many changing developers over long periods. Example (but they are not the only one): https://news.ycombinator.com/item?id=18442941 -- changing or adding anything becomes increasingly burdensome.

I would like to post that every time somebody warns of the dangers of AI for maintainability. We are long past that point, long before AI. Businesses made the conscious decision that it is okay for quality to deteriorate, they'll squeeze profits from it for as long as possible and then they assume something new has already come along anyway. The few business still relying in that technical-debt-heavy product are still offered service, for large fees.

AI is just more of the same. When it becomes too hard to maintain they'll just create a new software product. Pretty much like other things in the material world work too, e.g. housing, or gadgets, or fashion. AI actually supports this even more, if new software can be created faster than old code can be maintained that's quite alright for the money-making oriented people. It is harder to sell maintenance than something new at least once every decade anyway.

With great love to your comment, this has the same vibes as the infamous 2007 Dropbox comment: https://news.ycombinator.com/item?id=9224

I'd also argue that the context for an agent message is not the commit/release for the codebase on which it was run, but often a commit/release that is yet to be set up. So there's a bit of apples-to-oranges in terms of release tagging for the log/trace.

It's a really interesting problem to solve, because you could in theory try to retroactively find which LLM session, potentially from days prior, matches a commit that just hit a central repository. You could automatically connect the LLM session to the PR that incorporated the resulting code.

Though, might this discourage developers from openly iterating with their LLM agent, if there's a panopticon around their whole back-and-forth with the agent?

Someone can, and should, create a plug-and-play system here with the right permission model that empowers everyone, including the Programmer-Archaeologists (to borrow shamelessly from Vernor Vinge) who are brought in to "un-vibe the vibe code" and benefit from understanding the context and evolution.

But I don't think that "just dump it in clickhouse" is a viable solution for most folks out there, even if they have the infrastructure and experience with OTel stacks.

I'm currently experimenting with trying to do this through documentation and project planning. Two core practices I use are a docs/roadmap/ directory with an ordered list of milestone documents and a /docs/retros/ directory with dated retrospectives for each session. I'm considering adding architectural decision records as a dedicated space for documenting how things evolve. The quote fta could be handled by the ADR records if they included notes on alternatives that were tried and why they didn't work as part of the justification for the decision that was made.

The trouble with this quickly becomes finding the right ones to include in the current working session. For milestones and retros it's simple: include the current milestone and the last X retros that are relevant but even then you may sometimes want specific information from older retros. With ADR documents you'd have to find the relevant ones somehow and the same goes for any other additional documentation that gets added.

There is clearly a need for some standardization and learning which techniques work best as well as potential for building a system that makes it easy for both you and the LLM to find the correct information for the current task.

> agentic capabilities are very much on a roll-your-own-in-elisp basis

I use gptel-agent[1] when I want agentic capabilities. It includes tools and supports sub-agents, but I haven't added support for Claude skills folders yet. Rolling back the chat is trivial (just move up or modify the chat buffer), rolling back changes to files needs some work.

[1] https://github.com/karthink/gptel-agent

Amp, Cursor and OpenCode are the three that I'm most behind on I think. So many tools, so little time!

Running "rg" in your ~/.claude/ directory is a good starting point, but it's pretty inconvenient without a nicer UI for viewing the results.

LLMs notoriously don't learn anything - they reset to a blank slate every time you start a new conversation.

If you want them to learn you have to actively set them up to do that. The simplest mechanism is to use a coding agent tool like Claude Code and frequently remind it to make notes for itself, or to look at its own commit history, or to search for examples in the codebase that is available to it.

If by "these things" you mean large language models: they are not learning. Famously so, that's part of the problem.

No, we’re the ones who are learning.

There’s some utility to instructing them to ‘remember’ via writing to CLAUDE.md or similar, and instructing them to ‘recall’ by reading what they wrote later.

But they’ll rarely if even do it on their own.

> It always confuses me when I see shared chats with prompts and interactions that have proper capitalization, punctuation, grammar, etc.

I've tried and fail to write this in a way that won't come across as snobbish but it is not the intent.

It's a matter of standards. Using proper language is how I think. I'm incapable of doing otherwise even out of laziness. Pressing the shift key and the space bar to do it right costs me nothing. It's akin to shopping carts in parking lots. You won't be arrested or punished for not returning the shopping cart to where it belongs, you still get your groceries (the same results), but it's what you do in a civilized society and when I see someone not doing it that says things to me about who they are as a person.

> It helps a lot if you treat LLMs like a computer program instead of a human.

If one treats an LLM like a human, he has a bigger crisis to worry about than punctuation.

> It always confuses me when I see shared chats with prompts and interactions that have proper capitalization, punctuation, grammar, etc

No need for confusion. I'm one of those who does aim to write cleanly, whether I'm talking to a man or machine. English is my third language, by the way. Why the hell do I bother? Because you play like you practice! No ifs, buts, or maybes. You start writing sloppily because you go, "it's just an LLM!" You'll silently be building a bad habit and start doing that with humans.

Pay attention to your instant messaging circles (Slack and its ilk): many people can't resist hitting send without even writing a half-decent sentence. They're too eager to submit their stream of thought fragments. Sometimes I feel second-hand embarrassment for them.

I've always used "proper" sentences for LLMs since day 1. I think I do a good job at not anthropomorphizing them. It's just software. However, that doesn't mean you have to use it in the exact same ways as other software. LLMs are trained on mostly human-made texts, which I imagine are far more rich with proper sentences than Google search queries. I don't doubt that modern models will usually give you at least something sensible no matter the query, but I always assumed that the results would be better if the input was more similar to its training data and was worded in a crystal-clear manner, without trying to get it to fill the blanks. After all, I'm not searching for web pages by listing down some disconnected keywords, I want a specific output that logically follows from my input.

Very much this. My guess is that common words like article have very impact as they just occurs too frequently. If the LLM can generate a book, then your prompt should be like the index of that book instead of the abstract.

It makes sense if you think of a prompt not as a way of telling the LLM what to do (like you would with a human), but instead as a way of steering its "autocomplete" output towards a different part of the parameter space. For instance, the presence of the word "mysql" should steer it towards outputs related to MySQL (as seen on its training data); it shouldn't matter much whether it's "mysql" or "MYSQL" or "MySQL", since all these alternatives should cluster together and therefore have a similar effect.

Greetings, thanks, and other pleasantries feel rather pointless.

Punctuation, capitalization, and such less so. I may be misguided, but on the set of questions and answers on the internet, I'd like to believe there is some correlation between proper punctuation and the quality of the answer.

Enough that, on longer prompts, I bother to at least clean up my prompts. (Not so often on one-offs, as you say. I treat it similar to Google: I can depend on context for the LLM to figure out I mean "phone case" instead of "phone vase.")

Well, seeing as these things will become our AI overlords someday — I find hedging my bets with thank you and please helpful.

I know what you mean, it's the uncanny valley. But we don't need to "pretend" that it is a machine. It is a goddamned machine. Surely, only two unclouded brain cells can help us reach this conclusion?!

Yuval Noah Harari's "simple" idea comes to mind (I often disagree with his thinking, as he tends to make bold and sweeping statements on topics well out of his expertise area). It sounds a bit New Age-y, but maybe it's useful in the context of LLMs:

"How can you tell if something is real? Simple: If it suffers, it is real. If it can't suffer, it is not real."

An LLM can't suffer. So no need to get one's knickers in a twist with mental gymnastics.

> I think pretending it to be a machine, on the same level as a coffee maker does help setting the right boundaries.

Why would you say pretending? I would say remembering.

I'd argue that doing something you don't like with people you're not into is a L. Loneliness isn't optimal but for some people it's the lesser evil. I'm married though, so I have a floor, I'm sure some people are lonely enough to benefit from being around people even under the worst of circumstances.

99% of the US population have access to 4G, caring about 3G is a wasted effort. The point I’m trying to make about tech stack is users don’t care if you used Gemini, ChatGPT or Claude to generate code.

As someone who hasn’t converted to SSR yet. My main reason why I am switching is SEO, the performance increase is a plus though.

> more productive for me based on measurable metrics such as code contribution to my projects

Isn‘t it generally agreed upon that counting contributions, LoC or similar metrics is a very bad way to gauge productivity?

> and this year has had a tremendous amount of progress in collectively figuring out how to use this new programming language effectively.

Hence the lost year. Instead of productively building things, we spent a lot of resources on trying to figure out how to build things.

This is just the Blockchain and Web3 (NTF) crazes all over again on the surface.

Every single grifter from those times is slapping AI on everything that moves or doesn't move.

But the difference is that the blockchain was (and still is) a solution looking for a problem. LLMs can solve actual problems today.

https://tools.simonwillison.net/

Here you go https://ampcode.com/news/social-coding

>Where is the resulting software?

Everywhere.

Remember Satya Nadella estimating 30% of code at Microsoft was written by AI? That was March. At this point it's ubiquitous—and invisible.

The only future I see is that prompts will become so refined that you give them all the requirements an they write the code for ya. And that prompt language has to be unambiguous and so detailed that you loop around and re-invented a programming language, just this time it wastes tons of water.

Difference: companies are not pushing their employees to use stack overflow. Stack overflow doesn't waste massive amounts of water and energy. Stack overflow does not easily abuse millions of copyrights in a second by scraping without permission.

I am not contesting that stackoverflow isn't bad in many regards, but to equate that to massive PRs or code changes done via AI slop is a different level. At worst, you might get a page or two out of stack overflow but still need to stitch it together yourself.

With LLMs you can literally ask it to generate entire libraries without activating a single neuron in your nogging. Those two do NOT compare in the slightest.

Sounds more like just adding a ton of wasted time for the reviewer to read through those discussions. At least summarize it yourself, e.g. "After discovering manpage XYZ, it became clear that the correct usage of this function is fooBar()".

It's not just that. There's a lot of (maybe useful) info that's lost without the entire session. And even if you include a jsonl of the entire session, just seeing that is not enough. It would be nice to be able to "click" at some point and add notes / edit / re-run from there w/ changes, etc.

Basically we're at a point where the agents kinda caught up to our tooling, and we need better / different UX or paradigms of sharing sessions (including context, choices, etc)