So the markdown is scaffolding the app generates deterministically.

The ## heading is built from the block's timestamps, the app name and the window title. The frontmatter is per-day boilerplate. The file:/url: line is the window's backing document where the app exposes one. The captured text underneath is written exactly as the tree handed it over: plain lines, no reconstruction.

That lossiness is also why the file/URL references exist. Trying to rebuild a document's formatting from its accessibility tree is a losing game, so instead each block records where the real document lives, and the LLM reading the file can open the original if the fragments aren't enough. "Plain markdown" was meant as "a markdown file you can open anywhere", not "faithful markdown conversion of what you saw"

Because you then have the macOS orange screen sharing warning/icon. I don't really want to record my screen, just the text is enough.

Project reasons aside, macOS has perfectly capable, local, native OCR that you can access programmatically, without having to install something else.

I hope in near future _that_ layer of abstraction -- looking at a fairly standard application window with minor UI variations and reasoning about what area / labels within the UI mean what (possibly paired with app documentation) -- could probably become a light-weight fine-tuned vision model it itself that can run fully locally.

I haven't read about how the Codex Appshots work yet, but this can be used to extract text properly. I guess. How does this idea look to you?

Yeah don't get your hope up too much. I need to push through more cleaning up of what's captured. Let me know how it goes, keen for some feedback

One of the reasons people like TUIs is because the text is always just right there.