gonna try this out this week, one ask: when a wall does beat it, return "blocked, here's why" instead of empty, avoiding silent fetch failures.

Token reduction is useful, but knowing whether the agent actually accessed the real page is even more important.

stripping to markdown with Jina Reader or Trafilatura before passing to the agent cuts that 68k down to ~3-5k for most Wikipedia pages, and handles the JS-rendered case too.

one of the early jina.ai products was/is reader api, and they trained ReaderLM for this purpose. definitely a good idea to check out existing implementations

> on js-rendered and some anti-bot pages it returns nothing

yeah, and my (human piloted) browser gets blocked by so many websites now. I routinely get "Sorry." when trying to log into hn.

sigh.

And how much water?