Good point. The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — things like CDP leak fixes so Cloudflare doesn't block you mid-session. It's not about bypassing access restrictions.

Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we should make that clearer in the README.

Unless I'm totally misunderstanding something it's not xml but special tokens for the tokenizer someone smarter than me might know https://medium.com/@nisarg.nargund/why-special-tokens-matter...

Yeah that's a good observation. XML's closing tags give the model structural anchors during generation — it knows where it is in the nesting. JSON doesn't have that, so the deeper the nesting the more likely the model loses track of brackets.

We see this especially with arrays of objects where each object has optional nested fields. For complex nested objects, the model can get all items well formatted but one with an invalid field of wrong type. That's why we put effort into the repair/recovery/sanitization layer — validate field-by-field and keep what's valid rather than throwing everything out.

Hardly matters, this isn't a problem that you'd have these days with modern LLMs.

Also, a model can always use a proxy to turn your tool calls into XML

And feed you back json right away and you wouldn't even know if any transformation did take place.

HTML -> markdown -> LLM is standard practice. We strip elements like aside, embed, head , iframe etc. the criteria is conservatively set to avoid removing too many elements (especially in extractMain mode)

https://github.com/lightfeed/extractor/blob/main/src/convert...

I have used gemma 3 and had good results.

Once Gemini 3 flash drops the preview suffix, will update the examples. Thank you for the pointer.

What kind of LLMs are you using? In structured output mode?

In this library we recover nullable and optional fields, invalid elements in nested array, bad urls, repair incomplete JSONs. If these issues are what you see, yes it should work for your case.

Have you considered recaptcha v2 and similar? Proof of work might slow them down. Sounds pretty bad. Would be great if Cloudflare, Datadome, etc. were doing this for you and thus banning these devices for everyone.

What crawlers are using residential proxies?

Put things to perspective - Gemini 2.5 flash is 0.3/1M tokens - assuming each page is 700 tokens and output is not much you are looking at $210 for 1M pages

The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — fixing CDP leaks, removing automation flags, etc. For sites behind Cloudflare or Datadome, that alone usually isn't enough — you'll need residential proxies and proper browser fingerprints on top. The library supports connecting to remote scraping browsers via WebSocket and proxy configuration for those cases.

In context of e-commerce web extraction, invalid JSON can occur especially in edge cases, for example:

price: z.number().optional() -> price: “n/a”

url: z.string().url().nullable() -> url: “not found”

It can also be one invalid object (e.g. missing required field, truncated input) in an array causing the entire output to fail.

The unique contribution here is we can recover invalid nullable or optional field, and also remove invalid nested objects in an array.

Good point. The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — things like CDP leak fixes so Cloudflare doesn't block you mid-session. It's not about bypassing access restrictions.

Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we should make that clearer in the README.