I use this too and, not to detract from your enthusiasm, it's not exactly no-strings-attached. There's a token limit on free use and you can't use it for any commercial purposes. Luckily the pricing for unrestricted use is reasonable though at 2 cents per million tokens.

People will also want to note that it's LLM-powered which has pros and cons. One pro being that you can download and run their model yourself for non commercial use cases: https://huggingface.co/jinaai/reader-lm-1.5b

Thanks, Jina actually looks quite nice for use in LLMs.

I also provide a REST API [1] that you can use for free (within limits). However you have get an API Key by registering with Github (see reason below).

---

The demo was previously hosted on Vercel. Someone misused the demo and send ~5 million requests per day. And would not stop — which quickly brought me over the bandwidth limits of Vercel. And bandwidth is really really expensive!

So that is the reason for requiring API Keys and hosting it on a VPS… Lessons learned!

[1] https://html-to-markdown.com/api

For clarity: I'm a pandoc diehard (especially because it's written by a philosopher!) but it intentionally doesn't approach this level of functionality, AFAIK.

Oh very nice to hear, thank you very much!

That’s actually a great idea!

I personally use Raindrop for bookmarking articles. But I can’t find stuff in the search.

The other day “Embeddings are underrated” was on HN. That would actually be a good approach for finding stuff later on. Using webhooks, converting to markdown, generating embedding and then enjoying a better search. You just gave me the idea for my next weekend project :-)

There are two parts to it:

1) convert html to markdown

This is what my library specifically addresses, and I believe it handles this task robustly. There was a lot of testing involved. For example, I used the CommonCrawl Dataset to automatically catch edge cases.

2) Identify article content

This is the more challenging aspect. You need to identify and extract the main content while removing peripheral elements (navigation bars, sidebars, ads, etc.)

For example, the top of the markdown document will have lots of links from the navbar otherwise.

Mozilla's "Readability" project (and its various ports) is the most used solution in this space. However, it relies on heuristic rules that need adjustments to work on every website.

---

The html-to-markdown project in combination with some heuristic would be great match! There is actually a comment below [1] about this topic. Feel free to contact me if you start this project, would be happy to help!

[1] https://news.ycombinator.com/item?id=42094012

> built something similar for the Elixir

We interact with the web so much that it’s worth having such a library in every language... Great that you took the time and wrote one for the Elixir community!

Feel free to contact me if you want to ping-pong some ideas!

> feeding LLMs web page data

Exactly, that one use case that got quite popular. There is also the feature of keeping specific HTML tags (e.g. <article> and <footer>) to give the LLM a bit more context about the page.

Why not just give the html to the llm?

My final university project was about a clean-up-approach on the HTML nodes before sending it to the html-to-markdown converter. But that was extremely difficult and dependent on some heuristics that had to be tweaked.

Your idea of comparing multiple pages would be a great approach. It would be amazing if you build something like this! This would enable so many more use cases... For example a better “send to kindle” (see other comment from rty32 [1]).

[1] https://news.ycombinator.com/item?id=42093964

If it doesn't have to be in Kotlin there is flexmark:

* https://github.com/vsch/flexmark-java/tree/master/flexmark-h... * https://github.com/vsch/flexmark-java

Same idea I guess, but it's Aaron's has been broken for years - and probably for the best because it didn't stop people specifying things like "file:////etc/passwd" as the URL to export to markdown.

Yeah good point, that's actually difficult. They use many `<span>` html tags to color individual words and syntax.

But I wrote logic to handle that. It probably needs to be adapted at some point, but works surprisingly well. Have a look at the testdata files ("code.in.html" and "code.out.md" files [1]).

Feel free to give it a try & let me know if you notice any edge cases!

[1] https://github.com/JohannesKaufmann/html-to-markdown/blob/ma...

Pandoc is amazing. Especially because they support so many formats.

And their html to markdown converter is (in my opinion) still the best right now.

But html-to-markdown is getting close. My goal is to cover more edge cases than pandoc...

It doesn't. For that you would need to execute a full headless browser first, extract the HTML (document.body.innerHTML after the page has finished loading can work) and process the result.

If you're already running a headless browser you may as well run the conversion in JavaScript though - I use this recipe pretty often with my shot-scraper tool: https://shot-scraper.datasette.io/en/stable/javascript.html#... - adding https://github.com/mixmark-io/turndown to the mix will get you Markdown conversion as well.

We do that with Urlbox’s markdown feature: https://urlbox.com/extracting-text

That is unfortunately out of scope. I like the philosophy of doing one thing really well.

But nowadays—with Playwright and Puppeteer—there are great choices for Browser automation.

I used https://github.com/mozilla/readability for this

oops, refreshed the page and saw other comments addressing this! nevermind!