Ask HN: Is Common Crawl used exhaustively by any search engine?

8 pointsposted 18 hours ago
by n1xis10t

Item id: 45794812

2 Comments

agentbox

11 hours ago

To my knowledge, no public search engine indexes the full Common Crawl corpus. Projects like Neeva (before shutting down) and some academic prototypes used parts of it for evaluation, but none have managed to process all 300B pages continuously.

The biggest practical barriers are deduplication, spam filtering, and keeping the index fresh — CC snapshots are monthly but the quality varies a lot.

For experimentation, you can look at projects like CCNet, ElasticSearch’s open-source pipelines, or small-scale engines such as Marginalia Search, which use subsets for niche purposes.

n1xis10t

4 hours ago

For freshness, I wonder how much their news crawls (which I’m pretty sure are weekly) would help.

Thanks for the suggestions. Have you worked at all with the Common Crawl?