Select * the internet: DuckDB queries 100B rows in 30s

4 pointsposted 7 hours ago
by dmkii

1 Comments

dmkii

7 hours ago

You can basically SELECT * the internet by querying the public Common Crawl dataset on S3 with DuckDB. It's a great way to explore how the AI-pilled web (Vercel, Lovable, Replit, Bolt, Cloudflare, GitHub) compare to the traditional Wordpress-Blogspot hegemony.