Google really doesn't have a leg to stand on here. They scrape the Internet. They replace content against the wishes of users multiple different times, such as with AMP. Their entire business model recently has been to provide you answers they learned from scraping your website and now they want to sue other people who are doing the same.

Data wants to be free. They knew that once.

EDIT: Also to be clear I am not saying they can't win legally. I'm sure they can do legal games and could shop around until they were successful. They are in the wrong conceptually.

I bet SerpApi is getting more business than ever due to the Streisand effect. I hadn't heard about them, but if I want an API for Google results I'm definitely going to choose the one that was so hard for Google to block that they had to sue them instead. I see on their website they even advertise a "legal shield" where they assume scraping liability for their customers.

"Google follows industry-standard crawling protocols, and honors websites’ directives over crawling of their content."

Is that true with how they trained Gemini? Doesn't everyone with a foundational model scrape the web relentlessly without regard for robots.txt?

Reminds me of (the ironic AI summary) https://www.google.com/search?channel=entpr&q=celebritynetwo...

Testimony https://medium.com/@brianwarner/celebritynetworths-statement...

CNW ended up putting up content for fake celebrity's after declining Google's request for API usage to prove that Google was scraping them.

I had an idea - take SerpAPI and save top-10 or 20 links for many queries (millions), and put that in a RAG database. Then it can power a local LLM do web search without ever touching Google.

The index would just point a local crawler towards hubs of resources, links, feeds, and specialized search engines. Then fresh information would come from the crawler itself. My thinking is that reputable sites don't appear every day, if you update your local index once every few months it is sufficient.

The index could host 1..10 or even 100M stubs, each one touching on a different topic, and concentrating the best entry points on the web for that topic. A local LLM can RAG-search it, and use an agent to crawl from there on. If you solve search this way, without Google, and you also have local code execution sandbox, and local model, you can cut the cord. Search was the missing ingredient.

You can still call regular search engines for discovery. You can build your personalized cache of search stubs using regular LLMs that have search integration, like ChatGPT and Gemini, you only need to do it once per topic.

Pre the AI era at least morally they would have a stronger case, but now this is kind of hypocritical.

They also started caring about this, probably because they don't want their competitors to get the same data as they have.

google will lose, and I'm surprised they are even trying. hiQ v. LinkedIn already settled this: scraping public web pages isn’t “unauthorized access,” even if the site says no via robots.txt or ToS. Those aren’t locks.

Isn't search engine results a product that Google offers? [1] I find the argument quite strange that website owners agreed to Google being able to do anything with that data beyond displaying them in their search results when they wrote that robots.txt maybe ten years ago, but others shall not access those results programatically.

I certainly did not and find using the content google scraped from my website for money or AI (which they also sell on a token basis) more questionable than some third party offering API access to it.

[1] https://docs.cloud.google.com/generative-ai-app-builder/docs...

I bet the core of the problem for Google is that more folks use programmatic access to search which is not great on their side. Naturally you end up using Serp or other similar search APIs as they are great for the job. I believe this is also an issue especially in cases where search is performed on behalf of the user (scripts, ai tools). Google is just losing ground here, why would they bother otherwise, think what will happen to their stock if the search usage will drop? Another thing is that this builds pressure to whoever is integrating such a search tool in their products, clearly Google wants to grab that market as well.

well well well, how the turntables.

I'm not sure of the legality but I definitely appreciate their product. This lawsuit seems odd because google themselves scrape content for their indexes. From what I see SerpApi is really just providing a machine interface that Google themselves refuses to provide users and visibility into SERPs which is also something that users should have available to them.

I'm probably just being naive though...

In other news, Tailwind had to layoff their team due to lack of spending on new web sites as Google Search AI is answering search requests from scraped data without sending visitors to origin sites.

Very rich coming from an entity which scanned your neighborhood without asking for permission

is this post written by gemini? Cmon google, it wasnt even 10 mins long to write this.

This is why I stopped using google wherever possible - they pushed the frontier of useful fair use and copyright precedents and established that things on the public internet displayed to the public without a login mechanism are fair game for scraping. The US supreme court ruled that you have to incorporate authentication and not simply serve your content to the public internet if you want to restrict usage.

Then they bend over backwards and do the "but not like that!" crap with their legal team and swing their wealth and influence around to screw over other companies and people, and a vast majority of it just vanishes, gets memory holed, with NDAs and out of court settlements, so you never get to see the full scope of harm they inflict unless you're watching like a hawk and catch the headlines before they get disappeared.

Google needs to be broken up and we need to legislate the dismantling of the current adtech regime, with a privacy and sovereignty respecting digital bill of rights that puts the interests of individual citizens above that of giant corporate blobs and the mass surveillance data industry.

Move fast and break things unless they are the things owned by billionaires and gigacorps instead of stuff owned by normal people

Pot calling the kettle black

Now that the rocks are out in the glass houses, can someone please pelt one at Perplexity?

https://blog.cloudflare.com/perplexity-is-using-stealth-unde...

Google scrapes so what even is this? Beyond that I think it is unreasonable and monopolistic that Google can use all this data (like YouTube) to bolster their AI products but no one else can. It just means the megacorp will keep being megacorp and smaller players are doomed to have to work much harder and get very lucky. It’s not fair competition. So I view scraping Google as necessary for our society.

Related:

Reddit Accuses 'Data Scraper' Companies of Stealing Its Information

https://news.ycombinator.com/item?id=45695433

Our Response to Reddit, Inc. vs. SerpApi, LLC: Defending the First Amendment

https://news.ycombinator.com/item?id=45739889

From the filing

> SerpApi’s answer to SearchGuard is to mask the hundreds of millions of automated queries it is sending to Google each day to make them appear as if they are coming from human users. SerpApi’s founder recently described the process as “creating fake browsers using a multitude of IP addresses that Google sees as normal users.”

> Defendant SerpApi, LLC (“SerpApi”) offers services that “scrape” this copyrighted content and more from Google, using deceptive means to automatically access and take it for free at an astonishing scale and then offering it to various customers for a fee. In doing so, SerpApi acquires for itself the valuable product of Google’s labors and investment in the content, and denies Google’s partners compensation for their works

this has to be satire. Is Google not the #1 entity guilty of exactly this?

> SerpApi deceptively takes content that Google licenses from others

They have a different definition of "licensing" than most people I guess. Aren't site operators complaining about Google using this "licensed" content in AI overviews... not to mention the scraping for AI model training.

The pot is calling the kettle black.

SerpApi wouldn't even be a thing if Google offered an equivalent API...

What’s the difference between scraping and malicious scraping? Does google engage in scraping or malicious scraping? Do the AI companies engage in scraping or malicious scraping?

yoink

* that's the sound of a ladder being yanked up

If google provided a useable reasonably priced API, then SerpApi wouldn't exist.

Disgusting behavior by google. Scraping is google's whole business.

And then pretending that they're fighting for other people's copyright is just the cherry on top of the pile of hypocrisy.

[dead]

[dead]

Google can eat a bag of dicks.

Their entire ai model was scraped.