> The I guess the site is either useless, or can be used in combination with customer's own research that includes sites not included in the comparison. The site is still useful, it's just not exhaustive.
There is a significant difference between having the prices for 85% of retailers because you haven't figured out how to get pricing from 15% of them and having the prices for 5% of retailers because the biggest ones block you from getting their prices. The amount of work you can save the consumer, and thereby the usefulness of the service to them, changes by a dramatic amount.
> Also, maybe the source that isn't scrape-able becomes less popular as a result of not being included in the price comparison list. If they're always more expensive, then neither you nor they have any advantage in listing on your site. If they're always cheaper, then your site may have no purpose to serve.
Now consider the possibility that they sometimes have the best price and sometimes don't.
> The lack of a certain set of data may itself be a data point, and should be used as marketing material for or against the company resisting the scraping.
Suppose you go to the price comparison site to look for the best price, the best price in its data set is $22, then you go to Amazon and they have it for $19 because it doesn't have their prices. The customer then starts checking Amazon in addition to the price comparison site. Their price is only actually lower 15% of the time, but another 75% of the time it's exactly the same, so the customers who don't want to keep checking two sites start checking only Amazon instead of only the price comparison site, at which point Amazon gets to charge them a higher price on 10% of stuff, or a higher price on more than 10% once people have been trained not to check. And prevent them from patronizing a random competitor when their price is exactly the same
Causing that to happen is the reason they don't want their prices in the comparison site. If not being listed there hurt them then they wouldn't be trying to prevent scraping.
> I'm a consumer who happily does significant research before a buying decision, going to various sites and checking prices, warranties, model numbers, reviews, availability, delivery times, and all the shit. And prefer doing the research myself than trusting a price comparison site, so I'm not the target market, thus my bias in my commentary and opinion on that topic
Consider also that you may be able to justify doing this when buying electronics, or choosing which brand of something to buy on a recurring basis, but if you just want the best price on the product you already know you want, it's crazy to spend hours to save cents. But completely sensible to switch to a search box that would save you 10 cents on every $2 purchase by giving you price-sorted results from multiple retailers who all sell the same products, if it's allowed to exist.