twosdai
30 minutes ago
Are you using residential proxies? How do you handle websites that don't want to be scraped.
EG if I start passing in Linkedin pages what is your expectation of the result that people would see per profile.
EDIT:
Congrats on the launch seriously hard work, just wanting to understand your scraping stance more. I've worked with a lot of tools on this, didn't mean for my initial comment to be adversarial.
TheYahiaBakour
23 minutes ago
hi, no worries at all, it came off perfectly fine
yes, we use residential proxies + all requests go are js-rendered, we maintain a caching layer which is 95%+ opted into by customers
it's all included in the credit price, great value compared to alternatives, the business model does rely on scale and our margin gets better the more requests we serve (esp infra cost for k8 + browser fleet)
to answer your e.g., yes public linkedin pages will work fine, anything behind a login we don't really support out of the box until we can figure out a safe way to do so, since that's where red lines are drawn
we step in whenever we see our service is hitting a website more than it should, this usually means reaching out to a customer for clarity on why they are not opting into the cache, we have alot of safeguards around fraud/spam and will let someone know if their request pattern looks like they're causing harm