> Who uses service providers like this?

I use change detection to monitor all sorts of websites for changes. Some of my favorite authors don't have RSS. I always set up price monitoring for any big ticket item I'm considering like appliances so I can see how their pricing changes over time. I also use scrapers for websites that don't have an API. I like having all of my purchase history indexed in a database where I can do analysis.

> These kinds of services inevitably make the web more human-hostile and expensive.

I would rather not have to spend more time circumventing stupid bot detection things. I would be more than happy to pay for access to some of this data that I cannot access any other way.. but sure, let's keep burning resources on a cat and mouse game that scrapers will always be able to win.

Whether or not scrapping publically available websites is unethical is probably up for debate. In some cases at least, courts have found it to be legal, even when the site is throwing up technical barriers or issues cease and desists.

What is likely unethical is the fact that they offer residential proxies. The residential providers of those proxies are frequently not aware they’ve been opted in to provide such a service.

I built a similar system for an identity protection service that automated removing PII from directory websites like whitepages. Which was less ethical, stealth browser automation or monetized privacy invasion?

> Seems very unethical, no? Who uses service providers like this? The whole point of anti-bot measures is to get rid of bots - you are not wanted there.

Unethical just because it does something someone else doesn't want? I guess it depends on why and what the intention is. I don't have time to sit 24/7 in front of a computer to get a ticket to some events, does that mean it's unethical for me to use my own bot so I can purchase a ticket to bands I'm a fan of? Probably not. But if I did so for scalping purposes? Then yeah, I'd agree it's unethical.

The whole point of anti-anti-bot measures is to be able to do things even if others don't think that thing should be automated, so from the hacker news audience, I think quite a lot of us have at one point or another engaged in stuff like that. Doing so merely for profits of course stinks, but for you to be able to have a fighting chance against scalpers? Probably OK.

> Who uses service providers like this?

People who don't want their headless browser to get blocked?

> Seems very unethical, no? Who uses service providers like this? The whole point of anti-bot measures is to get rid of bots - you are not wanted there.

I'm familiar with companies automating access to software only accessible via the web with poor/no API support. This is software they pay (usually a lot of money) for, and usually has built in captchas to guard logins. They aren't a large enough customer to ask the removal of these captchas or whitelabelled (just one out of many SaaS tenants), so they simply work around that restriction.

> Seems very unethical, no?

I don't think one can judge it ethically without considering the context. Are we talking about mass automated scraping? Or are we talking about me trying to get a good deal by scraping local used car dealership listing once per day for my personal need (just so I don't have to do it manually)?

One of these is strictly more ethical, but both will be blocked by Cloudflare for example. I'd happily use such service in my personal case.

Obviously don't know what percentage represents "legit" use cases vs other more morally questionable, but in our case we have a cms where content team can include external links and we need to verify periodically whether those links work or not, which is not as easy as making get requests with a client.

(I haven't tried this out yet.) My use case would be to take a snapshot of each HN story. This is surprisingly hard, because most websites prevent bots from doing that.

For example, Claude has a lot of trouble reading HN's front page. HN itself is fine, but the moment you ask it to pick out an article, it often chokes. The website has put up a verification captcha, or it's a paywall, etc. Paywalls can be bypassed by reading HN comments and looking for archive links. But those archives often block bots too, so you're back to square one.

Whether it's unethical is an interesting question. I believe I should have the right to do what I want with internet content, as long as I'm not abusive. Merely having a bot isn't abusive. It would be one thing if the bot is hammering a server or vacuuming up training data, but having a bot at all is presently very hard.

This service caught my attention because it could potentially solve the problem I'm running into. Simply taking snapshots of articles that hit HN shouldn't be so hard, but it is. HN sends millions of views to websites; one bot taking a snapshot isn't going to make a difference. I don't think it counts as "unethical" just because we're going against the website owner's wishes. When you post content to the internet, you sign up to share that content with everyone, other than what's denied by robots.txt. If it's not blacklisted by robots.txt, it should be possible for well-behaved bots to access.

I don't expect very many people here to care about the poor bot creators. Most of the bot creators are malicious anyway. But I personally lament the loss of being able to write a program that can process information from the browser in arbitrary ways. You should be able to, yet we're buying into the notion that it's okay for website owners to say "this content is only accessible by approved bots like Google, and everyone else can sod off."

HN proves it doesn't need to be like that. It gets dozens of millions of page views a day, a lot of which is bot traffic. HN only uses captchas for creating accounts or logging in. You're free to scrape any content as long as you respect the crawl delay of 30 seconds specified in robots.txt, and don't try to visit links that perform actions a human would take (like adding things to favorites or voting). That's how the internet should work: just deliver content.

I briefly tried to do his job where it was scraping steam for CS GO skins (think a knife skin for $2,000.00) and yeah trying to find proxy poviders/get around the ip limit... tough one but market for it people paying for the tool (not mine).

Antibot measure also block real users at the slightest change they don't like. Anti-fingerprinting measure? You're a bot. Adblockers? You're a bot.

There's no ethical consumption of... ad supported content.

Exactly these crappy companies like browser use is causing more captcha etc.. All these scraper companies should've been regulated heavily. They use residential proxy creating incentive for hacking IOT devices etc..

I wish simpler bots existed for consumers. I want to know when someone replies to me, when a price drops, when airlines open new seat reservations, when a new seat opens for a college class, when a concert is coming to my area for a musician I listen to, when my local grocer has new stock, when a new Hyatt offer is available in a city I want to visit, etc. doesn’t mean I’m abusive. I can have it check once a day. In almost all those cases, I want to spend money with the business but I don’t want to manually check

The people who've been in charge of the web (i.e., mostly the browser makers, but also the owners of the most popular sites) have made decisions that are IMHO severely anti-user. Although these anti-user design decisions have been accumulating for 30 years, users have had no alternative because all the content was on the web with way to get it other than to visit web sites with a web browser.

Now that there is an alternative (namely AI) people (including me) are flocking to the alternative. You want frame this as unethical bots versus ethically-acceptable human site visitors, but the main motivation for the use of scraping bots these days is to provide services (i.e, AI-based question answering) that users (like me) consider far superior to going directly to web sites for information because visiting web sites with a web browser is a frustrating tedious experience.

Web archival/preservation services/projects that need to get past captchas and other bot checks are a prime target for a service like this... but I think their main customers are people just mass scraping parts of the internet for less altruistic reasons.

Once again I'd like to remind that violating Terms of Service isn't the same as violating some moral ethics. They are literally just expectations with no enforceable or legal boundaries.

For example I could write in my Terms of Service that you do not view more than one page on my website and expect you to send me a written permission to read the rest. I don't expect anybody to follow and I sure don't think less of those that do.

The push for verified IDs is not related to this, its more of a politically motivated attempt at selling fear to justify more surveillance.

Yeah pretty new stuff - official it’s still not recommended but works really well so far! Finally we don’t have to run baremetal

We decided to maintain Chromium as engine for stealth purposes.

Browsers like LightPanda lack stealth at all, they are trivial to detect. There are ways to make Chromium more performant, by removing everything that you don't need.

We believe that Chromium can reach that performance without starting an entire engine from scratch, and without losing stealth, a top priority for us.

The language is not the problem, C++ is as performant as Zig, but Chromium bloat is huge, agree on that.

Yes, warm pool work, but our goal is to replace them at all.

Warm pools are nice but at the end they also consume resources, And you need to always keep the pool warm, starting browsers to balance, etc...

With the upcoming changes we will keep Chromium startup and the VM will be ready in 50ms, defeating warm pools at all

Also some customers need special parameters and features, increasing warm pools complexity. The happy path will be fast but the edge case will be extremely slow , and we want to guarantee fast speeds to matter which features you need on the requested browser.

Not all use cases require all the features that we built

Few issues we had with lambas: - Limited running time (15 min), we support up to 4 hours (we can run longer if needed) - Price - Lack of snapshotting mechanisms - Lack of low-level control over the running host

But yeah, lambda is way more than enough for most common use cases automating the web

Your solution sounds very expensive.

You can take a snapshot of a microVM and roll back. I've never heard of this being done with containers.

If you follow the kernel mailing list container breakout exploits are currently a weekly occurrence

yes only c8i, m8i and r8i instance types support it. It is called nested virtualization[1]

[1] https://aws.amazon.com/about-aws/whats-new/2026/02/amazon-ec...

When we had need of quite big machines (AWS metal instances), we've found the performance differential between metal, and the equivalent size VM was 10-20% for CPU heavy workloads.

Checkpoint with Chromium running is possible and will be our next step.

Main blockers right now is fingerprint injection and profile injection, solved already.

It's always a balance of engineering effort & gains. Post-Chromium snapshot let's us save 200ms, which is not that important for 99% of use-cases, but that will come soon since it brings some other benefits (like CPU footprint)

Profiling and tools used are already included with Chromium, they provide nice debugging tools

It's not cheaper, slower startup, we lose full control and the environment is not optimized to run Chromium, so we also lose performance

I've experimented with Android Browsers. The problem is that android VMs are super heavy compared to the resources needed to run just Chromium

Startups are absurdly slow, isolation is harder, etc...

Android bloat is insane, you need to run the entire Java VM to start the browser... It's also harder to fingerprint, and at scale that's something that we need for Browser Use

Cool experiment but not yet production ready, at least for us

Shameless plug, we have the infra for exactly that use-case. Reach out if you're interested. Email in profile.

Why do this? I can’t see that it would be better than running chrome.

You have to be a bit more restrictive today yes, but if you weren't already overrun with bots and hacking attempts while hosting a public service many years ago, you probably weren't hosting a even medium-popular website in the first place. Same thing goes today. Slap a rate limit on it and be done with it.

Our previous solution (Unikraft) did not supported auto-scaling

That's why we moved to a fully in-house solution with Firecracker and auto-scaling on EC2

Docker doesn't provide any security. You install Docker on your local laptop, and the container you spin up when you execute `docker run` interacts with your laptop's kernel directly. It provides logical isolation between containers but provides zero protection for your host kernel (assuming you decide to install Docker on a remote server instead).

Firecracker provides an isolation between the host kernel, on the one hand, and the guest microVM, on the other hand. So on AWS, you use an Amazon Machine Image (AMI) to specify the OS and other components and libraries installed on an EC2 server such as c5.metal, or if you're using nested virtualization, you can use c8i, s8i, or m8i instances at a discount of about 80%-90% at some performance and other cost, and you bundle Linux along with the Firecracker binary. Then you compile a build artifact including `rootfs` for the Firecracker baked image which is the microVM image (analogous to a Docker image that results from executing `docker build`). But the microVM process has its own virtual kernel and is a guest on the host machine. So for instance, you can place Docker inside the microVM, then the container is executing against the microVM kernel, not the host EC2 kernel. Communication is achieved securely between the two using `vsock` and probably something like `socat` so that data travels, say, from guest RAM to host RAM directly to an S3 quarantine bucket, for instance, without ever touching the host's kernel or filespace.

So I've been playing and tweaking for a while with running different browsers in containers. And it took a long while to get working well, but it's doable.

The only issue is scaling, the containers aren't super quick to start (so we keep a spare container ready) and there's plenty of other issues. Also docker isn't really a security boundary so there's issues and concerns there.

Their competitive advantage is not so much running the browser but rather making the browser undetectable.

They boast a large residential proxy network too, which tells you all you need to know.

Or processes. Chrome has builtin process isolation for every browser tab. It starts up darn near instantly, and scores as 'pretty good' as far as sandboxing is concerned.

Docker does not isolate, consumes more resources and is slower

Startup time probably. They can start firecracker from a snapshot state.

docker is not a security boundary but a resource boundary.

Our browsers beat competitors in performance too. Chrome uses mainly CPU, not GPU

We support GPU via software tho

they kind of do.. gcp has their lambda equivalent which i believe comes with chromium preinstalled, its how major search tools like jina work, sure thre problaby somethign about session management that they probably neuter to prevent abuse though

not google but cloudflare has a similar product - though I am not sure how good it is

They have IMO: https://web.archive.org/web/20180823072111/https://cloud.goo...

They just don't have access to giant pools of residential IPs, so too many sites end up blocking all the cloud providers by IP range/ASN anyway, even if they could get through a captcha.

Really depends on the server specs. Tab amount relies entirely in memory & CPU availability, not in the infra that runs behind the scenes

But yeah, in one server we can fit hundreds of browsers, or even thousands if we use bigger servers. And each one of them with dozens of tabs, no issue

Browsers can't be reused between customers. They contain sensitive and private data. Everything needs to be isolated and ephemeral.

Assuming CRIU can checkpoint and restore Chrome, and especially recent versions of Chrome, just fine, is a little bit of stretch.

We persist profiles to maintain sessions if needed, this includes cookies, session storage and everything needed to keep your account logged in

well that's how browser agents work in a nutshell lol