Isn’t “just post the prompt” a joke? That only works if the content was one-shot, and I bet that (much like code) most isn’t.

Same as sites the block "wget" as a user agent, but then you can pass in a -U "Cheetos/10" and sail right in.

I'd be fine if they throttled the connection, or a contact page. Every little bit helps build a high trust society.

It certainly wasn't a mistake in the beginning, but it's certainly a mistake now.

"The HTTP User-Agent header was a mistake form the beginning."

User-Agent spoofing was added to NCSA Mosaic in 1996. The public www was three years old and text-only clients were still in widespread use

https://raw.githubusercontent.com/alandipert/ncsa-mosaic/mas...

NCSA Mosaic is the early graphical browser that begat Netscape Navigator that begat Firefox. Later came Internet Explorer, Safari, Chrome and so on

What was the point of spoofing in 1996

Maybe it was just for fun, judging by the examples in the "mosaic-spoof-agents" file

As a matter of practice, by default I do not send a user-agent header. I only send the minumum headers required. For me, that's almost always 1-2 for GET and 4 for POST

For the vast majority of websites I have accessed,^1 this header minimisation has zero effect on the success of the HTTP request

1. For example, I have used a database of sites submitted to HN as way to test if header minimisation affects HTTP request success

Generally I do not use a web browser to make HTTP requests. I perform text processing on the HTML, JSON or whatever is returned, using custom utilities. I store information in SQLite. I read this information as plain text, preferably 7-bit ASCII. I dislike UTF-8

Onlilne debates about "browser fingerprinting" always seem to focus on trying to "blend in", e.g., via "spoofing"

As such, because browser continue to get more bloated with "features", online commenters argue in favor of sending more and more data points to servers that can be used to create a fingerprint instead of reducing the amount of data sent

Because, according to their reasoning (or lack thereof), sending less data would "stand out"

True, but it's generally easier to "spoof" a client that sends less data than one that sends more. And the number of sites that require a user-agent header is still smaller than the number that don't

There is actually/obviously. It lets you identify bugs in well meaning clients. e.g. you can identify if IE v1.2.3 isn't handling form submissions correctly. It lets you serve content that works for clients with known issues or limitations.

When distributing software, it’s very useful to know which OS and which architecture the user is on in order to present them with the appropriate file.

I feel like this is with 2026 view where browsers are so mutually compatible.

In the bad old days there were so many differences between html, css and js behaviors that if you wanted your site to be nice you had to change it for the browser. The way css padding worked wasn't even the same. Feature detection was rarely viable for any of this.

No user agent would probably have only entrenched IE6 dominance even more by blocking you from deliberately making a site that works at all on other browsers (including IE7 for that matter)

I get them all the time on Firefox on windows using a vpn.

As much as I would like pricing APIs as well, the economics don’t add up for retail, and it would effectively lead to a scalper’s market.

A regular Joe consumer shouldn’t have to learn python to get thier foot in the door because some arbitrage purchaser has automated a ‘best deal’ purchasing bot based on real time pricing APIs.

scraping, however, is not intrinsically a scam.

Always was.

Has the internet made the average human’s life better? As someone that lived in both eras I have to say no. If you want to improve humanity greatly and prevent currently 8.3 billion people’s misery and counting, you build a time machine and prevent Tim Berners-Lee from inventing it in the first place. But it wouldn’t help, because someone else would still invent this torture nexus of hyperconnected human minds.

Do you think this could drive a shift towards more efficient web tech? I.e. more static websites, or websites served by faster software?

> we see customers that are constantly burned by abusive scrapers and the scrapers aren't slowing down

So, I have two dumb questions:

1) Can't the customer rate-limit connections? If the "abuse" in scraping is the number of requests... limit the number of requests?

2) There is probably a market now for federated authentication where the provider gives legal guarantees of anonymized fingerprint in exchange for either payment or selling data. This would provide an identifier for rate-limiting requests, and be pretty much the same as "Sign in with Google", but without identifying the user to the customer site. Are there too many problems with this that make it unfeasible, or would your company/customers be open to such a solution?

Tracking humans isn't necessary. Fingerprinting is necessary to tracking humans.

I was a bit puzzled by your second sentence, so I searched around a bit and... do I have this right?

- There’s a well-known way (“Ziv’s rounding”) to get (among other things) a correctly rounded double-precision pow(), but in bad cases it can get slow, meaning really quite slow in practice and we’ve got no idea how slow in the worst case (nobody knows what the worst case is).

- There’s a recent, guaranteed-correct way[1] to get (specifically) a correctly rounded double-precision pow(), but the last step requires “enough” precision and we’ve got no idea how much that actually is, so CORE-MATH uses 256 bits of mantissa and crashes if that turns out not to be enough (no such cases are currently known).

- (Bonus) For most special functions [not just the bivariate ugly duckling of pow()], there’s essentially no hope of getting a correctly rounded quad-precision version anytime soon.

[1] https://inria.hal.science/hal-04159652v2

At what point are we going to realize that computers will never work for floating point math. If your numbers are not exact there will always be a better solution for what you are doing that does not involve a computer.

Sure, but you'd use a low-precision approximation for that; you don't need full double precision

I didn't recommend fixed point for simpler HW - I recommended it for better precision (if you know what you are doing). First, a point I didn't make, is that if you have 32 bits of fixed, you get way more precision than with a 32 bit float. But I can think of a pretty common case where a 24 bit int would win against a 32bit float: convolution filters. If you have a filter whose inputs are supposed to sum up to 1 (which is the most common case), integer computations mean that, even with internal overflows, the end result will be correct. In contrast, with floats, you can lose precision. If you apply said operation 10000x recursively (say, you are 'stepping' a simulation), those errors can add up bigtime.

> Floats are also much easier to get right for your coworkers that aren't numerical analysts.

That one is true, however, when you have people, such as EEs who really care about precision, and know the theory behind it, then floats are often not the obvious choice. It has other advantages, like your calculation running the exact same regardless of CPU and/or compiler, which I'm sure a lot of analysts care about. Afaik finance people don't even use floats for things like account balances, because you can't represent something like 0.1$ exactly.

Fixed point has basically no language support, and is very hard to get right, but sometimes you need to do that.

Do you have any subject matter expertise in quantization errors? Like doing simulations or DSP work? Not trying to be antagonistic, just figure out where you're coming form.

> The vast majority of processors are optimized for floats now and some operations (e.g. division) are actually faster.

This seems backwards. Hardware is optimized for floats because people use floats. If people used fixed point, hardware would become optimized for that instead.

Given an equal number of transistors, I'm pretty sure fixed point would be a lot faster on equally optimized hardware for almost all operations.

fixed-point provides uniform precision, exact integer-scaled arithmetic, is deterministic whereas floating point is more convenient but its not a panacea

it's very common to have to implmeent algorithms on microcontrollers with no floating point or on FPGAs...

Well, there's an arbitrage opportunity if I ever saw one

Please share with the class :)

Ooh, the disclaimer is new, it wasn't there originally: https://web.archive.org/web/20260712212920/https://scrapfly....

Sounds needlessly divisive.

Why not criticize the content instead of the source or medium?

> Why give up information when you don't have to?

Yeah, I generally agree with that viewpoint, but you are giving up that information, even without JS.

Your TCP stack can be fingerprinted through long enough observation. Windows, Mac, and Linux all look different on a network level. It's not as simple `if XYZ then OS = Windows`, it's more holistic/probabilistic, but it's possible nonetheless.

e.g. One thing that bugs me about arch linux is that they recently changed the kernel default TCP keepalive time to be shorter (much shorter[1]), making arch users stand out a lot. So, not only can a fingerprinter identify your OS, they might be able to pin down your exact distro based on TCP behavior alone.

I guess my point is that hiding the OS would be a massive amount of effort to plug a hole that cannot be plugged without effectively controlling the entire OS. TB/MVB is limited in what it can do by itself.

[1] https://rfc.archlinux.page/0051-tcp-keepalive/

Well it's not really an airtag, I don't "attach" anything to your browser when I check what its GPU can do.

That's just a description of you that I share with my other stores. Casinos, Target, Burger King, etc all do this when you get 86'd, for example.

Yes, and we need a name for the fallacy of GP. It's a recurring theme.

Which is famously done by casinos. But in practice many businesses big and small do share intelligence with each other about problematic customers who shoplift etc.

Holy slippery slope, Batman. Just because you don't have a right to know everything about you, follow you home, and see what you bought elsewhere, doesn't mean I'm not allowed to observe how your [thing claiming to be a browser] behaves when it connects to my website.

If you want to make it harder to guess whether you're human or a malicious bot, you're going to find yourself clicking on more than just traffic lights and bicycles in the near future.

There's nothing wrong with websites choosing to comply with particular privacy standards. This should be voluntary and used as a trust signal. A website that promises to follow certain standards should be legally accountable if it breaks that promise, but it should not be mandatory to make such promises.

With companies like these, "plausible deniability" is more likely.

But they added ad clicks tracking API ("Attribution Reporting API") and interests tracking API ("Topics API") into the browser which work behind user's back and against their interests.