I was probably not the first to have to do that, we knew what apps our customers used, making them better was the whole point of the operation

I did the architectural design for the SuperMac cards. I figured out what needed to be accelerated, dropping code into people's machines to see where the cycles were going. Others did the physical design for the first 2 cards, I did the design of the chip in the Thunder and later cards (designed the data paths and state machines and a full simulation, someone else actually laid the gates)

If your card has a SQD01 on it it's my work. It peaks at 1.5Gb/s on solid fills

To be fair this was Excell 25 years ago, may no longer be true.

One of the other bugs (the Quark/ATM one) was also because of the programmers were worried about writing over stuff that hadn't been completely erased, the Quark guys wrote a string with 2 spaces at the end through a box that masked the end of the string, the ATM font renderer saw it couldn't fit the text so it split it in half and tried again so it drew N/2 N/4 N/8 ... strings. It spent all it's time in the 68k's multiply instructions figuring out how wide the strings (and substrings) were, our fancy 24-bit character rendering hardware was an afterthought

There's a good chance it was Excel's workaround for some other GPU's buggy behavior.

In all of the software you’ve written, are you aware of how many on-screen pixels you’ve overdrawn?

> To be fair excel would erase places white that it wanted to write up to 9 times before it drew any black pixels

I feel like I'm having a stroke trying to read this, what does it mean??

I swear if Sun Microsystems was still around, their machines would still ship with 8-bit pseudocolor and you'd have to pay an extra $3k for 24-bit.

I got the extra vram in my LC to allow for 24-bit color but it was dog slow. The 16 bit data path didn't help. If I wanted it, I'd get things done in 8 bit or mono until it was ready, then switch to 24 bit for the final look.

Yeah, even in Linux we were doing these things with X Windows bit depths.

8 bit psuedo color, so the color palette switched with every focus-follows-mouse window boundary crossing. 16 bit direct color with banding but no more palette psychedlia.

This was equal parts to make it faster and to allow for higher framebuffer resolutions with limited VRAM.

16 bits? Luxury. I had 6 colors when I was a kid and was happy to have them.

Yeah those low core global system variables (including a readable/writeable 0) at fixed addresses were very much a thing, they were a bad design decision made for the original Macs with almost no memory, and made running more than one app (switcher/multifinder) a difficult transition back in the day. Someone wasn't planning ahead

I also worked on the original A/UX port for the Mac II, some hardware (like the IWM) required tiny buzzy loops, we ran into one bug where using the floppy caused ADB to freeze, but only on the release machines, not the prototypes all our engineers had, turned out there was hardware that made access to the VIA faster by pulling the clock in for 1 cycle, if you sat in a loop reading the timer in the via to measure a sector time for the IWM in too tight a loop it upped the output clock from the VIA to the ADB chip and over clocked it ....

They were most likely just bugs. Quite possibly really stupid bugs.

Not every bug results in the program doing the wrong thing, they often just make the program do the right thing very slowly.

And nobody notices, since it still produces the right result.

It's not necessarily stupid code in the game, but something the C library is doing that it probably shouldn't.

If the stream is buffered, then all operations, including fread, are supposed to go through the buffer.

All three of these should issue buffer-sized reads to the operating system:

1. A loop which calls getc(stream) 65536 times.

2. fread(buf, 1, 65536, stream)

3. fread(buf, 65536, 1, stream)

The more direct behavior of fread should only kick in if the stream is configured as unbuffered.

I would say that the way low-level reads are issued to the host operating system is a "visible effect" of the program, so I suspect this may actually be a matter of conformance. I.e. it's not okay to issue those reads however the stream library wants as long as the data is read.

That's not how I remember these events when they were playing out. I distincly remember social media posts warning about the dangers of modifying game files, plus refusal to acknowledge the issue. Note there were 2 full weeks between the blog post and the update mentioning the bounty. I'm pretty sure the massive community outrage in between has played a role in it. But I don't have any sources and I was wrong about at least one thing (lack of attribution), so I'm okay assuming I'm wrong about everything else too.

> The standard says that fread calls fgetc multiple times for each object:

>> For each object, size calls are made to the fgetc function and the results stored, in the order read, in an array of unsigned char exactly overlaying the object

Aha! That phrase led me to https://man7.org/linux/man-pages/man3/fread.3p.html. I consulted https://man7.org/linux/man-pages/man3/fread.3.html and https://man.openbsd.org/fread.3. Neither mentions that.

Now, I checked https://cplusplus.com/reference/cstdio/fread/. It doesn’t mention it, either.

⇒ this appears to be POSIX-specific.

Finally, if somebody implements fread as “For each object, size calls are made to the fgetc function”, it doesn’t matter whether you ask for 1 object of size 65,536 or 65,536 objects of size 1; both would call fgetc 65,536 times.

I think it’s pretty rare for files to be unbuffered like that. AFAIK it’s mostly stderr that ends up unbuffered, at least on Unix-like systems.

I used to make it a general rule to start all my optimisation of any network code by running strace and look for excessive read's and write's, because you'd be shocked how many did stuff like that if they didn't know the length of a string, or to read the length first, instead of reading into a buffer.

I had to convince people with benchmarks regularly that, yes, you could write the handful of lines to do proper user-space buffering and trivially run rings around any code that did extra context switches, because a lot of people didn't realise the cost difference between system calls and calling their own functions.

This included, by the way, the MySQL client library, at one point, which would do small read for length fields instead of larger non-blocking reads into a buffer all the time

That's different: you're talking about the application code, like OP.

But I think the parent comment's point is that the issue is in the implementation of fread itself in the standard library. It's perfectly reasonable for an application to pass it 1, 65536 (i.e. one byte, up to 65536 times) and expect it not to issue 65536 separate OS calls.

He didn't write it.

The C runtime authors did (presumably Microsoft, if it's MSVCRT).

He's hooking into ReadFile, a layer below the stdlib. By the time it reaches the hook, it's already split.

Its slowness is also a function of security software or any other file system "filters" (I believe they're called) are installed.

For example, I run TortoiseGit which has a caching feature which is supposed to make it faster at showing what to commit. Disabling it increases the number of items I can delete per second in my Windows Explorer from about 1000 to about 3000 while making not making TortoiseGit operations meaningfully slower (that I can tell).

This is a Dev Drive [0] on my machine, it would probably be slower on my C: drive which has full Windows Defender real time file scanning.

[0]: https://learn.microsoft.com/windows/dev-drive/

Windows Explorers zip implementation also seams to do 1 byte reads by the speed is has compared to every other zip implementation.

There are no small freads in the story, whatever implements those freads supposedly split them up into many calls. But that sound more like a problem of that implementation than the fread callers as size == 1 is correct when you are reading a bag of bytes.

If I have a buffer of bytes, and I intend to treat the content of that buffer as individual bytes, what is semantically wrong with "read 65k 1-byte-sized items into this buffer"? Wouldn't it be a bit unnatural to express it as "read one item whose size is 65k"?

But the args aren't necessarily swapped just because they end up in a slow case in some implementation.

Yes, it divides the bytes read by the element size to get the return value.

Which is the obvious reason you'd pass an element size of 1: you want to know how many bytes were read.

That's a case of the driver cheating but there are also lots of cases where the game is just full of bugs that the driver has to work around in order to not be blamed for them.

I'm curious, what's the ratio of:

- deciding to inform the game developer & wait for reply vs not waiting for reply vs just fixing it yourself without informing the developer; and

- if informed: developer actually fixing it vs only saying they would fix it vs no reply whatsoever (not counting automated "thank you for your inquiry" replies, in cases where you don't already have more direct channels to the dev than email)

I've always kind of wondered this because in a way, it's kind of weird that it's fixed for them, at least for new releases / games actively being developed.

(Full disclosure: I'm a game developer myself, with a very high interest in engine plumbing & dev [including graphics], though finding a job for the latter is easier said than done.)

then the driver gets updated and the game either continues to optimize (wrong) or branches out into code that was written before that driver came out and generally wasn't that well tested, and the circle continues...

It's the life of a (game) developer...

In general, because it's a flag that says to do things in an incorrect but faster way. It's like -ffast-math. The applications for which it's intended don't do anything where the incorrectness matters. Some random application falsely labeled hl2.exe may or may not.

> What it should do is ensure some things not relevant to Half-Life 2 were not done, thus getting better performance for this game in particular, but there is no guarantee that same optimizations work for other applications or games, so one should not expect an overall improvement.

I can't quite parse this. Yes, there is no guarantee that the optimizations will work for another game, which is precisely why you can expect an improvement with hl2. With non-hl2, you may get an improvement, you may not, and you may get incorrect behavior.

Everything else is not the same, but hl2 doesn't use the stuff that's different.

I wouldn't be surprised if it made other games on the Source engine faster, but everything else slower.

gpu drivers detect games, among other thing by looking at executable names

then driver "optimizes" behavior, sometimes dishonestly (reducing precision), sometimes honestly (working around game engine stupidity)

Because everyone reported performance metrics using it as a benchmark. Higher number = more sales.

At least these actually make things faster usually.

Who has been validating that approach to solve their own optimization target?