pizlonator
8 months ago
It’s cool to see this kind of analysis, even if it’s analyzing a totally bogus benchmark.
If you want to compare language runtimes, compilers, or CPUs then you have to pick a larger workload than just one loop. So, if a microbenchmark is an experiment, then it is a truly bad experiment indeed.
Reason: loops like this are easy for compilers to analyze, in a way that makes them not representative of real code. The hard part of writing a good compiler is handling the hard to analyze cases, not loops like this. So, if a runtime does well on a bullshit loop like this then it doesn’t mean that it’ll do well on real stuff.
(Source: I wrote a bunch of the JSC optimizations including the loop reshaping and the modulo ones mentioned by this post.)
ergeysay
8 months ago
> loops like this are easy for compilers to analyze, in a way that makes them not representative of real code
Which makes it a perfectly fine benchmark to measure whether a particular compiler implements these optimisations. The benchmark also highlights fine implementation details. I did not know about Dart's interrupt checks, for instance.
I see these microbenchmarks as genuinely useful, as I can analyse them, the logic behind them, and apply results in interpreter design. Consider [0] for example. Any sane compiler would do this kind of optimisation, but I've seen only one production interpreter (daScript) doing it.
[0] https://ergeysay.github.io/optimising-interpreters-fusion.ht...
pkolaczk
8 months ago
And what if the runtime does poorly on even such a simple loop? Go is surprisingly slower here than Java and Kotlin.
I agree with the author of the blog here - microbenchmarks are just experiments and they can be very useful if you do proper analysis of the results. You can definitely learn something about the runtimes even from such a simple for loop benchmark.
solarkraft
8 months ago
> The hard part of writing a good compiler is handling the hard to analyze cases, not loops like this
But it means that it can handle the easy case and if one is already bad at that it indicates that it won’t do well on harder ones either.
My takeaway from 1 billion loops wasn’t exactly “js is always fast” but “js can be fast while python will always be slow” (talking about their main interpreters of course).
brabel
8 months ago
> It’s cool to see this kind of analysis, even if it’s analyzing a totally bogus benchmark.
Yeah but that bogus benchmark has become viral on social networks, and people are even writing their own shitty "experiments" based on that one.
This post is absolutely wonderful in that it doesn't shit on the original benchmark too much, while explaining patiently how to do it right. Hopefully, the person who started with the bogus benchmark, as well as people following in his footsteps, will learn something important (I myself have posted benchmarks before - though I believe much higher quality - but failed to properly analyse the results as shown here).
Notice that the bogus benchmark ended up catching the attention of a Dart maintainer and looks like they found their compiler was missing some easy optimisations... so it may end up having been helpful for every Dart user in the world!