Modern Perfect Hashing

97 pointsposted a day ago
by bariumbitmap

16 Comments

eschaton

18 minutes ago

There’s an explanation of how to implement a perfect hash generator in “Hacker’s Delight” that you can read to take a known set of keys and produce a consistent perfect hash at runtime.

It’s a very worthwhile thing to keep in your back pocket if you’re going to deal with hashes whose keys are 100% knowable at runtime but not before.

adzm

13 hours ago

Make sure to read the post linked right at the beginning as well: http://0x80.pl/notesen/2023-04-30-lookup-in-strings.html as well as the magic bitboards linked, too https://www.chessprogramming.org/Magic_Bitboards

Though honestly this post really needed some numbers and benchmarks.

Sesse__

13 hours ago

I never really finished the project, thus only the rough qualitative benchmarks you get at the bottom (measured mostly by profiling and size(1)); I saw that it wasn't enough of a win in the larger context where I needed it, thus it made sense to stop early instead of making exhaustive benchmarks.

The blog post was mainly for curious readers, I'm surprised it hit HN at all. :-)

jibal

12 hours ago

gperf is very limited in the number of keys it can handle as opposed to, say, https://burtleburtle.net/bob/hash/perfect.html

Sesse__

12 hours ago

Well, again, different problem constraints, different solutions. Seemingly that tool can handle larger sets than gperf (although it claims gperf stops at a couple hundred, which is an exaggeration; try it with the first 1000 lines of /usr/dict/words and it's nearly instant, and with the first 10k it needs 35 seconds or so), but it also says the runtime is even slower. My goal was to have faster runtime, not handle more keys. YMMV.

jibal

12 hours ago

Not an exaggeration, just written when machines were a lot slower. Anyway, more work in this space is always welcome, so thanks.

o11c

11 hours ago

One thing not mentioned: very often "give up and allow a not-quite-perfect hash" is a reasonable solution.

hinkley

4 hours ago

I believe there are some high concurrency hash tables out there where the data structure contains two tables, and so each get results in a constant number of fetches, but the count is almost always greater than one. But if it avoids concurrency issues that ends up being acceptable.

I remember Cliff Click presenting one that was lockless and if I'm recalling correctly, where capacity growth operations could happen in parallel with reads. And possibly writes.

mgaunard

13 hours ago

What's most annoying with gperf and similar tools is that they aren't really suited to applications where the set of keys is known at runtime during initialization.

zvr

39 minutes ago

I've written code that, during initialization, after all keys have been collected, essentially called gperf to create the lookup function, compiled it, and then dynamically loaded it, so that the (long-running) program would be able to use it.

mgaunard

24 minutes ago

That means you need to deploy with a compiler etc., not ideal.

hinkley

3 hours ago

Maybe this is one of those situations where compile time code execution wins out. Instead of needing one solution for runtime and one for a priori knowledge, you just run the runtime code during the build process and Bob's your uncle.

jibal

12 hours ago

hinkley

3 hours ago

If I'm understanding this correctly, you're essentially hashing twice. You get a perfect hash on the second hash call, and don't worry about one on the first.

There are other hash tables that use double hashing but they don't guarantee a lack of collisions. They just make them highly improbable.

anitil

5 hours ago

I wrote a python library that would build a perfect hash at run-time, it was basically the stupidest way you could do it - it would shell out to gperf to build the library, compile it to a shared library, then link the entry points in (I think with ctypes? I can't remember).

It was just for fun, but in the end even ignoring the startup costs of all of that, the default performance of python's dictionary was better.