Show HN: Entropy-Guided Loop – How to make small models reason

33 pointsposted 4 days ago
by andrewmonostate

3 Comments

mountainriver

4 days ago

Deep Entropix vibes

andrewmonostate

3 days ago

Thanks for bringing this up! Good catch on the similarities! Yes, both use entropy/uncertainty to allocate compute intelligently.

From what I understand, Entropix is an entropy-aware decoder - it monitors token entropy during generation and dynamically adjusts sampling or spawns parallel CoT branches at high-uncertainty points. It's a decoding-time intervention.

My approach doesn't touch decoding at all. I:

1. Generate normally (standard sampling)

2. Capture logprobs + top-k alternatives

3. Check if perplexity/entropy/confidence triggers exceed thresholds

4. If yes, do ONE refinement pass with an "uncertainty report" showing the model exactly which tokens were uncertain + their alternatives + context

The key difference: Entropix steers the ship while sailing; my loop reviews the voyage log and decides whether to make one correction pass. No branching, no custom samplers, deterministic cost (0 or 1 extra pass).

They're actually complementary - you could use Entropix entropy-aware sampling for initial generation and still apply a refinement loop afterward. Same underlying signal (entropy), different control points! The result of combining both should be outstanding! I will test it soon.