One cause of "AI" chatbots' accuracy being so poor?

1 pointsposted 6 hours ago
by chrisjj

Item id: 49934256

4 Comments

verdverm

5 hours ago

my hypothesis is they are using flash and quant'd models to keep up with demand in the hardware crunch

You see the same pattern across open weight models and homelab setups, as we try to fit the models on devices while preserving capability and getting reasonable throughput

chrisjj

4 hours ago

Trying to preserve something one cannot reliably measure. I wish you luck!

verdverm

4 hours ago

> cannot reliably measure

there are lots of things you can measure, designing tasks for evals is an art (which I'm trying to learn)

chrisjj

3 hours ago

Sure but I said reliably measure.