Bonsai: A 27B reasoning model on a 16 GB M2 Mac, with Ferrox

3 pointsposted 9 hours ago
by antonellof

1 Comments

antonellof

9 hours ago

Run a 27B reasoning model locally on a 16GB M2 Mac. Ferrox + ternary quantization delivers 5.9GB models without sacrificing much quality.