LLM continuous batching host scheduler in Rust (191k tok/s on 1 vCPU)

2 pointsposted 4 hours ago
by bmartin-systems
(github.com)

1 Comments