Show HN: Shimmytok – Pure Rust GGUF tokenizer (no C++, no extra files)

2 pointsposted 7 hours ago
by MKuykendall
(github.com)

2 Comments

holg

5 hours ago

Great work, sounds very promising, i did do some AI with burn and candle and always needed to create/convert the own models for this purpose, this could really help, thanks