Show HN: Shimmytok – Pure Rust GGUF tokenizer (no C++, no extra files)

2 pointsposted 25 days ago
by MKuykendall

2 Comments

holg

25 days ago

Great work, sounds very promising, i did do some AI with burn and candle and always needed to create/convert the own models for this purpose, this could really help, thanks