I've been studying these a bunch for a project in university. Last week I went over the derivation of the ELBO for a couple hours and it was a very fun and elucidating exercise.
Once you give names to the larger mathematical structures and understand them a bit better it becomes quite simple. I wish some of the blogs/papers I'd read had named "Importance Sampling".
The probability notation can be pretty confusing too. Sometimes it's hard to understand the "types" of some variables. But I'm inexperienced.
ChatGPT was surprisingly helpful. If you put in the work to truly understand the where the gaps are in your mental model (which parts aren't completely intuitive), it can do an amazing job filling in the gaps.
Something I've wondered, maybe I should just do it if I can find some time, but... given DeepSeek's nice results on using rendered text as input, I'm wondering if anyone has given serious research efforts towards image-based diffusion methods for text.
As in, instead of all the complexities induced by discrete token generation, just generate the image of the text using standard image diffusion methods, then convert it to text.
If you used a single, monospace font, I bet this would be even pretty efficient, because the OCR problem becomes basically just direct template matching.
But I guess probably there is already a paper out there, I haven't searched. I'd be curious to know if it compares on par with token-based methods.
I've been wondering why we can't skip the entire token embedding step and just feed the model raw Unicode. I suppose the reason for that is efficiency and it's related to your question. Makes one wonder what other steps we can do more efficiently in code than in neurons.
There’s has a ton of work on character or byte level encodings for llms. The problem is you expand your input tokens by 3-4x. Expensive.
Also, you still need token embeddings (I think you might be confused how that works).
I think it would make training take much much longer but the models would be able to tell you how many Rs are in strawberry.
Yes. By the way I think it is a pity that all the tutorials start with token embedding; by teaching LLMs like that, you don't learn why they are necessary.
I’d like to see more of these models.
I’ve been using diffusion Gemma and it is very fast on GPUs in output token/sec.
In the diffusion Gemma whitepaper, they say they could have done better with more time and compute.
Even with those caveats, it is very uses-able as a local model.
good stuff, no mention of confidence tho, recommend having a look at diffusiongemma and others.