The title is not wrong, but it also doesn't feel correct either. What they do here is they use a pre-trained model to guide the training of a 2nd model. Of course, that massively speeds up training of the 2nd model. But it's not like you can now train a diffusion model from scratch 20x faster. Instead, this is a technique for transplanting an existing model onto a different architecture so that you don't have to start training from 0.

Diffusion models are already being evaluated using pretrained SSL models à la CLIP Score [1]. So it makes sense that one would incorporate that directly into training the model from scratch.

[1] https://huggingface.co/docs/diffusers/en/conceptual/evaluati...

I wonder how well this technique works if the distribution of the training dataset between the diffusion model and the image encoder is quite different, for example if you use DinoV2 as the encoder but train the diffusion model on anime.

Still waiting for a competitive diffusion llm