I imagine operating in the latent space would make the decomposition truly useful/controllable.

It's a lossy compression/decompression problem too.