We are surprisingly close to prompt-to-model in computer vision