Speeding up end-to-end inference with self-improving agents