I Shouldn't Need an LLM to Explain My LLM