When Your Chatbot Is So Eager to Help, It Forgot Whose Side It's On