A/B Testing Could Lead LLMs to Retain Users Instead of Helping Them