Set up an overnight job that sanitizes snapshots of prod data — replacing PII or other sensitive data with randomly generated cruft — and loads that into a pre-prod environment for use in debugging. For easiest troubleshooting, have it preserve primary keys; and have customer service look up the PKs of affected users’ records.

This won’t work for all classes of bug, but helps a lot.

Also helpful: extensive logging, telemetry, and exception capture with not just stack but context; session recordings to video (the service we use for that blanks out user inputs for privacy purposes but lets you see where things went wrong).

25 years ago I worked in a troubleshoot team for some time. The team members all had read authorization for all production files and databases. We did a first analysis of the problem and then send it on to the responsible maintainer without the production data. I had to sign a special NDA to be a member of that team.

[dead]