Ask HN: Running UPDATEs in production always feels heavier than it should

3 pointsposted 8 hours ago
by Lucy_Bai

Item id: 46762394

3 Comments

scrapheap

5 hours ago

That fear is good, it means you know that you're taking a risk - I'd worry if you had to touch production and didn't feel that fear.

If you only have the one or two production systems/databases, and you're having to do this more than a couple of times each year, then you could certainly benefit for improving your tooling.

But if you've inherited a large number of production systems, each with their own local database instance. Then implementing new tooling will be a long process that you might not be able to get Management to agree to.

When I have to touch a production database, I usually use the following steps:

0. Follow your organisation's Change Processes (i.e. if you need to put in a change request and get approval then do that)

1. Snapshot the server itself (VMs are great for this)

2. Backup the database (just be careful not to leave that backup anywhere open, and remember to clear it up when you no longer need it)

3. Write the SELECT version of your SQL first (If you need to delete a record from the userSessions table with an ID of 123, then start by writing

  SELECT * FROM userSessions WHERE id=123;
4. If that shows you just the rows you'd expect then convert it to the delete from of the SQL (making sure that you don't change any part of the WHERE clause)

  DELETE FROM userSessions WHERE id=123;
5. Get someone else, who also understands the database, to check your SQL

6. Run your SQL

7. Do all your sanity checks to make sure that production is working as expected.

8. If it does go wrong then you have more ammunition to use when trying to convince Management to spend some resources on improving your tooling for these sort of changes.

necovek

7 hours ago

I apply "better practices and tooling", advice you are not looking for ;)

pestatije

6 hours ago

yeah definitely don't ever touch production like that, use best practices, and tooling

my 2c