What is the worst database accident that happened to you in production? [closed]

Question

For example: Updating all rows of the customer table because you forgot to add the where clause.

1. What was it like, realizing it and reporting it to your coworkers or customers?
2. What were the lessons learned?

Answer 1

I think my worst mistake was

truncate table Customers
truncate table Transactions

I didnt see what MSSQL server I was logged into, I wanted to clear my local copy out...The familiar "OH s**t" when it was taking significantly longer than about half a second to delete, my boss noticed I went visibily white, and asked what I just did. About half a mintue later, our site monitor went nuts and started emailing us saying the site was down.

Lesson learned? Never keep a connection open to live DB longer than absolutly needed.

Was only up till 4am restoring the data from the backups too! My boss felt sorry for me, and bought me dinner...

Answer 2

I work for a small e-commerce company, there's 2 developers and a DBA, me being one of the developers. I'm normally not in the habit of updating production data on the fly, if we have stored procedures we've changed we put them through source control and have an officially deployment routine setup.

Well anyways a user came to me needing an update done to our contact database, batch updating a bunch of facilities. So I wrote out the query in our test environment, something like

update facilities set address1 = '123 Fake Street'
    where facilityid in (1, 2, 3)

Something like that. Ran it in test, 3 rows updated. Copied it to clipboard, pasted it in terminal services on our production sql box, ran it, watched in horror as it took 5 seconds to execute and updated 100000 rows. Somehow I copied the first line and not the second, and wasn't paying attention as I CTRL + V, CTRL + E'd.

My DBA, an older Greek gentleman, probably the grumpiest person I've met was not thrilled. Luckily we had a backup, and it didn't break any pages, luckily that field is only really for display purposes (and billing/shipping).

Lesson learned was pay attention to what you're copying and pasting, probably some others too.

Answer 3

A junior DBA meant to do:

delete from [table] where [condition]

Instead they typed:

delete [table] where [condition]

Which is valid T-Sql but basically ignores the where [condition] bit completely (at least it did back then on MSSQL 2000/97 - I forget which) and wipes the entire table.

That was fun :-/