A funny IT tale and curry soda

Well here's a funny IT story I thought I would share. A client does an upgrade to their live environment on a Friday night and everything seemed to go OK. Monday is a bank holiday and so it isn't until Tuesday that they realise that no users can log in. A support ticket is logged and investigations take place but it looks like it's a permissions issues at the client end. Turns out they had turned on some new firewall rules on Saturday which killed things.

The next day the same client is confused because the database upgrade script appears to have wiped out all their data back to the beginning of May. When told that this was not possible it then transpired that said client had upgraded an old test database and then copied it over their live production database without having first made a backup of said live database thus wiping out four months worth of real live data. Not a lot that could be done with that state of affairs.

Then someone realises that we have a copy of a client backup from July and so perhaps we could upgrade that and ship it back to them and at least get a couple of months data restored. Sadly it turned out that the client wasn't very good at backups either and the file that we had could not be restored because they had made a technical error when creating the backup in the first place.

It's hard to believe that in this day and age it is still possible for people to upgrade live environments without first making adequate backups and having a rollback procedure in place. This is an IT (and database admin) 101 sort of thing. There is nothing that can be done and sadly the client will have to try and manually recreate four months worth of data. I feel sorry for those guys at the sharp edge of this one but it has made a very amusing story to share round the office. If you think you are having a bad day, just think how bad these guys must feel LOL

In a completed unrelated matter we had a very odd experience tonight. I had ordered a bottle of curry flavoured ramune (that's soda to those that don't know). It did smell vaguely of curry. It tasted like fizzy turmeric and ginger. It was a very strange taste. It wasn't delicious but it wasn't revolting either. Saying that, I won't be buying that particular flavour again!