When to trust that gut feeling...

Posted by Michael Doyon / on 11/06/2009 11:14:00 PM / /

This week has emphasized some common knowledge that I went against earlier this week. Monday morning I wasn't feeling well but figured I could make it through the day. When I got to work, I was informed of a bug in our production code that I had introduced a few months ago. Though the problem had existed for a few months, it was considered critical and needed to be fixed immediately. Despite not feeling up to par, I began investigating the issue. As my day progressed, I began to feel worse yet I continued to work at the problem. By mid-day, I was nearly complete with a solution one of my team members suggested despite having a gut feeling it wasn't quite right. I couldn't explain how it wasn't right or prove it wasn't the perfect solution so I went with it. I rushed through what I felt was a complete solution and did some quick testing. Everything seemed to work and I needed to go home. I felt like I was beginning to develop a fever.

I quickly deployed the changes to our TEST environment and updated my team on what I had done. I then went home and went to bed. I basically slept through the night and called in sick on Tuesday. My technical lead confirmed that QA had signed off on the hot fix I prepared and formalized the paperwork for deployment. It went out early Tuesday afternoon seeming to have resolved the existing issue.

Fast forward a few days to today...Friday. Whatever bug I had earlier in the week has long since passed. Around 2:45pm today, I received an email depicting an exception that the application I had worked on was causing. Again, it was deemed as critical for billing and payroll purposes. Needless to say, it needed to be fixed immediately. Investigating the bug revealed that the hot fix that was pushed out on Tuesday has now caused this new issue which requires a hot fix. I tried my best to resolve the issue by the end of day but with each fix I added a new error occurred. I certainly wasn't going to push out a hot fix that is known to be flawed and hasn't been tested at 5:00pm on a Friday. The fix will have to wait until Monday.

I think there are a few important lessons to be learned here:
1. Don't be a hero. If you aren't feeling well, just call in sick.
2. Don't work on critical code unless you are thinking clearly.
3. Don't rush critical fixes no matter how trivial they may seem.
4. All code should be tested. Changes resolving a critical issue MUST be tested thoroughly! At the very least, test the fix itself and perform some basic regression testing. A full regression test should be performed if possible.
5. Trust your gut! If you have a bad feeling about something, there is likely a reason for it.