Investigate all problems

I just came into office, a few days after the last release of our product, feeling relaxed and thinking about the new release. Just then my colleague, who was working on introducing a new LDAP backend, and was testing the migration code which he had written came up to me and reported a flaw in the upgrade/migrate of the just released version. As I have been sharing with you guys, we have a well defined process of making a product release with clear steps and forms which are filled up and recorded to help track back when required. I looked up the release records and found that the particular test which would have trapped this flaw was performed and the tester had reported a success on that. Surprise...Surprise...Surprise. Was it an oversight by a tester or was it a flaw in the process. So lets get to the bottom of things...

I've been very inspired by the program "Nat Geo investigates" on the National Geographic channel, where they do stories on investigations carried out for plane crashes, space shuttle crashes, and other such life threatening disasters. Last week, I saw how the investigators (combined from several countries) looked into a Swiss air crash in the Atlantic, spent over $50 million and 4 yours of painstaking work to recover the splintered millions of pieces of the aircraft to get to the bottom of the cause of the crash. Amazing...Its no wonder that the airlines industry is six sigma.

We have been following similar principals, where we analyze each issue, waste, problem and see how it can be fixed by technology, processes and training (in this order) In this particular instance, it turned out that the process form had an item to confirm that the logs are clean after upgrade....Hello...Which logs, how to check if they are clean...How much time has elapsed since the test before you check the log (logs rotate due to automated cron jobs and leave behind an empty log file which will have no errors). Improved the form immediately and retrained the QA team.

Every problem is an opportunity to improve the system which drives your business. Don't waste it.