Hexawise went down today. I apologize. It was my fault.

By Justin Hunter · Jan 30, 2013

Hexawise was down for all of our users for most of the last hour just now. I apologize for the inconvenience caused. It was entirely my fault. I wanted to give you, our users, an explanation.

When Gmail, Twitter, and Amazon's EC2 service experience outages that impact large percentages of their users, executives at those firm sometimes need a couple days to analyze exactly what wrong and produce a report explaining the failure to their users. No complex or time-consuming analysis is required here. I messed up.

Here's the honest but somewhat embarrassing truth about what happened:

  • When I founded Hexawise, 4 years ago today, I registered the domain name Hexawise.com personally.

  • Each year, we need to pay an annual fee of $10 to keep our domain name renewed.

  • Each year, I pay it.

  • Except this year.

  • This year, I took my eye off the ball when I went on vacation last week to experience the Maha Kumbh Mela (a massive Hindu pilgrimage that takes place at the Ganges river once every 12 years that has been referred to as "the world's largest gathering of humanity").

  • I missed an email from our domain registrar highlighting that payment was due as I was setting off to the Kumbh Mela.

  • Just returning from vacation today, I realized that the site was down, jumped on the phone with Sean Johnson, our CTO, who explained what had happened (and that he didn't have access to the account I use to make the annual payment). I promptly paid it and things returned to normal.

 

 

450px-Third Shahi Snan in Hari Ki Pauri

 

Kumbh Mela (image from Wikipedia commons)

 

Here's what we'll do differently in the future to ensure this won't happen again:

  • I've updated credit card info with our domain name registrar and put this payment on auto-renew.

  • I've put calendar reminders on my calendar and our CFO's for the coming years; we'll both personally proactively confirm that it has been made.

  • I've put the email address of the domain registrar into my VIP email folder to ensure I won't miss any future emails from them.

Again, I apologize for the inconvenience caused. I've let down both you (users of Hexawise), and our engineers (who have kept Hexawise up >99.9% of the time even through hundreds of updates to Hexawise that they've put into production within the last four years). They're regularly going to great lengths to ensure the site is up so that you can generate efficient and effective software tests 24/7/365 using Hexawise. You and our engineers deserve better than a CEO who dropped the ball and brought the site down today. I hope you'll excuse this lapse. I won't make this same mistake again. We will continue to strive to keep site uptime as close to 100% as we possibly can.

 

Justin Hunter

Founder and CEO of Hexawise