Some of you who visited this website Friday might have been greeted with a placeholder page telling you that WorkersCompensation.com was “Being hosted free” at GoDaddy.com. That page also had a button where GoDaddy would help you buy our domain name. Some of you may be wondering how that could happen.
I am wondering the same damn thing.
GoDaddy.com is a registrar. They are one of many companies that license domain names, and they then register them with the internet network at large. They are, essentially, a traffic cop helping people to their destination. GoDaddy was a disruptor in the industry when they came on the scene more than a decade ago. When I first started internet development, way back in 1995, domain names could only be purchased in one place, and it cost $100 to secure a name for 2 years. Changes made by ICAAN, the internet “governing board”, allowed more companies to enter the market, and GoDaddy broke many barriers – most notably price. Today, a domain name costs less than $10 a year, and GoDaddy deserves credit for that result. Today we hold and manage over 250 domain names with them.
So how did Friday’s incident unfold? Before I can get into that, I should give a brief, simplified “Internet 101” so that the story is better understood. I’ll try not to bore you:
Every website has an underlying IP (Internet Protocol) address that is it’s true “location” on the internet. The primary IP for this site is 12.47.46.154 (paste that in your browser address bar and you will see what I mean). These numbers are what computers really use to navigate the network and get you to your desired destination. A registrar is the service that converts domain names to IP addresses via nameservers. A nameserver is created with the registrar and assigned an IP value. For some time, the primary nameserver for this website was ns12.workerscompensation.com. It was assigned a value of 12.47.46.154.
You see where this is headed?
When we create a nameserver, the registrar is supposed to notify an entire network of Domain Name Servers (DNS) of it’s existence. That way, the entire network will know that, when you type in workerscompensation.com, it is assigned to ns12.workerscompensation.com, which can be found at 12.47.46.154 – and voila, here we are. Each server has within it a DNS Zone file, further directing traffic by IP (email,ftp, etc.), which may or may not be on that specific machine.
Now that we have made the internet clearly understood, we are ready to explain why we were hosted free on GoDaddy Friday…..
To handle and support our traffic and related data services, we maintain multiple servers in different geographic locations. This is done for both redundancy and disaster recovery, and has worked well for us for a number of years. The text you are reading today, uploaded from Sarasota, FL, likely came from Minneapolis, MN. The ads to your right, however, came from a server in Dallas, TX. All of our servers back up nightly (with what we call a “tarball” in the web world) to a center in Hartford, CT. We have yet another server in Las Vegas, not synced with any other server and completely unconnected to our network. It simply idles with data from our primary machines, waiting for traffic to be sent its way. It is our “doomsday” location; our “go to” place when something terrible happens elsewhere.
And that is where our troubles began.
For years we have effectively been able to manage our growth and subsequent server upgrades by creating appropriate nameservers and assigning them to new machines. However, in early July when we had a network problem in Minnesota, we discovered to our chagrin that the nameservers pointing to another backup server in Hartford were not working. Subsequent testing with various centers and servers led us to suspect that new nameservers created at GoDaddy were not apparently being registered as such across the DNS system. They didn’t work.
We called tech support a couple times on this. Now, when you call tech support anywhere, you usually get a polite person who will try to ascertain your problem even though they have no clue how to solve it once that is accomplished. Then they put you on hold for 20 minutes while they run around the office trying to find someone who can solve your problem. Those people are called “Level 2 Tech Support” – and you will never, ever get to speak to them. The solution that our Level 1 tech person proposed was for us to purchase GoDaddy’s new Virtual Nameserver program, where we set our domains to their servers and they create fake nameservers that appear in our domain records. Beyond that the issue remained largely unresolved.
I am sure the fact they are selling a service that would fix a problem with the free service is completely coincidental.
There was another option, related to the solution our tech support person proposed. We have the option of pointing our domains to GoDaddy’s master servers, and using their DNS Zone file to direct traffic. The upside to this is powerful: extremely fast and flexible propagation. When we activate a nameserver it can take 24 hours for the DNS to update and make the changes visible to all (assuming it works). Using the registrar zone file allows us to redirect in mere minutes, simply by changing an IP within the record. The potential downside is, if the registrar has a technical problem and goes offline, we might too, since we rely on their servers to provide that redirection.
We debated that option, and decided that such an event is unlikely, due to the size, scope and stability of their operation. Last Wednesday we made the switch. We had only to wait until Friday for the first problem to occur. We have been unable to find out what happened, but a technical problem at GoDaddy caused the “loss” our DNS Zone files for this domain. Without that directional data, our site went offline. Fortunately, all of our revenue services, WorkCompResearch, Virtual Claims Kit and FlashForm SSL, are on other domains and were unaffected. Level 1 tech support, after putting us on hold for some time, informed us that the zone file would be restored within 30 minutes. We specifically inquired if they still held the custom direction data we had programmed in the system. He assured us it would.
It didn’t. It contained default IP info resulting in the “parked free” page some of you saw. We realized this and remedied it as quickly as possible, but the damage was done. So that is how we came to be parked free with offered assistance to sell our domain name.
GoDaddy over the years has brought innovative services to the industry and has been a key in lowering costs for the industry. I like innovators. I like disruptive forces within the economy. However, low cost does not always equate to value, particularly if that service does not work as designed. As pleasant as it may be for us to be greeted by Danica Patrick every time we visit their site, I would much prefer our sites work as intended. That can only happen when our registrar performs their role as required.
This week we will start testing domains with a different registrar, and hope to resolve the overall nameserver issue. I do not present myself as a network expert by any stretch of the imagination. I depend a great deal on what others tell me. Our network centers have been pointing fingers at the registrar, and the registrar has pointed fingers at the network centers. All I know is that something we’ve done for years doesn’t work the way it used to.
Finding the solution for that will be the best value of all.