Yesterday, the backbone internet provider to our primary Network Operations Center (NOC) in Minneapolis, MN, experienced extensive technical malfunctions. This resulted in a service outage for that NOC for much of the day. While several of our services, including FlashForm SSL, CompGuide XML and our corporate email system, remained largely operable through our Dallas NOC, the primary website and our WorkCompResearch service lost critical systems and went offline. This situation was not related to the planned server upgrades scheduled to begin next week.
We sincerely apologize for the inconvenience this may have caused.
Our total down time was just about 12 hours. It is the longest outage we've experienced since we started the site in 1999. We know that the internet is not a perfect system. Things break, and networks sometimes don't function as intended. Outages have and will continue to occur, but we do try to learn from every experience.
We learned a few things yesterday that can help us in the future, and I promise that we will be taking steps to minimize future events of this nature. We are modifying our upgrade schedule to include some systems previously not included. This will add more redundancy to better allow us to respond to individual NOC issues. We also will be establishing server capacity at a NOC in the northeast that should give us additional backup capacity and support. These steps should help us better deal with issues as they arise in the future.
WorkersCompensation.com has become a reliable information workhorse for the workers' comp industry. People count on us to be there when they need us. We did not live up to that goal yesterday. But I assure you, we will continue to make every effort not to disappoint. We appreciate your continued support and trust, and look forward to serving you for many years to come.