In IT, there are always important discussions and investments regarding Disaster Recovery. Just like companies around the world, we take disaster recovery seriously, which is reflected in our thorough investment in things like our data center. Varsity has two data centers. One is located in the central valley near Sacramento and the other is near Salt Lake City. We selected our data centers based on the level of redundancy.
Our primary data center in California has redundant gigabit fiber to the Internet, redundant battery backup and cooling, and 12 one megawatt diesel generators. It’s safe to say that our facility, with its bio-metric scanners and raised floor, is a top facility designed to withstand serious outages and failures.
But no system is impervious to failures, which we were reminded of on Friday morning in regard to our voice services. To our happy surprise, there were no failures at the data center or in our infrastructure. As it turns out, the fiber for our voice service provider was down. It took about eight hours for the provider to identify the issue, which was located somewhere in Sacramento. Yes, the fiber had been damaged. But why?
It was shot. The provider reported that the fiber optic cable in an above ground junction box had been damaged by a bullet.
Well, I can safely say that we did not account for a bullet to be a potential risk, which just goes to show that it is impossible to plan for every scenario. In our case, it tested our risk tolerance for only having one voice service provider. In the coming months we will be adding redundant voice service providers over different connections.
