We are a west-coast (San Francisco) firm. Our website, which is
hosted by Verio, has been down since at least this morning. The
problem appears to be a "proactive shutdown" by Verio's data center in
Boca Raton, Florida. Since Verio provides hosting services for
millions of customers, I expect that this action in Boca Raton is
having internet-wide implications and affecting many, many businesses.
The Verio website shows the following status messages:
10/24/2005 7:00 pm ET update
To Verio shared hosting customers:
Verio is continuing to work with the building/facilities management on
an on-site technical assessment of the generator system. Our team will
be at work throughout the evening as needed to continue work on power
and service restoration. A new status update will be posted here at
8:00 pm ET.
Many of our shared hosting customers have been impacted by Hurricane
Wilma's effects in the south Florida area. Our Boca Raton data center
facility did experience power loss this morning, and backup
generators, which were tested just four days ago and found to be
working properly, are not operational at this time. At this time, we
have proactively powered down servers to minimize impact to our shared
servers located in the Boca Raton data center. We will restore these
servers once power comes back online. In the meantime, we are
aggressively working with the local power company to assess when power
will be restored, and we are working with the building facilities
management to restore service to backup generators as well. We will
continue to provide updates and any time estimates on this site as we
have them.
Thank you for your patience during this time. We understand the impact
of this action to your business, and the Verio Disaster Recovery team
is diligently working to have power and service restored as quickly as
possible.
----------------------
Our website came up sometime this morning. The following message was
posted on Verio's status page:
10/25/2005
10:00 am ET update
To Verio's domestic and international hosting and colocation customers
impacted by Hurricane Wilma and the Florida data center service
interruption:
Our Boca Raton facility is currently operational on backup generator
power with services in the process of being restored. All servers
either have already been powered on or are in the process of coming
online. We are continuing to monitor any issues during this process in
an effort to minimize impact to the servers and data. We are also
continuing to monitor the restoration of primary power service to
minimize any impact should the facility transfer over from generator
power to primary municipal power automatically. At this time, that is
a possibility and one we are watching closely due to possible service
interruption.
Due to weather conditions related to Hurricane Wilma, our south
Florida datacenter lost primary and backup power at approximately
9:40am ET, Monday October 24th. Verio's other datacenters, including
premier datacenters located throughout the US, Europe, and Asia, were
not impacted by this outage.
At approximately 8:40am ET on October 24th, Verio lost municipal power
to the south Florida facility. The facility has a system of five
secondary generators which supply power in the event of a power
outage. The secondary backup system began providing power when
municipal power was lost. When issues with the backup system became
apparent, the system was immediately converted to battery backup and
systems operated for as long as possible. After battery backups were
depleted, Verio electively turned off systems in the datacenter to
minimize impact to servers and data.
Due to extremely difficult weather conditions outside the facility,
technicians were unable to effectively access the backup generators
for much of the morning and early afternoon. As technicians were able
to access the system, they began doing an initial assessment, and the
generator vendor was also able to dispatch technicians to assist with
repairs. Due to hazardous conditions, including high winds, power
lines, and debris, the vendor technicians arrived on-site during the
early evening hours, October 24th.
After diagnosing the problem and effecting the appropriate remedy, the
generators were sequentially brought back on line starting at
approximately 1:40 am ET on October 25th. Once the generator plant was
producing the correct voltage and frequency with the requisite
stability, power was switched to the data center's split-system (DX)
computer room air conditioning (CRAC) units, and simultaneously to the
chilled water plant that feeds the bulk of the facility's cooling
infrastructure. Once the cooling system restored safe operating
temperatures in the raised floor areas, Verio was able to energize the
data center network equipment, before proceeding to begin powering up
the shared hosting and colocation environments at approximately 7:10
am am ET on October 25th.
In preparation for Hurricane Wilma, a full battery of tests was
conducted on the generators and systems. These tests were last
conducted on October 19th, just days prior to Hurricane Wilma passing
over our facility. These tests all came back positive, with no
issues. We are continuing to fully investigate these issues in our
follow up efforts.
Robert Weller
Robert Weller (rweller@h-e.com)