Datacenter flooding
Oct 29 2012 08:27:43 PM PT
--------
InterNAP has sent us the following notification:
Please be advised that the NYM-EXT1 facility is experiencing significant flooding in the sub-basement of the 75 Broad Street building as a result of Hurricane Sandy. The flooding has submerged the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level. The available fuel reserves on the mezzanine level are estimated to support customer loads for approximately 5-7 hours. Once this fuel supply has been exhausted the generator will no longer be able to sustain operation and critical customer power loads will be lost. We urge all Internap customers to take necessary remote action, if shutting down equipment in a more graceful fashion is possible before the outage occurs.
The building itself is being evacuated and no remote hands support will be available to assist in any equipment shutdown. Life safety is our number one priority and we are making plans to completely exit the facility. No customer access to the building is possible at this time either. Due to the evacuation, Internap will not be able to provide any exact updates on when the fuel will be exhausted and critical customers loads lost, but as noted, we believe it will take place in approximately 5-7 hours from now.
We hope that flood waters will recede enough that generators can function and/or that utility power can be restored before this happens, but we can't tell if that will be the case.
If a complete power outage takes place, our managed services should not be significantly affected beyond the downtime. This is because all are configured to come back online properly when power is restored, and managed game servers (with the exception of Minecraft servers, which are snapshotted hourly) rarely write to disk, meaning that disk corruption is unlikely. However, unmanaged services would also go down. If you run an unmanaged VDS or machine, we recommend that you back up critical files (such as databases) that might be writing to disk at the time of any later abrupt shutdown, and that you consider stopping those services in advance to be safe.
Oct 29 2012 08:27:43 PM PT
--------
InterNAP has sent us the following notification:
Please be advised that the NYM-EXT1 facility is experiencing significant flooding in the sub-basement of the 75 Broad Street building as a result of Hurricane Sandy. The flooding has submerged the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level. The available fuel reserves on the mezzanine level are estimated to support customer loads for approximately 5-7 hours. Once this fuel supply has been exhausted the generator will no longer be able to sustain operation and critical customer power loads will be lost. We urge all Internap customers to take necessary remote action, if shutting down equipment in a more graceful fashion is possible before the outage occurs.
The building itself is being evacuated and no remote hands support will be available to assist in any equipment shutdown. Life safety is our number one priority and we are making plans to completely exit the facility. No customer access to the building is possible at this time either. Due to the evacuation, Internap will not be able to provide any exact updates on when the fuel will be exhausted and critical customers loads lost, but as noted, we believe it will take place in approximately 5-7 hours from now.
We hope that flood waters will recede enough that generators can function and/or that utility power can be restored before this happens, but we can't tell if that will be the case.
If a complete power outage takes place, our managed services should not be significantly affected beyond the downtime. This is because all are configured to come back online properly when power is restored, and managed game servers (with the exception of Minecraft servers, which are snapshotted hourly) rarely write to disk, meaning that disk corruption is unlikely. However, unmanaged services would also go down. If you run an unmanaged VDS or machine, we recommend that you back up critical files (such as databases) that might be writing to disk at the time of any later abrupt shutdown, and that you consider stopping those services in advance to be safe.
Comment