Announcement

Collapse
No announcement yet.

Datacenter flooding -- Oct 29 2012 08:27:43 PM PT

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Datacenter flooding -- Oct 29 2012 08:27:43 PM PT

    Datacenter flooding
    Oct 29 2012 08:27:43 PM PT

    --------

    InterNAP has sent us the following notification:

    Please be advised that the NYM-EXT1 facility is experiencing significant flooding in the sub-basement of the 75 Broad Street building as a result of Hurricane Sandy. The flooding has submerged the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level. The available fuel reserves on the mezzanine level are estimated to support customer loads for approximately 5-7 hours. Once this fuel supply has been exhausted the generator will no longer be able to sustain operation and critical customer power loads will be lost. We urge all Internap customers to take necessary remote action, if shutting down equipment in a more graceful fashion is possible before the outage occurs.

    The building itself is being evacuated and no remote hands support will be available to assist in any equipment shutdown. Life safety is our number one priority and we are making plans to completely exit the facility. No customer access to the building is possible at this time either. Due to the evacuation, Internap will not be able to provide any exact updates on when the fuel will be exhausted and critical customers loads lost, but as noted, we believe it will take place in approximately 5-7 hours from now.

    We hope that flood waters will recede enough that generators can function and/or that utility power can be restored before this happens, but we can't tell if that will be the case.

    If a complete power outage takes place, our managed services should not be significantly affected beyond the downtime. This is because all are configured to come back online properly when power is restored, and managed game servers (with the exception of Minecraft servers, which are snapshotted hourly) rarely write to disk, meaning that disk corruption is unlikely. However, unmanaged services would also go down. If you run an unmanaged VDS or machine, we recommend that you back up critical files (such as databases) that might be writing to disk at the time of any later abrupt shutdown, and that you consider stopping those services in advance to be safe.

    #2
    Roll on November!




    I'm not insane. I'm just overwhelming!

    ·····••••• Support Cainslair. Donate here!•••••·····
    ·····••••• and get extra options! •••••·····

    Comment


      #3
      Who the hell puts generators in the basement? It's the same stupid idea they had at my work -- which is by the way, 11 feet of standing water in the sub basement and basement. Awesome.

      At least I don't have to go into work for probably a few weeks. But I'm part of the DR team so I will be working regardless

      Comment


        #4
        Originally posted by Shyatic View Post
        Who the hell puts generators in the basement? It's the same stupid idea they had at my work -- which is by the way, 11 feet of standing water in the sub basement and basement. Awesome.

        At least I don't have to go into work for probably a few weeks. But I'm part of the DR team so I will be working regardless
        In nyc not many places you can put the generators unless the building was build for the loads a generator would need for a higher floor.

        Comment


          #5
          Our NYC location went offline at approximately 11:48am EDT (lasting much longer than we expected!), signaling to us that the diesel fuel for InterNAP's generators has been exhausted and that the pumps in the sub-basement are still not working to bring more up. At this time, InterNAP does not know how long it will take before the facility can be brought back online, but as soon as we receive a time estimate from them, we will post it.

          Comment


            #6
            Our NYC location remains offline and InterNAP is working to bring fuel to the generators in order to turn back up the power. They are declining to provide any sort of ballpark ETR at this time.

            Comment


              #7
              If the server is in the city(downtown)..we're screw. the whole city is flooded and with no power.
              Hit 'em like comic books, kaboom! pow pow!

              Comment


                #8
                We got gas!! We are back !!

                Comment


                  #9
                  InterNAP has restored power to our NYC location now (it's back to running on generators, and they say that the fuel supply will be consistent, preventing further outages). Most services at this location are back online, but we're working on a single machine that didn't load Windows properly. If you have a dedicated machine at this location that is not working, please let us know, and we can check it out for you.

                  Comment


                    #10
                    Originally posted by Shyatic View Post
                    Who the hell puts generators in the basement? It's the same stupid idea they had at my work -- which is by the way, 11 feet of standing water in the sub basement and basement. Awesome.

                    At least I don't have to go into work for probably a few weeks. But I'm part of the DR team so I will be working regardless
                    The generators are not in the basement, the pumps to move more fuel up to them is. They probably have a massive storage tank down there.

                    Comment


                      #11
                      InterNAP has related to us the following:

                      The generators have been refueled, and engineers have successfully restored power to the site via generator. Please begin checking your systems now. There is still no customer access being granted at this time so please do not make plans to visit the data center until you have been notified by Internap that building access has been restored.

                      Followed by:

                      Subsequent to power returning via generator as previously notified, the power was lost again briefly. The onsite engineers are investigating and work to stabilize power is ongoing. The NOC will continue to update customers as the work continues, and confirm once Remote Hands services are available.

                      Followed by:

                      As previously notified, the NYM-EXT1 facility has had generator power restored, and engineers have continued to work to stabilize both the generator and additional site infrastructure in support of our customers.

                      At this time site engineers have been able to confirm that the generator is running normally, and is at full fuel capacity. The brief drop in generator that took place has been fully investigated and it is not expected to be a concern moving forward. Additionally site engineers have returned the UPS systems to service and confirmed that the UPS' are operating normally and helping to support critical loads. Site engineers have also returned all CRAC units to service at this time. The local site engineers believe that the generator and UPS systems are operating in a stable fashion at this time and as noted, we have ample fuel on site and are coordinating additional fuel delivery.

                      From what they have said, we don't expect further power outages. However, some of InterNAP's upstreams in NYC are still offline and may go up and down, so you may see higher-than-normal pings or slight packet loss at times as InterNAP has to route around them and compensate.

                      After the last notification, we worked to systematically bring all services back online. All but one of our game server machines are online right now (the customers on the machine that is offline are being contacted separately). Voice servers are also working normally, and VDSes should all be online. If you have a unmanaged dedicated machine of your own at this location and you are not able to connect to it, please contact us ASAP so that we can pull up its console and investigate.

                      We greatly appreciate your patience as we and InterNAP dealt with the damage from this powerful storm! Total downtime for most services was a bit under 15 hours (11:48am EDT through 2:15am EDT). If you require a day's SLA credit for the downtime, please let us know, and we will be happy to issue this.

                      Comment


                        #12
                        I had the server going this morning then it suddenly went empty around 6AM, now I know why:

                        New NYC connectivity loss (but not loss of power)
                        Oct 31 2012 03:37:22 AM PT We lost contact with all our equipment in NYC again at approximately 5:48am EDT this morning (Oct. 31). InterNAP has not sent out an official announcement yet about this problem, but their NOC answered our telephone inquiry by stating that one of InterNAP's other facilities in NYC is without power, and as a result, our facility does not have connectivity. They tell us that the generators are still working properly for us, which means that your service is continuing to run in the background and should simply pop back online when they get the link fixed, continuing as it was at the start of the outage.

                        --------------

                        I have a big conference call this morning I'm preparing for we need someone else to jumpstart us again.....

                        Comment

                        Cain's Lair Forums Statistics

                        Collapse

                        Topics: 26,182   Posts: 269,814   Members: 6,176   Active Members: 2
                        Welcome to our newest member, 28Farrell8.

                        Today's Birthdays

                        Collapse

                        Top Active Users

                        Collapse

                        There are no top active users.

                        More Posts

                        Collapse

                        • Reply to 6 years
                          by Apache Warrior
                          6 Mar 2024, 08:29 AM
                        • Reply to 6 years
                          by Sirex
                          I think there is like a magical time span when bikes become worth a fortune.

                          Yeah thought its now or never haha
                          Just hit 2.5K miles...
                          5 Mar 2024, 04:37 PM
                        • Reply to Hey Guys...It's BrundleFly
                          by Sirex
                          Hey Brundle nice to see your post I remember you very well hows thing with you.

                          Not a busy place here these days but its still up and occasionally...
                          5 Mar 2024, 04:34 PM
                        • Reply to Hell Let Loose
                          by Sirex
                          Hey nice to see you

                          Its not a busy place, least not as busy as it once was but you still get the odd message posted every few weeks...
                          5 Mar 2024, 04:31 PM
                        • Reply to I had Open Heart Surgery!!
                          by Sirex
                          Wow its amazing how well the body can cope with things it deams normal.

                          Hope your recovering well and back to full steam.

                          OH...
                          5 Mar 2024, 04:28 PM
                        Working...
                        X