Emergency Network Updates
Hurricane Sandy - Final Planned Update - 8:40 AM EST - 11/03/12
Valued Zayo Customer,
Commercial power has been restored to the 60 Hudson Street building in New York, and all of the areas occupied by Zayo within the building have successfully transferred back to commercial power. At this time, all Zayo systems within 60 Hudson Street are functioning normally, inclusive of networking equipment, cooling equipment, and AC / DC power backup systems. As the commercial power outage at 60 Hudson Street was the last remaining network impairment of significance related to Hurricane Sandy, this will be the final planned update that will be distributed related to the storm event.
A final summary of Zayo’s major facilities in the affected area is as follows:
- 111 8th Avenue, New York: Commercial power was restored to the building on this afternoon, and the Zayo areas on both the 3rd floor (Suites 303 and 311A) and 7th floor are operating on commercial power with emergency generator backup. Power has remained stable since throughout the evening, and all cooling and additional backup systems (DC power plants and Uninterruptible Power Supplies) are functioning as expected. All systems are operating normally and no further work is planned at this location aside from routine maintenance of the generators which is expected to be completed on Saturday morning. A single customer experienced impact for 2 hours related to the storm event.
- 60 Hudson St, New York: Commercial power was restored to the building and sequentially activated within the building on a floor by floor basis, beginning at approximately 5:30am Eastern time on Saturday morning. All Zayo facilities within the building are operating normally with emergency generators fueled and capable of restoring AC power if there are any fluctuations in the commercial power feeds. No impact to customer services was experienced at this site related to the storm and associated absence of commercial power.
- 165 Halsey St, Newark: Facility remains on AC power and stable at this time, all systems are operating normally. No impact to customer services was experienced at this site related to the storm and associated absence of commercial power.
- 75 Broad St, New York: Power was restored to 75 Broad in the late afternoon, and the Zayo suites have returned to normal operation with battery backup systems nearing full charge after several days of discharging. All networking equipment is operating normally and as customer equipment is restored, customers are encouraged to contact the Zayo NCC at 866-236-2824 if any issues are encountered with Zayo provided services.
Regarding fiber restoration efforts, only minor issues remain which are affecting a limited number of Fiber to the Tower locations in Pennsylvania and individual customer services in isolated areas in the New York metropolitan area with a high level summary as follows:
Pennsylvania
o Philadelphia Metro area –
§ A limited number of fiber cuts and power outages remain impacting the Fiber to the Tower (cellular backhaul) network, though great progress was made in normalizing this section of the network between power restoration by the utilities and the repair of additional fiber cuts. Zayo will continue to work directly with the wireless providers in the region to prioritize remaining cell sites to complete repairs.
New York
o New York City region
§ Only isolated fiber issues remain, primarily those identified as power is restored to end user locations; any customers suspecting fiber connectivity issues after returning their equipment to service should contact the Zayo NCC at 866-236-2824 for troubleshooting assistance.
If any customers have services that are currently impacted and not currently being addressed by Zayo personnel, please report those to the NCC at 866-236-2824 or via email at NCC@zayo.com so we can resolve those issues as quickly as possible.
All of us at Zayo thank you for your continued patience as we worked through those issues impacting or potentially impacting the services of our customers. We understand that you rely upon us to support your mission critical communication infrastructure, which is of the greatest importance in times of emergency such as this. If at any time you all have suggestions as to how we might support your services more effectively or if you have concerns with any aspect of our support then please let me know.
Best regards,
Greg Hadlock
SVP, Network Control Center | ZayoBandwidth
Hurricane Sandy - Final Planned Update - 7:58 AM EST - 11/03/12
Valued Zayo Customer -
Zayo is pleased to report that all remaining major network impairments were resolved within the past 24 hours, making this the final planned update that will be distributed related to Hurricane Sandy.
During the past day, power was restored to approximately one million subscribers, dropping the number of consumers without power from 4.5 million Thursday afternoon to 3.5 million late Friday. A view of the remaining outages by state is provided below, and the current situation report for Hurricane Sandy as well as future versions can be found on the Department of Energy website here: http://www.oe.netl.doe.gov/named_event.aspx?ID=67
From a network perspective, the most significant progress made by the power utilities on Friday was the restoration of power in the lower and central Manhattan area - power was restored to the major telecommunications facilities of 111 8th Avenue and 60 Hudson Street, both of which were transferred back to commercial power without issue. Elsewhere in the Zayo network, power utilities completed the repairs necessary for Zayo to replace damaged fiber spans in Pennsylvania and New Jersey, bringing closure to those remaining fiber cuts that were impacting multiple unprotected services.
A current status of Zayo’s major facilities is as follows:
- 111 8th Avenue, New York: Commercial power was restored to the building on this afternoon, and the Zayo areas on both the 3rd floor (Suites 303 and 311A) and 7th floor are operating on commercial power with emergency generator backup. Power has remained stable since throughout the evening, and all cooling and additional backup systems (DC power plants and Uninterruptible Power Supplies) are functioning as expected. All systems are operating normally and no further work is planned at this location aside from routine maintenance of the generators which is expected to be completed on Saturday morning. A single customer experienced impact for 2 hours related to the storm event.
- 60 Hudson St, New York: Commercial power was restored to the building at approximately 5:30am Eastern time on Saturday morning, and all Zayo facilities within the building are operating normally with emergency generators fueled and capable of restoring AC power if there are any fluctuations in the commercial power feeds. No impact to customer services was experienced at this site related to the storm and associated absence of commercial power.
- 165 Halsey St, Newark: Facility remains on AC power and stable at this time, all systems are operating normally. No impact to customer services was experienced at this site related to the storm and associated absence of commercial power.
- 75 Broad St, New York: Power was restored to 75 Broad in the late afternoon, and the Zayo suites have returned to normal operation with battery backup systems nearing full charge after several days of discharging. All networking equipment is operating normally and as customer equipment is restored, customers are encouraged to contact the Zayo NCC at 866-236-2824 if any issues are encountered with Zayo provided services.
Regarding fiber restoration efforts, current updates are as follows:
Pennsylvania
o Bethlehem –
§ While power utilities did not fully complete the repair between Bethlehem and Philadelphia on Friday as expected, the utility made enough progress that Zayo was able to replace the majority of the damaged cable on a permanent route, and complete the span with a temporary restoration. All services on this route have restored and full redundancy restored to SONET and Ethernet rings in the area. Zayo will complete a permanent repair of the fiber in a coordinated maintenance window with affected customers notified via Zayo’s standard maintenance process. There is no perceived risk related to the temporary fiber restoration at this time, and final repairs by the power company are expected to be completed not later than Monday.
o Philadelphia Metro area –
§ A limited number of fiber cuts and power outages remain impacting the Fiber to the Tower (cellular backhaul) network, though great progress was made in normalizing this section of the network between power restoration by the utilities and the repair of additional fiber cuts. Zayo will continue to work directly with the wireless providers in the region to prioritize remaining cell sites to complete repairs.
New Jersey
o Madison –
§ The power company completed the repair of the pole line infrastructure, and Zayo fiber repairs were completed this afternoon, restoring all affected unprotected services
New York
o New York City region
§ Only isolated fiber issues remain, primarily those identified as power is restored to end user locations; any customers suspecting fiber connectivity issues after returning their equipment to service should contact the Zayo NCC at 866-236-2824 for troubleshooting assistance.
If any customers have services that are currently impacted and not currently being addressed by Zayo personnel, please report those to the NCC at 866-236-2824 or via email at NCC@zayo.com so we can resolve those issues as quickly as possible.
All of us at Zayo thank you for your continued patience as we worked through those issues impacting or potentially impacting the services of our customers. We understand that you rely upon us to support your mission critical communication infrastructure, which is of the greatest importance in times of emergency such as this. If at any time you all have suggestions as to how we might support your services more effectively or if you have concerns with any aspect of our support then please let me know.
Best regards,
Greg Hadlock
SVP, Network Control Center | ZayoBandwidth
4500 S 129th East Avenue | Suite 132| Tulsa, OK 74134
Direct: 303.731.6662 | ghadlock@zayo.com|LinkedIn
Join the Network Operations Group on LinkedIn
As always, our management team is available 24/7 and a current contact and escalation list can be downloaded here:
Service at 111 8th: Commercial Power Fully Restored - 8:37 PM EST - 11/02/12
Dear Valued Zayo Customer-
As a final update to the commercial power restoration at 111 8th Avenue, commercial power has been restored to all Zayo facilities within the building. At this time, the Zayo equipment and colocation spaces on both the 3rd floor (Suites 303 & 311A) and 7th floor are operating on stable commercial power with emergency generators on standby. The supporting cooling and DC power systems are functioning properly and the Zayo areas within the facility are all operating under normal commercial power conditions.
Unless there are significant changes in status, our next update will be sent later tonight with further details regarding power restoration at 60 Hudson and inclusive of overall network status.
As always, our management team is available 24/7 and a current contact and escalation list can be downloaded here:
Service at 111 8th: Commercial Power Restored - 7:24 PM EST - 11/02/12
Dear Valued Zayo Customer-
As an update to the commercial power restoration at 111 8th Avenue, commercial power has fully restored and stabilized to the building. At this time, the Zayo equipment and colocation space on the 3rd floor area (Suites 303 & 311A) is operating on stable commercial power with the emergency generator on standby. The 7th floor suite continues to run on generator backup at this time, as the commercial power feed to the 7th floor was disconnected to introduce the temporary 2 megawatt generator as a back-up protection. Zayo will initiate a planned transition this evening to return the 7thfloor suite to commercial power. As demonstrated yesterday, the 7th floor facility is completely supported by backup DC power systems, so no customer impacted is expected during this operation.
As an additional update, preliminary information indicates commercial power will be restored to 60 Hudson Street between 11:30pm and 12:00am tonight (Friday night into Saturday morning). We are currently confirming this information and we will have technical personnel on site when the transition back to commercial power occurs.
As always, our management team is available 24/7 and a current contact and escalation list can be downloaded here:
Service at 111 8th: Commercial Power Restoring - 6:20 PM EST - 11/02/12
Dear Valued Zayo Customer-
This is a real-time communication regarding the status of the 111 8th Avenue facility in New York. As of a few moments ago, commercial power was beginning to restore at the 111 8th Avenue location. At this time, the commercial power is not yet stable and all Zayo facilities continue to run on generator power. Once commercial power becomes stable, the 3rd floor suite will automatically return to commercial power with the backup generator on emergency standby. The larger 7th floor suite, which experienced generator issues yesterday, will continue to operate on generator power until Zayo disconnects the temporary 2 megawatt generator installed last night and restores connectivity to commercial power. This activity is currently planned for this evening but further communications will be sent before this work takes place. The return to commercial power is NOT expected to cause any customer impact, and all Zayo equipment will continue to have both emergency generator and DC power backup systems in the event of commercial AC power failure.
As always, our management team is available 24/7 and a current contact and escalation list can be downloaded here:
Update #7: Service at 111 8th Ave., Suite 734 - 9:08 p.m. EST - 11/01/12
As previously noted, maintenance on fuel supply to the Generator serving the 7th floor was successful and all systems have remained stable for 2+ hours. All cooling systems have been placed on-line and temperatures have returned to normal operating levels. We are pleased to report that despite the events of the day, no customer experienced a single minute of downtime for either network or colocation services at 111 8th Ave.
Zayo will move forward with the installation of the backup generator unit as an added precaution. The unit is scheduled to arrive onsite later this evening and should be installed during the night. This is added protection for the 5-hour battery protection at this location.
ConEd estimates restoration of commercial power at 111 8th Ave location during the Sunday to Monday timeframe. As such, Zayo will maintain added staffing levels both at the site as well as in the NCC to monitor any events.
We appreciate our customers’ patience as we continue to work through to full restoration from Hurricane Sandy. As always, we will provide updates with any new developments.
Update #6: Service at 111 8th Ave., Suite 734 - 7:15 p.m. EST - 11/01/12
The maintenance to the fuel delivery system was completed successfully; all Zayo systems remained operational during the maintenance as they failed over to battery back-up as expected. All systems are back on the generator at this time. Cooling infrastructure will be brought back on line in a phased manner to ensure the generator remains stable. The first unit was just brought online and the temperature has already decreased eight degrees in the equipment area. All services to Zayo Network equipment in support of customer transport services are stable at this time.
Zayo engineers will remain onsite monitoring the system throughout the evening to ensure the environment remains stable. Zayo will provide another update once the temperatures have returned to normal operating conditions.
ConEd has indicated that commercial power is expected to be brought back on-line at 111 8th Ave in 3-4 days. As further, more detailed information is made available, Zayo will provide additional clarity on timing.
Update #5: Service at 111 8th Ave., Suite 734 - 5:55 p.m. EST - 11/01/12
Zayo's services continue to be stable at this time. The temperature in the Zayo suite is also stable and has not risen over the past few hours. Equinix is planning to make a permanent repair to the fuel delivery system at 5.45pm EST. The repair is intended to restore the generator to full performance and allow the Air Conditioning system to be re-activated.
The generator will be disabled during this repair and Zayo's infrastructure will be transferred to DC battery power. Service will not be affected during this transition. As previously communicated, we have approximately five hours of back up battery support time. The Equinix repair is expected to take approximately 30 minutes.
We will provide further updates once the repair is completed.
Update #4: Service at 111 8th Ave., Suite 734 - 4:34 p.m. EST - 11/01/12
We’re making progress on issues caused by a faulty fuel line that began this morning at our colocation facility at 111 8th Ave, in Suite 734. Temperatures have stabilized and we have a constant supply of AC power at the site. We believe that repairs can be implemented without any interruption to customer services.
We will continue to host a customer bridge on this event until the situation is stable, with dial-in information above.
Temperatures jumped quickly after the initial problem. We took action by opening windows on the upper levels and setting up multiple fans to circulate air. The general temperature has been stabilized at 95-100 degrees. This has bought us time to implement emergency plans for back-up power sources and to isolate the root cause of the fuel supply issue. Given the relative stability of the current environment and in coordination with Equinix, we've elected not to attempt to bring cooling back on-line so as to avoid any potential collateral impact to Generator operation.
The root cause of the problem with the generator was a faulty isolation valve, allowing air into the fuel supplied to the generator, resulting in sub-optimal operation. The generator was no longer able to support the load of both the networking equipment and the site air conditioning systems. A temporary repair of the isolation valve was attempted, which marginally improved fuel flow. The generator will need to be taken offline to implement permanent repairs and Zayo is coordinating with the landlord to perform permanent repairs. Zayo has a 2 megawatt generator en route to the site that will be parked on the street and connected to the electrical system, supporting both the networking equipment and site cooling. Our current estimate is that this plan will be implemented within the next three to five hours.
We have Electrical Engineers on site prepping implementation of roll-up generator tie-in and have a confirmed implementation plan for tying the generator into electrical infrastructure. We are waiting on timing updates for generator delivery and continue to work through other logistics. In the event of any disruption, Zayo's DC plant has an estimated five hours run-time to support all Zayo network and customer colocation services. With these plans and back-up plans established, we plan to defer a permanent fix until the back-up 2MW roll-in generator arrives and is tied into site.
Update #3: Service at 111 8th Ave., Suite 734 - 2:29 p.m. EST - 11/01/12
At approximately 2:00pm EST, the generator serving suite 734 for Zayo/zColo and Equinix was taken off-line to implement a temporary fix. The intention of the temporary fix was to stabilize fuel pump and flow to generator in an attempt to return portion of cooling on-line. All Zayo network and customer equipment successfully transitioned to Zayo's DC battery plant with no interruption to customers. Zayo's DC plant provides for an estimated 5 hours run-time (Zayo owned - separate form Equinix), should the Generator have to be taken off-line for maintenance or if there should be a failure.
At approximately 2:12pm EST, the generator was reengaged and teams are assessing stability to bring some portion of cooling back on-line.
With mitigation efforts temperatures have stabilized in a range of 95-100 degrees fahrenheit and have been generally unchanged over past 2 hours at those levels.
All Zayo network equipment supporting transport services and customer colocated equipment is stable and operational at this time.
Update #2: Service at 111 8th Ave., Suite 734 - 1:12 p.m. EST - 11/01/12
Power delivery to Zayo network equipment and colocation customers remains in place with no customer outages to this point, however temperatures in the location have risen above 100 degrees. Zayo is pursuing a number of paths to mitigate service impact including working with building owner and Equinix to resolve root cause of issue. As a back-up plan, Zayo has secured a 2MW roll-in generator that is in transit from New Jersey. If building infrastructure is not restored, Zayo's intention would be to tie its 7th floor services and cooling infrastructure directly to this back-up power source and maintain this through full commercial utility restoration. Due to traffic congestion, it is difficult to estimate when the generator will arrive in NYC, but we expect it to take 2.5-3.5 hours to come into the city, and roughly an hour to connect it with Zayo infrastructure.
We are pursuing several paths, however this is a back-stop if other avenues are not achieved.
Further updates will follow as we continue to action across several fronts.
Update #1: Service at 111 8th Ave., Suite 734 - 12:23 p.m. EST - 11/01/12
This is an urgent communication regarding the status of the 111 8th Avenue facility in New York. At this time, there are technical difficulties with the fuel pumping system to a subset of generators within the building, affecting the AC power supplied to Zayo’s space on the 7th floor. While the networking equipment remains powered at this time, the reduced power generation has impacted the cooling systems in this area of the building and the rising temperatures present a risk to networking equipment owned by Zayo and collocated customers.
Zayo is taking emergency measures to mitigate this issue which includes deployment of fans within the facility to eliminate hotspots, powering down any equipment that is not currently supporting live customer traffic, and evaluating the next steps for further load reductions which will include powering down redundant networking equipment and cards to operate in a simplex condition consuming less power. These measures, if required, may impact IP routing performance and result in suboptimal routing, or potential packet loss during peak traffic periods but may be necessary to prevent service outages or otherwise have redundant equipment remained undamaged to expedite recovery following a worst case scenario. The current ambient temperature in the 7th floor space is 93 degrees and slowing rising and we will continue to do everything within our power to prevent or otherwise minimize customer impact.
Please note that this situation currently does NOT impact customers collocated in suites 303 or 311A. We will continue provide updates via the website and via email as further information becomes available.
