July 25, 2020 9:35AM EDT
[Monitoring] Facilities has continued to work on the water main break but all work is not fully completed yet. As of 7/14 water was restored to
Love
MaRC
MRDC
CRC Fire Suppression
As of 7/25 significant progress has been made but not fully completed.
July 14, 2020 12:20PM EDT
[Monitoring] As an update, the expected date of resolution for the water main break is now Friday 7/17/2020. In the interim, all IT services have not been affected by the water main break as a workaround was put in place when the event initially occurred.
July 4, 2020 9:07AM EDT
[Monitoring] As an update, the water main will not be fixed until sometime Monday 7/6 at the earliest. OIT has been able to function business as usual but will continue to monitor the water main break and respond accordingly if there are any problems.
June 29, 2020 10:53AM EDT
[Monitoring] As of Saturday ( 6/27) evening, the water main break was not fixed, but it was isolated.
This meant that water pressure for campus was returned close to normal which in turn allowed OIT to restore HPC 's ( High Performance Computing ) services close to 100% with one research cluster still remaining off as of today, Monday 6/29. These services had been proactively shut down in order to reduce the amount of cooling needed to support the datacenters and all of OIT services.
HPC's service restoration was done on Sunday 6/28 starting at 0900 in the morning and completed as of noon that day.
OIT was also able to return CATV back to normal service as well on Saturday 6/27.
No IT services other than HPC and CATV were affected by the water main break.
This includes but is not limited to mail, Canvas, www.gatech.edu, Banner, enterprise backups, Endpoint, Vlab, and Buzzport, All of these services remained functional and are still functional.
The BCDC and Coda datacenters were not affected either.
Going forward, the water main break is expected to be fixed on Friday, July 3rd during the holiday weekend. OIT will continue to monitor our datacenter temperatures 24x7 and respond accordingly as needed.
If you have questions, please submit a ticket here
https://gatech.service-now.com/technology?id=sc_cat_item&sys_id=360bf344db574050391b9837db9619a0&sysparm_category=05009bc4db934050391b9837db961957
and select Datacenters for the service you are having an issue with
Or you may call 404-894-4669 to reach OIT Datacenter Operations
June 27, 2020 6:13PM EDT
[Identified] Around 1400 today, there was a major water main break at the intersection of Hemphill and Ferst Dr. near the MaRC Building. This caused a significant amount of flooding for the area and threatened damage to buildings in the immediate vicinity such as Center St., the Baker building, and other buildings that are close by.
GTPD and Facilities responded immediately and began working with Atlanta Watershed to address the situation. So, far no major services have been affected other than HPC and CATV. OIT in an effort to proactively avoid the loss of any IT services has shut down a significant portion of the HPC ( High Performance Compute ) load in Rich as well as most of CATV. This was done to reduce the amount of cooling required to support IT services in an effort to lessen the demand on the campus cooling supply ( campus loop ).
While HPC is down, many research jobs will not run and access to existing research will not be available. We apologize for the inconvenience and are working to avoid any further loss of services.
Other IT services have remained up such as Office365 and mail, the Campus Network, www.gatech.edu, Buzzport, Oscar, Banner, Financials, vCenter, Vlab, enterprise networking, enterprise backups , Passport, Identity Management, storage, and others have all remained in service.
If you have questions or concerns please call 404-894-4669 or file a ticket via https://gatech.service-now.com/technology?id=sc_cat_item&sys_id=360bf344db574050391b9837db9619a0&sysparm_category=05009bc4db934050391b9837db961957
and select "Data Centers" for the Service
