Georgia Tech IT
Active Incident

Updated a few seconds ago

CEDAR – Temporarily Pause Adding/Writing DataDegraded Performance

Incident Status

Degraded Performance

Components

Pace

Services

Cedar Storage


September 28, 2026 4:56PM EDT
MONITORING

PACE is continuing to expunge snapshots to make space available, and is looking into accelerating the schedule for bringing the new capacity online with the goal of 2 additional PB being available by end of week. At this time, our recommendations for users are still the same.

September 28, 2026 12:58PM EDT
MONITORING

Deletions over the weekend reduced CEDAR's capacity from 93% to 90% and traffic rates are currently low. We continue to ask users to not add data to CEDAR for now, and will have more clear recommendations after meeting with the vendor this afternoon

September 25, 2026 4:29PM EDT
MONITORING

PACE and the CEDAR vendor have identified and begun implementing measures to create additional capacity on CEDAR. More capacity is expected to become available Sunday evening as automated cleanup processes continue to run. We have verified that storage space is beginning to be reclaimed and will continue to closely monitor capacity utilization and system performance as these cleanup activities progress. At this time, these actions are expected to reduce pressure on the system and improve performance for affected users. As a precaution, users should continue to avoid creating, modifying, or transferring large amounts of data unless necessary. Read-only operations remain safe to perform. Additional updates will be provided as more information becomes available

September 25, 2026 1:17PM EDT
MONITORING

We are closely monitoring storage capacity for the CEDAR service, which is approaching its available limits. A capacity expansion is planned for mid-October, but usage is expected to continue increasing before that work is completed. At this time, a small number of users have reported delays when accessing or updating data. Based on current performance trends, additional users may begin to experience similar issues. OIT is working with the vendor to identify options for recovering capacity and reducing the risk of service degradation. We are also requesting that users avoid creating, modifying, or transferring large amounts of data unless necessary to help preserve service stability. Users should not delete data at this time. Impact: Users may experience slower-than-normal performance when accessing, updating, or storing data in CEDAR."

September 25, 2026 12:49PM EDT
MONITORING

Dear CEDAR users, The CEDAR storage system is approaching capacity. The addition of 2.4 PB of new storage is expected by mid-October. When possible, please temporarily stop writing and modifying data to CEDAR while mitigations are implemented. Users should not delete data – pausing writing and modifying data while work is carried out with the vendor is sufficient. Read-only operations are safe to continue. Thank you for helping us keep CEDAR’s performance optimal. Please contact us at pace-support@oit.gatech.edu with questions. We will follow up with additional details on CEDAR’s expansion once they are available. Thank you, The PACE Team

Academic Services

Operational

Access Control

Operational

Audio Visual (AV) Technology

Operational

Buzzcard

Operational

Campus Services

Operational

Chat and Remote Meetings

Operational

Cloud and File Storage

Operational

Data and Reporting

Operational

Data Center

Operational

Email and Calendaring

Operational

Endpoint Infrastructure

Operational

Generative AI

Operational

GTPE

Operational

Identity

Operational

Infrastructure Technologies

Operational

Network

Operational

Pace

Degraded Performance

Printing and Copying Services

Operational

Servicenow

Operational

Student Information Systems

Operational

Other

Operational

Student Services

Operational

External Services

1Password

AWS CloudFront

AWS EC2

Azure

Box

DocuSign

Dropbox

Duo

Honorlock

Instructure (Canvas)

Let's Encrypt

MS Office365

Qualtrics

Salesforce

Sectigo

Zoom

Scheduled Maintenance

Dynamically Populated Microsoft 365 Groups Moving to New Automation (CHG0037944)Planned Maintenance

Schedule

October 1, 2026 8:00PM - 11:00PM EDT

Components

Email and Calendaring

Services

Office 365

Description

Microsoft is retiring the 'memberOf' dynamic membership rule operator on November 3, 2026. A number of GT departments rely on this functionality to dynamically populate Microsoft 365 Groups and Microsoft Teams. The OIT Business Solutions team will be implementing new automation to replace this functionality during this maintenance window. No action is needed by customers, but impacted customers' CSRs/IT staff have been notified of this change (CHG0037944).
PACE October Maintenance PeriodPlanned Maintenance

Schedule

October 5, 2026 6:00AM - October 8, 2026 11:59PM EDT

Components

Pace

Services

Phoenix Cluster, Firebird Cluster, ICE Cluster, Globus Endpoints

Description

WHEN IS IT HAPPENING? PACE's next Maintenance Period starts at 6:00AM on Monday, 10/05/2026, and is tentatively scheduled to conclude by 11:59PM on Thursday, 10/08/2026. PACE will release each cluster (Phoenix, Firebird, and ICE) as soon as maintenance work is complete. WHAT DO YOU NEED TO DO? As usual, jobs with resource requests that would be running during the Maintenance Period will be held until after the maintenance by the scheduler. During this Maintenance Period, access to all PACE-managed compute systems (Phoenix, Firebird, and ICE) and all storage locations except for CEDAR will be unavailable. Please plan accordingly for the projected downtime. WHAT IS HAPPENING? ITEMS REQUIRING USER ACTION/AWARENESS: [Phoenix] Reconfigure Phoenix scratch Lustre device. To reconfigure the device, all scratch data was temporarily migrated to different storage device. Data will be migrated back in this maintenance period to complete this work. While no action is necessary, users are reminded to not store difficult to reproduce or otherwise important files in scratch. [all] Upgrade GPU drivers For V100 and RTX6000 models: nvidia-driver: 580-dkms For all other models: nvidia-driver: 615-open; cuda: 13.4.2 [CEDAR] CEDAR Capacity Expansion Work to expand CEDAR’s capacity by 2.4PB is ongoing. When possible, users are asked to continue to avoid creating, modifying, or transferring large amounts of data unless necessary to help preserve service stability until the addition is complete in mid-October. Users will be notified via email once the new addition is available. ITEMS NOT REQUIRING USER ACTION: [all] Preparatory work for Research Hall expansion and cooling loop redundancy [all] Updates to all operating systems [all] Modify swap usage [ice] Upgrade storage systems versions WHY IS IT HAPPENING? Regular maintenance periods are necessary to reduce unplanned downtime and maintain a secure and stable system. WHO IS AFFECTED? All users across all PACE clusters. WHO SHOULD YOU CONTACT FOR QUESTIONS? Please contact PACE at pace-support@oit.gatech.edu with questions or concerns. You may read this message on our blog. Thank you, The PACE Team
Welcome to Georgia Tech's IT Service Status Page Don't see your issue posted here? Let us know! Administrative Services Center Email: support@oit.gatech.edu Phone: 404-385-1111 Location: Atlanta Campus, Clough Building Room 215