Georgia Tech IT
Active Incident

Updated a few seconds ago

GlobalProtect VPN - User/Group-based Policy Issues Operational

Incident Status

Operational

Components

Network

Services

Firewall, VPN


March 31, 2025 10:57AM EDT
[Monitoring] The cause has been identified and a fix has been implemented as of 10:55am. Users should now be able to access all resources protected by user-based policies as normal.

March 31, 2025 10:46AM EDT
[Investigating] OIT Network Engineering is aware of issues affecting a subset of GlobalProtect VPN users who are unable to access resources protected by user/group-based firewall policy (also known as User-ID). As a result, many users are not able to access various systems and applications. The cause of the issue is being actively investigated at this time. Further updates will be provided as progress is made.
Phoenix access disabled due to storage issuesOperational

Incident Status

Operational

Components

Academic Services

Services

PACE


March 31, 2025 4:12PM EDT
[Monitoring] As the Phoenix project storage system has stabilized, we have restored login access via ssh and resumed starting jobs. The cost for the jobs running during the performance degradation will not count towards the March usage. The Phoenix OnDemand portal can again be used to access project and scratch space. Any user still receiving a "Proxy Error" should contact pace-support@oit.gatech.edu for an individual reset of their OnDemand session. Globus file transfers have resumed. We have determined that transfers to/from home, scratch, and CEDAR storage were inadvertently paused, and we apologize for any confusion. Any paused transfer should have automatically resumed. The PACE team continues to monitor the storage system for any further issues. We are working with the vendor to identify the root cause and prevent future performance degradation. Please contact us at pace-support@oit.gatech.edu with any questions. We appreciate your patience during this unexpected outage.

March 31, 2025 12:38PM EDT
[Investigating] To limit the impact of the current Phoenix project filesystem issues, we have implemented the following changes to expedite troubleshooting and limit impact to currently running jobs: New Logins to Phoenix Login Nodes are Paused We have prevented new login attempts to the Phoenix login nodes. Users that are currently logged in will be able to stay logged onto the system. Phoenix Jobs Prevented from Starting Jobs that are in the queue but that have not yet started have been paused to prevent them from starting. These submitted jobs will remain in the queue. Jobs that are currently running may experience decreased performance if using project storage. We are doing our best to prioritize the successful completion of these jobs. Open OnDemand (OOD) Users of Phoenix OOD can log in and interact with only their home directory. Project and scratch space are not available. Some users of Open OnDemand may be unable to reach this service and are experiencing "Proxy Error" messages. We are investigating the root cause of this issue. Globus File Transfer Paused for Project Space File transfers to/from project storage on Globus have been paused. Other Globus transfers (Box, DropBox, and OneDrive cloud connectors; scratch; home; and CEDAR) will continue. The PACE team is working as quickly as we are able to diagnose the current issues with support from our filesystem vendor. We aim to resume normal operation of the Phoenix cluster as quickly as we are able. We will continue to share updates as we have them and apologize for this unexpected service outage.

March 31, 2025 9:13AM EDT
[Investigating] Summary: Performance of Phoenix project storage is currently degraded. Details: Two of our MDS (MetaData Servers) rebooted early Monday morning, March 31, and load averages are unusually high on one of them. Impact: Researchers may experience significant slowness in read & write performance on Phoenix project storage until we are able to mitigate the issue. Conda environments located in project storage may be very slow to load (even if the python script to run is located elsewhere) or fail to activate, while attempts to view project storage files via the OnDemand web portal may time out.

Academic Services

Operational

Campus Services

Operational

Campus Services - ITG

Operational

Campus Audio Visual Services

Operational

Email and Calendaring

Operational

Identity

Operational

Network

Operational

Web Hosting

Operational

Enterprise Data Services

Operational

IT Service Management

Operational

VideoConferencing

Operational

GTPE

Operational

Cloud Storage

Operational

Other

Operational

External Services

AWS CloudFront

AWS EC2

Azure

Box

DocuSign

Dropbox

Duo

Honorlock

Instructure (Canvas)

Kaltura

Let's Encrypt

MS Office365

Salesforce

Sectigo

Zoom

Welcome to Georgia Tech's IT Service Status Page Don't see your issue posted here? Let us know! Administrative Services Center Email: support@oit.gatech.edu Phone: 404-385-1111 Location: Atlanta Campus, Clough Building Room 215