Active Incident

Updated a few seconds ago

Incident Status

Operational

Components

Academic Services

Services

PACE



May 17, 2024 3:53PM EDT
[Monitoring] We have not experienced issues since Wednesday, May 15th, and continue to monitor scheduler performance. Mitigations are in place to ensure services will restart as needed to continue processing jobs.

May 13, 2024 2:54PM EDT
[Identified] Intermittent failures of the Phoenix scheduler.

May 13, 2024 2:33PM EDT
[Identified] We are experiencing intermittent failures of the Slurm scheduler on Phoenix. Jobs may start but fail to actually run, experience longer-than-normal completion times, encounter other failure modes. We are investigating the issue and will update as we learn more.

Incident Status

Degraded Performance

Components

Web Hosting

Services

Plesk, WAF - Web Application Firewall



April 24, 2024 8:47AM EDT
[Monitoring] OIT is continuing to monitor the situation, and there have been no significant updates yet. Support teams are still working with the vendor to get a more permanent fix to the issue.

April 16, 2024 3:35PM EDT
[Monitoring] While the workaround to address the WAF issue on the the Plesk Webhosting complex on 4/5 did restore service, it unfortunately also is causing an IP forwarding issue which is affecting the functionality of some of the sites. Support is continuing to work with the vendor to restore the WAF to service so that the webhosting complex can be brought back to its original state.

April 12, 2024 2:51PM EDT
[Monitoring] The workaround that was put in to restore connectivity to plesk hosted websites continues to be working. However, a permanent solution to the underlying causing is still being investigated and worked on with the vendor. Once a working solution is found, tested, and implemented, this issue will be cleared . Until then, we will continue to run off the workaround, and support is continuously monitoring the situation for any new issues that could spring up from the workaround.

April 6, 2024 12:34AM EDT
[Monitoring] The workaround was successful, and sites should be accessible once again. Support is continuing to work with the vendor to address the underlying issue for a more permanent fix.

April 5, 2024 11:33PM EDT
[Identified] Support is continuing to work with the vendor to try and fix the issue with the WAF. However, network team is currently getting ready to implement a workaround to restore connectivity to the affected sites as a temporary measure until a fix for the WAF issue can be found. Once the workaround is implemented, approximately 30 minutes of down time on the hosted sites is expected.

April 5, 2024 1:51PM EDT
[Identified] Staff are still working with the vendor to resolve the WAF issue. If students need to access Canvas, they can reach it by going to https://gatech.instructure.com/

April 5, 2024 11:06AM EDT
[Identified] The vendor's patch was unsuccessful in resolving the issue and will rollback their patch. Efforts are being made to prevent any downtime, but it is possible that there may be a brief outage of up to 10 minutes. Support is continuing to work with the vendor to troubleshoot the problem.

April 5, 2024 9:23AM EDT
[Identified] The vendor is applying a patch to address the communications issues with the Web Application Firewall. Efforts are being made to prevent any downtime, but it is possible that there may be a brief outage of up to 10 minutes.

April 5, 2024 8:33AM EDT
[Identified] Staff have discovered a communications issue between the Plesk hosts and the WAF (web application firewall) that is causing sites hosted on Plesk to be slow or unreachable. Staff are continuing to work with the vendor to resolve this issue. Sites affected include but are not limited to: buzzport.gatech.edu canvas.gatech.edu crc.gatech.edu ctl.gatech.edu dining.gatech.edu diversity.gatech.edu finance.gatech.edu honors.gatech.edu hosting.gatech.edu housing.gatech.edu ipat.gatech.edu

April 5, 2024 8:00AM EDT
[Investigating] Several sites hosted on the webplesk are unreachable due to issues with the WAF (web application firewall) including but not limited to : buzzport.gatech.edu canvas.gatech.edu crc.gatech.edu ctl.gatech.edu dining.gatech.edu diversity,gatech.edu housing.gatech.edu ipat.gatech.edu Staff are continuing to investigate with the vendor to resolve this issue.

April 4, 2024 4:58PM EDT
[Investigating] Several sites hosted on the webplesk are unreachable due to issues with the WAF (web application firewall) including but limited to : canvas.gatech.edu crc.gatech.edu ctl.gatech.edu dining.gatech.edu diversity,gatech.edu housing.gatech.edu ipat.gatech.edu

April 4, 2024 2:29PM EDT
[Monitoring] Service is currently restored and support is monitoring the situation. If you are continuing to experience issues, or if you have questions or concerns, please file a ticket via services.gatech.edu or contact the Administrative Services Center at 404/385-1111.

April 4, 2024 11:50AM EDT
[Identified] Related to the previous issue, web-plesk53 is now serving incorrect SSL certificates to sites hosted on the server. Support is working to resolve the issue.

April 4, 2024 9:49AM EDT
[Monitoring] Several sites hosted off of web-plesk53 were unreachable due to issues with the WAF (web application firewall) including: -Housing.gatech.edu, textbooks.math.gatech.edu, -undergradresearch.gatech.edu, -faset.gatech.edu, -transitionprograms.gatech.edu, -policylibrary.gatech.edu, -policies.gatech.edu, -alan.ece.gatech.edu -gtcad.gatech.edu -hasler.ece.gatech.edu Service is currently restored and support is monitoring the situation.

Academic Services

Operational

Campus Services

Operational

Campus Services - ITG

Operational

Campus Audio Visual Services

Operational

Email and Calendaring

Operational

Identity

Operational

Network

Operational

Web Hosting

Operational

Enterprise Data Services

Operational

IT Service Management

Operational

VideoConferencing

Operational

GTPE

Operational

Cloud Storage

Operational

External Services

Scheduled Maintenance

Schedule

May 30, 2024 7:00AM - 9:00AM EDT

Components

Email and Calendaring

Services

Microsoft Exchange Online

Description

WHAT IS HAPPENING? The Collaboration and Productivity Solutions team in OIT will be enhancing Georgia Tech's email security by updating email domains from their current SPF configuration of "SoftFail" to "Fail". The domains that will be updated are: • edm.gatech.edu • erp.gatech.edu • itg.gatech.edu • pace.gatech.edu • resnet.gatech.edu • services.gatech.edu • vtc.gatech.edu • mail.gatech.edu • oit.gatech.edu • auxtech.gatech.edu • is.gatech.edu • ts.gatech.edu • itsea.gatech.edu • security.gatech.edu • tag.gatech.edu • wrecktechs.gatech.edu Changing the SPF record from "SoftFail" to "Fail" has the potential to impact email deliverability. Email sent from unauthorized mail servers may be marked as junk by the receiving mail server or rejected outright. (More information on SPF is available in ServiceNow: https://gatech.service-now.com/home?id=kb_article_view&sysparm_article=KB0025892.) WHO IS AFFECTED BY THIS CHANGE? Only email sent from the domains listed above will be impacted if the sending service has not been properly authenticated. The Collab team has reviewed our DMARC reports since the previous change from "Neutral" to "SoftFail" to ensure that all legitimate servers and 3rd party mail senders have been properly configured. WHAT DO YOU NEED TO DO AFTER THIS CHANGE? Please report any unexpected email bounces for the domains listed above to the Collaboration and Productivity team at collab-support@gatech.gatech.edu. WHAT IF YOU HAVE QUESTIONS? If there are any questions, please contact the Collaboration and Productivity Solutions team at collab-support@gatech.edu.

Schedule

June 11, 2024 7:00AM - 9:00AM EDT

Components

Email and Calendaring

Services

Microsoft Exchange Online

Description

WHAT IS HAPPENING? The Collaboration and Productivity Solutions team in OIT will be enhancing Georgia Tech's email security by updating email domains from their current DMARC configuration of p=none to p=quarantine; pct=25. The domains that will be updated are: edm.gatech.edu erp.gatech.edu itg.gatech.edu pace.gatech.edu resnet.gatech.edu services.gatech.edu vtc.gatech.edu mail.gatech.edu oit.gatech.edu auxtech.gatech.edu is.gatech.edu ts.gatech.edu itsea.gatech.edu security.gatech.edu tag.gatech.edu wrecktechs.gatech.edu DMARC (Domain-based Message Authentication, Reporting & Conformance) is an email authentication protocol that is designed to give Georgia Tech the ability to protect our domains from spoofing and other unauthorized use. (More information about DMARC is available in ServiceNow: https://gatech.service-now.com/home?id=kb_article_view&sysparm_article=KB0023372.) When a DMARC policy is set to "quarantine," it means that emails that fail DMARC authentication will be treated as suspicious by the receiving mail server. This can result in the email being sent to the recipient's spam or junk folder, rather than being delivered to their inbox. WHO IS AFFECTED BY THIS CHANGE? Only email sent from the domains listed above will be impacted if the sending service has not been properly authenticated with SPF or DKIM. Additionally, only a percentage of mail will be impacted based on the “pct” or percentage tag value. Remaining mail will receive the lower "none" policy. The Collab team has reviewed our DMARC reports prior to this change to ensure that legitimate mail flow will not be impacted by the change. The team will continue to monitor DMARC reports after the change to ensure there are no mail flow issues. WHAT DO YOU NEED TO DO AFTER THIS CHANGE? Please report any unexpected email bounces for the domains listed above to the Collaboration and Productivity team at collab-support@gatech.gatech.edu. WHAT IF YOU HAVE QUESTIONS? If there are any questions, please contact the Collaboration and Productivity Solutions team in OIT at collab-support@gatech.edu.

Schedule

June 25, 2024 7:00AM - 10:00AM EDT

Components

Email and Calendaring

Services

Microsoft Exchange Online

Description

WHAT IS HAPPENING? The Collaboration and Productivity Solutions team in OIT will be enhancing Georgia Tech's email security by updating email domains from their current DMARC configuration of p=quarantine; pct-25; to p=quarantine; pct-50;. The domains that will be updated are: edm.gatech.edu erp.gatech.edu itg.gatech.edu pace.gatech.edu resnet.gatech.edu services.gatech.edu vtc.gatech.edu mail.gatech.edu oit.gatech.edu auxtech.gatech.edu is.gatech.edu ts.gatech.edu itsea.gatech.edu security.gatech.edu tag.gatech.edu wrecktechs.gatech.edu DMARC (Domain-based Message Authentication, Reporting & Conformance) is an email authentication protocol that is designed to give Georgia Tech the ability to protect our domains from spoofing and other unauthorized use. (More information about DMARC is available in ServiceNow: https://gatech.service-now.com/home?id=kb_article_view&sysparm_article=KB0023372.) When a DMARC policy is set to "quarantine," it means that emails that fail DMARC authentication will be treated as suspicious by the receiving mail server. This can result in the email being sent to the recipient's spam or junk folder, rather than being delivered to their inbox. WHO IS AFFECTED BY THIS CHANGE? Only email sent from the domains listed above will be impacted if the sending service has not been properly authenticated with SPF or DKIM. Additionally, only a percentage of mail will be impacted based on the “pct” or percentage tag value. Remaining mail will receive the lower "none" policy. The Collab team has reviewed our DMARC reports prior to this change to ensure that legitimate mail flow will not be impacted by the change. The team will continue to monitor DMARC reports after the change to ensure there are no mail flow issues. WHAT DO YOU NEED TO DO AFTER THIS CHANGE? Please report any unexpected email bounces for the domains listed above to the Collaboration and Productivity team at collab-support@gatech.gatech.edu. WHAT IF YOU HAVE QUESTIONS? If there are any questions, please contact the Collaboration and Productivity Solutions team in OIT at collab-support@gatech.edu.

Schedule

July 2, 2024 7:00AM - 10:00AM EDT

Components

Email and Calendaring

Services

Microsoft Exchange Online

Description

WHAT IS HAPPENING? The Collaboration and Productivity Solutions team in OIT will be enhancing Georgia Tech's email security by updating email domains from their current DMARC configuration of p=quarantine; pct=50; to p=quarantine; pct=75;. The domains that will be updated are: edm.gatech.edu erp.gatech.edu itg.gatech.edu pace.gatech.edu resnet.gatech.edu services.gatech.edu vtc.gatech.edu mail.gatech.edu oit.gatech.edu auxtech.gatech.edu is.gatech.edu ts.gatech.edu itsea.gatech.edu security.gatech.edu tag.gatech.edu wrecktechs.gatech.edu DMARC (Domain-based Message Authentication, Reporting & Conformance) is an email authentication protocol that is designed to give Georgia Tech the ability to protect our domains from spoofing and other unauthorized use. (More information about DMARC is available in ServiceNow: https://gatech.service-now.com/home?id=kb_article_view&sysparm_article=KB0023372.) When a DMARC policy is set to "quarantine," it means that emails that fail DMARC authentication will be treated as suspicious by the receiving mail server. This can result in the email being sent to the recipient's spam or junk folder, rather than being delivered to their inbox. WHO IS AFFECTED BY THIS CHANGE? Only email sent from the domains listed above will be impacted if the sending service has not been properly authenticated with SPF or DKIM. Additionally, only a percentage of mail will be impacted based on the “pct” or percentage tag value. Remaining mail will receive the lower "none" policy. The Collab team has reviewed our DMARC reports prior to this change to ensure that legitimate mail flow will not be impacted by the change. The team will continue to monitor DMARC reports after the change to ensure there are no mail flow issues. WHAT DO YOU NEED TO DO AFTER THIS CHANGE? Please report any unexpected email bounces for the domains listed above to the Collaboration and Productivity team at collab-support@gatech.gatech.edu. WHAT IF YOU HAVE QUESTIONS? If there are any questions, please contact the Collaboration and Productivity Solutions team in OIT at collab-support@gatech.edu.
Welcome to Georgia Tech's IT Service Status Page Don't see your issue posted here? Let us know! Administrative Services Center Email: support@oit.gatech.edu Phone: 404-385-1111 Location: Atlanta Campus, Clough Building Room 215