You are here

Home

Microsoft Service Issues

Mon, 11/25/2024 - 8:15am -- David Savoie

UPDATE Nov 26, 2024, 7:30 AM

Everything should be restored and working correclty.  Contact the IT Service Desk if you continue to have problems.

 

Microsoft is reporting severice degradation with several of its cloud based system which may affect desktop applications as well  Services affected include:

  • Exchange Online
  • Microsoft 365 suite
  • Microsoft Power Automate
  • Microsoft Purview
  • Microsoft Teams
  • SharePoint Online
  • Universal Print


Current status
Nov 25, 2024, 7:30 AM CST
We've isolated the root cause and have developed a patch, which we're testing on a subset of components before full deployment. We're making sure that once we deploy the script, we're able to ensure a full recovery.

Nov 25, 2024, 8:27 AM CST
We’re monitoring the progress of the fix, which has been deployed to approximately 60% of the affected environments. We’re continuing our manual restarts on the remaining impacted machines.

We will continue to update as more information becomes available.

Nov 25, 2024, 11:37 AM CST

We're continuing to reroute traffic to alternate infrastructure and have reinitiated targeted server restarts to ensure the fix takes effect as expected. We're monitoring to confirm the restarts proceed successfully. We don't yet have an estimated time to resolution; however, we'll provide one as soon as it becomes available.

Nov 25, 2024, 1:15PM CST

Our mitigative actions haven't provided relief as expected, and a portion of infrastructure remains in an unhealthy state. We determined that some of the targeted server restarts did not succeed due to processing issues, which are under investigation. We’re currently focused on spreading traffic to healthy infrastructure, and we're seeing some recovery.

Nov 25, 2024 3:00PM CST

Services are showing significant signs of restoration following our infrastructure optimizations to redistribute traffic and restart affected servers, upwards of 90 percent recovery as we continue to increase processing capability. We're continuing to perform appropriate optimizations on the subset of remaining affected machines to resolve the issue for all users.

Nov 25, 2024 4:00 PM CST

We identified a change in the environment that resulted in an influx in request retries routed through affected servers. Our optimizations, which enhanced the infrastructure's processing capabilities, continue to provide incremental relief. We're monitoring the service and continuing our work to perform any follow-up actions or opening additional workstreams needed to fully resolve the problem. We understand the significant impact of this event to your organization, we're treating this issue with the highest priority, and we're working to provide relief as soon as possible.

Nov 25, 2024 6:30PM CST

The service is continuing to incrementally recover. We're forecasting that impact will be fully remediated by Monday, November 25, 2024 at 9:00 PM CST , though we're closely monitoring our progress and will provide an updated timeline should the estimate change.

Nov 25, 2024, 4:59AM

From monitoring service telemetry, most users should now experience relief. We’ve completed our optimizations and we're continuing our period of extended monitoring to ensure the availability remains stable.