Incident with Copilot
Incident
Report for GitHub
Resolved
On July 13, 2024 between 00:01 and 19:27 UTC the Copilot service was degraded. During this time period, Copilot code completions error rate peaked at 1.16% and Copilot Chat error rate peaked at 63%. Between 01:00 and 02:00 UTC we were able to reroute traffic for Chat to bring error rates below 6%. During the time of impact customers would have seen delayed responses, errors, or timeouts during requests. GitHub code scanning autofix jobs were also delayed during this incident.
A resource cleanup job was scheduled by Azure OpenAI (AOAI) service early July 13th targeting a resource group thought to only contain unused resources. This resource group unintentionally contained critical, still in use, resources that were then removed. The cleanup job was halted before removing all resources in the resource group. Enough resources remained that GitHub was able to mitigate while resources were reconstructed.
We are working with AOAI to ensure mitigation is in place to prevent future impact. In addition, we will improve traffic rerouting processes to reduce time to mitigate in the future.
A resource cleanup job was scheduled by Azure OpenAI (AOAI) service early July 13th targeting a resource group thought to only contain unused resources. This resource group unintentionally contained critical, still in use, resources that were then removed. The cleanup job was halted before removing all resources in the resource group. Enough resources remained that GitHub was able to mitigate while resources were reconstructed.
We are working with AOAI to ensure mitigation is in place to prevent future impact. In addition, we will improve traffic rerouting processes to reduce time to mitigate in the future.
Update
Copilot is operating normally.
Update
Our upstream provider continues to recover and we expect services to return to normal as more progress is made. We will provide another update by 20:00 UTC.
Update
Our upstream provider is making good progress recovering and we are validating that services are nearing normal operations. We will provide another update by 18:00 UTC.
Update
Our upstream provider is gradually recovering the service. We will provide another update at 23:00 UTC.
Update
We are continuing to wait on our upstream provider to see full recovery. We will provide another update at 11:00 UTC
Update
The error rate for Copilot chat requests remains steady at less than 10%. We are continuing to investigate with our upstream provider.
Update
Copilot is experiencing degraded performance. We are continuing to investigate.
Update
We have applied several mitigations to Copilot chat, reducing errors to less than 10% of all chat requests. We are continuing to investigate the issue with our upstream provider.
Update
Copilot chat is experiencing degraded performance, impacting up to 60% of all chat requests. We are continuing to investigate the issue with our upstream provider.
Update
Copilot chat is currently experiencing degraded performance, impacting up to 60% of all chat requests. We are investigating the issue.
Update
Copilot is experiencing degraded availability. We are continuing to investigate.
Update
Copilot API chat experiencing significant failures to backend services
Investigating
We are investigating reports of degraded performance for Copilot
This incident affected: Copilot.
