On September 1, 4Degrees experienced an extended service availability incident caused by infrastructure failures within Google Cloud Platform (GCP).
We first detected customer impact at approximately 10:07 AM CT. Service recovered briefly from approximately 10:15–10:24 AM CT, and from 10:54–11:42 AM CT before becoming unavailable again. We fully restored service at approximately 1:33 PM CT.
We know customers rely on 4Degrees throughout their workday, and this level of disruption falls short of the reliability we expect to provide. We’re sorry for the impact.
What happened
Google Cloud experienced a significant networking incident affecting infrastructure in its us-central1 region, where portions of 4Degrees' infrastructure operate, along with several other regions. Google has not indicated that all of those incidents shared a common root cause.
Google reported elevated packet loss and errors affecting a broad set of services, including Cloud Run, Cloud SQL, Virtual Private Cloud, Cloud NAT, Google Kubernetes Engine, Compute Engine, and others. Google’s preliminary analysis found that routine network-fabric maintenance unexpectedly disrupted network capacity within part of the us-central1-b zone. The primary Google incident began at approximately 9:44 AM CT and Google declared all affected products recovered at approximately 1:52 PM CT.
4Degrees is designed with redundant infrastructure so that the failure of an individual component or availability zone does not result in an extended customer outage. That architecture provided some resilience during today’s incident, including periods of temporary recovery, but it did not provide sufficient isolation from the broader Google Cloud disruption.
Our existing failover design relies on a limited number of independent cloud failure domains and assumed a more isolated regional or geographic impact than we experienced today. As a result, we were unable to move the full 4Degrees application stack to unaffected infrastructure quickly enough to maintain continuous service. During the incident, our team also began provisioning infrastructure in an additional tertiary region to restore service through a completely separate failure domain. By the time that environment was operational, however, the affected Google Cloud resources in our existing regions had recovered, allowing us to restore service there first.
Timeline
- ~9:44 AM CT: Google Cloud reports the beginning of the underlying us-central1 networking incident.
- ~10:07 AM CT: 4Degrees detects elevated errors and customer-facing availability issues.
- ~10:15 AM CT: 4Degrees service recovers.
- ~10:27 AM CT: Availability issues return as the underlying infrastructure incident continues.
- ~10:54 AM CT: 4Degrees service recovers.
- ~11:42 AM CT: Availability issues return as the underlying infrastructure incident continues.
- ~1:33 PM CT: 4Degrees service is restored and remains stable.
- ~1:52 PM CT: Google reports that all products affected by its us-central1 incident have recovered.
What we're changing
We are expanding that architecture so that future provider-level failures have a substantially smaller impact on 4Degrees. Specifically, we are expanding our failover footprint across additional independent Google Cloud regions and failure domains, reducing our dependency on any small set of underlying infrastructure.
We are also reviewing the dependencies required for the complete application to operate from secondary and tertiary regions, and adding more rigorous failure testing to verify that we can continue operating when an entire portion of our cloud infrastructure becomes unavailable.
Going forward
Our goal isn't simply to recover more quickly from this exact failure mode. It's to ensure that 4Degrees can continue operating through a much broader range of cloud-provider failures without requiring customers to wait for the underlying provider to recover.
We'll complete the additional redundancy work and validate it through simulated regional failures. We appreciate your patience during today's disruption and apologize again for the impact.
