Root Cause Analysis: Redis Cluster Startup Failures – US3 Cluster

Incident Date: October 20, 2025
Duration: 18:36 UTC – 20:41 UTC
Status: Resolved

Summary

On October 20, 2025, customers experienced a significant outage affecting the US3 cluster.

The disruption began with increased latency and 503 errors before escalating into full service downtime as Redis clusters in the US3 region failed to start successfully during an infrastructure update.

Service was fully restored at 20:41 UTC after engineers identified and resolved the underlying startup issue, confirming stability across all Redis clusters.

Impact

Between 18:36 UTC and 20:41 UTC, customers experienced:

• Major outage in the US3 cluster, impacting message delivery and connection reliability
• Elevated error rates and timeouts across dependent APIs
• Temporary need for customers to switch to alternate clusters to maintain service continuity

No customer data was lost. However, applications relying solely on the affected cluster experienced full downtime for a large portion of the incident.

Root Cause

The outage was caused by a failure of Redis clusters to start correctly following an infrastructure update due to a missing configuration flag upon startup.

An unexpected upgrade to the Docker runtime running our Redis cluster introduced a breaking change that prevented container startup for certain Redis deployments. When replacement Redis instances in the US3 redis-main cluster were launched, they failed initialization checks and repeatedly restarted, rendering the cluster unavailable.

The incompatibility remained undetected until a routine node replacement in the US3 cluster introduced new Redis instances to the cluster.

Detection and Response

Monitoring systems first detected increased error rates and latency at 18:36 UTC, followed by a rise in 503 responses from the affected APIs.

Timeline of events:

• 18:36 UTC – Increased latency and 503 errors observed in US3 cluster
• 19:01 UTC – Engineering began investigation into Redis startup failures
• 19:04 UTC – Incident declared a major outage; mitigation efforts initiated
• 19:59 UTC – Root cause identified and configuration fix applied
• 20:20 UTC – Services operational; monitoring for recovery stability
• 20:41 UTC – All Redis nodes confirmed healthy; incident resolved

Resolution

The engineering team:

• Implemented a temporary fix to the Redis environments ensure Redis instances could initialize successfully
• Blocked further automated replacements in other clusters until validated
• Verified recovery and stability across all Redis clusters

After deployment of the fix, Redis instances in the US3 redis-main cluster started correctly and full service was restored.

Preventative Actions

To prevent recurrence, the team has:

• Rolled out a permanent fix across all Redis clusters used in all regions
• Planned a long-term remediation to modernize Redis image packaging for compatibility with current and future Docker releases

Next Steps and Commitment

We are conducting a broader review of infrastructure upgrade processes to better detect runtime incompatibilities before they impact production.

We apologize for the disruption this incident caused. Ensuring reliability and transparency remains our highest priority, and we continue to strengthen our processes to maintain consistent, predictable service for all Pusher customers.