Summary

On Sunday, December 5th, 2021 at 00:00:00 UTC, the SSL certificate on the public domain *.pushnotificaiton.pusher.com expired. As a result, all requests for device subscription and notification publishing APIs failed with a “certificate expired” error. Our engineers got paged and started investigating the situation. The issue was promptly identified and the team worked on creating a new certificate. After the new certificate got deployed, the coming requests stopped receiving errors and the system went back to normal which concluded the incident.

That was an internal process failure where we failed to update the SSL certificate on time before it expired. Better monitoring will be applied to this SSL certificate and others. In addition to that, an investigation for using certificate auto-renewal tools is in progress.

Impact:

• Device subscription requests using client SDKs failed during the incident. They get automatically retried for Android and iOS SDKs but not for Web or Safari SDK.
• Notification publishing API requests from the Server SDK failed during the incident.

Timeline:

All times are in UTC on the 5th of December 2021:

• 00:00 UTC the SSL certificate for *.pushnotifications.pusher.com expired
• 00:25 UTC Our engineering on-call team was paged due to integration tests failing, the team immediately started investigating the situation.
• 00:35 UTC After analyzing the Pusher Beams monitoring stack and logs of the integration test, the issue was identified and the team started working on creating a new SSL certificate.
• 02:20 UTC The new certificate was deployed and the system went back to normal.