Request failures to APNs

Incident Report for Pusher

Postmortem

At 19:15 UTC, approximately 50% of requests from Beams to APNs started failing because a certificate was served by APNs that Beams did not trust. The certificate that was signed by GeoTrust Global CA, but this CA certificate has been removed from the ca-certificates package (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962596). Our hosting provider appears to have performed an automated package upgrade of ca-certificates to the version that does not include GeoTrust Global CA.

To fix this problem, we manually added the GeoTrust Global CA certificate to the set of certificates trusted by Beams.

We were alerted to this issue by our automated monitoring system, but the out-of-hours alert threshold caused it to take too long to fire. We have adjusted this threshold so that we are alerted sooner if this issue happens again in the future.

Posted Feb 09, 2021 - 17:04 UTC

Resolved

This incident has been resolved.

Posted Feb 08, 2021 - 21:59 UTC

Identified

Some APNs servers are serving certificates that we are not able to validate with our server's CA certificate store.

We are investigating workarounds.

Posted Feb 08, 2021 - 20:47 UTC

Monitoring

About 50% of our requests to APNs are failing with internal errors.

We retry pusher notification trigger requests, so a higher percentage of trigger requests to our systems should be getting through.

Device registration requests will be retried by the Beams SDKs.

Posted Feb 08, 2021 - 20:26 UTC

This incident affected: Beams.