At 19:15 UTC, approximately 50% of requests from Beams to APNs started failing because a certificate was served by APNs that Beams did not trust. The certificate that was signed by
GeoTrust Global CA, but this CA certificate has been removed from the
ca-certificates package (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962596). Our hosting provider appears to have performed an automated package upgrade of
ca-certificates to the version that does not include
GeoTrust Global CA.
To fix this problem, we manually added the
GeoTrust Global CA certificate to the set of certificates trusted by Beams.
We were alerted to this issue by our automated monitoring system, but the out-of-hours alert threshold caused it to take too long to fire. We have adjusted this threshold so that we are alerted sooner if this issue happens again in the future.