Emails delayed
Incident Report for Pusher

We use a job queue and workers processes to send limit notifications and usage summaries (as both emails or webhooks).

On 2020-07-05 10:00 UTC we ran a long running background job using the same queue as the limit notification and usage summary jobs. This long running job blocked other jobs from running, and lead to a delay of up to 14 hours before limit notifications and usage summaries were sent.

To prevent this from happening in the future, we will:

  • Add metric to track job queue depth and alert on this
  • Use a dedicated queue for limit notification jobs
Posted Jul 08, 2020 - 09:27 UTC

Limit notifications (emails and webhooks) and usage summary emails were delayed by up to 14 hours over the past 2 days (2020-07-05 and 2020-07-06). This issue has now been resolved.
Posted Jul 05, 2020 - 09:00 UTC