We use a job queue and workers processes to send limit notifications and usage summaries (as both emails or webhooks).

On 2020-07-05 10:00 UTC we ran a long running background job using the same queue as the limit notification and usage summary jobs. This long running job blocked other jobs from running, and lead to a delay of up to 14 hours before limit notifications and usage summaries were sent.

To prevent this from happening in the future, we will:

• Add metric to track job queue depth and alert on this
• Use a dedicated queue for limit notification jobs