tag:status.pusher.com,2005:/historyPusher Status - Incident History2024-03-19T08:48:46ZPushertag:status.pusher.com,2005:Incident/200680552024-02-25T04:11:20Z2024-02-25T04:11:20ZPusher US2 webhooks delayed<p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>04:11</var> UTC</small><br><strong>Resolved</strong> - We have identified the issue and resolved it. Webhooks are behaving as expected again.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>01:02</var> UTC</small><br><strong>Investigating</strong> - We are investigating a delay in Webhooks.</p>tag:status.pusher.com,2005:Incident/200618402024-02-24T13:36:33Z2024-02-24T13:36:35ZPusher EU webhooks delayed<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>13:36</var> UTC</small><br><strong>Resolved</strong> - All delayed EU webhooks completed processing at approximately 1030 UTC. This incident is resolved.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>05:16</var> UTC</small><br><strong>Update</strong> - We have deployed a fix for this issue. The delivery of the delayed webhooks continues.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>01:51</var> UTC</small><br><strong>Identified</strong> - We are continuing to see delays in webhooks. We are working on implementing an additional fix.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>00:29</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results as the backload is processed off.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>00:25</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>23:53</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.pusher.com,2005:Incident/200625632024-02-24T13:35:10Z2024-02-24T13:35:11ZPusher US2 webhooks delayed<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>13:35</var> UTC</small><br><strong>Resolved</strong> - All delayed US webhooks completed processing at approximately 730 UTC. This incident is resolved.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>05:16</var> UTC</small><br><strong>Update</strong> - We have deployed a fix for this issue. The delivery of the delayed webhooks continues.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>02:29</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.pusher.com,2005:Incident/200523192024-02-23T00:36:09Z2024-02-23T00:36:11ZPusher US3 cluster outage<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>00:36</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>00:34</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>23:11</var> UTC</small><br><strong>Identified</strong> - The team identified the problem. Most connections are stable now but some of them may get reconnected still. The team is working on a permenant solution.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>22:24</var> UTC</small><br><strong>Investigating</strong> - Pusher US3 cluster is experiencing disruptions for client socket connections. Our team is looking into it.</p>tag:status.pusher.com,2005:Incident/198388822024-01-26T15:37:00Z2024-01-26T16:14:24ZChannels Stats Missing Data<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>15:37</var> UTC</small><br><strong>Resolved</strong> - Between 15:37 - 16:01 UTC we encountered an issue with our stats pipeline that resulted in missing metrics. Customers may notice data missing for this period in the Pusher Dashboard metrics along with any configured integration dashboards(DataDog and Librato). <br />The issue has been resolved and stats are being published correctly since 16:01.</p>tag:status.pusher.com,2005:Incident/193369582023-12-05T18:11:07Z2023-12-05T18:11:08ZPublic EU cluster outage<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>18:11</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>17:47</var> UTC</small><br><strong>Monitoring</strong> - The public EU cluster has been restored and all services are now back online. We will continue to monitor the cluster health.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>17:36</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>17:34</var> UTC</small><br><strong>Investigating</strong> - The public EU cluster is currently experiencing an outage which affects customers in this region. For customers in other regions, there is no impact. We are actively working on restoring the public EU cluster.</p>tag:status.pusher.com,2005:Incident/192661382023-12-01T21:20:36Z2023-12-01T21:20:37ZUS2 Customers may be experiencing latency issues with or unable to utilize the Channels WebSocket and REST API<p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>21:20</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>20:53</var> UTC</small><br><strong>Monitoring</strong> - The fix has been implemented and latency has improved.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>20:46</var> UTC</small><br><strong>Identified</strong> - The team has identified the cause of the latency and are actively working to resolve the high latency issues.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>20:31</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue in US2 where some clients may be experiencing latency issues or unable to utilize the Channels WebSocket and REST API.</p>tag:status.pusher.com,2005:Incident/192490582023-11-30T11:17:12Z2023-11-30T11:17:13ZCustomers using Datadog exporter feature at pusher channels might experience a small data loss or delay on the dashboards<p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>11:17</var> UTC</small><br><strong>Resolved</strong> - This incident is being resolved</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>10:30</var> UTC</small><br><strong>Investigating</strong> - Our engineering team has detected an issue that is preventing all data to be published to DataDog. We are currently investigating the root cause</p>tag:status.pusher.com,2005:Incident/192428562023-11-29T16:09:02Z2023-11-29T16:09:03ZHigh latency on channels US2 API<p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>16:09</var> UTC</small><br><strong>Resolved</strong> - Our engineering team has confirmed this issue has been mitigated. We apologise for any inconvenience this may have caused you.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>15:48</var> UTC</small><br><strong>Identified</strong> - Our engineering team is working on rolling out an improvement to resolve the high latency on the US2 cluster.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>15:17</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.pusher.com,2005:Incident/191948012023-11-22T14:27:38Z2023-11-22T14:27:39ZA subset of webhooks may not be successfully delivered - EU cluster only<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>14:27</var> UTC</small><br><strong>Resolved</strong> - Our engineering team has rolled out a fix and has confirmed webhooks delivery is fully operational on the affected EU cluster.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>13:08</var> UTC</small><br><strong>Investigating</strong> - Our engineering team is actively investigating an issue that is impacting the delivery of a specific set of webhooks. This is only affecting our EU cluster</p>tag:status.pusher.com,2005:Incident/191781342023-11-20T10:41:05Z2023-11-20T15:20:08Zintermittent issues with publishing messages in EU cluster<p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>10:41</var> UTC</small><br><strong>Resolved</strong> - This issue has been resolved</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>10:30</var> UTC</small><br><strong>Monitoring</strong> - The issue has been identified and a fix is released</p><p><small>Nov <var data-var='date'>19</var>, <var data-var='time'>10:15</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating the issue</p>tag:status.pusher.com,2005:Incident/191298382023-11-15T19:02:44Z2023-11-15T19:02:44ZWebhook delivery issue - Channels EU cluster<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>19:02</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>18:46</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>18:36</var> UTC</small><br><strong>Investigating</strong> - We're currently investigating a webhook delivery issue in the EU cluster, impacting a small portion of traffic.</p>tag:status.pusher.com,2005:Incident/191173422023-11-14T17:36:04Z2023-11-14T17:36:06ZChannels EU cluster having intermittent issues<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>17:36</var> UTC</small><br><strong>Resolved</strong> - The cluster is now operational following the implementation of a fix at 17:13 UTC. Our team is actively monitoring the results. The primary contributing factor was identified as a DNS-related issue. In order to determine the root cause, a thorough investigation will be conducted and a detailed post-mortem will be published.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>16:16</var> UTC</small><br><strong>Update</strong> - We are continuing our investigation into the ongoing issue affecting EU cluster. Our team is actively working to mitigate the problem. While the cluster is slowly recovering, it has not yet reached a stable state.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>15:27</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing an outage affecting API, WebSocket, and Webhook delivery services. Our team is actively investigating the issue</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>14:31</var> UTC</small><br><strong>Monitoring</strong> - All systems are now operational. However, clients utilising fallback transports may still experience intermittent issues. We are actively working to address the remaining issues.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>14:07</var> UTC</small><br><strong>Update</strong> - Webhook delivery was temporarily affected, resulting in delays for some customers. Delivery is returning to normal.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>13:43</var> UTC</small><br><strong>Update</strong> - We saw 3 short bursts of errors related to service restarts with limited customer impact in the last 2 hours. We are continuing to investigate the issue.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>12:55</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.pusher.com,2005:Incident/189219652023-10-25T04:00:00Z2023-10-25T09:42:59ZDisruption in Channels stats integration<p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>Resolved</strong> - Between 06:10 and 09:14 UTC, we encountered an issue that temporarily disrupted the export of data to our stats integrations as well as our own dashboard. As a result, customers using our stats integration to relay metrics to platforms like Datadog and Librato may observe a gap in their data, and the same gap may be visible in our own dashboard.</p>tag:status.pusher.com,2005:Incident/188965092023-10-23T09:22:11Z2023-10-23T09:22:11ZMarketing website is unreachable for some visitors<p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>09:22</var> UTC</small><br><strong>Resolved</strong> - The marketing website (pusher.com) should now be reachable for all visitors, and this incident is now resolved. Pusher services were available and unaffected the entire time.</p><p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>08:24</var> UTC</small><br><strong>Investigating</strong> - Our marketing website (pusher.com) is unreachable for some visitors. All Pusher services are unaffected and working as expected. We are investigating the issue with the marketing website.</p>tag:status.pusher.com,2005:Incident/188417402023-10-18T14:30:00Z2023-10-18T15:40:03ZPartial outage in eu cluster<p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>14:30</var> UTC</small><br><strong>Resolved</strong> - Between 14:30 and 14:45 UTC, the Channels API experienced a partial outage. At its peak, approximately 32% of requests to publish a new message failed and most clients had to reconnect.</p>tag:status.pusher.com,2005:Incident/188113882023-10-17T23:20:34Z2023-10-17T23:20:37ZInsights and graphs not available in neither customer and admin view<p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>23:20</var> UTC</small><br><strong>Resolved</strong> - Most customers have had their graphs restored. Over the next few days, the team will persist in their efforts to ensure that all customers' graph issues are resolved, albeit outside the incident process.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>12:54</var> UTC</small><br><strong>Update</strong> - We've observed an improvement in processing speed, and for numerous customers, the data is now readily available and accurate on the Dashboard. We are actively working on the issue.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>07:57</var> UTC</small><br><strong>Update</strong> - Some data processing is taking longer than expected. We are actively working on the issue.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>16:29</var> UTC</small><br><strong>Update</strong> - The data replay is ongoing. We expect it to be completed in the next 6h.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>14:08</var> UTC</small><br><strong>Update</strong> - We are now replaying old data in order to process it. This may cause intermittent delays in usage stats across the dashboard.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>09:28</var> UTC</small><br><strong>Monitoring</strong> - The team has restored the service and is currently monitoring</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>08:57</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue that is preventing the telemetry to be pushed to the customer and admin views, making the graphs not available</p>tag:status.pusher.com,2005:Incident/185379362023-09-19T07:57:11Z2023-09-19T07:57:11ZElevated API Errors<p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>07:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>19:56</var> UTC</small><br><strong>Update</strong> - Our vendor continues to investigate and work towards resolution of this issue. We are continuing to see elevated error rates on our public cluster during this time.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>19:26</var> UTC</small><br><strong>Identified</strong> - AWS has confirmed an issue in the us-west region. We are actively assessing the impact on our customers, who may experience elevated error rates during this time.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>19:18</var> UTC</small><br><strong>Update</strong> - Customers hosted on our us3 cluster are currently experiencing elevated error rates. Our team is currently looking into the issue.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>19:03</var> UTC</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status.pusher.com,2005:Incident/184370302023-09-11T06:01:20Z2023-09-11T06:01:21Z[Urgent] MT1 cluster maintenance<p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>06:01</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>16:03</var> UTC</small><br><strong>Scheduled</strong> - We will be undertaking urgent maintenance on our MT1 cluster to resolve an issue that can cause client connections to be rejected. During this window clients may experience periods where they are unable to create and maintain connections to the MT1 cluster.</p>tag:status.pusher.com,2005:Incident/184370132023-09-09T05:02:51Z2023-09-09T05:02:51Z[Urgent] MT1 and EU cluster maintenance<p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>05:02</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>04:27</var> UTC</small><br><strong>Verifying</strong> - We have deployed the intended changes to the MT1 and EU clusters. We are continuing to monitor.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>16:02</var> UTC</small><br><strong>Scheduled</strong> - We will be undertaking urgent maintenance on our MT1 and EU clusters to resolve an issue that can cause client connections to be rejected. During this window clients may experience periods where they are unable to create and maintain connections to these clusters.</p>tag:status.pusher.com,2005:Incident/184225082023-09-07T15:28:06Z2023-09-07T15:28:17ZMT1 Cluster issues<p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>15:28</var> UTC</small><br><strong>Resolved</strong> - This issue is now resolved and the service is operational once more. <br />An RCA will be published once we conclude internal investigations into the issue.</p><p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>14:38</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>14:23</var> UTC</small><br><strong>Investigating</strong> - We are investigating reports of issues with our MT1 cluster.</p>tag:status.pusher.com,2005:Incident/184094712023-09-06T21:24:18Z2023-09-06T21:24:18ZMT1 cluster outage<p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>21:24</var> UTC</small><br><strong>Resolved</strong> - This issue is now resolved, with connections being established successfully across both websocket and fallback protocols. We will share a post mortem when the full investigation is complete.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>20:07</var> UTC</small><br><strong>Update</strong> - We've noticed some improvements, although the cluster isn't completely stable yet. Our team is working to resolve the issue.<br /><br />We're currently able to accept more connections through WebSocket transport. However, webhook delivery and fallback transports continue to experience higher impact.<br /><br />Our customer support team has been assisting some of our customers in redirecting their traffic to alternative US-based clusters. (US2 and US3)</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>17:12</var> UTC</small><br><strong>Monitoring</strong> - We have implemented a fix and are monitoring progress as we see Websocket connections reliablity improve. <br />We anticipate that our fallback protocols, SockJS and XHR, will improve over the course of the next 30 minutes.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>16:08</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>14:55</var> UTC</small><br><strong>Investigating</strong> - We are investigating an issue with clients unable to establish a connection to the mt1 cluster.</p>tag:status.pusher.com,2005:Incident/184000512023-09-05T23:03:09Z2023-09-05T23:03:09ZMT1 Elevated Errors<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>23:03</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>22:52</var> UTC</small><br><strong>Monitoring</strong> - We have implemented a fix and are monitoring the impact - so far the system looks operational.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>22:06</var> UTC</small><br><strong>Investigating</strong> - We are seeing that error rates have increased once more and are continuing to investigate.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>21:41</var> UTC</small><br><strong>Monitoring</strong> - The cluster has recovered. We are monitoring the results.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>21:11</var> UTC</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors on MT1 and are currently looking into the issue.</p>tag:status.pusher.com,2005:Incident/181527702023-08-15T12:30:00Z2023-08-15T13:53:13ZElevated error rates on mt1 cluster<p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>12:30</var> UTC</small><br><strong>Resolved</strong> - Between 12:33 and 13:05 UTC customers may have encountered an increase in error rates when interacting with the Channels HTTP API. The error rate has since subsided. We are investigating the platform to identify the underlying cause.</p>tag:status.pusher.com,2005:Incident/180389782023-08-02T17:30:00Z2023-08-03T07:42:40ZPartial disruption in DataDog metrics exporter<p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>17:30</var> UTC</small><br><strong>Resolved</strong> - An incident occurred yesterday during a maintenance operation, resulting in disruptions to our stats exporter. Some customers might have gaps in their DataDog metrics between 18:30 and 19:30 UTC.</p>