Pusher status

11:23 UTC WebHooks stopped being sent

11:57 UTC WebHooks recovered

I'm afraid that missing WebHooks have been lost and cannot be delivered, so if you rely on WebHooks to synchronize state you will need to use the appropriate API calls to fetch the current state.

13:00 UTC Some internal configuration errors have resulted in presence and WebHook related data becoming inconsistent for some channels. We're currently working on a fix. ETA 14:30 UTC.

We're currently experiencing some issues with delivering webhooks. We'll update this when we know more.

We're seeing elevated rates of http error 500 for some apps at the moment, but we're working hard on this issue.

We're just looking into an issue with increased latency at the moment.

02:30 UTC Issues are affecting service. Currently investigating

03:00 UTC Service is back to normal.

Currently facing issue opening new connections. We are investigating.

03:29 UTC Systems are back to normal.

03:55 UTC Some stats data is currently missing in dashboards between about 02:30 - 03:00 UTC. This data will be recovered over the next 24 hours.

We're currently experiencing issues delivering webhooks. We're aware of the problem and are working on a fix.

We're currently resolving some problems in our dashboard. All other systems are totally unaffected by this.

We're currently experiencing issues with messaging, further information will follow.

Pusher is currently having websocket connectivity issues. We apologize for the inconvenience.

We're currently looking into reports of increased latency, and are taking steps that should resolve this soon.

The pusher webhooks notification service exhibited problems for ca 30 minutes due to a faulty server. We apologize for any inconvenience caused.

One of our components temporarily stopped responding, and service was disrupted for ~10 minutes. The issue has been resolved and we don't expect further re-occurrences.

Last night one of our system components experienced failover. While the service continued running, some user's authentication keys took longer than expected to replicate. We're sorry for any inconvenience for those affected, and we will look into why this happened.

We are currently experiencing issues with connectivity in AWS. We are investigating.

07:47 UTC We are mainly back up and running. Core real time services are working except for posting to the API via SSL (this is due to having to switch away from using Amazon ELBs which are still causing issues). The management site will recover when the RDS it relies upon recovers...

11:10 UTC Site is now available again

11:15 UTC SSL requests are now supported again for the API. All systems are now functioning correctly.

We had some load balancer failures between 11:30 - 11:40 UTC which caused WebSocket connections to fail. This is now resolved

On a number of occasions today, latency has increased beyond acceptable levels. We are looking into the cause, but suspect a faulty node that we are working to replace. Sorry for the inconvenience.

0930 UTC Lost network connectivity due to an AWS issue, waiting for the root cause.

0945 UTC Service is operating normally

During an upgrade to our infrastructure to increase capacity, we began having problems delivering messages. We are working to fix this.

We will keep you informed.

We are investigating and will keep you updated.

UPDATE:

We have resolved this issue. Message delivery is back to normal.

We are working to fix this.

Non-SSL connections work fine.

UPDATE:

SSL connections, and the rest of the Pusher service, are now operating normally.

One of our load balancers is down. We are working to bring it back up. Until we fix the problem, it will take longer than usual for clients to connect.

UPDATE:

The load balancer that was down is now back up.

We're currently experiencing an issue which may result in connections being closed. We're working to resolve this ASAP.

We've been experiencing some intermittent connectivity issues in the last hour. We're looking into it.

We are looking into this problem at the moment, and will update the status when we know more.

We have suffered the loss of one of our websocket load balancers

This load balancer has been removed from DNS, but you may experience extra connection attempts till the change propagates

Last night at around 7:40 PM PDT, an Amazon availability zone lost connectivity. This affected us for about 15 minutes. Connectivity was restored, and we are now operating normally. Sorry for any inconvenience.

We're currently investigating - update coming soon

11:11UTC We seem to be experiencing an issue with Amazon's elastic load balancer

11:24UTC We have switched to a new elastic load balancer however DNS will take some time to propagate to api.pusherapp.com. The API can temporarily be accessed from production-api-v3-2075882950.us-east-1.elb.amazonaws.com.

We experienced some issue with Amazon's elasic load balancer this morning which caused much higher than usual API request times. The mean latency was approaching 1s between 9:30 UTC & 12:30 UTC and was highly variable. All requests were being handled, but the increased timeout probably resulted in some requests timing out.

We solved this issue by provisioning a new ELB instance, and are addressing the route cause with Amazon.

Sorry for the inconvenience

We saw several issues with API timeouts and socket connection timeouts between 2am and 5am this morning. We're still trying to work out the cause of these issues. Currently Pusher is operating normally.

For approximately 15 minutes the socket processes were not running. No messages were delivered during this period (around 4pm GMT).

This was caused by all socket processes failing, and then a failure of our process monitoring to restart them. We're continuing to investigate this to ensure that it cannot happen again. Our apologies for this disruption to the service.

UPDATE This issue has resurfaced again (11:30pm GMT). We have restarted the affected processes, but we are still working on the permanent solution.

As of approximately 20:00 GMT, our service has stopped relaying messages from the API to the connected clients. This seems to be a problem with our message bus that we are trying to resolve. We will update this status when it has been resolved.

Sorry for any inconvenience.

UPDATE:

We have managed to get the issue resolved but are continuing to look into why it happened, and preventing it from happening in future.

For approximately 5 minutes yesterday the mean latency of Pusher exceeded 500ms which is unacceptable to us. This was caused by a sharp spike which saw us handle approximately 10 times more traffic during an hour than our previous maximum. We quickly started another instance and brought mean latencies back down below 100ms. We apologise for any inconvenience.

On the weekend we experienced failures on several of our socket servers. The result was that some clients were unable to connect and in some cases messages were incorrectly routed.

Solution:

This was due to a bug in our redis client library which we have now upgraded. The problem should not happen again.