Pusher status

We are looking into this problem at the moment, and will update the status when we know more.

We have suffered the loss of one of our websocket load balancers

This load balancer has been removed from DNS, but you may experience extra connection attempts till the change propagates

Last night at around 7:40 PM PDT, an Amazon availability zone lost connectivity. This affected us for about 15 minutes. Connectivity was restored, and we are now operating normally. Sorry for any inconvenience.

We're currently investigating - update coming soon

11:11UTC We seem to be experiencing an issue with Amazon's elastic load balancer

11:24UTC We have switched to a new elastic load balancer however DNS will take some time to propagate to api.pusherapp.com. The API can temporarily be accessed from production-api-v3-2075882950.us-east-1.elb.amazonaws.com.

We experienced some issue with Amazon's elasic load balancer this morning which caused much higher than usual API request times. The mean latency was approaching 1s between 9:30 UTC & 12:30 UTC and was highly variable. All requests were being handled, but the increased timeout probably resulted in some requests timing out.

We solved this issue by provisioning a new ELB instance, and are addressing the route cause with Amazon.

Sorry for the inconvenience

We saw several issues with API timeouts and socket connection timeouts between 2am and 5am this morning. We're still trying to work out the cause of these issues. Currently Pusher is operating normally.

For approximately 15 minutes the socket processes were not running. No messages were delivered during this period (around 4pm GMT).

This was caused by all socket processes failing, and then a failure of our process monitoring to restart them. We're continuing to investigate this to ensure that it cannot happen again. Our apologies for this disruption to the service.

UPDATE This issue has resurfaced again (11:30pm GMT). We have restarted the affected processes, but we are still working on the permanent solution.

As of approximately 20:00 GMT, our service has stopped relaying messages from the API to the connected clients. This seems to be a problem with our message bus that we are trying to resolve. We will update this status when it has been resolved.

Sorry for any inconvenience.

UPDATE:

We have managed to get the issue resolved but are continuing to look into why it happened, and preventing it from happening in future.

For approximately 5 minutes yesterday the mean latency of Pusher exceeded 500ms which is unacceptable to us. This was caused by a sharp spike which saw us handle approximately 10 times more traffic during an hour than our previous maximum. We quickly started another instance and brought mean latencies back down below 100ms. We apologise for any inconvenience.

On the weekend we experienced failures on several of our socket servers. The result was that some clients were unable to connect and in some cases messages were incorrectly routed.

Solution:

This was due to a bug in our redis client library which we have now upgraded. The problem should not happen again.