Issue is now resolved and our service is back to normal
Posted Dec 18, 2014 - 22:16 UTC
Update
Amazon announced they have resolved the issue and we can see everything are going back to normal too. We will continue to monitor the situation for a couple of hours.
"1:11 PM PST Between 11:36 AM and 12:47 PM PST, we experienced elevated error rates resolving DNS names for some AWS resources in the US-EAST-1 Region. The issue has been resolved and the service is operating normally." via http://status.aws.amazon.com/
Posted Dec 18, 2014 - 21:19 UTC
Update
Amazon acknowledged the issue and they are looking into it.
"12:41 PM PST We are investigating reports of problems resolving some DNS names in the US-EAST-1 Region." via http://status.aws.amazon.com/
We will continue to monitor the situation closely and the mitigation suggested earlier still work.
Posted Dec 18, 2014 - 20:47 UTC
Update
DNS problems in fact persist, but appear to be limited to certain availability zones. The mitigation strategy suggested earlier still holds and we are continuing to monitor the situation and explore more active mitigation strategies.
Posted Dec 18, 2014 - 20:33 UTC
Monitoring
api.pusherapp.com appears to be resolving correctly now, though some clients may have cached the failure and require time for this to expire.
Posted Dec 18, 2014 - 20:29 UTC
Identified
We are seeing a drop in requests to our rest API. The issue started at 7:36 UTC.
Amazon appears to have stopped resolving our DNS name to our ELB when queried from within AWS.
Our ELB can be reached directly at production-api-v4-1909248546.us-east-1.elb.amazonaws.com as a mitigation until AWS resolve this issue.
Posted Dec 18, 2014 - 20:11 UTC
Investigating
We are experiencing some issues with our REST API endpoint api.pusherapp.com. We are investigating this further as we speak.