*** The backlog in the delivery queue should be resolving as we ramp up delivery to the affected ISPs.
We are getting reports of major outages at:
- Spectrum (also includes Charter, RoadRunner, Time Warner) across most of the East Coast and parts of the midwest.
- Verizon outage in Texas and parts of the New England.
- Cablevision/Optimum across U.S.
- Cox.net out in New England area
Along with these major providers Level3, a backbone provider, is also experiencing outages.
While these outages are not directly affecting our servers, if you see mail stuck in queue or fail/bounce due to time outs it is due to being unable to connect to a mail provider due to these outages.
Update: Hardware failure, remediated with hot swap replacement. Queued sends should begin to resolve over the next hour. Services are being restored.
As of 1:06 PM EST we began experiencing a service outage. Our engineers are currently working to restore all services.
Update 2: Resolution has been confirmed.
Update: Servers are beginning to be accessible. We will do a final update when it is confirmed resolved.
As of 2:35 pm EST one of our COLO datacenters is experiencing a network outage.
Update: The router has been replaced and Campaigns are getting pushed out. You may see some delay on the Statistics as they are getting caught up.
As of 9:48 a.m. EST a router failed causing our servers to be cut off from the internet. We are in the process of replacing it. The expected resolution time is 11:30 a.m. EST.
Update: The AOL issues are beginning to go away and should normalize over the next few days.
If you are experiencing a high number of bounces at AOL, know that our deliverability team is currently working with them to get this resolved.
Update: Previews are now working as intended. Any Preview sent before 10:10 pm EST will need to be re-sent.
Campaign Previews are currently delayed. Our team is looking into the cause of this and will push out any delayed Previews as soon as possible. The delay has not affected live Campaigns.
As of this morning, IP reputation monitor ReturnPath is not returning Sender Scores for IP addresses across the internet causing Comcast’s automatic blocking systems to block all mail.
On May 1 Facebook will discontinue forwarding @facebook.com email addresses to forwarding accounts. We will no longer attempt to deliver email addresses that contain @facebook.com to avoid deliverability or bounce issues for our customers.
The @facebook.com email address will still be on your Mailing Lists unless you Deactivate them.
Update: Tracking is getting pushed out to Statistics right now. It may take a few hours to be completely caught up.
Investigating – Early this morning our servers that push the Statistic data for Campaigns is delayed in pushing the Open and Click data to your Campaign Statistics.
Update – Deliverability has made progress with the Microsoft team. Bounces are coming down. The initial find is the Microsoft spam algorithm incorrectly flagged IP addresses as “spammy” due to the mail stream increases beyond the threshold’s limits. We will continue to mitigate with them until all mail is flowing freely again.
Investigating – We are suddenly seeing high bounce rates coming from Microsoft accounts (hotmail.com, msn.com, outlook.com, live.com) tonight. Our deliverability team is currently in communication with Microsoft to determine the cause of this.