Category statusupdate

On May 23, 2023 starting at 8pm we have a planned hardware migration scheduled. This is necessary to better serve our customers as we grow. Down time is expected to begin around 8pm EST May 23 and end 4pm EST May 24.

During this period, access to our services will be extremely limited. Access to the login page will be down. Campaigns should not be scheduled during this period as well. They will not send.

Campaigns that have gone out prior to this period will still function as expected. Images, Open tracking, and Click tracking will still be up.

We apologize for this extended down time.

Update: Service has been fully restored.

We are aware of reports that the front facing interface is currently inaccessible for some users. The team is currently working to resolve this as soon as possible.

Update: Hardware replacement was completed by 4:25 pm EST 2/24/2022.

We have scheduled down time for hardware migration at 2:30 PM EST on 2/24/2022.  Estimated time is 60 minutes.  Updates will be posted here.

We will have scheduled down time for hardware replacement at 3PM EST today.  Estimated time is 30 minutes.  Updates will be posted here.

**Update 1

We got started at 3:13 and hardware has been swapped, performing testing now.

**Update 2

Hardware has been replaced along with additional hardware that was out of tolerance.

Service should now be restoring, and we will continue to monitor.

You may see some down periods as we confirm configurations.

**Update 3

Replaced hardware seems to have stabilized, but we will be monitoring for any continuing issues.

**Update 4

Brief downtime now for some reboots to initialize.

**Update **

Recovery is complete and services are back online. Campaigns that were scheduled to go out during the downtime were paused and will need to be manually restarted or rescheduled.

SSL certificates for each custom url domain are being generated. If you encounter a SSL error, please wait a few minutes and check again.

**Update**

Contingency hardware is online, and backup restoration is in process.  The weather event has prevented our datacenter provider from correcting the hardware issue in person within a known time frame.

Our contingency plan has continued, and the failover server is being restored from our early morning backup.

Any scheduled campaigns for today, should begin to be delivered as soon as restoration is complete.

ETA restoration of service is 1200 EST.

We are aware of an intermittent network issue effecting our data center, we are working to stabilize the service.

Due to winter weather events in the DFW area immediate recover has been hampered.

We are proceeding with secondary recovery offsite and will provide updates as they become available.

This is now resolved.

There is currently a hardware problem that is being investigated. You may experience intermittent outages while we work on fixing this.

Update: This error has been resolved.

We are aware of an error with the Campaign Content Editor as a result of a recent udpate. The team is working to resolve as quickly as possible.

*** The backlog in the delivery queue should be resolving as we ramp up delivery to the affected ISPs.

We are getting reports of major outages at:

• Spectrum (also includes Charter, RoadRunner, Time Warner) across most of the East Coast and parts of the midwest.
• Verizon outage in Texas and parts of the New England.
• Cablevision/Optimum across U.S.
• Cox.net out in New England area

Along with these major providers Level3, a backbone provider, is also experiencing outages.

While these outages are not directly affecting our servers, if you see mail stuck in queue or fail/bounce due to time outs it is due to being unable to connect to a mail provider due to these outages.

Update: Hardware failure, remediated with hot swap replacement.  Queued sends should begin to resolve over the next hour. Services are being restored.

As of 1:06 PM EST we began experiencing a service outage. Our engineers are currently working to restore all services.

Update 2: Resolution has been confirmed.

Update: Servers are beginning to be accessible. We will do a final update when it is confirmed resolved.

As of 2:35 pm EST one of our COLO datacenters is experiencing a network outage.