serviceupdate

We will have scheduled down time for hardware replacement at 3PM EST today.  Estimated time is 30 minutes.  Updates will be posted here.

**Update 1

We got started at 3:13 and hardware has been swapped, performing testing now.

**Update 2

Hardware has been replaced along with additional hardware that was out of tolerance.

Service should now be restoring, and we will continue to monitor.

You may see some down periods as we confirm configurations.

**Update 3

Replaced hardware seems to have stabilized, but we will be monitoring for any continuing issues.

**Update 4

Brief downtime now for some reboots to initialize.

**Update **

Recovery is complete and services are back online. Campaigns that were scheduled to go out during the downtime were paused and will need to be manually restarted or rescheduled.

SSL certificates for each custom url domain are being generated. If you encounter a SSL error, please wait a few minutes and check again.

**Update**

Contingency hardware is online, and backup restoration is in process.  The weather event has prevented our datacenter provider from correcting the hardware issue in person within a known time frame.

Our contingency plan has continued, and the failover server is being restored from our early morning backup.

Any scheduled campaigns for today, should begin to be delivered as soon as restoration is complete.

ETA restoration of service is 1200 EST.

We are aware of an intermittent network issue effecting our data center, we are working to stabilize the service.

Due to winter weather events in the DFW area immediate recover has been hampered.

We are proceeding with secondary recovery offsite and will provide updates as they become available.

Update: Hardware failure, remediated with hot swap replacement.  Queued sends should begin to resolve over the next hour. Services are being restored.

As of 1:06 PM EST we began experiencing a service outage. Our engineers are currently working to restore all services.

Update: The router has been replaced and Campaigns are getting pushed out. You may see some delay on the Statistics as they are getting caught up.

As of 9:48 a.m. EST a router failed causing our servers to be cut off from the internet. We are in the process of replacing it. The expected resolution time is 11:30 a.m. EST.

Update: The service disruption started at about 1:00 pm EST and resolved at about 2:00 pm EST.
The practical impact of this in most cases was that if a Comcast subscriber attempted to access web interface using HTTPS that attempt was likely to fail.

As far as we can tell, this issue did not impact any unencrypted protocols like HTTP or SMTP, so most outgoing emails and incoming clicks/opens were unaffected.

The problem was resolved on its own.

Comcast users may have intermittent difficulty in accessing the Inboxfirst web interface due to a localized Comcast outage near the data center.