Date

May 23 and May 24, 2017 GMT

Description

Customers Using SMS OOB (Out Of Band/2FA) Did Not Receive Messages 

Duration

May 23 for 2 hours & May 24 for 7 hours

Note: Our customers have reported some issues outside of the times which our third party provider provided. This could be due to the fact that the events leading up to this interruption may have been spotty before the actual time defined.

Affected components

Any Silo or Toolbox login attempts where OOB was required

Affected customers

All Authentic8 provided OOB customers

Root cause analysis

Nexmo, who is the underlying service provider for SMS OOB verification, experienced problems delivering SMS messages during the times listed. This meant that users who invoked OOB authentication  during these times were not able to access the system. This caused a secondary problem where users, who did not realize that SMS was delayed, began initiating large numbers of requests. That triggered an anti-abuse mechanism which effectively locked the offending users out of the platform. 



Event Log

2019-05-23 17:00 GMT Authentic8 began receiving reports from customers that they were not receiving their OOB codes in a timely manner. Upon investigation, it was determined that the third party provider was having an outage. 


2019-05-23 21:00 GMT Nexmo, our SMS OOB provider, reported services working normally again.


2019-05-24 13:00 GMT Authentic8 began receiving reports from customers that they were not receiving their OOB codes in a timely manner. Upon investigation, it was determined that the third party provider was again having an outage. 


2019-05-24 21:00 GMT Nexmo reported services working normally again.


Resolution

Nexmo reported that their suppliers identified an issue impacting SMS deliveries sent using short codes in the USA.  This was resolved.


Moving Forward

We have decided to implement a backup provider to make this part of our service more robust. We expect to have this implemented by 5/31/19.