May 23 and May 24, 2017 GMT
Customers Using SMS OOB (Out Of Band/2FA) Did Not Receive Messages
May 23 for 2 hours & May 24 for 7 hours
Note: Our customers have reported some issues outside of the times which our third party provider provided. This could be due to the fact that the events leading up to this interruption may have been spotty before the actual time defined.
Any Silo or Toolbox login attempts where OOB was required
All Authentic8 provided OOB customers
Root cause analysis
Nexmo, who is the underlying service provider for SMS OOB verification, experienced problems delivering SMS messages during the times listed. This meant that users who invoked OOB authentication during these times were not able to access the system. This caused a secondary problem where users, who did not realize that SMS was delayed, began initiating large numbers of requests. That triggered an anti-abuse mechanism which effectively locked the offending users out of the platform.
2019-05-23 17:00 GMT Authentic8 began receiving reports from customers that they were not receiving their OOB codes in a timely manner. Upon investigation, it was determined that the third party provider was having an outage.
2019-05-23 21:00 GMT Nexmo, our SMS OOB provider, reported services working normally again.
2019-05-24 13:00 GMT Authentic8 began receiving reports from customers that they were not receiving their OOB codes in a timely manner. Upon investigation, it was determined that the third party provider was again having an outage.
2019-05-24 21:00 GMT Nexmo reported services working normally again.
Nexmo reported that their suppliers identified an issue impacting SMS deliveries sent using short codes in the USA. This was resolved.
We have decided to implement a backup provider to make this part of our service more robust. We expect to have this implemented by 5/31/19.