Date | May 23 and May 24, 2017 GMT |
Description | Customers Using SMS OOB (Out Of Band/2FA) Did Not Receive Messages |
Duration | May 23 for 2 hours & May 24 for 7 hours Note: Our customers have reported some issues outside of the times which our third party provider provided. This could be due to the fact that the events leading up to this interruption may have been spotty before the actual time defined. |
Affected components | Any Silo or Toolbox login attempts where OOB was required |
Affected customers | All Authentic8 provided OOB customers |
Root cause analysis | Nexmo, who is the underlying service provider for SMS OOB verification, experienced problems delivering SMS messages during the times listed. This meant that users who invoked OOB authentication during these times were not able to access the system. This caused a secondary problem where users, who did not realize that SMS was delayed, began initiating large numbers of requests. That triggered an anti-abuse mechanism which effectively locked the offending users out of the platform. |
Event Log
2019-05-23 17:00 GMT Authentic8 began receiving reports from customers that they were not receiving their OOB codes in a timely manner. Upon investigation, it was determined that the third party provider was having an outage.
2019-05-23 21:00 GMT Nexmo, our SMS OOB provider, reported services working normally again.
2019-05-24 13:00 GMT Authentic8 began receiving reports from customers that they were not receiving their OOB codes in a timely manner. Upon investigation, it was determined that the third party provider was again having an outage.
2019-05-24 21:00 GMT Nexmo reported services working normally again.
Resolution
Nexmo reported that their suppliers identified an issue impacting SMS deliveries sent using short codes in the USA. This was resolved.
Moving Forward
We have decided to implement a backup provider to make this part of our service more robust. We expect to have this implemented by 5/31/19.