Date        
December 11, 2018
Description    
Customers unable to access the Authentic8 service in NYC
Duration    
422 minutes (15:11-22-13 UTC)
Affected components    
Access to Silo and Toolbox was temporarily unavailable in NYC      
Affected customers    
All customers in the New York region
Root cause analysis
NYC provider decommissioned the wrong network port



Event Log


On 12/11/2018 at 15:11:00 UTC Authentic8 monitoring detected that the NYC load balancer was unresponsive. Customers in the New York region also began reporting issues connecting to the Authentic8 service.
 
Within a few minutes, NYC inbound traffic was directed to redundant data center in Ashburn for the duration of the incident. An investigation was started with the service provider which ultimately identified the root cause. The network port and IP associated with the load balancer VIP was mistakenly decommissioned.  


Resolution


Service to NYC was restored on12/11/2018 at 22:13:00 UTC.  
 
The service provider rebuilt the circuit and moved routes back to NYC. This was an unexpected and large oversight by a new provider we’ve recently added. We’re currently performing a full audit with that vendor to identify and classify key components so this doesn’t happen again.


Moving Forward


We will apply Authentic8 owned IP space to the NYC data center.  This will be complete by 12/14/18.  Once this is complete, we will move customers back to this data center.  


Request for information from you


-    If you were impacted and your experience doesn’t match what was described, please share your experience with us.
-    Please let us know if you want to be notified when the issue was resolved.