Incident - Service Degradation for New York City Region

   

Date

July 24, 2020

Description

Customers within the New York City region were unable to establish new Silo sessions.

Duration

Approximately 1 hour and 53 minutes (9:15 AM to 11:02 AM PT)

Affected components

Customers in the New York City region may have experienced delays and/or failures in starting new Silo sessions.

Affected customers

Anyone trying to connect to the New York City App Servers. 

Root cause analysis

A failure in our automation allowed several mis-configured servers to be placed in service that were unable to start new sessions due to name resolution issues despite passing health check tests.

 

 Event Log

On 07/24/2020 at 9:15 AM PT, Authentic8 received reports from two customers who were unable to connect to the service. Authentic8’s Site Reliability team began to investigate the reports.

     

Resolution

On 07/24/2020 at 11:02 AM, Authentic8’s Site Reliability team fully resolved the name server configuration issue. All new connections to NYC app servers were successful. 

 

Moving Forward

Authentic8 has repaired the error in our infrastructure automation  that allowed this misconfiguration and has implemented redundant monitoring to provide additional methods of assessing the server health. Improvements to the product are underway to further improve the health check tests to understand this condition and mark the server as unavailable if name resolution is not possible.

  


Additional Notes  

Please contact Support if you have any additional questions and/or require further information.