On 12th Feb 2024, from 03:57 GMT to 04:36 GMT a fraction of facial similarity checks and of document checks requiring manual review for some customers in the EU region experienced an increase in turnaround time. Delayed checks were then processed and at 05:15 GMT the system was back to normal.
An erroneous configuration was released on 9th Feb 2024 for a service responsible for handling checks requiring manual review leading to most replicas of the service to eventually fail.
03:57 GMT: Decrease in throughput of manual reviews in EU.
04:05 GMT: The On-call team was alerted to an increase in error rate on a specific service.
04:36 GMT: Fix is applied and system was back to normal operation.
05:15 GMT: Completion of all backlog of checks generated during incident window.
Reverted the erroneous configuration.
Review internal documentation for preventing that from happening.
Reviewed monitoring for alerting Engineering more promptly.