A change to document classification logic led to 30% of the reports not being automatically classified. Those reports were sent for manual review, leading to an increase in turn around time. The change was reverted shortly after and automated processing returned to normal. Impacted checks were re-run, leaving a residual backlog of reports requiring manual review that was cleared a few hours later.
A change was deployed with a minor error that was not apparent in testing. This led to the model being unable to classify documents reliably, resulting in a significant portion of the reports being sent for review.
13:45 UTC: We deploy a change to our document classification logic. A signifiant portion of traffic starts being sent for manual review.
14:30 UTC: We are notified about a drop in automation rate and increase in manual queues.
14:37 UTC: Having identified the cause and reverted the change, automation rate returns to normal levels.
15:15 UTC: We continue to monitor. Impacted reports are rerun to assist clearing down manual backlog.
17:29 UTC: All reruns completed, positively impacting manual queues.
We are adding new monitoring capabilities on automation rates for a faster notification and response time.
We are reviewing our recovering process so we are able to perform reruns of impacted reports faster.
We are adding non regression tests to the classification logic to prevent similar incidents.