Incident date: 22 June 2026
Region: EU (eu-west-1)
Affected EU services: API, Dashboard, Applicant Form, Document Verification,
Facial Similarity, Watchlist, Identity Enhanced, Webhooks, Known Faces, Autofill,
QES and Device Intelligence.
Customer impact: ~16:50–17:00 UTC (acute degradation); ~17:00–17:20 UTC (backlog recovery)

Summary

On 22 June 2026, from approximately 16:50 UTC, a shared database cluster serving our EU region came under severe load and could not reliably serve queries for about 10 minutes. EU services returned elevated errors, and processing throughput briefly fell to ~20–35% of normal levels, with many subcomponents of our system (e.g., Facial Similarity report processing) being entirely disrupted, some others less heavily impacted (e.g., Document report processing). Service recovered by 17:01 UTC, the database fully stabilizing after an automatic failover (~17:05–17:07 UTC). A resultant report backlog was cleared by ~17:20 UTC.

Requests in flight during the acute degradation window may have failed unless retried; queued background work was processed automatically once the database recovered.

Root cause

The incident was triggered by a routine database storage-reclamation task following standard scheduled data-deletion processing. This task normally completes without issue; why it failed on this occasion remains under investigation, although we observed that it was processing a larger-than-usual backlog. We have a support case open with our cloud provider to confirm a definitive root cause.

The reclamation task began to compete with normal application queries, which slowed as the database struggled to keep up. Applications opened more and more connections, leading to connection saturation and causing queries across the affected services to fail.

The database stabilized when an automatic failover to a healthy standby was triggered; the contention fully resolving with the failover to a new instance.

Timeline (UTC)

• 16:50 — Our monitoring detected errors and elevated latency across EU services caused by resource contention on a shared database cluster.
• 16:53 — We start to see improvements, but system still not acting at normal levels of performance.
• ~17:00 — Customer-facing errors subsided and processing resumed as the contention eased.
• 17:01 — On-call engineers opened an incident and continued investigations.
• ~17:05–17:07 — The database performed an automatic failover to a healthy instance, which reset the overloaded writer and fully stabilised the cluster. The failover was triggered because of resource contention (out of memory) caused by the heavy vacuuming in the preceding minutes of the incident. Once the impacting vacuum operations had finished freeing up resources, we had started to see signs of improvement (16:53—17:01), but added latency in the feedback loop and aggregation window at AWS still decided to trigger the failover, even though we were already in a recovering state.
• 17:15–17:20 — Requests that had queued during the incident were worked through and the backlog returned to normal.
• 17:21–17:56 — We monitored the recovery and confirmed processing remained at full capacity.

Remedies

• Reviewing connection limits and pooling so a single service cannot saturate a shared database, and evaluating dedicated database clusters per product to remove cross-service impact.
• Changing large historical-data deletions to run in smaller, throttled batches, and tuning database maintenance to avoid large catch-up operations.
• Adding earlier, proactive alerting on database memory, connections and load so we can intervene before customer impact.
• Continue working with our cloud provider on a definitive root cause.