Report creation failure in EU and US regions

Incident Report for Onfido

Postmortem

Summary

Onfido experienced an outage on Feb 24th, 14:14 UTC, for 9 minutes. Client API requests to upload documents and create checks returned an error response during this period. While this also impacted our SDK traffic, our logs indicate a very small number of user sessions failed to complete, as they were largely recovered by re-tries.

Root Causes

During preparations to rollout a planned database update, part of the change was inadvertently pushed to our production environment before it was ready. This was due to a misconfigured release pipeline.

Timeline

14:14 UTC: A change to a database is inadvertently pushed to production
14:15 UTC: The team responsible for this upgrade is alerted to an increase in database errors
14:17 UTC: The problem is identified, and a fix is released
14:22 UTC: The fix is fully applied to impacted regions

Remedies

The offending pipeline is being corrected and an investigation will be done to assess if any other pipeline suffers from the same misconfiguration. Additional controls will be established to avoid such misconfigurations in future.

Posted Feb 27, 2025 - 16:37 UTC

Resolved

Onfido experienced a critical outage across all regions on Feb 24th, 2025, at 14:15 UTC, lasting 9 minutes (resolved on Feb 24th, 2025, 14:24 UTC). During this time, any request made to our systems failed, resulting in no applicants or reports being created and no uploads being processed.

We take pride in running a robust, reliable service and are working hard to prevent this from happening again. Once we conclude our investigation, we will publish a detailed postmortem.

Posted Feb 24, 2025 - 14:36 UTC

This incident affected: Europe (onfido.com) (API, Dashboard, Document Verification, Facial Similarity, Watchlist, Identity Enhanced, Known faces, QES, Device Intelligence), USA (us.onfido.com) (API, Dashboard, Document Verification, Facial Similarity, Watchlist, Identity Enhanced, Known faces, Device Intelligence), and Canada (ca.onfido.com) (API, Dashboard, Document Verification, Facial Similarity, Known faces, Device Intelligence).