On 12th September 2024, from approximately 16:15 UTC to 17:45 UTC, we experienced a period of instability that impacted applicant updates (68%), applicant deletions (100%) and document check completion rates (40%). Because our clients and partners use our applicant endpoints in various ways, this didn’t uniformly affect them across the board. It’s estimated it impacted the creation of at most 14% of our verifications overall during that period, and slightly less on Studio (11%) in our European region.
This was followed by an extended period during which many historic check PDFs were unavailable for retrieval via the dashboard and API. Full correction of this issue was completed at 08:30 UTC on September 14th.
The maximum available key size was exceeded in a database table used by one of our services to store historical applicant data. Due to this, the application was unable to save new data for existing applicants (not affecting new applicants).
Due to the volume of data stored by this table, while correcting this issue, priority was given to restore service availability at the expense of historical data retrieval, which prevented download check PDFs of older applicants until it was fully restored on September 14th at 8:30 UTC.
(times in UTC)
September 12th
16:15 - Incident started
16:23 - Issue was identified
17:45 - Issue was fixed
September 14th
08:30 - Historical data backfill completed
Review key types and growth rates of other tables across our data stores. Monitor table growth and migrate to required larger key types.