On September 24th from 10:33 UTC to 11:14 UTC customers faced a very high error rate when creating new document Reports.
Reports created before 10:33 UTC and non-document reports continued to be processed. For the duration of the incident only 10% of new document reports requests were accepted and processed.
At 10:33 we started the release of an infrastructure change. A migration from a legacy deployment system to a newer standard one.
The new configuration contained an typographical error that removed some permissions for an internal service. As a result, the service wasn’t able to interact with other infrastructure components. That service is in charge of handling documents, making any document media upload or download fail for our customers.