Summary

The Dashboard was intermittently unavailable in the EU region from 15:30 UTC to 18:30 UTC due to an outage in the underlying data store that backs its search index.

Root Causes

A partition on the data store's shards caused increased latency on 20% of write and read operations, which ultimately timed out. This in turn prevented the Dashboard from loading whenever a timeout for these operations was reached.

Timeline

15:30 UTC: Dashboard (EU) started having intermittent usage issues

16:30 UTC: We identified the root cause in the search index data store

17:45 UTC: Fix was applied to the search index data store

18:30 UTC: Incident was resolved

Remedies

As immediate remediation the partition was manually restored, resulting in normal read/write times. The data that failed to be written during the incident period was eventually made consistent automatically.

We are committed to preventing such issues from having such an impact in the future. Therefore, we will:

• Prevent issues in the data store from blocking access to unrelated Dashboard features, as a means of graceful service degradation;
• Include specific alarms for shard partitions to allow for faster action by the engineering team.