We apologize for any inconvenience caused by this incident. A summary of the events and measures taken follows.
15:06 We deploy a new version of our systems which includes the removal of several database indices, determined to be unused.
15:13 At time of updating database indices we saw a sudden increase in database query times and our services started to experience timeouts. Our engineers start a troubleshooting process in order to identify and solve the issue.
15:20 We post a status message notifying about the incident.
15:26 We identify the increased database load as a result of the indices which were removed during the deployment. The increased load on our database prevents web requests to our application from being successfully processed.
15:28 The database queries which cause the increased load are identified.
15:48 We release a new version of our systems in which the database indices are recreated and monitor the results.
16:08 The indices are recreated but we still experience a high database load. We continue to identify a solution to mitigate the problem.
16:25 We perform optimizations of the causal queries and monitor the results. The action allows our systems to recover and return to normal operation.
16:42 The incident is marked as resolved after a period of successful monitoring.
We released a new version of our systems in which the removal of several database indices were included. It was mistakenly assumed that the indices were unused and could be safely removed. The removal caused an increased load on our database which prevented our systems from successfully handling web requests.
The issue was solved by recreating the indices and performing optimizations on the database queries which was causing the increased loads.
To prevent this from happening again, we have started an initiative that will review our process for how we implement database changes. More specifically, we will review what is required when changing a database index and how we can identify whether it is in use or not.
From a longer perspective, we will implement improvements to our database architecture in order to reduce the impact that database difficulties have on our system as a whole.
You are most welcome to contact us via email for more information: support@kundo.se