Availability problems
Incident Report for Kundo
Postmortem

Performance issues in our main database

We apologize for any inconvenience caused by this incident. A summary of the events and measures taken follows.

Summary

An update to the system resulted in an increased load on the database, resulting i degraded performance in several of our products.

Timeline (in CET)

10:09 An update affecting the database is done in our systems

10:10 The first customer is affected by degraded performance

10:12 We identified that there is an issue and started working on reverting the change

10:37 All systems are available again

What happened?

We did an update to improve database performance. The update caused an initial increase in load on the database before all our caching functionality was in effect, resulting in an overload of the database.

Actions to mitigate impact of incidents like these in future

  • Improving the speed of reverting changes to the system
  • Improving our monitoring

The update causing the issue has been released in smaller parts, resulting in a lower load on the database.

Posted Jun 05, 2023 - 10:21 CEST

Resolved
Kundo is now working as normal and we have confirmed that the incident has been resolved.
Posted May 26, 2023 - 10:44 CEST
Monitoring
A fix has been implemented and we are monitoring the results. The dashboard and Forum is reachable but might be slower than usually for a few minutes.
Posted May 26, 2023 - 10:38 CEST
Update
We are continuing to work on a fix for this issue.
Posted May 26, 2023 - 10:31 CEST
Identified
We have identified the problem and are currently implementing a solution.
Posted May 26, 2023 - 10:28 CEST
Investigating
We are currently investigating a problem where Kundo Dashboard is unreachable for some users. The investigation is ongoing and we will update this message when we have more information.
Posted May 26, 2023 - 10:24 CEST
This incident affected: Kundo, Dashboard, and Forum.