Problems with Kundo Knowledge and Help Center
Incident Report for Kundo
Postmortem

Problems accessing Knowledge and Help Center

We apologize for any inconvenience caused by this incident. A summary of the events and measures taken follows.

Summary

Kundo was again the target of an attack causing performance issues and temporary unavailability in Knowledge and Help Center.

With help from our previous learnings and actions taken from the earlier attacks the impact of the attack was severely limited. Our new ways of blocking attackers proved effective and we managed to get the system up and running smoothly.

Timeline (in CET)

13:14 An alarm about reduced performance in Knowledge is triggered.

13:30 Our incident process is initiated and troubleshooting is started by the whole development team.

13:33 - The root cause of the incident is identified to be an unusual large amounts of requests to our servers making Knowledge unreachable.

13:44 - Traffic load is back at normal levels and Knowledge is available again

13:51 - We block requests from identified attackers.

14:08 - We identify that a misconfiguration led to degraded performance across all Kundo services

14:09 - Misconfiguration is fixed

14:14 - All systems including the Knowledge are confirmed to available and Incident status is set to Resolved

What happened?

Kundo was the target of an attack consisting of large amounts of security vulnerability scannings of Knowledge and Help Center. The large amount of requests made Knowledge and Help Centers intermittently inaccessible. 

To mitigate the problem the malicious requests from the attackers were blocked.

Actions to mitigate impact of failures like these in future

Our previous actions from last week have made sure we are more suited to cater attacks like this and the impact on the system was severely limited compared to the incident the week before.

Based on the learnings from this incident we have now also implemented further ways to block and limit the impact of attacks.

Further details about last weeks incidents 

During the last couple of weeks we have seen a large increase in the amount of security attacks on IT infrastructure in the society which has also been well covered in the media. Kundo is no exception and even though we had many preventive measures in place already we have not been able to keep the system availability on the levels we normally take pride in delivering.

Based on the learnings from the incidents we have made sure that the system is even more robust and scalable - which has now been proved over a series of events.

We apologize for any inconvenience the incidents may have caused and give our sincere promise to continuously evolve Kundo and make it a better product each and every week.

You are most welcome to contact us via email for more information: support@kundo.se

Posted Feb 24, 2023 - 13:48 CET

Resolved
This incident has been resolved.
Posted Feb 16, 2023 - 13:48 CET
Monitoring
The problem is now resolved and we are monitoring the situation.
Posted Feb 16, 2023 - 13:45 CET
Investigating
Some users are experiencing problem accessing Knowledge / Help Center at this moment in time. We are investigating the issue.
Posted Feb 16, 2023 - 13:33 CET
This incident affected: Knowledge and Help Center.