8th May 2020

Emergency maintenance (FRA01), scheduled 1 year ago

Since yesterday we are facing issues with our IP address allocation system which is responsible for assigning IP addresses to containers. Due to what we think a split-brain, this system is inconsistent state causing it to function improperly.

So far we have done anything possible to repair the situation, without affecting the runtime of our containers. But after 24 hours of debugging we have to come to the conclusion that probably the only way is to get out this situation is to reinitialize every node in the affected cluster meaning we have to move all running containers to other nodes.

Normally moving containers between nodes will hardly be noticed, but as we expect to do it multiple times there is a good change your applications will be affected by this.

This evening at 23:00, Central European Time (CET) we will start the operation, causing your containers to be restarted once or multiple times. As there are a lot of containers to be restarted we cannot be sure how long this will take, but we do everything within our power to get the platform into the correct state again.

We sincerely apologize for any inconvenience caused and that we have to perform this emergency maintenance on such a short notice without a proper announcement.