We sincerely apologize for the loss of service you had to endure last night between 18:30 and 20:00 EEST in our FI-HEL1 data centre.

While we realise this post mortem will not excuse us for the loss of service, we do wish it explains our actions and what we will do to avoid such issues in the future.

All of UpCloud’s FI-HEL1 servers lost contact to the internet at 18.30 EEST. Our monitoring software alerted us to this immediately and we began to investigate the issue. We updated our Status page on the incident at 18.42 EEST while our staff continued their work.

At about 18.54 EEST we had narrowed this incident down to a distributed denial of service attack.

We mitigated the attack by 19:45 EEST so that 3/4 of the servers were back online. By 20:00 EEST all servers were connected normally.

The attack continued late into the night, but it did not affect our customers due to filtering and restrictions we put in place.

We will be issuing all affected customers compensation according to our Service Level Agreement. We will send out a separate e-mail to all administrative contacts with the amount as we have credited the accounts.

While we cannot pre-emptively prepare to avoid all possible outages, we do prepare ourselves and our corporate processes so we can fix them as soon as possible. During such incidents we try to communicate to our best extent (without affecting repair work) on unaffected third party services, such as our status page and Twitter -account.

You can follow us on Twitter with @upcloudcom and subscribe to service updates at http://status.upcloud.com.

Again, we are truly sorry for the loss of service.

Joel Pihlajamaa
Chief Technology Officer

Antti Vilpponen
General Manager