Monitoring - All the cloud servers in UK-LON1 have now been migrated to more stable platforms. We had to restart few individual servers had as they encountered unexpected errors during the maintenance. We apologize the inconvenience.
If your server is still experiencing any issues please first check server's status with Control Panel's console and if necessary try to apply shutdown and restart. If the problem persists please contact our support.
Jun 27, 00:28 UTC
Update - About 67% of the maintenance is done.
Jun 26, 22:50 UTC
Update - About 67% of the maintenance is done. If you have any servers in that DC that are not behaving as expected it is safe to give a full shutdown/restart request from our control panel. This will not affect our work, and in most cases, the shutdown/restart can fix any issues that might be related to this maintenance.
Jun 26, 15:29 UTC
Identified - Since June 13th, we have had multiple incidents of compute node crashes in our London data centre. We would like to give some insight behind these, as we owe our users an explanation for this unusual situation.

Earlier this year in May, a new Intel CPU vulnerability was discovered (https://upcloud.com/blog/mds-vulnerabilities/), which prompted us to swiftly run security updates throughout our infrastructure to protect our users data. This is something we have had to do in the past for similar Intel CPU vulnerabilities such as Meltdown (https://upcloud.com/blog/intel-cpu-vulnerability-meltdown/), and which we did with great success.

Around the same time, we also rolled out new versions of our infrastructure software, in preparation of new features and products that we are currently working on. Unfortunately, as we discovered, the combination of certain hardware in the London data centre, our new software features, together with the Intel CPU vulnerability mitigations, proved to cause instability over time in specific situations.

UK-LON1 has been the only data centre to experience these situations and we have been working tirelessly to identify the root cause and administer a fix. We are confident that we have identified the root cause at this point, and have already begun rolling out updates. These updates require no actions on our users part, and will not cause any disturbance to your services at UpCloud.

The rollout is estimated to conclude within the next 36 hours, by the end of Thursday, 27th of June, 2019.

We will post any future updates here, so please make sure to subscribe or check back for any updates.

We are truly sorry for the loss of service our users have endured.

Antti Vilpponen
CEO
Jun 26, 11:49 UTC
UpCloud Control Panel Operational
API Operational
DE-FRA1: Network Connections Operational
DE-FRA1: Virtualization Hosts Operational
DE-FRA1: Storage Backends Operational
FI-HEL1: Network Connections Operational
FI-HEL1: Virtualization Hosts Operational
FI-HEL1: Storage Backends Operational
FI-HEL2: Network Connections Operational
FI-HEL2: Virtualization Hosts Operational
FI-HEL2: Storage Backends Operational
NL-AMS1: Network Connections Operational
NL-AMS1: Virtualization Hosts Operational
NL-AMS1: Storage Backends Operational
SG-SIN1: Network Connections Operational
SG-SIN1: Virtualization Hosts Operational
SG-SIN1: Storage Backends Operational
UK-LON1: Network Connections Operational
UK-LON1: Virtualization Hosts Operational
UK-LON1: Storage Backends Operational
US-CHI1: Network Connections Operational
US-CHI1: Virtualization Hosts Operational
US-CHI1: Storage Backends Operational
US-SJO1: Network Connections Operational
US-SJO1: Virtualization Hosts Operational
US-SJO1: Storage Backends Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
We will be conducting maintenance of the UpCloud API and control panel, to further improve reliability and security. The estimated time of the maintenance is up to 1 hour, and during this window, both the API and control panel will be partly unavailable.
Posted on Jun 26, 12:12 UTC
Past Incidents
Jun 26, 2019
Resolved - This incident has been resolved.
Jun 26, 11:50 UTC
Identified - We have identified an issue with one of the compute nodes in our London data centre. The problem may cause affected cloud servers to become slow or unresponsive.
We are working to bring all affected servers back online and resolve the situation as soon as possible.

If you need to contact our Support at support@upcloud.com
Jun 26, 11:19 UTC
Resolved - This incident has been resolved.
Jun 26, 09:12 UTC
Monitoring - We have restored all affected cloud servers to normal operating status and continue to monitor the situation.
If your server is still not fully functional, please contact our support at support@upcloud.com.
We apologise for the incident.
Jun 26, 08:28 UTC
Identified - We have identified an issue with one of the compute nodes in our London data centre. The problem may cause affected cloud servers to become slow or unresponsive.
We are working to bring all affected servers back online and resolve the situation as soon as possible.
Jun 26, 07:49 UTC
Jun 25, 2019
Resolved - This incident has been resolved.
Jun 25, 16:53 UTC
Identified - We have identified an issue with one of the compute nodes in our London data centre. The problem may cause affected cloud servers to become slow or unresponsive.
We are working to bring all affected servers back online and resolve the situation as soon as possible.
Jun 25, 16:39 UTC
Resolved - This incident has been resolved.
Jun 25, 10:32 UTC
Monitoring - We have restored all affected cloud servers to normal operating status and continue to monitor the situation.
If your server is still not fully functional, please contact our support at support@upcloud.com.
We apologise for the incident.
Jun 25, 10:32 UTC
Identified - We have identified an issue with one of the compute nodes in our London data centre. The problem may cause affected cloud servers to become slow or unresponsive.
We are working to bring all affected servers back online and resolve the situation as soon as possible.
Jun 25, 10:04 UTC
Jun 24, 2019
Resolved - This incident has been resolved.
Jun 24, 12:32 UTC
Investigating - Some users might experience problems with our Control Panel and public website due to third party provider network issues. We are working to mitigate the problem.
Jun 24, 11:33 UTC
Jun 23, 2019

No incidents reported.

Jun 22, 2019
Resolved - The incident has been resolved and all the affected cloud servers have been brought back online. If your server still experiences any problems please contact our support.
Jun 22, 12:35 UTC
Update - Unexpected hardware failure affected multiple compute hosts in London data centre this morning. Most of the impacted cloud servers have already been brought back online. We continue working with the remaining ones and expect everything to be running in the next few hours.
Jun 22, 12:12 UTC
Identified - We have identified an issue with one of the compute nodes in our London data centre. The problem may cause affected cloud servers to become slow or unresponsive.
We are working to bring all affected servers back online and resolve the situation as soon as possible.
Jun 22, 05:42 UTC
Jun 21, 2019

No incidents reported.

Jun 20, 2019
Resolved - This incident has been resolved.
Jun 20, 09:35 UTC
Monitoring - We have restored all affected cloud servers to normal operating status and continue to monitor the situation.
If your server is still not fully functional, please contact our support at support@upcloud.com.
We apologise for the incident.
Jun 20, 09:25 UTC
Identified - We have identified an issue with one of the compute nodes in our London data centre. The problem may cause affected cloud servers to become slow or unresponsive.
We are working to bring all affected servers back online and resolve the situation as soon as possible.
Jun 20, 09:10 UTC
Jun 19, 2019

No incidents reported.

Jun 18, 2019
Resolved - This incident has been resolved, we apologise for the inconvenience caused.
Jun 18, 06:10 UTC
Monitoring - All servers have been restored. We will continue to monitor.
Jun 18, 03:45 UTC
Investigating - We are investigating a compute node issue that is affecting some cloud servers in our LON data centre.
Jun 18, 03:11 UTC
Jun 17, 2019
Resolved - This incident has been resolved.
Jun 17, 04:18 UTC
Identified - The issue has been identified and a fix is being implemented.
Jun 17, 04:13 UTC
Investigating - We are investigating a compute node issue that is affecting some cloud servers in our LON data centre.
Jun 17, 03:40 UTC
Jun 16, 2019
Resolved - This incident has been resolved, we apologise for the inconvenience caused.
Jun 16, 16:07 UTC
Identified - The issue has been identified and a fix is being implemented.
Jun 16, 15:12 UTC
Investigating - We are investigating a compute node issue that is affecting some cloud servers in our LON data centre.
Jun 16, 15:12 UTC
Jun 15, 2019

No incidents reported.

Jun 14, 2019
Resolved - This incident has been resolved.
Jun 14, 20:36 UTC
Update - We are continuing to work on a fix for this issue.
Jun 14, 20:23 UTC
Identified - We have identified an issue with one of the compute nodes in our London data centre. The problem may cause affected cloud servers to become slow or unresponsive.
We are working to bring all affected servers back online and resolve the situation as soon as possible.
Jun 14, 20:13 UTC
Jun 13, 2019
Resolved - This incident has been resolved.
Jun 13, 23:03 UTC
Monitoring - All servers have been restored. We will continue to monitor.
Jun 13, 22:40 UTC
Update - We are continuing to work on a fix for this issue.
Jun 13, 22:04 UTC
Identified - We have identified an issue with one of the compute nodes in our London data center. The problem may cause affected cloud servers to become slow or unresponsive.
We are working to bring all affected servers back online and resolve the situation as soon as possible.
Jun 13, 22:03 UTC
Resolved - This incident has been resolved.
Jun 13, 22:03 UTC
Update - Compute Node Issue has been resolved.
Jun 13, 20:36 UTC
Identified - We have identified an issue with one of the compute nodes in our London data centre. The problem may cause affected cloud servers to become slow or unresponsive.

We are working to bring all affected servers back online and resolve the situation as soon as possible.
Jun 13, 20:02 UTC
Resolved - This incident has been resolved.
Jun 13, 13:04 UTC
Update - Telia has still problems in their end which means non-optimal routing and increased latency to some destinations from Singapore Cloud Servers. We will continue to follow the situation with them and aim to restore full connectivity as soon as it's possible.
Jun 13, 11:25 UTC
Monitoring - One of our Internet transit providers in Singapore, Telia, reported a problem in their side. In addition there were some global routing issues at another provider. Connections to Cloud Servers located in Singapore might still be slower than usual from some locations. We will continue to monitor the situation and make necessary changes to restore optimal network performance.
Jun 12, 17:00 UTC
Identified - We have identified an issue with network connectivity in our Singapore data centre. The affected cloud servers may experience slow connections or become unreachable.
We are working to restore connectivity and resolve the situation as soon as possible.
Jun 12, 16:21 UTC
Resolved - Payment gateway has resumed normal operations and credit card payments should now go through again.
Jun 13, 11:22 UTC
Identified - We are currently experiencing issues with our 3rd party payments system. Some users may be unable to add further credits to their account during this time.
Jun 13, 09:48 UTC