Incident Feed - Arbutus

Arbutus Volume Storage Issue

2025-08-03 15:20

RESOLVED - Volume storage is operating normally again, though latency may remain high for a little while.

--------------------------

We are currently experiencing an issue with the storage backend for volumes in the Arbutus Cloud. Some volume data or volume actions may currently be unusable.

Elevated Volume Latency

None

We are experiencing an increase in volume latency while our storage cluster recovers from some hardware failures. Client impacts to volume performance and usablility are expected.

Increased Volume Latencies

None

Arbutus volumes are experiencing elevated read/write latencies due to the system recovering from a hardware failure.

Increase Volume Latency

2025-09-13 03:12

Arbutus is experiencing a period of increased storage volume latency due to hardware failure.

Arbutus Object Storage Performance Degredation

None

Update: The issue has been resolved and service has returned to normal.

The Arbutus Object Storage Service is currently experiencing a slow down impacting traffic. The issue is being investigated.

Intermittent Arbutus Object Storage Interruptions

2025-12-02 03:40

We are currently seeing some interruptions with processing object storage requests.

Power outage - Panne électrique

None

All systems are back online.

-----------------------------------------

Update 2026-01-19 13:00 Pacific

With many workloads coming back from a cold start, there is heightened pressure on the network and storage latency of Arbutus Cloud. We expect some latency sensitive workloads may be impacted for the next 24h until workloads normalize.

-----------------------------------------

Update 2026-01-19 00:15 Pacific

Arbutus Cloud has been restored, and any remaining issues are being tracked down and resolved by the team. All previously running VMs will have been restarted, so their current state will have to be verified on a case by case basis. If you are still experiencing issues please submit a ticket.

-----------------------------------------

Update 2026-01-18 15:30 Pacific

Arbutus Cloud restoration continues, and many compute hypervisors and their VMs are available now, with further nodes coming online throughout the day. DBAAS is also available.

Further updates to come.

-----------------------------------------

Update 2026-01-18 01:00 Pacific

Power has been restored to the site. Volume, CephFS storage, and S3 access has been fully restored.
The openstack control plane is up and available, however only the persistent hypervisors and their VMs have been brought back online.
Compute instances will start to be brought back online starting in the AM Pacific time.

Thank-you for your patience as we restore full service.

-----------------------------------------

There is a power outage at this site. Il y a une panne de courant sur ce site.

Arbutus Power Maintenance

2026-02-21 21:00

The Arbutus site is undergoing power maintenance over the next few days. No impact is expected.

Network Maintenance

None

On March 4, there will be local network maintenance between 7:00am-8:00am PST. Arbutus traffic is not expected to be impacted, but access to TSM and local clients will be unavailable for about 30 minutes during the windows.

Arbutus Power Maintenance

None

The Arbutus site is undergoing power maintenance over the next few days. No impact is expected.

Arbutus Object Storage Slowness

None

We are experiencing intermittent slowness on Legacy Arbutus Object Storage (https://object-arbutus.cloud.computecanada.ca/) due to heavy traffic. New Arbutus Object Storage (https://object-arbutus.alliancecan.ca/) is not impacted.

Network Maintenance

None

On May 12, there will be local network maintenance between 7:00am-8:00am PST. Arbutus traffic is not expected to be impacted, but access to TSM and local clients will be unavailable for about 30 minutes during the windows.

Authentication Issues

None

We're currently investigating authentication issues on new Arbutus cloud. Access to the API and dashboard appear to be impacted, but running VMs are unaffected.

Arbutus OpenStack Maintenance

2026-05-06 21:00

There will be internal OpenStack Database maintenance occurring between 10AM-2PM Pacific Time. No impacts are expected as a result of this work.

Intermittent Arbutus Object Storage

2026-05-11 21:00

We are experiencing some interruptions on legacy Arbutus Object Storage, some requests are taking a long time to return.

Problems Starting Instances

None

We're seeing problems starting instances on Arbutus and are investigating.

Arbutus Cloud Manila Issues

None

We are currently experiencing issues with the Manila service in the cloud. Currently no new CephFS shares are able to be created. We are investigating.

Legacy Arbutus Control Plane Maintenance

2026-05-29 23:00

Some components in the Legacy Arbutus control plane are undergoing maintenance. No user impact is expected.

Software upgrades to OpenStack control plane

2026-06-01 22:10

OpenStack control plane software is having a component upgraded and this could cause a brief outage for authentication for the web interface (Horizon) and for the api. Only a minute or less of impact is expected.

Problems with new virtual machine creation

2026-06-03 23:33

We are currently investigating issues with virtual machine creation. Newly launched instances may take a long time to respond to console access in Horizon and be available for ssh access after creation. Investigation is ongoing and no ETA for resolution yet.

Problem was located in the storage system and fixed.

Control plane networking outage

2026-06-19 00:15

The database that underpins the virtual networking in Arbutus is being moved to faster storage which should result in fewer issues that impact the networking for Arbutus. Expected network interruption is ~5-10 min. Full details for the expected impact are below.

Expected Impact During the Change

Existing VM traffic (data plane): Unaffected during StatefulSet and pod deletion — OVS flow tables are cached in the kernel on each compute node and continue to forward traffic without the OVN control plane
OVN control plane unavailability (~5–10 min): During pod and PVC deletion and while new pods initialize, no new OVN logical flow updates will be processed
New VM boots, port creation, network/router operations, floating IP associations, security group changes: Temporarily queued or delayed — Neutron API remains available but changes will not be reflected in the data plane until the sync completes
Data plane blackout (~15–20 min): Once OVS agents on compute nodes detect loss of the OVN controller connection, cached flows are flushed — existing VM traffic will be interrupted for the duration of the Neutron sync repair
Sync / restoration: neutron-ovn-db-sync-util repopulates ~2,400 ports, ~307 routers, ~531 security groups, ~298 floating IPs, and ~313 networks from Percona XtraDB; estimated sync duration is 5–10 minutes based on a dry-run conducted in advance
Estimated total maintenance window: ~20–30 minutes from start to full data plane restoration