Incident Feed - Nibi

Power outage - Panne électrique

2025-07-26 14:00

There is a power outage at this site. / Il y a une panne de courant sur ce site.

Outage - Panne

2025-07-29 15:00

Nibi is currently offline due to an unknown issue. We are investigating.

---

Nibi est présentement hors ligne à cause d'un problème dont la cause est inconnue. Nous analysons le système.

Node outage for circuit-breaker upgrades

2025-08-09 16:00

Compute nodes will be powered-off during an upgrade of their electrical feed. This will not affect login access or storage.

A reservation is in place which will prevent jobs from starting during the outage, but you are free to submit jobs that will run after nodes have returned to service.

Jobs running at the time of the outage will be killed, and will need to be resubmitted.

We expect Nibi compute nodes to be available again around noon on Friday, the 8th.

Stability Issues related to file systems

2025-08-21 00:30

Nibi is currently experiencing stability issues related to our CVMFS and NFS file systems. Jobs and nodes may fail, become temporarily inaccessible, or perform below expectations. We are working to resolve the issue.

Emergency power outage affecting compute/gpu nodes

2025-09-09 21:38

An emergency power outage TODAY will affect all Nibi compute nodes (including GPU).

This means all jobs will be killed at 4pm, and you'll need to resubmit any such jobs. Currently queued jobs will be started when the cluster returns to service.

We do not have a firm estimate of how long the outage will last - perhaps an hour, but there will be some time required after that to ensure that nodes are operating correctly before allowing them to run jobs again.

Planned Outage - Arrêt planifié

2025-09-29 16:00

Nibi requires some final configuration, updates, and benchmarking to be conducted. The CPU nodes will be unavailable from 9am on Sept 22 until noon on Sept 26th. During this time the GPU nodes, login nodes, storage and Nibi cloud will remain available for use. Queued jobs will remain in slurm waiting for these CPU nodes to return to service.

Scheduler problem - Problème d'ordonnanceur

2025-10-02 11:23

After the outage, problems related to Slurm and MPI are occurring (eg: "Incompatible plugin version"). We are working to fix these issues and will avoid starting new jobs until they are solved.

GPU Nodes Issue

2026-01-01 06:00

There is a problem with Nibi's GPU nodes that will cause GPU jobs to fail with something like "Unexpected error from cudaGetDeviceCount()".
We are working on an update and fix, until then, no GPU jobs are currently running and no new GPU jobs will start.

CPU Node partial outage - Arrêt partiel des nœuds CPU

2025-11-20 15:08

UPDATE: November 20th, 2025 - Repairs to the cooling system have been completed and the cluster is now fully operational.

Some of the Nibi CPU nodes may need to be shutdown to facilitate repairs to the immersion cooling tanks.

This does not affect your ability to login, access files, or run jobs; it only means that the cluster will be able to run fewer CPU (not GPU) jobs.

---

Une portion des nœuds de calculs CPU de Nibi pourrait être mise à l’arrêt pour faciliter des réparations.

Vous pourrez toujours vous connecter à la grappe, accéder à vos fichiers et exécuter des tâches de calcul ; le nombre de tâches CPU exécutées par la grappe sera toutefois réduit. (Les tâches GPU ne seront pas affectées.)

Cluster access is currently unavailable

2025-11-10 14:28

There is an ongoing issue with the Nibi cluster that is currently preventing users from logging in. This issue does not affect any running jobs.

Scheduler problem - Problème d'ordonnanceur

2026-01-16 17:00

A problem is causing job failures with unexpected "Permission denied" errors for a subset of users and nodes.

We believe these failures have stopped.

Access is now restricted to users who requested access - L'accès est maintenant réservé aux personnes qui ont demandé l'accès

2026-01-23 18:09

We have discovered that a misconfiguration allowed users who had not requested access to Nibi. This has been fixed. Access to Nibi can be requested here:

https://ccdb.alliancecan.ca/me/access_systems

Nous avons découvert qu'un problème de configuration permettait aux usagers n'ayant pas demandé accès à Nibi d'y avoir accès. Ceci a été corrigé. L'accès à Nibi peut être demandé ici :

https://ccdb.alliancecan.ca/me/access_systems

Scheduler problem - Problème d'ordonnanceur

2026-02-09 15:05

The job scheduler is misbehaving, which may impair the ability to start new jobs. We are investigating. / L'ordonnanceur a un problème, ce qui peut empêcher de démarrer de nouvelles tâches. Nous investiguons actuellement ce qui se passe.

Nibi Cloud Dashboard Update

2026-02-13 14:58

We’re currently seeing intermittent dashboard issues. Running virtual machines are operating normally and are not impacted.

Scheduler problem - Problème d'ordonnanceur

2026-02-13 16:00

The ultimate trigger of this problem seems to have been a power issue early this morning. It caused a cascade of effects involving VMs being moved between hypervisors, some VMs rebooting, possibly some storage issues, and affected a number of daemons (sssd, munge, slurmd on compute nodes, slurm controller and DB VMs) that would give user-visible errors.

Power outage - Panne électrique

2026-03-28 23:10

We are investigating a power outage at Nibi, all compute nodes are unavailable atm.

Operating, but some operations may be disrupted.

2026-04-30 14:30

Nibi is online and operating, but some operations may be disrupted.

Specifically, there was an initial, emergency mitigation of a recent security flaw which involved disabling set-uid features. This would have prevented any user of fusermount (sshfs, squashfs, some others), and may have affected virtual desktop creation (via OnDemand, including the Jupyterlab component).

A final fix for the security issue is in place and we'll be restoring these specific edge cases shortly.

some login and robot failures

2026-05-06 16:30

A problem relating to security settings caused connection failures for login and robot nodes during this time. We apologize for the disruption!

/nearline filesystem problem - Problème avec le système de fichiers /nearline

2026-05-14 20:27

There is a problem with the /nearline filesystem.

Impacts: slow access or no access
Cause: To be determined
Solution: To be determined
Estimate time to solution: To be determined

======

Il y a un problème avec le système de fichiers /nearline.

Impacts : accès lents ou accès bloquants
Cause : à déterminer
Solution : à déterminer
Heure estimée de résolution : à déterminer

Filesystem problem - Problème de système de fichiers

2026-06-02 22:30

There is a problem with the /nearline filesystem on Nibi, which seems to be unresponsive. We are investigating. / Il y a un problème avec le système de fichiers /nearline sur la grappe Nibi, qui ne réagit pas correctement. Nous investiguons actuellement ce qui se passe.

Regional power-reduction effort

2026-07-01 00:00

At the request of power authorities, we're reducing power usage between 6-8pm tonight.

Your job may be delayed in starting if its expected runtime intersects with this period. No running jobs will be affected.

Reduced throughput / power shedding

2026-07-07 02:00

The power utility has requested that we reduce power usage between 4-8PM today.

The scheduler has been configured to drain nodes during this time, so your job may be delayed in starting. Note that already running jobs will not be affected - neither will short jobs which end before the event.