2026-07-07 20:19:09+00:00 1390 <p>Most of the work has been completed. <br></p><p>The Fir login nodes can be reached via fir.alliancecan.ca</p><p>The filesystem is the same filesystem that you can see on cedar.</p><p>You can find the documentation about node configurations here: https://docs.alliancecan.ca/wiki/Fir</p><p>Due to power and cooling restraints we are working to reduce the load on cedar so that we can bring all of fir's cpu nodes online</p><p><font color="#000000"><span style="background-color: rgb(255, 255, 0);">Large memory nodes</span> are <b>now available</b></font></p><p><font color="#000000"> <span style="background-color: rgb(255, 255, 0);">robot.fir.alliancecan.ca&nbsp;</span></font><font color="#ffffff"><span style="background-color: rgb(255, 255, 0);"><span style="background-color: rgb(255, 255, 255);"> </span></span></font><font color="#000000">is <b>now available</b></font></p><p><font color="#000000"> <span style="background-color: rgb(255, 255, 0);">email notifications </span>will become available soon.</font></p><p>Thank you for your patience</p> None 2025-08-08 21:53 1400 <p><br class="Apple-interchange-newline">The power contractors have requested that we reduce power usage between Monday August, 25 and Friday Aug 29th.</p><p>The scheduler has been configured to drain nodes during this time, so your job may be delayed in starting.&nbsp; Note that already running jobs will not be affected - neither will short jobs which end before the event.</p><div><br></div> 2025-09-04 22:13 2025-08-25 14:38 1403 <p>There is a problem with the filesystem on Fir, which is showing highly variable response times to interactive commands such as 'ls' and 'cd'.&nbsp;<span style="font-size: 1rem;">We are investigating.</span></p><p>Il y a un problème avec le système de fichiers sur Fir, qui affiche des temps de réponse très variables aux commandes interactives telles que « ls » et « cd ».&nbsp;&nbsp;<span style="font-size: 1rem;">Nous investiguons actuellement ce qui se passe.</span></p><p><span style="font-size: 1rem;"><br></span></p><p><span style="font-size: 1rem;">The issue has been resolved</span></p> 2025-08-28 23:48 2025-08-27 18:00 1410 <p><br></p><p>There is a problem with the /nearline filesystem.</p><p> <br></p><ul> <li>Impacts: file restoration is not functioning</li> <li>Cause: To be determined</li> <li>Solution: To be determined</li> <li>Estimate time to solution: To be determined</li> </ul><p> <br></p><p>======</p><p> <br></p><p>Il y a un problème avec le système de fichiers /nearline.</p><p> <br></p><ul> <li>Impacts : accès lents ou accès bloquants</li> <li>Cause : à déterminer</li> <li>Solution : à déterminer</li> <li>Heure estimée de résolution : à déterminer</li> </ul><p><br></p> 2025-09-03 01:32 2025-09-02 21:44 1423 <p><br> </p><p>There is a problem with the /home filesystem.</p><ul> <li>Impacts: slow access or no access</li> <li>Cause: To be determined</li> <li>Solution: To be determined</li> <li>Estimate time to solution: To be determined</li> </ul><p>======</p><p>Il y a un problème avec le système de fichiers /home.</p><ul> <li>Impacts : accès lents ou accès bloquants</li> <li>Cause : à déterminer</li> <li>Solution : à déterminer</li> <li>Heure estimée de résolution : à déterminer</li> </ul> 2025-09-18 00:00 2025-09-17 21:41 1424 <p>SFU will move external routing service to 400G path on Monday 9PM. We expect only a short WAN outage. Running jobs will not be affected.&nbsp;</p><p>The work is completed.</p> None 2025-09-23 04:00 1436 <p>Some compute nodes do not mount correctly CVMFS resulting in error like:</p><p><span style="color: rgb(0, 0, 0); font-family: &quot;Times New Roman&quot;; font-size: medium;">/cvmfs/soft.computecanada.ca/custom/software/lmod/lmod/libexec/lmod: No such file or directory</span></p><p><br></p> 2025-10-06 23:11 2025-10-04 04:00 1443 <p>There is a partial power outage at this site. / Il y a une panne de courant&nbsp;partielle sur ce site.</p><p>It will have caused some jobs to abnormally terminate.&nbsp;</p><p>We are investigating the cause.</p> 2025-10-18 21:36 2025-10-18 18:30 1453 <p>Due to the failure of the tape library, /nearline restore is unavailable until further notice. A support ticket to vendor was opened.</p><p>---</p><p>À cause d’une défaillance de la bibliothèque de bandes magnétiques, `/nearline` n’est pas disponible jusqu’à nouvel ordre. Nous travaillons avec le vendeur pour résoudre ce problème.</p> 2025-11-05 23:30 2025-11-04 18:30 1456 <p><br></p><p>There is a problem with the /nearline filesystem. File restore from tapes is not working after the repair of the tape library. We are investigating the issue.</p><p>/nearline restore is working now.</p> 2025-11-09 18:18 2025-11-06 05:42 1458 <p>Fir Lustre filesystem servers crashed and are unable to recover at this time. We will contact the vendor to resolve this issue.</p><p>---</p><p>Les serveurs du système de fichiers réseau sont en panne. Nous travaillons avec le vendeur pour résoudre ce problème.</p> 2025-11-09 22:00 2025-11-07 16:19 1473 <p>Fir's lustre filesystem is currently unresponsive due to high load on the MDS serer.</p><p>The team is working on restoring the services back to normal</p><p>---</p><p>Le système de fichiers ne répond plus à cause d’une surcharge.</p><p>Nous travaillons à rétablir le service.</p> 2025-12-01 17:43 2025-11-24 17:28 1478 <p>Access to login nodes is disrupted. We are investigating. / L'accès aux noeuds de connexion est interrompu. Nous investiguons actuellement ce qui se passe.<br></p> 2025-12-01 00:04 2025-11-30 18:00 1480 <p>Fir external network connection will be moved from dark fiber to a DWDM channel between Dec. 2 1PM to 2PM. Network connection will failover to the backup 400G link during the outage. We do not expect any significant loss of network connection during the change.</p><p>---</p><p>Le réseau externe de Fir sera mis à jour le 2 décembre entre 13:00 et 14:00. Pendant cette période, les connexions seront routées vers le lien de secours à 400G.&nbsp; Nous n’anticipons pas de perte de données réseau significatives pendant ce changement.</p> 2025-12-02 22:30 2025-12-02 21:00 1485 <p>Due to a linecard failure at SFU campus core router, access from SFU campus to FIR is currently unavailable. This outage does not affect any off campus traffic to FIR</p><p>SFU IT fixed the issue.</p> 2025-12-06 16:18 2025-12-06 09:00 1495 <p>Access to login nodes is disrupted. We are investigating. / L'accès aux noeuds de connexion est interrompu. Nous investiguons actuellement ce qui se passe.<br></p> 2025-12-23 21:04 2025-12-23 19:55 1506 <p>Access to login nodes is disrupted. We are investigating. / L'accès aux noeuds de connexion est interrompu. Nous investiguons actuellement ce qui se passe.<br></p> 2026-01-10 17:55 2026-01-10 01:00 1515 <p>Fir's nearline backup service will be undergoing maintenance on 2026-02-04&nbsp;</p><p>During this time restores from tape will be unavailable.</p><p>The work is expected to be completed by the end of the day.</p><p>======</p><p>Il y aura une maintenance du service de sauvegarde nearline de Fir le 4 février.</p><p>Pendant ce temps, il ne sera pas possible de restaurer des données à partir des sauvegardes sur ruban.</p><p>Nous planifions que le service sera de retour en fin de journée.</p> None 2026-02-04 16:00 1523 <p>There is a problem with the (CHOOSE TYPE(S): /home /project /scratch ) filesystem on SYSTEM, which seems to be unresponsive. We are investigating. / Il y a un problème avec le système de fichiers (CHOOSE TYPE(S): /home /project /scratch ) sur la grappe SYSTEM, qui ne réagit pas correctement. Nous investiguons actuellement ce qui se passe.<br></p> None 2026-01-30 18:35 1538 <p>Hello,</p><p>Unfortunately, this morning we noticed that fir's lustre filesystem became unstable again. We are aware that there is currently a regression bug in lustre that causes this issue. We are investigating with the vendor and are working on bringing the filesystem back into service.</p><p>Thank you for your patience</p> 2026-02-20 21:05 2026-02-20 17:18 1541 <p>The process took much longer than anticipated.&nbsp; The vendor had requested that we wait so they could gather some analysis.&nbsp; We resolved this current issue overnight.</p><p><br></p><p>There is a problem with the /home /project /scratch filesystems on Fir, which seems to be unresponsive. We are investigating. / Il y a un problème avec le système de fichiers /home /project /scratch sur la grappe Fir, qui ne réagit pas correctement. Nous investiguons actuellement ce qui se passe.</p><p>We are actively monitoring and the fix is known, it will just take a relatively short period of time.</p><p>Thanks for your patience.</p> 2026-02-26 10:02 2026-02-25 22:05 1543 <p>There is a problem with the /home /project /scratch filesystems on Fir, which seems to be unresponsive. We are investigating. / Il y a un problème avec le système de fichiers /home /project /scratch sur la grappe Fir, qui ne réagit pas correctement. Nous investiguons actuellement ce qui se passe.</p><p>We are actively working to restore service and the fix is known.</p><p>Thanks for your patience.</p> 2026-04-13 19:18 2026-04-13 19:17 1562 <p>Update:</p><p>Power was restored; the power outage took all compute nodes offline and running jobs were, as a result, cancelled.&nbsp;&nbsp;</p><p>We took the opportunity to patch a filesystem bug which will hopefully make the filesystem quite a bit more stable than before.</p><p>-----------------</p><p><br></p><p>There is a power outage at this site. / Il y a une panne de courant sur ce site.<br></p> 2026-03-23 23:00 2026-03-23 20:48 1563 <p><span style="font-size: 1rem;">CDU is back online and the GPUs that were affected are also back online.</span></p><p><span style="font-size: 1rem;"><br></span></p><p><span style="font-size: 1rem;">Fir is currently partially offline due to a cooling issue. Roughly 50% of the GPUs are offline after a Cooling Distribution Unit (CDU) suffered a failure during routine maintenance and testing. We are working with the vendor to resolve the issue.</span></p> 2026-03-25 21:00 2026-03-24 20:34 1565 <p><br></p><p>Fir will be at reduced capacity from Friday March 27, 2026 through March 30th. During this time, you will not be able to log in or run jobs on the cluster. Any jobs that are running at the time of the outage will be terminated and you will need to re-submit them once the cluster is back online.</p> 2026-03-30 22:35 2026-03-28 01:13 1586 <p>Following a security incident, we have temporarily suspended login and JupyterHub access. We are actively working to resolve the situation and will provide further details shortly.</p> <p>Thank you for your patience and understanding.</p> <p>======</p> <p>Suite à un incident de sécurité, nous avons procédé à la fermeture temporaire des accès aux noeuds de connexion et à JupyterHub. Nous travaillons activement à résoudre la situation et reviendrons vers vous avec plus de détails sous peu.</p> <p>Merci de votre compréhension.</p><p></p> <p><br></p> 2026-05-01 16:06 2026-04-29 21:14 1594 <p>Due to cooling system maintenance.</p><p>Fir will be unavailable on May 12 for the whole day. During this time, login node service may be frequently unavailable as we upgrade the storage.&nbsp; Compute nodes and interactive jobs will be completely unavailable as we attempt to upgrade other services during this outage time.&nbsp; Any jobs that are running at the time of the outage will be terminated and you will need to re-submit them once the cluster is back online.</p> 2026-05-14 00:00 2026-05-12 11:00 1606 <p>ptrace functionality has been restored as another mitigation is in place.</p><p><br></p><p>We have temporarily disabled the `ptrace` function due to a security issue. Tools that use `ptrace` do not work at this time, notably debuggers (GDB...) and profilers (perf, Valgrind...).</p><p>---</p><p>Nous avons temporairement désactivé la fonction `ptrace` à cause d’un problème de sécurité. Les outils qui utilisent `ptrace` ne fonctionnent pas en ce moment, notamment les débogueurs (GDB...) et profileurs (perf, Valgrind...).</p> 2026-05-26 05:47 2026-05-14 14:01 1614 <p>There is a problem with the filesystem on Fir, which seems to be unresponsive. We are investigating. / Il y a un problème avec le système de fichiers sur la grappe Fir, qui ne réagit pas correctement. Nous investiguons actuellement ce qui se passe.<br></p> None 2026-05-21 20:01