Fil des news

Reboot of the ssh gateways due to security reasons…

There's was something “broken” with /scratch/Cral on Cascade cluster.

I still don't know precisely why it was not working on some nodes and working on others (and I do not like not knowing). But it is now working on all nodes (proper mounting is on it's way, 220~ish nodes todo).

There will be a total power down today (Tuesday 26th of May 2026) of the HPC power lines in the datacenter to (hopefully) track down the root causes of last week events… Expect a whole day power down, starting 9h00.

EDIT 11h45: testing done.

And in the middle of this, there's a (big) security upgrade…

GW will restart, all (remaining) login nodes will restart also.

EDIT 16h45: restarted. With a lot of nodes still “broken” (~1/5) for reasons.

important: There's is something broken with “/scratch/Cral” on Cascade cluster. It is on my todolist.

Yesterday end-of-afternoon/evening, the whole HPC power system went down, with the cooling system.

We are trying to restart the HPC system from now. Be patient…

EDIT 11:30: I still have glitches on fileservers, none for homes, so @comp & @visu are restarting.

EDIT 13:30: Lake cluster is ongoing a full restart, Cascade cluster will follow soon

EDIT 14:30: Lake cluster is UP, with scratch ; Cascade cluster is still restarting, with delayed scratch mount ; slurm is restarting, with all jobs either requeued or failed (mostly failed, hope is a dead weight).

EDIT 17:00 youpi! (no.) There will be a total power down of the HPC power lines in the datacenter to (hopefully) track down the root causes of yesterday events… Expect a whole day power down→ Mardi 26-05-2026, starting 9h00.

(╯è.é)╯︵ ┻━┻

There is a rebuild in progress on data7 (Bio: lbmc, ciri, igfl, rdp & al.) and it started with a temporary suspension of all I/O (~25mn) of the main pool of disks (don't ask, I don't know why).