20260521 / Power down in datacenter

Yesterday end-of-afternoon/evening, the whole HPC power system went down, with the cooling system.

We are trying to restart the HPC system from now. Be patient…

EDIT 11:30: I still have glitches on fileservers, none for homes, so @comp & @visu are restarting.

EDIT 13:30: Lake cluster is ongoing a full restart, Cascade cluster will follow soon

EDIT 14:30: Lake cluster is UP, with scratch ; Cascade cluster is still restarting, with delayed scratch mount ; slurm is restarting, with all jobs either requeued or failed (mostly failed, hope is a dead weight).

EDIT 17:00 youpi! (no.) There will be a total power down of the HPC power lines in the datacenter to (hopefully) track down the root causes of yesterday events… Expect a whole day power down.

(╯è.é)╯︵ ┻━┻