====== 20260521 / Power down in datacenter ======

Yesterday end-of-afternoon/evening, the whole HPC power system went down, with the cooling system.

We are trying to restart the HPC system from now. Be patient...

EDIT 11:30: I still have glitches on fileservers, none for homes, so @comp & @visu are restarting.

EDIT 13:30: Lake cluster is ongoing a full restart, Cascade cluster will follow soon

EDIT 14:30: Lake cluster is UP, with scratch ; Cascade cluster is still restarting, with delayed scratch mount ; slurm is restarting, with all jobs either requeued or failed (mostly failed, hope is a dead weight).

**EDIT 17:00** youpi! (no.) There will be a **total power down** of the HPC power lines in the datacenter to (hopefully) track down the root causes of yesterday events... **Expect a whole day power down-> Mardi 26-05-2026, starting 9h00**.


(╯è.é)╯︵ ┻━┻



{{tag> power incident }}