Fil des news

An infiniband switch died this morning (mostly in E5-GPU), cutting off scratches and MPI connections. It has been replaced.

REMINDER:

• starting yesterday:
  • E5 is in DRAIN mode, to be modified:
  • E5 wall time will be reduced to 4 hours walltime
  • new partition E5-short, with 30 minutes walltime
  • new partition E5-long, with 8 days walltime
• starting today (this morning): CANCELLED
  • Lake-flix and Cascade-flix will be in DRAIN mode, to be renamed Lake-premium and Cascade-premium

These modifications might take the entire week (due to draining jobs).

Lake-flix and Cascade-flix will be renamed Lake-premium and Cascade-premium later (probably during mandatory October 2024 shutdown)

Documentation has been updated and reflect futur state.

• starting today:
  • E5 is in DRAIN mode, to be modified
  • E5 wall time will be reduced to 4 hours
  • new partition E5-short, with 30 minutes walltime
  • new partition E5-long, with 8 days walltime
• starting tomorrow:
  • Lake-flix and Cascade-flix will be in DRAIN mode, to be renamed Lake-premium and Cascade-premium

These modifications might take the entire week (due to draining jobs).

• As a test, starting tonight, following partitions will no longer be in “hold mode” (PartitionDown) during the day (6h→22h) :
  • E5, Lake, Lake-bigmem, Epyc, Cascade.

• starting next week: (probably Tuesday 11th of June)
  • E5 wall time will be reduced to 4 hours
  • new partition E5-short, with 30 minutes walltime
  • Lake-flix and Cascade-flix will be renamed Lake-premium and Cascade-premium

reminder: Access to Lake-bigmem is subject to authorization, use our forms to ask.

slurmctl is back ONLINE

TL;DR:

There is a known bug in our version of slurm, where in a large array job, if two subtasks fail at the same time, one will be left stuck in FAIL/REQUEUE mode indefinitely. This can segfault the slurm controller at restart (like when rotating log, for example).

And things go sideways in the accounting database very fast (it took only 3 seconds to hang the database and segfault the controller).

Workaround:

• on our part, a daily script to cleanup the jobs states
• on YOUR part : do not submit job arrays on multiples partitions, stick to one only.

• BOFH excuses, Chapter 6 -Solar Flares-

So, following an “electronic shower” of some sort, a network device decided to stop last night, cutting access to ssh gateways. It has been handled, with hammer force.

• slurm controller is DOWN, we are working on it

database badly corrupted by a job array, manual cleaning in progress.