20141007 / /scratch on E5-2670

Explanations will be long, take a coffee, a tea, or quit ;o)

Last week, we experienced a threshold effect (again) while adding new nodes to E5 cluster. It lead us to reboot a big part of debian nodes. Doing this open a Murphy's box (you know the law ? It's the same, with a ribbon on it).

• an electrical problem appear where we add the new nodes, on the same distribution unit of a block of x5570, including their scratch servers.

1. > unexpected reboots while trying to figure out what's going on.

• new E5 nodes cannot connect to “old” infiniband switchs (firmware problem)

1. > nodes need a new infiniband card firmware, and reboot
2. > OS kernel need an upgrade, which need to be propagated
3. > some infiniband cables wont work anymore (wires need an upgrade, maybe)

• new E5 infiniband switch cannot connect to “old” infiniband switchs (firmware problem)

1. > old switchs need a new firmware, and reboot
2. > a special machine, with a very early OS, with experimental libs is needed to upgrade switchs

• old E5 nodes bios is incompatible with new infiniband card firmware…

1. > all E5 nodes need to be bios upgraded, and reboot

• with all theses, scratch has become incoherent, checkfs needed…

And now ? Upgrades are (mostly) OK, scratch has been checked and is OK. Some E5 nodes are not OK, and have been pushed away from queuing system.