| Les deux révisions précédentesRévision précédenteProchaine révision | Révision précédenteDernière révisionLes deux révisions suivantes |
| en:documentation:tutorials:scratch [2018/05/25 12:58] – cpetit | en:documentation:tutorials:scratch [2021/11/18 15:20] – [Network topology of PSMN] ltaulell |
|---|
| The ''/scratch'' is a **//temporary//** workspace that you can use to copy your input data and write your calculation outputs. | The ''/scratch'' is a **//temporary//** workspace that you can use to copy your input data and write your calculation outputs. |
| The pros of working in the ''/scratch'' are the following: | The pros of working in the ''/scratch'' are the following: |
| * offers a bigger volume than the volume available in your ''/home'' space (currently 2 ''/scratch'' space of ~ 35To are available) | * offers a bigger volume than the volume available in your ''/home'' space (many ''/scratch'' workspaces are available), |
| * increase processing speed of reads/writes | * increase processing speed of reads/writes, |
| * increase the simultaneous I/O on the volume. | * increase the simultaneous I/O. |
| \\ | \\ |
| | |
| | Scratches are **shared and common storages**, their performances depends on your proper use. Hence, they must not contain: |
| | |
| | * documentation (lots of useless small files) ''=> /home'', |
| | * symbolic links (tiny files), |
| | * programs or exe (useless I/O) ''=> /home''. |
| | |
| | <note tip>**Exclusively input, temporary or output data**</note> |
| | |
| | |
| Two types of ''/scratch'' are available: | Two types of ''/scratch'' are available: |
| |
| | ===== Global scratch of a cluster ===== |
| |
| ==== • The global /scratch ==== | As we have various clusters of different generations of processors (Lake, the newest, E5 and X5, the former), we have several global scratches. |
| |
| Subdivided into 2 ''/scratch'' space : | ==== Cluster Lake ==== |
| |
| * <wrap hi> E5 scratch </wrap>, common to every ** E5-* ** queues | The ''/scratch/Lake/'', common to ** CLG* and SLG* ** queues is available for all [[documentation:clusters:hardware&#cluster_lake|cluster Lake nodes]]. |
| |
| This ''/scratch'' is accessible to all [[documentation:clusters:hardware&#cluster_equip_meso_ou_e5|Équip@méso nodes]] and to the front nodes **[[documentation:clusters:services|e5-2670comp1 & 2]]**, **[[documentation:clusters:services|e5-2667v4comp1 & 2]]**, | It contains ''/scratch/Lake/lake-gfs-scratch'', as a empty check file. |
| \\ as well as the visualization server **[[documentation:tutorials:visualisation|r730visu]]**. | |
| |
| It contains the empty verification file: ''/scratch/e5-2670-gfs-scratch''. | It is accessible from **[[documentation:clusters:services|cluster Lake front machines]]** |
| |
| | <note tip> |
| | **-At last-, ''/scratch/Lake'' is available** |
| | </note> |
| |
| * <wrap hi> X5 scratch </wrap>, common to every ** x55/x56* ** queues | <code> |
| | /scratch/ |
| | ├── Lake/ (common to Lake cluster) |
| | ├── disk/ (local to some servers) |
| | ├── Chimie/ (reserved to chimie usage) |
| | ├── Bio/ (reserved to biologie usage) |
| | ... |
| | └── Project_name/ (reserved to some servers, |
| | with dedicated hardware) |
| |
| This ''/scratch'' is accessible to all [[documentation:clusters:hardware&#cluster_x55_x56|x55/x56 nodes]] and to the front nodes **[[documentation:clusters:services|x5570comp1 & 2]]**. | </code> |
| |
| It contains the empty verification file: ''/scratch/x5570-gfs-scratch''. | ==== Cluster E5 ==== |
| | |
| | The ''/scratch/E5N/'', common to ** E5-* ** queues is available for all [[documentation:clusters:hardware&#cluster_e5|cluster E5 nodes]]. |
| | |
| | It contains ''/scratch/E5N/E5N-gfs-scratch'', as a empty check file. |
| | |
| | It is accessible from **[[documentation:clusters:services|cluster E5 front machines]]** |
| | |
| | |
| | <code> |
| | /scratch/ |
| | ├── E5N/ (E5 scratch, common to E5 cluster) |
| | ├── nvme/ (local to some servers) |
| | ├── ssd/ (local to some servers) |
| | ... |
| | └── Project_name/ (local to some servers, |
| | with dedicated hardware) |
| | |
| | </code> |
| | ==== Cluster X5 ==== |
| | |
| | The ''/scratch/X5/'', common to ** X5-* ** queues is available for all [[documentation:clusters:hardware&#cluster_x55_x56|cluster x55/x56 nodes]]. |
| | |
| | It contains ''/scratch/X5/x5570-gfs-scratch'', as a empty check file. |
| | |
| | It is accessible from **[[documentation:clusters:services|cluster x55/x56 front machines]]** |
| | |
| | <code> |
| | /scratch/ |
| | └── X5/ (X5 scratch, common to X5 cluster) |
| | </code> |
| | |
| | |
| | ==== Scratch local to some nodes ==== |
| | |
| | Some local ''/scratch'' are available only on some nodes. [[documentation:clusters:hardware|See detailled description]] or ask staff. |
| | |
| | |
| | ==== Network topology of PSMN ==== |
| |
| |
| \\ | \\ |
| The diagram below represents the integration of these two ''/ scratch'' spaces in the PSMN infrastructure. | The diagram below represents the integration of these ''/scratch'' spaces in the PSMN infrastructure. |
| \\ | \\ |
| |
| {{ :documentation:tutorials:psmn_network_synoptic.png?direct&660 |}} | {{ :documentation:tutorials:psmn_network_synoptic_a3_2021.png?direct&660 |}} |
| |
| <hidden> | |
| {{ :documentation:tutorials:psmn-network-synoptic.png?direct&640 |}} | |
| </hidden> | |
| \\ | \\ |
| |
| ===== Using the /scratch: manual copy ===== | ===== Using the /scratch: manual copy ===== |
| |
| For the global ''/scratch'', before submitting your job, copy your inputs into a folder on the ''/scratch'' (//ie// ''/scratch/login/whatever/''). | For global ''/scratch'', before submitting your job, copy your inputs into a folder on the ''/scratch/'' (//ie// ''/scratch/<cluster>/<login>/whatever/''). |
| The simplest is to reproduce the same hierarchy of folders and files by changing '' /home'' by ''/scratch'' (copy with '' cp -r ''). | |
| | The simplest is to reproduce the same hierarchy of folders and files by changing '' /home'' by ''/scratch/<cluster>''. |
| |
| Do not forget to modify your submitting files consequently. | Do not forget to modify your submitting files consequently. |
| ===== Using the /scratch : automatic copy ===== | ===== Using the /scratch : automatic copy ===== |
| |
| Pour les ''/scratch'' locaux, vous ne savez pas quel noeud vous sera attribué (c'est valable aussi pour le scratch global, mais moins génant). Vous ne pouvez donc pas copier vos données vers un noeud en particulier à l'avance. Il faut intégrer la copie ''/scratch''<->''/home'' dans le script soumis. | For local ''/scratch'' you don't know which node will be assigned to your job (it is also valid for the global scratch, but less annoying). Thus, You cannot copy your data to a particular node in advance. You must integrate the copy ''/scratch'' <-> ''/home'' in the submitted script. |
| | |
| |
| **Vous trouverez [[documentation:examples:submit_scripts&#job_parallele_variantes|des exemples sur cette page]].** | **Examples are available [[documentation:examples:submit_scripts&#job_parallele_variantes|here]].** |
| |
| |
| Nous n'avons pas d'exemple de script qui fasse le ménage de manière automatique. De même, si vous ne vous sentez pas à l'aise avec ces scripts, n'hésitez pas à venir en discuter autour d'un café, le lundi après-midi. | We don't have a sample script that cleans up automatically. Similarly, if you don't feel comfortable with these scripts, do not hesitate to come and talk about during a Tuesday coffee session. |
| |
| |
