Différences

Ci-dessous, les différences entre deux révisions de la page.

Lien vers cette vue comparative

Les deux révisions précédentesRévision précédente
Prochaine révision
Révision précédente
en:documentation:tutorials:scratch [2020/02/10 16:09] – [cluster X5] ltaulellen:documentation:tutorials:scratch [2023/12/12 12:57] (Version actuelle) – supprimée ltaulell
Ligne 1: Ligne 1:
-====== What is /scratch volume? ====== 
- 
- 
-The ''/scratch'' is a **//temporary//** workspace that you can use to copy your input data and write your calculation outputs. 
-The pros of working in the ''/scratch'' are the following: 
-  * offers a bigger volume than the volume available in your ''/home'' space (many ''/scratch'' workspaces are available), 
-   * increase processing speed of reads/writes, 
-   * increase the  simultaneous I/O. 
-\\ 
- 
-Scratches are **shared and common storages**, their performances depends on your proper use. Hence, they must not contain: 
- 
-  * documentation (lots of useless small files) ''=> /home'', 
-  * symbolic links (tiny files), 
-  * programs or exe (useless I/O) ''=> /home''. 
- 
-<note tip>**Exclusively input, temporary or output data**</note> 
- 
- 
-Two  types of ''/scratch'' are available: 
- 
-<note warning> 
-**Incomplete informations, because of new cluster (Lake)** 
-</note> 
-===== Global scratch of a cluster ===== 
- 
-As we have various clusters of different generations of processors (Lake, the newest, E5 and X5, the former), we have several global scratches. 
- 
-==== Cluster Lake  ==== 
- 
-The ''/scratch/Lake/'', common to ** CLG* and SLG* ** queues is available for all [[documentation:clusters:hardware&#cluster_lake|cluster Lake nodes]]. 
- 
-It contains ''/scratch/Lake/lake-gfs-scratch'', as a empty check file. 
- 
-It is accessible from **[[documentation:clusters:services|cluster Lake front machines]]** 
- 
-<note tip> 
-**Chimie et Bio scratches are already available** 
-</note> 
- 
-<code> 
-/scratch/ 
-     ├── Lake/         (yet to come, common to Lake cluster) 
-     ├── disk/         (local to some servers) 
-     ├── Chimie/       (reserved to chimie usage) 
-     ├── Bio/          (reserved to biologie usage) 
-     ... 
-     └── Project_name/ (reserved to some servers,  
-                        with dedicated hardware) 
- 
-</code> 
- 
-==== cluster E5 ==== 
- 
-The ''/scratch/E5/'', common to ** E5-* ** queues is available for all [[documentation:clusters:hardware&#cluster_e5|cluster E5 nodes]]. 
- 
-It contains ''/scratch/E5/e5-2670-gfs-scratch'', as a empty check file. 
- 
-It is accessible from **[[documentation:clusters:services|cluster E5 front machines]]** 
- 
- 
-<code> 
-/scratch/ 
-     ├── E5/           (E5 scratch, common to E5 cluster) 
-     ├── nvme/         (local to some servers) 
-     ├── ssd/          (local to some servers) 
-     ... 
-     └── Project_name/ (local to some servers,  
-                        with dedicated hardware) 
- 
-</code> 
-==== cluster X5 ==== 
- 
-The ''/scratch/X5/'', common to ** X5-* ** queues is available for all [[documentation:clusters:hardware&#cluster_x55_x56|cluster x55/x56 nodes]]. 
- 
-It contains ''/scratch/X5/x5570-gfs-scratch'', as a empty check file. 
- 
-It is accessible from **[[documentation:clusters:services|cluster x55/x56 front machines]]** 
- 
-<code> 
-/scratch/ 
-     └── X5/           (X5 scratch, common to X5 cluster) 
-</code> 
- 
- 
-==== scratch local to some nodes ==== 
- 
-Some local ''/scratch'' are available only on some nodes. [[documentation:clusters:hardware|See detailled description]] or ask staff. 
- 
- 
-==== Network topology of PSMN ==== 
- 
- 
-\\ 
-The diagram below (<wrap warning>NOT up-to-date</wrap>) represents the integration of these two ''/ scratch'' spaces in the PSMN infrastructure. 
-\\ 
- 
-{{ :documentation:tutorials:psmn_network_synoptic.png?direct&660 |}} 
- 
-<hidden> 
-{{ :documentation:tutorials:psmn-network-synoptic.png?direct&640 |}} 
-</hidden> 
-\\ 
- 
- 
-==== • Locals /scratch: ==== 
- 
-These locals ''/scratch'' are only avaiable in some nodes of the cluster [[documentation:clusters:hardware&#clusters_mutualises|see more complete description ]]. 
- 
- 
-====== How to use the scratch ====== 
-===== Using the /scratch: manual copy ===== 
- 
-For the global ''/scratch'', before submitting your job, copy your inputs into a folder on the ''/scratch'' (//ie// ''/scratch/login/whatever/''). 
-The simplest is to reproduce the same hierarchy of folders and files by changing '' /home'' by ''/scratch'' (copy with '' cp -r ''). 
- 
-Do not forget to modify your submitting files consequently.  
- 
-At the end of the run, retrieve your results manually, in the ''/ scratch'', from any of the corresponding front nodes. 
- 
-===== Using the /scratch : automatic copy ===== 
- 
-For local ''/scratch'' you don't know which node will be assigned to your job, it is also valid for the global scratch, but less annoying). Thus, You cannot copy your data to a particular node in advance. You must integrate the copy ''/scratch'' <-> ''/home'' in the submitted script. 
- 
- 
- 
-**Examples are available  [[documentation:examples:submit_scripts&#job_parallele_variantes|here]].** 
- 
- 
-We don't have a sample script that cleans up automatically. Similarly, if you don't feel comfortable with these scripts, do not hesitate to come and talk about during a Tuesday coffee session. 
- 
- 
-===== Technological transfer ===== 
- 
-<WRAP info> The global /scratch (glusterfs) were deployed thanks to  the upstream work, recipes and integrations made on the CBP experimental platform.</WRAP> 
  
en/documentation/tutorials/scratch.1581350980.txt.gz · Dernière modification : 2020/08/25 15:58 (modification externe)