Jobs that do a lot of I/O operations on a shared cluster filesystem like /scratch
can severely slow down the whole system. If your job does not use multiple nodes and is not reading and writing very large files, it might be a good idea to move input and output files to the /tmp
folder on the compute node itself.
/tmp
is a RAM based filesystem, meaning that anything you store there is actually stored in memory. So space is quite limited. Currently all jobs on a node can use at most 20% of the total system memory for space in /tmp
. If you need more space, you should consider using /dev/shm
, where you can use up to 50% of the total system memory per job.
–mem
option