10 Job submission in qsub and related environments

It is your own responsibility to correctly submit jobs to your HPC infrastructure. Because of the wide variety of configurations available, we cannot write a script that will be able to work with all or even a high fraction of such systems. If you have a way of doing this for every line in a text file, then use that. qsub job arrays (qsub -t) are the standard way of doing this, but they did not work correctly with our version of torque.

However, we have provided a script that works on our institutional HPC machine using qsub. It may require minor or major modification to work with other systems - use it cautiously! It can be found in the scripts directory.

This creates a

For other systems, you might want to consider the unix command ‘split’. This can split the command list into e.g. parallel processing units. If you can run commands in batches of 8, then:

generates files with names like example_commandfile1_split<aa-...> each containing 8 lines from the command file.