NICA-Scheduler
If you have time-consuming tasks, many simple tasks or a lot of files to process, you can use batch systems on distributed clusters, such as SLURM on the HybriLIT platform including Govorun supercomputer and JINR CICC complex or Sun Grid Engine (SGE) on the NICA ncx-cluster to essentially accelerate your work.
If you know how to work with SLURM (SLURM on HybriLIT), Sun Grid Engine (SGE User Guide) and Torque systems (doc index), you can use directly sbatch or qsub commands on the clusters to parallelize data processing. Simple examples of user jobs for SLURM, SGE and Torque can be found here. Otherwise, NICA-Scheduler has been developed to simplify running user tasks in parallel.
NICA-Scheduler is a module package developed for the ROOT- and FairRoot-based frameworks, such as the BmnRoot and MpdRoot software. It uses an existing batch system (SLURM, SGE and Torque are supported) to distribute user jobs on clusters (as well as supports multithreading on local machines), and simplifies parallel job execution without knowledge of the batch system. NICA-Scheduler can be used as a submodule inside your project or can be compiled as a standalone package.
Jobs for distributed execution are described and passed to NICA-Scheduler as XML files, e.g.:
$ nica-scheduler job_description.xml
Additional options can be used to run the scheduler:
-h/–help : produce help message
-f/–file arg : set XML file path with a job description
-d/–debug : debug mode with detailed output
–db/–database arg : predefined settings for the Unified Condition Database of the experiment (now arg = bmn supported only)
An example of the NICA-Scheduler job description:
<job name="reco_job">
<macro path="$VMCWORKDIR/macro/run/run_reco.C" start_event=”0” event_count=”1000”>
<file input="$VMCWORKDIR/macro/run/sim1.root" output="$VMCWORKDIR/macro/run/dst1.root"/>
<file input="$VMCWORKDIR/macro/run/sim2.root" output="$VMCWORKDIR/macro/run/dst2.root"/>
<file sim_input="energy=3,gen=dcmqgsm" output="$VMCWORKDIR/macro/run/dst_${counter}.root"/>
</macro>
<run mode=“global" count=“25" config=“~/build/config.sh"/>
</job>
The XML description of a job to run in a batch system starts with <job> tag and ends with closing </job> tag. The attribute 'name' defines the name of the job to identify it. The attribute 'batch' defines the batch system of the cluster. The possible values: ‘_sge_’ for Sun Grid Engine, ‘_slurm_’ for SLURM and ‘_torque_’ for the Torque batch system. Attention! The default value is ‘slurm’ (SLURM batch system) to run on a cluster, where SLURM is deployed. In addition, predefined settings of the Unified Condition Database for the experiment can be used via 'database' attribute (at present, only 'bmn' value is supported).
Tag <macro> sets information on a ROOT-macro to be executed:
- path – a path of the ROOT macro for distributed execution.
Important! Since a job can be started by the Scheduler on any machine of the cluster, the path must point to a shared space (e.g. _/eos/nica/users/$USER/_ volumes on the NICA cluster). This argument is required.
- name – a name of the macro to refer to it from another macro in the same job. The referring macro uses this name in the macro_input attribute of its <file> tag to take the output files of the named macro as its input files. This argument is optional.
- start_event – a number of the start event to process for all the input files. This argument is optional.
- event_count – a number of events to process for all the input files. This argument is optional.
- add_args – additional last arguments of the ROOT macro, if required.
Tag <file> is included inside the <macro> tag and contains information on input and output files to process by the macro:
- input – a path to one input file or a set of files in case of regular expressions (?,*,+) to be processed.
- file_input – a path to a text file containing a list of input files separated by new lines.
- json_input – a path to a JSON file containing a list of input files written as an array of strings, e.g. ["file1.root", "file2.root"].
- job_input – a name of one of the previous jobs, whose output files are input files for the current macro.
- macro_input – a name of one of the previous macros in the same job, whose output files are input files for the current macro.
- sim_input – a string to specify a list of input simulation files forming from the Unified Condition Database if it is used.
- exp_input – a string to specify a list of input experimental (raw) files forming from the Unified Condition Database if it is used.
- output – a path to result files.
Important! Since a user job can be started by the NICA-Scheduler on any machine of the cluster, the path must point to a shared space, e.g. EOS volume being available on all cluster nodes.
- start_event – a number of the start event specific for the file. This argument is optional.
- event_count – a number of events of the file to be processed. This argument is optional.
Important! If the start_event (or event_count) is set in <macro> and <file> tags then the value in the <file> tag is selected as preferable.
- parallel_mode – a processor count to parallelize event processing for the given input file. This argument is optional.
- merge – whether merge partial output files in the parallel_mode. Default value is false, possible values of the attribute: "false" (not merge result files), "true" (merge result files to one output file and delete the partial results), "preserve" (merge result files to one output file and not delete the partial results), "chain" (create the output file containing a TChain object including all partial files).
An example of sim_input string is "energy=3,gen=urqmd", where the following parameters mean as follows: collision energy of selected events is equal to 3 GeV and event generator is UrQMD. All possible parameters are described in database console utilities. As a list of input files to be processed can include a lot of files in one <file> tag, special variables can be used in the output argument to set a list of output files:
${counter} = counter is corresponding to a regular input file to be processed, the counter starts with 1 and increases by 1 for the next input file.
${input} = absolute path of the input file.
${file_name} = input file name without extension.
${file_name_with_ext} = input file name with extension.
${file_dir_name} = directory name of the input file.
${batch_temp_dir} = path to a batch temp directory.
${user} = username who runs the job.
${first_number} = first number in the input file path.
${last_number} = last number in the input file path.
${first_number_fn} = first number in the input file name (without path).
Also, you can use an additional possibility to exclude first ('~' symbol after colon) or last ('-' symbol after colon) characters for the above special variables, e.g. ${file_name_with_ext:~N} is used as an input file name with extension but without first N chars, or ${file_name:-N} - input file name without last N chars.
In case of mass data processing or production, input files are often pre-copied to an intermediate fast storage before processing on the cluster. The possibility is implemented via <put>, <get> and <clean> tags included into the <file> tag. The <put> tag is used to preliminary copy input files to a temporary storage, and the <get> tag – copy the result files back from the storage. In addition, the <clean> tag can be used to list auxiliary files (except temporary input file and result one deleted by default), which will be removed after the processing. The <put> and <get> tags contain the following attributes:
- command – copy command (possible with parameters) used to transfer files to (<put> tag) or from (<get> tag) an intermediate storage, e.g. "xrdcp -f -p -N".
- directory – a directory path at the intermediate storage to put input files and get result ones.
- extras – additional result file paths or masks separated by comma to copy from the intermediate storage to the directory of the output file after the processing (only for the <get> tag). A relative path is considered relative to the <get> directory. This argument is optional.
The <clean> tag uses only path attribute to list additional files to be deleted after processing, e.g. some log or dump files.
Tag <run> describes run parameters and allocated resources for the job. It can contain the following arguments:
- mode – an execution mode. ‘_global_’ value specifies distributed processing on a cluster with the batch system, ‘_local_’ – multi-threaded execution on a user multicore computer, ‘_container_’ – same as ‘_global_’, but jobs computed in container environment. The default value is ‘local’.
- count – a maximum number of processors allocated for the job. If the count is greater than a number of files to be processed, the count of the allocated processors is assigned to the number of the files. The default value is 1. You can assign the processor count to 0, then all cores of the user machine are employed in case of the local mode or all processors of the batch system are requested in case of the global mode.
- queuelimit – a maximum number of jobs submitted by the current user to the batch queue: the scheduler waits for a free slot in the queue before submitting the next job. It is used only in the global (batch) modes. The default value is 0 (unlimited). This argument is optional.
- arraylimit – a maximum number of tasks in one batch array job: if the task count of the job is greater than the limit, the job is submitted by portions as separate array jobs. It is used only in the global (batch) modes. The default value is 0 (unlimited). This argument is optional.
- config – comma separated list of config file paths, where the config represents a bash file setting environment variables (e.g., ROOT environment variables), which will be executed before running the macros. This argument is optional.
- queue – a queue name of the batch system to be used for running the job if it is not as default, e.g. "mpd@bfsrv.jinr-t1.ru" for LIT CICC cluster queue.
- qos – quality of service (QOS) for each job if required.
- hosts – selected host names separated by comma to process the job, e.g. "ncx201,ncx202" - for running the job only on ncx201.jinr.ru and ncx202.jinr.ru nodes of the NICA cluster. Add '~' (tilda) symbol at the first place for listing only host names, which will be excluded from the job processing.
- priority – a priority of the job (an integer in the range -1023 to 1024 with default value: 0). It is optional argument.
- memory1 – a number of operative memory being allocated per one processor slot, but not less than 1 MB, e.g. "4 GB". It is optional.
- logs – a log file path for multi-threaded (local) mode. This argument is optional.
- core_dump - whether dump core to 'core.*' files in case of task failures. The default value is false, possible values: "false", "true".
- container_os - base OS of the container. Works only with ‘_container_’ mode. ‘_alma9_’ value specifies the AlmaLinux 9 environment. ‘_centos7_’ – CentOS 7 environment. The default value is ‘_alma9_’.
If you want to run a non-ROOT (arbitrary) command by the NICA-Scheduler, use <command> tag instead of the <macro> one with argument line that means command line for distributed execution by a scheduling system. An example of a job with <command> tag:
<job>
<command line="show_simulation_files energy=5-9"/>
<run mode="global" config="~/build/config.sh"/>
</job>
The <command> tags can also be combined with <macro> tags in one job: all the tags are executed in the order, in which they are written in the job description. Note! If the job contains macros processed in parallel, each command is executed in every thread (in case of the local mode) or in every task of the batch array job (in case of the global mode). Another example of a job for local multi-threaded execution:
<job>
<macro path=“$VMCWORKDIR/macro/run/run_reco.C">
<file input=“$VMCWORKDIR/macro/run/sim1.root" output="$VMCWORKDIR/macro/run/dst1.root“ start_event=”0” event_count=”0”/>
<file input="$VMCWORKDIR/macro/run/sim2.root" output="$VMCWORKDIR/macro/run/dst2.root“ start_event=”0” event_count=”1000” parallel_mode=“5” merge=“true”/>
</macro>
<run mode="local" count=“6" config=“~/build/config.sh" logs="processing.log"/>
</job>
An XML job description for the NICA-Scheduler can contain more than one job, and a list of user jobs for distributed execution. In this case <job> tags are included in a general <jobs> tag. Dependencies can be set between different jobs, so the job depending on another one will not start its execution until the latter ends. To set dependency, use 'dependency' attribute assigned to the name of another job in the <job> tag. In addition, a NICA-Scheduler job can contain more than one macro to execute, just use several <macro> tags in the job.
The examples directory of the NICA-Scheduler contains various XML examples. If you have any questions on the NICA-Scheduler, please, feel free to email to gerts.nosp@m.en@j.nosp@m.inr.r.nosp@m.u.
Repository layout
| File / directory | Description |
| src | Source code of the nica-scheduler program and the helper scripts and macro for merging the partial results |
| settings.h | Settings of the Unified Condition Database used for the sim_input and exp_input attributes |
| examples | Examples of the XML job descriptions and plain batch jobs |
| proof | Scripts for running PROOF on Demand (PoD) clusters |
| cmake | Build of the scheduler inside the ROOT-based frameworks or as a standalone package |
Inside BmnRoot the scheduler is built by services/CMakeLists.txt: the nica-scheduler executable is put to the bin directory of the build together with the merge.C and merge.sh helpers.
1.17.0