BM@N Statistics
Python script to calculate statistics for BM@N data processing:
- file size (--size): size of the data files in a given directory, total and per event;
- processing time (--time): duration of batch jobs from their log files, total and per event.
The results are drawn as histograms (the dashed line marks the mean value) and saved to a PNG file; the summary values are printed to the terminal. Event counts are taken from the Unified Condition Database (UniConDa).
Requirements and installation
- Python 3.6 or newer;
- Python packages matplotlib, numpy, psycopg2 and uproot;
- network access to the UniConDa server (bmn-unidb.jinr.ru).
The packages can be installed with pip install -r requirements.txt
Running from a virtual environment:
python3 -m venv ~/.venv
source ~/.venv/bin/activate
pip install -r requirements.txt
cd [BMNROOT_PATH]/services/statistics/
python3 stats.py --size --dir /eos/nica/bmn/exp/raw/run8/beam/BMN_2022/ --config config-size-exp-raw.json --recursive
Usage
python3 stats.py [-h] --dir DIR [--size] [--time] [--recursive] --config CONFIG
[--output [OUTPUT]] [--batch]
options:
-h, --help Show this help message and exit
--dir, -d DIR Directory to explore
--size, -s Compute file size statistics
--time, -t Compute time statistics from log files
--recursive, -r Search files in subdirectories too
--config, -c CONFIG Path to JSON config file
--output, -o [OUTPUT] Path to output PNG file, default is ./output.png;
'-o' without a value shows the plot in a window instead
--batch, -b Batch mode: do not open graphical windows, only save the plot
to the output file (use it on the cluster nodes without a display)
At least one of --size and --time must be given. If both are given, the same directory and config file are used for both statistics, and all four histograms are drawn in one picture.
JSON config file
Ready-made configs:
| Config | Purpose |
| config-size-exp-raw.json | size of experimental raw files (.data), with file and event size limits |
| config-size-exp.json | size of experimental files (.data, .root), no limits |
| config-size-sim.json | size of simulation files (.r12, .root) |
| config-time.json | processing time from batch job logs (.out) |
Required parameters:
- extensions - data or log file extensions to process (use ["*"] to include all files)
- db_user, db_pass, db_name, db_host - UniConDa credentials (db_reader / uni_db / bmn-unidb.jinr.ru)
Optional parameters for both statistics:
- dpi - dpi of the output picture, default is 300
- folders_ignore - list of strings; files whose path contains any of them are skipped
Optional parameters for --size:
- source - exp (default) for experimental files or sim for simulation files, see below
- file_size_limit - min:max size of files to be processed, for example 100B:20GB
event_size_limit - min:max file size per event, for example 300KB:900KB
Units are B, KB, MB, GB, TB (case-insensitive, 1 KB = 1024 B); an empty string means no limit. Files outside the limits are skipped.
Optional parameters for --time:
- success - how many times the line Macro finished successfully must appear in a log to consider the job successful, default is 1 (e.g. 2 for a job running two macros)
- failure - a string marking a failed job even if the success line is present, for example File::Recover; empty by default
Histogram binning, units, regular expressions for log parsing and the names of the output lists are set in file_size/config.py and log_time/config.py.
How the statistics are computed
File size - the size of every file with a matching extension is taken from the file system. The size per event uses the event count from UniConDa:
- exp - the raw_file table, found by the file name (without directory). Therefore the size per event is computed only for raw files registered in the database; for other files (e.g. digi or DST) only the file size statistics are shown.
- sim - the simulation_file table, found by the full file path. The path must be exactly as stored in the database, so pass --dir as an absolute path, e.g. /eos/nica/bmn/sim/gen/....
For .root files with a known event count, the number of entries of the bmndata tree is compared with it, and the file is skipped if they differ. A file without an event count in the database is still counted in the file size histogram, but not in the size per event histogram.
Processing time - every log file is parsed for the following lines:
- Start date: <date> and End date: <date> - the output of date written by the batch scripts of NICA-Scheduler; the processing time is the difference between them;
- Macro finished successfully - the success marker printed at the end of the BmnRoot macros;
- Processing <macro>.C("<input file>", ...)... - printed by ROOT when the macro starts, gives the name of the processed input file;
- the run number is taken from the first line with .root matching _run<number> (e.g. mpd_run_Top_7830_ev0_p0.root), and the event count is taken from the run_ table for this run (the latest period if the run number exists in several periods). If the run is not found, the number from a line ... number of events ... <N> in the log is used.
Logs without the success marker or with the failure string are excluded from the histograms.
Output
- the PNG file given by --output (default ./output.png); the script then tries to open it with xdg-open (not in the --batch mode);
- unsuccessful_list.txt in the current directory: for --size - files that were skipped or have no event count in the database; for --time - input files of the failed jobs (can be used to resubmit them);
- successful_list.txt in the current directory (--time only) - input files of the successful jobs.
The lists are overwritten on every run. See output-size-sample.png and output-time-sample.png for sample results.
Examples
Time:
cd [BMNROOT_PATH]/services/statistics/
python3 stats.py --time --dir /eos/nica/bmn/users/$USER/logs/ --config config-time.json
Size - experimental raw files:
cd [BMNROOT_PATH]/services/statistics/
python3 stats.py --size --dir /eos/nica/bmn/exp/raw/run8/beam/BMN_2022/ --config config-size-exp-raw.json --recursive
Size - simulation files:
cd [BMNROOT_PATH]/services/statistics/
python3 stats.py --size --dir /eos/nica/bmn/sim/gen/DCMSMM/DCMSMM_XeCsI_3.9AGeV_mb_10k/ --config config-size-sim.json --recursive
The examples subdirectory contains configs and resulting plots for raw, digi and DST files of RUN7.
1.17.0