Size and content of ROOT files
Tools to inspect BmnRoot output files (simulation, digi, DST): what takes the disk space, and whether two files hold the same data.
| File | Purpose | Requires |
| calculate_size.C | Disk space taken by the bmndata tree and by each of its branches | BmnRoot environment |
| compare_files.C | Compare the data content of two ROOT files ignoring metadata; content hash of a file | BmnRoot environment (class dictionaries) |
| compare_files.py | The same comparison and hash on the raw data, without ROOT | Python 3 and uproot (pip install uproot) |
The ROOT macros need the BmnRoot environment to be loaded first:
source ~/bmnroot/build/config.sh
The file paths passed to the macros are expanded, so ~/... and $VMCWORKDIR/... can be used in them.
calculate_size.C: sizes of the tree branches
Prints the size on disk of the bmndata tree and of every top-level branch (including all its sub-branches), e.g. to estimate how much a new branch or a change of a data format costs in the output files.
root -l -b -q '$VMCWORKDIR/macro/profiling/size/calculate_size.C("$VMCWORKDIR/macro/run/bmndst.root")'
Example output for a DST file (shortened):
The TTree "bmndata" takes 4.17 MB on disk
Its branch "DstEventHeader." takes 2.71 KB on disk
Its branch "BmnTrigInfo." takes 16.51 KB on disk
...
Its branch "BmnSiliconHit" takes 293.07 KB on disk
Its branch "BmnSiliconUpperCluster" takes 1.08 MB on disk
...
Its branch "StsHit" takes 943.04 KB on disk
...
Its branch "BmnGlobalTrack" takes 123.75 KB on disk
...
How the sizes are calculated:
- Branch: the compressed size of the baskets (the event data) of the branch and all its sub-branches, plus the compressed size of its metadata: the TBranch object with its sub-branches and leaves, streamed into a TMemFile with the compression settings of the input file. For an uncompressed branch, the uncompressed size of its baskets is used.
- Tree: the size of the tree header stored in the file (it holds the metadata of all branches), plus the baskets of all branches. The branch sizes therefore add up approximately to the tree size.
Notes:
- Only the tree header is read, not the event data, so even large files are fast: a 1 GB digi file takes about a second.
- Only the bmndata tree is counted. Other objects of the file (FairGeoParSet with the geometry, parameter containers, run headers, ...) are not, so the file is larger than the tree: the DST file above is 4.39 MB, of which the tree takes 4.17 MB.
- The units are binary: 1 KB = 1024 B, 1 MB = 1024 KB.
- If the file does not exist, or has no bmndata tree, or the tree is empty, the macro prints an error and does nothing.
compare_files.C: compare two files with ROOT
A byte comparison (cmp, md5sum) reports two ROOT files as different even when they hold the same data: the file header stores the file name and modification time, and every TKey and TBasket stores its creation time. The macro compares the data instead. It reads back every key of both files (the highest cycle of each key, recursing into subdirectories) and compares the objects by serializing them into memory buffers byte by byte. Trees are compared entry by entry, branch by branch. TProcessID keys hold only a time-based GUID and are skipped.
A typical use is to check that a refactoring or an optimization did not change the results of the simulation or reconstruction.
# full comparison of the data content
root -l -b -q '$VMCWORKDIR/macro/profiling/size/compare_files.C("file1.root", "file2.root")'
# coarse comparison: entry counts and uncompressed sizes only
root -l -b -q '$VMCWORKDIR/macro/profiling/size/compare_files.C("file1.root", "file2.root", kTRUE)'
# content hash of a single file (full or coarse)
root -l -b -q -e '.L $VMCWORKDIR/macro/profiling/size/compare_files.C' -e 'hash_file("file.root")'
root -l -b -q -e '.L $VMCWORKDIR/macro/profiling/size/compare_files.C' -e 'hash_file("file.root", kTRUE)'
Every compared object gets a line starting with OK, DIFF or WARN. The last line is the verdict: The files are IDENTICAL (...) or The files are DIFFERENT. For example:
DIFF /DstRunHeader (DstRunHeader): different size (188 vs 182 bytes uncompressed)
OK /TimeBasedBranchList (TList): same size of 21 bytes
DIFF /bmndata: different number of entries (5 vs 2000)
The files are DIFFERENT
Notes:
- The full comparison needs the dictionaries of the stored classes, so run it in the BmnRoot environment. A key of an unknown class is skipped with a WARN line.
- In the full mode, the comparison of a tree stops at the first difference and reports its branch and entry.
- The coarse mode (kTRUE) needs no class dictionaries and is fast, but equal sizes do not guarantee equal content: for example, a changed value of a fixed-size type is not detected.
- hash_file() reduces the same data to an MD5 hash, so files that compare_files() reports as IDENTICAL get equal hashes. A hash is only as reproducible as the streamers of the stored classes: a class whose serialized form contains addresses or uninitialized memory gives a different hash every time. Never compare a coarse hash with a full one.
- Reading FairGeoParSet builds the geometry, so TGeoManager info messages in the output are expected.
compare_files.py: compare two files without ROOT
The same comparison done with uproot on the raw data: for every tree, it compares the decompressed basket data of every branch; for any other object, it compares the raw streamed bytes of its key. The data are never deserialized into objects, so neither ROOT nor the BmnRoot libraries are needed. This makes the script handy on machines without BmnRoot and in CI.
python3 compare_files.py file1.root file2.root # full comparison
python3 compare_files.py file1.root file2.root --size-only # coarse comparison
python3 compare_files.py --hash file.root # content hash of a file
python3 compare_files.py --hash file1.root file2.root # hashes of two files, compared
The exit code is 0 if the files are identical (or the hashes are equal, or a single hash was computed), 1 if they differ, and 2 on error, so the script can be used in scripts and CI jobs.
Differences from compare_files.C:
- All differing branches of a tree are reported, each with the first differing entry if it can be determined.
- --size-only compares only the entry counts and uncompressed data sizes without decompressing the data, with the same limitation as the coarse mode of the macro.
- The hash is computed from the raw payload, not from re-serialized objects, so it is not comparable with the hash of compare_files.C. The two are built with different scheme tags, so they never match by accident.
1.17.0