Classes
class	CustomFormatter

class	FileAccessException

Functions
	get_process_output (process_string)

	get_process_retcode (process_string)

	get_process_full (process_string)

	check_file_exists_ssh (file_path, ssh_data)
	GET FILE INFORMATION ### check whether file exists using SSH protocol.

	check_file_exists_xrd (file_path, xrootd_url)

	check_file_exists_eos (file_path, xrootd_url)

	get_file_sshsize (file_path, ssh_data)

	get_file_xrdsize (file_path, xrootd_url)

	get_file_eossize (file_path, xrootd_url)

	get_file_MD5c (file_path)

	get_file_MD5a (file_path)

	get_file_adler32c (file_path)

	get_file_adler32a (file_path)

	get_file_xrdsum (file_path, xrootd_url)

	get_file_info (file_path, storage_url, protocol_type, ssh_con, start_tmp_dir, match_checksum=0)

	ls_dir_xrd (file_path, xrootd_url)
	GET FILE LIST ### list XRootD directory.

	ls_dir_eos (file_path, xrootd_url)

	get_files_xrd (dir, xrootd_url)

	get_files_eos (dir, xrootd_url)

	cp_file_ssh (source_file_path, destination_file_path, source_ssh_prefix, destination_ssh_prefix)
	COPY FILE ### copy file via SCP (in case of ssh-key access).

	cp_file_ssh_protocol (source_file_path, destination_file_path, connection_ssh_prefix, transfer_protocol, copy_prefix, is_source_copy_prefix=False)

	cp_file_xrd (source_file_path, destination_file_path, source_storage_url, destination_storage_url)

	cp_file_eos (source_file_path, destination_file_path, source_storage_url, destination_storage_url)

	rm_file_xrd (file_path, xrootd_url)
	REMOVE FILE ### remove remote file via XRootD.

Variables
int	CHECKSUM_BLOCKSIZE = 102410241024

list	storage_dictionary = ['ddc','ncx', 'cicc']

	format

	datefmt

	handlers

Function Documentation

◆ check_file_exists_eos()

function_set.check_file_exists_eos	(	file_path,
		xrootd_url
	)

Definition at line 102 of file function_set.py.

References get_process_retcode().

◆ check_file_exists_ssh()

function_set.check_file_exists_ssh	(	file_path,
		ssh_data
	)

GET FILE INFORMATION ### check whether file exists using SSH protocol.

Definition at line 81 of file function_set.py.

References get_process_full().

◆ check_file_exists_xrd()

function_set.check_file_exists_xrd	(	file_path,
		xrootd_url
	)

Definition at line 94 of file function_set.py.

References get_process_full().

Referenced by function_set.FileAccessException.__str__(), and get_file_info().

◆ cp_file_eos()

function_set.cp_file_eos	(	source_file_path,
		destination_file_path,
		source_storage_url,
		destination_storage_url
	)

Definition at line 476 of file function_set.py.

References get_process_full().

Referenced by function_set.FileAccessException.__str__(), and get_file_info().

◆ cp_file_ssh()

function_set.cp_file_ssh	(	source_file_path,
		destination_file_path,
		source_ssh_prefix,
		destination_ssh_prefix
	)

COPY FILE ### copy file via SCP (in case of ssh-key access).

ssh_prefix format: "user@host"

Definition at line 408 of file function_set.py.

References get_process_full().

Referenced by function_set.FileAccessException.__str__(), and get_file_info().

◆ cp_file_ssh_protocol()

function_set.cp_file_ssh_protocol	(	source_file_path,
		destination_file_path,
		connection_ssh_prefix,
		transfer_protocol,
		copy_prefix,
		is_source_copy_prefix = `False`
	)

Definition at line 435 of file function_set.py.

References get_process_full().

◆ cp_file_xrd()

function_set.cp_file_xrd	(	source_file_path,
		destination_file_path,
		source_storage_url,
		destination_storage_url
	)

Definition at line 463 of file function_set.py.

References get_process_full().

Referenced by function_set.FileAccessException.__str__(), get_file_info(), and transfer_raw_data.transfer_file().

◆ get_file_adler32a()

function_set.get_file_adler32a ( file_path )

 Get file Adler32 checksum (read whole file at once)

:param file_path: file path
:returns: Adler32 file checksum
:raises FileAccessException: if there is any error while getting file checksum (message + integer code)

Definition at line 220 of file function_set.py.

◆ get_file_adler32c()

function_set.get_file_adler32c ( file_path )

 Get file Adler32 checksum (read splitted by chunks)

:param file_path: file path
:returns: Adler32 file checksum
:raises FileAccessException: if there is any error while getting file checksum (message + integer code)

Definition at line 195 of file function_set.py.

Referenced by function_set.FileAccessException.__str__(), and get_file_info().

◆ get_file_eossize()

function_set.get_file_eossize	(	file_path,
		xrootd_url
	)

 Get file size in bytes via EOS

:param file_path: file path
:param xrootd_url:  XRootD/EOS URL, e.g. 'root://eos.jinr.ru//eos/nica/bmn...'
:returns: file size in bytes
:raises FileAccessException: if there is any error while getting file size (message + integer code)

Definition at line 140 of file function_set.py.

References get_process_full().

Referenced by function_set.FileAccessException.__str__(), and get_file_info().

◆ get_file_info()

function_set.get_file_info	(	file_path,
		storage_url,
		protocol_type,
		ssh_con,
		start_tmp_dir,
		match_checksum = `0`
	)

 Get file information (size and checksum) using different protocols (in case of errors: file_size and/or file_checksum is None)

:param file_path: input file path to get its information
:param storage_url: storage URL, where the input file is located
:param protocol type: 0 - local file; 1 - SSH protocol; 2 - XRootD protocol; 3 - EOS protocol
:param ssh_con: SSH prefix as 'user@server'
:param start_tmp_dir: temporary directory to copy file for checking
:param match_checksum: 0 - do not match checksums, 1 - fast comparing checksum if it exists in the File System, 2- always manual recalculating checksum before comparing
:returns: 
   - 1 - file size (integer, in bytes);
   - 2 - file checksum (string);
   - 3 - error code (0 - no errors, 1 - file not found, 2 - file read error, 3 - error while getting checksum, <0 - internal errors)
   - 4 - error message (in case of any errors)

Definition at line 268 of file function_set.py.

References check_file_exists_xrd(), cp_file_eos(), cp_file_ssh(), cp_file_xrd(), get_file_adler32c(), get_file_eossize(), get_file_sshsize(), get_file_xrdsize(), and get_file_xrdsum().

◆ get_file_MD5a()

function_set.get_file_MD5a ( file_path )

 Get file MD5 checksum (read all at once)

:param file_path: file path
:returns: MD5 file checksum
:raises FileAccessException: if there is any error while getting file checksum (message + integer code)

Definition at line 176 of file function_set.py.

◆ get_file_MD5c()

function_set.get_file_MD5c ( file_path )

 Get file MD5 checksum (read splitted by chunks)

:param file_path: file path
:returns: MD5 file checksum
:raises FileAccessException: if there is any error while getting file checksum (message + integer code)

Definition at line 156 of file function_set.py.

◆ get_file_sshsize()

function_set.get_file_sshsize	(	file_path,
		ssh_data
	)

 Get file size in bytes via SSH (in case of ssh-key access)

:param file_path: file path
:param ssh_data:  ssh login format: "user@host"
:returns: file size in bytes
:raises FileAccessException: if there is any error while getting file size (message + integer code)

Definition at line 108 of file function_set.py.

References get_process_full().

Referenced by function_set.FileAccessException.__str__(), and get_file_info().

◆ get_file_xrdsize()

function_set.get_file_xrdsize	(	file_path,
		xrootd_url
	)

 Get file size in bytes via XRootD

:param file_path: file path
:param xrootd_url:  XRootD URL, e.g. 'root://eos.jinr.ru//eos/nica/bmn...'
:returns: file size in bytes
:raises FileAccessException: if there is any error while getting file size (message + integer code)

Definition at line 124 of file function_set.py.

References get_process_full().

Referenced by function_set.FileAccessException.__str__(), get_file_info(), and transfer_raw_data.transfer_file().

◆ get_file_xrdsum()

function_set.get_file_xrdsum	(	file_path,
		xrootd_url
	)

 Get file checksum via XRootD

:param file_path: file path
:param xrootd_url:  XRootD/EOS URL, e.g. 'root://eos.jinr.ru//eos/nica/bmn...'
:returns: file checksum (Adler32 at the NICA storages)
:raises FileAccessException: if there is any error while getting file checksum (message + integer code)
:raises ValueError: if returned checksum has incorrect format (non-HEX value)

Definition at line 246 of file function_set.py.

References get_process_full().

Referenced by function_set.FileAccessException.__str__(), get_file_info(), and transfer_raw_data.transfer_file().

◆ get_files_eos()

function_set.get_files_eos	(	dir,
		xrootd_url
	)

Definition at line 391 of file function_set.py.

References get_files_eos(), and ls_dir_eos().

Referenced by function_set.FileAccessException.__str__(), and get_files_eos().

◆ get_files_xrd()

function_set.get_files_xrd	(	dir,
		xrootd_url
	)

Definition at line 378 of file function_set.py.

References get_files_xrd(), get_process_retcode(), and ls_dir_xrd().

Referenced by function_set.FileAccessException.__str__(), and get_files_xrd().

◆ get_process_full()

function_set.get_process_full ( process_string )

Definition at line 70 of file function_set.py.

Referenced by function_set.FileAccessException.__str__(), check_file_exists_ssh(), check_file_exists_xrd(), cp_file_eos(), cp_file_ssh(), cp_file_ssh_protocol(), cp_file_xrd(), get_file_eossize(), get_file_sshsize(), get_file_xrdsize(), get_file_xrdsum(), and rm_file_xrd().

◆ get_process_output()

function_set.get_process_output ( process_string )

Definition at line 59 of file function_set.py.

◆ get_process_retcode()

function_set.get_process_retcode ( process_string )

Definition at line 64 of file function_set.py.

Referenced by function_set.FileAccessException.__str__(), check_file_exists_eos(), and get_files_xrd().

◆ ls_dir_eos()

function_set.ls_dir_eos	(	file_path,
		xrootd_url
	)

Definition at line 373 of file function_set.py.

Referenced by function_set.FileAccessException.__str__(), and get_files_eos().

◆ ls_dir_xrd()

function_set.ls_dir_xrd	(	file_path,
		xrootd_url
	)

GET FILE LIST ### list XRootD directory.

Definition at line 368 of file function_set.py.

Referenced by function_set.FileAccessException.__str__(), and get_files_xrd().

◆ rm_file_xrd()

function_set.rm_file_xrd	(	file_path,
		xrootd_url
	)

REMOVE FILE ### remove remote file via XRootD.

Definition at line 490 of file function_set.py.

References get_process_full().

Variable Documentation

◆ CHECKSUM_BLOCKSIZE

int function_set.CHECKSUM_BLOCKSIZE = 1024*1024*1024

Definition at line 11 of file function_set.py.

◆ datefmt

function_set.datefmt

Definition at line 17 of file function_set.py.

◆ format

function_set.format

Definition at line 16 of file function_set.py.

◆ handlers

function_set.handlers

Definition at line 18 of file function_set.py.

◆ storage_dictionary

list function_set.storage_dictionary = ['ddc','ncx', 'cicc']

Definition at line 12 of file function_set.py.

Classes

Functions

Variables

Function Documentation

◆ check_file_exists_eos()

◆ check_file_exists_ssh()

◆ check_file_exists_xrd()

◆ cp_file_eos()

◆ cp_file_ssh()

◆ cp_file_ssh_protocol()

◆ cp_file_xrd()

◆ get_file_adler32a()

◆ get_file_adler32c()

◆ get_file_eossize()

◆ get_file_info()

◆ get_file_MD5a()

◆ get_file_MD5c()

◆ get_file_sshsize()

◆ get_file_xrdsize()

◆ get_file_xrdsum()

◆ get_files_eos()

◆ get_files_xrd()

◆ get_process_full()

◆ get_process_output()

◆ get_process_retcode()

◆ ls_dir_eos()

◆ ls_dir_xrd()

◆ rm_file_xrd()

Variable Documentation

◆ CHECKSUM_BLOCKSIZE

◆ datefmt

◆ format

◆ handlers

◆ storage_dictionary