Version of collatedFileOperation with multiple read/write ranks. More...

Inheritance diagram for hostCollatedFileOperation:

Collaboration diagram for hostCollatedFileOperation:

Public Member Functions
	TypeName ("hostCollated")
	Runtime type information. More...

	hostCollatedFileOperation (bool verbose=false)
	Default construct. More...

	hostCollatedFileOperation (const Tuple2< label, labelList > &commAndIORanks, const bool distributedRoots, bool verbose=false)
	Construct from communicator with specified io-ranks. More...

virtual	~hostCollatedFileOperation ()
	Destructor. More...

virtual void	storeComm () const
	Transfer ownership of communicator to this fileOperation. Use with caution. More...

Public Member Functions inherited from collatedFileOperation
	TypeName ("collated")
	Runtime type information. More...

	collatedFileOperation (bool verbose=false)
	Default construct. More...

	collatedFileOperation (const Tuple2< label, labelList > &commAndIORanks, const bool distributedRoots, bool verbose=false)
	Construct from communicator with specified io-ranks. More...

virtual	~collatedFileOperation ()
	Destructor. More...

virtual fileName	objectPath (const IOobject &io, const word &typeName) const
	Generate disk file name for object. Opposite of filePath. More...

virtual bool	writeObject (const regIOobject &, IOstreamOption streamOpt=IOstreamOption(), const bool writeOnProc=true) const
	Writes a regIOobject (so header, contents and divider). More...

virtual void	flush () const
	Forcibly wait until all output done. Flush any cached data. More...

virtual word	processorsDir (const IOobject &) const
	Actual name of processors dir. More...

virtual word	processorsDir (const fileName &) const
	Actual name of processors dir. More...

Public Member Functions inherited from masterUncollatedFileOperation
	TypeName ("masterUncollated")
	Runtime type information. More...

	masterUncollatedFileOperation (bool verbose=false)
	Default construct. More...

	masterUncollatedFileOperation (const Tuple2< label, labelList > &commAndIORanks, const bool distributedRoots, bool verbose=false)
	Construct from communicator with specified io-ranks. More...

virtual	~masterUncollatedFileOperation ()
	Destructor. More...

virtual bool	mkDir (const fileName &, mode_t=0777) const
	Make directory. More...

virtual bool	chMod (const fileName &, const mode_t) const
	Set the file mode. More...

virtual mode_t	mode (const fileName &, const bool followLink=true) const
	Return the file mode. More...

virtual fileName::Type	type (const fileName &, const bool followLink=true) const
	Return the file type: DIRECTORY, FILE or SYMLINK. More...

virtual bool	exists (const fileName &, const bool checkGzip=true, const bool followLink=true) const
	Does the name exist (as DIRECTORY or FILE) in the file system? More...

virtual bool	isDir (const fileName &, const bool followLink=true) const
	Does the name exist as a DIRECTORY in the file system? More...

virtual bool	isFile (const fileName &, const bool checkGzip=true, const bool followLink=true) const
	Does the name exist as a FILE in the file system? More...

virtual off_t	fileSize (const fileName &, const bool followLink=true) const
	Return size of file. More...

virtual time_t	lastModified (const fileName &, const bool followLink=true) const
	Return time of last file modification. More...

virtual double	highResLastModified (const fileName &, const bool followLink=true) const
	Return time of last file modification. More...

virtual fileNameList	readDir (const fileName &, const fileName::Type=fileName::FILE, const bool filtergz=true, const bool followLink=true) const
	Read a directory and return the entries as a string list. More...

virtual bool	cp (const fileName &src, const fileName &dst, const bool followLink=true) const
	Copy, recursively if necessary, the source to the destination. More...

virtual bool	ln (const fileName &src, const fileName &dst) const
	Create a softlink. dst should not exist. Returns true if. More...

virtual bool	mv (const fileName &src, const fileName &dst, const bool followLink=false) const
	Rename src to dst. More...

virtual bool	mvBak (const fileName &, const std::string &ext="bak") const
	Rename to a corresponding backup file. More...

virtual bool	rm (const fileName &) const
	Remove a file, returning true if successful otherwise false. More...

virtual bool	rmDir (const fileName &dir, const bool silent=false, const bool emptyOnly=false) const
	Remove a directory and its contents. More...

virtual fileName	filePath (const bool checkGlobal, const IOobject &io, const word &typeName, const bool search) const
	Search for an object. checkGlobal : also check undecomposed case. More...

virtual fileName	dirPath (const bool checkGlobal, const IOobject &io, const bool search) const
	Search for a directory. checkGlobal : also check undecomposed. More...

virtual fileNameList	readObjects (const objectRegistry &db, const fileName &instance, const fileName &local, word &newInstance) const
	Search directory for objects. Used in IOobjectList. More...

virtual bool	readHeader (IOobject &, const fileName &, const word &typeName) const
	Read object header from supplied file. More...

virtual autoPtr< ISstream >	readStream (regIOobject &, const fileName &, const word &typeName, const bool readOnProc=true) const
	Reads header for regIOobject and returns an ISstream to read the contents. More...

virtual bool	read (regIOobject &, const bool masterOnly, const IOstreamOption::streamFormat format, const word &typeName) const
	Top-level read. More...

virtual autoPtr< ISstream >	NewIFstream (const fileName &) const
	Generate an ISstream that reads a file. More...

virtual autoPtr< OSstream >	NewOFstream (const fileName &pathname, IOstreamOption streamOpt=IOstreamOption(), const bool writeOnProc=true) const
	Generate an OSstream that writes a file. More...

virtual autoPtr< OSstream >	NewOFstream (IOstreamOption::atomicType atomic, const fileName &pathname, IOstreamOption streamOpt=IOstreamOption(), const bool writeOnProc=true) const
	Generate an OSstream that writes a file. More...

virtual label	addWatch (const fileName &) const
	Add watching of a file. Returns handle. More...

virtual bool	removeWatch (const label) const
	Remove watch on a file (using handle) More...

virtual label	findWatch (const labelList &watchIndices, const fileName &) const
	Find index (or -1) of file in list of handles. More...

virtual void	addWatches (regIOobject &, const fileNameList &) const
	Helper: add watches for list of regIOobjects. More...

virtual fileName	getFile (const label) const
	Get name of file being watched (using handle) More...

virtual void	updateStates (const bool masterOnly, const bool syncPar) const
	Update state of all files. More...

virtual fileMonitor::fileState	getState (const label) const
	Get current state of file (using handle) More...

virtual void	setUnmodified (const label) const
	Set current state of file (using handle) to unmodified. More...

virtual instantList	findTimes (const fileName &, const word &) const
	Get sorted list of times. More...

virtual IOobject	findInstance (const IOobject &io, const scalar startValue, const word &stopInstance) const
	Find instance where IOobject is. More...

virtual void	setTime (const Time &) const
	Callback for time change. More...

virtual void	sync ()
	Forcibly parallel sync. More...

const HashPtrTable< DynamicList< instant > > &	times () const noexcept
	Return cached times. More...

Public Member Functions inherited from fileOperation
	TypeName ("fileOperation")
	Runtime type information. More...

	fileOperation (const label comm, const labelUList &ioRanks=labelUList::null(), const bool distributedRoots=false)
	Construct from communicator, optionally with specified io-ranks and/or distributed roots. More...

	fileOperation (const Tuple2< label, labelList > &commAndIORanks, const bool distributedRoots=false)
	Construct from communicator with specified io-ranks. More...

	declareRunTimeSelectionTable (autoPtr, fileOperation, word,(bool verbose),(verbose))

	declareRunTimeSelectionTable (autoPtr, fileOperation, comm,(const Tuple2< label, labelList > &commAndIORanks, const bool distributedRoots, bool verbose),(commAndIORanks, distributedRoots, verbose))

virtual	~fileOperation ()=default
	Destructor. More...

label	comm () const noexcept
	Communicator to use. More...

label	comm (label communicator) const noexcept
	Set communicator to use [mutable]. Negative values are a no-op. More...

bool	distributed () const noexcept
	Distributed roots (parallel run) More...

bool	distributed (bool on) const noexcept
	Set distributed roots on/off [mutable]. More...

const labelList &	ioRanks () const noexcept
	The list of IO ranks (global ranks) More...

InfoProxy< fileOperation >	info () const noexcept
	Return info proxy, used to print information to a stream. More...

virtual bool	good () const
	True if the fileOperation can be considered valid. At the moment, primarily used to detect the dummy fileOperation. More...

virtual bool	broadcastCopy (const label comm, const bool writeOnProc, const fileName &src, const fileName &dst) const
	Read dir/file (recursively if necessary) on master of the communicator, send and write contents to all 'writeOnProc' processors with local file name. More...

virtual fileName	filePath (const fileName &, const bool checkGzip=true, const bool followLink=true) const
	Search for a file or directory. Use IOobject version in preference. More...

label	nProcs () const noexcept
	Overall number of processors, from UPstream::nProcs() or detected from directories/results. More...

label	nProcs (const label numProcs) noexcept
	Set number of processor directories/results. More...

virtual label	nProcs (const fileName &dir, const fileName &local="", const label wantedNProcs=0) const
	Get number of processor directories/results. More...

fileName	processorsCasePath (const IOobject &io, const word &procDir) const
	Generate path (like io.path) from root+casename with any. More...

fileName	processorsPath (const IOobject &io, const word &instance, const word &procDir) const
	Generate path (like io.path) with provided instance and any. More...

fileName	processorsPath (const fileName &, const word &procDir) const
	Operating on fileName: replace processorXXX with procDir. More...

label	setNProcs (label numProcs)
	Same as nProcs. More...

Additional Inherited Members
Public Types inherited from fileOperation
enum	pathType : int { NOTFOUND = 0, ABSOLUTE, OBJECT, WRITEOBJECT, PROCUNCOLLATED, PROCBASEOBJECT = PROCUNCOLLATED + 1, PROCOBJECT = PROCBASEOBJECT + 1, PARENTOBJECT, FINDINSTANCE, PROCUNCOLLATEDINSTANCE, PROCBASEINSTANCE, PROCINSTANCE }
	Enumeration for the location of an IOobject. More...

typedef Tuple2< fileName, Tuple2< pathType, int > >	dirIndex
	Augment fileName with pathType and local offset. More...

typedef List< dirIndex >	dirIndexList

typedef IntRange< int >	procRangeType
	For addressing a range of processors, identical to UPstream::rangeType. More...

Static Public Member Functions inherited from fileOperation
static refPtr< fileOperation >	null ()
	Reference to a dummy file handler. More...

static autoPtr< fileOperation >	New (const word &handlerType, bool verbose=false)
	Select fileHandler-type. Uses defaultFileHandler if the handlerType is empty. More...

static autoPtr< fileOperation >	New (const word &handlerType, const Tuple2< label, labelList > &commAndIORanks, const bool distributedRoots, bool verbose=false)
	Select fileHandler-type. Uses defaultFileHandler if the handlerType is empty. More...

static autoPtr< fileOperation >	New (const fileOperation &origHandler, const boolUList &useProc, bool verbose=false)
	Replicate the given fileHandler properties with a subset of (global) ranks. More...

static autoPtr< fileOperation >	New (const fileOperation &origHandler, const bitSet &useProc, bool verbose=false)
	Replicate the given fileHandler properties with a subset of (global) ranks. More...

static autoPtr< fileOperation >	NewUncollated ()
	The commonly used uncollatedFileOperation. More...

static const fileOperation &	fileHandler ()
	Return the current file handler. Will create the default file handler if necessary. More...

static refPtr< fileOperation >	fileHandler (std::nullptr_t)
	Delete current file handler. More...

static refPtr< fileOperation >	fileHandler (refPtr< fileOperation > &newHandler)
	Replace the current file handler. More...

static refPtr< fileOperation >	fileHandler (refPtr< fileOperation > &&newHandler)
	Replace the current file handler. More...

static refPtr< fileOperation >	fileHandler (autoPtr< fileOperation > &&newHandler)
	Replace the current file handler. More...

static int	cacheLevel () noexcept
	Return cache level. More...

static int	cacheLevel (int level) noexcept
	Set cache level (0 = off). More...

static int	nProcsFilter () noexcept
	Return collated 'processorsDDD' filtering. More...

static int	nProcsFilter (int level) noexcept
	Set collated 'processorsDDD' filtering (0 = off). More...

static instantList	sortTimes (const fileNameList &dirEntries, const word &constantName="constant")
	Sort directory entries according to time value,. More...

static bool	uniformFile (const fileNameList &names)
	True if the file names are identical. False on an empty list. More...

static bool	uniformFile (const label comm, const fileName &name)
	True if the file name is identical on all ranks. More...

static label	splitProcessorPath (const fileName &objectPath, fileName &path, fileName &procDir, fileName &local, procRangeType &group, label &nProcs)
	Split objectPath into part before 'processor' and part after. More...

static label	detectProcessorPath (const fileName &objPath)
	Detect processor number from '/aa/bb/processorDDD/cc'. More...

static labelRange	subRanks (const labelUList &mainIOranks)
	Get (contiguous) range/bounds of ranks addressed within the given main io-ranks. More...

static labelList	getGlobalHostIORanks ()
	Get list of global IO master ranks based on the hostname. It is assumed that each host range is contiguous. More...

static labelList	getGlobalIORanks ()
	Get list of global IO ranks from FOAM_IORANKS env variable. If set, these correspond to the IO master ranks. More...

Static Public Attributes inherited from collatedFileOperation
static float	maxThreadFileBufferSize
	Max size of thread buffer size. This is the overall size of. More...

Static Public Attributes inherited from masterUncollatedFileOperation
static float	maxMasterFileBufferSize
	Max size of parallel communications. Switches from non-blocking. More...

Static Public Attributes inherited from fileOperation
static const Enum< pathType >	pathTypeNames_

static word	processorsBaseDir = "processors"
	Return the processors directory name (usually "processors") More...

static word	defaultFileHandler
	Name of the default fileHandler. More...

static refPtr< fileOperation >	fileHandlerPtr_
	The currently active file handler. Avoid accessing directly. More...

Protected Member Functions inherited from collatedFileOperation
void	printBanner (const bool withRanks=false) const
	Print banner information, optionally with io ranks. More...

bool	appendObject (const regIOobject &io, const fileName &pathName, IOstreamOption streamOpt) const
	Append to processorsNN/ file. More...

Protected Member Functions inherited from masterUncollatedFileOperation
template<class Type >
Type	scatterList (const UList< Type > &, const int, const label comm) const

template<class Type , class FileOp >
Type	masterOp (const fileName &fName, const FileOp &fop, const int tag, const label comm) const

template<class Type , class FileOp >
Type	masterOp (const fileName &src, const fileName &dest, const FileOp &fop, const int tag, const label comm) const

virtual fileName	filePathInfo (const bool checkGlobal, const bool isFile, const IOobject &io, const dirIndexList &pDirs, const bool search, pathType &searchType, word &processorsDir, word &instance) const
	Search (locally!) for object; return info on how it was found. More...

fileName	localObjectPath (const IOobject &, const pathType &searchType, const word &processorsDir, const word &instancePath) const
	Construct filePath. More...

bool	exists (const dirIndexList &, IOobject &io) const
	Helper: check IO for local existence. Like filePathInfo but. More...

Protected Member Functions inherited from fileOperation
fileMonitor &	monitor () const
	Get or create fileMonitor singleton. More...

refPtr< dirIndexList >	lookupAndCacheProcessorsPath (const fileName &objectPath, const bool syncPar) const
	Lookup name of processorsDDD using cache. More...

virtual refPtr< dirIndexList >	lookupProcessorsPath (const fileName &objectPath) const
	Lookup name of processorsDDD using cache. More...

bool	exists (IOobject &io) const
	Does IOobject exist? Is either a directory (empty name()) or a file. More...

bool	isIOrank (const label proci) const
	Is proci a master rank in the communicator (in parallel) or a master rank in the IO ranks (non-parallel) More...

void	printRanks () const
	Helper: output which ranks are IO. More...

Static Protected Member Functions inherited from masterUncollatedFileOperation
static word	findInstancePath (const instantList &timeDirs, const instant &t)
	Equivalent of Time::findInstance. More...

static void	readAndSend (const fileName &filePath, const labelUList &recvProcs, PstreamBuffers &pBufs)
	Read file contents and send to processors. More...

static autoPtr< ISstream >	read (IOobject &io, const label comm, const bool uniform, const fileNameList &filePaths, const boolUList &readOnProcs)
	Read files on comms master. More...

Static Protected Member Functions inherited from fileOperation
static void	mergeTimes (const instantList &extraTimes, const word &constantName, instantList &times)
	Merge two times. More...

static bool	isFileOrDir (const bool isFile, const fileName &)
	Helper: check for file (isFile) or directory (!isFile) More...

static label	getManagedComm (const label communicator)
	Construction helper: check for locally allocated communicator. More...

Protected Attributes inherited from collatedFileOperation
OFstreamCollator	writer_
	Threaded writer. More...

Protected Attributes inherited from masterUncollatedFileOperation
HashPtrTable< DynamicList< instant > >	times_
	Cached times for a given directory. More...

Protected Attributes inherited from fileOperation
label	comm_
	Communicator to use. More...

label	nProcs_
	Overall number of processors. More...

bool	distributed_
	Distributed roots (parallel run) More...

const labelList	ioRanks_
	The list of IO ranks (global ranks) More...

HashTable< dirIndexList >	procsDirs_
	Detected processors directories. More...

std::unique_ptr< fileMonitor >	monitorPtr_
	File-change monitor for all registered files. More...

Static Protected Attributes inherited from fileOperation
static int	cacheLevel_
	Cache level (eg, for caching time directories). Default: 1. More...

Detailed Description

Version of collatedFileOperation with multiple read/write ranks.

In parallel it will assume ranks are sorted according to hostname and the lowest rank per hostname will be the IO rank. The output directories will get a unique name processors<N>_<low>-<high> where N is the overall number of processors and low and high is the range of ranks contained in the files. Each of these subsets uses its own communicator.

Instead of using the hostnames the IO ranks can be assigned using the FOAM_IORANKS environment variable (also when running non-parallel), e.g. when decomposing into 4:

    FOAM_IORANKS='(0 2)' decomposePar -fileHandler hostCollated
    FOAM_IORANKS='0,2'   decomposePar -fileHandler hostCollated
    FOAM_IORANKS='0 2'   decomposePar -fileHandler hostCollated

will generate

    processors4_0-1/  : containing data for processors 0 to 1
    processors4_2-3/  : containing data for processors 2 to 3

Environment

FOAM_ENV : list of io-ranks as plain space or comma separated list or as an OpenFOAM formatted list. Eg, '(0 4 8)'

See also: Foam::collatedFileOperation

Source files

Definition at line 79 of file hostCollatedFileOperation.H.

Constructor & Destructor Documentation

◆ hostCollatedFileOperation() [1/2]

hostCollatedFileOperation ( bool verbose = false )

explicit

Default construct.

Definition at line 107 of file hostCollatedFileOperation.C.

◆ hostCollatedFileOperation() [2/2]

hostCollatedFileOperation	(	const Tuple2< label, labelList > &	commAndIORanks,
		const bool	distributedRoots,
		bool	verbose = `false`
	)

explicit

Construct from communicator with specified io-ranks.

Definition at line 124 of file hostCollatedFileOperation.C.

◆ ~hostCollatedFileOperation()

~hostCollatedFileOperation ( )

virtual

Destructor.

Definition at line 151 of file hostCollatedFileOperation.C.

References UPstream::freeCommunicator().

Here is the call graph for this function:

Member Function Documentation

◆ TypeName()

TypeName ( "hostCollated" )

Runtime type information.

◆ storeComm()

void storeComm ( ) const

virtual

Transfer ownership of communicator to this fileOperation. Use with caution.

Reimplemented from collatedFileOperation.

Definition at line 142 of file hostCollatedFileOperation.C.

The documentation for this class was generated from the following files:

src/OpenFOAM/global/fileOperations/collatedFileOperation/hostCollatedFileOperation.H
src/OpenFOAM/global/fileOperations/collatedFileOperation/hostCollatedFileOperation.C

Public Member Functions

Additional Inherited Members