Skip to content
Snippets Groups Projects
Verified Commit 02b804b1 authored by Adam Caprez's avatar Adam Caprez
Browse files

Drop header and reorder.

parent b4ce891f
Branches
No related tags found
1 merge request!331Drop header and reorder.
......@@ -22,6 +22,14 @@ operations, such as testing and running applications, one should use an
lots of threads for compiling applications, or checking the job status multiple times a minute.
## File Systems
* **No POSIX file system performs well with an excessive number of files**, as each file operation
requires opening and closing, which is relatively expensive.
* Moreover, network data transfer operations that involve frequent scanning (walking) of every
file in a set for syncing operations (backups, automated copying) can become excessively taxing for
network file systems, especially at scale.
* Large numbers of files can take an inordinate amount of time to transfer in or out of network
file systems during data migration operations.
* **Computing workflows can be negatively impacted by unnecessarily large numbers of file operations**, including file transfers.
* Some I/O intensive jobs may benefit from **copying the data to the fast, temporary /scratch
file system local to each worker nodes**. The */scratch* directories are unique per job, and
are deleted when the job finishes. Thus, the last step of the batch script should copy the
......@@ -36,15 +44,6 @@ all the necessary files need to be either moved to a permanent storage, or delet
disk, in your program.** This approach stresses the file system and may cause general issues.
Instead, consider reading and writing large blocks of data in memory over time, or
utilizing more advanced parallel I/O libraries, such as *parallel hdf5* and *parallel netcdf*.
#### Large numbers of files considerations
* **No POSIX file system performs well with an excessive number of files**, as each file operation
requires opening and closing, which is relatively expensive.
* Moreover, network data transfer operations that involve frequent scanning (walking) of every
file in a set for syncing operations (backups, automated copying) can become excessively taxing for
network file systems, especially at scale.
* Large numbers of files can take an inordinate amount of time to transfer in or out of network
file systems during data migration operations.
* **Computing workflows can be negatively impacted by unnecessarily large numbers of file operations**, including file transfers.
## Internal and External Networks
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment