SEARCH · Engineering Papers

Results for “FILES”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

All sources NASA NTRS DOE OSTI

DOE OSTI2025

2D reactive transport model of shale chemical weathering and biogeochemical fluxes along a mountainous hillslope, East River Watershed, Colorado: Input files and simulation results

This data package contains input files and simulation results for a two-dimensional (2D) reactive transport model used to quantitatively analyze the coupled hydrological and biogeochemical processes governing shale weathering and associated biogeochemical fluxes under realistic environmental conditions in the high-elevation East River Watershed. These data support the conclusions presented in Stolze et al. (Water Resources Research, under review), "Model-based interpretation of solute exports and carbon partitioning during shale weathering in a mountainous hillslope". The model simulates atmospheric-subsurface gas exchange, subsurface water flow, and shale weathering processes under dynamic, year-scale conditions along a shale-underlain hillslope located in the East River watershed. The simulations were performed using the PFLOTRAN flow and reactive transport code and executed on the Perlmutter supercomputer to leverage its large-scale parallel computing capabilities. The data package contains two zipped folders, "model_input_files" and "simulation_results", and one readme.txt file. "model_input_files" contains the necessary input files to run the calibrated base-base model presented in Stolze et al. (Water Resources Research, under review). "simulation_results" contains a single hdf5 file ("Output_2D_hillslope_model.h5") which includes the results of simulation performed using the base-case model. This file can be opened with HDFView 3.1.4, Python, or MATLAB. "readme.txt" contains relevant information about the base-case model and provides guidelines on how to run the associated input files provided in the folder "model_input_files". Furthermore, readme.txt provides information regarding the model results provided in "Output_2D_hillslope_model.h5" such as matrix dimensionality and output units. Field datasets used to evaluate model performance were collected at three monitoring wells located along a hillslope transect (PLM1, PLM2, and PLM3). Dissolved ion concentration data were collected from November 2016 to October 2021 for Ca, Mg, DIC, Na, K, SO4 (Dong et al., 2025 - dic_npoc_data_2014_2024.zip - DOI:10.15485/1660459; Williams et al., 2025 - anion_data_2014_2024.zip - DOI:10.15485/1668054; Dong et al., 2025 - cation_data_2014_2024.zip - DOI:10.15485/1668055). Note that we used the files named er_PLM1_xx_yy, er_PLM2_xx_yy, and er_PLM3_xx_yy where xx stands for the name of the aqueous species and yy stands for the depth where the measurements were performed. Soil water content ([0 - 1] m) and water table depth were collected from November 2016 to October 2021 (Wan et al., 2024 - Dynamic_water_table__depthsFig2b.csv and Soil_water_content_Fig4e.csv - DOI:10.15485/2322567). Gaseous CO2 concentration were collected from October 2020 to December 2021(Wan et al., 2024 - Soil_CO2_concentrations_Fig4h.csv - DOI:10.15485/2322567) Gaseous CO2 flux from the subsurface to the atmosphere were collected in the vicinity of PLM2 from October 2019 to May 2022 (Wu et al., 2025). Soil microbial biomass concentration was measured from August 2016 to June 2017 (Sorensen et al., 2019 - 2017_East_River_Pumphouse_Microbial_Biomass__1_.csv - DOI:10.15485/1577267) All field data are published as CSV files compatible with Microsoft Excel, MATLAB, and Python, or as text files. The coordinates of the monitoring wells and the CO2(g) flux sensor in the coordinate system WGS84 are: -PLM1: [38.9197710 ; -106.9492750] -PLM2: [38.9201580 ; -106.9487170] -PLM3: [38.9207843 ; -106.9483668] -PLM4: 38.9210060 ; -106.9479528] -CO2(g) flux sensor: [38.9199180 ; -106.9489906] ------------------------------------------------------------------------------------------- This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231. This research used resources of the National Energy Research Scientific Computing Center (NERSC), a Department of Energy User Facility using NERSC award BER-ERCAP 23980, BER-ERCAP 28550, and BER-ERCAP 33789.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2021

ESS-DIVE Reporting Format for Comma-separated Values (CSV) File Structure

The ESS-DIVE reporting format for Comma-separated Values (CSV) file structure is based on a combination of existing guidelines and recommendations including some found within the Earth Science Community with valuable input from the Environmental Systems Science (ESS) Community. The CSV reporting format is designed to promote interoperability and machine-readability of CSV data files while also facilitating the collection of some file-level metadata content. Tabular data in the form of rows and columns should be archived in its simplest form, and we recommend submitting these tabular data following the ESS-DIVE reporting format for generic comma-separated values (CSV) text format files. In general, the CSV file format is more likely accessible by future systems when compared to a proprietary format and CSV files are preferred because this format is easier to exchange between different programs increasing the interoperability of a data file. By defining the reporting format and providing guidelines for how to structure CSV files and some field content within, this can increase the machine-readability of the data file for extracting, compiling, and comparing the data across files and systems.Data package files are in .csv, .png, and .md. Open the .csv with e.g. Microsoft Excel, LibreOffice, or Google Sheets. Open the .md files by downloading and using a text editor (e.g., notepad or TextEdit). Open the .png in e.g. a web browser, photo viewer/editor, or Google Drive.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2013

Small File Aggregation with PLFS [Slides]

Today’s computational science demands have resulted in ever larger parallel computers, and storage systems have grown to match these demands. Parallel file systems used in this environment are increasingly specialized to extract the highest possible performance for large I/O operations, at the expense of other potential workloads. While some applications have adapted to I/O best practices and can obtain good performance on these systems, the natural I/O patterns of many applications result in the generation of a huge number of small files, the creation of which is poorly served by current parallel file systems at very large scale. This paper describes a new technique for optimizing small file access in parallel file systems for these very large scale systems. The idea is to use a virtual parallel log-structure file system on the compute nodes in order to aggregate large numbers of small files in compute node memory and then stream their data sequentially to a much smaller number of physical files on an underlying parallel file system. The technique is implemented and evaluated using PLFS as the aggregating middleware. We evaluate our system with micro-benchmarks on a local OSX filesystem and with an MPI extension of the standard Postmark to provide results at scale on both Lustre and PanFS parallel filesystems. We observe as much as a 33x improvement in small file create rates on a single host, and 30x improvement in small file write rates, compared to a baseline Lustre configuration on a leadership computing platform using 16,384 cores and achieve an unprecedented create rate of 200 million files per second.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Athena-I CUBIT Journal Files

The Monte Carlo N-Particle (MCNP)1 transport code version 6 (also known as MCNP6) has the capability for tracking particles on unstructured mesh (UM) geometry models embedded into con structive solid geometry (CSG) cells. This feature has been developed for performing calculations of complex geometry models because manually creating CSG models is a time-consuming and error prone process as the complexities of geometries increase. The UM capability was originally designed to work with UM models created with the Abaqus software and ASCII input files that it generates. The MCNP code version 6.0 and later can process UM models formatted as Abaqus input files. Starting with version 6.3, MCNP can also process HDF5 mesh input files. External codes must be used to generate Abaqus input files for MCNP UM calculations. CUBIT, the Sandia National Laboratory automated mesh generation toolkit, can generate a UM model formatted as an Abaqus input file. However, the Abaqus input files exported from CUBIT cannot be used for MCNP simulations because it lacks the proper syntax. A Python script was developed to convert an Abaqus file created by CUBIT to an Abaqus file format that MCNP can process. Creating UM models for complex geometries is not an easy task. The process of creating UM models in CUBIT for MCNP simulations is detailed in. CUBIT provides several user interface options including a graphical user interface (GUI) and a command line interface. A GUI provides an easy way to use CUBIT without learning the CUBIT command syntax. When using CUBIT with either interface option, command lines are written into an ASCII file known as a journal file; this journal file can be edited and archived so that it can be played back in CUBIT to automatically generate a UM model. This report describes the CUBIT journal files of the UM models developed for Athena-I. The Athena platform, an energy-tuning assembly, was developed to spectrally shape the National Ignition Facility (NIF) deuterium-tritium fusion neutron source to a thermonuclear (fusion) plus prompt fission neutron spectrum with capability to act as a short-pulse neutron source. MCNP6 was used for the Athena experiment design analysis.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2015

Methods and apparatus for multi-resolution replication of files in a parallel computing system using semantic information

Techniques are provided for storing files in a parallel computing system using different resolutions. A method is provided for storing at least one file generated by a distributed application in a parallel computing system. The file comprises one or more of a complete file and a sub-file. The method comprises the steps of obtaining semantic information related to the file; generating a plurality of replicas of the file with different resolutions based on the semantic information; and storing the file and the plurality of replicas of the file in one or more storage nodes of the parallel computing system. The different resolutions comprise, for example, a variable number of bits and/or a different sub-set of data elements from the file. A plurality of the sub-files can be merged to reproduce the file.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Accelerating shared file checkpoint with local burst buffers

A data management system and method for accelerating shared file checkpointing. Written application data is aggregated in an application data file created in a local burst buffer memory at a compute node, and an associated data mapping built index to maintain information related to the offsets into a shared file at which segments of the application data is to be stored in a parallel file system, and where in the buffer those segments are located. The node asynchronously transfers a data file containing the application data and the associated data mapping index to a file server for shared file storage. The data management system and method further accelerates shared file checkpointing in which a shared file, together with a map file that specifies how the shared file is to be distributed, is asynchronously transferred to local burst buffer memories at the nodes to accelerate reading of the shared file.

Gooding, Thomas↗

DOE OSTI2019

Measurements and analytics of wide-area file transfers over dedicated connections

Distributed scientific and big-data computations are becoming increasingly dependent on access to remote files. Wide-area file transfers are supported by two basic schemes: (i) application-level tools, such as GridFTP, that provide transport services between file systems housed at geographically separated sites, and (ii) file systems mounted over wide-area networks, using mechanisms such as LNet routers that make them transparently available. In both cases, the file transfer performance critically depends on the configuration consisting of host, file, IO, and disk subsystems, which are complex by themselves, as well as on their complex compositions implemented using buffers and IO-network data transitions. We present extensive file transfer rate measurements collected over dedicated 10 Gbps connections with 0-366 ms round-trip times, using GridFTP and XDD file transfer tools, and Lustre file system extended over wide-area networks using LNet routers. Our test configurations are composed of: three types of host systems; XFS, Lustre, and ext3 file systems; and Ethernet and SONET wide-area connections. We present analytics based on the convexity-concavity of throughput profiles which provide insights into throughput and its superior or inferior trend compared to linear interpolations. We propose the utilization-concavity coefficient, a scalar metric that characterizes the overall performance of any file transfer method consisting of specific configuration and scheme. Our results enable performance optimizations by highlighting the significant roles of (i) buffer sizes and parallelism in GridFTP and XDD, and (ii) buffer utilization and credit mechanism in LNet routers.

Rao, Nageswara↗

DOE OSTI2019

FGR 13 Dose Conversion Factor Files

The purpose of this document is to discuss the construction of two MACCS dose conversion factor (DCF) files in some detail, an older file created in 2007 named FGR13DCF.inp and a newer file created in 2018 called FGR13GyEquiv_RevA.inp. Very briefly, the difference between the two files is that the older file follows the standard conventions of assigning a radiation weighting factor of 20 for alpha radiation for all tissues and organs; whereas, the newer file complies with the FGR 13 health effects modeling and uses modified radiation weighting factors (referred to as relative biological effectiveness (RBE) factors) for alpha radiation of 10 for breast and of 1 for red bone marrow. During an intermediate period the creation of these two DCF files, a file called FGR13GyEquiv was created and used for the SOARCA calculations. This file was not released to the MACCS user community, but it is also discussed briefly in this document. DCF files are used by MACCS to convert air and ground radionuclide concentrations to doses to an organ or to the whole body. The MACCS calculation considers duration and shielding factor for each exposure pathway including cloudshine, groundshine, inhalation, and ingestion. Dose coefficients for cloudshine and groundshine are expressed as dose rates; dose coefficients for inhalation and ingestion are expressed as committed doses. In this document, dose coefficients (newer ICRP terminology) is used to describe the values contained in dose conversion factor (older ICRP terminology) files.

63 RADIATION, THERMAL, AND OTHER ENVIRON. POLLUTAN↗

NASA NTRS1998

Long-Term file activity patterns in a UNIX workstation environment

As mass storage technology becomes more affordable for sites smaller than supercomputer centers, understanding their file access patterns becomes crucial for developing systems to store rarely used data on tertiary storage devices such as tapes and optical disks. This paper presents a new way to collect and analyze file system statistics for UNIX-based file systems. The collection system runs in user-space and requires no modification of the operating system kernel. The statistics package provides details about file system operations at the file level: creations, deletions, modifications, etc. The paper analyzes four months of file system activity on a university file system. The results confirm previously published results gathered from supercomputer file systems, but differ in several important areas. Files in this study were considerably smaller than those at supercomputer centers, and they were accessed less frequently. Additionally, the long-term creation rate on workstation file systems is sufficiently low so that all data more than a day old could be cheaply saved on a mass storage device, allowing the integration of time travel into every file system.

Gibson, Timothy J.↗

NASA NTRS2013

Software to Compare NPP HDF5 Data Files

This software was developed for the NPOESS (National Polar-orbiting Operational Environmental Satellite System) Preparatory Project (NPP) Science Data Segment. The purpose of this software is to compare HDF5 (Hierarchical Data Format) files specific to NPP and report whether the HDF5 files are identical. If the HDF5 files are different, users have the option of printing out the list of differences in the HDF5 data files. The user provides paths to two directories containing a list of HDF5 files to compare. The tool would select matching HDF5 file names from the two directories and run the comparison on each file. The user can also select from three levels of detail. Level 0 is the basic level, which simply states whether the files match or not. Level 1 is the intermediate level, which lists the differences between the files. Level 2 lists all the details regarding the comparison, such as which objects were compared, and how and where they are different. The HDF5 tool is written specifically for the NPP project. As such, it ignores certain attributes (such as creation_date, creation_ time, etc.) in the HDF5 files. This is because even though two HDF5 files could represent exactly the same granule, if they are created at different times, the creation date and time would be different. This tool is smart enough to ignore differences that are not relevant to NPP users.

Wiegand, Chiu P.↗

DOE OSTI2014

Storing files in a parallel computing system based on user-specified parser function

Techniques are provided for storing files in a parallel computing system based on a user-specified parser function. A plurality of files generated by a distributed application in a parallel computing system are stored by obtaining a parser from the distributed application for processing the plurality of files prior to storage; and storing one or more of the plurality of files in one or more storage nodes of the parallel computing system based on the processing by the parser. The plurality of files comprise one or more of a plurality of complete files and a plurality of sub-files. The parser can optionally store only those files that satisfy one or more semantic requirements of the parser. The parser can also extract metadata from one or more of the files and the extracted metadata can be stored with one or more of the plurality of files and used for searching for files.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2015

Methods and apparatus for capture and storage of semantic information with sub-files in a parallel computing system

Techniques are provided for storing files in a parallel computing system using sub-files with semantically meaningful boundaries. A method is provided for storing at least one file generated by a distributed application in a parallel computing system. The file comprises one or more of a complete file and a plurality of sub-files. The method comprises the steps of obtaining a user specification of semantic information related to the file; providing the semantic information as a data structure description to a data formatting library write function; and storing the semantic information related to the file with one or more of the sub-files in one or more storage nodes of the parallel computing system. The semantic information provides a description of data in the file. The sub-files can be replicated based on semantically meaningful boundaries.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2016

Measurements of file transfer rates over dedicated long-haul connections

Wide-area file transfers are an integral part of several High-Performance Computing (HPC) scenarios. Dedicated network connections with high capacity, low loss rate and low competing traffic, are increasingly being provisioned over current HPC infrastructures to support such transfers. To gain insights into these file transfers, we collected transfer rate measurements for Lustre and xfs file systems between dedicated multi-core servers over emulated 10 Gbps connections with round trip times (rtt) in 0-366 ms range. Memory transfer throughput over these connections is measured using iperf, and file IO throughput on host systems is measured using xddprof. We consider two file system configurations: Lustre over IB network and xfs over SSD connected to PCI bus. Files are transferred using xdd across these connections, and the transfer rates are measured, which indicate the need to jointly optimize the connection and host file IO parameters to achieve peak transfer rates. In particular, these measurements indicate that (i) peak file transfer rate is lower than peak connection and host IO throughput, in some cases by as much as 50% or lower, (ii) xdd request sizes that achieve peak throughput for host file IO do not necessarily lead to peak file transfer rates, and (iii) parallelism in host IO and TCP transport does not always improve the file transfer rates.

log-haul connections↗

DOE OSTI2016

Measurements of file transfer rates over dedicated long-haul connections

Rao, Nageswara S↗

DOE OSTI2018

AutoGen: World's Fastest Building Energy Model Creator - Text Replacement for Variables for EnergyPlus files

AutoGen is the world's fastest EnergyPlus input file generator based on a base input file and sets of input variables. It has a simple user interface where a user can specify the directories and names of base input EnergyPlus file, list of input parameters to vary and metrics with the range of input values and a directory where the files will be generated. AutoGen uses OpenMP API to generate a large number of input files. It takes a base EnergyPlus input files (or a group of base Energyplus input files) along with a text or excel file with an input parameter to be varied and sensitivity metrics (i.e. the range of each input variables) of the input variables. It then uses Open PMP to perform the parallelization to generate the EnergyPlus files for all the permutation and combinations if input carriables as per the sensitivity metrics file. With AutoGen millions of files can be generated within a few minutes.

Adams, Mark B↗

DOE OSTI2019

File transfers between machines without target CPU intervention

Techniques for transferring files between machines include creating a zero-length target file on non-volatile storage, truncating the file to a desired size, and allocating storage on the non-volatile storage for each block of the target file. The technique also includes determining a logical block address (LBA) for each location in the target file. The technique further includes sending a request to an input/output (I/O) node to transfer a source file to the non-volatile storage, where the request includes a mapping between the LBAs and file offsets. The technique includes opening the source file and a block device at the I/O node. The technique further includes reading each block from the source file and writing each block to the target file on the non-volatile storage utilizing the block device, and then closing the source file and the block device.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

ESS-DIVE Reporting Format for File-level Metadata

The ESS-DIVE reporting format for file-level metadata (FLMD) provides granular information at the data file level to describe the contents, scope, and structure of the data file to enable comparison of data files within a data package. The FLMD are fully consistent with and augment the metadata collected at the data package level. We developed the FLMD template based on a review of a small number of existing FLMD in use at other agencies and repositories with valuable input from the Environmental Systems Science (ESS) Community. Also included is a template for a CSV Data Dictionary where users can provide file-level information about the contents of a CSV data file (e.g., define column names, provide units). Files are in .csv, .xlsx, and .md. Templates are in both .csv and .xlsx (open with e.g. Microsoft Excel, LibreOffice, or Google Sheets). Open the .md files by downloading and using a text editor (e.g. Notepad or TextEdit). Though we provide Excel templates for the file-level metadata reporting format, our instructions encourage users to 'Save the FLMD template as a CSV following the CSV Reporting Format guidance'. In addition, we developed the ESS-DIVE File Level Metadata Extractor which is a lightweight python script that can extract some FLMD fields following the recommended FLMD format and structure.

54 ENVIRONMENTAL SCIENCES↗

NASA NTRS1995

MPI-IO: A Parallel File I/O Interface for MPI Version 0.3

Thanks to MPI [9], writing portable message passing parallel programs is almost a reality. One of the remaining problems is file I/0. Although parallel file systems support similar interfaces, the lack of a standard makes developing a truly portable program impossible. Further, the closest thing to a standard, the UNIX file interface, is ill-suited to parallel computing. Working together, IBM Research and NASA Ames have drafted MPI-I0, a proposal to address the portable parallel I/0 problem. In a nutshell, this proposal is based on the idea that I/0 can be modeled as message passing: writing to a file is like sending a message, and reading from a file is like receiving a message. MPI-IO intends to leverage the relatively wide acceptance of the MPI interface in order to create a similar I/0 interface. The above approach can be materialized in different ways. The current proposal represents the result of extensive discussions (and arguments), but is by no means finished. Many changes can be expected as additional participants join the effort to define an interface for portable I/0. This document is organized as follows. The remainder of this section includes a discussion of some issues that have shaped the style of the interface. Section 2 presents an overview of MPI-IO as it is currently defined. It specifies what the interface currently supports and states what would need to be added to the current proposal to make the interface more complete and robust. The next seven sections contain the interface definition itself. Section 3 presents definitions and conventions. Section 4 contains functions for file control, most notably open. Section 5 includes functions for independent I/O, both blocking and nonblocking. Section 6 includes functions for collective I/O, both blocking and nonblocking. Section 7 presents functions to support system-maintained file pointers, and shared file pointers. Section 8 presents constructors that can be used to define useful filetypes (the role of filetypes is explained in Section 2 below). Section 9 presents how the error handling mechanism of MPI is supported by the MPI-IO interface. All this is followed by a set of appendices, which contain information about issues that have not been totally resolved yet, and about design considerations. The reader can find there the motivation behind some of our design choices. More information on this would definitely be welcome and will be included in a further release of this document. The first appendix contains a description of MPI-I0's 'hints' structure which is used when opening a file. Appendix B is a discussion of various issues in the support for file pointers. Appendix C explains what we mean in talking about atomic access. Appendix D provides detailed examples of filetype constructors, and Appendix E contains a collection of arguments for and against various design decisions.

Corbett, Peter↗