Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “File transfer”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

NOvA Inclusive Nue CC Cross Section Data Release

NOvA inclusive electron neutrino cross section results presented at Neutrino2020Exposure:8.09E20 protons-on-target, neutrino-enhanced beamData release contains 3 ROOT files containing the double-differential measurement with respect to electron angle (cos theta) and electron energy, a single-differential measurement with respect to squared four-momentum transfer (Q2), and the total cross section as a function of neutrino energy. The 1D cross-section measurements include the same kinematic phase space restrictions present in the double-differential analysis.The files, xsec_{dThdE/Enu/dQ2}.root, contain the extracted cross section and associated covariance matrices for the double-differential cross section with respect to the observed electron kinematics, dThdE, neutrino energy, Enu, and squared four-momentum transfer, Q^2.In each file there will be the following histograms:— TH2D/TH1D xsec: unfolded measured cross section— TH2D xsec_cov: cross section covariance matrix— TH2D stat_cov: statistical covariance matrixIncludes text files, xsec_{dThdE/Enu/dQ2}.txt show the corresponding histograms in plain text format.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2020

Distributed Computing for the Project 8 Experiment

The Project 8 collaboration aims to measure the absolute neutrino mass or improve on the current limit by measuring the tritium beta decay electron spectrum. We present the current distributed computing model for the Project 8 experiment. Project 8 is in its second phase of data taking with a near continuous data rate of 1Gbps. The current computing model uses DIRAC (Distributed Infrastructure with Remote Agent Control) for its workflow and data management. A detailed meta-data assignment using the DIRAC File Catalog is used to automate raw data transfers and subsequent stages of data processing. The DIRAC system is deployed on containers managed using a Kubernetes cluster to provide a scalable infrastructure. A modified DIRAC Site Director provides the ability to submit jobs using Singularity on opportunistic High-Performance Computing (HPC) sites.

Distributed Computing, Kubernetes, DIRAC, Project ↗
DOE OSTI2015

Data handling with SAM and art at the NO v A experiment

During operations, NOvA produces between 5,000 and 7,000 raw files per day with peaks in excess of 12,000. These files must be processed in several stages to produce fully calibrated and reconstructed analysis files. In addition, many simulated neutrino interactions must be produced and processed through the same stages as data. To accommodate the large volume of data and Monte Carlo, production must be possible both on the Fermilab grid and on off-site farms, such as the ones accessible through the Open Science Grid. To handle the challenge of cataloging these files and to facilitate their off-line processing, we have adopted the SAM system developed at Fermilab. SAM indexes files according to metadata, keeps track of each file's physical locations, provides dataset management facilities, and facilitates data transfer to off-site grids. To integrate SAM with Fermilab's art software framework and the NOvA production workflow, we have developed methods to embed metadata into our configuration files, art files, and standalone ROOT files. A module in the art framework propagates the embedded information from configuration files into art files, and from input art files to output art files, allowing us to maintain a complete processing history within our files. Embedding metadata in configuration files also allows configuration files indexed in SAM to be used as inputs to Monte Carlo production jobs. Further, SAM keeps track of the input files used to create each output file. Parentage information enables the construction of self-draining datasets which have become the primary production paradigm used at NOvA. In this study we will present an overview of SAM at NOvA and how it has transformed the file production framework used by the experiment.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2014

Release of Continuous Representation for S(α,β) ACE Data

For low energy neutrons, the default free gas model for scattering cross sections is not always appropriate. Molecular effects or crystalline structure effects can affect the neutron scattering cross sections. These effects are included in the S(α; β) thermal neutron scattering data and are tabulated in file 7 of the ENDF6 format files. S stands for scattering. α is a momentum transfer variable and is an energy transfer variable. The S(α; β) cross sections can include coherent elastic scattering (no E change for the neutron, but specific scattering angles), incoherent elastic scattering (no E change for the neutron, but continuous scattering angles), and inelastic scattering (E change for the neutron, and change in angle as well). Every S(α; β) material will have inelastic scattering and may have either coherent or incoherent elastic scattering (but not both). Coherent elastic scattering cross sections have distinctive jagged-looking Bragg edges, whereas the other cross sections are much smoother. The evaluated files from the NNDC are processed locally in the THERMR module of NJOY. Data can be produced either for continuous energy Monte Carlo codes (using ACER) or embedded in multi-group cross sections for deterministic (or even multi-group Monte Carlo) codes (using GROUPR). Currently, the S(α; β) files available for MCNP use discrete energy changes for inelastic scattering. That is, the scattered neutrons can only be emitted at specific energies— rather than across a continuous spectrum of energies. The discrete energies are chosen to preserve the average secondary neutron energy, i.e., in an integral sense, but the discrete treatment does not preserve any differential quantities in energy or angle.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2018

NUMA-Aware Thread Scheduling for Big Data Transfers over Terabits Network Infrastructure

The evergrowing trend of big data has led scientists to share and transfer the simulation and analytical data across the geodistributed research and computing facilities. However, the existing data transfer frameworks used for data sharing lack the capability to adopt the attributes of the underlying parallel file systems (PFS). LADS (Layout-Aware Data Scheduling) is an end-to-end data transfer tool optimized for terabit network using a layout-aware data scheduling via PFS. However, it does not consider the NUMA (Nonuniform Memory Access) architecture. In this paper, we propose a NUMA-aware thread and resource scheduling for optimized data transfer in terabit network. First, we propose distributed RMA buffers to reduce memory controller contention in CPU sockets and then schedule the threads based on CPU socket and NUMA nodes inside CPU socket to reduce memory access latency. We design and implement the proposed resource and thread scheduling in the existing LADS framework. Experimental results showed from 21.7% to 44% improvement with memory-level optimizations in the LADS framework as compared to the baseline without any optimization.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

TRAILS Output Files

Overview This data repository contains ZIP files that store compressed versions of the output of running the WaterPaths utility planning and management tool in the DU Re-Evaluation mode (to download the tool, please see this GitHub repository). The tool was used to simulate the six-utility North Carolina Research Triangle problem. Details on the contents of each ZIP file can be seen below. Data details Temporal range: Weekly data for 2,344 weeks from 2015 to 2060 (45 years). Spatial range: Six water utilities in the North Carolina Research Triangle region (0: Chapel Hil/OWASA, 1: Durham, 2: Cary, 3: Raleigh, 4: Pittsboro, and 5: Chatham) File types: CSV and OUT Different solutions available The solution numbers correspond to the different pathway strategies (henceforth referred to as "solutions") discussed in paper's main and supporting text (abstract and link to the paper here). They are as follows: Sol92: The Durham-focused pathway strategy Sol132: The Raleigh-focused pathway strategy Sol140: The regionally-robust pathway strategy Objectives files These files can be accessed by unzipping solXX_objectives_pathways.zip that contains 1,000 Objectives_RDMXX_solsXX_to_XX.csv files. Each CSV file will consist of a row representing all the objective values for that specific solution, while every six columns represents the reliability, restriction frequency, infrastructure net present value ($ mil), peak financial cost, worst-case cost, and unit cost ($ per MG; in that order) for each of the six utilities. There will be 1,000 such files, denoting the performance of the six utilities across the 1,000 deeply uncertain states of the world (DU SOWs). Pathway files These files can be accessed by unzipping solXX_objectives_pathways.zip that contains 1,000 Pathways_sXX_RDMXX.out file. Each OUT corresponds to the set of infrastructure being triggered in a specific DU SOW, and each file will have the name file will consist of four tab-delimited columns that are described as follows: Realization: The realization in which an infrastructure options being triggered utility: The utility currently triggering infrastructure week: The week in which a specific infrastructure option is being triggered infra.: The infrastructure option being triggered If the OUT file contains only the header line, no infrastructure was triggered for that specific DU SOW. Policies files These files can be obtained by unzipping Policies.zip. Each of the 1,000 CSV files within the unzipped folder will contain weekly water use restriction policies for all 1,000 hydroclimatic realizations within a specific DU SOW. The column structure is as follows: 0rest_m: restriction multiplier for utility 0 (values between 0 and 1) 1rest_m: restriction multiplier for utility 1 (values between 0 and 1) 2rest_m: restriction multiplier for utility 2 (values between 0 and 1) 3rest_m: restriction multiplier for utility 3 (values between 0 and 1) 4rest_m: restriction multiplier for utility 4 (values between 0 and 1) 5rest_m: restriction multiplier for utility 5 (values between 0 and 1) 0transf: transfer volume for utility 0 (in MGD) 1transf: transfer volume for utility 1 (in MGD) 2transf: transfer volume for utility 2 (in MGD) 3transf: transfer volume for utility 3 (in MGD) 4transf: transfer volume for utility 4 (in MGD) 5transf: transfer volume for utility 5 (in MGD) Water Sources files These files can be obtained by unzipping WaterSources_subset.zip. Each of the 100 CSV files within the unzipped folder will contain weekly state variables at each water source for all 1,000 hydroclimatic realizations within a specific DU SOW. The column structure is as follows: Xvolume: available water volume from source X (in MGD) Xs_area: surface area of source X (in ACF) Xdemand: demand drawn from a water source from source X (in MGD) Xup_spill: upstream spillage from source X (in MGD) Xww_inflow: wastewater inflow from source X (in MGD) Xcatch_inflow: upstream catchment inflow to source X (in MGD) Xevap: evaporation multiplier for source X (values between 0 and 1) Xds_spill: downstream spillage from source X (in MGD) X_Y_alloc_cap: the allocated capacity from source X to utility Y (values between 0 and 1) X_Y_alloc_dem: the allocated demand from source X to utility Y (values between 0 and 1) Xtrmt_alloc_Y: the allocated treatment capacity from source X to utility Y (values between 0 and 1) Utilities files These files can be obtained by unzipping Utilities_subset.zip. Each of the 100 CSV files within the unzipped folder will contain weekly state variables at each utility for all 1,000 hydroclimatic realizations within a specific DU SOW. The column structure is as follows: Xst_vol: total available storage volume of utility X (in MG) Xcapacity: total storage capacity of utility X (in MG) Xnet_inf: : net inflow for all storage infrastructure for utility X (in MGD) Xst_rof: short term ROF for utility X (values between 0 and 1) Xst_stor_rof: short-term storage ROF for utility X (values between 0 and 1) Xst_trmt_rof: short-term treatment ROF for utility X (values between 0 and 1) Xlt_rof: long-term ROF for utility X (values between 0 and 1) Xlt_stor_rof: long-term storage ROF for utility X (values between 0 and 1) Xlt_trmt_rof: long-term treatment ROF for utility X (values between 0 and 1) Xrest_demand: restricted demand for utility X (in MGD) Xunrest_demand: unrestricted demand for utility X (in MGD) Xunfulf_demand: unfulfilled demand for utility X (in MGD) Xwastewater: wastewater return for utility X (in MGD) Xtreat_capacity: total treatment capacity for utility X (in MG) Xcont_fund: reserve (contingency) fund balance for utility X Xins_pout: insurance payout for utility X (% annual volumetric revenue) Xins_price: insurance price for utility X (% annual volumetric revenue) Xinfra_npv: infrastructure net present value for utility ($mil) Xst_vol: total available storage volume of utility X (in MG) Xdebt_serv: debt service for utility X (usually once per year if the infrastructure is triggered; % annual volumetric revenue) Xstor_vol: total stored volume (in MGD) Xobs_ann_dem: observed annual demand for utility X (in MGD) Xproj_dem: projected annual demand for utility X (in MGD) Xpv_debt_serv: present value of debt service payments for utility X (% annual volumetric revenue) Xgross_rev: gross revenue for utility X ($mil) Acknowledgment IM3 is a multi-institutional effort led by Pacific Northwest National Laboratory and supported by the U.S. Department of Energy's Office of Science as part of research in MultiSector Dynamics, Earth and Environmental Systems Modeling Program.

Artificial Intelligence↗
DOE OSTI2019

Understanding data motion in the modern HPC data center

The utilization and performance of storage, compute, and network resources within HPC data centers have been studied extensively, but much less work has gone toward characterizing how these resources are used in conjunction to solve larger scientific challenges. To address this gap, we present our work in characterizing workloads and workflows at a data-center-wide level by examining all data transfers that occurred between storage, compute, and the external network at the National Energy Research Scientific Computing Center over a three-month period in 2019. Using a simple abstract representation of data transfers, we analyze over 100 million transfer logs from Darshan, HPSS user interfaces, and Globus to quantify the load on data paths between compute, storage, and the wide-area network based on transfer direction, user, transfer tool, source, destination, and time. We show that parallel I/O from user jobs, while undeniably important, is only one of several major I/O workloads that occurs throughout the execution of scientific workflows. We also show that this approach can be used to connect anomalous data traffic to specific users and file access patterns, and we construct time-resolved user transfer traces to demonstrate that one can systematically identify coupled data motion for individual workflows.

Lockwood, GK↗
DOE OSTI2025

A machine-learning-driven data labeling pipeline for scientific analysis in MLExchange

This study introduces a novel labeling pipeline to accelerate the labeling process of scientific data sets by using artificial intelligence (AI)-guided tagging techniques. This pipeline includes a set of interconnected web-based graphical user interfaces (GUIs), where Data Clinic and MLCoach enable the preparation of machine learning (ML) models for data reduction and classification, respectively, while Label Maker is used for label assignment. Throughout this pipeline, data can be accessed through a direct connection to a file system or through Tiled for access through Hypertext Transfer Protocol (HTTP). Our experimental results present three use cases where this labeling pipeline has been instrumental for the study of large X-ray scattering data sets in the area of pattern recognition, the remote analysis of resonant soft X-ray scattering data and the fine-tuning process of foundation models. These use cases highlight the labeling capabilities of this pipeline, including the ability to label large data sets in a short period of time, to perform remote data analysis while minimizing data movement and to enhance the fine-tuning process of complex ML models with human involvement.

Chavez, Tanny (ORCID:0000000193172896)↗
DOE OSTI2022

funcX: Federated Function as a Service for Science

Here, funcX is a distributed function as a service (FaaS) platform that enables flexible, scalable, and high performance remote function execution. Unlike centralized FaaS systems, funcX decouples the cloud-hosted management functionality from the edge-hosted execution functionality. funcX's endpoint software can be deployed, by users or administrators, on arbitrary laptops, clouds, clusters, and supercomputers, in effect turning them into function serving systems. funcX's cloud-hosted service provides a single location for registering, sharing, and managing both functions and endpoints. It allows for transparent, secure, and reliable function execution across the federated ecosystem of endpoints-enabling users to route functions to endpoints based on specific needs. funcX uses containers (e.g., Docker, Singularity, and Shifter) to provide common execution environments across endpoints. funcX implements various container management strategies to execute functions with high performance and efficiency on diverse funcX endpoints. funcX also integrates with an in-memory data store and Globus for managing data that may span endpoints. We motivate the need for funcX, present our prototype design and implementation, and demonstrate, via experiments on two supercomputers, that funcX can scale to more than 130000 concurrent workers. We show that funcX's container warming-aware routing algorithm can reduce the completion time for 3,000 functions by up to 61% compared to a randomized algorithm and the in-memory data store can speed up data transfers by up to 3x compared to a shared file system.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

AutoSIM

AutoSIM enables several instances of an executable to run in parallel on a directory of input files and consolidate results, at scale on the world's fastest supercomputers. The software accomplishes this by balancing compression, transmission, and computation of input files, executables, and output files to allow scalable parallel runs of any executable. The software breaks-up simulations into a configurable number of "churns" or sequential instances of executable invocations on a configurable number of cores/node. It sends the executable, supporting files, and appropriate number of input files to each node for independent execution, and then transfers the executable results to a shared disk for post-processing analysis. This MPI-based code has been demonstrated using EnergyPlus, DOE's flagship whole-building simulation engine, but was architected to allow scalable runtime of any software for any application (e.g. sensitivity analysis and uncertainty quantification). The code has also been demonstrated to scale on the Titan supercomputer using 131,072 processors to run 524,288 EnergyPlus annual simulations, and write 45TB of data to disk in 68 mins 08 secs.

New, Joshua R↗
DOE OSTI2024

Processing MCNP Elemental Edit Outputs

The Monte Carlo N-Particle (MCNP) transport code version 6 (also known as MCNP6) has the capability for tracking particles on unstructured mesh (UM) geometry models embedded into constructive solid geometry (CSG) cells. A UM geometry is a collection of elements representing a solid geometry. The first step of MCNP UM modeling is using other software packages to create a finite element mesh representation of a solid 3D geometry. Computer-aided design (CAD) or computer-aided manufacturing (CAM) software is typically used to create a solid geometry model, which is later imported into mesh generation software to create a UM model. The MCNP UM feature was originally designed for models generated by the Abaqus/CAE software. The MCNP code version 6.0 and later can process UM models formatted as Abaqus input files. MCNP can process a UM model consisting of several different element types including linear tetrahedral or hexahedral elements and calculate quantities of interest such as flux and energy deposition at elements. An MCNP UM simulation provides high-fidelity elemental edit (i.e., tally) outputs, which can be further used in multiphysics calculations. The MCNP UM feature was used for multiphysics simulations where quantities of interest calculated by MCNP are used as inputs for heat transfer calculations in Abaqus. MCNP6.3 can produce two types of elemental edit output (EEOUT) file formats: ASCII and HDF5. An EEOUT file type must be requested on an EMBED card while output type (flux or energy deposition) must be requested on an EMBEE card. We wrote Python3 scripts to extract energy deposition values in an ASCII or HDF5 EEOUT file and compute a heat flux profile for an Abaqus heat transfer calculation.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

A case study on parallel HDF5 dataset concatenation for high energy physics data analysis

In High Energy Physics (HEP), experimentalists generate large volumes of data that, when analyzed, helps us better understand the fundamental particles and their interactions. This data is often captured in many files of small size, creating a data management challenge for scientists. In order to better facilitate data management, transfer, and analysis on large scale platforms, it is advantageous to aggregate data further into a smaller number of larger files. However, this translation process can consume significant time and resources, and if performed incorrectly the resulting aggregated files can be inefficient for highly parallel access during analysis on large scale platforms. In this paper, we present our case study on parallel I/O strategies and HDF5 features for reducing data aggregation time, making effective use of compression, and ensuring efficient access to the resulting data during analysis at scale. We focus on NOvA detector data in this case study, a large-scale HEP experiment generating many terabytes of data. Here, the lessons learned from our case study inform the handling of similar datasets, thus expanding community knowledge related to this common data management task.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2016

Predicting Fueled Experiment Swelling and Thermal Expansion Using STAR-CCM+

An experiment being designed at Idaho National Laboratory (INL) has an unusual temperature requirement. The surface temperature must remain above 300 C over the irradiation lifetime. The fuel in this experiment is composed of uranium-nitride (UN) tri-structural-isotropic (TRISO) particles encased in a cylindrical, SiC matrix. Increased swelling of SiC at lower temperatures is the basis for this lower temperature limit. For a typical thermal analysis, the focus is on some upper limit of the center-line temperature. This tends to be a more straight-forward process where the analyst makes a series of worst-case assumptions and demonstrates that the experiment - even under those conditions - cannot exceed the maximum temperature. The model used to ensure the experiment will always have a surface temperature in excess of 300 C has been developed using STAR-CCM+. Notable properties of the simulation include - Full 3D fluid and heat transfer modeling; - Geometry input taken directly from design engineer's CAD files; - Fuel swelling and thermal conductivity as functions of displacements per atom (DPA) and temperature; - Radial thermal expansion of fuel, rodlet, and capsule; - Jump gap and surface roughness implemented using equations from BISON. These things together have yielded a model with sufficient detail to predict minimum surface temperatures with the greatest precision attainable with available data. The only notable approximation is the zirconia insulator, which will actually be a sphere. This particular shape was chosen to expedite meshing, while maintaining a realistic degree of physical contact with adjacent components. Given temperature and irradiation dependent data for thermal conductivity and fuel swelling of a SiC matrix, it was possible to perform an in-depth thermal analysis to find an optimal configuration of radial gas gaps. Of greatest importance was that the fuel surface temperature remain above 300 C. This criteria was met with a healthy margin, even when considering machining tolerances of ±1 mil on the gas gaps.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2026

Kinetic Model of HoxEFU reduction by NADH [SWR-26-087]

This repository is used to release code generated for manuscripts on the Photosynthetic Energy Transduction core program. This code simulates the reduction of HoxEFU by NADH. The electro transfer rate constants for the simulation are specified in the .csv files. The two .csv files correspond tot he two models described in Dawson et al. Cell. Rep. Phys. Sci. 2026. The code utilizes a chemical master equation, a set of differential equations, defining the time evolution of the oxidation and reduction kinetics of NAD+, NADH, a FMN flavin, and a set of iron sulfur clusters. The kinetics of HoxEFU reduction by NADH are evaluated by numerical integration of the chemical master equation using a variable-time-step Runge-Kutta algorithm.

Dahl, Peter [National Laboratory of the Rockies (N↗
DOE OSTI2014

MPI-Bash: Parallel scripting right from the Bourne-Again Shell (Bash)

Parallel, scientific applications running on massively parallel supercomputers commonly produce large numbers of large data files. While parallel filesystems improve the performance of file generation, postprocessing activities such as archiving and compressing the data are often run sequentially (and therefore slowly), squandering the supercomputer's vast performance. Consequently, data that take hours to generate may take days to postprocess. Writing and debugging parallel postprocessing programs is often too time-consuming to be worthwhile, especially for one-off postprocessing operations. What the community needs is a simple mechanism for cobbling together scripts that distribute postprocessing work across a supercomputer's many nodes. MPI-Bash attempts to supply that mechanism by augmenting the GNU project's command shell, Bash, with support for the Message Passing Interface (MPI) and, if available, the Libcircle library for work distribution and load balancing. MPI-Bash makes it easy to parallelize bash scripts that run a set of Linux commands independently over a large number of input files. Because MPI-Bash includes various MPI functions for data transfer and synchronization, it is not limited to embarrassingly parallel workloads but can incorporate phased operations (i.e., all workers must finish operation X before any worker is allowed to begin operation Y).

97 MATHEMATICS AND COMPUTING↗