Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “File transfer”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
DOE OSTI2015

AsyncStageOut: Distributed User Data Management for CMS Analysis

AsyncStageOut (ASO) is a new component of the distributed data analysis system of CMS, CRAB, designed for managing users' data. It addresses a major weakness of the previous model, namely that mass storage of output data was part of the job execution resulting in inefficient use of job slots and an unacceptable failure rate at the end of the jobs. ASO foresees the management of up to 400k files per day of various sizes, spread worldwide across more than 60 sites. It must handle up to 1000 individual users per month, and work with minimal delay. This creates challenging requirements for system scalability, performance and monitoring. ASO uses FTS to schedule and execute the transfers between the storage elements of the source and destination sites. It has evolved from a limited prototype to a highly adaptable service, which manages and monitors the user file placement and bookkeeping. To ensure system scalability and data monitoring, it employs new technologies such as a NoSQL database and re-uses existing components of PhEDEx and the FTS Dashboard. We present the asynchronous stage-out strategy and the architecture of the solution we implemented to deal with those issues and challenges. The deployment model for the high availability and scalability of the service is discussed. The performance of the system during the commissioning and the first phase of production are also shown, along with results from simulations designed to explore the limits of scalability.

Riahi, H.↗
DOE OSTI2016

A Bloom Filter Based Scalable Data Integrity Check Tool for Large-Scale Dataset

Large scale HPC applications are becoming increasingly data intensive. At Oak Ridge Leadership Computing Facility (OLCF), we are observing the number of files curated under individual project are reaching as high as 200 millions and project data size is exceeding petabytes. These simulation datasets, once validated, often needs to be transferred to archival system for long term storage or shared with the rest of the research community. Ensuring the data integrity of the full dataset at this scale is paramount important but also a daunting task. This is especially true considering that most conventional tools are serial and file-based, unwieldy to use and/or can't scale to meet user's demand.To tackle this particular challenge, this paper presents the design, implementation and evaluation of a scalable parallel checksumming tool, fsum, which we developed at OLCF. It is built upon the principle of parallel tree walk and work-stealing pattern to maximize parallelism and is capable of generating a single, consistent signature for the entire dataset at extreme scale. We also applied a novel bloom-filter based technique in aggregating signatures to overcome the signature ordering requirement. Given the probabilistic nature of bloom filter, we provided a detailed error and trade-off analysis. Using multiple datasets from production environment, we demonstrated that our tool can efficiently handle both very large files as well as many small-file based datasets. Our preliminary test showed that on the same hardware, it outperforms conventional tool by as much as 4×. It also exhibited near-linear scaling properties when provisioned with more compute resources.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2014

The Shell-Model Code NuShellX@MSU

Use of the code NuShellX@MSU is outlined. It connects to the ENSDF data files for automatic comparisons to energy level data. Operator overlaps provide predictions for spectroscopic factors, two-nucleon transfer amplitudes, nuclear moments, gamma decay and beta decay.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Dataset for "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models" Willard et al. (2024)

This data release provides all data and code used in the paper " "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models" Willard et al. (2024)" to model stream temperature, evaluate, and assess results. The associated manuscript explores current open questions in prediction in ungauged and unmonitored basins concerning top-down versus bottom-up approaches, tradeoffs between data available and input requirements, and the appropriate representation of catchment attributes as inputs to deep learning models. Modeling was done primarily with long short-term memory (LSTM) models, and stream site coverage spans 1362 locations across the conterminous United States. The data is organized into these items items:Code repository and data for the paper " "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models" Willard et al. (2024)".Code: stream_temp_ml_regionalization.zip contains the code repositoryData to run the code: - data_dir.zip -- contains all files that should be moved to the "DATA_DIR" variable defined in the "set_env_vars.sh" script in the code repository- metadata_dir.zip -- contains all files that should be moved to the "METADATA_DIR" variable defined in the "set_env_vars.sh" script in the code repository- error_analysis_attribute_and_groundwater_dir.zip - workflows for the extended error analysis by stream attribute and groundwater influenceData produced by the code and used in the paper:- outputs_dir.zip - contains model output and results (outputs_dir/results), model weights (outputs_dir/models), and all other outputs used for the paper including feature importances.To cite this code, please use the following BibTeX or MLA entries:bibtex:@misc{willard2024streamdata, author = {Jared Willard and Fabio Ciulla and Helen Weierbach and Vipin Kumar and Charuleka Varadharajan}, title = {Dataset for "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models"}, year = {2024}, doi = {10.15485/2448016}, publisher = {ESS-DIVE Repository}, url = {https://doi.org/10.15485/2448016}}MLA: Willard, Jared, et al. Dataset for "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models". 2024. ESS-DIVE Repository, doi:10.15485/2448016.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Device-Centric Ransomware Detection using Machine Learning-Based Memory Forensics for Smart Inverters

Ransomware attacks are the fastest-growing form of cyberattacks worldwide. Recently, ransomware attacks have targeted industrial control systems (ICSs), including power grids. Lessons learned from recent incidents in ICSs show that ransomware groups can deliver ransomware into not only the organization’s control servers, but also the operational technology (OT) devices such as smart inverters and smart grid devices. This paper proposes a machine learning (ML)- based memory forensics method enabling the detection of ransomware binaries stored in the memory of a commercial smart inverter. Device firmware binary files are extracted from a Serial Peripheral Interface (SPI) flash memory, and samples of both benign and ransomware binaries are generated by a binary manipulation method and a real-world ransomware encryption, separately. A deep transfer learning (DTL) method is used to retrain a convolutional neural network (CNN)-based ransomware detection algorithm using the generated samples. The experimental result validates that the proposed ML-based memory forensics method can accurately detect ransomware files.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

THM: the MOOSE thermal hydraulics module

The MOOSE Thermal Hydraulics Module (THM) is designed to facilitate the development of thermal hydraulic system models. It provides the capability to assemble networks of coupled components such as pipes, junctions, valves, turbomachinery, and heat exchangers. Its library of components supports a single-phase, compressible flow model based on a variable-area formulation of the Euler equations of gas dynamics and discretized using a finite volume scheme. THM offers a flexible system for specifying closures such as friction factors or heat transfer coefficients, allowing the user to choose from built-in correlations or define their own in the input file. A control logic system can be used to control input parameters, necessary for implementing transient scenarios and mirroring real control systems in thermal hydraulic systems. THM can be coupled with other MOOSE-based applications for multiphysics calculations. This talk will give an introduction to the capabilities of THM and provide some examples of its usage and validation.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2025

MOOSE Thermal-Hydraulics Module - MOOSE workshop

The MOOSE Thermal Hydraulics Module (THM) is designed to facilitate the development of thermal hydraulic system models. It provides the capability to assemble networks of coupled components such as pipes, junctions, valves, turbomachinery, and heat exchangers. Its library of components supports a single-phase, compressible flow model based on a variable-area formulation of the Euler equations of gas dynamics and discretized using a finite volume scheme. THM offers a flexible system for specifying closures such as friction factors or heat transfer coefficients, allowing the user to choose from built-in correlations or define their own in the input file. A control logic system can be used to control input parameters, necessary for implementing transient scenarios and mirroring real control systems in thermal hydraulic systems. THM can be coupled with other MOOSE-based applications for multiphysics calculations. This training will give an introduction to the capabilities of THM and provide some examples of its usage and validation.

97 - MATHEMATICS AND COMPUTING↗
DOE OSTI2013

GEO2D - Two-Dimensional Computer Model of a Ground Source Heat Pump System

This file contains a zipped file that contains many files required to run GEO2D. GEO2D is a computer code for simulating ground source heat pump (GSHP) systems in two-dimensions. GEO2D performs a detailed finite difference simulation of the heat transfer occurring within the working fluid, the tube wall, the grout, and the ground. Both horizontal and vertical wells can be simulated with this program, but it should be noted that the vertical wall is modeled as a single tube. This program also models the heat pump in conjunction with the heat transfer occurring. GEO2D simulates the heat pump and ground loop as a system. Many results are produced by GEO2D as a function of time and position, such as heat transfer rates, temperatures and heat pump performance. On top of this information from an economic comparison between the geothermal system simulated and a comparable air heat pump systems or a comparable gas, oil or propane heating systems with a vapor compression air conditioner. The version of GEO2D in the attached file has been coupled to the DOE heating and cooling load software called ENERGYPLUS. This is a great convenience for the user because heating and cooling loads are an input to GEO2D. GEO2D is a user friendly program that uses a graphical user interface for inputs and outputs. These make entering data simple and they produce many plotted results that are easy to understand. In order to run GEO2D access to MATLAB is required. If this program is not available on your computer you can download the program MCRInstaller.exe, the 64 bit version, from the MATLAB website or from the Geothermal Data Repository. This is a free download which will enable you to run GEO2D.

15 GEOTHERMAL ENERGY↗
DOE OSTI2020

Recent evolutions in the LTDP CDF project

In the latest years, CNAF (the national center of the Italian Institute for Nuclear Physics INFN dedicated to Research and Development on Information and Communication Technologies) has been working on the Long Term Data Preservation (LTDP) project for the CDF experiment, active at Fermilab from 1990 to 2011. The main aims of the project are to protect the most relevant part of the CDF RUN-2 data collected between 2001 and 2011 and already stored on tape at CNAF (4 PB), as well as to ensure the availability and the access to the analysis facility to those data over time. Lately, the CDF database, hosting information about CDF datasets such as their structure, file locations and metadata, has been imported from Fermilab to CNAF. Also, the Sequential Access via Metadata (SAM) station data handling tool for CDF data management, that allows to manage data transfers and to retrieve information from the CDF database, has been properly installed and configured at CNAF. This was a fundamental step in the perspective of a complete decommissioning of CDF services on Fermilab side. An access system has been designed and tested to submit CDF analysis jobs, using CDF software distributed via CERN Virtual Machine File System (CVMFS) and requesting delivery of CDF files stored on CNAF tapes, as well as data present only on Fermilab storage archive. Moreover, the availability and the correctness of all CDF data stored on CNAF tapes has been verified. This paper describes all these recent evolutions in detail, presenting the future plans for the LTDP project at CNAF.

Fornari, Federico↗
DOE OSTI2025

HPC Campaign Management: Remote data access with user-defined error bound using ADIOS and ZFP

Remote access to large-scale scientific datasets, like those generated by combustion simulations or other high-performance computing (HPC) applications, presents a significant challenge. Downloading entire datasets is often impractical due to their size and the bandwidth limitations of typical networks. To address this challenge, we propose a novel approach that enables efficient remote access to large datasets distributed across multiple facilities. Our method enables technologies to download only the data values of a select variable, in a select region of interest, to a user-defined accuracy. For this purpose, we extended the ADIOS IO library to provide read functions with user-defined accuracy, a remote data server that understands multidimensional selections of specific variables, steps and accuracy from an ADIOS dataset, and which uses lossy compression on the remote site to reduce the data to be transferred back to the client. In addition, our extension of the ADIOS library collects metadata from multiple datasets in small files called Campaign Archives, which can be shared among project participants on any HPC, cloud or laptop, and which can easily facilitate the discovery of content and pointers to the data location as well as remote access to the data by local tools as if data was local. This feature called Campaign Management, enables a group of scientists to manage related datasets stored in multiple files, across multiple facilities as if it was in a single file/database. We demonstrate the effectiveness of our approach using a 1.5 TB dataset from the S3D combustion simulation on Frontier at the Oak Ridge Leadership Facility. Even a single variable from this dataset, at 64 GB, is too large to be processed on a standard laptop. We show two different reading patterns for 2D plots and 3D visualization, with careful settings that a scientist studying combustion data would do and show that running the same Python scripts on Frontier directly takes comparable time than running them on the local laptop with remote access to the data on Frontier.

Podhorszki, Norbert [ORNL] (ORCID:000000019647542X↗
DOE OSTI2015

Cluster-to-cluster data transfer with data compression over wide-area networks

The recent emergence of ultra high-speed networks up to 100 Gb/s has posed numerous challenges and has led to many investigations on efficient protocols to saturate 100 Gb/s links. However, end-to-end data transfers involve many components, not only protocols, affecting overall transfer performance. These components include disk I/O subsystem, additional computation associated with data streams, and network adapters. For example, achievable bandwidth by TCP may not be implementable if disk I/O or CPU becomes a bottleneck in end-to-end data transfer. In this paper, we first model all the system components involved in end-to-end data transfer as a graph. We then formulate the problem whose goal is to achieve maximum data transfer throughput using parallel data flows. We also propose a variable data flow GridFFP XIO stack to improve data transfer with data compression. Our contributions lie in how to optimize data transfers considering all the system components involved rather than in accurately modeling all the system components involved. Our proposed formulations and solutions are evaluated through experiments on the ESnet 100G testbed and a wide-area cluster-to-cluster testbed. The experimental results on the ESnet 100G testbed show that our approach is several times faster than Globus Online-8 x faster for datasets with many 10 MB files and 3-4 x faster for other datasets of larger size files. The experimental results on the cluster-to-cluster testbed show that our variable data flow approach is up to 4 x faster than a normal cluster data transfer.

Disk-to-Disk Transfer↗
DOE OSTI2022

StaNdaRT: a repository of standardised test models and outputs for supernova radiative transfer

We present the first results of a comprehensive supernova (SN) radiative-transfer (RT) code-comparison initiative (StaNdaRT), where the emission from the same set of standardised test models is simulated by currently used RT codes. We ran a total of ten codes on a set of four benchmark ejecta models of Type Ia SNe. We consider two sub-Chandrasekhar-mass (M tot = 1.0 M ⊙ ) toy models with analytic density and composition profiles and two Chandrasekhar-mass delayed-detonation models that are outcomes of hydrodynamical simulations. We adopt spherical symmetry for all four models. The results of the different codes, including the light curves, spectra, and the evolution of several physical properties as a function of radius and time are provided in electronic form in a standard format via a public repository. We also include the detailed test model profiles and several Python scripts for accessing and presenting the input and output files. We also provide the code used to generate the toy models studied here. In this paper, we describe the test models, radiative-transfer codes, and output formats in detail, and provide access to the repository. We present example results of several key diagnostic features.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2022

HPXML Version Translator

HPXML is a consensus data transfer standard for residential buildings. Over several years, multiple versions of the standard have been released. This tool accepts an HPXML file in an older version and translates it to a newer version.

Merket, Noel↗
DOE OSTI2015

SU-E-T-138: Automated Chart Review Module Including Cross-Vendor Data Transfer Verification Developed for IHE-RO Initiative

Purpose: To follow the Integrating Healthcare Enterprise - Radiation Oncology (IHE-RO) initiative of proper cross-vendor technology integration, an automated chart checker (ACC) was developed. ACC compares extracted data from an approved patient plan in the Eclipse treatment planning system (TPS) against data existing in the Mosaiq treatment management system (TMS). ACC automatically analyzes these parameters using built-in quality checklists to provide further aid in chart review. Methods: Eclipse TPS data are obtained using Eclipse scripting API (ESAPI) while Mosaiq TMS data are obtained from a radiotherapy-treatment-planning (RTP) file. Using this information, ACC identifies TPS-TMS discrepancies in 18 primary beam parameters including MU, energy, jaw positions, gantry angle, table angle, accessories, and bolus for up to 31 beams. Next, approximately 40 items from traditional quality checklists are evaluated such as prescription consistency, DRR graticule placement, plan approval status, global max dose, and dose tracking coefficients. Parameters were artificially modified to determine if ACC would detect an error in data transfer and to test each component of quality checklists. Results: Using ESAPI scripting and RTP file-processing, ACC was able to properly aggregate data from TPS and TMS for up to 31 beams. Errors were artificially introduced into each plan parameter, and ACC was able to successfully detect all of them within seconds. Next, ACC was able to successfully detect mistakes in the chart by identifying deviations with its quality checklists, within seconds. Conclusion: ACC effectively addresses the potential issue of faulty cross-vendor data transfer, as described by IHE-RO. In addition, ACC was also able to detect deviations from its built-in quality checklists. ACC is already an invaluable tool for efficient and standardized chart review and will continue to improve as its incorporated checklists become more comprehensive.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2016

A new generation of x-ray spectrometry UHV instruments at the SR facilities BESSY II, ELETTRA and SOLEIL

A novel type of ultra-high vacuum instrument for X-ray reflectometry and spectrometry-related techniques for nanoanalytics by means of synchrotron radiation (SR) has been constructed and commissioned at BESSY II. This versa-tile instrument was developed by the PTB, Germany’s national metrology institute, and includes a 9-axis manipulator that allows for an independent alignment of the samples with respect to all degrees of freedom. In addition, it integrates a rotational and translational movement of several photodiodes as well as a translational movement of a beam-geometry-defining aperture system. Thus, the new instrument enables various analytical techniques based on energy dispersive X-ray detectors such as reference-free X-Ray Fluorescence (XRF) analysis, total-reflection XRF, grazing-incidence XRF, in addition to optional X-Ray Reflectometry (XRR) measurements or polarization-dependent X-ray absorption fine structure analyses (XAFS). Samples having a size of up to (100 × 100) mm{sup 2}; can be analyzed with respect to their mass deposition, elemental, spatial or species composition. Surface contamination, nanolayer composition and thickness, depth pro-file of matrix elements or implants, nanoparticles or buried interfaces as well as molecular orientation of bonds can be accessed. Three technology transfer projects of adapted instruments have enhanced X-Ray Spectrometry (XRS) research activities within Europe at the synchrotron radiation facilities ELETTRA (IAEA) and SOLEIL (CEA/LNE-LNHB) as well as at the X-ray innovation laboratory BLiX (TU Berlin) where different laboratory sources are used. Here, smaller chamber requirements led PTB in cooperation with TU Berlin to develop a modified instrument equipped with a 7-axis manipulator: reduced freedom in the choice of experimental geometry modifications (absence of out-of-SR-plane and reference-free XRS options) has been compensated by encoder-enhanced angular accuracy for GIXRF and XRR.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2024

GeoCricket

SAND2025-12229O Geospatial Critical Infrastructure and Census Data Stockpile Tool (GeoCricket) is a set of functions that collect critical infrastructure and census data for use in the Resilient Node Cluster Analysis Tool (ReNCAT) and Quantum Geographic Information System Social Burden Calculator. It can also act to inform other place-based work. The code queries public-facing Representational State Transfer (REST) servers to collect geospatial data related to a specific area. It then exports that data as standard geographic information system file types or as a .csv file. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Haines, John↗
DOE OSTI2018

Erratum: Polarization transfer observables in elastic electron-proton scattering at Q 2 =2.5 , 5.2, 6.8, and 8.5GeV 2 [Phys. Rev. C 96 , 055203 (2017)]

Subsequent to the release of our original paper, we discovered in the context of preparing our technical supplement [1] for journal publication that a typographical error had existed in the text file that the analysis program used to construct the beam polarization "database" for both the original analysis, published in Ref. [2], and our final analysis. The electron-beam polarization P e and the analyzing power A y cancel exactly in the ratio R, which is proportional to the ratio P t / P l of the transferred polarization components. On the other hand, the extraction of the relative e dependence of P l / P Born l relies on knowledge of the beam polarization. As such, data taking was interrupted roughly every two days during the GEp - 2γ experiment to perform invasive measurements of the beam polarization using the Hall C Moller polarimeter [3].

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2014

HPC s Pivot to Data

Computer centers such as NERSC and OLCF have traditionally focused on delivering computational capability that enables breakthrough innovation in a wide range of science domains. Accessing that computational power has required services and tools to move the data from input and output to computation and storage. A ''pivot to data'' is occurring in HPC. Data transfer tools and services that were previously peripheral are becoming integral to scientific workflows. Emerging requirements from high-bandwidth detectors, high-throughput screening techniques, highly concur- rent simulations, increased focus on uncertainty quantification, and an emerging open-data policy posture toward published research are among the data-drivers shaping the networks, file systems, databases, and overall compute and data environment. In this paper we explain the pivot to data in HPC through user requirements and the changing resources provided by HPC with particular focus on data movement. For WAN data transfers we present the results of a study of network performance between centers.

97 MATHEMATICS AND COMPUTING↗