Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data requests”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Accessing Data Federations with CVMFS

Data federations have become an increasingly common tool for large collaborations such as CMS and Atlas to efficiently distribute large data files. Unfortunately, these typically are implemented with weak namespace semantics and a non-POSIX API. On the other hand, CVMFS has provided a POSIX-compliant read-only interface for use cases with a small working set size (such as software distribution). The metadata required for the CVMFS POSIX interface is distributed through a caching hierarchy, allowing it to scale to the level of about a hundred thousand hosts. In this paper, we will describe our contributions to CVMFS that merges the data scalability of XRootD-based data federations (such as AAA) with metadata scalability and POSIX interface of CVMFS. We modified CVMFS so it can serve unmodified files without copying them to the repository server. CVMFS 2.2.0 is also able to redirect requests for data files to servers outside of the CVMFS content distribution network. Finally, we added the ability to manage authorization and authentication using security credentials such as X509 proxy certificates. We combined these modifications with the OSGs StashCache regional XRootD caching infrastructure to create a cached data distribution network. Here, we will show performance metrics accessing the data federation through CVMFS compared to direct data federation access. Additionally, we will discuss the improved user experience of providing access to a data federation through a POSIX filesystem.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Latency hiding for caches

A technique for accessing a memory having a high latency portion and a low latency portion is provided. The technique includes detecting a promotion trigger to promote data from the high latency portion to the low latency portion, in response to the promotion trigger, copying cache lines associated with the promotion trigger from the high latency portion to the low latency portion, and in response to a read request, providing data from either or both of the high latency portion or the low latency portion, based on a state associated with data in the high latency portion and the low latency portion.

97 MATHEMATICS AND COMPUTING↗

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific applications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application-level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher prediction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific ap- plications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Net- works (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate band- widths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer perfor- mance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application- level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher pre- diction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗

TAZeR: Hiding the Cost of Remote I/O in Distributed Scientific Workflows

A perennial bottleneck in distributed workflow analytics is long access latencies for remote data. We ask the question: assuming that data must be accessed remotely, can latencies be hidden? We present TAZeR, a framework that reduces data access latency while increasing data reuse. TAZeR transparently converts POSIX I/O into operations that interleave application work with data transfer, i.e. read, prefetching, and write stage-out. TAZeR ensures read data moves directly to application memory without synchronous intervention (soft zero-copy). TAZeR uses distributed bandwidth-aware staging to exploit reuse across application tasks and to manage the capacity constraints of fast hierarchical storage. We evaluate TAZeR on a High Energy Physics workflow that requests remote data at 48 Gb/s (over two 1 Gb/s WAN links) using complex access patterns. TAZeR is 12× and 22× faster than XRootD (state-of-the-art) and file copies (current approach), respectively; and within 7% of optimal. We discuss conditions when TAZeR can hide I/O accesses; and evaluate performance as effective staging sizes change.

Tazer, remote I/O, data staging↗

On Performance Prediction of Big Data Transfer in High-performance Networks

Big data generated by large-scale scientific and industrial applications need to be transferred between different geographical locations for remote storage, processing, and analysis. High-speed dedicated connections provisioned in High-performance Networks (HPNs) are increasingly utilized to carry out such big data transfer. HPN management highly relies on an important capability of performance (mainly throughput) prediction to reserve sufficient bandwidth and meanwhile avoid over-provisioning that may result in unnecessary resource waste. This capability is critical to improving the resource (mainly bandwidth) utilization of dedicated connections and meeting various user requests for data transfer. Conventional methods conduct performance prediction by fitting prior observed transfer history with predefined loss functions, without considering unobservable latent factors such as competing loads on end hosts. Such latent factors also have a significant impact on the application-level data transfer performance, which may result in an inaccurate prediction model. In this paper, we first investigate the impact of latent factors and propose a clustering-based method to eliminate their negative impact on performance prediction. We then develop a robust machine learning-based performance predictor by: i) incorporating the proposed latent factor elimination method into data preprocessing, and ii) adopting a customized domain guided loss function. Extensive experimental results show that our predictor achieves significantly higher prediction accuracy than several other state-of-the-art methods.

Liu, Wuji↗

Final Report for DOE Award ER25756

The SciDAC-funded Center for Enabling Distributed Petascale Science (CEDPS) was established to address technical challenges that arise due to the frequent geographic distribution of data producers (in particular, supercomputers and scientific instruments) and data consumers (people and computers) within the DOE laboratory system. Its goal is to produce technical innovations that meet DOE end-user needs for (a) rapid and dependable placement of large quantities of data within a distributed high-performance environment, and (b) the convenient construction of scalable science services that provide for the reliable and high-performance processing of computation and data analysis requests from many remote clients. The Center is also addressing (c) the important problem of troubleshooting these and other related ultra-high-performance distributed activities from the perspective of both performance and functionality

97 MATHEMATICS AND COMPUTING↗

Exploratory analysis and performance prediction of big data transfer in High-performance Networks

Big data transfer in large-scale scientific and business applications is increasingly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) via advance bandwidth reservation. Provisioning agents need to carefully schedule data transfer requests, compute network paths, and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized, could be simply wasted due to the exclusive access during the approved time window, and cause extra overhead and complexity for resource management. This calls for accurate performance prediction to reserve bandwidths that match actual needs and avoid over-provisioning. We employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements collected in the past several years from data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated testbeds. We first analyze the performance patterns in response to a comprehensive list of parameters in end-host systems, network connections, and data transfer applications, which motivate the use of machine learning and also help us identify the effects of latent factors. We then propose threshold- and clustering-based methods to eliminate negative effects of latent factors in data preprocessing and build a robust performance predictor based on customized domain-oriented loss functions. The performance of the proposed methods is verified by extensive experiments using SVR and RFR as well as theoretical analysis of the general performance bound.

97 MATHEMATICS AND COMPUTING↗

DayCent data and results for "Robust paths to net greenhouse gas mitigation and negative emissions via advanced biofuels"

DayCent data and results for:J. L. Field, T. L. Richard, E. A. Smithwick, H. Cai, M. S. Laser, D. S. LeBauer, S. P. Long, K. Paustian, Z. Qin, J. J. Sheehan, P. Smith, M. Q. Wang, L. R. Lynd, Robust paths to net greenhouse gas mitigation and negative emissions via advanced biofuels. Proceedings of the National Academy of Sciences (2020). https://doi.org/10.1073/pnas.1920877117 This zip file contains a UNIX-format DayCent model executable, input files, automation code, and associated directory structure necessary to re-produce the DayCent analysis underlying the manuscript. The main script 'autodaycent.py' (written for Python 2.7) opens an interactive command line routine that facilitates:* Calibrating the DayCent pine growth model.* Initializing DayCent for a set of case studies sites.* Executing an ensemble of model runs representing case study site reforestation, grassland restoration, or conversion to switchgrass cultivation. * Results analysis & generation of manuscript Fig. 3. Note that the interactive analysis code requires that all input files to be contained in the directory structure as uploaded, without modification. Executable versions of the DayCent model (https://www.nrel.colostate.edu/projects/daycent/) compatible with other operating systems are available upon request.Please send questions/comments to John.L.Field@gmail.com

Agro-ecosystem function and prediction↗

C4MIP – The Coupled Climate–Carbon Cycle Model Intercomparison Project: Experimental protocol for CMIP6

Coordinated experimental design and implementation has become a cornerstone of global climate modelling. Model Intercomparison Projects (MIPs) enable systematic and robust analysis of results across many models, by reducing the influence of ad hoc differences in model set-up or experimental boundary conditions. As it enters its 6th phase, the Coupled Model Intercomparison Project (CMIP6) has grown significantly in scope with the design and documentation of individual simulations delegated to individual climate science communities. The Coupled Climate–Carbon Cycle Model Intercomparison Project (C4MIP) takes responsibility for design, documentation, and analysis of carbon cycle feedbacks and interactions in climate simulations. These feedbacks are potentially large and play a leading-order contribution in determining the atmospheric composition in response to human emissions of CO 2 and in the setting of emissions targets to stabilize climate or avoid dangerous climate change. For over a decade, C4MIP has coordinated coupled climate–carbon cycle simulations, and in this paper we describe the C4MIP simulations that will be formally part of CMIP6. While the climate–carbon cycle community has created this experimental design, the simulations also fit within the wider CMIP activity, conform to some common standards including documentation and diagnostic requests, and are designed to complement the CMIP core experiments known as the Diagnostic, Evaluation and Characterization of Klima (DECK). C4MIP has three key strands of scientific motivation and the requested simulations are designed to satisfy their needs: (1) pre-industrial and historical simulations (formally part of the common set of CMIP6 experiments) to enable model evaluation, (2) idealized coupled and partially coupled simulations with 1 % per year increases in CO 2 to enable diagnosis of feedback strength and its components, (3) future scenario simulations to project how the Earth system will respond to anthropogenic activity over the 21st century and beyond. This study documents in detail these simulations, explains their rationale and planned analysis, and describes how to set up and run the simulations. Particular attention is paid to boundary conditions, input data, and requested output diagnostics. It is important that modelling groups participating in C4MIP adhere as closely as possible to this experimental design.

54 ENVIRONMENTAL SCIENCES↗

SU-D-BRD-07: Automatic Patient Data Audit and Plan Quality Check to Support ARIA and Eclipse

Purpose: To ensure patient safety and treatment quality in RT departments that use Varian ARIA and Eclipse, we developed a computer software system and interface functions that allow previously developed electron chart checking (EcCk) methodologies to support these Varian systems. Methods: ARIA and Eclipse store most patient information in its MSSQL database. We studied the contents in the hundreds database tables and identified the data elements used for patient treatment management and treatment planning. Interface functions were developed in both c-sharp and MATLAB to support data access from ARIA and Eclipse servers using SQL queries. These functions and additional data processing functions allowed the existing rules and logics from EcCk to support ARIA and Eclipse. Dose and structure information are important for plan quality check, however they are not stored in the MSSQL database but as files in Varian private formats, and cannot be processed by external programs. We have therefore implemented a service program, which uses the DB Daemon and File Daemon services on ARIA server to automatically and seamlessly retrieve dose and structure data as DICOM files. This service was designed to 1) consistently monitor the data access requests from EcCk programs, 2) translate the requests for ARIA daemon services to obtain dose and structure DICOM files, and 3) monitor the process and return the obtained DICOM files back to EcCk programs for plan quality check purposes. Results: EcCk, which was previously designed to only support MOSAIQ TMS and Pinnacle TPS, can now support Varian ARIA and Eclipse. The new EcCk software has been tested and worked well in physics new start plan check, IMRT plan integrity and plan quality checks. Conclusion: Methods and computer programs have been implemented to allow EcCk to support Varian ARIA and Eclipse systems. This project was supported by a research grant from Varian Medical System.

60 APPLIED LIFE SCIENCES↗

Code Release for “Unlocking Extreme Space Weather through Advanced Modeling of Legacy Vela Spacecraft” ER

The software being developed for this project has two mains aims. First, a Bayesian Model Calibration (BMC) procedure is being developed to calibrate a spallation model that simulates protons hitting a spacecraft orbiting earth to real data. The procedure will be developed for general data (there is no data release requested as part of this code release). Second, an inverse physics modeling task is being undertaken to map the number of resulting neutrons observed from this process to the expected number of protons that hit the model. This second task is of statistical interest; to publish on it, the code will need to be open source.

Murph, Alexander↗

Predicted Radiation Output from Several Kilograms of Plutonium Oxide

Neutron and gamma signatures from 3.44 kg of plutonium oxide were predicted with MCNP6 in anticipation of a passive diagnostic measurement campaign. Calculated radiation fields are visualized through use of “FMESH” type tallies. This work is effectively an extension of previous efforts and application of MISC for the photon source term is the main advancement. Large sources of modeling uncertainty include physical can dimensions and thus internal volume and density estimates. Material age is unknown and will also affect the passive gamma signature. The measurement campaign will provide information for model refinement and subsequent simulations. The authors have requested passive data collection on the pair of cans with the largest and smallest mass and such data will be valuable for benchmarking purposes.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Results for the First Quarter Calendar Year 2018 Tank 50 Salt Solution Sample

In this Technical Report, the chemical and radionuclide contaminant results from the First Quarter Calendar Year 2018 (CY18) sample of Tank 50 salt solution are presented in tabulated form. The information from this characterization will be used by Savannah River Remediation (SRR) for the transfer of aqueous waste from Tank 50 to the Saltstone Production Facility, where the waste will be treated and disposed of in the Saltstone Disposal Facility. This Technical Report compares results, where applicable, to Saltstone Waste Acceptance Criteria (WAC) limits and targets. The chemical and radionuclide contaminant results from the characterization of the First Quarter CY18 sampling of Tank 50 were requested by SRR personnel via a Task Technical Request (TTR) 2 and details of the testing are presented in the Savannah River National Laboratory (SRNL)Task Technical and Quality Assurance Plan (TT QAP). This Technical Report is part of Deliverable 2 relating to Task 1 from the SRR request. 2 Data pertaining to the regulatory limits for Resource Conservation and Recovery Act (RCRA) metals will be documented at a later time per the TTQAP for the Tank 50 Saltstone task.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Distributed metadata servers in a file system with separate metadata servers for file metadata and directory metadata

A file system is provided having a plurality of distributed metadata servers with separate metadata servers for file metadata and directory metadata. An exemplary distributed metadata storage system comprises a plurality of compute nodes, wherein at least one compute node comprises a file system client that generates a directory metadata request and/or a file metadata request; a plurality of directory metadata servers to serve directory metadata in response to a given directory metadata request; and a plurality of file metadata servers to serve file metadata in response to a given file metadata request. The plurality of directory metadata servers are distinct from the plurality of file metadata servers. The compute nodes optionally comprise an object client that requests object data from an object store system. The number of directory metadata servers and the number of file metadata servers are optionally determined independently to balance for different workloads, and can be dynamically readjusted for evolving workloads.

97 MATHEMATICS AND COMPUTING↗

Trajectory analysis with geometric features

Described herein are various technologies pertaining to extracting one or more features from trajectory data recorded during motion of a body, and further, generating a n-dimensional feature vector based upon the one or more extracted features. The n-dimensional feature vector enables expedited analysis of the trajectory data from which the feature vector was generated. For example, rather than having to analyze a trajectory curve comprising a large number of time-position data points, the n-dimensional feature vector can be compared with one or more search parameters to facilitate clustering of the trajectory data associated with the n-dimensional feature vector with other trajectory data which also satisfies the search request. The trajectory data can be plotted on a screen in combination with the n-dimensional feature vector, and other pertinent information. The trajectory data, etc., can be displayed using heat maps or other graphical representation.

Wilson, Andrew T.↗

Trajectory prediction via a feature vector approach

Described herein are various technologies pertaining to extracting one or more features from trajectory data recorded during motion of a body, and further, generating a n-dimensional feature vector based upon the one or more extracted features. The n-dimensional feature vector enables expedited analysis of the trajectory data from which the feature vector was generated. For example, rather than having to analyze a trajectory curve comprising a large number of time-position data points, the n-dimensional feature vector can be compared with one or more search parameters to facilitate clustering of the trajectory data associated with the n-dimensional feature vector with other trajectory data which also satisfies the search request. The trajectory data can be plotted on a screen in combination with the n-dimensional feature vector, and other pertinent information. The trajectory data, etc., can be displayed using heat maps or other graphical representation.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Silent store detection and recording in memory storage

An aspect includes receiving a write request that includes a memory address and write data. Stored data is read from a memory location at the memory address. Based on determining that the memory location was not previously modified, the stored data is compared to the write data. Based on the stored data matching the write data, the write request is completed without writing the write data to the memory and a corresponding silent store bit, in a silent store bitmap is set. Based on the stored data not matching the write data, the write data is written to the memory location, the silent store bit is reset and a corresponding modified bit is set. At least one of an application and an operating system is provided access to the silent store bitmap.

97 MATHEMATICS AND COMPUTING↗