Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,585 records · Page 88
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Demystifying asynchronous I/O Interference in HPC applications

With increasing complexity of HPC workflows, data management services need to perform expensive I/O operations asynchronously in the background, aiming to overlap the I/O with the application runtime. However, this may cause interference due to competition for resources: CPU, memory/network bandwidth. The advent of multi-core architectures has exacerbated this problem, as many I/O operations are issued concurrently, thereby competing not only with the application but also among themselves. Furthermore, the interference patterns can dynamically change as a response to variations in application behavior and I/O subsystems (e.g. multiple users sharing a parallel file system). Without a thorough understanding, I/O operations may perform suboptimally, potentially even worse than in the blocking case. To fill this gap, here we investigate the causes and consequences of interference due to asynchronous I/O on HPC systems. Specifically, we focus on multi-core CPUs and memory bandwidth, isolating the interference due to each resource. Then, we perform an in-depth study to explain the interplay and contention in a variety of resource sharing scenarios such as varying priority and number of background I/O threads and different I/O strategies: sendfile, read/write, mmap/write underlining trade-offs. The insights from this study are important both to enable guided optimizations of existing background I/O, as well as to open new opportunities to design advanced asynchronous I/O strategies.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

High-performance web services for querying gene and variant annotation

Efficient tools for data management and integration are essential for many aspects of high-throughput biology. In particular, annotations of genes and human genetic variants are commonly used but highly fragmented across many resources. Here, we describe MyGene.info and MyVariant.info, high-performance web services for querying gene and variant annotation information. These web services are currently accessed more than three million times permonth. They also demonstrate a generalizable cloud-based model for organizing and querying biological annotation information. MyGene.info and MyVariant.info are provided as high-performance web services, accessible at http://mygene.info and http://myvariant.info. Both are offered free of charge to the research community.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Ten simple rules for getting and giving credit for data

This article attempts to summarize current best practices that support the movement towards enabling researchers to cite and receive credit for their data. The authors are a small representation of the people and organizations trying to make this happen, and we acknowledge that it is not possible to capture all efforts behind this endeavor in 10 Simple Rules. We encourage interested readers to dive deeper by providing related resources along the way.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Towards scalable dataframe systems

Dataframes are a popular abstraction to represent, prepare, and analyze data. Despite the remarkable success of dataframe libraries in R and Python, dataframes face performance issues even on moderately large datasets. Moreover, there is significant ambiguity regarding dataframe semantics. In this paper we lay out a vision and roadmap for scalable dataframe systems. To demonstrate the potential in this area, we report on our experience building Modin, a scaled-up implementation of the most widely-used and complex dataframe API today, Python's pandas. With pandas as a reference, we propose a simple data model and algebra for dataframes to ground discussion in the field. Given this foundation, we lay out an agenda of open research opportunities where the distinct features of dataframes will require extending the state of the art in many dimensions of data management. We discuss the implications of signature dataframe features including flexible schemas, ordering, row/column equivalence, and data/metadata fluidity, as well as the piecemeal, trial-and-error-based approach to interacting with dataframes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Hindsight logging for model training

In modern Machine Learning, model training is an iterative, experimental process that can consume enormous computation resources and developer time. To aid in that process, experienced model developers log and visualize program variables during training runs. Exhaustive logging of all variables is infeasible, so developers are left to choose between slowing down training via extensive conservative logging, or letting training run fast via minimalist optimistic logging that may omit key information. As a compromise, optimistic logging can be accompanied by program checkpoints; this allows developers to add log statements post-hoc, and "replay" desired log statements from checkpoint---a process we refer to as hindsight logging. Unfortunately, hindsight logging raises tricky problems in data management and software engineering. Done poorly, hindsight logging can waste resources and generate technical debt embodied in multiple variants of training code. In this paper, we present methodologies for efficient and effective logging practices for model training, with a focus on techniques for hindsight logging. Our goal is for experienced model developers to learn and adopt these practices. To make this easier, we provide an open-source suite of tools for Fast Low-Overhead Recovery (flor) that embodies our design across three tasks: (i) efficient background logging in Python, (ii) adaptive periodic checkpointing, and (iii) an instrumentation library that codifies hindsight logging for efficient and automatic record-replay of model-training. Model developers can use each flor tool separately as they see fit, or they can use flor in hands-free mode, entrusting it to instrument their code end-to-end for efficient record-replay. Our solutions leverage techniques from physiological transaction logs and recovery in database systems. Evaluations on modern ML benchmarks demonstrate that flor can produce fast checkpointing with small user-specifiable overheads (e.g. 7%), and still provide hindsight log replay times orders of magnitude faster than restarting training from scratch.

Computer Science↗
DOE OSTI2022

ESS-DIVE Reporting Format for Dataset Package Metadata

ESS-DIVE’s (Environmental Systems Science Data Infrastructure for a Virtual Ecosystem) dataset metadata reporting format is intended to compile information about a dataset (e.g., title, description, funding sources) that can enable reuse of data submitted to the ESS-DIVE data repository. The files contained in this dataset include instructions (dataset_metadata_guide.md and README.md) that can be used to understand the types of metadata ESS-DIVE collects. The data dictionary (dd.csv) follows ESS-DIVE’s file-level metadata reporting format and includes brief descriptions about each element of the dataset metadata reporting format. This dataset also includes a terminology crosswalk (dataset_metadata_crosswalk.csv) that shows how ESS-DIVE’s metadata reporting format maps onto other existing metadata standards and reporting formats.Data contributors to ESS-DIVE can provide this metadata by manual entry using a web form or programmatically via ESS-DIVE’s API (Application Programming Interface). A metadata template (dataset_metadata_template.docx or dataset_metadata_template.pdf) can be used to collaboratively compile metadata before providing it to ESS-DIVE.Since being incorporated into ESS-DIVE’s data submission user interface, ESS-DIVE’s dataset metadata reporting format, has enabled features like automated metadata quality checks, and dissemination of ESS-DIVE datasets onto other data platforms including Google Dataset Search and DataCite.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Data for publication: "A fresh take: Seasonal changes in terrestrial freshwater inputs impact salt marsh hydrology and vegetation dynamics"

This data repository contains data associated with the manuscript "A fresh take: Seasonal changes in terrestrial freshwater inputs impact salt marsh hydrology and vegetation dynamics". This study was conducted at the Elkhorn Slough National Estuarine Research Reserve in Watsonville, California from October 2019 - June 2022. We sought to understand the role of shallow freshwater inputs from adjacent uplands on salt marsh hydrologic behavior and vegetation productivity. This dataset contains CSV files of the following: daily salt marsh subsurface water level and pore water conductivity, monthly vegetation survey measurements, soil core data. Estuary surface water level, conductivity, and local precipitation were downloaded from the National Estuarine Research Reserve System (Centralized Data Management Office) at https://cdmo.baruch.sc.edu/.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

IDAES-PSE 2.6.0 Release

The Institute for the Design of Advanced Energy Systems (IDAES) Integrated Platform is a versatile computational environment offering extensive process systems engineering (PSE) capabilities for optimizing the design and operation of complex, interacting technologies and systems. IDAES enables users to efficiently search vast, complex design spaces to discover the lowest cost solutions while supporting the full process modeling lifecycle, from conceptual design to dynamic optimization and control. The extensible, open platform empowers users to create models of novel processes and rapidly develop custom analyses, workflows, and end-user applications. IDAES-PSE 2.6.0 Release Highlights Upcoming Changes IDAES will be switching to the new Pyomo solver interface in the next release. Whilst this will hopefully be a smooth transition for most users, there are a few important changes to be aware of. The new solver interface uses a different version of the IPOPT writer (“ipopt_v2”) and thus any custom configuration options you might have set for IPOPT will not carry over and will need to be reset. By default, the new Pyomo linear presolver will be activated with ipopt_v2. Whilst are working to identify any bugs in the presolver, it is possible that some edge cases will remain. IDAES will begin deploying a new set of scaling tools and APIs over the next few releases that make use of the new solver writers. The old scaling tools and APIs will remain for backward compatibility but will begin to be deprecated. New Models, Tools and Features New Intersphinx extension automatically linking Jupyter notebook examples to project documentation New end-to-end diagnostics example demonstrated on a real problem New complementarity formulation for VLE with cubic equations of state, backward compatibility for old formulation New solver interface with presolve (ipopt_v2) in support of upcoming changes to the initialization and APIs methods, with default set to ipopt to maintain backwards compatibility; this will deprecate once all examples have been updated New forecaster and parameterized bidder methods within grid integration library Updated surrogates API and examples to support Keras 3, with backwards compatibility for older formats such as TensorFlow SavedModel (TFSM) Updated costing base dictionary to include the 2023 cost year index value Updated ProcessBlock to include information on the constructing block class Updated Flowsheet Visualizer to allow visualize() method to return value and functions Bug Fixes Fixed bug in the Modular Property Framework that would cause errors when trying to use phase-based material balances with phase equilibria. Fixed bug in Modular Properties Framework that caused errors when initializing models with non-vapor-liquid phase equilibria. Fixed typos flagged by June update to crate-ci/typos and removed DMF-related exceptions Minor corrections of units of measurement handling in power plant waste/transport costing expressions, control volume material holdup expressions, and BTX property package parameters Fixed throwing >7500 numpy deprecation warnings by replacing scalar value assignment with element extraction and item iteration calls Testing and Robustness Migrated slow tests (>10s) to integration, impacting test coverage but also yielding a nearly 30% decrease in local test runtime Pinned pint to avoid issues with older supported Python versions Pinned codecov versions to avoid tokenless upload behavior with latest version Bumped extensions to version 3.4.2 to allow pointing to non-standard install location Deprecations and Removals Python 3.8 is no longer supported. The supported Python versions are 3.9 through 3.12 The Data Management Framework (DMF) is no longer supported. Importing idaes.core.dmf will cause a deprecation warning to be displayed until the next release The SOFC Keras surrogates have been removed. The current version of the SOFC surrogate model in the examples repository is a PySMO Kriging model.

AS↗
DOE OSTI2024

Conclusions and outlook

In this chapter, we provide notes on the state of metadata collection and organization practices and reflect on the achievements of the MetaCO task group. We also include an utility perspective on how to use this report in daily data management practice and a short guide is provided to show how metadata practices can be initiated. The chapter, and the report, is concluded within an outlook, sketching future progress and opportunities on the horizon.

Aguado, Daniel↗
DOE OSTI2017

SDAV Institute Final report

Final report of the Scalable Data Management, Analysis and Visualization Institute

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2016

Energy Policy Case Study - California: Renewables and Distributed Energy Resources

The purpose of this document is to present a case study of energy policies in California related to power system transformation and renewable and distributed energy resources (DERs). Distributed energy resources represent a broad range of technologies that can significantly impact how much, and when, electricity is demanded from the grid. Key policies and proceedings related to power system transformation and DERs are grouped into the following categories: 1.Policies that support achieving environmental and climate goals 2.Policies that promote deployment of DERs 3.Policies that support reliability and integration of DERs 4.Policies that promote market animation and support customer choice. Major challenges going forward are forecasting and modeling DERs, regulatory and utility business model issues, reliability, valuation and pricing, and data management and sharing.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2017

Fission in R-processes Elements (FIRE) - Annual Report: Fiscal Year 2017

The goal of the FIRE topical collaboration in nuclear theory is to determine the astrophysical conditions of the rapid neutron capture process (r-process), which is responsible for the formation of heavy elements. This will be achieved by including in r-process simulations the most advanced models of fission (spontaneous, neutron-induced, beta-delayed) that have been developed at LLNL and LANL. The collaboration is composed of LLNL (lead) and LANL for work on nuclear data (ground-state properties, fission, beta-decay), BNL for nuclear data management, and the university of Notre Dame and North Carolina State University for r-process simulations. Under DOE/NNSA agreement, both universities receive funds from the DOE Office of Science, while national laboratories receive funds directly from NA221.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2017

Out Brief for the Structural Reliability Partnership Workshop

The Structural Reliability Partnership Workshop was held in Albuquerque, NM on August 29-30, 2017 and was hosted by Sandia National Laboratories. Attendees were present from academia, industry and several other national laboratories. The workshop kicked off with an introduction to the SRP to familiarize potential members with what the purpose, structure and benefits would be to their organization. Technical overviews were given on several topics by attendees from each sector – national labs, universities and industry – to provide a snapshot of the type of work that is currently being conducted on structural reliability. Attendees were then given the opportunity to suggest and discuss potential Challenge Scenario topics. Three were ultimately decided upon as being the most important: Additive Manufacturing, Hydrogen Pipeline Steels, and Bolted Joined Structures. These were then analyzed using Quad Charts to determine What, How, Who, and Why these areas would be further investigated. Rather than restricting future research to only one area, the option was left open to investigate both the top two, depending on interest and cost associated with hosting such an event. More informal collaboration may be undertaken for the third topic if members have time and interest. Other items discussed pertained to the organization, structure and policies of the Partnership. Topics including Data Management, IP, and mechanisms of partnering/information sharing were touched upon but final decisions were not made. Further action is needed before this can be done. Action items were outlined and assigned, where possible. The next workshop is to be held in early August 2018 in Boulder, CO and is to be hosted by NIST. In the interim, quarterly updates are to take place via WebEx to maintain a line of communication and to ensure progress on both the administrative and technical tasks.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2017

AToM (Advanced Tokamak Modeling)

This final report for the AToM project documents the contributions from General Atomics over the period 1 September 2014 to 31 August 2017. This includes significant progress on the development of tokamak integrated modeling frameworks (OMFIT and IPS), and development of new workflows for computing plasma profile (temperature, density, etc) evolution. Significant effort was also carried out in the high-performance computing arena, by adding the COGENT edge-simulation code to the AToM portfolio, and performing optimization of the CGYRO plasma turbulence code. Issues of data management and user outreach are also documented.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2017

Successful Completion of FY18/Q1 ASC L2 Milestone 6355: Electrical Analysis Calibration Workflow Capability Demonstration.

The Sandia Analysis Workbench (SAW) project has developed and deployed a production capability for SIERRA computational mechanics analysis workflows. However, the electrical analysis workflow capability requirements have only been demonstrated in early prototype states, with no real capability deployed for analysts’ use. This milestone aims to improve the electrical analysis workflow capability (via SAW and related tools) and deploy it for ongoing use. We propose to focus on a QASPR electrical analysis calibration workflow use case. We will include a number of new capabilities (versus today’s SAW), such as: 1) support for the XYCE code workflow component, 2) data management coupled to electrical workflow, 3) human-in-theloop workflow capability, and 4) electrical analysis workflow capability deployed on the restricted (and possibly classified) network at Sandia. While far from the complete set of capabilities required for electrical analysis workflow over the long term, this is a substantial first step toward full production support for the electrical analysts.

42 ENGINEERING↗
DOE OSTI2018

Nevada National Security Site Integrated Groundwater Sampling Plan, Revision 1

The purpose is to provide a comprehensive, integrated approach for collecting and analyzing groundwater samples to meet the needs and objectives of the DOE/EM Nevada Program’s UGTA Activity. Implementation of this Plan will provide high-quality data required by the UGTA Activity for ensuring public protection in an efficient and cost-effective manner. The Plan is designed to ensure compliance with the UGTA Quality Assurance Plan (QAP) (NNSA/NFO, 2015); Federal Facility Agreement and Consent Order (FFACO) (1996, as amended); and DOE Order 458.1, Radiation Protection of the Public and the Environment (DOE, 2013). The Plan’s scope comprises sample collection and analysis requirements relevant to assessing both the extent of groundwater contamination from underground nuclear testing and impact of testing on water quality in downgradient communities. This Plan identifies locations to be sampled by CAU and location type, sampling frequencies, sample collection methodologies, and the constituents to be analyzed. In addition, the Plan defines data collection criteria such as well purging, detection levels, and accuracy requirements/recommendations; identifies reporting and data management requirements; and provides a process to ensure coordination between NNSS groundwater sampling programs for sampling analytes of interest to UGTA. Information used in the Plan development—including the rationale for selection of wells, sampling frequency, and the analytical suite—is discussed under separate cover (N-I, 2014) and is not reproduced herein. This Plan does not address compliance for those wells involved in a permitted activity. Sampling and analysis requirements associated with these wells are described in their respective permits and are discussed in NNSS environmental reports (see Section 5.2). In addition, sampling for UGTA CAUs that are in the Closure Report (CR) stage are not included in this Plan. Sampling requirements for these CAUs are described in the CR. Frenchman Flat is currently the only UGTA CAU in the CR stage. Sampling requirements for this CAU are described in Underground Test Area (UGTA) Closure Report for Corrective Action Unit 98: Frenchman Flat Nevada National Security Site, Nevada (NNSA/NFO, 2016).

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2018

SIERRA Multimechanics Module: Aria Thermal Theory Manual - Version 4.50.

This document describes the theoretical foundation of thermal analysis in Sierra Mechanics. The SIERRA Multimechanics Module: Aria, henceforth referred to as Aria for brevity, was developed at Sandia National Laboratories under the ASC program, and approximates linear and nonlinear continuum models of heat transfer. Aria uses the SIERRA Framework, which provides data management services commonly required by computational mechanics software, and facilitates the development of coupled, multi-mechanics applications for massively parallel computers. The mathematical models in Aria are based heavily on those of COYOTE, a well-established thermal analysis program that was also developed at Sandia and its ASC code predecessor, Calore. Aria, Calore and COYOTE share a significant body of numerical methods, which are described in detail by Reddy and Gartling. Throughout this document, the terms software and implementation are synonymous with the Aria thermal-fluid analysis computer program. Whether one uses Aria to perform heat transfer analysis, or in developing a new capability for the Aria application, this document provides the information needed understand the existing numerical algorithm implementations. Justification for the fundamental assumptions of heat transfer, nor derivation of the energy conservation equations are included in this document. For a more thorough theoretical background, one is referred to one of the many available textbooks, e.g. Another reference, which is freely available in downloadable electronic form, is Lienhard and Lienhard.

97 MATHEMATICS AND COMPUTING↗