Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,351 records · Page 75
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Towards scalable dataframe systems

Dataframes are a popular abstraction to represent, prepare, and analyze data. Despite the remarkable success of dataframe libraries in R and Python, dataframes face performance issues even on moderately large datasets. Moreover, there is significant ambiguity regarding dataframe semantics. In this paper we lay out a vision and roadmap for scalable dataframe systems. To demonstrate the potential in this area, we report on our experience building Modin, a scaled-up implementation of the most widely-used and complex dataframe API today, Python's pandas. With pandas as a reference, we propose a simple data model and algebra for dataframes to ground discussion in the field. Given this foundation, we lay out an agenda of open research opportunities where the distinct features of dataframes will require extending the state of the art in many dimensions of data management. We discuss the implications of signature dataframe features including flexible schemas, ordering, row/column equivalence, and data/metadata fluidity, as well as the piecemeal, trial-and-error-based approach to interacting with dataframes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Hindsight logging for model training

In modern Machine Learning, model training is an iterative, experimental process that can consume enormous computation resources and developer time. To aid in that process, experienced model developers log and visualize program variables during training runs. Exhaustive logging of all variables is infeasible, so developers are left to choose between slowing down training via extensive conservative logging, or letting training run fast via minimalist optimistic logging that may omit key information. As a compromise, optimistic logging can be accompanied by program checkpoints; this allows developers to add log statements post-hoc, and "replay" desired log statements from checkpoint---a process we refer to as hindsight logging. Unfortunately, hindsight logging raises tricky problems in data management and software engineering. Done poorly, hindsight logging can waste resources and generate technical debt embodied in multiple variants of training code. In this paper, we present methodologies for efficient and effective logging practices for model training, with a focus on techniques for hindsight logging. Our goal is for experienced model developers to learn and adopt these practices. To make this easier, we provide an open-source suite of tools for Fast Low-Overhead Recovery (flor) that embodies our design across three tasks: (i) efficient background logging in Python, (ii) adaptive periodic checkpointing, and (iii) an instrumentation library that codifies hindsight logging for efficient and automatic record-replay of model-training. Model developers can use each flor tool separately as they see fit, or they can use flor in hands-free mode, entrusting it to instrument their code end-to-end for efficient record-replay. Our solutions leverage techniques from physiological transaction logs and recovery in database systems. Evaluations on modern ML benchmarks demonstrate that flor can produce fast checkpointing with small user-specifiable overheads (e.g. 7%), and still provide hindsight log replay times orders of magnitude faster than restarting training from scratch.

Computer Science↗
DOE OSTI2022

ESS-DIVE Reporting Format for Dataset Package Metadata

ESS-DIVE’s (Environmental Systems Science Data Infrastructure for a Virtual Ecosystem) dataset metadata reporting format is intended to compile information about a dataset (e.g., title, description, funding sources) that can enable reuse of data submitted to the ESS-DIVE data repository. The files contained in this dataset include instructions (dataset_metadata_guide.md and README.md) that can be used to understand the types of metadata ESS-DIVE collects. The data dictionary (dd.csv) follows ESS-DIVE’s file-level metadata reporting format and includes brief descriptions about each element of the dataset metadata reporting format. This dataset also includes a terminology crosswalk (dataset_metadata_crosswalk.csv) that shows how ESS-DIVE’s metadata reporting format maps onto other existing metadata standards and reporting formats.Data contributors to ESS-DIVE can provide this metadata by manual entry using a web form or programmatically via ESS-DIVE’s API (Application Programming Interface). A metadata template (dataset_metadata_template.docx or dataset_metadata_template.pdf) can be used to collaboratively compile metadata before providing it to ESS-DIVE.Since being incorporated into ESS-DIVE’s data submission user interface, ESS-DIVE’s dataset metadata reporting format, has enabled features like automated metadata quality checks, and dissemination of ESS-DIVE datasets onto other data platforms including Google Dataset Search and DataCite.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Data for publication: "A fresh take: Seasonal changes in terrestrial freshwater inputs impact salt marsh hydrology and vegetation dynamics"

This data repository contains data associated with the manuscript "A fresh take: Seasonal changes in terrestrial freshwater inputs impact salt marsh hydrology and vegetation dynamics". This study was conducted at the Elkhorn Slough National Estuarine Research Reserve in Watsonville, California from October 2019 - June 2022. We sought to understand the role of shallow freshwater inputs from adjacent uplands on salt marsh hydrologic behavior and vegetation productivity. This dataset contains CSV files of the following: daily salt marsh subsurface water level and pore water conductivity, monthly vegetation survey measurements, soil core data. Estuary surface water level, conductivity, and local precipitation were downloaded from the National Estuarine Research Reserve System (Centralized Data Management Office) at https://cdmo.baruch.sc.edu/.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

IDAES-PSE 2.6.0 Release

The Institute for the Design of Advanced Energy Systems (IDAES) Integrated Platform is a versatile computational environment offering extensive process systems engineering (PSE) capabilities for optimizing the design and operation of complex, interacting technologies and systems. IDAES enables users to efficiently search vast, complex design spaces to discover the lowest cost solutions while supporting the full process modeling lifecycle, from conceptual design to dynamic optimization and control. The extensible, open platform empowers users to create models of novel processes and rapidly develop custom analyses, workflows, and end-user applications. IDAES-PSE 2.6.0 Release Highlights Upcoming Changes IDAES will be switching to the new Pyomo solver interface in the next release. Whilst this will hopefully be a smooth transition for most users, there are a few important changes to be aware of. The new solver interface uses a different version of the IPOPT writer (“ipopt_v2”) and thus any custom configuration options you might have set for IPOPT will not carry over and will need to be reset. By default, the new Pyomo linear presolver will be activated with ipopt_v2. Whilst are working to identify any bugs in the presolver, it is possible that some edge cases will remain. IDAES will begin deploying a new set of scaling tools and APIs over the next few releases that make use of the new solver writers. The old scaling tools and APIs will remain for backward compatibility but will begin to be deprecated. New Models, Tools and Features New Intersphinx extension automatically linking Jupyter notebook examples to project documentation New end-to-end diagnostics example demonstrated on a real problem New complementarity formulation for VLE with cubic equations of state, backward compatibility for old formulation New solver interface with presolve (ipopt_v2) in support of upcoming changes to the initialization and APIs methods, with default set to ipopt to maintain backwards compatibility; this will deprecate once all examples have been updated New forecaster and parameterized bidder methods within grid integration library Updated surrogates API and examples to support Keras 3, with backwards compatibility for older formats such as TensorFlow SavedModel (TFSM) Updated costing base dictionary to include the 2023 cost year index value Updated ProcessBlock to include information on the constructing block class Updated Flowsheet Visualizer to allow visualize() method to return value and functions Bug Fixes Fixed bug in the Modular Property Framework that would cause errors when trying to use phase-based material balances with phase equilibria. Fixed bug in Modular Properties Framework that caused errors when initializing models with non-vapor-liquid phase equilibria. Fixed typos flagged by June update to crate-ci/typos and removed DMF-related exceptions Minor corrections of units of measurement handling in power plant waste/transport costing expressions, control volume material holdup expressions, and BTX property package parameters Fixed throwing >7500 numpy deprecation warnings by replacing scalar value assignment with element extraction and item iteration calls Testing and Robustness Migrated slow tests (>10s) to integration, impacting test coverage but also yielding a nearly 30% decrease in local test runtime Pinned pint to avoid issues with older supported Python versions Pinned codecov versions to avoid tokenless upload behavior with latest version Bumped extensions to version 3.4.2 to allow pointing to non-standard install location Deprecations and Removals Python 3.8 is no longer supported. The supported Python versions are 3.9 through 3.12 The Data Management Framework (DMF) is no longer supported. Importing idaes.core.dmf will cause a deprecation warning to be displayed until the next release The SOFC Keras surrogates have been removed. The current version of the SOFC surrogate model in the examples repository is a PySMO Kriging model.

AS↗
DOE OSTI2024

Conclusions and outlook

In this chapter, we provide notes on the state of metadata collection and organization practices and reflect on the achievements of the MetaCO task group. We also include an utility perspective on how to use this report in daily data management practice and a short guide is provided to show how metadata practices can be initiated. The chapter, and the report, is concluded within an outlook, sketching future progress and opportunities on the horizon.

Aguado, Daniel↗
DOE OSTI2020

Fission In R-process Elements (FY20 Q2 Quarterly Report)

The goal of the FIRE topical collaboration in nuclear theory is to determine the astrophysical conditions of the rapid neutron capture process (r-process), which is responsible for the formation of heavy elements. This will be achieved by including in r-process simulations the most advanced models of fission (spontaneous, neutron-induced, β-delayed) that have been developed at LLNL and LANL. The collaboration is composed of LLNL (lead) and LANL for fission work, BNL for nuclear data management, and the university of Notre Dame and North Carolina State University for r-process simulations. Under DOE/NNSA agreement, both universities receive funds from DOE Office of Science, while national laboratories receive funds directly from NA221.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2020

Infrasound Needs from a Future Geophysical Facility

Since the start of the SAGE/GAGE era, infrasound has become increasingly popular as both dedicated infrasound deplayments or coupled with other geophysical sensors for monitoring and research purposes, with interdisciplinary applications in both Earth and Atmospheric science. Infrasound can be used to study and monitor a variety of hazards that occur at or near the surface, including volcanic eruptions, earthquakes, and rapid gravity-driven mass movements. Because of this, it can help address some of the questions raised by recent community vision initiatives such as SZ4D (increase the understanding of the processes underlying subduction geohazards) and ERUPT (further our understanding of volcanic systems to help inform eruption forecasting). Many of the recommendations included here are topics of focus at workshops and conferences, including most recently at the November 2019 CONVERSE infrasound workshop. As NSF plans the successor to SAGE/GAGE, we advocate including infrasound in addition to the already-supported fields of seismology and geodesy. The two main features that we would like to see in the new facility are: 1) a community pool of infrasound instrumentation, and 2) the continuation and expansion of data management resources provided by the IRIS DMC.

58 GEOSCIENCES↗
DOE OSTI2020

Bio-Restore Workshop Summary Report

This report summarizes the results of a Bioenergy Technologies Office-sponsored public workshop held at Argonne National Laboratory on September 25–26, 2019. The purpose of the Bio-Restore Workshop was to bring together experts to discuss the current state of the technology and to identify data and R&D relevant to the bio-restore concept. Specific topics included using dedicated energy crops and other biomass sources to provide environmental benefits, implementing algae technologies (e.g., wastewater treatment, direct bloom harvest, turf scrubbers, macroalgae), quantifying and valorizing ecosystem services, developing low-cost sensors and data management systems, and integrating “bio-restore biomass” with supply chain needs. For the purposes of the workshop, the term “bio-restore biomass” refers to terrestrial and algal biomass that provides environmental benefits when produced or harvested. For terrestrial biomass, this primarily includes energy crops (herbaceous and woody), other cellulosic biomass, and waste biomass. For algal biomass, this includes wastewater treatment algal biomass, algal blooms, algal turf scrubbers/attached growth systems, and macroalgae. Types of biomass not of interest to the workshop discussions include oilseed crops, crops generally grown for food or feed, and algae grown in artificial light conditions or other energy-intensive cultivation designs.

09 BIOMASS FUELS↗
DOE OSTI2020

Fission In R-process Elements - FY2020

The goal of the FIRE topical collaboration in nuclear theory is to determine the astrophysical conditions of the rapid neutron capture process (r-process), which is responsible for the formation of heavy elements. This will be achieved by including in r-process simulations the most advanced models of fission (spontaneous, neutron-induced, β-delayed) that have been developed at LLNL and LANL. The collaboration is composed of LLNL (lead) and LANL for work on nuclear data (ground-state properties, fission, β-decay), BNL for nuclear data management, and the university of Notre Dame and North Carolina State University for r-process simulations. Under DOE/NNSA agreement, both universities receive funds from the DOE Office of Science, while national laboratories receive funds directly from NA221.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2020

Development of Monitoring Techniques for Binderjet Additive Manufacturing of Silicon Carbide Structures

The Transformational Challenge Reactor (TCR) program is leveraging additive manufacturing (AM) technologies to fabricate nuclear components to be assembled into a fully functional microreactor core. Compared with traditional manufacturing technologies, AM technologies allow (1) observation of the manufacturing process at a much higher resolution in real-time using in situ monitoring technologies to capture the sensor signature that scientifically describes each event occurring over time and space and (2) validation of the manufacturing process quality using domain-informed data analytics techniques as a potential qualification and certification methodology for the final component. This report provides an update on the program work on binder jetting in situ process monitoring and associated data analytics results, as well as sample placement and tracking for the subsequent chemical vapor infiltration (CVI) process. Examples are provided to illustrate the progress. Elements of the Digital Thread and data management are discussed in the main document, and an extensive supplemental material section is provided detailing the Digital Platform, as well as its implementation and components. In conclusion the path forward for the next fiscal year is discussed.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2020

Fission In R-process Elements (Q4 FY2020 Quarterly Report)

The goal of the FIRE topical collaboration in nuclear theory is to determine the astrophysical conditions of the rapid neutron capture process (r-process), which is responsible for the formation of heavy elements. This will be achieved by including in r-process simulations the most advanced models of fission (spontaneous, neutron-induced, β-delayed) that have been developed at LLNL and LANL. The collaboration is composed of LLNL (lead) and LANL for fission work, BNL for nuclear data management, and the university of Notre Dame and North Carolina State University for r-process simulations. Under DOE/NNSA agreement, both universities receive funds from DOE Office of Science, while national laboratories receive funds directly from NA221.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2020

Development of Monitoring Techniques for Laser Powder Bed Additive Manufacturing of Metal Structures (Progress Report)

The Transformational Challenge Reactor (TCR) program is leveraging additive manufacturing (AM) technologies to fabricate nuclear grade components to be assembled into a fully functional microreactor core. Compared to traditional manufacturing technologies, additive manufacturing technologies allow (1) observation of the manufacturing process at a much higher resolution in real time using in situ monitoring technologies to capture the sensor signature that scientifically describes each event occurring over time and space, and (2) validation of the manufacturing process quality using domain-informed data analytics techniques as a potential qualification and certification methodology for the final component. This report provides an update on the program work on laser powder bed fusion in-situ process monitoring and associated data analytics results. Examples are provided to illustrate the progress. Elements of the Digital Thread and data management are discussed in the main document, and an extensive supplemental material section is provided detailing the Digital Platform, as well as its implementation and components. In conclusion the path forward for the next fiscal year is discussed.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2020

Report on Progress of correlation of in-situ and ex-situ data and the use of artificial intelligence to predict defects

The Transformational Challenge Reactor (TCR) program is leveraging additive manufacturing (AM) technologies to fabricate nuclear components which will be assembled into a fully functional microreactor core. Compared with traditional manufacturing technologies, AM technologies allow (1) real-time observation of the manufacturing process at a much higher resolution using in-situ monitoring technologies to capture the sensor signatures that scientifically describe each event occurring over time and space and (2) validation of the manufacturing process quality using domain-informed data analytics techniques as a potential qualification and certification methodology for the final component. This report provides an update on the program work on in-situ and ex-situ data correlation and associated data analytics results. Examples are provided to illustrate progress with respect to laser powder bed fusion (L-PBF), binder jetting, computed tomography (CT) reconstruction, and mechanical testing. Elements of the Digital Thread and data management infrastructure are discussed in the main document, and an extensive supplemental appendix is provided detailing the Digital Platform, as well as its implementation and subcomponents. In conclusion, the path forward for the next fiscal year is also discussed.

42 ENGINEERING↗
DOE OSTI2020

ESGF project plan for ramping down development activities and transition the project and data to a new team

This document summarizes the current software development activities carried by the ESGF team with planned delivery dates before the transition of the project to a new team. The work we started on the publishing, search and retrieval, and the backend for indexing and data management that can be deployed using Kubernetes are planned to be delivered within the next three months; however, this assumes that the current team members will continue to work on the project. The activities are listed in Table 1 by priority and in case we lose team members, development of lower priority tasks can be paused and passed to the new team with its current status. Team members can be re-assigned based on their availability and transition work needed. Table 2 below describe activities necessary to help project transition to the new team. Deadlines for these activities are before October 1 st of 2022 or the completion of a successful transition. Support during the transition period assumes that we will have 3 key ESGF members and the ESGF PI supported at the 50% level or less for the duration of the transition. The level of support is an estimate, and it depends on several factors that cannot be decided at this time. The total cost of the proposed work and support activities for fiscal year 2021 is $1,270K with $230K carry over to fiscal year 2022. The cost for the support work listed in Table 2 is the sum of the two rows “HW support” and “project support, maintenance, and transition efforts”, colored blue, in Table1 which adds up to $720K. This cost is included in the $1,270K cost in table 1.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Fission In R-process Elements (Q1 FY2021)

The goal of the FIRE topical collaboration in nuclear theory is to determine the astrophysical conditions of the rapid neutron capture process (r-process), which is responsible for the formation of heavy elements. This will be achieved by including in r-process simulations the most advanced models of fission (spontaneous, neutron-induced, β-delayed) that have been developed at LLNL and LANL. The collaboration is composed of LLNL (lead) and LANL for fission work, BNL for nuclear data management, and the university of Notre Dame and North Carolina State University for r-process simulations. Under DOE/NNSA agreement, both universities receive funds from DOE Office of Science, while national laboratories receive funds directly from NA221

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

Fission In R-Process Elements

The goal of the FIRE topical collaboration in nuclear theory is to determine the astrophysical conditions of the rapid neutron capture process (r-process), which is responsible for the formation of heavy elements. This will be achieved by including in r-process simulations the most advanced models of fission (spontaneous, neutron-induced, β-delayed) that have been developed at LLNL and LANL. The collaboration is composed of LLNL (lead) and LANL for fission work, BNL for nuclear data management, and the university of Notre Dame and North Carolina State University for r -process simulations. Under DOE/NNSA agreement, both universities receive funds from DOE Office of Science, while national laboratories receive funds directly from NA221.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗