Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “science data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Pseudonymization at Scale: OLCF’s Summit Usage Data Case Study

The analysis of vast amounts of data and the processing of complex computational jobs have traditionally relied upon high performance computing (HPC) systems, which offer reliable and efficient management of large-scale computational and data resources. Understanding these analyses’ needs is paramount for designing solutions that can lead to better science, and similarly, understanding the characteristics of the user behavior on those systems is important for improving user experiences on HPC systems. A common approach to gathering data about user behavior is to extract workload characteristics from system log data available only to system administrators. Recently at Oak Ridge Leadership Computing Facility (OLCF), however, we unveiled user behavior about the Summit supercomputer by collecting data from a user’s point of view with ordinary Unix commands.In this paper, we discuss the process, challenges, and lessons learned while preparing this dataset for publication and submission to an open data challenge. The original dataset contains personal identifiable information (PII) about the users of OLCF which needed be masked prior to publication, and we determined that anonymization, which scrubs PII completely, destroyed too much of the structure of the data to be interesting for the data challenge. We instead chose to pseudonymize the dataset, which reduced the linkability of the dataset to the users’ identities. Pseudonymization is significantly more computationally expensive than anonymization, and the size of our dataset, which is approximately 175 million lines of raw text, necessitated the development of a parallelized workflow that could be reused on different HPC machines. We demonstrate the scaling behavior of the workflow on two leadership class HPC systems at OLCF, and we show that we were able to bring the overall makespan time from an impractical 20+ hours on a single node down to around 2 hours. As a result of this work, we release the entire pseudonymized dataset and make the workflows and source code publicly available.

Maheshwari, Ketan↗
DOE OSTI2023

BOXKIT

SF-23-067 BoxKit is a library that provides building blocks to parallelize and scale data science, high performance computing, and machine learning applications for block-structured datasets. Spatial data from simulations and experiments can be accessed and managed using tools available in this library when working with more data analysis oriented packages like SciKit (https://github.com/scikit-learn/scikit-learn) and FlowNet (https://github.com/NVIDIA/flownet2-pytorch)

DHRUV, AKASH↗
DOE OSTI2025

NDMAS

Overview of Current ART-GCR Data: Fuel Fabrication, Irradiation Monitoring (Fuel & Graphite – near real-time for HDG-1), Post-Irradiation Examination (Fuel & Graphite), Graphite Characterization (Baseline and Irradiated), High Temperature Metals Mechanical Tests, Design, Methods, and Validation Data, Japan Atomic Energy Agency’s High Temperature Test Reactor (HTTR), Argonne National Laboratory’s Natural convection Shutdown heat removal Test Facility (NSTF), Oregon State University’s High Temperature Test Facility (HTTF), Generation IV International VHTR Materials Handbook, Additional related data, and Advanced Test Reactor operations (near real-time).

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2020

Editorial: Achieving Water-Energy-Food Nexus Sustainability: A Science and Data Need or a Need for Integrated Public Policy?

The benefits of addressing the water, energy, and food sectors in an integrated manner is gaining significant recognition. An integrated approach can provide improved resource use efficiencies, more coherent environmental policies, and an overall strategy for achieving sustainability in the three sectors, as outlined in the United Nations Sustainable Development Goals (SDG) 2 (Food), 6 (Water), and 7 (Energy). Societies are concerned with ensuring food security, avoiding wars over water, and creating opportunity by ensuring access to energy. To be effective, however, this approach needs to be adopted at all levels of societies including government, private and civil society and reinforced by management and planning methods. This special issue identifies different approaches that are either being conceptualized or tested to support the Water-Energy-Food (WEF) Nexus approach. The articles contribute to answering the question, “Is achieving Water-Energy-Food (WEF) Nexus Sustainability a science and data need or an integrated public policy need?” In either case, both natural and social sciences will need to combine to support science issues or integrated policy issues. The papers explore the ways in which science, data, and policy development could help to define integrative principles and policies for the three sectors. This approach could expand beyond the water, energy, and food sectors to include health, environment, trade, commerce, and international assistance thereby providing broad support to the SDGs. Moreover, this issue demonstrates that data combined with new technologies (tools and models) can support better decision-making when adopted by governments and the sectors.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2023

Specifications and a Prototype Software to Demonstrate a Data Catalog for Hanford Datasets

Environmental management activities at the Hanford Site produce extensive data about site conditions, contaminants, and cleanup activities. Managing, archiving, and accessing that data requires a high degree of collaboration among site contractors and a high level of awareness by project managers and staff. The Hanford Site has a range of databases (e.g., Hanford Environmental Information System [HEIS]) and their associated user interfaces (e.g., Environmental Dashboard Application [EDA], Virtual Library [VL]), as well as other document management systems (e.g., Integrated Document Management System [IDMS]). However, Hanford lacks a single unified resource to find data (which itself comes in multiple formats) amongst the multiple disparate systems, not to mention ad hoc data not contained in an official repository/database.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Integrated Hydro-Terrestrial Modeling: Development of a National Capability

Water is one of our most important natural resources and is essential to our national economy and security. Multiple federal government agencies have mission elements that address national needs related to water. Each water-related agency champions a unique science and/or operational mission focused on advancing a portion of the nation’s ability to meet our water-related challenges. These diverse mission needs have engendered a rich and extensive base of water-related data and modeling capabilities. While useful for their intended purposes, these capabilities are not well integrated to address complex regional problems and overarching national problems. These major investments by a number of federal agencies, however, lay the foundation for an integrated hydro-terrestrial modeling and data infrastructure that will enhance knowledge, understanding, prediction, and management of the nation’s diverse water challenges. Creating a more seamless national hydro-terrestrial modeling and data capacity presents an enormous opportunity to advance operational as well as research capabilities leading to more effective water management. Advances are necessary not only in operational tools for forecasting but also in research to identify and resolve knowledge and data gaps that lead to unacceptable uncertainties in forecast outcomes. As such, close coordination across scientific, operational, and resource management communities is required. To this end, an interagency workshop on “Integrated Hydro-Terrestrial Modeling: Development of a National Capacity” was held at the National Science Foundation (NSF) headquarters in Alexandria, Virginia in September 2019, led jointly by the NSF, the U.S. Department of Energy (DOE) and the U.S. Geological Survey (USGS) and with broader interagency support provided through an interagency steering committee. This workshop provided a venue to bring together representatives of water-related agencies and their scientific partners (including university researchers) to initiate and refine a vision for a national Integrated Hydro-Terrestrial Modeling (IHTM) and data infrastructure and advance ideas towards its development. The workshop was designed to address three critical foci to advance the development of a national IHTM capacity: “Priority Water Challenges” around which to motivate and initiate development; Technical and methodological obstacles related to data and modeling; Organizational, structural, and cultural barriers that heretofore have impeded integration of capabilities across the federal and research landscapes. The following “Priority Water Challenge” domain areas represent targets for initiating development of the IHTM and were identified and selected in alignment with priorities of the administration’s Water Sub-Cabinet: (1) Nutrient loading, hypoxia, and harmful algal blooms; (2) Water availability in the western United States; and (3) Extreme weather-related water hazards. These water challenges span agency mission boundaries and encompass a broad range of geographies, complex system dynamics and feedbacks, and critical processes spanning hydrological, climatic, and biophysical systems as well as land-use/land-cover, agricultural, built infrastructure, societal, economic, and decisional environments. These three Priority Water Challenges cannot be fully addressed without leveraging complementary and synergistic capabilities across multiple agencies.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2023

Data Archive and Portal (DAP) Platform for Solid Phase Processing Technologies

The scale and speed of data generated by modern scientific experiments have constantly challenged the research community to store, curate, manage and optimally use it to drive scientific discoveries. In this work, we have developed a data archive and portal (DAP) platform including analytics capabilities to collect, curate, and manage data and metadata stream for solid phase processing (SPP) techniques. We successfully hosted around ~347K files of data related to processing parameters, microscopic images, and spectroscopic data related to solid phase processing. The DAP platform for SPP will establish an enduring capability to support machine learning and grow collaboration at the intersection of materials science and data science.

36 MATERIALS SCIENCE↗
DOE OSTI2022

RCSB Protein Data bank: Tools for visualizing and understanding biological macromolecules in 3D

Abstract Now in its 52nd year of continuous operations, the Protein Data Bank (PDB) is the premiere open‐access global archive housing three‐dimensional (3D) biomolecular structure data. It is jointly managed by the Worldwide Protein Data Bank (wwPDB) partnership. The Research Collaboratory for Structural Bioinformatics Protein Data Bank (RCSB PDB) is funded by the National Science Foundation, National Institutes of Health, and US Department of Energy and serves as the US data center for the wwPDB. RCSB PDB is also responsible for the security of PDB data in its role as wwPDB‐designated Archive Keeper. Every year, RCSB PDB serves tens of thousands of depositors of 3D macromolecular structure data (coming from macromolecular crystallography, nuclear magnetic resonance spectroscopy, electron microscopy, and micro‐electron diffraction). The RCSB PDB research‐focused web portal ( RCSB.org ) makes PDB data available at no charge and without usage restrictions to many millions of PDB data consumers around the world. The RCSB PDB training, outreach, and education web portal ( PDB101.RCSB.org ) serves nearly 700 K educators, students, and members of the public worldwide. This invited Tools Issue contribution describes how RCSB PDB (i) is organized; (ii) works with wwPDB partners to process new depositions; (iii) serves as the wwPDB‐designated Archive Keeper; (iv) enables exploration and 3D visualization of PDB data via RCSB.org ; and (v) supports training, outreach, and education via PDB101.RCSB.org . New tools and features at RCSB.org are presented using examples drawn from high‐resolution structural studies of proteins relevant to treatment of human cancers by targeting immune checkpoints.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

LANL Contributions to pyiron: An integrated development environment for computational materials science

To support and accelerate the development of simulation protocols in atomistic modelling, we introduce an integrated development environment (IDE) for computational materials science called pyiron (http://pyiron.org). The pyiron IDE combines a web based source code editor, a job management system for build automation, and a hierarchical data management solution. The core components of the pyiron IDE are pyiron objects based on an abstract class, which links application structures such as atomistic structures, projects, jobs, simulation protocols and computing resources with persistent storage and an interactive user environment. The simulation protocols within the pyiron IDE are constructed using the Python programming language. To highlight key concepts of this tool as well as to demonstrate its ability to simplify the implementation and testing of simulation protocols we discuss two applications. In these examples we show how pyiron supports the whole life cycle of a typical simulation, seamlessly combines ab initio with empirical potential calculations, and how complex feedback loops can be implemented. While originally developed with focus on ab initio thermodynamics simulations, the concepts and implementation of pyiron are general thus allowing to employ it for a wide range of simulation topics.

Janssen, Jan↗
DOE OSTI2023

Curifactory: A research experiment manager

Curifactory is a command line tool and framework for organizing Python experiment code, configuration parameters, and results. It is an opinionated and lightweight approach to workflow management infrastructure and is primarily intended to support researchers conducting experiments on one machine. This software was developed to support the reproducibility of results for several data science projects in the Nuclear Nonproliferation Division at Oak Ridge National Laboratory. Curifactory is intended to be a general framework and is not specific to machine learning or data science. It can aid in any field in which experiments are primarily computation-based studies and can be implemented in Python (e.g., high-energy physics, astronomy, computational chemistry). Here, the design emphasizes the automated caching of intermediate data analysis artifacts to speed up development involving computationally intensive tasks. It also allows for data provenance and experiment reproduction. Individual experiment runs are tracked through logs and their output reports, and entire copies of a run with all cached data and metadata can be exported for others to run using Curifactory on another machine. Curifactory experiments can either be integrated into a project from the beginning or can be written on top of an existing codebase without needing significant modification. A few important views of the Curifactory library can be seen in Figure 1.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Publishing Environmental Assessment and Management Science: Crossing the Hurdles

Benefits accrue to scientists, resource managers, companies, and policymakers when environmental scientists publish in peer-reviewed journals. However, environmental scientists and practitioners face challenges, including the sometimes low value placed on journal articles, institutional vested interests in outcomes, and the changing priorities of employers and project sponsors. Confidentiality agreements can also lead scientists to assume publication is not an option. Case studies may be viewed by potential authors as too routine for peer-reviewed journals. On the basis of 30 years of experience, we suggest that publishing hurdles can be overcome and that environmental scientists have a range of options. The topics of manuscripts can include not only results from case studies and perspectives based on them but also byproducts of assessments, including definitions, plans, monitoring methods and models, and decision frameworks. Environmental scientists have unique opportunities to move science forward with their practical knowledge if they can move across the institutional, logistical, data-related, and content-related hurdles.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Machine learning enabling high-throughput and remote operations at large-scale user facilities

Imaging, scattering, and spectroscopy are fundamental in understanding and discovering new functional materials. Contemporary innovations in automation and experimental techniques have led to these measurements being performed much faster and with higher resolution, thus producing vast amounts of data for analysis. These innovations are particularly pronounced at user facilities and synchrotron light sources. Machine learning (ML) methods are regularly developed to process and interpret large datasets in real-time with measurements. However, there remain conceptual barriers to entry for the facility general user community, whom often lack expertise in ML, and technical barriers for deploying ML models. Herein, we demonstrate a variety of archetypal ML models for on-the-fly analysis at multiple beamlines at the National Synchrotron Light Source II (NSLS-II). We describe these examples instructively, with a focus on integrating the models into existing experimental workflows, such that the reader can easily include their own ML techniques into experiments at NSLS-II or facilities with a common infrastructure. The framework presented here shows how with little effort, diverse ML models operate in conjunction with feedback loops via integration into the existing Bluesky Suite for experimental orchestration and data management.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Accelerating Scientific Workflows on HPC Platforms with In Situ Processing

Scientific workflows drive most modern large-scale science breakthroughs by allowing scientists to define their computations as a set of jobs executed in a given order based on their data dependencies. Workflow management systems (WMSs) have become key to automating scientific workflows-executing computational jobs and orchestrating data transfers between those jobs running on complex high-performance computing (HPC) platforms. Traditionally, WMSs use files to communicate between jobs: a job writes out files that are read by other jobs. However, HPC machines face a growing gap between their storage and compute capabilities. To address that concern, the scientific community has adopted a new approach called in situ, which bypasses costly parallel filesystem I/O operations with faster in-memory or in-network communications. When using in situ approaches, communication and computations can be interleaved. In this work, we leverage the Decaf in situ dataflow framework to accelerate task-based scientific workflows managed by the Pegasus WMS, by replacing file communications with faster MPI messaging. We propose a new execution engine that uses Decaf to manage communications within a sub-workflow (i.e., set of jobs) to optimize inter-job communications. We consider two workflows in this study: (i) a synthetic workflow that benchmarks and compares file- and MPI-based communication; and (ii) a realistic bioinformatics workflow that computes mu-tational overlaps in the human genome. Experiments show that in situ communication can improve the bioinformatics workflow execution time by 22% to 30% compared with file communication. Our results motivate further opportunities and challenges for bridging traditional WMSs with in situ frameworks.

Decaf↗
DOE OSTI2021

Atmospheric Radiation Measurement (ARM) User Facility 2020 Decadal Vision

The Atmospheric Radiation Measurement (ARM) user facility was established in 1989 by the U.S Department of Energy (DOE) Office of Biological and Environmental Research (BER) to provide an observational basis for studying the Earth’s climate. ARM began collecting observations in 1992 and was designated a user facility in 2003. The facility includes a network of extensively instrumented long-term fixed-location observatories and mobile facilities. The ARM facility also includes an aerial component to augment these ground-based measurements. Because of the diversity of in situ and remotely obtained observations, ARM’s data management infrastructure is designed to collect, process, and deliver data to the research community.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Architecture of a Data Portal for Publishing and Delivering Open Data for Atmospheric Measurement

Atmospheric data are collected by researchers every day. Campaigns such as GOAmazon 2014/2015 and the Amazon Tall Tower Observatory collect essential data on aerosols, gases, cloud properties, and meteorological parameters in the Brazilian Amazon basin. These data products provide insights and essential information for analyzing and predicting natural processes. However, in Brazil, it is estimated that more than 80% of the scientific data collected are not published due to the lack of web portals that collect and store these data. This makes it difficult, or even impossible, to access and integrate the data, which can result in the loss of significant amounts of information and significantly affect the understanding of the overall data. To address this problem, we propose a data portal architecture and open data deployment that enable Big Data processing, human interaction, and download-oriented approaches with tools that help users catalog, publish and visualize atmospheric data. Thus, we describe the architecture developed, based on the experience of the Atmospheric Radiation Measurement Data Center, which incorporates the principles of FAIR, the infrastructure and content management system for managing scientific data. The portal partial results were tested with environmental data from contaminated areas at the University of São Paulo. Overall, this data portal creates more shared knowledge about atmospheric processes by providing users with access to open environmental data.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2023

Toward a Smart Metaverse City: Immersive Realism and 3D Visualization of Digital Twin Cities

Metaverse and its related extended reality technologies can enable immersive, realistic, and participatory visualization of 3D data, and their use and the potential within a smart city can be effective for supporting urban research and urban operations management. This book chapter describes a vision for prototyping a “Smart Metaverse City” to combine the unique advantage of the Metaverse technology with the two-way connectivity of a digital twin city application. This synergy aims to create a virtual environment for immersive geovisualization to help researchers and the public understand the complex urban system through science-based and data-driven approaches. This book chapter selectively reviews past technological and paradigm advancements for collecting, analyzing, and visualizing 3D urban big data. Then we present a prototyping Geographic Information System (GIS) framework, together with some relevant data sources and open-source web technologies, to help researchers create a smart Metaverse city. We demonstrate our vision and discuss its application opportunities through a real-world example, a digital twin city developed at the Oak Ridge National Laboratory, to facilitate participatory, smart, and sustainable campus management.

Xu, Haowen↗
DOE OSTI2023

Optimizing Patient-Specific Medication Regimen Policies Using Wearable Sensors in Parkinson’s Disease

Effective treatment of Parkinson’s disease (PD) is a continual challenge for healthcare providers, and providers can benefit from leveraging emerging technologies to supplement traditional clinic care. We develop a data-driven reinforcement learning (RL) framework to optimize PD medication regimens through wearable sensors. We leverage a data set of n = 26 PD patients who wore wrist-mounted movement trackers for two separate six-day periods. Using these data, we first build and validate a simulation model of how individual patients’ movement symptoms respond to medication administration. We then pair this simulation model with an on-policy RL algorithm that recommends optimal medication types, timing, and dosages during the day while incorporating human-in-the-loop considerations on medication administration. The results show that the RL-prescribed medication regimens outperform physicians’ medication regimens, despite physicians having access to the same data as the RL agent. To validate our results, we assess our wearable-based RL medication regimens using n = 399 PD patients from the Parkinson’s Progression Markers Initiative data set. We show that the wearable-based RL medication regimens would lead to significant symptom improvement for these patients, even more so than training RL policies directly from this data set. In doing so, we show that RL models from even small data sets of wearable data can offer novel, generalizable clinical insights and medication strategies, which may outperform those derived from larger data sets without wearable data.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2020

Inferred inflow forecast horizons guiding reservoir release decisions across the United States

Abstract. Medium- to long-range forecasts often guide reservoir release decisions to support water management objectives, including mitigating flood and drought risks. While there is a burgeoning field of science targeted at improving forecast products and associated decision support models, data describing how and when forecasts are applied in practice remain undeveloped. This lack of knowledge may prevent hydrological modelers from developing accurate reservoir release schemes for large-scale, distributed hydrology models that are increasingly used to assess the vulnerabilities of large regions to hydrological stress. We address this issue by estimating seasonally varying, regulated inflow forecast horizons used in the operations of more than 300 dams throughout the conterminous United States (CONUS). For each dam, we take actual forward observed inflows (perfect foresight) as a proxy for forecasted flows available to the operator and then identify for each week of the year the forward horizon that best explains the release decisions taken. Resulting “horizon curves” specify for each dam the inferred inflow forecast horizon as a function of the week of the water year. These curves are analyzed for strength of evidence for contribution of medium- to long-range forecasts in decision making. We use random forest classification to estimate that approximately 80 % of large dams and reservoirs in the US (1553±50 out of 1927 dams with at least 10 Mm3 storage capacity) adopt medium- to long-range inflow forecasts to inform release decisions during at least part of the water year. Long-range forecast horizons (more than 6 weeks ahead) are detected in the operations of reservoirs located in high-elevation regions of the western US, where snowpack information likely guides the release. A simulation exercise conducted on four key western US reservoirs indicates that forecast-informed models of reservoir operations may outperform models that neglect the horizon curve – including during flood and drought conditions.

54 ENVIRONMENTAL SCIENCES↗