SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
The data generated from scientific workflows often become unusable due to the lack or incompleteness of information required for processing and analyzing the data. Reproducibility of scientific data and workflows facilitates efficient processing and analyses. A key to enabling reproducibility is to capture the end-to-end workflow life cycle, and any contextual metadata and provenance.
Curifactory is a command line tool and framework for organizing Python experiment code, configuration parameters, and results. It is an opinionated and lightweight approach to workflow management infrastructure and is primarily intended to support researchers conducting experiments on one machine. This software was developed to support the reproducibility of results for several data science projects in the Nuclear Nonproliferation Division at Oak Ridge National Laboratory. Curifactory is intended to be a general framework and is not specific to machine learning or data science. It can aid in any field in which experiments are primarily computation-based studies and can be implemented in Python (e.g., high-energy physics, astronomy, computational chemistry). Here, the design emphasizes the automated caching of intermediate data analysis artifacts to speed up development involving computationally intensive tasks. It also allows for data provenance and experiment reproduction. Individual experiment runs are tracked through logs and their output reports, and entire copies of a run with all cached data and metadata can be exported for others to run using Curifactory on another machine. Curifactory experiments can either be integrated into a project from the beginning or can be written on top of an existing codebase without needing significant modification. A few important views of the Curifactory library can be seen in Figure 1.
xCDAT (Xarray Climate Data Analysis Tools) is an open-source Python package that extends Xarray (Hoyer & Hamman, 2017) for climate data analysis on structured grids. xCDAT streamlines analysis of climate data by exposing common climate analysis operations through a set of straightforward APIs. Some of xCDAT’s key features include spatial averaging, temporal averaging, and regridding. These features are inspired by the Community Data Analysis Tools (CDAT) library (Dean N. Williams et al., 2009) (D. N. Williams, 2014) (Doutriaux et al., 2019) and leverage powerful packages in the Xarray ecosystem including xESMF (Zhuang et al., 2023), xgcm (Abernathey et al., 2022), and CF xarray (Cherian et al., 2023). To ensure general compatibility across various climate models, xCDAT operates on datasets that are compliant with the Climate and Forecast (CF) metadata conventions (Hassell et al., 2017).
Research in the data-intensive discipline of high energy physics (HEP) often relies on domain-specific digital contents. Reproducibility of research relies on proper preservation of these digital objects. This paper reflects on the interpretation of principles of Findability, Accessibility, Interoperability, and Reusability (FAIR) in such context and demonstrates its implementation by describing the development of an end-to-end support infrastructure for preserving and accessing Universal FeynRules Output (UFO) models guided by the FAIR principles. UFO models are custom-made python libraries used by the HEP community for Monte Carlo simulation of collider physics events. Our framework provides simple but robust tools to preserve and access the UFO models and corresponding metadata in accordance with the FAIR principles.
Research in the data-intensive discipline of high energy physics (HEP) often relies on domain-specific digital contents. Reproducibility of research relies on proper preservation of these digital objects. This paper reflects on the interpretation of principles of Findability, Accessibility, Interoperability, and Reusability (FAIR) in such context and demonstrates its implementation by describing the development of an end-to-end support infrastructure for preserving and accessing Universal FeynRules Output (UFO) models guided by the FAIR principles. UFO models are custom-made python libraries used by the HEP community for Monte Carlo simulation of collider physics events. Our framework provides simple but robust tools to preserve and access the UFO models and corresponding metadata in accordance with the FAIR principles.
Sensors are increasingly deployed for process monitoring and control. These produce on-line measurements at a high frequency, in parallel with low-frequency laboratory measurements. Compared to laboratory practices, sensor data quality assessment and control practices are far less structured at most utilities. This leads to inaccurate sensor data with unknown uncertainty factors.This chapter shows how to establish standard operating procedures (SOPs) to support sensor data quality assessment and control and subsequent maintenance actions by producing relevant sensor metadata. Furthermore, SOPs are provided for the most commonly used wastewater quality sensors, inspired by utility and academic best practices. This chapter builds on definitions provided in Chapter 3 and provides additional definitions specifically related to sensors maintenance. Chapter 6 complements the methods in this chapter, which are based on reference measurements, with data-analytical techniques.
This chapter provides a comprehensive overview of the main steps for algorithmic sensor signal quality assessment, which can enhance the decision-making process for water resource recovery facility (WRRF) operation and optimization. It introduces the concept of redundancy as the basis for data quality assessment. It also explains the typical data processing pipeline, which consists of preliminary analysis, data pre-processing, and specific algorithmic approaches. Each of these processes is presented and discussed in three separate sections. Importantly, this chapter introduces the main approaches for data quality assessment, provides guidelines for selecting the most suitable one and the key performance indicators to evaluate them and explains how to collect metadata through such an algorithmic approach.
The project "Analyzing and visualizing next generation climate data" adds block-structured (mosaic) grid support, parallel processing, and 2D/3D curvilinear interpolation to the open-source UV-CDAT climate data analysis tool. Block structured grid support complies to the Gridspec extension submitted to the Climate and Forecast metadata conventions. It contains two parts: aggregation of data spread over multiple mosaic tiles (M-SPEC) and aggregation of temporal data stored in different files (F-SPEC). Together, M-SPEC and F-SPEC allow users to interact with data stored in multiple files as if the data were in a single file. For computational expensive tasks, a flexible, multi-dimensional, multi-type distributed array class allows users to process data in parallel using remote memory access. Both nodal and cell based interpolation is supported; users can choose between different interpolation libraries including ESMF and LibCF depending on the their particular needs.
Through phase 1 and phase 2 SBIR grants, s fully functional I/O Controller and communication protocol for version 4 of EPICS is completed. This new software architecture provides a flexible and extendible architecture. Version 4 is implemented fully in Java. The performance metrics look promising. The final portion of phase 2 is to optimize the communication mechanisms. Subsequent work on different aspects of this are required to provide a viable solutions in various areas. Version 3 of EPICS is able to provide a platform for implementing channel based control, because the channel and attributes for time stamping, alarm, display and control were narrow, well defined, and complete. To extend EPICS functionality beyond this, it is necessary to define attributes needed for archive data, array, image data, and directory services. The proper handling of several array types enables the development of middle layer servers such as orbit and bump control in accelerators. Phase 1 should produce a well defined, reviewed, and agreed upon definition of the metadata required for these services. A Phase 2 grant would provide tools that implemented archiving, general array, imaging, and directory applications.
SPOCS implements a graph-based ortholog prediction method to generate a simple tab-delimited table of orthologs, and in addition, html files that provide a visualization of the ortholog/paralog relationships to which gene/protein expression metadata may be overlaid.
As part of this project work, researchers from Vanderbilt University, Fermi National Laboratory and Illinois Institute of technology developed a real-time cluster fault-tolerant cluster monitoring framework. This framework is open source and is available for download upon request. This work has also been used at Fermi Laboratory, Vanderbilt University and Mississippi State University across projects other than LQCD. The goal for the scientific workflow project is to investigate and develop domain-specific workflow tools for LQCD to help effectively orchestrate, in parallel, computational campaigns consisting of many loosely-coupled batch processing jobs. Major requirements for an LQCD workflow system include: a system to manage input metadata, e.g. physics parameters such as masses, a system to manage and permit the reuse of templates describing workflows, a system to capture data provenance information, a systems to manage produced data, a means of monitoring workflow progress and status, a means of resuming or extending a stopped workflow, fault tolerance features to enhance the reliability of running workflows. Requirements for an LQCD workflow system are available in documentation.
The goals of NECWGDN were to establish integrated geospatial databases that interfaced with existing open-source (water.html) environmental data server technologies (e.g., HydroDesktop) and included ecological and human data to enable evaluation, prediction, and adaptation in coastal environments to climate- and human-induced threats to the coastal marine resources within the Gulf of Maine. We have completed the development and testing of a "test bed" architecture that is compatible with HydroDesktop and have identified key metadata structures that will enable seamless integration and delivery of environmental, ecological, and human data as well as models to predict threats to end-users. Uniquely this database integrates point as well as model data and so offers capacities to end-users that are unique among databases. Future efforts will focus on the development of integrated environmental-human dimension models that can serve, in near real time, visualizations of threats to coastal resources and habitats.
Provenance is dened as information about the origin of objects, a concept that applies to both physical and digital objects and often overlaps both. The use of provenance in systems designed for research is an important but forgotten feature. Provenance allows for proper and exact tracking of information, its use, its lineage, its derivations and other metadata that are important for correctly adhering to the scien- tic method. In our project's prescribed use of provenance, researchers can determine detailed information about the use of sensor data in their experiments on ORNL's Flexible Research Platforms (FRPs). Our project's provenance system, Provenance Data Management System (ProvDMS), tracks information starting with the creation of information by an FRP sensor. The system determines station information, sensor information, and sensor channel information. The system allows researchers to derive generations of experiments from the sensor data and tracks their hierarchical flow. Key points can be seen in the history of the information as part of the information's workflow. The concept of provenance and its usage in science is relatively new and while used in other cases around the world, our project's provenance diers in a key area. To keep track of provenance, most systems must be designed or redesigned around the new provenance system. Our system is designed as a cohesive but sepa- rate entity and allows for researchers to continue using their own methods of analysis without being constrained in their ways in order to track the provenance. We have designed ProvDMS using a lightweight provenance library, Core Provenance Library (CPL) v.6 In addition to keeping track of sensor data experiments and its provenance, ProvDMS also provides a web-enabled visualization of the inheritance.
The exploration of large parameter spaces in search of problem solution and uncertainty quantifcation produces very large ensembles of data. Processing ensemble data will continue to require more resources as simulation complexity and HPC platform throughput increase. More tools are needed to help provide rapid insight into these data sets to decrease manual processing time by the analyst and to increase knowledge the data can provide. One such tool is Tecplot Chorus, whose strengths are visualizing ensemble metadata and linked images. This report contains the analysis and conclusions from evaluating Tecplot Chorus with an example problem that is relevant to Sandia National Laboratories. This report documents a preliminary evaluation of Tecplot Chorus for analyzing ensemble data from CTH simulations. The project that funded this report and evaluation is also evaluating and guiding development with SNL’s Slycat. Slycat and Tecplot Chorus each have their strengths, weaknesses, and overlapping capabilities. It is quite likely that, as the scale of ensemble data increases, both of these tools (and possibly others) will be needed for different processing goals. This report will focus on Tecplot Chorus and its application to an example ensemble of data supplied by David J. Peterson and John P. Korbin; this example is of a flyer plate impact and weld study henceforth referred to as CTH Impact Example. This evaluation also defines a workflow for analysts that can help reduce the time and resources for processing ensemble data.
The US Department of Energy Office of Geothermal Technologies (OGT) is funding and overseeing the development of the National Geothermal Data System (NGDS), a distributed information system providing access to integrated data in support of, and generated in, all phases of geothermal development. NGDS is being built in an open paradigm and will employ state-of-the-art informatics approaches and capabilities to advance the state of geothermal knowledge in the US. This document presents guidelines related to provision and interchange of data assets in the context of the National Geothermal Data System. It identifies general specifications for NGDS catalog metadata and data content, and provides specific instructions for preparation and submission of data assets by OGT-funded projects.
The LLVM intermediate representation (IR) lacks semantic constructs for depicting common high-performance operations such as parallel and concurrent execution, communication and synchronization. Currently, representing such semantics in LLVM requires either extending the intermediate form (a signi cant undertaking) or the use of ad hoc indirect means such as encoding them as intrinsics and/or the use of metadata constructs. In this paper we discuss a work in progress to explore the design and implementation of a new compilation stage and associated high-level intermediate form that is placed between the abstract syntax tree and when it is lowered to LLVM's IR. This highlevel representation is a superset of LLVM IR and supports the direct representation of these common parallel computing constructs along with the infrastructure for supporting analysis and transformation passes on this representation.
The following summaries are provided as fulfillment of milestone M4SF-17SN080305022 and represent international coordination activities in disposal research funded by the US DOE Spent Fuel and Waste Storage and Technologies (SFWST) Campaign during Fiscal Year 2017: SFWST funded bi-lateral interactions with Taiwan, OECD-NEA Repository Metadata (RepMet) project, SFWST funded bi-lateral interactions with the Republic of Korea.
Over the last three decades DOE made a large investment in field-scale experiments in order to understand the role of terrestrial ecosystems in the global carbon cycle, and forecast how carbon cycling will change over the next century. The Smithsonian Environmental Research Center received one of the first awards in this program and managed two long-term studies (25 years and 10 years) with a total of approximately $10 million of support from DOE, and many more millions leveraged from the Smithsonian Institution and agencies such as NSF. The present DOE grant was based on the premise that such a large investment demands a proper synthesis effort so that the full potential of these experiments are realized through data analysis and modeling. The goal of the this grant was to archive legacy data from two major elevated carbon dioxide experiments in DOE databases, and to engage in synthesis activities using these data. Both goals were met. All datasets deemed a high priority for data synthesis and modeling were prepared for archiving and analysis. Many of these datasets were deposited in DOE’s CDIAC, while others are being held at the Oak Ridge National Lab and the Smithsonian Institution until they can be received by DOE’s new ESS-DIVE system at Berkeley Lab. Most of the effort was invested in researching and re-constituting high-quality data sets from a 30-year elevated CO 2 experiment. Using these data, the grant produced products that are already benefiting climate change science, including the publication of new coastal wetland allometry equations based on 9,771 observations, public posting of dozens of datasets, metadata and supporting codes from long-term experiments at the Global Change Research Wetland, and publication of two synthetic data papers on scrub oak forest responses to elevated CO 2 . In addition, three papers are in review or nearing submission reporting unexpected long-term patterns in ecosystem responses to elevated CO 2 and nitrogen in a coastal wetland.