Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data integration”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

OEDI—Solar Grid Integration Data and Analytics Library

As a part of the Open Energy Data Initiative, this effort aims to develop and demonstrate novel distribution state estimation, control optimization, and transient analysis as well as provide access to data, data integration, and mapping information. More specifically, the focus of the effort will be on physics-based distribution system state estimation, hybrid (physics-based and machine learning) distribution optimal power flow, and event detection/analysis for solar integration and analytics. This work will enable reproducible, robust, replicable, and generalizable R&D in simulation and emulation of solar system integration. These test models and datasets will provide an integrated library for developing and testing power system operation technologies. To make the library user-friendly, this project will provide data curation tools such as data translators, mapping scripts and APIs, database schemas and metadata, interfaces and user dashboard, source code for the reference algorithms, description of the use-cases/scenarios, and comprehensive information on all the assumptions.

14 SOLAR ENERGY↗
DOE OSTI2026

BASIN-3D Data Integration for Selected ARM Data Field Campaign Report

The purpose of this data services request was to demonstrate integration of the Atmospheric Radiation Measurement (ARM) User Facility’s “met” datastreams with time series data from other earth science data sources using the BASIN-3D data synthesis software tool. BASIN-3D is an open-source Python library that enables researchers to integrate data across configured public and private data sources. It provides a common query language for researchers to request measurement locations and time series data based on specified locations, variables, time period, statistics, aggregation, and data quality. BASIN-3D acquires the data that match the query from each configured data source and translates the results into harmonized vocabularies, thus reducing researchers' data-wrangling effort. In addition, because the queries are executed on demand, researchers can easily regenerate their synthesized data sets as new data and/or data updates become available, eliminating one-off data products. BASIN-3D can output data using a variety of different data structures for end-user applications including Python pandas data frames and hdf5 output formats.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Towards Lightweight Data Integration Using Multi-Workflow Provenance and Data Observability

Modern large-scale scientific discovery requires multidisciplinary collaboration across diverse computing facilities, including High Performance Computing (HPC) machines and the Edge-to-Cloud continuum. Integrated data analysis plays a crucial role in scientific discovery, especially in the current AI era, by enabling Responsible AI development, FAIR, Reproducibility, and User Steering. However, the heterogeneous nature of science poses challenges such as dealing with multiple supporting tools, cross-facility environments, and efficient HPC execution. Building on data observability, adapter system design, and provenance, we propose MIDA: an approach for lightweight runtime Multi-workflow Integrated Data Analysis. MIDA defines data observability strategies and adaptability methods for various parallel systems and machine learning tools. With observability, it intercepts the dataflows in the background without requiring instrumentation while integrating domain, provenance, and telemetry data at runtime into a unified database ready for user steering queries. We conduct experiments showing end-to-end multi-workflow analysis integrating data from Dask and MLFlow in a real distributed deep learning use case for materials science that runs on multiple environments with up to 276 GPUs in parallel. We show near-zero overhead running up to 100,000 tasks on 1,680 CPU cores on the Summit supercomputer.

Santos Souza, Renan↗
DOE OSTI2023

Towards Integrating Data Quality Assessments and Radiometer Uncertainty for Determining the Expanded Uncertainty of Three-Component Solar Radiation Measurements

Accurate solar irradiance data are fundamental for determining the design and performance characteristics of photovoltaic systems. The uncertainty of solar irradiance measurements depends on many factors including radiometer design, calibration, installation, maintenance, and operational environment. The key contributors to this uncertainty can be classified as the measurement uncertainty of a particular radiometer and the operational uncertainty determined for the time of measurement. Radiometer measurement uncertainty estimates (U R ) can be based on well-established methods used as part of the radiometer calibration process. Estimates of operational uncertainties (U o ) require consideration of additional site-specific factors that affect data quality. A method is needed for establishing the accuracy of solar irradiance data by integrating an existing data quality process and measurement uncertainty estimates for specific radiometers. An algorithm has been developed to integrate data quality analyses and measurement uncertainty estimates for three-component solar irradiance data: global horizontal (total hemispheric) irradiance, direct normal (beam) irradiance, and diffuse horizontal (sky) irradiance collected at one- to 60-minute intervals. The algorithm has been tested using one-minute irradiance measurements. The goal of the project is to distribute a user-friendly software package based on the new algorithm.

data integrity↗
DOE OSTI2023

An integrated data management and informatics framework for continuous drug product manufacturing processes: A case study on two pilot plants

The pharmaceutical industry continuously looks for ways to improve its development and manufacturing efficiency. In recent years, such efforts have been driven by the transition from batch to continuous manufacturing and digitalization in process development. To facilitate this transition, integrated data management and informatics tools need to be developed and implemented within the framework of Industry 4.0 technology. Here, in this regard, the work aims to guide the data integration development of continuous pharmaceutical manufacturing processes under the Industry 4.0 framework, improving digital maturity and enabling the development of digital twins. This paper demonstrates two instances where a data integration framework has been successfully employed in academic continuous pharmaceutical manufacturing pilot plants. Details of the integration structure and information flows are comprehensively showcased. Approaches to mitigate concerns in incorporating complex data streams, including integrating multiple process analytical technology tools and legacy equipment, connecting cloud data and simulation models, and safeguarding cyber-physical security, are discussed. Critical challenges and opportunities for practical considerations are highlighted.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Towards physics-inspired data-driven weather forecasting: integrating data assimilation with a deep spatial-transformer-based U-NET in a case study with ERA5

Abstract. There is growing interest in data-driven weather prediction (DDWP), e.g., using convolutional neural networks such as U-NET that are trained on data from models or reanalysis. Here, we propose three components, inspired by physics, to integrate with commonly used DDWP models in order to improve their forecast accuracy. These components are (1) a deep spatial transformer added to the latent space of U-NET to capture rotation and scaling transformation in the latent space for spatiotemporal data, (2) a data-assimilation (DA) algorithm to ingest noisy observations and improve the initial conditions for next forecasts, and (3) a multi-time-step algorithm, which combines forecasts from DDWP models with different time steps through DA, improving the accuracy of forecasts at short intervals. To show the benefit and feasibility of each component, we use geopotential height at 500 hPa (Z500) from ERA5 reanalysis and examine the short-term forecast accuracy of specific setups of the DDWP framework. Results show that the spatial-transformer-based U-NET (U-STN) clearly outperforms the U-NET, e.g., improving the forecast skill by 45 %. Using a sigma-point ensemble Kalman (SPEnKF) algorithm for DA and U-STN as the forward model, we show that stable, accurate DA cycles are achieved even with high observation noise. This DDWP+DA framework substantially benefits from large (O(1000)) ensembles that are inexpensively generated with the data-driven forward model in each DA cycle. The multi-time-step DDWP+DA framework also shows promise; for example, it reduces the average error by factors of 2–3. These results show the benefits and feasibility of these three components, which are flexible and can be used in a variety of DDWP setups. Furthermore, while here we focus on weather forecasting, the three components can be readily adopted for other parts of the Earth system, such as ocean and land, for which there is a rapid growth of data and need for forecast and assimilation.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

Data Integration in Poplar: ‘Omics Layers and Integration Strategies

Populus trichocarpa is an important biofuel feedstock that has been the target of extensive research and is emerging as a model organism for plants, especially woody perennials. This research has generated several large ‘omics datasets. However, only few studies in Populus have attempted to integrate various data types. This review will summarize various ‘omics data layers, focusing on their application in Populus species. Subsequently, network and signal processing techniques for the integration and analysis of these data types will be discussed, with particular reference to examples in Populus.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2019

Method and application of data integration at a nuclear power plant. Light Water Reactor Sustainability Program report

The data of a nuclear power plant (NPP) is stored in silos for the various plant tools. A previous study conducted by the LWRS program identified the different data sources that can benefit online monitoring in an NPP. These data have different structures and tools and are therefore used independently. Integrating data from various sources can result in multiple benefits to online monitoring by enhancing the ability to compare data, reduce statistical uncertainty due to a larger data set, increase data heterogeneity in space and time to reveal confounded information, develop intelligent methods, and share data. Data integration can result in direct cost saving by reducing the amount of effort needed to find needed information. It can also enable establishing a data warehouse, the backbone repository of data for an NPP, and is necessary to evolve NPPs to a holistic online monitoring approach and automate labor-intensive activities. Whether within an NPP or for a fleet of NPPs, deploying a data warehouse in a plant or data broker for a fleet of plants requires overcoming multiple challenges, including the lack of a standard form of storing and using data and the lack of integration methods to compare and map the different data sources. This effort demonstrates two methods and applications for data integration through an application integration pilot and a data fusion pilot. The application integration is targeted towards the software interfacing of data and data management. This approach is commonly used by data integrators. The data fusion pilot is targeted towards intelligently mapping or coupling data for optimized data use. The pilots were performed in collaboration with two NPPs.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2026

REDI – Readiness Engine for Data Integration

The Readiness Engine for Data Integration (REDI) is an open-source framework for automating, standardizing, and assessing the process of preparing scientific data for AI training. REDI implements a five-stage pipeline (ingest, preprocess, transform, structure, output) with per-stage provenance instrumentation via Flowcept, domain-aware transformation logic (PII anonymization, regridding, graph encoding, and more), and built-in readiness assessment and validation modes. REDI has been evaluated across climate, proteomics, materials science, and nuclear fusion datasets, demonstrating near-ideal parallel scaling to 100 nodes on OLCF's Frontier system. REDI is deployable as an agent-callable skill in coding environments such as Claude Code and OpenAI Codex, and is complemented by SetGo for FAIR compliance and catalog publication.

Brewer, Wesley [Oak Ridge National Laboratory (ORN↗
DOE OSTI2020

Understanding Impacts of Data Integrity Attacks on Transactive Control Systems

With the rapid growth of internet-connected smart devices capable of exchanging energy price information and adaptively controlling energy consumption of connected loads, the role of transactive control is expected to become more prominent in the modern grid. Transactive control systems integrate the wholesale and retail energy markets, and enable active participation of end users, thereby playing a key role in managing the rising number of distributed assets in the smart grid. The use of internet for the communication of data between the controllers at the building, distribution, and transmission levels makes the system susceptible to cyber-attacks. A skilled adversary can potentially manipulate the exchanged data with the intention of inflicting damage to the system. In this paper, four categories of metrics are put forth for evaluating the performance of the system when under data integrity attacks. A model for scaling-based data integrity attack has been described, and co-simulations have been performed on a 240 bus WECC transmission system model with detailed modeling of distribution systems at specific buses. Our results show that scaling-based data integrity attacks can have non-trivial impacts on the operational, financial, and comfort aspects of the transactive control system.

Smart grid, transactive control system, Buildings-↗
DOE OSTI2020

2018 Renewable Energy Grid Integration Data Book

The 2018 Renewable Energy Grid Integration Data Book identifies the status, key trends, challenges, and solutions of renewable energy grid integration in a highly visual format. It provides an overview of selected key grid integration metrics that represent complex interactions among generation characteristics, market rules, and environmental and safety factors, which may vary by geography, season, and time of day; and the data book presents metrics that either indicate how much variable renewable energy (VRE) is currently being integrated onto the grid or measure factors that may increase or decrease the challenges associated with integrating VRE generation onto the grid.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2019

Bioinformatics Resource Manager: A systems biology web tool for microRNA and omics data integration

Background: The Bioinformatics Resource Manager (BRM) is a web-based tool developed to facilitate identifier conversion and data integration for Homo sapiens (human), Mus musculus (mouse), Rattus norvegicus (rat), Danio rerio (zebrafish), and Macaca mulatta (macaque), as well as perform orthologous conversions among the supported species. In addition to providing a robust means of identifier conversion, BRM also incorporates a suite of miRNA-target databases upon which to query target genes or to perform reverse target lookups using gene identifiers. Results: BRM has the capability to perform cross-species identifier lookups across common identifier types, directly integrate datasets across platform or species by performing identifier retrievals in the background, and retrieve miRNA targets from multiple databases simultaneously and integrate the resulting gene targets with experimental mRNA data. Here we use workflows provided in BRM to integrate RNAseq data across species to identify common biomarkers of exposure after treatment of human lung cells and zebrafish to benzo[a]pyrene (BAP). We further use the miRNA Target workflow to experimentally determine the role of miRNAs as regulators of BAP toxicity and identify the predicted functional consequences of miRNA-target regulation in our system. The output from BRM can easily and directly be uploaded to freely available visualization tools for further analysis. From these examples, we were able to identify an important role for several miRNAs as potential regulators of BAP toxicity in human lung cells associated with cell migration, cell communication, cell junction assembly and regulation of cell death. Conclusions: Overall, BRM provides bioinformatics tools to assist biologists having minimal programming skills with analysis and integration of high-content omics’ data from various transcriptomic and proteomic platforms. BRM workflows were developed in Java and other open-source technologies and are served publicly using Apache Tomcat at https://cbb.pnnl.gov/brm/.

bioinformatics, MicroRNA, systems biology, Genomic↗
DOE OSTI2020

Data Integration Aggregated Model For Nuclear Deployment

The data of a nuclear utility or plant are stored in various isolated forms in different systems. These data have different structures and tools and are, therefore, used independently. The integration of data sources is often needed and is performed manually and on an as-needed basis. The integration of the data would enable creating a data warehouse for a single nuclear power plant (NPP) or multiple plants. A data model contains objects that make up a given domain along with their attributes and relationships and could be used as a standard for exchange between applications for simple integration and data aggregation. After a survey of existing models, it was determined that a new model was required to meet the scope of this problem. DIAMOND has been developed in the form of an ontology to meet these needs. An ontology is simply a data model that also acts as a taxonomy for the domain, organizing the objects in a hierarchy with the most general objects at the top.

Browning, JerenM↗
DOE OSTI2020

Detection and Diagnosis of Data Integrity Attacks in Solar Farms Based on Multilayer Long Short-Term Memory Network

Photovoltaic (PV) systems are becoming more vulnerable to cyber threats. In response to this emerging concern, developing cyber-secure power electronics converters has received increased attention from the IEEE Power Electronics Society that recently launched a cyber-physical-security initiative. Here this letter proposes a deep sequence learning based diagnosis solution for data integrity attacks on PV systems in smart grids, including dc–dc and dc–ac converters. The multilayer long short-term memory networks are used to leverage time-series electric waveform data from current and voltage sensors in PV systems. The proposed method has been evaluated in a PV smart grid benchmark model with extensive quantitative analysis. As a comparison, we have evaluated classic data-driven methods, including K-nearest neighbor, decision tree, support vector machine, artificial neural network, and convolutional neural network. Comparison results verify performances of the proposed method for detection and diagnosis of various data integrity attacks on PV systems.

42 ENGINEERING↗