Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “heterogeneous data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2019

The SkyView Data Model: Handling Heterogeneous Data Using Dynamic Software

The SkyView Virtual Telescope allows users to query over 75 distinct datasets of astronomical imagery including more than 190 distinct surveys. These holdings are extraordinarily diverse ranging over more than 15 orders of magnitude in frequency and5 orders of magnitude in resolution. To accommodate supporting these data, SkyView has developed a simple data model for surveys. Nor surprisingly, given the diverse origins of the datasets included, some surveys do not easily fit within this model. The conventional approach to dealing with such outliers is to build a layer on top of the actual data which more faithfully adheres to the data model. With SkyView we have used a somewhat alternative approach. We allow the data model to define new behaviors in the SkyView code itself either by customizing the use of existing capabilities or pointing to new classes that have non-standard behavior. This approach is feasible since in modern computer languages the code that is run can be selected dynamically with new classes completely overriding traditional behavior. This paper describes this approach and considers how it may become more useful as our software structures become increasingly dynamically linked.

Thomas McGlynn↗
DOE OSTI2021

A Lakehouse Architecture for the Management and Analysis of Heterogeneous Data for Biomedical Research and Mega-biobanks

Data Lakehouse is a new paradigm in data architectures that embodies and integrates already established concepts for the systematic management of disparate, large-scale data – a data lake for heterogeneous data management, use of open standards for high-performance querying, and systematic maintenance of the data "freshness". In addition to being a new concept, the data lakehouse is also still a conceptual construct. Many projects that use the lakehouse require maturing, empirical studies, and specific implementations. In this paper, we present our implementation of the data lakehouse concept in a biomedical research and health data analytics domain, and we discuss the implementation of some unique and novel features such as support for specialized access controls in support of HIPAA regulation and IRB protocols, and support for the FAIR standard.

Begoli, Edmon↗
DOE OSTI2024

Heterogeneous Data Type Mixture Model

This code performs estimation of a mixture model involving heterogeneous data types. The resulting model can be used to calculate likelihoods of underlying model components.

Kaplan, AlanD↗
DOE OSTI2026

Risk-Aware Measurement Synchronization and Recovery for DSSE With Heterogeneous Data Sources

Power distribution systems are increasingly integrating heterogeneous sensors with varying data reporting rates and types, which pose challenges to achieving observability at the desired temporal resolution of distribution system state estimation (DSSE). Multisensor failures caused by extreme events exacerbate these issues, introducing substantial uncertainties into DSSE. This article proposes a novel solution to these challenges by ensuring high-resolution system observability despite heterogeneous data sources and multisensor failures. First, a deep learning architecture combining long short-term memory (LSTM) and graph convolutional network (GCN) is employed to synchronize meters with different reporting rates, aiming to achieve system observability. A random-walk-model-based approach is introduced to generate pseudo-measurements while properly characterizing their uncertainties under multisensor failures. Finally, a disaster-risk-informed observability metric (RiOM) is defined to quantify the uncertainty associated with state estimation results. The proposed framework offers deeper insights into the system observability on the fly compared with conventional analysis. The effectiveness of the framework is demonstrated on an IEEE standard test case and a large-scale real-world distribution feeder in mid-Minnesota in the U.S.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

A Data Processing Pipeline To Extract A Knowledge Graph From Heterogeneous Data For Socio-technical Analysis Of Critical Infrastructure Influence

The code is written in Python and consists of the following pipeline that is implemented in Apache Airflow. This pipeline intends to understand the companies that are directly or indirectly involved with a type of critical infrastructure system at some point in that system's lifecycle. The pipeline takes a configuration file that specifies a list of initial companies to consider, a geographic region of interest, and a set of SEC form types as well as other data sources (e.g. CrunchBase) from which to extract entities and relations. There are four main components to this pipeline as currently implemented: Entity Extraction, Network Construction, Analysis, and Visualization. First, Entity Extraction, is implemented as the `topear-extract_organizations` Apache Airflow workflow. Given an initial query that specifies a geographic region of interest and a time interval, the software will extract CI facilities of interest and organizations that have a direct influence relationship to those facilities (e.g. ownership). During the course of the LDRD, we focused on Electric Vehicle charging stations and this information is available via the Department of Energy (DOE) database on fueling stations maintained by NREL. Within the context of the DOE CESER project, we have focused on Battery Energy Storage Systems (BESS). Second, the Network Extraction component will iteratively construct a social network graph given the set of organizations and people extracted in the previous step. Organizations (and eventually People if desired) are then fed as a query to the `topgear-construct_social_network` Apache Airflow workflow which given a set of initial companies and data sets (e.g. SEC EDGAR form types, OpenCorporates, Crunchbase). This Airflow workflow will iteratively query such data sources to discover relationships with new organizations and people. For example, this module can iteratively query SEC EDGAR for metadata that documents the number of each type of form for the given set of companies and their location. This forms metadata represents a catalog of data sources from SEC EDGAR for the extracted social network knowledge graph. The pipeline then downloads these forms from the website and saves them in a build directory for further processing. These documents are then parsed for entities and relations. Again, we note that in additional to SEC data sources, this step can also pull in information on organizations via API services such as CrunchBase and OpenCorporates or bulk data sources. At the end of this step, the resultant social network, the Critical Infrastructure network, and the edges that encode relationships between organizations and CI facilities, form the Adversarial Socio-Technical Network (ASTN) that informs the analysis. Third, the Analysis component processes these generated ASTN. Previously, that has included the ability to compare prevalence of different vendors for a given infrastructure component type across different regions as well as identify common public and private investors across those vendors. This was demonstrated for EV Charging Stations across several different metropolitan areas within an IEEE PES GridEdge publication. More recently, we have looked at ways to identify infrastructure owners and operators of BESS with the most nameplate capacity across different states as well as other indictors of risk resulting from changes in ownership over time. Finally, the Visualization component consists of an HTML/CSS/JS framework by which users can interact geospatial, operational, and organizational relationships across a given portfolio of Critical Infrastructure facilities. The objective is to provide a library of UI/UX modules that can be repurposed for stakeholder-specific dashboards. All of the modules are related via a common event model that enables UI actions in one view to percolate across the other views.

Weaver, Gabriel [Idaho National Laboratory (INL), ↗
DOE OSTI2024

Federated Machine Learning-Based Anomaly Detection System for Synchrophasor Network Using Heterogeneous Data Sets: Preprint

Synchrophasor technology is widely deployed in the energy management system to monitor the grid health at micro level and perform necessary corrective actions in real time; however, integrated phasor devices and data aggregators are exposed to several cybersecurity threats. This paper proposes a federated ML(FML)-based ADS to detect several data integrity attacks in the synchrophasor network. The proposed approach integrates the horizontal FML technique and consists of substation-based local models and a control center-based global model. The proposed methodology includes training local models using heterogeneous data sets that include network and grid information and updating the global model through multiple iterations by sharing model gradients. Finally, the trained global model is applied to identify cyberattacks, normal operation, and physical events. To validate the proof of concept, we used synthetic data sets generated by Mississippi State University and Oak Ridge National Laboratory for training and testing the classification models using the National Renewable Energy Laboratory's high performance computing resources. Our experimental results, computed through several performance measures, reveal that the proposed approach shows consistent performance during the binary, three-class, and multiclass classifications while ensuring privacy of synchrophasor data.

anomaly detection system↗
DOE OSTI2025

Graph-Learning-Assisted State and Event Tracking for Solar-Penetrated Power Grids with Heterogeneous Data Sources

Unlike transmission systems, distribution systems do not typically contain sufficient metering to enable real-time state estimation. The lack of sufficient real-time measurements prohibits accurate and timely monitoring of the state of distribution systems. As a result, control and optimal operation of distribution systems, especially those containing large numbers of renewable generation units are not possible without proper data and information about the current state of the system. The main motivation of this project is to address this shortcoming by developing an approach which provides “predicted” real-time measurements so that they can be used to execute a distribution system state estimator. Thus, the objective of the project is to make the distribution systems fully observable, such that the hosting capacity for solar generation can be accurately estimated, and unnecessary solar curtailments can be avoided. In order to accomplish this goal, the project investigated the use of a grid-model-informed machine learning (ML) tool which integrates heterogeneous data streams obtained from AMI meters, SCADA as well as PMU measurements and created synchronous measurement snapshots for the state estimator (SE); and developed a hybrid robust SE which provides not only accurate state estimates but also real-time feedback for the ML model refinement.

14 SOLAR ENERGY↗
DOE OSTI2024

3D reconstruction of an inertial-confinement fusion implosion with neural networks using multiple heterogeneous data sources

3D asymmetries are major degradation mechanisms in inertial-confinement fusion implosions at the National Ignition Facility (NIF). These asymmetries can be diagnosed and reconstructed with the neutron imaging system (NIS) on three lines of sight around the NIF target chamber. Conventional tomographic reconstructions are used to reconstruct the 3D morphology of the implosion using NIS [Volegov et al., J. Appl. Phys. 127, 083301 (2020)], but the problem is ill-posed with only three imaging lines of sight. Asymmetries can also be diagnosed with the real-time neutron activation diagnostics (RTNAD) and the neutron time-of-flight (nToF) suite. Since the NIS, RTNAD, and nToF each sample a different part of the implosion using different physical principles, we propose that it is possible to overcome the limitations of too few imaging lines of sight by performing 3D reconstructions that combine information from all three heterogeneous data sources. This work presents a new machine learning-based reconstruction technique to do just this. By using a simple physics model and group of neural networks to map 3D morphologies to data, this technique can easily account for data of multiple different types. A simple proof-of-principle is presented, demonstrating that this technique can accurately reconstruct a hot-spot shape using synthetic primary neutron images and a hot-spot velocity vector. In particular, the hot-spot’s asymmetry, quantified as spherical harmonic coefficients, is reconstructed to within ±4% of the radius in 90% of test cases. In the future, this technique will be applied to actual NIS, RTNAD, and nToF data to better understand 3D asymmetries at the NIF.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2024

Multitask methods for predicting molecular properties from heterogeneous data

Data generation remains a bottleneck in training surrogate models to predict molecular properties. We demonstrate that multitask Gaussian process regression overcomes this limitation by leveraging both expensive and cheap data sources. In particular, we consider training sets constructed from coupled-cluster (CC) and density functional theory (DFT) data. We report that multitask surrogates can predict at CC-level accuracy with a reduction in data generation cost by over an order of magnitude. Of note, our approach allows the training set to include DFT data generated by a heterogeneous mix of exchange–correlation functionals without imposing any artificial hierarchy on functional accuracy. More generally, the multitask framework can accommodate a wider range of training set structures—including the full disparity between the different levels of fidelity—than existing kernel approaches based on Δ-learning although we show that the accuracy of the two approaches can be similar. Consequently, multitask regression can be a tool for reducing data generation costs even further by opportunistically exploiting existing data sources.

Chemistry↗
DOE OSTI2026

ZENN: A thermodynamics-inspired computational framework for heterogeneous data–driven modeling

Traditional entropy-based methods—such as cross-entropy loss in classification problems—have long been essential tools for representing the information uncertainty and physical disorder in data and for developing artificial intelligence algorithms. However, the rapid growth of data across various domains has introduced new challenges, particularly the integration of heterogeneous datasets with intrinsic disparities. To address this, we introduce a zentropy-enhanced neural network (ZENN), extending zentropy theory into the data science domain via intrinsic entropy, enabling more effective learning from heterogeneous data sources. ZENN simultaneously learns both energy and intrinsic entropy components, capturing the underlying structure of multisource data. To support this, we redesign the neural network architecture to better reflect the intrinsic properties and variability inherent in diverse datasets. We demonstrate the effectiveness of ZENN on classification tasks and energy landscape reconstructions, showing its superior generalization capabilities and robustness-particularly in predicting high-order derivatives. In image and text classification tasks, ZENN demonstrates superior generalization by introducing a learnable temperature variable that models latent multisource heterogeneity, allowing it to surpass state-of-the-art models on CIFAR-10/100, BBC News, and AG News. As a practical application in materials science, we employ ZENN to reconstruct the Helmholtz energy landscape of Fe3Pt using data generated from density functional theory and capture key material behaviors, including negative thermal expansion and the critical point in the temperature–pressure space. Overall, this work presents a zentropy-grounded framework for data-driven machine learning, positioning ZENN as a versatile and robust approach for scientific problems involving complex, heterogeneous datasets.

36 MATERIALS SCIENCE↗
DOE OSTI2019

SONET: a semantic ontological network graph for managing points of interest data heterogeneity

Scalability, standardization, and management are important issues when working with very large Volunteered Geographic Information (VGI). VGI is a rich and valuable source of Points of Interest (POI) information, but its inherent heterogeneity in content, structure, and scale across sources present major challenges for interlinking data sources for analysis. To be useful at scale, the raw information needs to be transformed into a standardized schema that can be easily and reliably used by data analysts. In this work, we tackle the problem of unifying POI categories (e.g. restaurants, temple, and hotel) across multiple data sources to aid in improving land use maps and population distribution estimation as well as support data analysts wishing to fuse multiple data sources with the OpenStreetMap (OSM) mapping platform or working with projects that are already configured in the OSM schema and wish to add additional sources of information. Graph theory and its implementation through the SONET graph database, provides a programmatic way to organize, store, and retrieve standardized POI categories at multiple levels of abstraction. Additionally, it addresses category heterogeneity across data sources by standardizing and managing categories in a way that makes cross-domain analysis possible.

Palumbo, Rachel↗
DOE OSTI2019

Infrared and radar fusion detection method based on heterogeneous data preprocessing

The difficulty in infrared and radar fusion target detection lies in the pre-processing steps such as sample reduction and feature reduction of heterogeneous sensors. For the problem of data volume redundancy, HVS sample reduction is proposed to reduce the amount of data in the infrared data set. Infrared and radar joint feature vectors are constructed through nearest neighbor data association. For the problem of feature redundancy, a weighted mutual information feature selection method based on prior information is proposed according to the difference of feature sources. The fusion data set is then used for classification. The experiments show that the HVS-based sample reduction and the prior weighted feature selection achieved a higher detection probability and a shorter detection time, and improved the robustness in the case of large differences between the training set and the test set.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2019

Resilient Distributed Parameter Estimation With Heterogeneous Data

This paper studies resilient distributed estimation under measurement attacks. A set of agents each makes successive local, linear, noisy measurements of an unknown vector field collected in a vector parameter. The local measurement models are heterogeneous across agents and may be locally unobservable for the unknown parameter. An adversary compromises some of the measurement streams and changes their values arbitrarily. The agents' goal is to cooperate over a peer-to-peer communication network to process their (possibly compromised) local measurements and estimate the value of the unknown vector parameter. We present SAGE, the Saturating Adaptive Gain Estimator, a distributed, recursive, consensus + innovations estimator that is resilient to measurement attacks. We demonstrate that, as long as the number of compromised measurement streams is below a particular bound, then, SAGE guarantees that all of the agents' local estimates converge almost surely to the value of the parameter. The resilience of the estimator - i.e., the number of compromised measurement streams it can tolerate - does not depend on the topology of the inter-agent communication network. Finally, we illustrate the performance of SAGE through numerical examples.

42 ENGINEERING↗
DOE OSTI2021

Using heterogeneous data to identify signatures of dengue outbreaks at fine spatio-temporal scales across Brazil

Dengue virus remains a significant public health challenge in Brazil, and seasonal preparation efforts are hindered by variable intra- and interseasonal dynamics. Here, we present a framework for characterizing weekly dengue activity at the Brazilian mesoregion level from 2010–2016 as time series properties that are relevant to forecasting efforts, focusing on outbreak shape, seasonal timing, and pairwise correlations in magnitude and onset. In addition, we use a combination of 18 satellite remote sensing imagery, weather, clinical, mobility, and census data streams and regression methods to identify a parsimonious set of covariates that explain each time series property. The models explained 54% of the variation in outbreak shape, 38% of seasonal onset, 34% of pairwise correlation in outbreak timing, and 11% of pairwise correlation in outbreak magnitude. Regions that have experienced longer periods of drought sensitivity, as captured by the “normalized burn ratio,” experienced less intense outbreaks, while regions with regular fluctuations in relative humidity had less regular seasonal outbreaks. Both the pairwise correlations in outbreak timing and outbreak trend between mesoresgions were best predicted by distance. Our analysis also revealed the presence of distinct geographic clusters where dengue properties tend to be spatially correlated. Forecasting models aimed at predicting the dynamics of dengue activity need to identify the most salient variables capable of contributing to accurate predictions. Our findings show that successful models may need to leverage distinct variables in different locations and be catered to a specific task, such as predicting outbreak magnitude or timing characteristics, to be useful. This advocates in favor of “adaptive models” rather than “one-size-fits-all” models. The results of this study can be applied to improving spatial hierarchical or target-focused forecasting models of dengue activity across Brazil.

59 BASIC BIOLOGICAL SCIENCES↗