Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “clustering algorithm”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 721 records · Page 40
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Semi-Supervised Domain Adaptation for Cross-Survey Galaxy Morphology Classification and Anomaly Detection

In the era of big astronomical surveys, our ability to leverage artificial intelligence algorithms simultaneously for multiple datasets will open new avenues for scientific discovery. Unfortunately, simply training a deep neural network on images from one data domain often leads to very poor performance on any other dataset. Here we develop a Universal Domain Adaptation method DeepAstroUDA, capable of performing semi-supervised domain alignment that can be applied to datasets with different types of class overlap. Extra classes can be present in any of the two datasets, and the method can even be used in the presence of unknown classes. For the first time, we demonstrate the successful use of domain adaptation on two very different observational datasets (from SDSS and DECaLS). We show that our method is capable of bridging the gap between two astronomical surveys, and also performs well for anomaly detection and clustering of unknown data in the unlabeled dataset. We apply our model to two examples of galaxy morphology classification tasks with anomaly detection: 1) classifying spiral and elliptical galaxies with detection of merging galaxies (three classes including one unknown anomaly class); 2) a more granular problem where the classes describe more detailed morphological properties of galaxies, with the detection of gravitational lenses (ten classes including one unknown anomaly class).

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Robust Group Subspace Recovery: A New Approach for Multi-Modality Data Fusion

Robust Subspace Recovery (RoSuRe) algorithm was recently introduced as a principled and numerically efficient algorithm that unfolds underlying Unions of Subspaces (UoS) structure, present in the data. The union of Subspaces (UoS) is capable of identifying more complex trends in data sets than simple linear models. In this work, we build on and extend RoSuRe to prospect the structure of different data modalities individually. We propose a novel multi-modal data fusion approach based on group sparsity which we refer to as Robust Group Subspace Recovery (RoGSuRe). Relying on a bi-sparsity pursuit paradigm and non-smooth optimization techniques, the introduced framework learns a new joint representation of the time series from different data modalities, respecting an underlying UoS model. We subsequently integrate the obtained structures to form a unified subspace structure. The proposed approach exploits the structural dependencies between the different modalities data to cluster the associated target objects. The resulting fusion of the unlabeled sensors’ data from experiments on audio and magnetic data has shown that our method is competitive with other state of the art subspace clustering methods. The resulting UoS structure is employed to classify newly observed data points, highlighting the abstraction capacity of the proposed method.

47 OTHER INSTRUMENTATION↗
DOE OSTI2024

Wholesale Electricity Analysis via Simulation & Learning Experiments (WEASLE): Platform Development and Pilot Competition

This document reports the development of the Wholesale Electricity Analysis via Simulation and Learning Experiments (WEASLE) platform and the pilot competition that was conducted to test the platform. Due to the increasing reliance on variable renewable energy resources for bulk power, the pilot competition, called the Energy Storage Participation Algorithm Competition (ESPA-Comp), was used to test the effect of various market designs on storage utilization and market efficiency. Basic details of the platform are provided, including an overview of the market clearing engine, the battery dispatch and degradation models, electric grid topology and resource mix, and software architecture. Two market designs were tested: a two-settlement market analogous to typical ISO design today, and a multi-settlement market that allows additional forward-trading periods during the real-time market. Results from the pilot competition show that the storage bidding problem is nontrivial and is well suited for future challenges. We find that: 1) all four teams utilized different approaches to the bidding problem, 2) different methodological approaches led to substantially different offer behaviors, 3) resource profits are clustered by team and methodological approach, 4) simulated offers reduced market surplus by about 0.5%, 5) substantially different prices between two-settlement and multi-settlement markets albeit minimal difference in overall market surplus.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

CEGANN: CRYSTAL EDGE GRAPH ATTENTION NEURAL NETWORK

SF-22-156 Machine learning (ML) models and applications in materials design and discovery typically involve the use of feature representations or descriptors followed by a learning algorithm that maps them to user desired properties of interest. Most popular mathematical formulation-based descriptors are not unique across atomic environments and suffer from transferability issues across different application domains and/or material classes. The CEGANN code provides a unified interface to facilitate material characterization across materials across multiple scales (from atomic to mesoscale) and diverse classes of materials ranging from metals oxides, non-metals, and even hierarchical materials such as zeolites and semi ordered materials such as mesophases. CEGANN implements a Graph Attention Network (GAT) type convolution architecture. The details of network architecture can be found in the paper https://doi.org/10.48550/arXiv.2207.10168. The software comes with pretrained examples and dataset for the classification of the following representative systems: (1) Structure-level representation such as space group (2) Structural dimensionality (e.g., bulk, 2D, clusters etc.) (3) Grain boundary identification (4) Nucleation and growth of a zeolite polymorph (5) Characterization of binary mesophases and their phase transitions (6) Growth of ice. The code is written in python programming language.

CHAN, HENRYT↗
DOE OSTI2024

Plasma Surface Interactions: Predicting the Performance and Impact of Dynamic PFC Surfaces

The objective of this project is to develop, and integrate, high-performance simulation tools capable of predicting plasma-facing component (PFC) operating lifetime and the impact of the evolving surface morphology of tungsten-based PFCs on plasma contamination, including the dynamic recycling of fuel species and tritium retention, in future magnetic fusion devices. Establishing a fundamental physical understanding and developing predictive capabilities of plasma-surface interactions (PSI) requires simultaneously addressing complex and diverse physics occurring over a wide range of length (Angstroms to meters) and time (femtoseconds to years) scales, as well as integrating extensive physical processes across the plasma–surface interface. This requires development of not only detailed physics models and computational strategies at each scale, but also algorithms and methods to couple them effectively in a way that can be robustly validated. Deploying these tools requires the continued development and coupling of leadership-scale computational codes to describe the boundary plasma and the evolving PFC surface, as well as a host of simulations that bridge disparate scales to address complex physical and computational issues at the plasma–surface interface in multi-component materials systems for magnetic fusion energy development beyond ITER.

36 MATERIALS SCIENCE↗
DOE OSTI2024

FORESTR: Finding, Organizing, Representing, Explaining, Summarizing, and Thinning Random forests

Random forests have become popular models used for data driven predictions. As a result, random forests are currently used or being considered for high-consequence mission applications in national security, such as the prediction of yield from optical signals and malware detection. While random forests may provide accurate predictions, the complexity of the algorithm causes a lack of interpretability. Random forests are an ensemble of regression or decision trees. Individual regression and decision trees are interpretable, but ensembles are inherently difficult to interpret due to the compilation of many models. We aim to increase the interpretability of random forests by finding patterns in the ensemble of trees that can be used to “thin” (or remove) trees. As a starting point, in this report, we develop a new distance metric for quantifying the similarity between trees based on their topologies (i.e., shapes). We base the metric on a novel distance metric for graphs that is a proper mathematical distance, is invariant to transformations, has registration between graphs, and computes topological evolutions between graphs. We use the tree distance metric to compute tree statistics such as a “mean tree” and to identify clusters of trees. We apply the developed methodology to a toy dataset and a mission relevant product inspection dataset to demonstrate how the metric can provide insight into random forests. Furthermore, we discuss the limitations of the approach and ideas for future research into how the metric could be used as a thinning tool to develop less complex models.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Unified architecture for data-driven metadata tagging of building automation systems

This article presents a Unified Architecture (UA) for automated point tagging of Building Automation System (BAS) data, based on a combination of data-driven approaches. Advanced energy analytics applications—including fault detection and diagnostics and supervisory control—have emerged as a significant opportunity for improving the performance of our built environment. Effective application of these analytics depends on harnessing structured data from the various building control and monitoring systems, but typical BAS implementations do not employ any standardized metadata schema. While standards such as Project Haystack and Brick Schema have been developed to address this issue, the process of structuring the data, i.e., tagging the points to apply a standard metadata schema, has, to date, been a manual process. This process is typically costly, labor-intensive, and error-prone. In this work we address this gap by proposing a UA that automates the process of point tagging by leveraging the data accessible through connection to the BAS, including time-series data and the raw point names. The UA intertwines supervised classification and unsupervised clustering techniques from machine learning and leverages both their deterministic and probabilistic outputs to inform the point tagging process. Furthermore, we extend the UA to embed additional input and output data-processing modules that are designed to address the challenges associated with the real-time deployment of this automation solution. We test the UA on two datasets for real-life buildings: (i) commercial retail buildings and (ii) office buildings from the National Renewable Energy Laboratory (NREL) campus. We report the proposed methodology correctly applied 85–90% and 70–75% of the tags in each of these test scenarios, respectively for two significantly different building types used for testing UA's fully-functional prototype. The proposed UA, therefore, offers promising approach for automatically tagging BAS data as it reaches close to 90% accuracy. Further building upon this framework to algorithmically identify the equipment type and their relationships is an apt future research direction to pursue.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2020

Propagating sample variance uncertainties in redshift calibration: simulations, theory, and application to the COSMOS2015 data

ABSTRACT Cosmological analyses of galaxy surveys rely on knowledge of the redshift distribution of their galaxy sample. This is usually derived from a spectroscopic and/or many-band photometric calibrator survey of a small patch of sky. The uncertainties in the redshift distribution of the calibrator sample include a contribution from shot noise, or Poisson sampling errors, but, given the small volume they probe, they are dominated by sample variance introduced by large-scale structures. Redshift uncertainties have been shown to constitute one of the leading contributions to systematic uncertainties in cosmological inferences from weak lensing and galaxy clustering, and hence they must be propagated through the analyses. In this work, we study the effects of sample variance on small-area redshift surveys, from theory to simulations to the COSMOS2015 data set. We present a three-step Dirichlet method of resampling a given survey-based redshift calibration distribution to enable the propagation of both shot noise and sample variance uncertainties. The method can accommodate different levels of prior confidence on different redshift sources. This method can be applied to any calibration sample with known redshifts and phenotypes (i.e. cells in a self-organizing map, or some other way of discretizing photometric space), and provides a simple way of propagating prior redshift uncertainties into cosmological analyses. As a worked example, we apply the full scheme to the COSMOS2015 data set, for which we also present a new, principled SOM algorithm designed to handle noisy photometric data. We make available a catalogue of the resulting resamplings of the COSMOS2015 galaxies.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Capturing many-body correlation effects with quantum and classical computing

Theoretical descriptions of excited states of molecular systems in high-energy regimes are crucial for supporting and driving many experimental efforts at light source facilities. However, capturing their complicated correlation effects requires formalisms that provide a hierarchical infrastructure of approximations. These approximations lead to an increased overhead in classical computing methods and, therefore, decisions regarding the ranking of approximations and the quality of results must be made on purely numerical grounds. The emergence of quantum computing methods has the potential to change this situation. Here, in this study, we demonstrate the efficiency of the quantum phase estimator (QPE) in identifying core-level states relevant to x-ray photoelectron spectroscopy. We compare and validate the QPE predictions with exact diagonalization and real-time equation-of-motion coupled-cluster formulations, which are some of the most accurate methods for states dominated by collective correlation effects.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2023

Conserved unique peptide patterns (CUPP) online platform 2.0: implementation of +1000 JGI fungal genomes

Carbohydrate-processing enzymes, CAZymes, are classified into families based on sequence and three-dimensional fold. Because many CAZyme families contain members of diverse molecular function (different EC-numbers), sophisticated tools are required to further delineate these enzymes. Such delineation is provided by the peptide-based clustering method CUPP, Conserved Unique Peptide Patterns. CUPP operates synergistically with the CAZy family/subfamily categorizations to allow systematic exploration of CAZymes by defining small protein groups with shared sequence motifs. The updated CUPP library contains 21,930 of such motif groups including 3,842,628 proteins. The new implementation of the CUPP-webserver, https://cupp.info/, now includes all published fungal and algal genomes from the Joint Genome Institute (JGI), genome resources MycoCosm and PhycoCosm, dynamically subdivided into motif groups of CAZymes. This allows users to browse the JGI portals for specific predicted functions or specific protein families from genome sequences. Thus, a genome can be searched for proteins having specific characteristics. All JGI proteins have a hyperlink to a summary page which links to the predicted gene splicing including which regions have RNA support. The new CUPP implementation also includes an update of the annotation algorithm that uses only a fourth of the RAM while enabling multi-threading, providing an annotation speed below 1 ms/protein.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

A Multistage Stochastic Transmission Expansion Algorithm for Wide-Area Planning under Uncertainty

The overall objective for this project was to develop and demonstrate a set of methods for solving the transmission investment problem for a large network considering many possible scenarios of future conditions and multiple decision points when investments can be made. Project sub-objectives achieved this goal through a succession of extending the methods to apply to problems with increasing complexity or additional features, including the number of decision points, whether generation and transmission are co-optimized, and whether AC or DC power flow is used. A transmission model was developed for the Western Electric Coordinating Council (WECC) region, the high-voltage transmission system that serves the western third of the continental U.S. Using a dataset provided by WECC and by researchers from John Hopkins University, we have validated and demonstrated the model and used it to compare the new method for solving multi-stage stochastic transmission planning to several state-of-the-art techniques. The project has resulted in several key outcomes and achievements: The covariance-based method for choosing a small set of hours to represent short-term variability has superior performance in terms of accuracy to existing methods, including K-means clustering and Importance Sampling; The combined partitioning method for long-term uncertainty with the nested clustering approach for choosing representative hours for each long-term group has superior accuracy for equivalent computational effort compared with existing methods; Using the partitioning/clustering method combined with Sample Average Approximation provides both statistical bounds on the quality of the solution and at the same time, a complete investment plan for all contingencies in the full uncertainty set; no existing methods can provide both at the same time; The method is demonstrated to work well for choosing both transmission and generation investments; A variant on the method allows for both scenario selection and simultaneous correction for the error from the DC power flow approximation to provide a tractable method for AC power flow-based transmission planning under uncertainty; The method applied to the WECC case study demonstrates the additional value to the system operator and the consumer of identifying flexible investment options in the near-term decisions. In particular, the case study exhibits significant option value in postponing some transmission additions that appear useful but in some long-term system states create new congestion problems.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

Model Based Validation of Intelligent Powertrain Strategies for Connected and Automated Vehicles

Systems incorporating Vehicle to Everything (V2X) and conventional cellular based communication in vehicles can significantly help improve energy consumption via a combination of intelligent powertrain control strategies, smarter routing algorithms and driving in such a way as to minimize fuel economy and the emission of carbon dioxide, known as "eco-driving." In projects led by the Southwest Research Institute (SwRI), large-scale traffic simulations are created to model real-world scenarios with dynamic behavior that is reactive to imposed changes. Coupled with high fidelity powertrain models, the closed loop framework enables research and development of such Connected and Automated Vehicle (CAV) enabled technologies at scale. This paper will discuss a traffic system simulation environment that was built based on the High Street urban corridor in Columbus, Ohio. Eco-driving strategies were tested at scale on a variety of powertrain platforms – internal combustion engines, hybrid electric and fully electric vehicles. Furthermore, the paper will focus on hybrid electric powertrain modeling along with details on how the powertrain model was leveraged to develop a sophisticated clustering scheme to help down-select speed traces from large scale simulation studies for validation on vehicle dynamometer. Nominal energy consumption improvement around 12% was observed with good match between simulation studies and vehicle testing.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2022

funcX: Federated Function as a Service for Science

Here, funcX is a distributed function as a service (FaaS) platform that enables flexible, scalable, and high performance remote function execution. Unlike centralized FaaS systems, funcX decouples the cloud-hosted management functionality from the edge-hosted execution functionality. funcX's endpoint software can be deployed, by users or administrators, on arbitrary laptops, clouds, clusters, and supercomputers, in effect turning them into function serving systems. funcX's cloud-hosted service provides a single location for registering, sharing, and managing both functions and endpoints. It allows for transparent, secure, and reliable function execution across the federated ecosystem of endpoints-enabling users to route functions to endpoints based on specific needs. funcX uses containers (e.g., Docker, Singularity, and Shifter) to provide common execution environments across endpoints. funcX implements various container management strategies to execute functions with high performance and efficiency on diverse funcX endpoints. funcX also integrates with an in-memory data store and Globus for managing data that may span endpoints. We motivate the need for funcX, present our prototype design and implementation, and demonstrate, via experiments on two supercomputers, that funcX can scale to more than 130000 concurrent workers. We show that funcX's container warming-aware routing algorithm can reduce the completion time for 3,000 functions by up to 61% compared to a randomized algorithm and the in-memory data store can speed up data transfers by up to 3x compared to a shared file system.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Evaluating lightweight unsupervised online IDS for masquerade attacks in CAN

Vehicular controller area networks (CANs) are susceptible to masquerade attacks by malicious adversaries. In masquerade attacks, adversaries silence a targeted ID and then send malicious frames with forged content at the expected timing of benign frames. As masquerade attacks could seriously harm vehicle functionality and are the stealthiest attacks to detect in CAN, recent work has devoted attention to compare frameworks for detecting masquerade attacks in CAN. However, most existing works report offline evaluations using CAN logs already collected using simulations that do not comply with the domain’s real-time constraints. Here we contribute to advance the state of the art by presenting a comparative evaluation of four different non-deep learning (DL)-based unsupervised online intrusion detection systems (IDS) for masquerade attacks in CAN. Our approach differs from existing comparative evaluations in that we analyze the effect of controlling streaming data conditions in a sliding window setting. In doing so, we use realistic masquerade attacks being replayed from the ROAD dataset. We show that although evaluated IDS are not effective at detecting every attack type, the method that relies on detecting changes in the hierarchical structure of clusters of time series produces the best results at the expense of higher computational overhead. We discuss limitations, open challenges, and how the evaluated methods can be used for practical unsupervised online CAN IDS for masquerade attacks.

Anomaly detection↗
DOE OSTI2024

$\mathrm{COBRA}$ improves the completeness and contiguity of viral genomes assembled from metagenomes

Viruses are often studied using metagenome-assembled sequences, but genome incompleteness hampers comprehensive and accurate analyses. Contig Overlap Based Re-Assembly (COBRA) resolves assembly breakpoints based on the de Bruijn graph and joins contigs. Here we benchmarked COBRA using ocean and soil viral datasets. COBRA accurately joined the assembled sequences and achieved notably higher genome accuracy than binning tools. From 231 published freshwater metagenomes, we obtained 7,334 bacteriophage clusters, ~83% of which represent new phage species. Notably, ~70% of these were circular, compared with 34% before COBRA analyses. We expanded sampling of huge phages (≥200 kbp), the largest of which was curated to completion (717 kbp). Improved phage genomes from Rotsee Lake provided context for metatranscriptomic data and indicated the in situ activity of huge phages, whiB-encoding phages and cysC- and cysH-encoding phages. COBRA improves viral genome assembly contiguity and completeness, thus the accuracy and reliability of analyses of gene content, diversity and evolution.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Continuous-variable quantum computation of the O(3) model in 1+1 dimensions

We formulate the $O(3)$ non-linear sigma model in $1+1$ dimensions as a limit of a three-component scalar field theory restricted to the unit sphere in the large squeezing limit. This allows us to describe the model in terms of the continuous variable (CV) approach to quantum computing. Here we construct the ground state and excited states using the coupled cluster ansatz and find excellent agreement with the exact diagonalization results for a small number of lattice sites. We then present the simulation protocol for the time evolution of the model using CV gates, estimate the discretization error, and present numerical results obtained from a photonic quantum simulator. We expect that the methods developed in this work will be useful for exploring interesting dynamics for a wide class of sigma models and gauge theories, as well as for simulating scattering events on quantum hardware in the coming decade.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2020

PERSIANN Dynamic Infrared–Rain Rate (PDIR-Now): A Near-Real-Time, Quasi-Global Satellite Precipitation Dataset

This study presents the Precipitation Estimation from Remotely Sensed Information Using Artificial Neural Networks–Dynamic Infrared Rain Rate (PDIR-Now) near-real-time precipitation dataset. This dataset provides hourly, quasi-global, infrared-based precipitation estimates at 0.04° × 0.04° spatial resolution with a short latency (15–60 min). It is intended to supersede the PERSIANN–Cloud Classification System (PERSIANN-CCS) dataset previously produced as the near-real-time product of the PERSIANN family. We first provide a brief description of the algorithm’s fundamentals and the input data used for deriving precipitation estimates. Second, we provide an extensive evaluation of the PDIR-Now dataset over annual, monthly, daily, and subdaily scales. Last, the article presents information on the dissemination of the dataset through the Center for Hydrometeorology and Remote Sensing (CHRS) web-based interfaces. The evaluation, conducted over the period 2017–18, demonstrates the utility of PDIR-Now and its improvement over PERSIANN-CCS at all temporal scales. Specifically, PDIR-Now improves the estimation of rain/no-rain days as demonstrated by a critical success index (CSI) of 0.53 compared to 0.47 of PERSIANN-CCS. In addition, PDIR-Now improves the estimation of seasonal and diurnal cycles of precipitation as well as regional precipitation patterns erroneously estimated by PERSIANN-CCS. Finally, an evaluation is carried out to examine the performance of PDIR-Now in capturing two extreme events, Hurricane Harvey and a cluster of summer thunderstorms that occurred over the Netherlands, where it is shown that PDIR-Now adequately represents spatial precipitation patterns as well as subdaily precipitation rates with a correlation coefficient (CORR) of 0.64 for Hurricane Harvey and 0.76 for the Netherlands thunderstorms.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

Performance Results for Sensor Assignment Problem as Solved on a Multi-Node Cluster

An earlier report described a procedure for optimal sensor set selection and its implementation on a computational cluster. This new and innovative capability was developed to facilitate a reduction in operations staffing levels to improve plant economics. By automating surveillance and maintenance tasks through early detection of degrading sensors and equipment, staff can be more efficiently deployed. The method uses automated reasoning and domain knowledge in the form of the conservation equations to infer from plant measurements the state of equipment health. Inclusion of domain knowledge addresses the problem that exists with pure data-driven methods that there are no rigorous guidelines for determining what constitutes an adequate sensor set. Formalizing the procedure for sensor set selection as we have done results in a more reliable and explainable diagnosis of plant equipment health. Importantly, from the standpoint of the plant owner, personnel are provided with an early and explicit diagnosis of an equipment problem. That in principle automates the process and eliminates having to send personnel into the plant to find the cause as typically occurs when a data-driven method detects an anomaly. In this report we describe first results obtained using a computational cluster to solve the sensor set selection problem as framed above. The case described addresses the problem of equipment health monitoring in the high-pressure (HP) feedwater system of a pressurized light water reactor as seen through the eyes of our collaborating utility partner. Maintenance of this system can amount to millions of dollars per year if equipment health issues go undiagnosed and lead to loss of function. On examining the potential that is inherent in the installed sensor set for diagnosing equipment health degradation, it was found that greater fault resolution capability can be achieved using a sensor set that is 20 percent fewer in number. The take-away is that compared to the installed sensor set there exists a more strategic assignment of sensors that will furnish better health monitoring capability and with fewer sensors. Where the problem defies solution by manual inspection, as is the case here, one can be found by an algorithm. The solution was obtained in four hours using 30 computational cores. The HP feedwater problem as posed above illustrates the added value of approaching the sensor selection problem as one amenable to algorithmic solution. This problem is of interest to advanced reactor designers and to utilities that are setting up remote monitoring and diagnostic centers.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗