Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Qubit lattice algorithm simulations of the scattering of a bounded two dimensional electromagnetic pulse from an infinite planar dielectric interface

Qubit lattice algorithm (QLA) simulations are performed for a two-dimensional spatially bounded pulse propagating onto a plane interface between two dielectric slabs. QLA is an initial value scheme that consists of a sequence of unitary collision and streaming operators, with appropriate potential operators, that recover Maxwell equations in inhomogeneous dielectric media to the second order in the lattice discreteness. For the case of total internal reflection, there is transient energy transfer into the second medium due to the evanescent fields as the Poynting unit vector of the pulse is rotated from its incident to reflected direction. Because of the finite spatial extent of the pulse, a self-consistent Goos–Hanchen-type displacement along the interface is found without imposing any explicit interface boundary conditions on the fields. For normal incidence, the standard Fresnel coefficients are recovered for appropriately averaged QLA fields. Energy is conserved at all times to seven significant figures.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2026

Melter Tests to Define LAW Halide Concentrations, Phase 2

The present work represents the second of two phases to address the formation of secondary sulfate phases while processing high waste loading LAW formulations with elevated halide, chromium, and phosphate feed concentrations over a wide range of LAW waste types, including those with high potassium concentrations. The Test Plan for the present work outlines a series of tests designed to assess the formation of secondary phases over a range of sulfur and halide concentrations for two LAW streams and associated glass formulations as well as crucible testing to identify additives that have the potential to suppress the formation of secondary phases for wastes with high chlorine contents. The combined results from both phases of testing and from previous tests with these LAW waste streams will be used as inputs to future work to develop concentration limits for these components in the LAW glass property composition models and to support the modification and updating of the current WTP LAW glass formulation algorithm.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2024

Relevant biochar characteristics influencing compressive strength of biochar-cement mortars

To counteract the contribution of CO 2 emissions by cement production and utilization, biochar is being harnessed as a carbon-negative additive in concrete. Increasing the cement replacement and biochar dosage will increase the carbon offset, but there is large variability in methods being used and many researchers report strength decreases at cement replacements beyond 5%. This work presents a reliable method to replace 10% of the cement mass with a vast selection of biochars without decreasing ultimate compressive strength, and in many cases significantly improving it. By carefully quantifying the physical and chemical properties of each biochar used, machine learning algorithms were used to elucidate the three most influential biochar characteristics that control mortar strength: initial saturation percentage, oxygen-to-carbon ratio, and soluble silicon. These results provide additional research avenues for utilizing several potential biomass waste streams to increase the biochar dosage in cement mixes without decreasing mechanical properties.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Architecture for Quantum-in-the Loop Real-Time Simulations for Designing Resilient Smart Grids

With the power grid growing more complex every day with the inclusion of new sensors and regulatory approvals that enable end-users and small local developers to participate in the grid, it is becoming challenging for conventional smart grid simulation, emulation, and testing technologies to keep up. In this work, we propose that quantum-encoded real-time simulations can be helpful under the new paradigm and operational circumstances to solve optimization problems for power grids. By leveraging the principles of quantum mechanics, the proposed quantum-in-loop (QIL) framework will enable better and faster optimization solutions based on real-world, real-time data streams, facilitating the real-time planning and operations of electrical grids that rely on millions of distributed sensors and controllers. Furthermore, QIL will allow researchers and engineers to assist utilities in designing, developing, and de-risking algorithms to optimize power grid operation and resilience by considering inputs from millions of grid-connected devices. QIL framework is being developed to have a self-limiting triage mechanism, which will help engineers and practitioners identify fundamental physical limits on quantum processors, revealing what quantum algorithms can and cannot do in utility-specific use cases and must continue to count on classical high-performance computing infrastructure.

digital real-time simulation↗
DOE OSTI2022

Sensing Electrical Networks Securely & Economically (SENSE)

The growing adoption of distributed energy resources (DERs) like battery energy storage systems and roof top solar/PV and the rapid penetration of electric vehicles (EVs), the electric grid is undergoing a major transformation with elevated stress on legacy grid assets. Despite a lot of expenditure to address these challenges, both in dollars and manpower, utilities have not been able to receive the value that was promised. The gains have been most visible at the transmission and substation level, especially where the main objective was improving operational and economic efficiency for the utility. Improving visibility and control at a few select points enhances the existing and established paradigm of centralized command and control. With changing load patterns, load types and the overall transition to an “active grid”, the centralized control and coordination paradigm gets challenged. To address the challenges, a new architecture and mechanism is needed, one that supports decentralized control and decision making, extracting value streams at the grid edge, particularly as the changes are fueled by transitions occurring in the distribution system. To address this, a communications and data processing platform, “GAMMA” was developed and demonstrated through the project. At the heart of the platform, are distributed, intelligent edge nodes with sensing and compute capabilities, that can record and analyze information locally. They are embedded in sensors and actuators specific to different distribution system applications. Phase 1 of the project focused on developing novel sensor technology that can be used for monitoring utility pole top distribution transformers. The sensors were designed with the objective of being low-cost, communicating with the GAMMA cloud using novel “delay-tolerant” networking using Bluetooth and a secure mobile application. They were non-intrusive in nature so that they can be installed quickly in the field, resulting in overall low cost of deployment and operations. Following the successful completion of Phase 1, the team manufactured 100 units for a field demonstration in Phase 2. The field demonstration was carried out on two real feeder systems with the local utility partner. In total, 100 sensors were installed and operated over a period of 6 months in the state of Georgia. The platform is operational end to end, with the cloud infrastructure deployed on a distributed, serverless environment that can serve multiple data streams, an analytics engine and a portal to securely view the data from multiple assets. The data collected through the GAMMA Mobile Phone app showcased the viability of the novel delay tolerant networking architecture, and the data processing algorithms developed through the course of the project, were successful in extracting important information about the overall network, improving the utility’s visibility and situational awareness in the distribution feeder.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

Phoenix: A Scalable Streaming Hypergraph Analysis Framework

We present Phoenix, a scalable hypergraph analytics framework for data analytics and knowledge discovery that was implemented on the leadership class computing platforms at Oak Ridge National Laboratory (ORNL). Our software framework comprises a distributed implementation of a streaming server architecture which acts as a gateway for various hypergraph generators/external sources to connect. Phoenix has the capability to utilize diverse hypergraph generators, including HyGen, a very large-scale hypergraph generator developed by ORNL. Phoenix incorporates specific algorithms for efficient data representation by exploiting hidden structures of the hypergraphs. Our experimental results demonstrate Phoenix’s scalable and stable performance on massively parallel computing platforms. Phoenix’s superior performance is due to the merging of high-performance computing with data analytic.

Kurte, Kuldeep↗
DOE OSTI2026

Curation and Dissemination of Complex Multi-Modal Datasets for Radiation Detection, Localization, and Tracking

The PANDAWN sensor network in Chicago, IL, is a state-of-the-art testbed for networked, multi-modal sensing. It integrates AI/data science methods into its operation, from data acquisition to automated data labeling and curation workflows. The curation and dissemination of diverse multi-modal datasets will enable the development of new radiological/nuclear (R/N) detection, localization, and tracking algorithms and methods relevant across the nonproliferation mission space. This article first introduces the PANDAWN sensor network and the features that make it stand out from previous multi-modal data acquisition efforts. We then review the various data streams acquired on the PANDAWN nodes and present the implementation of an automated data curation pipeline that includes the labeling of radiation and contextual data streams. Here, we finally provide a short overview of different studies that leveraged the curated datasets.

Data curation↗
DOE OSTI2025

sdt (Solar Data Tools) [SWR-25-130]

Solar Data Tools (sdt) is an open-source Python library for analyzing PV power (and irradiance) time-series data. It was developed to enable analysis of unlabeled PV data, i.e. with no model, no meteorological data, and no performance index required, by taking a statistical signal processing approach in the algorithms used in the package’s main data processing pipeline. Solar Data Tools empowers PV system fleet owners or operators to analyze system performance a hundred times faster even when they only have access to the most basic data stream—power output of the system.

Meyers-Im, Bennet [National Laboratory of the Rock↗
DOE OSTI2025

A Morphological Model to Separate Resolved–Unresolved Sources in the DESI Legacy Surveys: Application in the LS4 Alert Stream

Separating resolved and unresolved sources in large imaging surveys is a fundamental step to enable downstream science, such as searching for extragalactic transients in wide-field time-domain surveys. Here we present our method to effectively separate point sources from the resolved, extended sources in the Dark Energy Spectroscopic Instrument (DESI) Legacy Surveys (LS). We develop a supervised machine learning model based on the Gradient Boosting algorithm XGBoost. The features input to the model are purely morphological and are derived from the tabulated LS data products. We train the model using ∼2 × 10 5 LS sources in the COSMOS field with HST morphological labels and evaluate the model performance on LS sources with spectroscopic classification from the DESI Data Release 1 (∼2 × 10 7 objects) and the Sloan Digital Sky Survey Data Release 17 (∼3 × 10 6 objects), as well as on ∼2 × 10 8 Gaia stars. A significant fraction of LS sources are not observed in every LS filter, and we therefore build a “Hybrid” model as a linear combination of two XGBoost models, each containing features combining aperture flux measurements from the “blue” (gr) and “red” (iz) filters. The Hybrid model shows a reasonable balance between sensitivity and robustness, and achieves higher accuracy and flexibility compared to the LS morphological typing. With the Hybrid model, we provide classification scores for ∼3 × 10 9 LS sources, making this the largest ever machine learning catalog separating resolved and unresolved sources. The catalog has been incorporated into the real-time pipeline of the La Silla Schmidt Southern Survey (LS4), enabling the identification of extragalactic transients within the LS4 alert stream.

astrostatistics↗
DOE OSTI2022

SNAD transient miner: Finding missed transient events in ZTF DR4 using k-D trees

Here we report the automatic detection of 11 transients (7 possible supernovae and 4 active galactic nuclei candidates) within the Zwicky Transient Facility fourth data release (ZTF DR4), all of them observed in 2018 and absent from public catalogs. Among these, three were not part of the ZTF alert stream. Our transient mining strategy employs 41 physically motivated features extracted from both real light curves and four simulated light curve models (SN Ia, SN II, TDE, SLSN-I). These features are input to a k-D tree algorithm, from which we calculate the 15 nearest neighbors. After pre-processing and selection cuts, our dataset contained approximately a million objects among which we visually inspected the 105 closest neighbors from seven of our brightest, most well-sampled simulations, comprising 89 unique ZTF DR4 sources. Our result illustrates the potential of coherently incorporating domain knowledge and automatic learning algorithms, which is one of the guiding principles directing the SNAD team. It also demonstrates that the ZTF DR is a suitable testing ground for data mining algorithms aiming to prepare for the next generation of astronomical data.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Real-time Event Detection Using Rank Signatures of Real-world PMU Data

Timely detection of power system events is a crucial task, which can facilitate the implementation of remedial actions to improve reliability, resiliency, and security of the system. Meanwhile, the widespread deployment of phasor measurement units (PMUs) makes it possible to develop data-driven event detection techniques. However, relying purely on data without incorporating domain knowledge for the event detection task in power systems poses substantial security and stability risks due to issues associated with data misinterpretation and model accuracy. In this regard, we propose a real-time event detection method using real-world PMU data by incorporating domain knowledge to adequately capture the event signatures. Specifically, we track the change in rank signatures of PMU data to accurately localize the events. To optimize the detection process, we incorporate an offline Bayesian optimization algorithm to tune the parameters by efficiently searching for the best values. The experiments using the real-world PMU dataset from a U.S. interconnection show that the proposed event detection approach can efficiently detect the events from PMU data streams with high accuracy.

Ghasemkhani, Amir↗
DOE OSTI2023

Online real-time learning of dynamical systems from noisy streaming data

Abstract Recent advancements in sensing and communication facilitate obtaining high-frequency real-time data from various physical systems like power networks, climate systems, biological networks, etc. However, since the data are recorded by physical sensors, it is natural that the obtained data is corrupted by measurement noise. In this paper, we present a novel algorithm for online real-time learning of dynamical systems from noisy time-series data, which employs the Robust Koopman operator framework to mitigate the effect of measurement noise. The proposed algorithm has three main advantages: (a) it allows for online real-time monitoring of a dynamical system; (b) it obtains a linear representation of the underlying dynamical system, thus enabling the user to use linear systems theory for analysis and control of the system; (c) it is computationally fast and less intensive than the popular extended dynamic mode decomposition (EDMD) algorithm. We illustrate the efficiency of the proposed algorithm by applying it to identify the Van der Pol oscillator, the chaotic attractor of the Henon map, the IEEE 68 bus system, and a ring network of Van der Pol oscillators.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Sage: parallel semi-asymmtric graph algorithms for NVRAMs

Non-volatile main memory (NVRAM) technologies provide an attractive set of features for large-scale graph analytics, including byte-addressability, low idle power, and improved memory-density. NVRAM systems today have an order of magnitude more NVRAM than traditional memory (DRAM). NVRAM systems could therefore potentially allow very large graph problems to be solved on a single machine, at a modest cost. However, a significant challenge in achieving high performance is in accounting for the fact that NVRAM writes can be much more expensive than NVRAM reads. In this paper, we propose an approach to parallel graph analytics using the Parallel Semi-Asymmetric Model (PSAM), in which the graph is stored as a read-only data structure (in NVRAM), and the amount of mutable memory is kept proportional to the number of vertices. Similar to the popular semi-external and semi-streaming models for graph analytics, the PSAM approach assumes that the vertices of the graph fit in a fast read-write memory (DRAM), but the edges do not. In NVRAM systems, our approach eliminates writes to the NVRAM, among other benefits. To experimentally study this new setting, we develop Sage, a parallel semi-asymmetric graph engine with which we implement provably-efficient (and often work-optimal) PSAM algorithms for over a dozen fundamental graph problems. We experimentally study Sage using a 48--core machine on the largest publicly-available real-world graph (the Hyperlink Web graph with over 3.5 billion vertices and 128 billion edges) equipped with Optane DC Persistent Memory, and show that Sage outperforms the fastest prior systems designed for NVRAM. Importantly, we also show that Sage nearly matches the fastest prior systems running solely in DRAM, by effectively hiding the costs of repeatedly accessing NVRAM versus DRAM.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Understanding the Impact of Data Staging for Coupled Scientific Workflows

We report the rate of data generated by cutting-edge experimental science facilities and large-scale simulations enabled by current high-performance computing (HPC) systems has continued to grow at a far greater pace than the development of the network and storage capabilities on which these systems rely. To cope with this challenge, scientist are moving toward the creation of autonomous experiments and HPC simulations using machine learning. However, efficiently moving, storing, and processing large amounts of data away from the point of origin presents an incredible challenge. In-memory computing, in situ analysis, data staging, and data streaming are recognized viable alternatives to traditional file-based methods for transferring data between coupled workflows. However, the performance trade-offs and limitations for these methods are not fully understood when used in HPC applications. This article presents a comprehensive performance assessment of the current solutions for data staging when applied to applications that are not necessary I/O intensive which makes them not ideal candidates for these methods. Our study is based on experiments running at scale on Oak Ridge National Laboratory's Summit supercomputer using applications and simulations that cover typical computational motifs and patterns. We investigated the usability and cost/benefit trade-offs of staging algorithms for HPC applications under different scenarios and highlight opportunities for optimizing the dataflow between coupled simulation workflows.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Signal Processing Based Method for Real-Time Anomaly Detection in High-Performance Computing

Performance anomalies can manifest as irregular execution times or abnormal execution events for many reasons, including network congestion and resource contention. Detecting such anomalies in real-time by analyzing the details of performance traces at scale is impractical due to the sheer volume of data High-Performance Computing (HPC) applications produce. In this paper, we propose formulating HPC performance anomaly detection as a signal-processing problem where anomalies can be treated as noise. We evaluate our proposed method in comparison with two other commonly used anomaly detection techniques of varying complexity based on their detection accuracy and scalability. Since real-time in-situ anomaly detection at a large scale requires lightweight methods that can handle a large volume of streaming data, we find that our proposed method provides the best trade-off. We then implement the proposed method in Chimbuko, the first online, distributed, and scalable workflow-level performance trace analysis framework. We compare our proposed signal-based anomaly detection algorithm with two other methods using a function of their accuracy, F1 score, and detection overhead. Our experiments demonstrate that our proposed approach achieves a 99% improvement for the benchmark datasets and a 93% improvement with Chimbuko traces.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2022

Resource Allocation for Single Carrier Massive MIMO Systems

Resource allocation in orthogonal frequency division multiplexing (OFDM) systems is performed through allocating blocks of subcarriers to each user. Even though OFDM is the primary waveform for 5G NR systems, research reports have noted that single carrier modulation (SCM) offers several advantages over OFDM in massive multiple input multiple output (MIMO) systems, making it a preferred candidate for some future applications such as massive machine type communications (mMTC). This paper presents a method for SCM resource allocation and the relevant information recovery algorithms at the receiver. Our emphasis is on cyclic prefixed SCM, where highly flexible and efficient frequency domain detection algorithms enable the operation of many simultaneous users in a massive MIMO uplink scenario. The proposed resource allocation method allows the number of users to exceed the number of antennas at the base station (BS). Each single carrier transmission is partitioned into L interleaved streams, and each user is allocated a number of such streams. One major benefit of SCM is that each data symbol is spread over the entire bandwidth. As such, the receiver performance is dictated by the average channel gain across the transmission band rather than the channel gain at a given frequency bin or a small group of frequencies. In the proposed setup, each stream may be thought of as a resource block in SCM, analogous to resource blocks in OFDM. Hence, in the context of this paper, the terms resource blocks and streams may be used interchangeably.

5G and Beyond Communications↗
DOE OSTI2023

Robust clustering of the local Milky Way stellar kinematic substructures with Gaia eDR3

Understanding local stellar kinematic substructures in the solar neighbourhood helps build a complete picture of the formation of the Milky Way, as well as an empirical phase space distribution of dark matter that would inform detection experiments. We apply the clustering algorithm HDBSCAN on the Gaia early third data release to identify a list of stable clusters in velocity space and action-angle space by taking into account the measurement uncertainties and studying the stability of the clustering results. We find 1405 (497) stars in 23 (6) robust clusters in velocity space (action-angle space) that are consistently not associated with noise. We discuss the kinematic properties of these structures and study whether many of the small clusters belong to a similar larger cluster based on their chemical abundances. They are attributed to the known structures: the Gaia Sausage-Enceladus, the Helmi Stream, and globular cluster NGC 3201 are found in both spaces, while NGC 104 and the thick disc (Sequoia) are identified in velocity space (action-angle space). Although we do not identify any new structures, we find that the HDBSCAN member selection of already known structures is unstable to input kinematics of the stars when resampled within their uncertainties. We therefore present the stable subset of local kinematic structures, which are consistently identified by the clustering algorithm, and emphasize the need to take into account error propagation during both the manual and automated identification of stellar structures, both for existing ones as well as future discoveries.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Spontaneous Raman–LIF–CO–OH measurements of species concentration in turbulent spray flames

This paper presents new measurements of species concentrations, temperature and mixture fraction in selected regions of a turbulent ethanol spray flame. The line-Raman–LIF–COsingle bondOH setup developed at the Sandia's Combustion Research Facility is utilised to probe regions of a spray flame where laser breakdown of liquid droplets is avoided and the remaining interferences can be corrected. The spray flame is stabilised on the piloted Sydney needle spray burner, where axial translation of the liquid injecting needle in the air-blast stream can transition the spray from dilute to dense. The solution to obtaining successful measurements is found to be multifaceted and includes: the appropriate selection of flame conditions; high sensitivity of the Raman detection system permitting reduced laser energies; development of a pre-processing algorithm to reject strong droplet interferences; and application of the hybrid matrix inversion method combined with wavelet denoising to account for interference corrections and noise at the very low signal levels obtained. Unique and necessary for the successful measurements reported in this paper, a pre-processing algorithm is outlined that removes data points corrupted with strong interferences from droplets. These interferences arise from a range of sources, but the most intense are due to the laser interaction with surrounding mist or liquid fragments, such that measurements near the jet centreline are corrupted and hence discarded. Reliable measurements of mixture fraction, temperature obtained from the sum of the species number densities, and species mole fractions are reported for regions in the flames sufficiently far from the centreline. The paper demonstrates the feasibility of the judicious use of Raman scattering in turbulent spray flames, the results of which will be extremely useful for validating numerical simulations.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗