Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

Data Assimilation for Robust UQ Within Agent-Based Simulation on HPC Systems

Agent-based simulation provides a powerful tool for in silico system modeling. However, these simulations do not provide built-in methods for uncertainty quantification (UQ). Within these types of models a typical approach to UQ is to run multiple realizations of the model then compute aggregate statistics. This approach is limited due to the compute time required for a solution. When faced with an emerging biothreat, public health decisions need to be made quickly and solutions for integrating near real-time data with analytic tools are needed. We propose an integrated Bayesian UQ framework for agent-based models based on sequential Monte Carlo sampling. Given streaming or static data about the evolution of an emerging pathogen this Bayesian framework provides a distribution over the parameters governing the spread of a disease through a population. These estimates of the spread of a disease may be provided to public health agencies seeking to abate the spread. By coupling agent-based simulations with Bayesian modeling in a data assimilation, our proposed framework provides a powerful tool for modeling dynamical systems in silico. We propose a method which reduces model error and provides a range of realistic possible outcomes. Moreover, our method addresses two primary limitations of ABMs: the lack of UQ and an inability to assimilate data. Our proposed framework combines the flexibility of an agent-based model with UQ provided by the Bayesian paradigm in a workflow which scales well to HPC systems. We provide algorithmic details and results on a simulated outbreak with both static and streaming data.

Spannaus, Adam [ORNL] (ORCID:0000000225213657)↗

Comparison of Supervised and Un-Supervised Machine Learning Algorithms for Threat Detection and Scintillator Performance for Radiation Portal Monitoring

Following the events of September 11, 2001, international border crossing have been equipped with radiation portal monitors (RPMs) to identify illicit radioactive material. Polyvinyl toluene (PVT) scintillators are commonly used due to their low cost and reasonable maintainability, however they offer low spectral resolution. Despite the fact that over twenty years has transpired since this event, radioisotopes are still typically identified by hand-crafted classification algorithms, e.g., total counts or energy windowing, and exhibit relatively poor performance in detecting threats at the low false alarm rates required to support the stream of commerce. While some improvement to performance has been realized via the use of supervised machine learning, these classification algorithms typically utilize simulations in lieu of real data due to the sparsity of data for one or more classes. Accordingly, the performance of these algorithms is somewhat less than optimal when examining experiments or simulations with model mismatch. Consequently, in this work, we examine the application of a number of unsupervised machine learning, anomaly detection based algorithms, to circumvent the inverse crime when analyzing spectroscopy data for RPMs. We also compare anomaly detection results with those obtained via the use of supervised classification detection ML algorithms when model mismatch is introduced between the simulated threat items utilized for training/testing. Finally, we compared the performance of the PVT scintillators to those obtained with higher resolution detectors using both anomaly detection and supervised classification algorithms.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

rustpix

rustpix is a high-performance, open-source Rust library with first-class Python bindings (via PyO3) for processing pixel-detector data in neutron imaging. It targets time-stamping detectors such as Timepix3 (TPX3) at ORNL's Spallation Neutron Source (VENUS beamline), where each detected neutron deposits charge across a cluster of pixels within a very high-rate event stream (96M+ hits/sec). rustpix parses TPX3 event data in parallel using memory-mapped I/O, offers four interchangeable clustering algorithms (ABS adjacency-based search, DBSCAN, graph/union-find connected components, and a parallel grid method), and extracts weighted, super-resolved centroids to produce neutron-event lists. A streaming architecture lets it process files larger than available memory. rustpix is distributed as a pip-installable Python package (with NumPy integration), Rust crates, a command-line tool, and an interactive GUI; it writes HDF5, Apache Arrow, and CSV; and it is designed to extend to TPX4 and other detector types. Released as open-source under the MIT License.

Zhang, Chen [Oak Ridge National Laboratory (ORNL),↗

The Phase-2 Upgrade of the CMS Data Acquisition

The High Luminosity LHC (HL-LHC) will start operating in 2027 after the third Long Shutdown (LS3), and is designed to provide an ultimate instantaneous luminosity of 7:5 × 10$^{34}$ cm$^{-2}$ s$^{-1}$, at the price of extreme pileup of up to 200 interactions per crossing. The number of overlapping interactions in HL-LHC collisions, their density, and the resulting intense radiation environment, warrant an almost complete upgrade of the CMS detector. The upgraded CMS detector will be read out by approximately fifty thousand highspeed front-end optical links at an unprecedented data rate of up to 80 Tb/s, for an average expected total event size of approximately 8 - 10 MB. Following the present established design, the CMS trigger and data acquisition system will continue to feature two trigger levels, with only one synchronous hardware-based Level-1 Trigger (L1), consisting of custom electronic boards and operating on dedicated data streams, and a second level, the High Level Trigger (HLT), using software algorithms running asynchronously on standard processors and making use of the full detector data to select events for offline storage and analysis. The upgraded CMS data acquisition system will collect data fragments for Level-1 accepted events from the detector back-end modules at a rate up to 750 kHz, aggregate fragments corresponding to individual Level- 1 accepts into events, and distribute them to the HLT processors where they will be filtered further. Events accepted by the HLT will be stored permanently at a rate of up to 7.5 kHz. This paper describes the baseline design of the DAQ and HLT systems for the Phase-2 of CMS.

Badaro, Gilbert↗

Machine Committee Framework for Power Grid Disturbances Analysis Using Synchrophasors Data

Events detection is a key challenge in power grid frequency disturbances analysis. Accurate detection of events is crucial for situational awareness of the power system. In this paper, we study the problem of events detection in power grid frequency disturbance analysis using synchrophasors data streams. Current events detection approaches for power grid rely on individual detection algorithm. This study integrates some of the existing detection algorithms using the concept of machine committee to develop improved detection approaches for grid disturbance analysis. Specifically, we propose two algorithms—an Event Detection Machine Committee (EDMC) algorithm and a Change-Point Detection Machine Committee (CPDMC) algorithm. Both algorithms use parallel architecture to fuse detection knowledge of its individual methods to arrive at an overall output. The EDMC algorithm combines five individual event detection methods, while the CPDMC algorithm combines two change-point detection methods. Each method performs the detection task separately. The overall output of each algorithm is then computed using a voting strategy. The proposed algorithms are evaluated using three case studies of actual power grid disturbances. Compared with the individual results of the various detection methods, we found that the EDMC algorithm is a better fit for analyzing synchrophasors data; it improves the detection accuracy; and it is suitable for practical scenarios.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Towards a self-driving trigger at the LHC: adaptive response in real time

Real-time data filtering and selection—or trigger—systems at high-throughput scientific facilities such as the experiments at the Large Hadron Collider must process extremely high-rate data streams under stringent bandwidth, latency, and storage constraints. Yet these systems are typically designed as static, hand-tuned menus of selection criteria grounded in prior knowledge and simulation. In this work, we further explore the concept of a self-driving trigger, an autonomous data-filtering framework that reallocates resources and adjusts thresholds dynamically in real-time to optimize signal efficiency, rate stability, and computational cost as instrumentation and environmental conditions evolve. We introduce a benchmark ecosystem to emulate realistic collider scenarios and demonstrate real-time optimization of a menu including canonical energy sum triggers as well as modern anomaly-detection algorithms that target non-standard event topologies using machine learning. Using simulated data streams and publicly available collision data from the Compact Muon Solenoid experiment, we demonstrate the capability to dynamically and automatically optimize trigger performance under specific cost objectives without manual retuning. Our adaptive strategy shifts trigger design from static menus with heuristic tuning to intelligent, automated, data-driven control, unlocking greater flexibility and discovery potential in future high-energy physics analyses.

Emami, Shaghayegh [Michigan U.] (ORCID:00090007589↗

A fast, dense Chebyshev solver for electronic structure on GPUs

Matrix diagonalization is almost always involved in computing the density matrix needed in quantum chemistry calculations. In the case of modest matrix sizes (≲4000), performance of traditional dense diagonalization algorithms on modern GPUs is underwhelming compared to the peak performance of these devices. This motivates the exploration of alternative algorithms better suited to these types of architectures. We newly derive, and present in detail, an existing Chebyshev expansion algorithm whose number of required matrix multiplications scales with the square root of the number of terms in the expansion. Focusing on dense matrices of modest size, our implementation on GPUs results in large speed ups when compared to diagonalization. Additionally, we improve upon this existing method by capitalizing on the inherent task parallelism and concurrency in the algorithm. Furthermore, this improvement is implemented on GPUs by using CUDA and HIP streams via the MAGMA library and leads to a significant speed up over the serial-only approach for smaller (≲1000) matrix sizes. Finally, we apply our technique to a model system with a high density of states around the Fermi level, which typically presents significant challenges.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

"Sintering" Models and In-Situ Experiments: Data Assimilation for Microstructure Prediction in SLS Additive Manufacturing of Nylon Components

Selective laser sintering methods are workhorses for additively manufacturing polymer-based components. The ease of rapid prototyping also means it is easy to produce illicit components. It is necessary to have a data-calibrated in-situ physical model of the build process in order to predict expected and defective microstructure characteristics that inform component provenance. Toward this end, sintering models are calibrated and characteristics such as component defects are explored. This is accomplished by assimilating multiple data streams, imaging analysis, and computational model predictions in an adaptive Bayesian parameter estimation algorithm. From these data sources, along with a phase-field model, bulk porosity distributions are inferred. Model parameters are constrained to physically-relevant search directions by sensitivity analysis, and then matched to predictions using adaptive sampling. Using this feedback loop, data-constrained estimates of sintering model parameters along with uncertainty bounds are obtained.

3D printing, Additive Manufacturing, polymer, sint↗

Developing ML/AI Methods for High-Throughput Characterization of Multiple-Sensor Streams of Tokamak Dynamics for High-Speed Control (Final Report)

This project evaluated and developed new mathematical and algorithmic techniques capable of handling (in real-time) the growing amounts of data generated by modern fusion research. While existing numerical linear algebra (NLA) methods provide the backbone to classical data analysis and algorithms, these methods fundamentally do not port to distributed architectures nor do they allow low-latency data reduction for control. Motivated by the needs for modern fusion reactors, this project explored and implemented new numerical methods to characterize plasma dynamics, respond in real-time to discharge evolution, and to process massive-scale data accurately and rapidly more fully. This project links expertise in multiple-sensor diagnostics of tokamak plasma dynamics from Columbia University’s Plasma Physics Laboratory with expertise in massive-scale data reduction and extreme data control algorithms at Columbia University’s Data Science Institute. This interdisciplinary project (i) applied machine learning methods, (ii) implemented a properly-trained neural-network for very fast processing of high-speed plasma videography, and (ii) developed the applied mathematical methods, based on randomized-NLA (rNLA) routines, for data analysis, reduction, and real-time control. The Columbia University High Beta Tokamak-Extended Pulse (HBT-EP) facility provided data to test new algorithms and partnership with Columbia University's Data Sciences Institute evaluated the broader use of new algorithms for many challenging control applications.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

The Sagittarius stream in Gaia Early Data Release 3 and the origin of the bifurcations

The Sagittarius dwarf spheroidal (Sgr) is a dissolving galaxy being tidally disrupted by the Milky Way (MW). Its stellar stream still poses serious modelling challenges, which hinders our ability to use it effectively as a prospective probe of the MW gravitational potential at large radii. Our goal is to construct the largest and most stringent sample of stars in the stream with which we can advance our understanding of the Sgr-MW interaction, focusing on the characterization of the bifurcations. We improved on previous methods based on the use of the wavelet transform to systematically search for the kinematic signature of the Sgr stream throughout the whole sky in the Gaia data. We then refined our selection via the use of a clustering algorithm on the statistical properties of the colour-magnitude diagrams. Our final sample contains more than 700 000 candidate stars and is three times larger than previous Gaia samples. With it, we have been able to detect the bifurcation of the stream in both the northern and southern hemispheres, which requires four branches (two bright and two faint) to fully describe the system. We present the detailed proper motion distribution of the trailing arm as a function of the angular coordinate along the stream, showing, for the first time, the presence of a sharp edge (on the side of the small proper motions) beyond which there are no Sgr stars. We also characterize the correlation between kinematics and distance. Finally, the chemical analysis of our sample shows that the faint branch of the bifurcation is more metal poor than the bright. We provide analytical descriptions for the proper motion trends as well as for the sky distribution of the four branches of the stream. Based on our analysis, we interpret the bifurcations as a misaligned overlap of the material stripped at the antepenultimate pericentre (faint branches) with the stars ejected at the penultimate pericentre (bright branch), given that Sgr just went through its perigalacticon. The source of this misalignment is still unknown, but we argue that models with some internal rotation in the progenitor – at least during the time of stripping of the stars that are now in the faint branches – are worth exploring.

79 ASTRONOMY AND ASTROPHYSICS↗

Estimate of the Mass and Radial Profile of the Orphan–Chenab Stream's Dwarf-galaxy Progenitor Using MilkyWay@home

We fit the mass and radial profile of the Orphan–Chenab Stream's (OCS) dwarf-galaxy progenitor by using turnoff stars in the Sloan Digital Sky Survey and the Dark Energy Camera to constrain N-body simulations of the OCS progenitor falling into the Milky Way on the 1.5 PetaFLOPS MilkyWay@home distributed supercomputer. We infer the internal structure of the OCS's progenitor under the assumption that it was a spherically symmetric dwarf galaxy composed of a stellar system embedded in an extended dark matter halo. We optimize the evolution time, the baryonic and dark matter scale radii, and the baryonic and dark matter masses of the progenitor using a differential evolution algorithm. The likelihood score for each set of parameters is determined by comparing the simulated tidal stream to the angular distribution of OCS stars observed in the sky. We fit the total mass of the OCS's progenitor to (2.0 ± 0.3) × 10 7 M ⊙ with a mass-to-light ratio of γ = 73.5 ± 10.6 and (1.1 ± 0.2) × 10 6 M ⊙ within 300 pc of its center. Within the progenitor's half-light radius, we estimate a total mass of (4.0 ± 1.0) × 10 5 M ⊙ . We also fit the current sky position of the progenitor's remnant to be (α, δ) = ((166.0 ± 0.9)°, (–11.1 ± 2.5)°) and show that it is gravitationally unbound at the present time. The measured progenitor mass is on the low end of previous measurements and, if confirmed, lowers the mass range of ultrafaint dwarf galaxies. Our optimization assumes a fixed Milky Way potential, OCS orbit, and radial profile for the progenitor, ignoring the impact of the Large Magellanic Cloud.

79 ASTRONOMY AND ASTROPHYSICS↗

Assessing Historical Variability of South Asian Monsoon Lows and Depressions With an Optimized Tracking Algorithm

Abstract Cyclonic low‐pressure systems (LPS) produce abundant rainfall in South Asia, where they are traditionally categorized as monsoon lows, monsoon depressions, and more intense cyclonic storms. The India Meteorological Department (IMD) has tracked monsoon depressions for over a century, finding a large decline in their number in recent decades, but their methods have changed over time and do not include monsoon lows. This study presents a fast, objective algorithm for identifying monsoon LPS and uses it to assess interannual variability and trends in reanalyses. Variables and thresholds used in the algorithm are selected to best match a subjectively analyzed LPS data set while minimizing disagreement between four reanalyses in a training period. The stream function of 850 hPa horizontal wind is found to be optimal in this sense; it is less noisy than vorticity and represents the complete nondivergent wind, even when flow is not geostrophic. Using this algorithm, LPS statistics are computed for five reanalyses, and none show a detectable trend in monsoon depression counts since 1979. Both the Japanese 55‐year Reanalysis (JRA‐55) and the IMD data set show a step‐like reduction in depression counts when they began using geostationary satellite data, in 1979 and 1982, respectively; the 1958–2018 linear trend in JRA‐55, however, is smaller than in the IMD data set, and its error bar includes 0. There are more LPS in seasons with above‐average monsoon rainfall and in La Niña years, but few other large‐scale modes of interannual variability are found to modulate LPS counts, lifetimes, or track length consistently across reanalyses.

54 ENVIRONMENTAL SCIENCES↗

Tactical Analysis for Calculating Contextual Risk at Boundaries: Summary of Laboratory Directed Research & Development Effort

The Tactical Analysis for Calculating Contextual Risk at Boundaries (TACCRAB) tool is an innovative digital twin (DT) platform and automated risk algorithm designed to transform operational decision-making in structured screening environments, with an initial focus on Southern Border Land Ports of Entry (POEs). The invention provides integration points for advanced artificial intelligence, predictive modeling, and real-time data analysis to produce a comprehensive risk management tool that enables proactive, data-informed security strategies. The core inventive features of TACCRAB center on its unique risk algorithm, which dynamically calculates contextual risk by synthesizing historical data, near real-time streaming data from the checkpoints themselves, and AI-generated predictions. Unlike traditional risk assessment methods, TACCRAB utilizes a DT to provide comprehensive operational insights, allowing stakeholders to visualize, simulate, and optimize checkpoint configurations with unprecedented speed and contextual awareness. TACCRAB's key innovation lies in its ability to combine multiple complex inputs - including technology detection probabilities, resource availability, screening pathway characteristics, and threat actor behavioral patterns - into a unified risk calculation and update these inputs based on changing operational and environmental conditions. By leveraging a DT that continuously updates and learns from linked data, TACCRAB can suggest adaptive mitigation strategies that minimize risk while maintaining operational efficiency. Particularly novel is the platform's approach to decision support, which goes beyond static risk assessment. The DT provides dynamic metrics such as wait times, resource allocation effectiveness, and potential emerging threat scenarios, enabling users to view sophisticated, relevant what-if simulations and optimize checkpoint operations in near real-time. The system's architecture allows for generalized application across different screening environments, such as secure facilities, ports of entry, and soft targets, making it a versatile tool for security and operational management. The invention distinguishes itself through its comprehensive integration of predictive modeling, AI-driven pattern discovery, and user-friendly interface design. By combining these elements, TACCRAB transforms complex risk data into actionable insights, supporting decision-makers at various organizational levels - from booth agents making split-second screening decisions to checkpoint managers optimizing the day's resource allocation to strategic planners managing long-term investments.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

Theory of an Automatic Seepage Meter and Ramifications for Applications

A new approach for measuring fluxes across surface water—groundwater interfaces was recently proposed. The Automatic Seepage Meter (ASM) is equipped with a precise water level sensor and digital memory that analyzes water level time series in a vertical tube inserted into a streambed. The ability to infer flux values with high temporal resolution relies on an accurate interpretation of water level dynamics inside the tube. Here, we reduce the three-dimensional hydrodynamic problem that describes the ASM water level in a variety of field conditions to a single ordinary differential equation. This novel general analytical solution for estimating ASM responses is more comprehensive and flexible than previous approaches and is applicable to the entire range of field conditions, including steady or transient stream stages, evaporation, rainfall, and noise. For example, our analysis determines the timing of the nonmonotonic ASM response to a monotonic linear stream stage variation and explains previously used empirical parabolic approximation for estimating fluxes. We present algorithms for simultaneous inference of vertical interface flux and hydraulic conductivity values together with an example code. We quantify how the accuracy of parameter estimation depends on test duration and noise amplitude and propose how our analysis can be used to optimize field test protocols. On this basis, changing the ASM geometry by increasing the radius and decreasing tube insertion depth may enable ASM field test protocols that estimate interface flux and hydraulic conductivity faster while maintaining desired accuracy. Potential applications of joint parameter estimation are suggested.

47 OTHER INSTRUMENTATION↗

A machine learning decision criterion for reducing scan time for hyperspectral neutron computed tomography systems

We present the first machine learning-based autonomous hyperspectral neutron computed tomography experiment performed at the Spallation Neutron Source. Hyperspectral neutron computed tomography allows the characterization of samples by enabling the reconstruction of crystallographic information and elemental/isotopic composition of objects relevant to materials science. High quality reconstructions using traditional algorithms such as the filtered back projection require a high signal-to-noise ratio across a wide wavelength range combined with a large number of projections. This results in scan times of several days to acquire hundreds of hyperspectral projections, during which end users have minimal feedback. To address these challenges, a golden ratio scanning protocol combined with model-based image reconstruction algorithms have been proposed. This novel approach enables high quality real-time reconstructions from streaming experimental data, thus providing feedback to users, while requiring fewer yet a fixed number of projections compared to the filtered back projection method. In this paper, we propose a novel machine learning criterion that can terminate a streaming neutron tomography scan once sufficient information is obtained based on the current set of measurements. Our decision criterion uses a quality score which combines a reference-free image quality metric computed using a pre-trained deep neural network with a metric that measures differences between consecutive reconstructions. The results show that our method can reduce the measurement time by approximately a factor of five compared to a baseline method based on filtered back projection for the samples we studied while automatically terminating the scans.

97 MATHEMATICS AND COMPUTING↗

Alert Classification for the ALeRCE Broker System: The Light Curve Classifier

We present the first version of the Automatic Learning for the Rapid Classification of Events (ALeRCE) broker light curve classifier. ALeRCE is currently processing the Zwicky Transient Facility (ZTF) alert stream, in preparation for the Vera C. Rubin Observatory. The ALeRCE light curve classifier uses variability features computed from the ZTF alert stream and colors obtained from AllWISE and ZTF photometry. We apply a balanced random forest algorithm with a two-level scheme where the top level classifies each source as periodic, stochastic, or transient, and the bottom level further resolves each of these hierarchical classes among 15 total classes. This classifier corresponds to the first attempt to classify multiple classes of stochastic variables (including core- and host-dominated active galactic nuclei, blazars, young stellar objects, and cataclysmic variables) in addition to different classes of periodic and transient sources, using real data. We created a labeled set using various public catalogs (such as the Catalina Surveys and Gaia DR2 variable stars catalogs, and the Million Quasars catalog), and we classify all objects with ≥6 g-band or ≥6 r-band detections in ZTF (868,371 sources as of 2020 June 9), providing updated classifications for sources with new alerts every day. For the top level we obtain macro-averaged precision and recall scores of 0.96 and 0.99, respectively, and for the bottom level we obtain macro-averaged precision and recall scores of 0.57 and 0.76, respectively. Updated classifications from the light curve classifier can be found at the ALeRCE Explorer website (http://alerce.online).

47 OTHER INSTRUMENTATION↗

A Reproducible Validation of Algorithms for Estimating Array Tilt and Azimuth from Photovoltaic Power Time Series

In this research, we assess the viability of four different, publicly available algorithms for estimating the azimuth and tilt parameters of solar photovoltaic systems using only the associated AC power time series data and site latitude-longitude coordinates. In this work, we curated a benchmarking data set of 44 fixed-tilt systems, comprising 275 measured AC power inverter data streams, with known azimuth and tilt parameters. Additionally, we isolated test cases in the data set with real-world issues, including shading and clipping, to determine how algorithm performance varies based on the presence of these phenomena. Using this data set for benchmarking, we evaluated the estimated vs. actual system characteristics for each algorithm, as well as the associated algorithm execution time using a standardized benchmarking process. The two highest performing algorithms were the Solar Data Tools and the PVWatts 5-based methods, which both achieved a median absolute error of approximately 5 and 1 degrees for azimuth and tilt, respectively. During run time analysis, the SDT method was approximately 5 times faster than the PVWatts 5-based method, with the median execution time for a stream varying between 6 and 8 seconds vs. a median run time of 31 seconds for the PVWatts 5-based method.

algorithm validation↗

Parallel Algorithms for Computing the Tensor-Train Decomposition

The tensor-train (TT) decomposition expresses a tensor in a data-sparse format used in molecular simulations, high-order correlation functions, and optimization. In this paper, we propose four parallelizable algorithms that compute the TT format from various tensor inputs: (1) Parallel-TTSVD for traditional format, (2) PSTT and its variants for streaming data, (3) Tucker2TT for Tucker format, and (4) TT-fADI for solutions of Sylvester tensor equations. We provide theoretical guarantees of accuracy, parallelization methods, scaling analysis, and numerical results. For example, for a d-dimension tensor in $\mathbb{R}$ $n\times∙∙∙$$\times$$n$ a two-sided sketching algorithm PSTT2 is shown to have a memory complexity of $O(n^{[d/2]})$, improving upon $O(n^{d—1})$ from previous algorithms.

97 MATHEMATICS AND COMPUTING↗