SEARCH · Engineering Papers

Results for “streaming algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

All sources NASA NTRS DOE OSTI

DOE OSTI2025

RICH (Robotic Interface Control & Handling) System for VULCAN

High flux neutron beam and high efficiency detectors warrant quick turn arounds of neutron diffraction measurements at the engineering materials diffractometer VULCAN. Efficient change and alignment of samples and automatic measurements at VULCAN are desired for better use of neutron beam time by users. In this work, we aim to develop a proof-of-concept Robotic Interactive Control & Handling System (RICH) for sample handling at VULCAN that could assist high throughput experiments and reduce the overhead time significantly. This was realized by a six-axis desktop robot with trained AI models. The trained AI models can recognize and locate samples in various types of positions in a live video stream. In addition, we developed smart algorithms which used our models on multiple cameras to recognize where samples are with respect to multiple point of views, and then using user-inputted parameters, align them to perform complex measurements.

42 ENGINEERING↗

DOE OSTI2020

Fast Data Processing for Hyperspectral Sensors on Small Platforms

Hyperspectral imaging is a very promising technology for nuclear proliferation detection. However, due to size and weight restrictions, small hyperspectral platforms such as satellites and small drones lack the on-board computing resources for accurate, real-time analysis of the enormous flow of data that a continuously operating hyperspectral sensor generates. This severely limits satellite systems, which can collect far more data than what they can telemeter, and hinders the ability of all platforms to adapt their missions on the fly in response to observations. This program addresses the hyperspectral data processing challenge through development of new, fast and accurate algorithms that produce data products in real time. The algorithms circumvent the major computational bottlenecks in existing processing streams, and would be incorporated in lightweight, power-efficient single-board computer systems. The toolkit of fast algorithms will be immediately useful in current and future hyperspectral systems being built by the Government and by private industry, including drone-based systems and satellite constellations that acquire timely global imagery.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

DOE OSTI2026

Data Summarization and Inference at Scale

This is the final report for the DOE ASCR grant SC-0022260, Data Summarization and Inference at Scale, PI: Alex Pothen, Purdue University. The goal of the project was to solve data-intensive and compute-intensive problems in the physical sciences, engineering, information science, data science, etc. by designing and implementing new algorithms that could work with a subset of the data. The four subgoals were: (a) The solution of problems where the data is too large to be stored in the memory of a computer. In this streaming model of computation, the data arrives as a stream of elements to the computer, each element is processed as it arrives, and a decision is made to discard the data or to store it; only a small subset of the data proportional to the size of the output solution is stored, and when all the data has been streamed, a solution to the problem is computed from the stored subset. (b) The use of machine learning methods to compute solutions to data-intensive problems. The use of GPUs is critical to obtain high performance on machine learning tasks, but their memory sizes are smaller relative to that of CPUs. For large-scale problems, the data is sampled many times, and small samples are used with repetition, for robustness, to compute solutions to inference tasks. This sampling reduces the memory required to solve the problem, but attention is needed to avoid slow convergence to the solutions, and reduced accuracy of inference. We propose submodular optimization, Large Language Models, and physics-informed neural networks to enable GPU computations here. (c) Modeling and visualization of high-dimensional data using interpretable features. Clinical proteomic data sets from immunology for the detection of cancer and other diseases are temporal and high-dimensional, and algorithms for visualizing these data sets using clinically interpretable features are lacking. We propose methods that compute distances based on the optimal transportation problem and graph edit distances to address this problem. We also propose the use of optimal transport-based distances, spatial statistics, and network structure to classify image data sets, We apply these algorithms to electron micrographs of the peripheral nervous system in the digestive tract. (d) The design of data-intensive algorithms on emerging architectures, specifically, noisy, intermediate-scale quantum (NISQ) devices. Quantum computers offer the possibility of exploring large solution spaces due to the principle of superposition, but current quantum computers are limited by few qubits, short coherence times due to noise, poor interconections among the qubits, etc. We propose the use of the divide and conquer paradigm to solve large-scale problems, wherein collections of small subproblems are solved on the quantum devices, and the solutions to the subproblems are integrated into a solution for the original problem on a classical computer.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Integration Development and Testing of Rear Transition Monitor for Beam Current Monitoring System

Addressing baseline effects in accelerator environments is crucial for accurate data acquisition and analysis, since baseline effects can obscure signal clarity and impact the reliability of beam current monitoring systems. There are many potential contributors to baseline noise, such as variations in beam dynamics, electromagnetic interference from nearby equipment, or RF interference. Previous applications of noise reduction systems don t sufficiently filter sources of asynchronous noise, so a new algorithm was implemented. A simulation dataset was created to replicate beam conditions and a Red Pitaya FPGA was used to collect data through the streaming application. A Python script was developed to implement noise reduction algorithms and efforts were made to integrate real-time data streaming with the Redis platform and Acnet Front End infrastructure.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2021

Hardening DOE R&D Software Tools for Web-based Visualization SBIR Phase I Final Report

Ubiquitous web-based visualization is essential to delivering large-scale data visualization to various stakeholders, from the scientist to the board member. These stakeholders will not tolerate a stalled application or a pop-up window asking them to wait for the processing to complete. They require a responsive and interactive visualization environment with high-quality imagery suitable for detailed analysis and boardroom presentations. At Kitware, Inc., we have accomplished web visualization to this point, leveraging state-of-the-art tools like HTML5, CSS3, SVG, Canvas, and WebGL. Solutions that leverage a combination of these technologies are necessary to handle workloads that vary significantly in data size efficiently. However, it is not always practical to move large data to the web client for visualization. Kitware's ParaView as a Service combines client-side visualization using both distributed processing and remote rendering on big data impractical to move. Existing distributed processing and remote rendering solution's interactivity is below the expectations of web-based applications. Our project examined proposed solutions to the areas outlined above in ParaView as a Service. We have investigated concurrent pipelines, streaming images, progressive rendering, and optimization of algorithms and data movement to address these concerns. For the Phase I project, we completed the proposed work plan. As a result, the project produced three prototypes of essential importance for web visualization and the ParaView as a Service community. We created a simple desktop application for an interactive streamline placement prototype, a web-based interactive streamline placement prototype, and a web-based progressive rendering utilizing raytracing prototype. These prototypes relied on the hardening of emerging software toolkits funded by the Department of Energy (DOE) Advanced Scientific Computing Research (ASCR) program (such as ParaView, VTK-m, and Mochi). We blended these components into web-based visualization prototypes that meet the industry's expectations for interactivity and responsiveness. The Phase I project had four essential focus areas: 1. Develop prototype ParaView as a Service backend server using asynchronous, non-blocking design principles. 2. Develop a prototype web application that uses the ParaView as a Service backend server for remote data visualization. 3. Implement image streaming with encoding/compression and progressive rendering capabilities in the proposed platform. 4. Evaluate the prototype developed and summarize observations, including the challenges and pitfalls of our approach. After our successful completion of Phase I, we are strongly positioned to propose a successful Phase II project.

Geveci, Berk↗

DOE OSTI2020

Unitary quantum lattice simulations for Maxwell equations in vacuum and in dielectric media

Utilizing the similarity between the spinor representation of the Dirac and the Maxwell equations that has been recognized since the early days of relativistic quantum mechanics, a quantum lattice algorithm (QLA) representation of unitary collision-stream operators of Maxwell's equations is derived for both homogeneous and inhomogeneous media. A second-order accurate 4-spinor scheme is developed and tested successfully for two-dimensional (2-D) propagation of a Gaussian pulse in a uniform medium whereas for normal (1-D) incidence of an electromagnetic Gaussian wave packet onto a dielectric interface requires 8-component spinors because of the coupling between the two electromagnetic polarizations. In particular, the well-known phase change, field amplitudes and profile widths are recovered by the QLA asymptotic profiles without the imposition of electromagnetic boundary conditions at the interface. The QLA simulations yield the time-dependent electromagnetic fields as the wave packet enters and straddles the dielectric boundary. QLA involves unitary interleaved non-commuting collision and streaming operators that can be coded onto a quantum computer: the non-commutation being the very reason why one perturbatively recovers the Maxwell equations.

Physics↗

DOE OSTI2026

Memory-Aware External Facelist Calculation: A Data-Parallel Atomic Hash Counting Approach

Unstructured volumetric meshes serve as fundamental data representations in various scientific simulations and analyses. They play a crucial role in representing complex computational domains and are essential for important numerical techniques, such as finite element analysis. Whenever such a mesh is read from a file, streamed in-situ, or generated by algorithms, scientific visualization libraries rely on calculating the external surface of a geometry, named “external facelist”, to produce a polygonal mesh for rendering. Consequently, external facelist calculation has become one of the most widely used algorithms in the scientific visualization domain, necessitating optimal performance. In this paper, we explore relevant work on external facelist calculation algorithms in two common visualization libraries, VTK and Viskores, assess their performance and memory constraints, and introduce a novel memory-aware external facelist calculation algorithm employing an atomic hash counting approach. This algorithm fully leverages Viskores' data-parallel primitive operations, facilitating its execution across diverse many-core architectures. Our algorithm features the lowest memory footprint on the GPU and the second-lowest on the CPU among all evaluated methods, and it also delivers the fastest performance on both CPU and GPU. It has been made available under an open-source license in the VTK and Viskores visualization systems.

Tsalikis, Spiros [Kitware] (ORCID:0000000151137195↗

DOE OSTI2024

Online randomized interpolative decomposition with a posteriori error estimator for temporal PDE data reduction

Traditional low-rank approximation is a powerful tool for compressing large data matrices that arise in simulations of partial differential equations (PDEs), but suffers from high computational cost and requires several passes over the PDE data. The compressed data may also lack interpretability thus making it difficult to identify feature patterns from the original data. Here, to address these issues, we present an online randomized algorithm to compute the interpolative decomposition (ID) of large-scale data matrices in situ. Compared to previous randomized IDs that used the QR decomposition to determine the column basis, we adopt a streaming ridge leverage score-based column subset selection algorithm that dynamically selects proper basis columns from the data and thus avoids an extra pass over the data to compute the coefficient matrix of the ID. In particular, we adopt a single-pass error estimator based on the non-adaptive Hutch++ algorithm to provide real-time error approximation for determining the best coefficients. As a result, our approach only needs a single pass over the original data and thus is suitable for large and high-dimensional matrices stored outside of core memory or generated in PDE simulations. A strategy to improve the accuracy of the reconstructed data gradient, when desired, within the ID framework is also presented. We provide numerical experiments on turbulent channel flow and ignition simulations, and on the NSTX Gas Puff Image dataset, comparing our algorithm with the offline ID algorithm to demonstrate its utility in real-world applications.

Column subset selection↗

DOE OSTI2021

Streaming Generalized Canonical Polyadic Tensor Decompositions

In this paper, we develop a method which we call OnlineGCP for computing the Generalized Canonical Polyadic (GCP) tensor decomposition of streaming data. GCP differs from traditional canonical polyadic (CP) tensor decompositions as it allows for arbitrary objective functions which the CP model attempts to minimize. This approach can provide better fits and more interpretable models when the observed tensor data is strongly non-Gaussian. In the streaming case, tensor data is gradually observed over time and the algorithm must incrementally update a GCP factorization with limited access to prior data. In this work, we extend the GCP formalism to the streaming context by deriving a GCP optimization problem to be solved as new tensor data is observed, formulate a tunable history term to balance reconstruction of recently observed data with data observed in the past, develop a scalable solution strategy based on segregated solves using stochastic gradient descent methods, describe a software implementation that provides performance and portability to contemporary CPU and GPU architectures and integrates with Matlab for enhanced usability, and demonstrate the utility and performance of the approach and software on several synthetic and real tensor data sets.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Energy Efficient Streaming Time Series Classification with Attentive Power Iteration

Efficiently processing time series data streams in real-time on resource-constrained devices offers significant advantages in terms of enhanced computational energy efficiency and reduced time-related risks. We introduce an innovative streaming time series classification network that utilizes attentive power iteration, enabling real-time processing on resource-constrained devices. Our model continuously updates a compact representation of the entire time series, enhancing classification accuracy while conserving energy and processing time. Notably, it excels in streaming scenarios without requiring complete time series access, enabling swift decisions. Experimental results show that our approach excels in classification accuracy and energy efficiency, with over 70% less consumption and threefold faster task completion than benchmarks. This work advances real-time responsiveness, energy conservation, and operational effectiveness for constrained devices, contributing to optimizing various applications.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Machine Learning Enabled Sensor Fusion for In-Situ Defect Detection in Laser Powder Bed Fusion

Laser Powder Bed Fusion (L-PBF) Additive Manufacturing (AM) is among the metal 3D printing technologies most broadly adopted by the manufacturing industry. The current industry qualification paradigm for critical-application L-PBF parts relies heavily on expensive non-destructive inspection techniques such as X-Ray Computed Tomography (XCT), which significantly limits the use-cases of L-PBF. In situ monitoring of the process promises a less expensive alternative to ex situ testing, but existing sensor technologies and data analysis techniques struggle to detect sub-surface flaws (e.g., porosity and cracking) on production-scale L-PBF printers. RTX Technologies Research Center (RTRC) has licensed ORNL’s Peregrine software package – a printer- and camera-agnostic data analytics tool designed specifically for detecting process anomalies using in situ data collected during powder bed printing. The goal of this project was to feed temporally rich, multi-modal sensor data, including visible light, integrated near infrared (NIR), and spatially mapped co-axial melt pool thermal emission data into Peregrine to enable detection of subsurface flaws. XCT data was used as ground truth training data to allow Peregrine’s deep learning algorithms to recognize anomalies in these complex data streams in both test artifacts and industrially relevant geometries. Completion of this program has seen the successful implementation of multi-modal, multi-layer sensor data footprints for training of machine learning models in Peregrine. Flaws detected in XCT data have been successfully detected directly from this in situ data footprint, and initial analyses of the in situ probability-of-detection has been conducted, showing performance levels commensurate with traditional non-destructive evaluation (NDE) methods. The in situ monitoring methodology was then applied to an industrially relevant component that was using post-build NDE, highlighting the utility of the proposed method for hard-to-inspect AM components. As a direct result of this program, two journal manuscripts [1], [2] have been published in Additive Manufacturing, with additional manuscripts planned following program completion.

36 MATERIALS SCIENCE↗

DOE OSTI2024

Lead tungstate calorimeters at Jefferson Lab and perspectives for the Electron–Ion Collider

Electromagnetic calorimeters based on PbWO4 scintillating crystals have a widespread applica- tion in experiments at different accelerator facilities such as CERN, FNAL, GSI, and Jefferson Lab. The unique properties of PbWO4 crystals, including a small radiation length and Molire radius, make them ideal for building high-granularity, radiation-hard detectors. This enables excellent spa- tial separation and energy resolution of reconstructed electromagnetic showers, making PbWO4 crystals the material of choice for numerous experiments. Lead tungstate calorimeters have been successfully used in several experiments at Jefferson Lab. Two large-scale detectors have been re- cently fabricated for future experiments : the Neutral Particle Spectrometer and the lead tungstate calorimeter of the GlueX detector. The future application of PbWO4 crystals in the ElectronIon Collider further highlights their ongoing importance in advancing experimental capabilities. In planning new experiments, the development of calorimeter instrumentation technologies becomes paramount. The integration of modern photodetectors, such as Silicon photomultipliers that are capable of operating in strong magnetic fields, and the implementation of streaming readout data acquisition systems, sophisticated shower reconstruction algorithms, and real-time data analysis are some examples of the continuously growing requirements of experimental setups. I will give an overview of the lead tungstate scintillating calorimeters and discuss some recent advancement in the calorimeter instrumentation.

Somov, Alexander↗

DOE OSTI2022

Automated Shift Detection in Sensor-Based PV Power and Irradiance Time Series

PV power and irradiance sensor-based measurements are prone to error, resulting in issues such as time series data shifts. In this research, a changepoint detection (CPD) algorithm that automatically detects data shifts in sensor-based time series is introduced. Data shift periods in 101 daily PV power and irradiance time series were labeled manually by two solar experts. These data streams represent sensor-based measurements, and display a variety of data shift behaviors. A changepoint detection algorithm was tuned using the 101 labeled data streams, with each model configuration's ability to detect labeled changepoints benchmarked using metrics such as F1-score, recall, and Rand Index. Best performing models on seasonality-corrected data streams include the Pruned Exact Linear (PELT) method, the Binary Segmentation method, and the Bottom-Up method, all scoring an average F1-score of 0.76 or greater at detecting labeled changepoints within a 30-day window across the labeled data sets. Pending approval, we plan to release the labeled data sets for this research on NREL's DuraMAT Data Hub, and the associated algorithm in the Python PVAnalytics package. By supplying the training sets and algorithm, we hope to encourage further development in this research space.

data shift↗

DOE OSTI2022

Automated Shift Detection in Sensor-Based PV Power and Irradiance Time Series: Preprint

PV power and irradiance sensor-based measurements are prone to error, resulting in issues such as abrupt time series data shifts. These shifts, which are usually unintentional, may be caused by software or hardware configuration changes on a PV system, and do not reflect an actual change in overall system performance. Locating these shifts and segmenting the associated time series aids in more accurate future PV analysis. In this research, an offline changepoint detection (CPD) algorithm that automatically detects these abrupt data shifts in sensor-based time series is introduced. Data shift periods in 101 daily PV power and irradiance time series were labeled manually by two solar experts. These data streams represent sensor-based measurements, and display a variety of data shift behaviors. A changepoint detection algorithm was tuned using the 101 labeled data streams, with each model configuration's ability to detect labeled changepoints benchmarked using metrics such as F1-score, recall, and Rand Index. Best performing models on seasonality-corrected data streams include the Pruned Exact Linear (PELT) method, the Binary Segmentation method, and the Bottom-Up method, all scoring an average F1-score of 0.76 or greater at detecting labeled changepoints within a 30-day window for the labeled data sets. To promote further research in this space, we are releasing the labeled data shift sets on U.S. Department of Energy's (DOE) DuraMAT Data Hub, and the associated algorithm in the Python PVAnalytics package.

changepoint detection↗

DOE OSTI2022

Automated Shift Detection in Sensor-Based PV Power and Irradiance Time Series

changepoint detection↗

DOE OSTI2024

Stellar Population Properties in the Stellar Streams around SPRC047

Abstract We have investigated the properties (e.g., age, metallicity) of the stellar populations of a ringlike tidal stellar stream (or streams) around the edge-on galaxy SPRC047 (z= 0.031) using spectral energy distribution (SED) fits to integrated broadband aperture flux densities. We used visual images in six different bands and Spitzer/IRAC 3.6μm data. We have attempted to derive best-fit stellar population parameters (metallicity, age) in three noncontiguous segments of the stream. Due to the very low surface brightness of the stream, we have performed a deconvolution with a Richardson–Lucy–type algorithm of the low spatial resolution 3.6μm IRAC image, thereby reducing the effect of the point-spread function aliasedemissionfrom the bright edge-on central galaxy at the locations of our three stream segments. Our SED fits that used several different star formation (SF) history priors, from an exponentially decaying SF burst to continuous SF, indicate that the age–metallicity–dust degeneracy is not resolved, most likely because of inadequate wavelength coverage and low signal-to-noise ratios of the low surface brightness features. We also discuss how future deep visual–near-infrared observations, combined with absolute flux calibration uncertainties at or below the 1% level, complemented by equally well absolute flux-calibrated observations in ultraviolet and mid-infrared bands, would improve the accuracy of broadband SED fitting results for low surface brightness targets, such as stellar streams around nearby galaxies that are not resolved into stars.

Astronomy & Astrophysics↗

DOE OSTI2026

Accelerating multigrid with streaming chiral SVD for Wilson fermions in lattice QCD

A modification to the setup algorithm for the multigrid preconditioner of Wilson fermions in lattice QCD is presented. A larger basis of test vectors than that used in regular multigrid is calculated by the smoother and truncated by singular value decomposition on the chiral components of the test vectors. The truncated basis is used to form the prolongation and restriction matrices of the multigrid hierarchy. This modification of the setup method is demonstrated to increase the convergence of linear solvers on an anisotropic lattice with m π ≈ 239 MeV from the Hadron Spectrum Collaboration and an isotropic lattice with m π ≈ 220 MeV from the MILC Collaboration. The lattice volume dependence of the method is also examined. Increasing the number of test vectors improves speedup up to a point, but storing these vectors becomes impossible in limited memory resources such as GPUs. To address storage cost, we implement a streaming singular value decomposition of the basis of test vectors on the chiral components and demonstrate a decrease in the number of fine level iterations by a factor of 1.7 for m q ≈ m crit

Iterative methods↗

DOE OSTI2025

Reconstruction framework advancements to support streaming for the ePIC detector at the EIC

The ePIC collaboration adopted the JANA2 framework to manage its reconstruction algorithms. This framework has since evolved substantially in response to ePIC’s needs. There have been three main design drivers: integrating cleanly with the Podio-based data models and other layers of the key4hep stack, enabling external configuration of existing components, and supporting timeframe splitting for streaming readout. The result is a unified component model featuring a new declarative interface for specifying inputs, outputs, parameters, services, and resources. This interface enables the user to instantiate, configure, and wire components via an external file. One critical new addition to the component model is a hierarchical decomposition of data boundaries into levels such as Run, Timeframe, PhysicsEvent, and Subevent. Two new component abstractions, Folder and Unfolder, are introduced in order to traverse this hierarchy, e.g. by splitting or merging. The pre-existing components can now operate at different event levels, and JANA2 will automatically construct the corresponding parallel processing topology. This means that a user may write an algorithm once, and configure it at runtime to operate on timeframes or on physics events. Overall, these changes mean that the user requires less knowledge about the framework internals, obtains greater flexibility with configuration, and gains the ability to reuse the existing abstractions in new streaming contexts.

Brei, Nathan [Thomas Jefferson National Accelerato↗