Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Relevant biochar characteristics influencing compressive strength of biochar-cement mortars

To counteract the contribution of CO 2 emissions by cement production and utilization, biochar is being harnessed as a carbon-negative additive in concrete. Increasing the cement replacement and biochar dosage will increase the carbon offset, but there is large variability in methods being used and many researchers report strength decreases at cement replacements beyond 5%. This work presents a reliable method to replace 10% of the cement mass with a vast selection of biochars without decreasing ultimate compressive strength, and in many cases significantly improving it. By carefully quantifying the physical and chemical properties of each biochar used, machine learning algorithms were used to elucidate the three most influential biochar characteristics that control mortar strength: initial saturation percentage, oxygen-to-carbon ratio, and soluble silicon. These results provide additional research avenues for utilizing several potential biomass waste streams to increase the biochar dosage in cement mixes without decreasing mechanical properties.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Architecture for Quantum-in-the Loop Real-Time Simulations for Designing Resilient Smart Grids

With the power grid growing more complex every day with the inclusion of new sensors and regulatory approvals that enable end-users and small local developers to participate in the grid, it is becoming challenging for conventional smart grid simulation, emulation, and testing technologies to keep up. In this work, we propose that quantum-encoded real-time simulations can be helpful under the new paradigm and operational circumstances to solve optimization problems for power grids. By leveraging the principles of quantum mechanics, the proposed quantum-in-loop (QIL) framework will enable better and faster optimization solutions based on real-world, real-time data streams, facilitating the real-time planning and operations of electrical grids that rely on millions of distributed sensors and controllers. Furthermore, QIL will allow researchers and engineers to assist utilities in designing, developing, and de-risking algorithms to optimize power grid operation and resilience by considering inputs from millions of grid-connected devices. QIL framework is being developed to have a self-limiting triage mechanism, which will help engineers and practitioners identify fundamental physical limits on quantum processors, revealing what quantum algorithms can and cannot do in utility-specific use cases and must continue to count on classical high-performance computing infrastructure.

digital real-time simulation↗
DOE OSTI2022

Sensing Electrical Networks Securely & Economically (SENSE)

The growing adoption of distributed energy resources (DERs) like battery energy storage systems and roof top solar/PV and the rapid penetration of electric vehicles (EVs), the electric grid is undergoing a major transformation with elevated stress on legacy grid assets. Despite a lot of expenditure to address these challenges, both in dollars and manpower, utilities have not been able to receive the value that was promised. The gains have been most visible at the transmission and substation level, especially where the main objective was improving operational and economic efficiency for the utility. Improving visibility and control at a few select points enhances the existing and established paradigm of centralized command and control. With changing load patterns, load types and the overall transition to an “active grid”, the centralized control and coordination paradigm gets challenged. To address the challenges, a new architecture and mechanism is needed, one that supports decentralized control and decision making, extracting value streams at the grid edge, particularly as the changes are fueled by transitions occurring in the distribution system. To address this, a communications and data processing platform, “GAMMA” was developed and demonstrated through the project. At the heart of the platform, are distributed, intelligent edge nodes with sensing and compute capabilities, that can record and analyze information locally. They are embedded in sensors and actuators specific to different distribution system applications. Phase 1 of the project focused on developing novel sensor technology that can be used for monitoring utility pole top distribution transformers. The sensors were designed with the objective of being low-cost, communicating with the GAMMA cloud using novel “delay-tolerant” networking using Bluetooth and a secure mobile application. They were non-intrusive in nature so that they can be installed quickly in the field, resulting in overall low cost of deployment and operations. Following the successful completion of Phase 1, the team manufactured 100 units for a field demonstration in Phase 2. The field demonstration was carried out on two real feeder systems with the local utility partner. In total, 100 sensors were installed and operated over a period of 6 months in the state of Georgia. The platform is operational end to end, with the cloud infrastructure deployed on a distributed, serverless environment that can serve multiple data streams, an analytics engine and a portal to securely view the data from multiple assets. The data collected through the GAMMA Mobile Phone app showcased the viability of the novel delay tolerant networking architecture, and the data processing algorithms developed through the course of the project, were successful in extracting important information about the overall network, improving the utility’s visibility and situational awareness in the distribution feeder.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

Phoenix: A Scalable Streaming Hypergraph Analysis Framework

We present Phoenix, a scalable hypergraph analytics framework for data analytics and knowledge discovery that was implemented on the leadership class computing platforms at Oak Ridge National Laboratory (ORNL). Our software framework comprises a distributed implementation of a streaming server architecture which acts as a gateway for various hypergraph generators/external sources to connect. Phoenix has the capability to utilize diverse hypergraph generators, including HyGen, a very large-scale hypergraph generator developed by ORNL. Phoenix incorporates specific algorithms for efficient data representation by exploiting hidden structures of the hypergraphs. Our experimental results demonstrate Phoenix’s scalable and stable performance on massively parallel computing platforms. Phoenix’s superior performance is due to the merging of high-performance computing with data analytic.

Kurte, Kuldeep↗
DOE OSTI2026

Curation and Dissemination of Complex Multi-Modal Datasets for Radiation Detection, Localization, and Tracking

The PANDAWN sensor network in Chicago, IL, is a state-of-the-art testbed for networked, multi-modal sensing. It integrates AI/data science methods into its operation, from data acquisition to automated data labeling and curation workflows. The curation and dissemination of diverse multi-modal datasets will enable the development of new radiological/nuclear (R/N) detection, localization, and tracking algorithms and methods relevant across the nonproliferation mission space. This article first introduces the PANDAWN sensor network and the features that make it stand out from previous multi-modal data acquisition efforts. We then review the various data streams acquired on the PANDAWN nodes and present the implementation of an automated data curation pipeline that includes the labeling of radiation and contextual data streams. Here, we finally provide a short overview of different studies that leveraged the curated datasets.

Data curation↗
DOE OSTI2025

sdt (Solar Data Tools) [SWR-25-130]

Solar Data Tools (sdt) is an open-source Python library for analyzing PV power (and irradiance) time-series data. It was developed to enable analysis of unlabeled PV data, i.e. with no model, no meteorological data, and no performance index required, by taking a statistical signal processing approach in the algorithms used in the package’s main data processing pipeline. Solar Data Tools empowers PV system fleet owners or operators to analyze system performance a hundred times faster even when they only have access to the most basic data stream—power output of the system.

Meyers-Im, Bennet [National Laboratory of the Rock↗
NASA NTRS1994

Some Practical Universal Noiseless Coding Techniques

Report discusses noiseless data-compression-coding algorithms, performance characteristics and practical consideration in implementation of algorithms in coding modules composed of very-large-scale integrated circuits. Report also has value as tutorial document on data-compression-coding concepts. Coding techniques and concepts in question "universal" in sense that, in principle, applicable to streams of data from variety of sources. However, discussion oriented toward compression of high-rate data generated by spaceborne sensors for lower-rate transmission back to earth.

Rice, Robert F.↗
NASA NTRSDate not supplied

AN INTRODUCTION TO THE GEONEX LEVEL-1G PRODUCTS: TOP-OF-ATMOSPHERE REFLECTANCE AND BRIGHTNESS TEMPERATURE

This paper introduces the GeoNEX (Geostationary-NASA Earth eXchange) Level-1G products of top-of-atmosphere (TOA) reflectance and brightness temperature. The products use data streams from the latest geostationary (GEO) sensors including the GOES-16/17 ABI and the Himawari-8/9 AHI. The GeoNEX processing pipeline starts by converting digital numbers to physical quantities with the latest radiometric calibration information. It integrates algorithms to automatically detect and remove residual geolocation errors, to estimate the pixel-wise data-acquisition time, and to accurately calculate the solar illumination angles for each pixel in the domain at every time step. The outputs are reprojected to a globally tiled common grid in geographic coordinates designed to facilitate inter-comparisons and/or synergies between the GeoNEX products and existing Earth observation datasets from polar-orbiting satellites. Therefore, the GeoNEX L1G products provide accurate and consistent TOA reflectance and brightness temperature datasets for scientific analyses and downstream product development.

Geostationary satellite, GOES-16, Himawari-8, NASA↗
DOE OSTI2025

A Morphological Model to Separate Resolved–Unresolved Sources in the DESI Legacy Surveys: Application in the LS4 Alert Stream

Separating resolved and unresolved sources in large imaging surveys is a fundamental step to enable downstream science, such as searching for extragalactic transients in wide-field time-domain surveys. Here we present our method to effectively separate point sources from the resolved, extended sources in the Dark Energy Spectroscopic Instrument (DESI) Legacy Surveys (LS). We develop a supervised machine learning model based on the Gradient Boosting algorithm XGBoost. The features input to the model are purely morphological and are derived from the tabulated LS data products. We train the model using ∼2 × 10 5 LS sources in the COSMOS field with HST morphological labels and evaluate the model performance on LS sources with spectroscopic classification from the DESI Data Release 1 (∼2 × 10 7 objects) and the Sloan Digital Sky Survey Data Release 17 (∼3 × 10 6 objects), as well as on ∼2 × 10 8 Gaia stars. A significant fraction of LS sources are not observed in every LS filter, and we therefore build a “Hybrid” model as a linear combination of two XGBoost models, each containing features combining aperture flux measurements from the “blue” (gr) and “red” (iz) filters. The Hybrid model shows a reasonable balance between sensitivity and robustness, and achieves higher accuracy and flexibility compared to the LS morphological typing. With the Hybrid model, we provide classification scores for ∼3 × 10 9 LS sources, making this the largest ever machine learning catalog separating resolved and unresolved sources. The catalog has been incorporated into the real-time pipeline of the La Silla Schmidt Southern Survey (LS4), enabling the identification of extragalactic transients within the LS4 alert stream.

astrostatistics↗
DOE OSTI2022

SNAD transient miner: Finding missed transient events in ZTF DR4 using k-D trees

Here we report the automatic detection of 11 transients (7 possible supernovae and 4 active galactic nuclei candidates) within the Zwicky Transient Facility fourth data release (ZTF DR4), all of them observed in 2018 and absent from public catalogs. Among these, three were not part of the ZTF alert stream. Our transient mining strategy employs 41 physically motivated features extracted from both real light curves and four simulated light curve models (SN Ia, SN II, TDE, SLSN-I). These features are input to a k-D tree algorithm, from which we calculate the 15 nearest neighbors. After pre-processing and selection cuts, our dataset contained approximately a million objects among which we visually inspected the 105 closest neighbors from seven of our brightest, most well-sampled simulations, comprising 89 unique ZTF DR4 sources. Our result illustrates the potential of coherently incorporating domain knowledge and automatic learning algorithms, which is one of the guiding principles directing the SNAD team. It also demonstrates that the ZTF DR is a suitable testing ground for data mining algorithms aiming to prepare for the next generation of astronomical data.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Real-time Event Detection Using Rank Signatures of Real-world PMU Data

Timely detection of power system events is a crucial task, which can facilitate the implementation of remedial actions to improve reliability, resiliency, and security of the system. Meanwhile, the widespread deployment of phasor measurement units (PMUs) makes it possible to develop data-driven event detection techniques. However, relying purely on data without incorporating domain knowledge for the event detection task in power systems poses substantial security and stability risks due to issues associated with data misinterpretation and model accuracy. In this regard, we propose a real-time event detection method using real-world PMU data by incorporating domain knowledge to adequately capture the event signatures. Specifically, we track the change in rank signatures of PMU data to accurately localize the events. To optimize the detection process, we incorporate an offline Bayesian optimization algorithm to tune the parameters by efficiently searching for the best values. The experiments using the real-world PMU dataset from a U.S. interconnection show that the proposed event detection approach can efficiently detect the events from PMU data streams with high accuracy.

Ghasemkhani, Amir↗
NASA NTRS1991

Exact and approximate solutions to the oblique shock equations for real-time applications

The derivation of exact solutions for determining the characteristics of an oblique shock wave in a supersonic flow is investigated. Specifically, an explicit expression for the oblique shock angle in terms of the free stream Mach number, the centerbody deflection angle, and the ratio of the specific heats, is derived. A simpler approximate solution is obtained and compared to the exact solution. The primary objectives of obtaining these solutions is to provide a fast algorithm that can run in a real time environment.

Hartley, T. T.↗
NASA NTRS1999

An O(Nm(sup 2)) Plane Solver for the Compressible Navier-Stokes Equations

A hierarchical multigrid algorithm for efficient steady solutions to the two-dimensional compressible Navier-Stokes equations is developed and demonstrated. The algorithm applies multigrid in two ways: a Full Approximation Scheme (FAS) for a nonlinear residual equation and a Correction Scheme (CS) for a linearized defect correction implicit equation. Multigrid analyses which include the effect of boundary conditions in one direction are used to estimate the convergence rate of the algorithm for a model convection equation. Three alternating-line- implicit algorithms are compared in terms of efficiency. The analyses indicate that full multigrid efficiency is not attained in the general case; the number of cycles to attain convergence is dependent on the mesh density for high-frequency cross-stream variations. However, the dependence is reasonably small and fast convergence is eventually attained for any given frequency with either the FAS or the CS scheme alone. The paper summarizes numerical computations for which convergence has been attained to within truncation error in a few multigrid cycles for both inviscid and viscous ow simulations on highly stretched meshes.

Thomas, J. L.↗
DOE OSTI2023

Online real-time learning of dynamical systems from noisy streaming data

Abstract Recent advancements in sensing and communication facilitate obtaining high-frequency real-time data from various physical systems like power networks, climate systems, biological networks, etc. However, since the data are recorded by physical sensors, it is natural that the obtained data is corrupted by measurement noise. In this paper, we present a novel algorithm for online real-time learning of dynamical systems from noisy time-series data, which employs the Robust Koopman operator framework to mitigate the effect of measurement noise. The proposed algorithm has three main advantages: (a) it allows for online real-time monitoring of a dynamical system; (b) it obtains a linear representation of the underlying dynamical system, thus enabling the user to use linear systems theory for analysis and control of the system; (c) it is computationally fast and less intensive than the popular extended dynamic mode decomposition (EDMD) algorithm. We illustrate the efficiency of the proposed algorithm by applying it to identify the Van der Pol oscillator, the chaotic attractor of the Henon map, the IEEE 68 bus system, and a ring network of Van der Pol oscillators.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1986

Development of an explicit multigrid algorithm for quasi-three-dimensional viscous flows in turbomachinery

A rapid quasi three-dimensional analysis was developed for blade-to-blade flows in turbomachinery. The analysis solves the unsteady Euler or thin layer Navier-Stokes equations in a body-fitted coordinate system. It accounts for the effects of rotation, radius change, and stream-surface thickness. The Baldwin-Lomax eddy-viscosity model is used for turbulent flows. The equations which are solved b a two-stage Runge-Kutta scheme made efficient by use of vectorization, a variable time-step, and a flux-based multigrid scheme, are described. A stability analysis is presented for the two-stage. Results for a flat-plate model problem show the applicability of the method to axial, radial, and rotating geometries. Results for a centrifugal impeller and a radial diffuser show that the quasi three-dimensional viscous analysis can be a practical design tool.

Chima, R. V.↗
NASA NTRS1983

Remote Sensing techniques used to characterize soil erosion in southwestern Sao Paulo state

Within randomly sampled squares of a 1 km x 1 km grid, rill/gullies frequency, land cover/land use type and shape of the slopes were extracted from aerial photographs of the Ribeirao Anhumas drainage basin. Mean slope gradient, stream frequency and slope length were calculated on topographic maps. Ground truth data on fine sand/coarse sand ratio and vegetation cover densities were obtained. The MSS-LANDSAT-2 data (CCTs) were analyzed using single-cell, cluster synthesis and slicer algorithms. Graphical and statistical analyses of the data indicate that different slope gradients and land cover/land use types are the most significant factors related to the soil erosion process. The digital analysis of MSS data allowed the association among gray level classes and vegetation cover classes, which defined seven classes. These gray level classes and slope gradient classes were used to rank erosion risk.

Parada, N. D. J.↗
NASA NTRS2011

Prioritized LT Codes

The original Luby Transform (LT) coding scheme is extended to account for data transmissions where some information symbols in a message block are more important than others. Prioritized LT codes provide unequal error protection (UEP) of data on an erasure channel by modifying the original LT encoder. The prioritized algorithm improves high-priority data protection without penalizing low-priority data recovery. Moreover, low-latency decoding is also obtained for high-priority data due to fast encoding. Prioritized LT codes only require a slight change in the original encoding algorithm, and no changes at all at the decoder. Hence, with a small complexity increase in the LT encoder, an improved UEP and low-decoding latency performance for high-priority data can be achieved. LT encoding partitions a data stream into fixed-sized message blocks each with a constant number of information symbols. To generate a code symbol from the information symbols in a message, the Robust-Soliton probability distribution is first applied in order to determine the number of information symbols to be used to compute the code symbol. Then, the specific information symbols are chosen uniform randomly from the message block. Finally, the selected information symbols are XORed to form the code symbol. The Prioritized LT code construction includes an additional restriction that code symbols formed by a relatively small number of XORed information symbols select some of these information symbols from the pool of high-priority data. Once high-priority data are fully covered, encoding continues with the conventional LT approach where code symbols are generated by selecting information symbols from the entire message block including all different priorities. Therefore, if code symbols derived from high-priority data experience an unusual high number of erasures, Prioritized LT codes can still reliably recover both high- and low-priority data. This hybrid approach decides not only "how to encode" but also "what to encode" to achieve UEP. Another advantage of the priority encoding process is that the majority of high-priority data can be decoded sooner since only a small number of code symbols are required to reconstruct high-priority data. This approach increases the likelihood that high-priority data is decoded first over low-priority data. The Prioritized LT code scheme achieves an improvement in high-priority data decoding performance as well as overall information recovery without penalizing the decoding of low-priority data, assuming high-priority data is no more than half of a message block. The cost is in the additional complexity required in the encoder. If extra computation resource is available at the transmitter, image, voice, and video transmission quality in terrestrial and space communications can benefit from accurate use of redundancy in protecting data with varying priorities.

Woo, Simon S.↗
NASA NTRSDate not supplied

A System to Provide Deterministic Flight Software Operation and Maximize Multicore Processing Performance: The Safe and Precise Landing – Integrated Capabilities Evolution (SPLICE) Datapath

A method and design are described for a system that processes multiple data streams, utilizing a multicore asymmetric processing architecture, that eliminates data interrupts to the application processors. The design supports a deterministic environment for flight software in NASA’s Safe and Precise Landing – Integrated Capabilities Evolution (SPLICE) project. The SPLICE project develops sensor, algorithm, and compute technologies for Precision Landing and Hazard Avoidance (PL&HA) capabilities. The compute technology for SPLICE is the Descent and Landing Computer (DLC). The DLC hosts several SPLICE algorithms with high computational resource requirements that must be executed in a real-time and deterministic manner. The software runs on a custom Single Board Computer (SBC), with a Xilinx Ultrascale+ Multiprocessor System-on-a-Chip (MPSoC). Input data for the flight software is from a variety of sensors, unique with respect to data rate and packet size. A data path between the SPLICE sensors and algorithms is designed to efficiently deliver this data to the flight software using the MPSoC asymmetric processing cores and Field Programmable Gate Array (FPGA) fabric. This is implemented in a manner that isolates the application processors running the flight software from interrupts associated with the input data. By leveraging real-time processors on the MPSoC, and a structure with the appropriate interfaces in the shared memory on the SBC, the flight software can use the full set of application processors. The available utilization for each processor in this set is also maximized for the SPLICE applications, providing a sufficiently deterministic execution environment without the cost and overhead of a real-time operating system.

heterogeneous processing system↗