Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 991 records · Page 55

Spectral Methods in Time-dependent Data Analysis

We aim to create a new model for time-dependent data analysis, named dynamical learning, that integrates data-driven manifold learning techniques with operator-theoretic methods from dynamical systems theory. This approach has the potential to deliver more efficient methods for analyzing time-dependent data, such as video streams, by naturally separating out the temporal and spatial features of the data. We aim to apply the newly developed methods to video surveillance data related to Sandia mission applications, and particularly focus on the problems of image segmentation and object tracking. This project ended early due to the departure of the PI from Sandia about 18 months into the project. Therefore, this document reports on partial progress towards the initial goals of the project. In addition, this document reports on part of the work conducted during the project; see the Appendix for a summary of all the work conducted during the 18 months.

97 MATHEMATICS AND COMPUTING↗

LANL Lightning Flash Time Duration Ground Truth

Video cameras were used to record cloud-to-ground (CG) lightning flash events on or near Los Alamos National Laboratory (LANL) over 3 days in August, 2018. Custom scripting software was created to automatically trigger camera recording off National Lightning Data Network (NLDN) streamed reporting of CG lightning within 12 miles of LANL. Cameras were sited near meteorology towers at LANL and 4k video captures were manually reviewed to extract still sequence images from 9 identified CG flashes. A camera frame rate of 30 fps was used to estimate time durations of 33-200 milliseconds over this natural lightning dataset.

58 GEOSCIENCES↗

Vulcan-Forge: Architecture and Design of a Multi-Modal Forensic Analysis Plugin for CALDERA

Forge and VULCAN together describe an open-architecture cybersecurity analysis ecosystem that unifies forensic artifact processing, detection engineering, and vulnerability intelligence within integrated platforms. Forge operates as a plugin for MITRE CALDERA, ingesting diverse evidence formats—including EVTX, PCAP/PCAPNG, CSV, JSON, YAML, XML, binaries, and archives—to construct a unified artifact graph enriched with severity scoring, TLP classification, and audit trails. It provides subsystems for artifact parsing, streaming structured-data visualization, NetworkMiner-based packet inspection, PE/.NET binary analysis, and LLM-assisted triage and rule generation, with outputs validated against CCCS-YARA and pySigma schemas. VULCAN complements this by serving as a cybersecurity analyst platform that integrates a Neo4j knowledge graph, Qdrant vector retrieval, SSVC-based triage, and a local LLM to deliver CVE intelligence and forensic analysis through a multi-source ingest pipeline drawing from NVD, CISA KEV, EPSS, MITRE ATT&CK, and CAPEC. Together, they bridge structured threat intelligence with automated forensic analysis and detection workflows.

97 MATHEMATICS AND COMPUTING↗

Progress towards accelerating HOMME on hybrid multi- core systems

The suitability of a spectral element based dynamical core (HOMME) within the Community Atmospheric Model (CAM) for GPU-based architectures is examined and initial performance results are reported. This work was done within a project to enable CAM to run at high resolution on next generation, multi-petaflop systems. The dynamical core is the present focus because it dominates the performance profile of our target problem. HOMME enjoys good scalability due to its underlying cubed-sphere mesh with full two-dimensional decomposition and the localization of all computational work within each element. The thread blocking and code changes that allow HOMME to effectively use GPUs are described along with a rewritten vertical remapping scheme which improves performance on both CPUs and GPUs. Validation of results in the full HOMME model is also described. Remaining issues affecting performance include optimizing the boundary exchanges for the case of multiple spectral elements being computed on the GPU and using multiple CUDA streams to overlap data transfers with computations.

99 GENERAL AND MISCELLANEOUS↗

Systems, methods, and devices for failure detection of one or more energy storage devices

An energy storage device management system can include a management portion for charging/discharging an energy storage device and an ultrasound interrogation portion for passing ultrasound energy through the energy storage device during charge/discharge cycles. A memory stores a stream of capture data instances derived from ultrasound energy exiting the energy storage device and baseline ultrasound data instances corresponding with the energy storage device during normal charging/discharging thereof. A processor can compare each capture data instance with the baseline ultrasound data and detect abnormal operating states of the energy storage device. A warning system can issue a notification when abnormal operating states are detected.

Kowalski, Jeffrey A.↗

ADCP Upsample

This R script takes reads the csv file of stream-wise velocity data and generates a new time series of with comparable but distinct noise using the Langevin method and library. It simulates a time series measured at 1/10 the sampling rate by calculating a moving average of 10, though other sampling rates can be calculated by adjusting the moving average window. A stationary signal is calculated by taking the difference of the original signal and the moving average, then normalizing by the standard deviation of the moving average. The stationary signal is used to calculate drift and diffusion coefficients. Those coefficients are used to generate a new stationary signal. The new stationary signal is un-normalized and added to the moving average (or new equivalent signal) to create a new up-sampled velocity time-series.

Green, Nathan↗

Data and scripts associated with a manuscript on a meta-analysis synthesizing stream biogeochemical response to wildfires across space and time (v2)

This data package is associated with the publication “Catchment characteristics modulate the influence of wildfires on nitrate and dissolved organic carbon in lotic systems across space and time: A meta-analysis” submitted to Global Biogeochemical Cycles (Cavaiani et al. 2025). This study uses meta-analytical techniques to evaluate the effect of wildfire on in-stream responses in burned and unburned watersheds. The study aims to provide additional insight into the range of responses and net influences that wildfires have on hydro-biogeochemistry across broad spatial scales, burn extents, and the persistence of water-quality change. This study compiles data and metadata from 18 total publications that includes 1) surface water geochemistry data (dissolved organic carbon; nitrate), 2) climate classifications, 3) year of the wildfire, 4) the time lag between when the fire occurred and when the sampling occurred, and 5) study design of the publication. In total, this meta-analysis draws data that spans 8 climate guilds, 3 biomes, 62 watersheds, and 20 unique wildfires. See Sites_meta_data.csv for citations of the papers used in this meta-analysis. All R scripts and the associated data can also be found on GitHub at This data package was originally published in March 2024. It was updated in April 2025 (v2; new and modified files). See the change history section in the readme for more details. This data package contains five primary folders that include the following: (1) inputs; (2) output for analysis; (3) initial plots; (4) R scripts; and (5) GIS data. The data package also contains a data dictionary (dd) that provides column header definitions and a file-level metadata (flmd) file that describes every file. The “inputs” folder contains a list of all publications identified during the formal web search and an indication of whether each publication was included in the final analysis. Additionally, it includes site-level metadata, catchment characteristics, and GIS data for all publications included in the final analysis. The “Output_for_analysis” folder contains all data frames and figures generated from each R script used for additional data analysis. The “initial_plots” folder includes all exploratory figures that will be included in a supplemental and figures that will be submitted with the manuscript for publication. The “R_scripts” folder contains the scripts that perform all the data manipulations, statistical analyses, and plots. The “gis_data” folder includes shape files for each fire included in this meta-analysis. This data package contains the following file types: csv, pdf, jpeg, cpg, dbf, prj, shp, shp.ea.iso.xml, shp.iso.xml, shx.

54 ENVIRONMENTAL SCIENCES↗

Iterative map-making with two-level preconditioning for polarized cosmic microwave background data sets: A worked example for ground-based experiments

Context. An estimation of the sky signal from streams of time ordered data (TOD) acquired by the cosmic microwave background (CMB) experiments is one of the most important steps in the context of CMB data analysis referred to as the map-making problem. The continuously growing CMB data sets render the CMB map-making problem progressively more challenging in terms of computational cost and memory in particular in the context of ground-based experiments with their operational limitations as well as the presence of contaminants. Aims. We study a recently proposed, novel class of the Preconditioned Conjugate Gradient (PCG) solvers which invoke two-level preconditioners in the context of the ground-based CMB experiments. We compare them against the PCG solvers commonly used in the map-making context considering their precision and time-to-solution. Methods. We compare these new methods on realistic, simulated data sets reflecting the characteristics of current and forthcoming CMB ground-based experiments. We develop a divide-and-conquer implementation of the approach where each processor performs a sequential map-making for a subset of the TOD. Results. We find that considering the map level residuals, the new class of solvers permits us to achieve a tolerance that is better than the standard approach by up to three orders of magnitude, where the residual level often saturates before convergence is reached. This often corresponds to an important improvement in the precision of the recovered power spectra in particular on the largest angular scales. The new method also typically requires fewer iterations to reach a required precision and therefore shorter run times are required for a single map-making solution. However, the construction of an appropriate two-level preconditioner can be as costly as a single standard map-making run. Nevertheless, if the same problem needs to be solved multiple times, for example, as in Monte Carlo simulations, this cost is incurred only once, and the method should be competitive, not only as far as its precision is concerned but also its performance.

79 ASTRONOMY AND ASTROPHYSICS↗

Near-memory data reorganization engine

A memory subsystem package is provided that has processing logic for data reorganization within the memory subsystem package. The processing logic is adapted to reorganize data stored within the memory subsystem package. In some embodiments, the memory subsystem package includes memory units, a memory interconnect, and a data reorganization engine ("DRE"). The data reorganization engine includes a stream interconnect and DRE units including a control processor and a load-store unit. The control processor is adapted to execute instructions to control a data reorganization. The load-store unit is adapted to process data move commands received from the control processor via the stream interconnect for loading data from a load memory address of a memory unit and storing data to a store memory address of a memory unit.

97 MATHEMATICS AND COMPUTING↗

National Energy Water Treatment & Speciation (NEWTS): A Water & Critical Mineral Database and Dashboard

The scarcity of water resources, the need for beneficial water reuse, and the challenges of wastewater treatment are becoming increasingly pressing in economic, social, and environmental domains. Addressing these concerns requires effective treatment strategies to manage wastewater streams and tackle environmental and economic issues. Furthermore, the recovery of critical minerals from the waste streams associated with energy production holds the promise of offsetting treatment costs and securing local sources of valuable minerals. However, relevant data on these waste streams are dispersed and challenging to locate. The process of ingesting such data into modeling software often involves multiple steps, requiring data restructuring to meet software-input requirements. The non-standardized reporting of water data makes data aggregation and reformatting a time-consuming process. Additionally, essential attributes necessary for modeling water treatment and mineral scale formation are frequently missing. Moreover, data gaps vary depending on the region of interest. Consequently, there is a pressing need for high-quality energy-water composition data that can be easily imported into water chemistry modeling software. To address this need, the National Energy Technology Laboratory has created the National Energy Water Treatment and Speciation (NEWTS) Database and Dashboard—a free online tool catering to community leaders and water researchers. NEWTS facilitates a comprehensive understanding of the composition of energy-related wastewater streams in the United States. The datasets provide detailed concentrations and speciation of major and minor aqueous compounds in energy-related wastewater streams, including power plant leachate, acid mine drainage, brackish water, and oil and gas produced water across the United States. Many of the aqueous species are critical minerals (Li, REEs) in high demand to modernize the world’s energy infrastructure. Many of the datasets also contain volumetric flow-rates needed to model the treatment and reuse scenarios in advanced aqueous chemistry software programs. The NEWTS Database and Dashboard offer public access to hitherto challenging-to-access datasets, presented in a standardized format that is tailored for easy input into aqueous chemistry modeling software. By performing the work needed to transform dispersed, disparate data sources into unified, model-ready datasets, NEWTS serves as an essential resource in advancing water treatment research and sustainable water resource management.

produced water management↗

Developing Fluorescence-Based Sensors to Support Rare Earth Element Separation

Rare earth elements (REEs) are essential to most renewable energy technologies. Unfortunately, as we transition to sustainable energy production, the demand for REEs is rapidly growing well beyond current rates of production. As a result, novel means of efficient, scalable, and easily adaptable methods for processing primary and recycle feedstocks are needed. Development and integration of sensors for highly selective in-line monitoring can support more efficient design and testing of such novel separation processes, as well as more cost-effective deployment of those separation flowsheets. Work here will explore the application of fluorescence spectroscopy, a highly sensitive and selective technique, to quantify multiple lanthanides in complex mixtures including known interferents or quenching agents. Results include identification of the optimal excitation wavelength and the limit of detection of various rare earth elements as well as the performance of data-science-based quantification approaches in streams where “unknowns” are present. Overall, the data science tools in conjunction with optical sensor data were able to quantify analytes in the presence of other lanthanides which can be anticipated in the actual industrial stream. Here we include characterization of lanthanides in a microfluidic device similar to those used in new process development. This study demonstrates the capability of utilizing fluorescence spectroscopy to quantify analytes in a complicated solution matrix, suggesting this is a successful approach for in-line monitoring to optimize the separation efficiency in an industrial stream.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Berkeley Nuclear Data Cloud (BDC) v2.3

The software was developed to serve and curate arbitrarily large datasets comprising data acquired from various mobile platforms. The software is contained in a number of server and client libraries. The former manage the ingestion, indexing, querying, and serving of the data. The latter libraries are distributed for Linux, Mac OSX, and Windows and enable users to interact with data downloaded from the service either in the form of an HDF5 file or streamed in a BSON data chunk. Using the Berkeley Data Cloud, researchers from varying fields can collaborate, compare results and curate both their raw data and the derived products of their analysis.

Cholia, Shreyas↗

Analysis of DWPF Sludge Batch 6 (Macrobatch 7): Pour Stream Glass Samples

The Defense Waste Processing Facility (DWPF) began processing Sludge Batch 6 (SB6), also referred to as Macrobatch 7 (MB7), in June 2010. SB6 is a blend of the heel of Tank 40 from Sludge Batch 5 (SB5), H-Canyon Np transfers and SB6 that was transferred to Tank 40 from Tank 51.1 SB6 was processed using Frit 418. Sludge is received into the DWPF Chemical Processing Cell (CPC) and is processed through the Sludge Receipt and Adjustment Tank (SRAT) and Slurry Mix Evaporator Tank (SME). The treated sludge slurry is then transferred to the Melter Feed Tank (MFT) and fed to the melter. During processing of each sludge batch, the DWPF is required to take at least one glass sample to meet the objectives of the Glass Product Control Program (GPCP) and to complete the necessary Production Records so that the final glass product may be disposed of at a Federal Repository. The DWPF requested various analyses of radioactive glass samples obtained from the melter pour stream during processing of SB6 as well as reduction/oxidation (REDOX) analysis of MFT samples to determine the impact of Argon bubbling. Sample analysis followed the Task Technical and Quality Assurance Plan (TTQAP) and an Analytical Study Plan (ASP). Four Pour Stream (PS) glass samples and two MFT slurry samples were delivered to the Savannah River National Laboratory (SRNL) from the DWPF. Table 1-1 lists the sample information for each pour stream glass sample. SB6 PS3 (S03472) was selected as the official pour stream sample for SB6 and full analysis was requested. This report details the visual observations of the as-received SB6 PS No.3 glass sample as well as results for the chemical composition, Product Consistency Test (PCT), radionuclide content, noble metals, and glass density. REDOX results will be provided for all four pour stream samples and vitrified samples of MFT-558 and MFT-568A. Where appropriate, data from other pour stream samples will be provided.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Towards understanding HPC users and systems: A NERSC case study

High performance computing (HPC) scheduling landscape currently faces new challenges due to thechanges in the workload. Previously, HPC centers were dominated by tightly coupled MPI jobs. HPCworkloads increasingly include high-throughput, data-intensive, and stream-processing applications. Asa consequence, workloads are becoming more diverse at both application and job levels, posing newchallenges to classical HPC schedulers. There is a need to understand the current HPC workloads andtheir evolution to facilitate informed future scheduling research and enable efficient scheduling in futureHPC systems.In this paper, we present a methodology to characterize workloads and assess their heterogeneity,at a particular time period and its evolution over time. We apply this methodology to the workloads ofthree systems (Hopper, Edison, and Carver) at the National Energy Research Scientific Computing Center(NERSC). We present the resulting characterization of jobs, queues, heterogeneity, and performance thatincludes detailed information of a year of workload (2014) and evolution through the systems’ lifetime(2010–2014).

Rodrigo, Gonzalo P.↗

From the Fire: A Deeper Look at the Phoenix Stream

We use 6 yr of data from the Dark Energy Survey to perform a detailed photometric characterization of the Phoenix stellar stream, a 15° long, thin, dynamically cold, low-metallicity stellar system in the Southern Hemisphere. We use natural splines, a nonparametric modeling technique, to simultaneously fit the stream track, width, and linear density. This updated stream model allows us to improve measurements of the heliocentric distance (17.4 ± 0.1 (stat.) ±0.8 (sys.) kpc) and distance gradient (−0.009 ± 0.006 kpc deg$^{−1}$) of Phoenix, which corresponds to a small change of 0.13 ± 0.09 kpc in heliocentric distance along the length of the stream. We measure linear intensity variations on degree scales, as well as deviations in the stream track on ∼2° scales, suggesting that the stream may have been disturbed during its formation and/or evolution. We recover three peaks and one gap in linear intensity along with fluctuations in the stream track. Compared to other thin streams, the Phoenix stream shows more fluctuations and, consequently, the study of Phoenix offers a unique perspective on gravitational perturbations of stellar streams. We discuss possible sources of perturbations to Phoenix, including baryonic structures in the Galaxy and dark matter subhalos.

79 ASTRONOMY AND ASTROPHYSICS↗

Paths to Sustainable Distributed Generation through 2050: Matching Local Waste Biomass Resources with Grid, Industrial, and Community Needs

This project identifies and evaluates current and future organic waste fuel types and matches these wastes with local grid, industrial, building power, and thermal energy needs. The team developed data on organic waste streams and thermal energy needs, and then constructed a techno-economic analysis model that can process the data and compute viable production of electricity, waste heat, biogas, and/or biomethane for a range of energy price points. Scenarios generated by the model include allocation of local organic waste streams to existing or new conversion infrastructure and are evaluated in a life-cycle assessment model to determine local and global environmental impacts. The project’s webbased tool provides rapid evaluation of potential waste-to-energy sites, including retrofits/expansions of existing facilities or entirely new construction, on the basis of resource availability, potential for waste heat use, and economic and environmental metrics.

09 BIOMASS FUELS↗

Timely Reporting of Heavy Hitters Using External Memory

Given an input stream S of size N, a Φ-heavy hitter is an item that occurs at least ΦN times in S. The problem of finding heavy-hitters is extensively studied in the database literature. In this work, we study a real-time heavy-hitters variant in which an element must be reported shortly after we see its T = Φ N-th occurrence (and hence it becomes a heavy hitter). We call this the Timely Event Detection (TED) Problem. The TED problem models the needs of many real-world monitoring systems, which demand accurate (i.e., no false negatives) and timely reporting of all events from large, high-speed streams with a low reporting threshold (high sensitivity). Like the classic heavy-hitters problem, solving the TED problem without false-positives requires large space (Ω (N) words). Thus in-RAM heavy-hitters algorithms typically sacrifice accuracy (i.e., allow false positives), sensitivity, or timeliness (i.e., use multiple passes). We show how to adapt heavy-hitters algorithms to external memory to solve the TED problem on large high-speed streams while guaranteeing accuracy, sensitivity, and timeliness. Our data structures are limited only by I/O-bandwidth (not latency) and support a tunable tradeoff between reporting delay and I/O overhead. With a small bounded reporting delay, our algorithms incur only a logarithmic I/O overhead. We implement and validate our data structures empirically using the Firehose streaming benchmark. Multi-threaded versions of our structures can scale to process 11M observations per second before becoming CPU bound. In comparison, a naive adaptation of the standard heavy-hitters algorithm to external memory would be limited by the storage device’s random I/O throughput, i.e., ≈100K observations per second.

97 MATHEMATICS AND COMPUTING↗

Revisit NGC 5466 tidal stream with Gaia , SDSS/SEGUE, and LAMOST

ABSTRACT By mining the data from Gaia Early Data Release 3, Sloan Digital Sky Survey/Sloan Extension for Galactic Understanding and Exploration Data Release 16, and Large Sky Area Multi-Object Fiber Spectroscopic Telescope Data Release 8, 11 member stars of the NGC 5466 tidal stream are detected and 7 of them are newly identified. To reject contaminators, a variety of cuts are applied in sky position, colour–magnitude diagram, metallicity, proper motion, and radial velocity. We compare our data to a mock stream generated by modelling the cluster’s disruption under a smooth Galactic potential plus the Large Magellanic Cloud (LMC). The concordant trends in phase space between the model and observations imply that the stream might have been perturbed by the LMC. The two most distant stars among the 11 detected members trace the stream’s length to 60° of sky, supporting and extending the previous length of 45°. Given that NGC 5466 is so distant and potentially has a longer tail than previously thought, we expect that the NGC 5466 tidal stream could be a useful tool in constraining the Milky Way gravitational field.

79 ASTRONOMY AND ASTROPHYSICS↗