Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 883 records · Page 49

A stream classification system for the conterminous United States

Stream classifications are important for understanding stream ecosystem diversity while also serving as tools for aquatic conservation and management. With current rates of land and riverscape modification within the United States (US), a comprehensive inventory and evaluation of naturally occurring stream habitats is needed, as this provides a physical template upon which stream biodiversity is organized and maintained. To adequately represent the heterogeneity of stream ecosystems, such a classification needs to be spatially extensive where multiple stream habitat components are represented at the highest resolution possible. Herein, we present a multi-layered empirically-driven stream classification system for the conterminous US, constructed from over 2.6 million stream reaches within the NHDPlus V2 stream network. The classification is based on emergent natural variation in six habitat layers meaningful at the stream-reach resolution: size, gradient, hydrology, temperature, network bifurcation, and valley confinement. To support flexibility of use, we provide multiple alternative approaches to developing classes and report uncertainty in classes assigned to stream reaches. The stream classification and underlying data provide valuable resources for stream conservation and research.

54 ENVIRONMENTAL SCIENCES↗

Early Fault Detection in Particle Accelerator Power Electronics Using Ensemble Learning

Early fault detection and fault prognosis are crucial to ensure efficient and safe operations of complex engineering systems such as the Spallation Neutron Source (SNS) and its power electronics (high voltage converter modulators). Following an advanced experimental facility setup that mimics SNS operating conditions, the authors successfully conducted 21 early fault detection experiments, where fault precursors are introduced in the system to a degree enough to cause degradation in the waveform signals, but not enough to reach a real fault. Nine different machine learning techniques based on ensemble trees, convolutional neural networks, support vector machines, and hierarchical voting ensembles are proposed to detect the fault precursors. Although all 9 models have shown a perfect and identical performance during the training and testing phase, the performance of most models has decreased in the next test phase once they got exposed to realworld data from the 21 experiments. The hierarchical voting ensemble, which features multiple layers of diverse models, maintains a distinguished performance in early detection of the fault precursors with 95% success rate (20/21 tests), followed by adaboost and extremely randomized trees with 52% and 48% success rates, respectively. The support vector machine models were the worst with only 24% success rate (5/21 tests). The study concluded that a successful implementation of machine learning in the SNS or particle accelerator power systems would require a major upgrade in the controller and the data acquisition system to facilitate streaming and handling big data for the machine learning models. In addition, this study shows that the best performing models were diverse and based on the ensemble concept to reduce the bias and hyperparameter sensitivity of individual models.

43 PARTICLE ACCELERATORS↗

WHONDRS River Corridor Sediment and Water Geochemistry and In Situ Sensor Data from 7 Perennial and 7 Intermittent Streams across San Antonio, Texas (v3)

This dataset supports a broader study examining the effects of intermittency on sediment respiration. The dataset provides sediment and surface water geochemistry and in situ sensor data from 7 perennial and 7 intermittent streams in San Antonio, Texas. Each stream/site was visited both in summer during base flow (July-September 2023) and winter during peak flow (January-February 2024). Related data were collected and will be published separately in collaboration with A. Veach. The data package was originally published in April 2025. It was updated in June 2025 (v2; modified and new files) and September 2025 (v3; modified files). See the change history section in the readme for more details. For details on how to navigate data packages generated by this project, see https://data.ess-dive.lbl.gov/portals/PNNLRiverCorridorSFA/About. This dataset is comprised of two folders of field photos and videos, one folder of raw Fourier transform ion cyclotron resonance mass spectrometry (FTICR-MS) data and one main data folder containing (1) file-level metadata; (2) data dictionary; (3) field metadata; (4) readme; (5) international generic sample number (IGSN) mapping file; (6) field protocol; (7) a subfolder with sample data; and (8) a subfolder with sensor data. The sample data subfolder contains (1) surface water and sediment dissolved organic carbon (DOC, measured as non-purgeable organic carbon, NPOC) data and averages; (2) surface water and sediment total nitrogen data and averages; (3) sediment grain size data; (4) sediment iron (II) data and averages; (5) wet sediment mass, dry sediment mass, water mass, and wet sediment volume in incubation and sediment ICR vials; (7) sediment incubation respiration rate data and averages; (8) normalized respiration rate data and averages; (9) methods codes; (10) sediment percent carbon and nitrogen; (11) sediment X-ray diffraction (XRD) data; (12) gravimetric moisture and averages; (13) a subfolder with sediment incubation respiration data, scripts, and plots; (14) surface water and sediment FTICR methods; and (15) a subfolder of 9.4 Tesla (9.4T) FTICR-MS data. This folder contains five subfolders, one containing the sediment .xml data files, one containing the water .xml files, one containing the sediment CoreMS output files, one containing the water CoreMS output files, and the other containing instructions and scripts for processing the files in CoreMS (https://github.com/EMSL-Computing/CoreMS). The sensor data subfolder contains (1) a subfolder with miniDOT dissolved oxygen and temperature data and plots; (2) miniDOT dissolved oxygen and temperature summary data; and (3) miniDOT installation methods. All files are .csv, .pdf, .R, .xml, .d, .html, .Rmd, .py, .cal, .json, .jpg, .jpeg, .png, .mov, or .mp4. CORRECTION: The data processing methods for FTICR described in “v3_WHONDRS_AV1_Methods_Codes.csv” mistakenly indicate that users should process the data in Formultitude. The corrected description should read: “Both unprocessed and processed data are provided to allow users flexibility in data processing. Instructions and scripts for processing the data using CoreMS are included.” CORRECTION: Carbon and nitrogen content are reported as percentages. The current column headers "01395_C_percent_per_mg" and "01397_N_percent_per_mg" are incorrect. These should read "01395_C_percent" and "01397_N_percent" and will be corrected in the next version of this data package.

54 ENVIRONMENTAL SCIENCES↗

Ultrafast structural response to charge redistribution within a photosinthetic reaction centre

This data set contains stream files, difference maps and SVD maps obtained from data processing of time-resolved serial femtosecond crystallography measurements performed at LCLS XFEL, used to construct the images and perform the analysis published in the paper "Ultrafast structural response to charge redistribution within a photosinthetic reaction centre". Two (2) time points plus a dark state from 2015 ('a' label) as well as five (5) time points plus a dark state from 2016 ('b' label) are provided.

CXI↗

FPGA-based compression of streaming x-ray photon correlation spectroscopy data

A data acquisition system to perform real-time background subtraction and lower-level-discrimination-based compression of streaming x-ray photon correlation spectroscopy (XPCS) data from a fast charge-coupled device (CCD) area detector has been built and put into service at the Advanced Photon source (APS) at Argonne National Laboratory. A commercial frame grabber with on-board field-programmable gate array (FPGA) was used in the design, and continuously processes 60 frames per second each consisting of 1,024 x 1,024 pixels with up to 64512 photon hits per frame.

43 PARTICLE ACCELERATORS↗

Once in a blue stream: Detection of recent star formation in the NGC 7241 stellar stream with MEGARA

Aims.In this work we study the striking case of a narrow blue stream with a possible globular cluster-like progenitor around the NGC 7241 galaxy and its foreground dwarf companion. We want to figure out if the stream was generated by tidal interaction with NGC 7241 or if it first interacted with the foreground dwarf companion and later both fell together toward NGC 7241. Methods.We used four sets of observations, including a follow-up spectroscopic study of this stream based on data taken with the MEGARA instrument at the 10.4 m Gran Telescopio Canarias using the integral field spectroscopy mode, the Mount Lemmon 0.80 m telescope, the Telescopio NazionaleGalileo, the DESI Imaging Legacy surveys, and GALEX archival data. We also used high-resolution zoomed-in cosmological simulations. Results.Our data suggest that the compact object we detected in the stream is a foreground Milky Way halo star. Near this compact object we detect emission lines overlapping a less compact, bluer, and fainter blob of the stream that is clearly visible in both ultraviolet and optical deep images. From its heliocentric systemic radial velocity derived from the [O III]λ5007 Å lines (V syst = 1548.58 ± 1.80 km s −1 ) and new UV and optical broadband photometry, we conclude that this overdensity could be the actual core of the stream, with an absolute magnitude ofM g ∼ −10 and ag − r = 0.08 ± 0.11, consistent with a remnant of a low-mass dwarf satellite undergoing a current episode of star formation. From the width of the stream and assuming a circular orbit, we calculate that the progenitor mass can be typical of a dwarf galaxy, but it could also be substantially lower if the stream is on a very radial orbit or if it was created by tidal interaction with the companion dwarf instead of with NGC 7241. These estimates also suggest that this is one of the lowest mass streams detected to date beyond the Local Group. Finally, we find that blue stellar streams containing star formation regions are commonly predicted by high-resolution cosmological simulations of galaxies lighter than the Milky Way. This scenario is consistent with the processes explaining the bursty star formation history of some dwarf satellites, which are followed by a gas depletion and a fast quenching once they enter within the virial radius of their host galaxies for the first time. Thus, it is likely that the stream’s progenitor is undergoing a star formation burst comparable to those that have shaped the star formation history of several Local Group dwarfs in the last few gigayears.

Astronomy & Astrophysics↗

Vis-SAGA: Visual Analytics for Situational Awareness of Grid Anomalies: Preprint

We describe supporting near real-time situational awareness of the electric distribution system by visualizing novel data from voltage sensors deployed on existing broadband cable television network equipment. Our scalable web-based visual analytics platform supports interactive geospatial exploration, time-series analysis, and summarization of grid behavior during potentially anomalous events. The broadband cable television sensor network provides observability of the electrical distribution system at a higher local spatial resolution than is typically available to most utilities, revealing the operational state of the network and aiding in the detection of abnormal behaviors or deviations from expected patterns, particularly across electric utility service areas. We outline the design and development of interactive geospatial and time-series visualization components and the scalable data services that supply metadata, historical, and real-time streams of sensor data across the network. We evaluate our platform during periods of extreme weather, demonstrating its ability to assist in detecting patterns of operation that affect power availability, quality, resiliency, and service restoration.

cable television↗

PlanetSense: A Real-time Streaming and Spatio-temporal Analytics Platform for Gathering Geo-spatial Intelligence from Open Source Data

Geospatial intelligence has traditionally relied on the use of archived and unvarying data for planning and exploration purposes. In consequence, the tools and methods that are architected to provide insight and generate projections only rely on such datasets. Albeit, if this approach has proven effective in several cases, such as land use identification and route mapping, it has severely restricted the ability of researchers to inculcate current information in their work. This approach is inadequate in scenarios requiring real-time information to act and to adjust in ever changing dynamic environments, such as evacuation and rescue missions. In this work, we propose PlanetSense, a platform for geospatial intelligence that is built to harness the existing power of archived data and add to that, the dynamics of real-time streams, seamlessly integrated with sophisticated data mining algorithms and analytics tools for generating operational intelligence on the fly. The platform has four main components – i) GeoData Cloud – a data architecture for storing and managing disparate datasets; ii) Mechanism to harvest real-time streaming data; iii) Data analytics framework; iv) Presentation and visualization through web interface and RESTful services. Using two case studies, we underpin the necessity of our platform in modeling ambient population and building occupancy at scale.

big data architecture↗

The nature and orbit of the Ophiuchus stream

The Ophiuchus stream is a recently discovered stellar tidal stream in the Milky Way. We present high-quality spectroscopic data for 14 stream member stars obtained using the Keck and MMT telescopes. We confirm the stream as a fast moving (v{sub los} ∼ 290 km s{sup −1}), kinematically cold group (σ{sub v{sub l{sub o{sub s}}}}≲1 km s{sup −1}) of α-enhanced and metal-poor stars ([α/Fe] ∼ 0.4 dex, [Fe/H] ∼ −2.0 dex). Using a probabilistic technique, we model the stream simultaneously in line-of-sight velocity, color–magnitude, coordinate, and proper motion space, and so determine its distribution in 6D phase-space. We find that the stream extends in distance from 7.5 to 9 kpc from the Sun; it is 50 times longer than wide, merely appearing highly foreshortened in projection. The analysis of the stellar population contained in the stream suggests that it is ∼12 Gyr old, and that its initial stellar mass was ∼2 × 10{sup 4} M{sub ⊙} (or at least ≳7 × 10{sup 3} M{sub ⊙}). Assuming a fiducial Milky Way potential, we fit an orbit to the stream that matches the observed phase-space distribution, except for some tension in the proper motions: the stream has an orbital period of ∼350 Myr, and is on a fairly eccentric orbit (e ∼ 0.66) with a pericenter of ∼3.5 kpc and an apocenter of ∼17 kpc. The phase-space structure and stellar population of the stream show that its progenitor must have been a globular cluster that was disrupted only ∼240 Myr ago. We do not detect any significant overdensity of stars along the stream that would indicate the presence of a progenitor, and conclude that the stream is all that is left of the progenitor.

79 ASTRONOMY AND ASTROPHYSICS↗

F3D Image Processing and Analysis for Many - and Multi-core Platforms

F3D is written in OpenCL, so it achieve[sic] platform-portable parallelism on modern mutli-core CPUs and many-core GPUs. The interface and mechanims to access F3D core are written in Java as a plugin for Fiji/ImageJ to deliver several key image-processing algorithms necessary to remove artifacts from micro-tomography data. The algorithms consist of data parallel aware filters that can efficiently utilizes[sic] resources and can work on out of core datasets and scale efficiently across multiple accelerators. Optimizing for data parallel filters, streaming out of core datasets, and efficient resource and memory and data managements over complex execution sequence of filters greatly expedites any scientific workflow with image processing requirements. F3D performs several different types of 3D image processing operations, such as non-linear filtering using bilateral filtering and/or median filtering and/or morphological operators (MM). F3D gray-level MM operators are one-pass constant time methods that can perform morphological transformations with a line-structuring element oriented in discrete directions. Additionally, MM operators can be applied to gray-scale images, and consist of two parts: (a) a reference shape or structuring element, which is translated over the image, and (b) a mechanism, or operation, that defines the comparisons to be performed between the image and the structuring element. This tool provides a critical component within many complex pipelines such as those for performing automated segmentation of image stacks. F3D is also called a "descendent" of Quant-CT, another software we developed in the past. These two modules are to be integrated in a next version. Further details were reported in: D.M. Ushizima, T. Perciano, H. Krishnan, B. Loring, H. Bale, D. Parkinson, and J. Sethian. Structure recognition from high-resolution images of ceramic composites. IEEE International Conference on Big Data, October 2014.

Source record↗

F3D Image Processing and Analysis for Many - and Multi-core Platforms

F3D is written in OpenCL, so it achieve[sic] platform-portable parallelism on modern mutli-core CPUs and many-core GPUs. The interface and mechanims to access F3D core are written in Java as a plugin for Fiji/ImageJ to deliver several key image-processing algorithms necessary to remove artifacts from micro-tomography data. The algorithms consist of data parallel aware filters that can efficiently utilizes[sic] resources and can work on out of core datasets and scale efficiently across multiple accelerators. Optimizing for data parallel filters, streaming out of core datasets, and efficient resource and memory and data managements over complex execution sequence of filters greatly expedites any scientific workflow with image processing requirements. F3D performs several different types of 3D image processing operations, such as non-linear filtering using bilateral filtering and/or median filtering and/or morphological operators (MM). F3D gray-level MM operators are one-pass constant time methods that can perform morphological transformations with a line-structuring element oriented in discrete directions. Additionally, MM operators can be applied to gray-scale images, and consist of two parts: (a) a reference shape or structuring element, which is translated over the image, and (b) a mechanism, or operation, that defines the comparisons to be performed between the image and the structuring element. This tool provides a critical component within many complex pipelines such as those for performing automated segmentation of image stacks. F3D is also called a "descendent" of Quant-CT, another software we developed in the past. These two modules are to be integrated in a next version. Further details were reported in: D.M. Ushizima, T. Perciano, H. Krishnan, B. Loring, H. Bale, D. Parkinson, and J. Sethian. Structure recognition from high-resolution images of ceramic composites. IEEE International Conference on Big Data, October 2014.

Source record↗

HPC ODA Commons [SWR-26-003]

HPC ODA Commons is a community-driven platform for standardizing HPC operational data analytics. HPC sites generate enormous volumes of operational data - scheduler logs, accounting records, monitoring streams - but turning that data into actionable insight is needlessly hard. Each site builds bespoke parsers, schemas, and evaluation pipelines. Results can't be compared across institutions. Promising analytics ideas stay siloed because there's no shared language for describing the data, the experiments, or the outcomes. HPC ODA Commons fixes this by establishing community-governed contracts - versioned schemas, canonical artifacts, and benchmark recipes - that make ODA workflows discoverable, reproducible, and comparable. It pairs these standards with a practical, CLI-first toolkit that lets operators and researchers go from raw logs to standardized results without sending data off-cluster.

Menear, Kevin [National Laboratory of the Rockies ↗

Real-time forecasting of electricity demand in a streams-based architecture with applications

A streams platform is used. Multiple streams of electricity usage data are received, each from an electrical meter providing periodic updates to electrical usage for devices connected to the electrical meter. Weather information is received corresponding to locations where the electrical meters are. Real-time predictive modeling of electricity demand is performed based on the received multiple streams of electricity usage data and the received weather information, at least by performing: updating a state space model for electrical load curves using the usage data from the streams and the weather, wherein the updating uses current load observations for the multiple streams for a current time period; and creating forecast(s) for the electricity demand. The forecast(s) of the electricity demand are output. Appliance-level predictions may be made and used, and substitution effects and load management functions may be performed.

Ghosh, Soumyadip↗

The Pristine survey – XVI. The metallicity of 26 stellar streams around the Milky Way detected with the STREAMFINDER in Gaia EDR3

ABSTRACT We use the photometric metallicities provided by the panoramic Pristine survey to study the veracity and derive the metallicities of the numerous stellar streams found by the application of the STREAMFINDER algorithm to the Gaia Early Data Release 3 data. All 26 streams present in Pristine show a clear metallicity distribution function, which provides an independent check of the reality of these structures, supporting the reliability of STREAMFINDER in finding streams and the power of Pristine to measure precise metallicities. We further present six candidate structures with coherent phase-space and metallicity signals that are very likely streams. The majority of studied streams are very metal-poor (14 structures with [Fe/H] < −2.0) and include three systems with [Fe/H] < −2.9 (C-11, C-19, and C-20). These streams could be the closest debris of low-luminosity dwarf galaxies or may have originated from globular clusters of significantly lower metallicity than any known current Milky Way globular cluster. Our study shows that the promise of the Gaia data for Galactic Archeology studies can be substantially strengthened by quality photometric metallicities, allowing us to peer back into the earliest epochs of the formation of our Galaxy and its stellar halo constituents.

79 ASTRONOMY AND ASTROPHYSICS↗