Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 667 records · Page 37
All sourcesNASA NTRSDOE OSTI
DOE OSTI2015

Security vulnerabilities of connected vehicle streams and their impact on cooperative driving

Autonomous vehicles capable of navigating unpredictable real-world environments with little human feedback are a reality today. Such systems rely heavily on onboard sensors such as cameras, radar/LIDAR, and GPS as well as capabilities such as 3G/4G connectivity and V2V/V2I communication to make real-time maneuvering decisions. Autonomous vehicle control imposes very strict requirements on the security of the communication channels used by the vehicle to exchange information as well as the control logic that performs complex driving tasks such as adapting vehicle velocity or changing lanes. This research presents a first look at the effects of security attacks on the communication channel as well as sensor tampering of a connected vehicle stream equipped to achieve CACC. Our simulation results show that an insider attack can cause significant instability in the CACC vehicle stream. We also outline how different countermeasures, such as downgrading to ACC mode, could potentially be used to improve the security and safety of the connected vehicle streams.

42 ENGINEERING↗
DOE OSTI2021

Leveraging In-Network Computing and Programmable Switches for Streaming Analysis of Scientific Data

With the emergence of programmable network devices that match the performance of fixed function devices, several recent projects have explored in-network computing, where the processing that is traditionally done outside the network is offloaded to the network devices. In-network computing has typically been applied to network functions (e.g., load balancing, NAT, and DNS), caching, data reduction/aggregation, and coordination/consensus functions. In some cases it has been used to accelerate stream-processing tasks that involve small payloads and simple operations. In this work we focus on leveraging in-network computing for stream processing of scientific datasets with large payloads that require complex operations such as floating-point computations and logarithmic functions. We demonstrate in-network computing for a real-world scientific application performing streaming normalization of a 2-D image from a light source experiment. We discuss the challenges we encountered and potential approaches to address them.

Sankaran, Ganesh↗
DOE OSTI2023

ERSAP: Toward Better NP Data-Stream Analytics With Flow-Based Programming

This paper presents an reactive, actor-model and FBP paradigm based framework that we develop to design data-stream processing applications for HEP and NP. This framework encourages a functional decomposition of the overall data processing application into small mono-functional artifacts. Artifacts that are easy to understand, develop, deploy and debug. The fact that these artifacts (actors) are programmatically independent they can be scaled and optimized independently, which is impossible to do for components of the monolithic application. One of the important advantages of this approach is fault tolerance where independent actors can come and go on the data-stream without forcing the entire application to crash. Furthermore, it also makes it is easy to locate the faulty actor in the data pipeline. Due the fact that the actors are loosely coupled, and that the data (inevitably) carries the context, they can run on heterogeneous environments, utilizing different accelerators. This paper describes the main design concepts of the framework and presents a ?proof of concept? application design and deployment results obtain processing on-beam calorimeter streaming data.

Gyurjyan, V.↗
DOE OSTI2017

Global synthesis of the temperature sensitivity of leaf litter breakdown in streams and rivers

Streams and rivers are important conduits of terrestrially derived carbon (C) to atmospheric and marine reservoirs. Leaf litter breakdown rates are expected to increase as water temperatures rise in response to climate change. The magnitude of increase in breakdown rates is uncertain, given differences in litter quality and microbial and detritivore community responses to temperature, factors that can influence the apparent temperature sensitivity of breakdown and the relative proportion of C lost to the atmosphere vs. stored or transported downstream. We synthesized 1025 records of litter breakdown in streams and rivers to quantify its temperature sensitivity, as measured by the activation energy (Ea, in eV). Temperature sensitivity of litter breakdown varied among twelve plant genera for which Ea could be calculated. Higher values of Ea were correlated with lower-quality litter, but these correlations were influenced by a single, N-fixing genus (Alnus). Ea values converged when genera were classified into three breakdown rate categories, potentially due to continual water availability in streams and rivers modulating the influence of leaf chemistry on breakdown. Across all data representing 85 plant genera, the Ea was 0.34 ± 0.04 eV, or approximately half the value (0.65 eV) predicted by metabolic theory. Our results indicate that average breakdown rates may increase by 5–21% with a 1–4 °C rise in water temperature, rather than a 10–45% increase expected, according to metabolic theory. Differential warming of tropical and temperate biomes could result in a similar proportional increase in breakdown rates, despite variation in Ea values for these regions (0.75 ± 0.13 eV and 0.27 ± 0.05 eV, respectively). The relative proportions of gaseous C loss and organic matter transport downstream should not change with rising temperature given that Ea values for breakdown mediated by microbes alone and microbes plus detritivores were similar at the global scale.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Salinization and sedimentation drive contrasting assembly mechanisms of planktonic and sediment‐bound bacterial communities in agricultural streams

Abstract Agriculture is the most dominant land use globally and is projected to increase in the future to support a growing human population but also threatens ecosystem structure and services. Bacteria mediate numerous biogeochemical pathways within ecosystems. Therefore, identifying linkages between stressors associated with agricultural land use and responses of bacterial diversity is an important step in understanding and improving resource management. Here, we use the Mississippi Alluvial Plain (MAP) ecoregion, a highly modified agroecosystem, as a case study to better understand agriculturally associated drivers of stream bacterial diversity and assembly mechanisms. In the MAP, we found that planktonic bacterial communities were strongly influenced by salinity. Tolerant taxa increased with increasing ion concentrations, likely driving homogenous selection which accounted for ~90% of assembly processes. Sediment bacterial phylogenetic diversity increased with increasing agricultural land use and was influenced by sediment particle size, with assembly mechanisms shifting from homogenous to variable selection as differences in median particle size increased. Within individual streams, sediment heterogeneity was correlated with bacterial diversity and a subsidy‐stress relationship along the particle size gradient was observed. Planktonic and sediment communities within the same stream also diverged as sediment particle size decreased. Nutrients including carbon, nitrogen, and phosphorus, which tend to be elevated in agroecosystems, were also associated with detectable shifts in bacterial community structure. Collectively, our results establish that two understudied variables, salinity and sediment texture, are the primary drivers of bacterial diversity within the studied agroecosystem, whereas nutrients are secondary drivers. Although numerous macrobiological communities respond negatively, we observed increasing bacterial diversity in response to agricultural stressors including salinization and sedimentation. Elevated taxonomic and phylogenetic bacterial diversity likely increases the probability of detecting community responses to stressors. Thus, bacteria community responses may be more reliable for establishing water quality goals within highly modified agroecosystems that have experienced shifting baselines.

DeVilbiss, Stephen E.↗
DOE OSTI2016

Spatial and successional dynamics of microbial biofilm communities in a grassland stream ecosystem

Biofilms represent a metabolically active and structurally complex component of freshwater ecosystems. Ephemeral prairie streams are hydrologically harsh and prone to frequent perturbation. Elucidating both functional and structural community changes over time within prairie streams provides a general understanding of microbial responses to environmental disturbance. In this study, we examined microbial succession of biofilm communities at three sites in a third-order stream at Konza Prairie over a 2- to 64-day period. Microbial abundance (bacterial abundance, chlorophyll a concentrations) increased and never plateaued during the experiment. Net primary productivity (net balance of oxygen consumption and production) of the developing biofilms did not differ statistically from zero until 64 days suggesting a balance of the use of autochthonous and allochthonous energy sources until late succession. Bacterial communities (MiSeq analyses of the V4 region of 16S rRNA) established quickly. Bacterial richness, diversity and evenness were high after 2 days and increased over time. Several dominant bacterial phyla (Beta-, Alphaproteobacteria, Bacteroidetes, Gemmatimonadetes, Acidobacteria, Chloroflexi) and genera ( Luteolibacter, Flavobacterium, Gemmatimonas, Hydrogenophaga ) differed in relative abundance over space and time. Bacterial community composition differed across both space and successional time. Pairwise comparisons of phylogenetic turnover in bacterial community composition indicated that early-stage succession (≤16 days) was driven by stochastic processes, whereas later stages were driven by deterministic selection regardless of site. Finally, our data suggest that microbial biofilms predictably develop both functionally and structurally indicating distinct successional trajectories of bacterial communities in this ecosystem.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Empirical Study of Focus-Plus-Context and Aggregation Techniques for the Visualization of Streaming Data

Analysis of streaming data often involves both real-time monitoring of incoming data as well as contextual awareness of data history. A focus-plus-context approach can support both goals, with variable levels of visual aggregation making it possible to provide a high level of detail for incoming and recent data while providing contextual information about recent history. Visual aggregation reduces data resolution in order to show the context of data over large periods of time within a limited display space. With a controlled experiment, we evaluated the effectiveness of different types of aggregation for four types of stream-analysis tasks. Overall, the results show that a focus-plus-context design has little negative impact on the ability to successfully monitor and analyze streaming data, making it possible to show longer periods of time than other approaches. However, visual aggregation can be problematic for trend recognition tasks. This research demonstrates how the effectiveness of the visualization depends on the specifics of the analysis task.

Ragan, Eric↗
DOE OSTI2022

Large Scale Caching and Streaming of Training Data for Online Deep Learning

The training of deep neural network models on large data remains a difficult problem, despite progress towards scalable techniques. In particular, there is a mismatch between the random but predetermined order in which AI flows select training samples and the streaming I/O patterns for which traditional HPC data storage (e.g., parallel file systems) are designed. In addition, as more data are obtained, it is feasible neither simply to train learning models incrementally, due to catastrophic forgetting (i.e., bias towards new samples), nor to train frequently from scratch, due to prohibitive time and/or resource constraints. In this paper, we study data management techniques that combine caching and streaming with rehearsal support in order to enable efficient access to training samples in both offline training and continual learning. We revisit state-of-art streaming approaches based on data pipelines that transparently handle prefetching, caching, shuffling, and data augmentation, and discuss the challenges and opportunities that arise when combining these methods with data-parallel training techniques. We also report on preliminary experiments that evaluate the I/O overheads involved in accessing the training samples from a parallel file system (PFS) under several concurrency scenarios, highlighting the impact of the PFS on the design of the data pipelines.

data pipelines↗
DOE OSTI2025

STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data

Error-bounded lossy compression is one of the most efficient solutions to reduce the volume of scientific data. For lossy compression, progressive decompression and random-access decompression are critical features that enable on-demand data access and flexible analysis workflows. However, these features can severely degrade compression quality and speed. To address these limitations, we propose a novel streaming compression framework that supports both progressive decompression and random-access decompression while maintaining high compression quality and speed. Our contributions are three-fold: (1) we design the first compression framework that simultaneously enables both progressive decompression and random-access decompression; (2) we introduce a hierarchical partitioning strategy to enable both streaming features, along with a hierarchical prediction mechanism that mitigates the impact of partitioning and achieves high compression quality—even comparable to state-of-the-art (SOTA) non-streaming compressor SZ3; and (3) our framework delivers high compression and decompression speed, up to 6.7 × faster than SZ3.

Wang, Daoce [University of Nebraska, Omaha]↗
DOE OSTI2019

Streaming Analytics Machine (SAM) v. 0.0.1

This is software developed in the course of my dissertation work, which was funded by Sandia. The software includes a library (the Streaming Analytics Machine or SAM) for expressing machine learning and subgraph matching over streaming data sources. It also includes a utility that will convert from a high-level language, the Streaming Analytics Language (SAL) to SAM. SAND2019-4322 M

Goodman, Eric↗
DOE OSTI2022

R-Stream 3.22.3-DOE

R-stream release 3.22.3 including the Glow and TVM front-ends, and the features developed in the Portacy 2B, X-MAT and STAPL projects. R-Stream is an automatic parallelization, source-to-source compiler. The TVM and Glow front-ends to R-Stream were developed to automatically map neural net models to parallel computers using openMP and CUDA (for Glow).

Meister, Benoit↗
DOE OSTI2022

Streaming Statistics

In the context of a larger effort for in situ data analytics, there is a need to calculate basic statistics metrics (e.g., count, mean, median) online as new data points become available. Originally, the code for such online, or in other words streaming, statistics was part of the TALASS (Topological Analysis of Large- Scale Simulations) library. We isolated the relevant code and created a standalone library from it called Streaming Statistics. We also added an ability to serialize and deserialize the statistics objects so that the library can be used in distributed, task-based processing. To use the Streaming Statistics library, the user chooses a statistic, constructs an object for it, and then "adds" values to it, which means the statistic is augmented.

Shudler, Sergei↗
DOE OSTI2020

Estimating Watershed Subsurface Permeability From Stream Discharge Data using Deep Neural Networks, Frontiers in Earth Science: Dataset

This data package contains watershed modeling inputs and outputs as well as deep neural networks training and testing results used in "Estimating Watershed Subsurface Permeability From Stream Discharge Data using Deep Neural Networks" (Cromwell et al., 2021). We train various deep neural network models with different architectures to predict subsurface permeability from stream discharge hydrograph at the watershed outlet. The training data are obtained from ensemble simulations of hydrographs corresponding to an permeability ensemble using a fully-distributed, integrated surface-subsurface hydrologic model. The trained model is then applied to estimate the permeability of the real watershed using its observed hydrograph at the outlet. Our study demonstrates that the permeabilities of the soil and geologic facies that make significant contributions to the outlet discharge can be more accurately estimated from the discharge data. Their estimations are also more robust with observation errors. Compared to the traditional ensemble smoother method, DNNs show stronger performance in capturing the nonlinear relationship between permeability and stream hydrograph to accurately estimate permeability. Our study sheds new light on the value of the emerging deep learning methods in assisting integrated watershed modeling by improving parameter estimation, which will eventually reduce the uncertainty in predictive watershed models. For detailed information regarding watershed model description and DNNs setup, please refer to Cromwell et al., 2021. The deep_learning.zip file contains the inputs and outputs for the different deep neural networks. The watershed_model.zip file contains the inputs and outputs for watershed simulation using ATS. The figures.zip file contains raw figures and their corresponding scripts. For a more detailed description, please see the README.md file within each compressed file.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Soil and Water Chemistry and Trace Metal Extractability and Speciation in Wetland Soils from Illinois and South Carolina and Stream Sediments from Tennessee

Dataset revised on October 15, 2021. This revision adds sulfur and iron X-ray absorption near-edge structure spectra for the wetland soils and stream sediments from the field areas. It also renames the sample locations in a way that is more intuitive to readers of the companion paper that is under review. Finally, the data filenames and organization have been updated in their labeling to parallel the data sources in the associated paper. The abstract text and methods were also revised to reflect the data that was added to the dataset.Trace metals are essential for microbially-mediated biogeochemical processes occurring in anoxic wetland soils and stream bed sediments, such as denitrification, methanogenesis, and mercury methylation. Low availability of these elements may potentially inhibit key components of anaerobic carbon and nitrogen cycling and contaminant transformation. The solid-phase speciation of trace metals likely plays an important role in controlling their bioavailability. Metal speciation is well studied in contaminated soils and sediments as well as those naturally elevated in trace metals. However, less is known regarding the chemical forms of trace metals in systems having concentrations similar to geological background levels, the very settings where metal limitations may be most prevalent. We have investigated trace metal concentrations, extractability, and solid-phase speciation in three freshwater subsurface aquatic systems: marsh wetland soils, riparian wetland soils, and the sediments of a streambed.Data are provided for marsh wetland soils at Argonne National Laboratory, riparian wetland soils in the Tims Branch watershed at Savannah River National Laboratory, and stream bed sediments from East Fork Poplar Creek near Oak Ridge National Laboratory. Soil and sediment elemental abundances, mineralogy, and extractable nutrients as well as dissolved major elements, anions, trace metals, and nutrients in the overlying surface waters are provided. In addition, the results of sequential chemical extraction for the trace metals cobalt, nickel, copper, and zinc from the soils and sediment are reported as well as X-ray absorption near-edge structure (XANES) spectra in these materials are reported. To aid interpretation of these data, XANES spectra of sulfur in the soils and sediments as well as both XANES and extended X-ray absorption fine structure (EXAFS) spectra of iron in these materials are reported. The data package also includes the XANES spectra of reference standards and a potential interferent in the measurements. All data are provided in text-based CSV format with header sections indicating the data contained in each file and the corresponding units. Note that "u" is used in place of Greek lower case mu to indicate the micro prefix on units.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Uptake and speciation of trace metal inputs to Wetland Soils from Illinois and South Carolina and Stream Sediments from Tennessee

Metals occur in all ecosystems, although their concentrations vary depending on their natural geologic conditions and surrounding human activities. In addition to metals intrinsically present from the geology of a particular system, metals can enter environmental systems from a wide variety of natural and anthropogenic sources, such as sediment re-suspension, mining operations, industrial processes, agricultural activities, and atmospheric deposition. While metals can be toxic at high concentrations, some metals serve as essential micronutrients for biogeochemical processes. Metal transport and availability in engineered and natural water systems depend on processes of adsorption/desorption, oxidation/reduction, dissolution/precipitation, and ligand complexation. Insights into the speciation of metals and their bioavailability will also help advance understanding of the roles of metals in the biogeochemical cycling of nutrients. We conducted batch experiments under anoxic conditions on soils and sediments collected from three different natural aquatic systems to understand their response to influxes of dissolved Cu, Ni and Zn. While soils and sediments from all sites could strongly bind added trace metals, there were substantial differences in trace metal uptake trends between different sites, especially for Cu. There was no distinct correlation between trace metal uptake and the total organic matter, iron, and sulfur content present in the samples. X-ray absorption spectroscopy indicated that the speciation of the freshly added metals taken up by the solids differs substantially from the speciation of the metals originally present in unamended samples. Cu sulfides dominated speciation at low loadings (1 µmol/g), whereas complexation to thiol groups and formation of metallic Cu governed speciation at high loadings (10 µmol/g). For Ni and Zn, adsorption to mineral surfaces and organic matter governed their speciation in materials from most sites. This study suggests that the background speciation of metals in natural aquatic systems is a poor predictor of the speciation and lability of metals introduced to terrestrial aquatic systems from anthropogenic or natural processes. Our findings imply that geochemical processes controlling trace metal speciation may vary considerably with metal loading in different natural systems. Data are provided for marsh wetland soils at Argonne National Laboratory (Marsh 1 and Marsh 2), riparian wetland soils (Riparian 1 and Riparian 2) in the Tims Branch watershed at Savannah River National Laboratory, and stream bed sediments (Stream 1 and Stream 2) from East Fork Poplar Creek near Oak Ridge National Laboratory. Data package includes the results of trace metal uptake experiments conducted on the selected sites for determining their capacity to immobilize metals under different loadings. XANES spectra for Cu, Ni, ad Zn at different loadings are included in the package. The abundance of different trace metal species obtained using linear combination fitting in ATHENA are also included in the data.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Model Data Archive Associated with Manuscript "Fire-altered Carbon Pools Create Disturbance Memory in Stream Dissolved Organic Carbon"

This data package supports the publication “Fire-altered Carbon Pools Create Disturbance Memory in Stream Dissolved Organic Carbon” by Li et al. (2026). The package contains processed model inputs, configuration files, restart files, simulation outputs, scripts, and visualization products used to evaluate post-fire dissolved organic carbon (DOC) dynamics in the Naches River Watershed, Washington, USA, following the 2021 Schneider Springs Fire. The modeling workflow couples ELM-BGC, the biogeochemistry-enabled Energy Exascale Earth System Model Land Model; ATS, the Advanced Terrestrial Simulator for integrated surface-subsurface hydrology; and PFLOTRAN, a reactive transport model for multicomponent aqueous geochemistry. Together, these models simulate how wildfire-induced changes in vegetation, litter, coarse woody debris, and soil organic matter influence DOC production, transport, and reaction from burned hillslopes to stream networks. The archive includes preprocessed meteorological, geospatial, hydrologic, and biogeochemical forcing data; ELM-BGC-derived DOC source terms; ATS mesh files; PFLOTRAN reactive-transport inputs; model configuration files; spin-up and transient restart files; watershed-scale diagnostic outputs; stream concentration time series; and figures or visualization files used to inspect and reproduce key results. File types include Hierarchical Data Format 5 (HDF5) files for gridded forcing and model-coupling data, model input and configuration files for ELM-BGC, ATS, and PFLOTRAN, restart and simulation-output files generated by the modeling workflow, tabular or time-series diagnostic outputs, scripts for post-processing and figure generation, and image or visualization products associated with the manuscript. Use of the package depends on the intended task. Re-running the simulations requires the relevant modeling software, including ELM-BGC, ATS, and PFLOTRAN as ATS's geochemical engine. Inspecting outputs and reproducing figures requires Python with scientific plotting libraries such as Matplotlib, and three-dimensional model outputs may be viewed with ParaView. Geographic information system files or maps may be inspected with ArcGIS Pro or comparable GIS software. The data package is intended to enable traceability, reuse, and partial reproduction of the coupled land-to-watershed hydro-biogeochemical modeling workflow used to test how wildfire disturbance affects terrestrial carbon pools and downstream DOC dynamics.

ATS↗
DOE OSTI2024

Ice-slurry production for solar-ice systems using supercooling with an in-stream crystallizer

Using ice slurry produced from supercooled water with an in-stream crystallizer opens a new path for solarice systems, increasing efficiency and reducing investment cost compared to ice-on-coil systems. Power needs are decoupled from the stored energy since the heat exchangers are not evenly distributed in the ice storage vessel but directly placed as the heat pump evaporator, which depends on the heat pump capacity and not on the ice storage volume. The novel ice storage design can serve as a heat source for a heat pump, where its evaporator is used to supercool water below 0 °C, or for refrigeration. The supercooled water at a temperature around −2 °C, maintained in a meta-stable liquid state, is triggered to nucleate in an in-stream crystallizer. The current design of the ice crystallizer allows for continuous operation of the system and scalability. Experimental results obtained with a gasketed plate heat exchanger used to supercool water and an in-stream crystallizer operated at a power of up to 6.5 kW are presented.

Thamm, Ann-Katrin↗
DOE OSTI2014

Special Analysis for the Disposal of the Idaho National Laboratory Unirradiated Light Water Breeder Reactor Rods and Pellets Waste Stream at the Area 5 Radioactive Waste Management Site, Nevada National Security Site, Nye County, Nevada

The purpose of this special analysis (SA) is to determine if the Idaho National Laboratory (INL) Unirradiated Light Water Breeder Reactor (LWBR) Rods and Pellets waste stream (INEL103597TR2, Revision 2) is suitable for disposal by shallow land burial (SLB) at the Area 5 Radioactive Waste Management Site (RWMS). The INL Unirradiated LWBR Rods and Pellets waste stream consists of 24 containers with unirradiated fabricated rods and pellets composed of uranium oxide (UO2) and thorium oxide (ThO2) fuel in zirconium cladding. The INL Unirradiated LWBR Rods and Pellets waste stream requires an SA because the 229 Th, 230 Th, 232 U, 233 U, and 234 U activity concentrations exceed the Nevada National Security Site (NNSS) Waste Acceptance Criteria (WAC) Action Levels.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗