Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “STREAMS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Evidence for C and Mg variations in the GD-1 stellar stream

ABSTRACT Dynamically cold stellar streams are the relics left over from globular cluster dissolution. These relics offer a unique insight into a now fully disrupted population of ancient clusters in our Galaxy. Using a combination of Gaia eDR3 proper motions, optical and near-UV colours, we select a sample of likely Red Giant Branch stars from the GD-1 stream for medium-low resolution spectroscopic follow-up. Based on radial velocity and metallicity, we are able to find 14 new members of GD-1, 5 of which are associated with the spur and blob/cocoon off-stream features. We measured C-abundances to probe for abundance variations known to exist in globular clusters. These variations are expected to manifest in a subtle way in globular clusters with such low masses ($\sim 10^4\,{\rm ~\textrm {M}_\odot }$) and metallicities ([Fe/H] ∼ −2.1 dex). We find that the C-abundances of the stars in our sample display a small but significant (3σ level) spread. Furthermore, we find ∼3σ variation in Mg-abundances among the stars in our sample that have been observed by APOGEE. These abundance patterns match the ones found in Galactic globular clusters of similar metallicity. Our results suggest that GD-1 represents another fully disrupted low-mass globular cluster where light-element abundance spreads have been found.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Redshift-space streaming velocity effects on the Lyman- α forest baryon acoustic oscillation scale

The baryon acoustic oscillation (BAO) scale acts as a standard ruler for measuring cosmological distances and has therefore emerged as a leading probe of cosmic expansion history. However, any physical effect that alters the length of the ruler can lead to a bias in our determination of distance and expansion rate. One of these physical effects is the streaming velocity, the relative velocity between baryons and dark matter in the early Universe, which couples to the BAO scale due to their common origin in acoustic waves at recombination. In this work, we investigate the impact of streaming velocity on the BAO feature of the Lyman- α forest auto-power spectrum, one of the main tracers being used by the recently commissioned Dark Energy Spectroscopic Instrument (DESI). To do this, we develop a perturbative model for Lyman- α flux fluctuations which is complete to second order for a certain set of fields, and applicable to any redshift-space tracer of structure since it is based only on symmetry considerations. We find that there are 8 biasing coefficients through second order. We find streaming velocity-induced shifts in the BAO scale of 0.081%–0.149% (transverse direction) and 0.053%–0.058% (radial direction), depending on the model for the biasing coefficients used. These are smaller than, but not negligible compared to, the DESI Lyman- α BAO error budget, which is 0.46% on the overall scale. Overall, the sensitivity of these results to our choice of bias parameters underscores the need for future work to measure the higher-order biasing coefficients from simulations, especially for future experiments beyond DESI.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

Mitigating Catastrophic Forgetting in Deep Learning in a Streaming Setting Using Historical Summary

Recent advancements in scientific equipment and the adaptation of electronics and the Internet of Things (IoT) in our everyday lives resulted in large and complex data production at a high rate. Making meaningful and timely knowledge discovery at a modest cost from this big data is difficult for computing power and storage limitations. Training deep learning models incrementally in a streaming setting can help us with overcoming these limitations. However, in a well-known phenomenon named catastrophic forgetting, incrementally trained models increasingly perform poorly on the past data. To mitigate catastrophic forgetting in training in a streaming setting, we propose constructing a historical summary over time and use the summary with newly arrived data during incremental training. We propose various data summarization techniques such as random sampling, micro clustering, coreset computation, and Auto Encoders to counteract catastrophic forgetting. We built a pipeline for incremental training with a historical summary for training deep learning models for streaming data. We demonstrate the effectiveness of historical summary in mitigating catastrophic forgetting using three case studies involving three different deep learning applications: an Artificial Neural Network (ANN) for classification task on MNIST dataset, a language model (RNN-LM) on the WikiText2 dataset, and a Convolutional Neural Network (CNN), ResNet50 to classify the ImageNet dataset. Through the training of the models, we observe that catastrophic forgetting is evident in ANN and CNN but not in an RNN. For the first task, our method recovers up to 47.9% lost accuracy due to catastrophic forgetting. For the third task, the historical summary recovers classification accuracy by up to 25%. For the second task, though there is not proof of catastrophic forgetting, the training performance (PPL) improves by up to 26% with historical summary.

Dash, Sajal↗
DOE OSTI2021

DynPaC: Coarse-Grained, Dynamic, and Partially Reconfigurable Array for Streaming Applications

Coarse-grained reconfigurable arrays (CGRAs) provide higher flexibility than application-specific integrated circuits (ASICs) and higher efficiency than fine-grained reconfigurable devices such as Field Programmable Gate Arrays (FPGAs). However, CGRAs are generally designed to support offloading of a single kernel. While their design, based on communicating functional units, appears to naturally suit streaming applications composed of multiple cooperating kernels, current approaches only statically partition the resources across kernels. However, streaming applications often are data-dependent, leading to variable kernel execution times depending on the input data and impacting the throughput of the entire pipeline if resources are statically allocated. Therefore, in this paper, we discuss the design of DynPaC — a coarse-grained, dynamically, and partially reconfigurable array for data-dependent streaming applications. We discuss the required software and hardware components to manage partial dynamic reconfiguration. We demonstrate that by supporting partial dynamic reconfiguration, we can obtain an average speedup of 1.44X for a representative set of applications w.r.t. static partitioning, with a limited area overhead (6.4% of the entire chip).

Tan, Cheng↗
DOE OSTI2021

Leveraging In-Network Computing and Programmable Switches for Streaming Analysis of Scientific Data

With the emergence of programmable network devices that match the performance of fixed function devices, several recent projects have explored in-network computing, where the processing that is traditionally done outside the network is offloaded to the network devices. In-network computing has typically been applied to network functions (e.g., load balancing, NAT, and DNS), caching, data reduction/aggregation, and coordination/consensus functions. In some cases it has been used to accelerate stream-processing tasks that involve small payloads and simple operations. In this work we focus on leveraging in-network computing for stream processing of scientific datasets with large payloads that require complex operations such as floating-point computations and logarithmic functions. We demonstrate in-network computing for a real-world scientific application performing streaming normalization of a 2-D image from a light source experiment. We discuss the challenges we encountered and potential approaches to address them.

Sankaran, Ganesh↗
DOE OSTI2023

ERSAP: Toward Better NP Data-Stream Analytics With Flow-Based Programming

This paper presents an reactive, actor-model and FBP paradigm based framework that we develop to design data-stream processing applications for HEP and NP. This framework encourages a functional decomposition of the overall data processing application into small mono-functional artifacts. Artifacts that are easy to understand, develop, deploy and debug. The fact that these artifacts (actors) are programmatically independent they can be scaled and optimized independently, which is impossible to do for components of the monolithic application. One of the important advantages of this approach is fault tolerance where independent actors can come and go on the data-stream without forcing the entire application to crash. Furthermore, it also makes it is easy to locate the faulty actor in the data pipeline. Due the fact that the actors are loosely coupled, and that the data (inevitably) carries the context, they can run on heterogeneous environments, utilizing different accelerators. This paper describes the main design concepts of the framework and presents a ?proof of concept? application design and deployment results obtain processing on-beam calorimeter streaming data.

Gyurjyan, V.↗
DOE OSTI2023

Salinization and sedimentation drive contrasting assembly mechanisms of planktonic and sediment‐bound bacterial communities in agricultural streams

Abstract Agriculture is the most dominant land use globally and is projected to increase in the future to support a growing human population but also threatens ecosystem structure and services. Bacteria mediate numerous biogeochemical pathways within ecosystems. Therefore, identifying linkages between stressors associated with agricultural land use and responses of bacterial diversity is an important step in understanding and improving resource management. Here, we use the Mississippi Alluvial Plain (MAP) ecoregion, a highly modified agroecosystem, as a case study to better understand agriculturally associated drivers of stream bacterial diversity and assembly mechanisms. In the MAP, we found that planktonic bacterial communities were strongly influenced by salinity. Tolerant taxa increased with increasing ion concentrations, likely driving homogenous selection which accounted for ~90% of assembly processes. Sediment bacterial phylogenetic diversity increased with increasing agricultural land use and was influenced by sediment particle size, with assembly mechanisms shifting from homogenous to variable selection as differences in median particle size increased. Within individual streams, sediment heterogeneity was correlated with bacterial diversity and a subsidy‐stress relationship along the particle size gradient was observed. Planktonic and sediment communities within the same stream also diverged as sediment particle size decreased. Nutrients including carbon, nitrogen, and phosphorus, which tend to be elevated in agroecosystems, were also associated with detectable shifts in bacterial community structure. Collectively, our results establish that two understudied variables, salinity and sediment texture, are the primary drivers of bacterial diversity within the studied agroecosystem, whereas nutrients are secondary drivers. Although numerous macrobiological communities respond negatively, we observed increasing bacterial diversity in response to agricultural stressors including salinization and sedimentation. Elevated taxonomic and phylogenetic bacterial diversity likely increases the probability of detecting community responses to stressors. Thus, bacteria community responses may be more reliable for establishing water quality goals within highly modified agroecosystems that have experienced shifting baselines.

DeVilbiss, Stephen E.↗
DOE OSTI2020

Empirical Study of Focus-Plus-Context and Aggregation Techniques for the Visualization of Streaming Data

Analysis of streaming data often involves both real-time monitoring of incoming data as well as contextual awareness of data history. A focus-plus-context approach can support both goals, with variable levels of visual aggregation making it possible to provide a high level of detail for incoming and recent data while providing contextual information about recent history. Visual aggregation reduces data resolution in order to show the context of data over large periods of time within a limited display space. With a controlled experiment, we evaluated the effectiveness of different types of aggregation for four types of stream-analysis tasks. Overall, the results show that a focus-plus-context design has little negative impact on the ability to successfully monitor and analyze streaming data, making it possible to show longer periods of time than other approaches. However, visual aggregation can be problematic for trend recognition tasks. This research demonstrates how the effectiveness of the visualization depends on the specifics of the analysis task.

Ragan, Eric↗
DOE OSTI2022

Large Scale Caching and Streaming of Training Data for Online Deep Learning

The training of deep neural network models on large data remains a difficult problem, despite progress towards scalable techniques. In particular, there is a mismatch between the random but predetermined order in which AI flows select training samples and the streaming I/O patterns for which traditional HPC data storage (e.g., parallel file systems) are designed. In addition, as more data are obtained, it is feasible neither simply to train learning models incrementally, due to catastrophic forgetting (i.e., bias towards new samples), nor to train frequently from scratch, due to prohibitive time and/or resource constraints. In this paper, we study data management techniques that combine caching and streaming with rehearsal support in order to enable efficient access to training samples in both offline training and continual learning. We revisit state-of-art streaming approaches based on data pipelines that transparently handle prefetching, caching, shuffling, and data augmentation, and discuss the challenges and opportunities that arise when combining these methods with data-parallel training techniques. We also report on preliminary experiments that evaluate the I/O overheads involved in accessing the training samples from a parallel file system (PFS) under several concurrency scenarios, highlighting the impact of the PFS on the design of the data pipelines.

data pipelines↗
DOE OSTI2025

STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data

Error-bounded lossy compression is one of the most efficient solutions to reduce the volume of scientific data. For lossy compression, progressive decompression and random-access decompression are critical features that enable on-demand data access and flexible analysis workflows. However, these features can severely degrade compression quality and speed. To address these limitations, we propose a novel streaming compression framework that supports both progressive decompression and random-access decompression while maintaining high compression quality and speed. Our contributions are three-fold: (1) we design the first compression framework that simultaneously enables both progressive decompression and random-access decompression; (2) we introduce a hierarchical partitioning strategy to enable both streaming features, along with a hierarchical prediction mechanism that mitigates the impact of partitioning and achieves high compression quality—even comparable to state-of-the-art (SOTA) non-streaming compressor SZ3; and (3) our framework delivers high compression and decompression speed, up to 6.7 × faster than SZ3.

Wang, Daoce [University of Nebraska, Omaha]↗
DOE OSTI2022

R-Stream 3.22.3-DOE

R-stream release 3.22.3 including the Glow and TVM front-ends, and the features developed in the Portacy 2B, X-MAT and STAPL projects. R-Stream is an automatic parallelization, source-to-source compiler. The TVM and Glow front-ends to R-Stream were developed to automatically map neural net models to parallel computers using openMP and CUDA (for Glow).

Meister, Benoit↗
DOE OSTI2022

Streaming Statistics

In the context of a larger effort for in situ data analytics, there is a need to calculate basic statistics metrics (e.g., count, mean, median) online as new data points become available. Originally, the code for such online, or in other words streaming, statistics was part of the TALASS (Topological Analysis of Large- Scale Simulations) library. We isolated the relevant code and created a standalone library from it called Streaming Statistics. We also added an ability to serialize and deserialize the statistics objects so that the library can be used in distributed, task-based processing. To use the Streaming Statistics library, the user chooses a statistic, constructs an object for it, and then "adds" values to it, which means the statistic is augmented.

Shudler, Sergei↗
DOE OSTI2020

Estimating Watershed Subsurface Permeability From Stream Discharge Data using Deep Neural Networks, Frontiers in Earth Science: Dataset

This data package contains watershed modeling inputs and outputs as well as deep neural networks training and testing results used in "Estimating Watershed Subsurface Permeability From Stream Discharge Data using Deep Neural Networks" (Cromwell et al., 2021). We train various deep neural network models with different architectures to predict subsurface permeability from stream discharge hydrograph at the watershed outlet. The training data are obtained from ensemble simulations of hydrographs corresponding to an permeability ensemble using a fully-distributed, integrated surface-subsurface hydrologic model. The trained model is then applied to estimate the permeability of the real watershed using its observed hydrograph at the outlet. Our study demonstrates that the permeabilities of the soil and geologic facies that make significant contributions to the outlet discharge can be more accurately estimated from the discharge data. Their estimations are also more robust with observation errors. Compared to the traditional ensemble smoother method, DNNs show stronger performance in capturing the nonlinear relationship between permeability and stream hydrograph to accurately estimate permeability. Our study sheds new light on the value of the emerging deep learning methods in assisting integrated watershed modeling by improving parameter estimation, which will eventually reduce the uncertainty in predictive watershed models. For detailed information regarding watershed model description and DNNs setup, please refer to Cromwell et al., 2021. The deep_learning.zip file contains the inputs and outputs for the different deep neural networks. The watershed_model.zip file contains the inputs and outputs for watershed simulation using ATS. The figures.zip file contains raw figures and their corresponding scripts. For a more detailed description, please see the README.md file within each compressed file.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Soil and Water Chemistry and Trace Metal Extractability and Speciation in Wetland Soils from Illinois and South Carolina and Stream Sediments from Tennessee

Dataset revised on October 15, 2021. This revision adds sulfur and iron X-ray absorption near-edge structure spectra for the wetland soils and stream sediments from the field areas. It also renames the sample locations in a way that is more intuitive to readers of the companion paper that is under review. Finally, the data filenames and organization have been updated in their labeling to parallel the data sources in the associated paper. The abstract text and methods were also revised to reflect the data that was added to the dataset.Trace metals are essential for microbially-mediated biogeochemical processes occurring in anoxic wetland soils and stream bed sediments, such as denitrification, methanogenesis, and mercury methylation. Low availability of these elements may potentially inhibit key components of anaerobic carbon and nitrogen cycling and contaminant transformation. The solid-phase speciation of trace metals likely plays an important role in controlling their bioavailability. Metal speciation is well studied in contaminated soils and sediments as well as those naturally elevated in trace metals. However, less is known regarding the chemical forms of trace metals in systems having concentrations similar to geological background levels, the very settings where metal limitations may be most prevalent. We have investigated trace metal concentrations, extractability, and solid-phase speciation in three freshwater subsurface aquatic systems: marsh wetland soils, riparian wetland soils, and the sediments of a streambed.Data are provided for marsh wetland soils at Argonne National Laboratory, riparian wetland soils in the Tims Branch watershed at Savannah River National Laboratory, and stream bed sediments from East Fork Poplar Creek near Oak Ridge National Laboratory. Soil and sediment elemental abundances, mineralogy, and extractable nutrients as well as dissolved major elements, anions, trace metals, and nutrients in the overlying surface waters are provided. In addition, the results of sequential chemical extraction for the trace metals cobalt, nickel, copper, and zinc from the soils and sediment are reported as well as X-ray absorption near-edge structure (XANES) spectra in these materials are reported. To aid interpretation of these data, XANES spectra of sulfur in the soils and sediments as well as both XANES and extended X-ray absorption fine structure (EXAFS) spectra of iron in these materials are reported. The data package also includes the XANES spectra of reference standards and a potential interferent in the measurements. All data are provided in text-based CSV format with header sections indicating the data contained in each file and the corresponding units. Note that "u" is used in place of Greek lower case mu to indicate the micro prefix on units.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Uptake and speciation of trace metal inputs to Wetland Soils from Illinois and South Carolina and Stream Sediments from Tennessee

Metals occur in all ecosystems, although their concentrations vary depending on their natural geologic conditions and surrounding human activities. In addition to metals intrinsically present from the geology of a particular system, metals can enter environmental systems from a wide variety of natural and anthropogenic sources, such as sediment re-suspension, mining operations, industrial processes, agricultural activities, and atmospheric deposition. While metals can be toxic at high concentrations, some metals serve as essential micronutrients for biogeochemical processes. Metal transport and availability in engineered and natural water systems depend on processes of adsorption/desorption, oxidation/reduction, dissolution/precipitation, and ligand complexation. Insights into the speciation of metals and their bioavailability will also help advance understanding of the roles of metals in the biogeochemical cycling of nutrients. We conducted batch experiments under anoxic conditions on soils and sediments collected from three different natural aquatic systems to understand their response to influxes of dissolved Cu, Ni and Zn. While soils and sediments from all sites could strongly bind added trace metals, there were substantial differences in trace metal uptake trends between different sites, especially for Cu. There was no distinct correlation between trace metal uptake and the total organic matter, iron, and sulfur content present in the samples. X-ray absorption spectroscopy indicated that the speciation of the freshly added metals taken up by the solids differs substantially from the speciation of the metals originally present in unamended samples. Cu sulfides dominated speciation at low loadings (1 µmol/g), whereas complexation to thiol groups and formation of metallic Cu governed speciation at high loadings (10 µmol/g). For Ni and Zn, adsorption to mineral surfaces and organic matter governed their speciation in materials from most sites. This study suggests that the background speciation of metals in natural aquatic systems is a poor predictor of the speciation and lability of metals introduced to terrestrial aquatic systems from anthropogenic or natural processes. Our findings imply that geochemical processes controlling trace metal speciation may vary considerably with metal loading in different natural systems. Data are provided for marsh wetland soils at Argonne National Laboratory (Marsh 1 and Marsh 2), riparian wetland soils (Riparian 1 and Riparian 2) in the Tims Branch watershed at Savannah River National Laboratory, and stream bed sediments (Stream 1 and Stream 2) from East Fork Poplar Creek near Oak Ridge National Laboratory. Data package includes the results of trace metal uptake experiments conducted on the selected sites for determining their capacity to immobilize metals under different loadings. XANES spectra for Cu, Ni, ad Zn at different loadings are included in the package. The abundance of different trace metal species obtained using linear combination fitting in ATHENA are also included in the data.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Model Data Archive Associated with Manuscript "Fire-altered Carbon Pools Create Disturbance Memory in Stream Dissolved Organic Carbon"

This data package supports the publication “Fire-altered Carbon Pools Create Disturbance Memory in Stream Dissolved Organic Carbon” by Li et al. (2026). The package contains processed model inputs, configuration files, restart files, simulation outputs, scripts, and visualization products used to evaluate post-fire dissolved organic carbon (DOC) dynamics in the Naches River Watershed, Washington, USA, following the 2021 Schneider Springs Fire. The modeling workflow couples ELM-BGC, the biogeochemistry-enabled Energy Exascale Earth System Model Land Model; ATS, the Advanced Terrestrial Simulator for integrated surface-subsurface hydrology; and PFLOTRAN, a reactive transport model for multicomponent aqueous geochemistry. Together, these models simulate how wildfire-induced changes in vegetation, litter, coarse woody debris, and soil organic matter influence DOC production, transport, and reaction from burned hillslopes to stream networks. The archive includes preprocessed meteorological, geospatial, hydrologic, and biogeochemical forcing data; ELM-BGC-derived DOC source terms; ATS mesh files; PFLOTRAN reactive-transport inputs; model configuration files; spin-up and transient restart files; watershed-scale diagnostic outputs; stream concentration time series; and figures or visualization files used to inspect and reproduce key results. File types include Hierarchical Data Format 5 (HDF5) files for gridded forcing and model-coupling data, model input and configuration files for ELM-BGC, ATS, and PFLOTRAN, restart and simulation-output files generated by the modeling workflow, tabular or time-series diagnostic outputs, scripts for post-processing and figure generation, and image or visualization products associated with the manuscript. Use of the package depends on the intended task. Re-running the simulations requires the relevant modeling software, including ELM-BGC, ATS, and PFLOTRAN as ATS's geochemical engine. Inspecting outputs and reproducing figures requires Python with scientific plotting libraries such as Matplotlib, and three-dimensional model outputs may be viewed with ParaView. Geographic information system files or maps may be inspected with ArcGIS Pro or comparable GIS software. The data package is intended to enable traceability, reuse, and partial reproduction of the coupled land-to-watershed hydro-biogeochemical modeling workflow used to test how wildfire disturbance affects terrestrial carbon pools and downstream DOC dynamics.

ATS↗
DOE OSTI2024

Ice-slurry production for solar-ice systems using supercooling with an in-stream crystallizer

Using ice slurry produced from supercooled water with an in-stream crystallizer opens a new path for solarice systems, increasing efficiency and reducing investment cost compared to ice-on-coil systems. Power needs are decoupled from the stored energy since the heat exchangers are not evenly distributed in the ice storage vessel but directly placed as the heat pump evaporator, which depends on the heat pump capacity and not on the ice storage volume. The novel ice storage design can serve as a heat source for a heat pump, where its evaporator is used to supercool water below 0 °C, or for refrigeration. The supercooled water at a temperature around −2 °C, maintained in a meta-stable liquid state, is triggered to nucleate in an in-stream crystallizer. The current design of the ice crystallizer allows for continuous operation of the system and scalability. Experimental results obtained with a gasketed plate heat exchanger used to supercool water and an in-stream crystallizer operated at a power of up to 6.5 kW are presented.

Thamm, Ann-Katrin↗
DOE OSTI2024

Exploring DAOS as a Burst Buffer for a 100 Gbps DAQ Real-Time Streaming System

We present an experimental evaluation of a burst buffer for a real-time DAQ streaming system designed to transmit instrument data to remote data centers. The system is based on EJ-FAT, a load balancing system capable of Nx 100Gbps streams, distributing data from event sources to processing nodes. We explore applying the DAOS system as a burst buffer to serve a number of purposes: improve resiliency, elasticity and add new functions into the processing pipeline. In the evaluation a sender transmits events over a 100Gbps network to a receiver integrated with DAOS to store the reassembled events using DAOS APIs. We evaluate the system for possible bottlenecks and provide end-to-end evaluation with a burst buffer using DAOS storage abstractions. We show that a receiver node can support 38.1 Gbps. This proves the viability of our approach and allows us to extend this work to investigate scale-out properties and new streaming optimizations.

Mei, Xinxin↗