Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 595 records · Page 33
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Uniform modelling of the stellar density of thirteen tidal streams within the Galactic halo

ABSTRACT We present the results of fitting a flexible stellar stream density model to a collection of thirteen streams around the Milky Way, using photometric data from DES, DECaLS, and Pan-STARRS. We construct density maps for each stream and characterize their tracks on the sky, width, and distance modulus curves along the length of each stream. We use these measurements to compute lengths and total luminosities of streams and identify substructures. Several streams show prominent substructures, such as stream broadening, gaps, large deviations of stream tracks, and sharp changes in stream densities. Examining the group of streams as a population, as expected we find that streams with globular cluster progenitors are typically narrower than those with dwarf galaxy progenitors, with streams around 100 pc wide showing overlap between the two populations. We also note the average luminosity of globular cluster streams is significantly lower than the typical luminosity of intact globular clusters. The likely explanation is that observed globular cluster streams preferentially come from lower luminosity and lower density clusters. The stream measurements done in a uniform manner presented here will be helpful for more detailed stream studies such as identifying candidate stream members for spectroscopic follow up and stellar stream dynamical modelling.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Trigger-DAQ and Slow Controls Systems in the Mu2e Experiment

The muon campus program at Fermilab includes the Mu2e experiment that will search for a charged-lepton flavor violating processes where a negative muon converts into an electron in the field of an aluminum nucleus, improving by four orders of magnitude the search sensitivity reached so far. Mu2e's Trigger and Data Acquisition System (TDAQ) uses otsdaq as its solution. Developed at Fermilab, otsdaq uses the artdaq DAQ framework and art analysis framework, under the-hood, for event transfer, filtering, and processing. otsdaq is an online DAQ software suite with a focus on flexibility and scalability, while providing a multi-user, web-based, interface accessible through the Chrome or Firefox web browser. The detector Read Out Controller (ROC), from the tracker and calorimeter, stream out zero-suppressed data continuously to the Data Transfer Controller (DTC). Data is then read over the PCIe bus to a software filter algorithm that selects events which are finally combined with the data flux that comes froma Cosmic Ray Ve to System (CRV). A Detector Control System (DCS) for monitoring, controlling, alarming, and archiving has been developed using the Experimental Physics and Industrial Control System (EPICS) Open Source Platform. The DCS System has also been itegrated into otsdaq. The installation of the TDAQ and the DCS systems in the Mu2e building is planned for 2021-2022, and a prototype has been built at Fermilab's Feynman Computing Center. We report here on the developments and achievements of the integration of Mu2e's DCS system into the online otsdaq software.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2026

Synoptic NO3 in Slate River Watershed, Colorado (2022)

The synoptic nitrate (NO3) dataset in the Slate River Watershed, Colorado consists of NO3 data collected at 19 locations three times during the summer of 2022. Stream samples were collected in early summer (early July), mid summer (late August), and late summer (late September). The samples include mainstem, tributary, and point source input water samples. These data were collected to evaluate spatiotemporal variability in stream NO3 during the summer, and evaluate anthropogenic controls on stream NO3 dynamics. This data package contains: (1) a csv of all NO3 samples and (2) a csv of locations for each sampling site. The dataset additionally includes a file-level metadata (flmd.csv) file that lists each file contained in the dataset with associated metadata; and a data dictionary (dd.csv) file that contains column/row headers used throughout the files along with a definition, units, and data type. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231.

EARTH SCIENCE > BIOSPHERE > ECOSYSTEMS > FRESHWATE↗
DOE OSTI2023

HERMES: A cyber-energy SAAS platform to provide real-time 2d and 3d interactive visualizations [SWR-23-15]

HERMES is a novel visualization SAAS platform for achieving real-time visualization of large-scale environments involving cyber-energy devices. The platform receives data from emulated environments which include physical hardware and network devices that represent a complex IT / OT system. The platform is capable of streaming, filtering, storing, and visualizing all data within the environment and scaling vertically and horizontally. This capability enables high fidelity visual analysis of events to be performed in real time as well as the collection and storage of historical data for forensic analysis.

Van Natta, Joshua↗
DOE OSTI2023

Model Inputs, Outputs, and Scripts associated with: “Combined effects of stream hydrology and land use on basin-scale hyporheic zone denitrification in the Columbia River Basin”

This data package is associated with the publication “Combined effects of stream hydrology and land use on basin‐scale hyporheic zone denitrification in the Columbia River Basin”, published in Water Resource Research (Son et al.2022) available at https://doi.org/10.1029/2021WR031131. This data package includes the key model inputs/outputs of the river corridor model for the Columbia River Basin (CRB) and the model source codes used in the manuscript. The model is a carbon-nitrogen-coupled river corridor model (RCM), and the model is used to quantify hyporheic zone (HZ) denitrification at the NHDPLUS stream reach scales. The RCM used in this study combines empirical substrate models derived from observations and three microbially driven reactions, including two-step denitrification and aerobic respiration, are considered within the HZ. The key input data of the model are exchange flux, residence time, and stream solute (dissolved organic carbon (DOC), dissolved oxygen (DO), and nitrate concentrations). These inputs are constant over time and represent long-term averaged values. This study uses the RCM to explore the spatial patterns of HZ denitrification across reaches with different sizes and land use in the CRB. Our main objective is to use the RCM as a virtual reality model, and the machine-learning models as surrogates that encapsulate the complexities of the physics-based model while identifying the importance of different variables that are not evident in the model conceptualization. We do not include a direct comparison of the modeled HZ denitrification and measurements; however, the RCM can capture the overall spatial patterns of the HZ denitrification because the model inputs and its reaction networks are based on well-established theory and a physical-based model. The combination of the model-based predictions and a machine-learning approach (e.g., random forest) is used to improve our understanding of what variables of the model are associated with spatial patterns of the modeled denitrification across reaches with different sizes and land uses, and to develop a proxy model using measurable variables to reproduce the simulated patterns.This dataset contains five folders: (1) model_inputs, (2) model_outputs, (3) Rscripts, (4) figures, and (5) model_codes. It also contains a readme, file level metadata (FLMD), and data dictionary (dd). Please see the FLMD for a list of all the files contained in this data package and descriptions for each. The model_inputs folder contains the model inputs used to drive the model simulations. The model_outputs folder contains key model output files from the river corridor model. The Rscripts folder contains the Rscripts for pre- and post- processing model results. The figures folder contains the raw figures associated with the manuscript. The model_codes folder includes key model source codes/input files. All files are .jpg, .jpeg, .out, .e, .od, .dat, .sub, .F90, .0, .R, .sbx, .cpg, .sbn, .shx, .shp, .dbf, .prj, .tfw, .tif, .xml, .pdf, or .csv.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

VDiSC: An Open Source Framework for Distributed Smart City Vision and Biometric Surveillance Networks

Recent global growth in the interest of smart cities has led to trillions of dollars of investment toward research and development. These connected cities have the potential to create a symbiosis of technology and society and revolutionize the cost of living, safety, ecological sustainability, and quality of life of societies on a world-wide scale. Some key components of the smart city construct are connected smart grids, self-driving cars, federated learning systems, smart utilities, large-scale public transit, and proactive surveillance systems. While exciting in prospect, these technologies and their subsequent integration cannot be attempted without addressing the potential societal impacts of such a high degree of automation and data sharing. Additionally, the feasibility of coordinating so many disparate tasks will require a fast, extensible, unifying framework. To that end, we propose the Distributed Smart City framework for Vision, or VDiSC. VDiSC serves as a unified biometric API harness that allows for seamless evaluation, deployment, and simple pipeline creation for heterogeneous biometric software. VDiSC additionally provides a fully declarative capability for defining and coordinating custom machine learning and sensor pipelines, allowing the distribution of processes across otherwise incompatible hardware and networks. VDiSC ultimately provides a way to quickly configure, hot-swap, and expand large coordinated or federated systems online without interruptions for maintenance. Because much of the data collected in a smart city contains Personally Identifying Information (PII), VDiSC also provides built-in tools and layers to ensure secure and encrypted streaming, storage, and access of PII data across distributed systems.

Brogan, Joel↗
DOE OSTI2023

Inference finds consistency between a neutrino flavor evolution model and Earth-based solar neutrino measurements

We continue examining statistical data assimilation (SDA), an inference methodology, to infer solutions to neutrino flavor evolution, for the first time using real - rather than simulated - data. The model represents neutrinos streaming from the Sun's center and undergoing a Mikheyev-Smirnov-Wolfenstein (MSW) resonance in flavor space, due to the radially-varying electron number density. The model neutrino energies are chosen to correspond to experimental bins in the Sudbury Neutrino Observatory (SNO) and Borexino experiments, which measure electron-flavor survival probability at Earth. In conclusion, the procedure successfully finds consistency between the observed fluxes and the model, if the MSW resonance - that is, flavor evolution due to solar electrons - is included in the dynamical equations representing the model.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2022

areaDetector/ADTimePix3

ADTimePix3 is an open‑source EPICS areaDetector driver for TimePix3 pixel detectors from Advanced Scientific Instruments (ASI). It provides a complete control and data‑acquisition interface between EPICS and the Serval server, supporting both real detectors (single‑chip and 2×2 quad configurations) and the TimePix3 emulator. The driver configures measurements via an HTTP/JSON REST API and acquires data through high‑rate TCP streams for raw events, images, and histograms. It offers real‑time preview and accumulated images, Time‑of‑Flight histogram processing, pixel masking (including Binary Pixel Configuration files and hot‑pixel tools), extensive health and status monitoring, and integration with standard areaDetector plugins for file saving and downstream analysis. ADTimePix3 is designed for Linux 64‑bit systems (tested on Ubuntu and RHEL) and is distributed under an MIT license with additional institutional disclaimers, enabling reuse and extension in photon science beamlines and other TimePix3‑based experimental facilities.

Gofron, Kaz↗
DOE OSTI2023

NEWTS EPA Gasification Dataset and Case Studies

Data from the US EPA Gasification Database containing gasification effluent data from selected U.S. coal power plants. Original data from: Nguyen, Dan-Tam, Eastern Research Group. Sep 29, 2015. Analytical Database for the Steam Electric Rulemaking - DCN SE05359. https://www.regulations.gov/document/EPA-HQ-OW-2009-0819-5640 Files include: Original data tab, processed data and charge balance information in "NEWTS_EPA_Gasification_Effluent_Dataset_all_tabs.xlsx" Individual .csv files for data in format for input into aqueous chemistry modeling software (OLI Systems and Geochemist's Workbench) Data input templates for modeling streams in OLI Systems and Geochemist's Workbench Case studies for selected solved streams modeled in OLI Systems and Geochemist's Workbench

gasification↗
DOE OSTI2022

Williston Basin Carbon Ore, Rare Earth, and Critical Minerals (CORE-CM) Program

The U.S. Department of Energy’s Office of Fossil Energy awarded 13 CORE-CM programs as part of the CORE-CM Initiative, designed to develop the technology and upstream and midstream supply chains to extract rare-earth elements (REEs) and critical minerals (CMs) from the nation’s coal supplies. The intent is to catalyze regional economic growth and job creation while strengthening the use of domestic resources. The Williston Basin CORE-CM Program aims to drive the expansion and transformation of coal and coal-based resource usage within the Williston Basin to produce REEs, CMs, and nonfuel carbon-based products (CBPs). The work constitutes Phase 1 of a long-term program. The objectives of Phase 1 are to identify the existing knowledge base and gaps and to develop a series of assessments/plans, including an initial basinal assessment; a characterization and data acquisition plan; a waste stream reuse plan; a basinal strategies assessment for infrastructure, industries, and business; a technology assessment, development, and field testing plan; a technology innovation center plan(s); and a stakeholder outreach and education plan. The goal of this project is to initiate the development of a new industry in the resource-rich Williston Basin that will catalyze economic growth and job creation through a coalition team of private industry, university, and state, local, and federal government entities. A coalition team of nearly 30 private industry, university, and state, local, and federal government partners was formed for this program. Partners include the University of North Dakota Energy & Environmental Research Center, Institute for Energy Studies, and Nistler College; North Dakota State University; Montana Tech University; Pacific Northwest National Laboratory; Critical Materials Institute; North Dakota Geological Survey; South Dakota Geological Survey; U.S. Geological Survey; North American Coal Corporation; BNI Energy; Basin Electric Power Cooperative; Minnkota Power Cooperative; and many more.

Kay, John P.↗
DOE OSTI2023

Williston Basin Carbon Ore, Rare Earth, and Critical Minerals (CORE-CM) Program

The U.S. Department of Energy’s (DOE’s) Office of Fossil Energy awarded 13 Carbon Ore, Rare-Earth and Critical Minerals (CORE-CM) programs as part of the CORE-CM Initiative, designed to develop the technology and upstream and midstream supply chains to extract rare-earth elements (REEs) and CMs from the nation’s coal supplies. The intent is to catalyze regional economic growth and job creation while strengthening the use of domestic resources. The Williston Basin CORE-CM Program aims to drive the expansion and transformation of coal and coal-based resource usage within the Williston Basin to produce REEs, CMs, and nonfuel carbon-based products (CBPs). The work constitutes Phase 1 of a long-term program. The objectives of Phase 1 are to identify the existing knowledge base and gaps and to develop a series of assessments/plans, including an initial basinal assessment; a characterization and data acquisition plan; a waste stream reuse plan; a basinal strategies assessment for infrastructure, industries, and business; a technology assessment, development, and field-testing plan; a technology innovation center plan(s); and a stakeholder outreach and education plan. Originally, the CORE-CM programs were to do little if any sample collection as part of the Phase 1 program. As part of the ongoing interactions with DOE regarding the progress of the 13 CORE-CM programs, DOE has decided to extend Phase 1 with additional funding to allow all of the Phase 1 CORE-CM programs to collect and characterize more samples that have potential to be sources of REEs and CMs. The goal of this project is to initiate the development of a new industry in the resource-rich Williston Basin that will generate economic growth and job creation through a coalition team of private industry; university; and state, local, and federal government entities. A coalition team of nearly 30 private industry; university; and state, local, and federal government partners was formed for this program. Partners include the University of North Dakota Energy & Environmental Research Center, Institute for Energy Studies, and Nistler College; North Dakota State University; Montana Tech University; Pacific Northwest National Laboratory; Critical Materials Institute; North Dakota Geological Survey; South Dakota Geological Survey; U.S. Geological Survey; North American Coal Corporation; BNI Energy; Basin Electric Power Cooperative; Minnkota Power Cooperative; and many more.

Kay, John P.↗
DOE OSTI2026

Historic climate, cosmogenic 10Be, denudation-rate, and geospatial datasets from the Pikes Peak region, Colorado, USA

This data package contains geographic information system (GIS) layers and tabular datasets associated with the study of elevation-dependent denudation rates on Pikes Peak in the Front Range of the Rocky Mountains, Colorado, USA. The package includes GIS layers used to produce the study-area map, including sample locations, sample watershed boundaries, the Pikes Peak batholith, Pleistocene glacier extent, weather station locations, and elevation and hillshade rasters, together with comma-separated value (CSV) tables and matching CSV data dictionaries. These mapped layers provide the geographic framework for interpreting denudation patterns across the Pikes Peak region and for relating sample locations to watershed geometry, bedrock setting, glacial history, and nearby climate stations. The first group of tables reports climate and geospatial context for the study area. These files include station-based temperature and precipitation data used to characterize elevational gradients in mean annual climate and monthly climate seasonality, sample locations, denudation-rate and topographic metrics, fixed frost-cracking model parameters, frost-cracking intensity and precipitation-frequency metrics, and stream-power inversion results. Together, these data provide the basis for evaluating how denudation varies with elevation, climate, and landscape form across sampled catchments on Pikes Peak. The second group of tables reports cosmogenic nuclide and erosion-model results used in the denudation analysis. Included files contain accelerator mass spectrometry (AMS) measurements for in situ-produced cosmogenic beryllium-10 (10Be), including sample identifiers, measured 10Be:9Be ratios, analytical uncertainties, carrier mass, quartz mass, blank corrections, blank-group statistics, and calculated 10Be concentrations and uncertainties. Additional tables summarize stream-power-law inversion results for sampled catchments, including optimized model parameters, predicted erosion rates, residual metrics, channel-pixel counts, and convergence status, as well as regression equations and summary statistics used to evaluate relationships among elevation, climate, frost cracking, precipitation forcing, and denudation rate. The package contains GIS files, comma-separated value files (.csv), Microsoft Excel files (.xlsx), CSV data dictionaries, a file-level metadata table, and a readme text file.

10Be cosmogenic nuclides↗
DOE OSTI2021

Mol* Viewer: modern web app for 3D visualization and analysis of large biomolecular structures

Abstract Large biomolecular structures are being determined experimentally on a daily basis using established techniques such as crystallography and electron microscopy. In addition, emerging integrative or hybrid methods (I/HM) are producing structural models of huge macromolecular machines and assemblies, sometimes containing 100s of millions of non-hydrogen atoms. The performance requirements for visualization and analysis tools delivering these data are increasing rapidly. Significant progress in developing online, web-native three-dimensional (3D) visualization tools was previously accomplished with the introduction of the LiteMol suite and NGL Viewers. Thereafter, Mol* development was jointly initiated by PDBe and RCSB PDB to combine and build on the strengths of LiteMol (developed by PDBe) and NGL (developed by RCSB PDB). The web-native Mol* Viewer enables 3D visualization and streaming of macromolecular coordinate and experimental data, together with capabilities for displaying structure quality, functional, or biological context annotations. High-performance graphics and data management allows users to simultaneously visualise up to hundreds of (superimposed) protein structures, stream molecular dynamics simulation trajectories, render cell-level models, or display huge I/HM structures. It is the primary 3D structure viewer used by PDBe and RCSB PDB. It can be easily integrated into third-party services. Mol* Viewer is open source and freely available at https://molstar.org/.

Sehnal, David↗
DOE OSTI2020

Dynamics of Gas Bubbles From a Submarine Hydrocarbon Seep Within the Hydrate Stability Zone

We validate a new model for mass transfer and bubble transport for natural seeps on the continental margins using an integrated observation of a seep at 883 m in the Northern Gulf of Mexico. In the model, mass transfer is assumed to transition from clean to dirty bubble mass transfer rates following a characteristic hydrate formation time that depends on the initial bubble surface area and the hydrate subcooling. We show that buoyancy-induced upwelling is negligible for the bubble stream. We initialize the model using precise data observed at the seafloor and validate the model results to optical and acoustic measurements within the bubble stream. The model accurately predicts the acoustic attenuation of the bubble flare by lateral spreading and bubble shrinkage throughout the water column. In conclusion, the model shows that up to 99.4% by mass of the released gases from this seep dissolves into the ocean within the hydrate stability zone.

03 NATURAL GAS↗
DOE OSTI2020

In situ compression artifact removal in scientific data using deep transfer learning and experience replay

The massive amount of data produced during simulation on high-performance computers has grown exponentially over the past decade, exacerbating the need for streaming compression and decompression methods for efficient storage and transfer of this data---key to realizing the full potential of large-scale computational science. Lossy compression approaches such as JPEG when applied to scientific simulation data realized as a stream of images can achieve good compression rates but at the cost of introducing compression artifacts and loss of information. This paper develops a unified framework for in situ compression artifact removal in which the fully convolutional neural network architectures are combined with scalable training, transfer learning, and experience replay to achieve superior accuracy and efficiency while significantly decreasing the storage footprint as compared with the traditional optimization-based approaches. We demonstrate the proposed approach and compare it with compressed sensing postprocessing and other baseline deep learning models using climate simulations and nuclear reactor simulations, both of which are driven by hyperbolic partial differential equations. Our approach when applied to remove the compression artifacts on the JPEG-compressed nuclear reactor simulation data (using a transfer-trained model that was pretrained on the climate simulation data and updated incrementally as the nuclear reactor simulation progressed), achieved a significant improvement---mean peak signal-to-noise ratio of 42.438 as compared with 27.725 obtained with the compressed sensing approach.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Radon Isotopes and Stable Water Isotopes from Coal Creek Watershed, Colorado (2021)

The radon isotope and stable water isotope data for Coal Creek Watershed, Colorado, consists of d2H, d18O, and 222Rn values from samples collected at 8 stream location along Coal Creek, samples from 7 groundwater springs within the watershed, and precipitation isotope samples collected by Next Generation Water Observing System (NGWOS) from a collector within the watershed. All stream and spring samples were collected between June and October, 2021, and precipitation isotope samples were collected between November 2020 and September 2021. These data were collected to evaluate how groundwater contributions to Coal Creek originating from a fractured hillslope and alluvial fan respond to summer monsoon rains and seasonal drying. Understanding of groundwater-surface water interactions in montane systems in critical for the future of water availability in the Western US as groundwater contributions are expected to become more important for sustaining summer stream flows. This data package contains: (1) a csv of all radon samples; (2) a csv of all stream and spring isotope samples; (3) a csv of precipitation isotope samples; and (4) a csv of locations for each sampling site. The dataset additionally includes a file-level metadata (flmd.csv) file that lists each file contained in the dataset with associated metadata; and a data dictionary (dd.csv) file that contains column/row headers used throughout the files along with a definition, units, and data type.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Hydrologic Model Data for the East Fork Poplar Creek Watershed Simulated with the Advanced Terrestrial Simulator (ATS): Streamflow and Network Expansion–Contraction Dynamics

This dataset supports hydrologic modeling and stream network expansion–contraction analysis for the East Fork Poplar Creek (EFPC) Watershed in Tennessee. It includes a Jupyter notebook for model setup, model configuration files, simulation outputs, and derived products used to evaluate model performance and investigate stream dynamics under varying hydrologic conditions. The dataset was generated using the Watershed Workflow Python package and the Advanced Terrestrial Simulator (ATS), enabling integrated surface–subsurface hydrologic simulations using a stream-aligned mesh. Outputs include high-resolution time series of streamflow, active network length, water table depth, and related hydrologic variables. Also included are spatially explicit stream persistency indices and classifications of reaches as perennial or non-perennial. These data facilitate reproducibility and support further research on stream intermittency and variability in network extent.The model data archive is organized in following directories:1) model_setup_inputsContains the Watershed Workflow Jupyter notebooks (accessed through any open source code editor), selected input datasets, and resulting ATS input files, including XML files (access through any open source code editor), computational mesh (.exo files can be viewed using Paraview), and meteorological forcing files (.h5 files can be accessed through h5py python package and HDFView open source software). 2) model_outputsIncludes ATS simulation outputs relevant to this study. Time series of spatially integrated or averaged variables (e.g., streamflow, water table depth) are provided as CSV files. Select spatial fields (e.g., ponded depth and water table depth) are saved as pickled Python objects to reduce file size, and can be accessed through pickle package in Python. Key geometry objects from Watershed Workflow—such as the surface mesh and river tree—are also included to support analysis of streamflow persistency and expansion–contraction dynamics. These files can also be accessed through Watershed Workflow Python package.3) model_evaluationProvides observed streamflow time series and field survey-based flow regime classifications used to evaluate model performance. Jupyter notebooks for processing ATS outputs and comparing model predictions with observations to build confidence in the model prior to scientific analysis are also included.4) Q_L_relationshipsContains workflows for generating time series of discharge, active network length, and related hydrologic variables used in the stream network expansion–contraction analysis. Includes routines for delineating baseflow-dominated periods. For each catchment, notebooks and processed data (as pickled DataFrames accessed through Pandas Python package) are provided. 5) figure_scriptsProvides the Jupyter notebooks used to generate the figures presented in the paper.

54 ENVIRONMENTAL SCIENCES↗