Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data streams”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18

Managing high-bandwidth real-time data storage

There exist certain systems which generate real-time data at high bandwidth, but do not necessarily require the long-term retention of that data in normal conditions. In some cases, the data may not actually be useful, and in others, there may be too much data to permanently retain in long-term storage whether it is useful or not. However, certain portions of the data may be identified as being vitally important from time to time, and must therefore be retained for further analysis or permanent storage without interrupting the ongoing collection of new data. We have developed a system, Mahanaxar, intended to address this problem. It provides quality of service guarantees for incoming real-time data streams and simultaneous access to already-recorded data on a best-effort basis utilizing any spare bandwidth. It has built in mechanisms for reliability and indexing, can scale upwards to meet increasing bandwidth requirements, and handles both small and large data elements equally well. We will show that a prototype version of this system provides better performance than a flat file (traditional filesystem) based version, particularly with regard to quality of service guarantees and hard real-time requirements.

99 GENERAL AND MISCELLANEOUS↗

Streaming Compression of Scientific Data via Weak-SINDy

Here, in this paper, a streaming weak-SINDy algorithm is developed specifically for compressing streaming scientific data. The production of scientific data, either via simulation or experiments, is undergoing a stage of exponential growth, which makes data compression important and often necessary for storing and utilizing large scientific data sets. As opposed to classical “offline” compression algorithms that perform compression on a readily available data set, streaming compression algorithms compress data “online” while the data generated from simulation or experiments is still flowing through the system. This feature makes streaming compression algorithms well suited for scientific data compression, where storing the full data set offline is often infeasible. This work proposes a new streaming compression algorithm, streaming weak-SINDy, which takes advantage of the underlying data characteristics during compression. The streaming weak-SINDy algorithm constructs feature matrices and target vectors in the online stage via a streaming integration method in a memory efficient manner. The feature matrices and target vectors are then used in the offline stage to build a model through a regression process that aims to recover equations that govern the evolution of the data. For compressing high-dimensional streaming data, we adopt a streaming proper orthogonal decomposition (POD) process to reduce the data dimension and then use the streaming weak-SINDy algorithm to compress the temporal data of the POD expansion. We propose modifications to the streaming weak-SINDy algorithm to accommodate the dynamically updated POD basis. By combining the built model from the streaming weak-SINDy algorithm and a small amount of data samples, the full data flow could be reconstructed accurately at a low memory cost, as shown in the numerical tests.

97 MATHEMATICS AND COMPUTING↗

Machine Learning and Social Media to Mine and Disseminate Big Scientific Data

One of the challenges in supplying the communities with wider access to scientific databases is the need for knowledge of database languages like Structured Query Language (SQL). Although the SQL language has been published in many forms, not everybody is able to write SQL queries. Another challenge is that it might not be practical to make the public aware of the structure of databases. There is a need for novice users to query relational databases using their natural language. To solve this problem, many natural language interfaces to structured databases have been developed. The goal is to provide a more intuitive method for generating database queries and delivering responses. Through social media, which makes it possible to interact with a wide section of the population, and with the help of natural language processing, researchers at the Atmospheric Radiation Measurement (ARM) Data Center at Oak Ridge National Laboratory (ORNL) have developed a concept to enable easy search and retrieval of data from several environmental data centers for the scientific community through social media.Using a machine learning framework that maps natural language text to thousands of datasets, instruments, variables, and data streams, the prototype system would allow users to request data through Twitter and receive a link (via tweet) to applicable data results on the project's search catalog tailored to their key words. This automated identification of relevant data from various petascale archives at ORNL could increase convenience, access, and use of the project's data by the broader community. In this paper we discuss how some data-intensive projects at ORNL are using innovative ways to help in data discovery.

Devarakonda, Ranjeet↗

In Situ Transmission Electron Microscopy: Signal processing challenges and examples

Transmission electron microscopy (TEM) is a powerful tool for imaging material structure and characterizing material chemistry. Recent advances in data collection technology for TEM have enabled high-volume and high-resolution data collection at a microsecond frame rate. Here, taking advantage of these advances in data collection rates requires the development and application of data processing tools, including image analysis, feature extraction, and streaming data processing techniques. In this article, we highlight a few areas in materials science that have benefited from combining signal processing and statistical analysis with data collection capabilities in TEM and present a future outlook on opportunities of integrating signal processing with automated TEM data analysis.

36 MATERIALS SCIENCE↗

StreamWorks - A system for Dynamic Graph Search

Acting on time-critical events by processing ever growing social media, news or cyber data streams is a major technical challenge. Many of these data sources can be modeled as multi-relational graphs. Mining and searching for subgraph patterns in a continuous setting requires an efficient approach to incremental graph search. The goal of our work is to enable real-time search capabilities for graph databases. This demonstration will present a dynamic graph query system that leverages the structural and semantic characteristics of the underlying multi-relational graph.

graph search, graph query↗

The LAKE model input dataset for three Arctic lakes

This dataset contains meteorological data collected for three Arctic lakes and compiled to satisfy input requirements of the LAKE 2.0 model. The dataset was generated to act as a benchmarking dataset for future model-data inter-comparisons. The LAKE 2.0 model simulates temperatures within the water later and the sedimentary layer of a lake. The LAKE2.0. is an open-source code and available to download via this weblike http://tesla.parallel.ru/Viktor/LAKE/-/wikis/LAKE-model (last visit July 14, 2021). The meteorological data are required to simulate the surface energy balance at the surface of a lake. This dataset includes a compilation of the meteorological data pulled from multiple data streams, including National Oceanic and Atmospheric Administration (NOAA) climate data, Circumarctic Lakes Observation Network (CALON) data, and the United States Geological Survey (USGS) data. The data were compiled for three Arctic lakes: FoxDen (66.55877, -164.45670), Atqasuk (70.452497, -156.951984), and Toolik (68.63150, -149.60740). Each meteorological data is in comma-delimited format (file extension ‘.dat’) and includes eight columns: Temperature [K], Pressure [Pa], longwave downward radiation [W/m2], shortwave downward radiation [W/m2], “U” wind speed [m/s], ”V” wind speed [m/s], humidity [kg/kg], precipitation [m/s]. In addition to the meteorological data file, we included setup and driver files. The Toolik lake is the deepest out of three lakes and has inflowing and outflowing groundwater data. InflowOutflowREADME.txt has more information about inflow and outflow flies. The other two lakes are much shallower and modeled as a closed system (i.e. no water inflow or outflow).

54 ENVIRONMENTAL SCIENCES↗

AGR-2 Data Qualification Interim Report

Projects for the very high temperature reactor (VHTR) Technology Development Office program provide data in support of Nuclear Regulatory Commission licensing of the VHTR. Fuel and materials to be used in the reactor are tested and characterized to quantify performance in high temperature and high fluence environments. The VHTR program established the NGNP Data Management and Analysis System (NDMAS) to manage and document VHTR data qualification, for storage of the data in a readily accessible electronic form, and to assist in the analysis and presentation of the data. This document gives the status of NDMAS processing and qualification of data associated with the initial reactor cycle (147A) of the second Advanced Gas Reactor (AGR-2) experiment which began on June 21, 2010. Because it is early in the AGR-2 experiment, data from only two AGR-2 data streams are reported on: Fuel Fabrication and Fuel Irradiation data. As of August 1, 2010, approximately 311,000 irradiation data records have been stored in NDMAS, and qualification tests are in progress. Preliminary information indicates that TC 2 in Capsule 2 failed prior to start of the experiment, and NDMAS testing has thus far identified only two invalid data values from the METSO data collection system Data from the Fission Product Monitoring System (FPMS) are not currently processed until after reactor cycle shutdown and have not yet been received. A description of the ATR operating conditions data associated with the AGR-2 experiment (e.g., power levels) are summarized in the AGR-1 data qualification report (INL/EXT-09-16460). Since ATR data are collected under ATR program data quality requirements (i.e., outside the VHTR program), the NGNP program and NDMAS do not take additional actions to qualify these data other than NDMAS capture testing. Data qualification of graphite characterization data collected under the Graphite Technology Development Project is reported in a separate status report (Hull 2010).

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Dendrometer data at The Morton Arboretum Forestry Plots 2019-2023

We are collecting long-term dendrometer data at The Morton Arboretum to determine seasonal growth patterns in trees. This data on stem growth patterns will eventually be integrated with other ongoing data streams to paint a broader picture of plant phenology. This data package contains the outputs from three types of dendrometer devices: ICT band dendrometer data is in the "ICT_data2020_2023.csv" file, TreeHugger band dendrometer data is in the "TreeHugger_data2019_2020.csv" file, and TOMST point dendrometer data is in the "TOMST_data2021_2023.csv" file. The TreeHugger and TOMST files contain corrected and raw uncorrected values, whereas the ICT data file contains only raw data. Initial tree size data at device installation is located in the "Initial_Tree_Size.csv" file. Additional information on units are contained within each data file's respective data dictionary, and the location metadata file contains the geographic locations of the 23 forestry plots at The Morton Arboretum as well as the measured tree species at each location.

54 ENVIRONMENTAL SCIENCES↗

Process Design and Techno-Economic Analysis of the Modular Staged Pressurized Oxy-Combustion (SPOC) Power Plant for Biomass

This work describes the process design and techno-economic analysis (TEA) of the modular SPOC power plant for biomass firing and coal-biomass co-firing. Two Rankine cycles were considered: a supercritical steam cycle (242 bar, 593°C, 593°C) with 550 MWe net output and a subcritical cycle (166 bar, 566°C, 566°C) with 200 MWe net output. For both cases, 95% carbon capture was modeled, and hybrid poplar biomass was chosen to generate carbon-negative power. In addition, the supercritical 500 MWe case included a 25% biomass co-firing (carbon neutral) case. For both cycles, a 100% Powder River Basin coal firing case was used for comparison purposes. In the SPOC process, oxygen is produced via a cryogenic air separation unit (ASU) and the heat generated from the compression of air is integrated into the steam cycle and utilized for boiler feed water pre-heating. Unique to the SPOC process, the boilers are pressurized and arranged in a series-parallel configuration, with minimized flue gas recirculation. The flue gas is cooled and scrubbed in the direct-contact cooler (DCC) column, and the moisture in the flue gas is condensed, leaving the bottom of the DCC at a sufficiently high temperature such that it can be used for boiler feed water pre-heating, improving plant thermal efficiency. Following drying and purification, CO2 in the flue gas is at the purity required for storage or utilization. The performance data were obtained from process modelling via Aspen Plus®. The stream data from Aspen Plus® were used as an input for the AACE Class 5 cost study. Ultimately, the capital costs, Levelized Cost of Electricity (LCOE), and cost of CO2 captured and avoided were obtained. The HHV efficiency of the carbon negative 550 MWe supercritical SPOC case (34.8%) was clearly above those reported by NETL for the BECCS baseline cases of supercritical pulverized coal with capture (B12B, 31.5%) and the 49% biomass co-firing case with capture (PA3, 29.2%). The HHV efficiency of the carbon-negative subcritical plant is also higher than the subcritical baseline PC plant with capture (case B11B.95) presented by NETL (32% vs 29.7%). The LCOE for the SPOC 100% biomass case was similar to the LCOE for the BECCS 49% biomass with carbon capture case ($147/MWh), and the SPOC carbon neutral case LCOE was lower ($110/MWh) than the cost for the NETL baseline SC coal firing case with 90% carbon capture ($114/MWh).

Magalhaes, Duarte↗

California - Leosphere Windcube 866 (120), Humboldt / Reviewed Data

The purpose of this dataset is to provide filtered, averaged lidar data and standardize the data format of various data streams from the buoy into NetCDF. The attached Lidar Buoy Data Dictionary provides further details on the various instruments mounted on the buoys, parameters measured by each instrument, and the frequency of data collection.

17 WIND ENERGY↗

Buoy - California - Wind Sentinel (130), Morro Bay - Processed Data

The purpose of the dataset is to provide preliminary filtered, averaged buoy data and standardize the data format of various data streams from the buoy into NetCDF. The attached Lidar Buoy Data Dictionary provides further details on the various instruments mounted on the buoys, parameters measured by each instrument, and the frequency of data collection.

17 WIND ENERGY↗

Lidar - California - Leosphere Windcube 866 (130), Morro Bay - Processed Data

The purpose of this dataset is to provide preliminary filtered, averaged lidar data and standardize the data format of various data streams from the buoy into NetCDF. The attached Lidar Buoy Data Dictionary provides further details on the various instruments mounted on the buoys, parameters measured by each instrument, and the frequency of data collection.

17 WIND ENERGY↗

California - Leosphere Windcube 866 (130), Morro Bay / Reviewed Data

The purpose of this dataset is to provide filtered, averaged lidar data and standardize the data format of various data streams from the buoy into NetCDF. The attached Lidar Buoy Data Dictionary provides further details on the various instruments mounted on the buoys, parameters measured by each instrument, and the frequency of data collection.

17 WIND ENERGY↗

Beam Stability R&D for the APS MBA Upgrade

Beam diagnostics required for the APS Multi-bend acromat (MBA) are driven by ambitious beam stability requirements. The major AC stability challenge is to correct rms beam motion to 10% the rms beam size at the insertion device source points from0.01 to 1000 Hz. The vertical plane represents the biggest challenge forAC stability, which is required to be 400 nm rms for a 4-micron vertical beam size. In addition to AC stability, long-term drift over a period of seven days is required to be 1 micron or less. Major diagnostics R&D components include improved rf beam position processing using commercially available FPGA-based BPM processors, new X-ray beam position monitors based on hard X-ray fluorescence from copper and Compton scattering off diamond, mechanical motion sensing to detect and correct long-term vacuum chamber drift, a new feedback system featuring a tenfold increase in sampling rate, and a several-fold increase in the number of fast correctors and BPMs in the feedback algorithm. Feedback system development represents a major effort, and we are pursuing development of a novel algorithm that integrates orbit correction for both slow and fast correctors down to DC simultaneously. Finally, a new data acquisition system (DAQ) is being developed to simultaneously acquire streaming data from all diagnostics as well as the feedback processors for commissioning and fault diagnosis. Results of studies and the design effort are reported.

APS MBA Upgrade↗

Characterization of throughput on the AXI DMA bus for burst data transfer over Ethernet

cThe Xilinx AXI Direct Memory Access (AXI DMA) module is an efficient solution for medium-speed data transfer in Xilinx SoC FPGAs, supporting data rates greater than 1000 Gbps even in very suboptimal operating modes. It facilitates direct transfer of AXI stream data into processor memory without constant software intervention, which reduces overhead and ensures consistent data logging. By utilizing the FPGA's available memory, large circular buffers (1-5 GiB) are used to buffer data and accommodate network limitations, enabling high-rate data bursts. In this study, we measured the performance of AXI DMA under conditions simulating its lowest practical data transfer speeds. The Arbitrary Length Data Sender was used to transmit AXI stream packets at 32-bit width and 100 MHz frequency, a narrow width and slow speed. Results show that the AXI DMA can transfer up to 3192.76 Mbps with large packet sizes but experiences reduced performance for smaller packets, as low as 2.6 Mbps for 4-byte packets. For Ethernet-limited applications, packet sizes between 8,000 and 16,000 bytes provided optimal transfer speeds of 874 to 1600 Mbps. These findings suggest that the AXI DMA is not the limiting factor in systems where packet sizes exceed 8,000 bytes.

43 PARTICLE ACCELERATORS↗

LANL CSSE L2: Case Study of In Situ Data Analysis in ASC Integrated Codes [Slides]

The massive, complex datasets generated by scientific simulations on next-generation computer architectures at extreme scale pose new challenges for data management and scientific understanding. A critical element in meeting these new challenges is in situ data analysis where information is reduced and analyzed as close to the point of generation as possible, for example, at run time or as the data stream to longer term storage. This milestone will demonstrate in situ data analysis in xRAGE, an ASC Eulerian Radiation Hydrodynamics code. We will work with the Eulerian Applications Project to identify target problems that will exercise physics of interest to the program, and we will work closely with users to maximize the achieved usability and productivity improvements. We will investigate enhancements to the in situ process that enable and improve techniques of data reduction and data triage in addition to improving in situ performance by using portable acceleration technology. We will report on their viability and performance in xRAGE or another ASC code. These products will provide ASC IC developers with reference codes and tools that will benefit DSW by advancing the technology for predictive science at extreme scales.

97 MATHEMATICS AND COMPUTING↗

AGR 5/6/7 Data Qualification Report for ATR Cycles 162B through 168A

This report provides the qualification status of experimental data for the Advanced Gas Reactor (AGR) 5/6/7 fuel irradiation. AGR-5/6/7 was conducted in the Advanced Test Reactor (ATR) at Idaho National Laboratory (INL) in support of development and qualification of tri-structural isotropic (TRISO) low-enriched fuel for use in high temperature gas-cooled reactors. The objectives of the AGR-5/6/7 experiments are to: (i) irradiate reference-design fuel particles to support fuel qualification, (ii) establish operating margins for the fuel beyond normal operating conditions, and (iii) provide irradiated-fuel performance data and irradiated-fuel samples for post-irradiation examination (PIE) and safety testing. The test train contains five separate capsules that were independently controlled and monitored. Each capsule contains multiple 12.51-mm-long compacts filled with low enriched uranium carbide/oxide (UCO) TRISO fuel particles. The primary objective of the AGR-5/6 test (Capsules 1, 2, 4, and 5) is to verify successful performance of the reference-design fuel under normal operating conditions. The AGR-7 test (Capsule 3) was designed to explore fuel performance at higher temperatures to demonstrate the capability of the fuel to withstand conditions beyond normal operating conditions in support of plant design and licensing. AGR 5/6/7 will also provide irradiated-fuel performance data on fission-gas release from failed particles during irradiation. The AGR-5/6/7 capsules were irradiated in the ATR northeast flux trap location. The experiment began on February 16, 2018 and ended on July 22, 2020, spanning nine ATR cycles over two and a half years. Thus, the AGR-5/6/7 fuel compacts were irradiated for a total of 360.9 effective full power days. The AGR 5/6/7 experiment was able to remain in the reactor core during all three Powered Axial Locator Mechanism (PALM) cycles (163A, 165A, and 167A) without overheating its fuel compacts. This report includes irradiation monitoring data from nine ATR Cycles: 162B, 163A, 164A, 164B, 165A, 166A, 166B, 167A, and 168A, as stored in the Nuclear Data Management and Analysis System (NDMAS). During irradiation, data records consisted of instantaneous measurements recorded every minute and provided by text files automatically every 2 hours. The AGR 5/6/7 data streams addressed in this report include thermocouple (TC) temperatures, sweep gas data (flow rates [capsule inlet, outlet, and downstream at detector], pressure, and moisture content), and Fission Product Monitoring System (FPMS) data (release rates and release to birth rate ratios [R/Bs]) for each of the five capsules. A total of 94,989,908 TC temperature and sweep gas data records were received and processed by NDMAS for AGR 5/6/7 irradiation. Of these records, 41,593,387 (or 43.7% of the total) met data collection and accuracy requirements and are labeled as Qualified. A total of 57,746,693 TC temperature readings were captured from 54 installed TCs. Among them, 10,034,676 TC temperature records (only 17.4%) were Qualified and 47,701,371 TC temperatures (or 82.6%) are Failed due to 48 TC failures (63.5%) and due to missing values (19.1%). To assess performance of the operational TCs, analysis of daily correlations between TCs found no evidence of virtual junction failure for any TCs. Analyses on control charts of TC temperature differences revealed trending in TC readings for TC2, 4, 5, and 13 in Capsule 3, but there is no conclusive indication of TC drift failure that caused those trends. Therefore, TC control charts are not used to disqualify TC data, but only for users’ consideration. For sweep gas flow rates, a total of 31,519,747 gas flow records (84.4%) are Qualified for use for AGR-5/6/7 experiment; 5,723,468 gas flow records (15.4%) are Failed due mostly to missing values; and 74,641 high sweep gas flow rates (0.2 %) are Trend. A large number of Failed missing TC temperature and gas flow values were caused by an error in the data output script that outputted a ‘NULL’ value when values were unchanged. This problem was fixed during the outage of Cycle 166B, which led to a substantially decreased number of missing values during the last three cycles. Nonetheless, a large amount of non-missing data remained because of the high data acquisition frequency (1-minute) and still provided sufficient data to effectively monitor the experiment as designed. For FPMS data, NDMAS received and processed fission product release and R/B data for nine ATR cycles, when ATR core reached full power during AGR 5/6/7 irradiation. These data consist of 110,388 release rate records and 110,388 R/B records for the twelve radionuclides (Kr 85m, Kr 87, Kr 88, Kr 89, Kr 90, Xe 131m, Xe 133, Xe 135, Xe 135m, Xe 137, Xe 138, and Xe 139) for each of the five capsules. Equivalent numbers of uncertainty records associated the release rates and R/B values were provided. To date, qualification status of the FPMS data stored in the NDMAS dat

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

System and method for wave prediction

A method and system for prediction of wave properties include collecting time series data streams from one or more wave measurement devices and processing the data using a wave-prediction algorithm to identify the frequency components of the data and compute wave parameters. The wave-field is propagated in space and time to predict wave height, speed, and velocity at a target location. A sliding window approach is used to continuously update the prediction in real-time.

Previsic, Mirko↗