SEARCH · Engineering Papers
Results for “Data Processing”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Internet of Things Data Characterization Process: Pattern of Life Behavioral Data Study
The HoneyBee™ TARDIS LDRD team completed a data scoping study that identified the initial processes and procedures to baseline the normal and expected behaviors during operability and interoperability of Internet of Things (IoT) device networks. This research is the initial step in developing a process (or methodology) to inform a much broader information framework incorporating machine learning to determine device pattern-of-life which enables the detection of abnormal IoT behaviors on an individual device, as well as in the context of a larger network.
NSO Processing Scripts (NSO wind loading data set processing scripts) [SWR-23-86]
NREL conducted comprehensive field measurements of the atmospheric turbulent wind conditions and the resulting structural wind loads on parabolic troughs at the Nevada Solar One plant from October 2021 to June, 2023. The measurement set-up included meteorological masts and structural load sensors on four trough rows. Additionally, NREL commissioned a lidar scanning the horizontal plane over the trough field. The high-resolution data set, characterizing the complex flow field and resulting structural loads, is available at https://data.openei.org/submissions/5938. The processing routines published here were used to create the data from the instruments' raw data.
On-the-fly Doppler broadening of unresolved resonance region cross sections
In this paper, two methods for computing temperature-dependent unresolved resonance region cross sections on-the-fly within continuous-energy Monte Carlo neutron transport simulations are presented. The first method calculates Doppler broadened cross sections directly from zero-temperature average resonance parameters. In a simulation, at each event that requires cross section values, a realization of unresolved resonance parameters is generated about the desired energy and temperature-dependent single-level Breit-Wigner resonance cross sections are computed directly via the analytical Ψ-x Doppler integrals. The second method relies on the generation of equiprobable cross section magnitude bands on an energy-temperature mesh. Within a simulation, the bands are sampled and interpolated in energy and temperature to obtain cross section values on-the-fly. Both of the methods, as well as their underlying calculation procedures, are verified numerically in extensive code-to-code comparisons. Energy-dependent pointwise cross sections calculated with the newly-implemented procedures are shown to be in excellent agreement with those calculated by a widely-used nuclear data processing code. Relative differences at or below 0.1% are observed. Integral criticality benchmark results computed with the proposed methods are shown to reproduce those computed with a state-of-the-art processed nuclear data library very well. In simulations of fast spectrum systems which are highly-sensitive to the representation of cross section data in the unresolved region, k-eigenvalue and neutron flux spectra differences of <10 pcm and <1.0% are observed, respectively. The direct method is demonstrated to be well-suited to the calculation of reference solutions — against which results obtained with a discretized representation may be assessed — as a result of its treatment of the energy, temperature, and cross section magnitude variables as continuous. Also, because there is no pre-processed data to store (only temperature-independent average resonance parameters) the direct method is very memory-efficient. Typically, only a few kB of memory are needed to store all required unresolved region data for a single nuclide. However, depending on the details of a particular simulation, performing URR cross section calculations on-the-fly can significantly increase simulation times. Alternatively, the method of interpolating equiprobable probability bands is demonstrated to produce results that are as accurate as the direct reference solutions, to within arbitrary precision, with high computational efficiency in terms of memory requirements and simulation time. Analyses of a fast spectrum system show that interpolation on a coarse energy-temperature mesh can be used to reproduce reference k-eigenvalue results obtained with cross sections calculated continuously in energy and directly at an exact temperature to within <10 pcm. Probability band data on a mesh encompassing the range of temperatures relevant to reactor analysis usually require around 100 kB of memory per nuclide. Finally, relative to the case in which probability table data generated at a single, desired temperature are used, minor increases in simulation times are observed when probability band interpolation is employed.
Identification of Natural Oscillation Modes for Purposes of Seismic Assessment and Monitoring of HPP Dams
The paper puts forward a method for processing data from detailed seismic assessments of HPP dams (dynamic tests). A detailed assessment (hundreds of observation points in dam galleries) is performed with consideration of operating dam equipment and the microseismic noise. It is shown that dynamic oscillation characteristics (natural oscillation frequencies and modes in the main dam axes, the velocities of propagation of elastic waves with given polarization, and so on.) can be determined with sufficient accuracy by using complex transfer functions and pulse characteristics. Monitoring data is processed using data from a detailed assessment, taking account of identified natural oscillation modes and determined ranges of natural frequencies. The spectra of characteristic frequencies thus obtained are used to choose substitution models and estimate the elastic characteristics of the “dam – rock bed” construction system, viz., the modulus of elasticity (the Young modulus), the Poisson ratio, the dam section stiffness with respect to shear, tension and compression and the elastic characteristics of the rock foundation.
Estimating Alarm Thresholds for Process Monitoring Data under Different Assumptions about the Data Generating Mechanism
Process monitoring (PM) for nuclear safeguards sometimes requires estimation of thresholds corresponding to small false alarm rates. Threshold estimation dates to the 1920s with the Shewhart control chart; however, because possible new roles for PM are being evaluated in nuclear safeguards, it is timely to consider modern model selection options in the context of threshold estimation. One of the possible new PM roles involves PM residuals, where a residual is defined as residual = data − prediction. This paper reviews alarm threshold estimation, introduces model selection options, and considers a range of assumptions regarding the data-generating mechanism for PM residuals. Two PM examples from nuclear safeguards are included to motivate the need for alarm threshold estimation. The first example involves mixtures of probability distributions that arise in solution monitoring, which is a common type of PM. The second example involves periodic partial cleanout of in-process inventory, leading to challenging structure in the time series of PM residuals.
Completion processing for data communications instructions
Completion processing of data communications instructions in a distributed computing environment, including receiving, in an active messaging interface (`AMI`) data communications instructions, at least one instruction specifying a callback function; injecting into an injection FIFO buffer of a data communication adapter, an injection descriptor, each slot in the injection FIFO buffer having a corresponding slot in a pending callback list; listing in the pending callback list any callback function specified by an instruction, incrementing a pending callback counter for each listed callback function; transferring payload data as per each injection descriptor, incrementing a transfer counter upon completion of each transfer; determining from counter values whether the pending callback list presently includes callback functions whose data transfers have been completed; calling by the AMI any such callback functions from the pending callback list, decrementing the pending callback counter for each callback function called.
Completion processing for data communications instructions
Completion processing of data communications instructions in a distributed computing environment, including receiving, in an active messaging interface (`AMI`) data communications instructions, at least one instruction specifying a callback function; injecting into an injection FIFO buffer of a data communication adapter, an injection descriptor, each slot in the injection FIFO buffer having a corresponding slot in a pending callback list; listing in the pending callback list any callback function specified by an instruction, incrementing a pending callback counter for each listed callback function; transferring payload data as per each injection descriptor, incrementing a transfer counter upon completion of each transfer; determining from counter values whether the pending callback list presently includes callback functions whose data transfers have been completed; calling by the AMI any such callback functions from the pending callback list, decrementing the pending callback counter for each callback function called.
Event-Based Processing of Neutron Scattering Data
Many of the world's time-of-flight spallation neutrons sources are migrating to the recording of individual neutron events. This provides for new opportunities in data processing, the least of which is to filter the events based on correlating them with logs of sample environment and other ancillary equipment. This paper will describe techniques for processing neutron scattering data acquired in event mode that preserve event information all the way to a final spectrum, including any necessary corrections or normalizations. This results in smaller final errors, while significantly reducing processing time and memory requirements in typical experiments. Results with traditional histogramming techniques will be shown for comparison.
GLBRC Soil Yearlong Incubation 13C-SIP-Lipidomics
Data package for Lipids represent a dynamic, yet stable pool of microbially-derived soil carbon This data is published under a CC0 license. The authors encourage data reuse and request attribution by referencing the below citations for the data packages and associated manuscript. Please cite as: Rempfert KR, Bell SL, Kasanke CP, Kyle JE, Hofmockel KS. 2025. GLBRC Soil Yearlong Incubation 13C-SIP-Lipidomics. [Data Set] PNNL DataHub. doi: Rempfert KR, Bell SL, Kasanke CP, Kyle JE, Hofmockel KS. 2025. MSV000097435: GLBRC soil yearlong incubation 13C-SIP-Lipidomics [Data Set] MassIVE. doi:10.25345/C57659T3K Rempfert KR, Bell SL, Kasanke CP, Kyle JE, Hofmockel KS. 2025. Lipids represent a dynamic, yet stable pool of microbially-derived soil carbon. In Prep This data package consists of compound-specific 13C SIP-lipidomics data from a yearlong tracer incubation experiment designed to investigate microbial lipid persistence in switchgrass bioenergy crop soils. In order to explore how lipid structure may modulate the persistence of C in soil lipids, we leveraged soils from two sites (Michigan - sandy texture, Wisconsin - silty texture) operated by the U.S. Department of Energy-funded Great Lakes Bioenergy Research Center (GLBRC). These sites had comparable climates, identical management practices, but contrasting soil textures, allowing us to assess the variability of lipid accrual or degradation in soils as well as provide insight regarding the degree to which edaphic properties may regulate the retention of soil lipids. Untargeted lipidomics analyses were performed to identify 13C-labeled lipids in the soil microbiome after long-term incubation. Soils were supplemented with 100 micrograms glucose per gram dry soil (99 atom % 13C or natural abundance for paired control) and incubated; samples were collected two months and one year after glucose addition. Lipid extracts (MPLEx) were analyzed by LC-MS/MS and identified using LIQUID. Calculation of isotopic enrichment of lipids was performed by targeted approach using TarMet to quantify lipid isotopologues and IsoCorrectoR to correct for natural abundance isotopes. Contents: Data package contents reported here are the first version and contain downstream analysis files for the raw LC-MS mass spectrometry files (.mzXML) deposited at the MassIVE database repository under accession MSV000097435 (80 experimental runs; 5.85 GB) | MassIVE DOI: 10.25345/C57659T3K. Support files include the additional data download 'Read Me' file containing data descriptor information. Reported data download contents are structured for compliance with project data sharing guidelines, community standards initiatives, and sponsor stakeholder policies supporting FAIR data principles. Data processing software, analysis tools, and data workflows are listed below corresponding to the host repository long-term location. Available Data Downloads (0.3 GB): "GLBRC soil yearlong incubation 13C-SIP-Lipidomics_readme.txt" - 'Read Me' data package content file (txt) "GLBRC_DataPackage_analysis files" - Data processing files (Rmd) and saved intermediate data processing outputs (rds, csv, xlsx) "GLBRC_13C_lipidomics_dataset.xlsx" - processed data in tabular format (xlsx) Linked Software: LIQUID LC-MS Analysis Software | 10.5281/zenodo.6459462 Lipid Mini-On Software Tools | 10.5281/zenodo.1492803 pmartR Omics Statistical Software | 10.5281/zenodo.6108667 xcms (v4.3.3) TarMet (v1.1.1) IsoCorrectoR (1.24.0) Funding Acknowledgments: This research was supported by an Early Career Research Program award funded by the U.S. Department of Energy, Office of Science, Office of Biological and Environmental Research (OBER) Genomic Science program under FWP 68292, FWP 07880 and EMSL Exploratory Research Project 51095. A portion of this work was performed in the William R. Wiley Environmental Molecular Sciences Laboratory, a national scientific user facility sponsored by OBER and located at Pacific Northwest National Laboratory (PNNL). PNNL is a multi-program national laboratory operated by Battelle for the DOE under Contract DE-AC05-76RLO1830.
HERO WEC 2024 Hydraulic Configuration Deployment Data
The following submission includes raw and processed data from the in water deployment of NREL's Hydraulic and Electric Reverse Osmosis Wave Energy Converter (HERO WEC), in the form of parquet files, TDMS files, CSV files, bag files and MATLAB workspaces. This dataset was collected in March 2024 at the Jennette's pier test site in North Carolina. This submission includes the following: - Data description document (HERO WEC FY24 Hydraulic Deployment Data Descriptions.doc) - This document includes detailed descriptions of the type of data and how it was processed and/or calculated. - Processed MATLAB workspace - The processed data is provided in the form of a single MATLAB workspace containing data from the full deployment. This workspace contains data from all sensors down sampled to 10 Hz along with all array Value Added Products (VAPs). - MATLAB visualization scripts - The MATLAB workspaces can be visualized using the file "HERO_WEC_2024_Hydraulic_Config_Data_Viewer.m/mlx". The user simply needs to download the processed MATLAB workspaces, specify the desired start and end times and run this file. Both the .m and .mlx file format has been provided depending on the user's preference. - Summary Data - The fully processed data was used to create a summary data set with averages and important calculations performed on 30-minute intervals to align with the intervals of wave resource data reported from nearby CDIP ocean observing buoys located 20km East of Jennette's pier and 40km Northeast of Jennette's pier. The wave resource data provided in this data set is to be used for reference only due the difference in water depth and proximity to shore between the Jennette's pier test site and the locations of the ocean observing buoys. This data is provided in the Summary Data zip folder, which includes this data set in the form of a MATLAB workspace, parquet file, and excel spreadsheet. - Processed Parquet File - The processed data is provided in the form of a single parquet file containing data from all HERO WEC sensors collected during the full deployment. Data in these files has been down sampled to 10 Hz and all array VAPs are included. - Interim Filtered Data - Raw data from each sensor group partitioned into 30-minute parquet files. These files are outputs from an intermediate stage of data processing and contain the raw data with no Quality Control (QC) or calculations performed in a format that is easier to use than the raw data. - Raw Data - Raw, unprocessed data from this deployment can be found in the Raw Data zip folder. This data is provided in the form of TDMS, CSV, and bag files in the original format output by the MODAQ system. - Python Data Processing Script - This links to an NREL public github repository containing the python script used to go from raw data to fully processed parquet files. Additional documentation on how to use this script is included in the github repository. This data set has been developed by the National Renewable Energy Laboratory, operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. Funding provided by the U.S. Department of Energy Office of Energy Efficiency and Renewable Energy Water Power Technologies Office.
Data Preparation Process for the Buildings Performance Database
The Buildings Performance Database (BPD) includes empirically measured data from a variety of data sources with varying degrees of data quality and data availability. The purpose of the data preparation process is to maintain data quality within the database and to ensure that all database entries have sufficient data for meaningful analysis and for the database API. Data preparation is a systematic process of mapping data into the Building Energy Data Exchange Specification (BEDES), cleansing data using a set of criteria and rules of thumb, and deriving values such as energy totals and dominant asset types. The data preparation process takes the most amount of effort and time therefore most of the cleansing process has been automated. The process also needs to adapt as more data is contributed to the BPD and as building technologies over time. The data preparation process is an essential step between data contributed by providers and data published to the public in the BPD.
Salton Sea Geodatabase for Geothermal Exploration Artificial Intelligence
These files contain the geodatabases related to Salton Sea Geothermal Field. It includes all input and output files used with the Geothermal Exploration Artificial Intelligence. Input and output files are sorted into three categories: raw data, pre-processed data, and analysis (post-processed data). In each of these categories there are six additional types of raster catalogs which are titled Radar, SWIR, Thermal, Geophysics, Geology, and Wells. The files are used with the Geothermal Exploration Artificial Intelligence for the Salton Sea Geothermal Site to identify indicators of blind geothermal systems. The included zip file is a geodatabase to be used with ArcGIS and the tar file is an inclusive database that encompasses the inputs and outputs for the Salton Sea Geothermal Site.
Desert Peak Geodatabase for Geothermal Exploration Artificial Intelligence
These files contain the geodatabases related to the Desert Peak Geothermal Field. It includes all input and output files used in the project. The files include data categories of raw data, pre-processed data, and analysis (post-processed data). In each of these categories there are six additional types of raster catalogs including Radar, SWIR, Thermal, Geophysics, Geology, and Wells. The files for the Desert Peak Geothermal Site are used with the Geothermal Exploration Artificial Intelligence to identify indicators of blind geothermal systems. The included zip file is a geodatabase to be used with ArcGIS and the tar file is an inclusive database that encompasses the inputs and outputs for the Desert Peak Geothermal Field.
Brady Geodatabase for Geothermal Exploration Artificial Intelligence
These files contain the geodatabases related to Brady's Geothermal Field. It includes all input and output files for the Geothermal Exploration Artificial Intelligence. Input and output files are sorted into three categories: raw data, pre-processed data, and analysis (post-processed data). In each of these categories there are six additional types of raster catalogs which are titled Radar, SWIR, Thermal, Geophysics, Geology, and Wells. These inputs and outputs were used with the Geothermal Exploration Artificial Intelligence to identify indicators of blind geothermal systems at the Brady Hot Springs Geothermal Site. The included zip file is a geodatabase to be used with ArcGIS and the tar file is an inclusive database that encompasses the inputs and outputs for the Brady Hot Springs Geothermal Site.
Data for Post-Access Report: Tidal Energy Resource Characterization and Model Validation via the Assessment of Cost-effective Spatially Distributed Current Measurements from microFloat Swarms
This project evaluated how high-resolution, spatially distributed field data can be used to refine and validate site-scale hydrodynamic simulations of tidal channels. Use of such spatially-distributed field observations or site-scale hydrodynamic simulations will be needed for producing accurate predictions of tidal energy production over larger arrays of tidal turbines. The data contained in this archive includes field observations of water velocity from a series of short surveys in Agate Pass taken on August 20th, 2020 using station-keeping ADCP (acoustic doppler current profiler) measurements, drifting downward-looking ADCPs, and microFloats, gathered by researchers at APL-UW under a previous project. It also includes high-resolution hydrodynamic model runs for the site during that time generated under TEAMER support. An explanatory file (.pdf format) describes the survey and field data structure. Processed data is stored in .mat format. Survey summary: start time, number of floats, water level. microFloat data: - Cleaned data for each float in survey, - Time series of position (lat., lon.), depth (relative to surface), and horizontal velocity (u, v) Drifting ADCP data: - Cleaned data for each drifting ADCP in survey - Time series of position (lat., lon.), sample depth (relative to surface), and observed water velocity (u,v,w) Stationary ADCP data: - Cleaned data for each station-keeping data set in survey - Time series of position (lat., lon.), sample depth (relative to surface), and observed water velocity (u,v,w) FVCOM (Finite Volume Coastal Ocean Model) model outputs: Water level and velocity, as well as derived variables (e.g., power density) from the final refined model simulations for the Agate Pass subdomain. Contained in NetCDF and ASCII files for the duration of August 18th and 20th 2020 field surveys. Additional scripts for processing the data and generating the figures in the report are included.
Impacts of processing decisions on TNSL cross sections and their applications
Thermal neutron scattering law (TNSL) data describe low-energy neutrons scattering off of bound materials, and can have a significant impact on modeling any system with slow neutrons, including nuclear reactors. Previous work to introduce TNSL data to neutron transport codes at LLNL focused on COG and TART, with the limitation that these codes require highly specialized data processing and formatting. We have recently increased efforts to process TNSL data with the LLNL nuclear data processing code FUDGE. FUDGE reads and writes the evaluated and processed files using the generalized nuclear database structure (GNDS). This process uncovered some significant difficulties in processing TNSL data, and unearthed assumptions made in current TNSL data processing that we have found inadequate. (authors)
On single-crystal total scattering data reduction and correction protocols for analysis in direct space
Data reduction and correction steps and processed data reproducibility in the emerging single-crystal total-scattering-based technique of three-dimensional differential atomic pair distribution function (3D-ΔPDF) analysis are explored. All steps from sample measurement to data processing are outlined using a crystal of CuIr 2 S 4 as an example, studied in a setup equipped with a high-energy X-ray beam and a flat-panel area detector. Computational overhead as pertains to data sampling and the associated data-processing steps is also discussed. Various aspects of the final 3D-ΔPDF reproducibility are explicitly tested by varying the data-processing order and included steps, and by carrying out a crystal-to-crystal data comparison. Situations in which the 3D-ΔPDF is robust are identified, and caution against a few particular cases which can lead to inconsistent 3D-ΔPDFs is noted. Although not all the approaches applied herein will be valid across all systems, and a more in-depth analysis of some of the effects of the data-processing steps may still needed, the methods collected herein represent the start of a more systematic discussion about data processing and corrections in this field.