SEARCH · Engineering Papers

Results for “Raw Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 613 records · Page 34

All sources NASA NTRS DOE OSTI

DOE OSTI2025

LCLS RF Station Phase Anomaly Candidate Dataset

A public anomaly detection dataset constructed from RF station faults for phase at SLAC's LCLS (Linac Coherent Light Source). We have compiled a dataset of the RF station diagnostic phase data and the beam-position monitor (BPM) signals, alongside the hand labels, for a labeled study period. The dataset consists of two HDF5 files (one for train and one for test) containing the raw data, two CSV files containing information about the candidates. The CSV file for the test dataset also contains the label.

Liang, Jia [Stanford Univ., CA (United States). In↗

DOE OSTI2024

Investigation of fast and efficient lossless compression algorithms for macromolecular crystallography experiments

Structural biology experiments benefit significantly from state-of-the-art synchrotron data collection. One can acquire macromolecular crystallography (MX) diffraction data on large-area photon-counting pixel-array detectors at framing rates exceeding 1000 frames per second, using 200 Gbps network connectivity, or higher when available. In extreme cases this represents a raw data throughput of about 25 GB s −1 , which is nearly impossible to deliver at reasonable cost without compression. Our field has used lossless compression for decades to make such data collection manageable. Many MX beamlines are now fitted with DECTRIS Eiger detectors, all of which are delivered with optimized compression algorithms by default, and they perform well with current framing rates and typical diffraction data. However, better lossless compression algorithms have been developed and are now available to the research community. Here one of the latest and most promising lossless compression algorithms is investigated on a variety of diffraction data like those routinely acquired at state-of-the-art MX beamlines.

36 MATERIALS SCIENCE↗

DOE OSTI2022

Modeling Urban Acoustic Noise in the Las Vegas, NV Region

Ambient infrasound noise in quiet, rural environments has been extensively studied and well-characterized through noise models for several decades. More recently, creating noise models for high-noise rural environments has also become an area of active research. However, far less work has been done to create generalized low-frequency noise models for urban areas. The high ambient noise levels expected in cities and other highly populated areas means that these environments are regarded as poor locations for acoustic sensors, and historically, sensor deployment in urban areas were avoided for this reason. However, there are several advantages to placing sensors in urban environments, including convenience of deployment and maintenance, and increasingly, necessity, as more previously rural areas become populated. This study seeks to characterize trends in low-frequency urban noise by creating a background noise model for Las Vegas, NV, using the Las Vegas Infrasound Array (LVIA): a network of eleven infrasound sensors deployed throughout the city. Data included in this study spans from 2019 to 2021 and provides a largely uninterrupted record of noise levels in the city from 0.1–500 Hz, with only minor discontinuities on individual stations. We organize raw data from the LVIA sensors into hourly power spectral density (PSD) averages for each station and select from these PSDs to create frequency distributions for time periods of interest . These frequency distributions are converted into probability density functions (PDFs), which are then used to evaluate variations in frequency and amplitude over daily to seasonal timescale s. In addition to PDFs, the median, 5 th percentile, and 95 th percentile amplitude values are calculated across the entire frequency range. This methodology follows a well-established process for noise model creation.

63 RADIATION, THERMAL, AND OTHER ENVIRON. POLLUTAN↗

DOE OSTI2023

Arctic shrub size and leaf traits, northern Alaska, summer 2017 (Version 2.0)

This data package contains leaf and size trait and environmental data collected from 170 plots of rapidly expanding shrub genera (Alnus, Betula, and Salix) and a widespread sedge (Eriophorum vaginatum) along a latitudinal and climate gradient in northern Alaska. The trait data were collected in summer 2017 and include leaf area, specific leaf area, leaf nitrogen concentration, leaf delta 15N, leaf delta 13C, shrub height, aboveground biomass, and the ratio of root biomass to aboveground biomass. These raw data support a submitted manuscript that examines the intraspecific variations of size, leaf and root traits of shrubs across the graminoid-dominated nutrient-poor arctic tundra and reveals the environmental drivers of deciduous shrub traits in tundra ecosystems (Fraterrigo et al., submitted). Data are presented by site (n=5) and patch (shrub or sedge plot) in separate csv files. The location data are provided in the “plot coordinates” file; all other files contain the data in the file title. Detailed methods are in the submitted manuscript. A companion data package contains root trait data collected simultaneously from the same plots (Fraterrigo and Chen, 2020). See the "Related references" section for more information.2023/07/20 Update: The latest version of this dataset publication is version 2.0. The latest version of the data package was updated to correct the leaf size trait.csv file.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2025

Differentially Private Map Matching (DPMM) v1.0

Human mobility trajectories provide valuable information for developing mobility applications, as they contain diverse and rich information about the users. User mobility data is valuable for various applications such as intelligent transportation systems (ITS), commercial business models, and disease-spread models. However, such spatio-temporal traces may pose a threat to user privacy. GPS trajectories in their raw form are not suitable for transportation studies, as they require matching locations with nearest road links — a process called map-matching. This software implements a differential privacy (DP)-based map-matching algorithm, called DPMM, that generates link-level location trajectories in a privacy-preserving manner to protect users' origin destinations (OD) and travel paths. OD privacy is achieved by injecting Planar Laplace noise to the user OD GPS points. Travel-path privacy is provided with randomized travel path construction using exponential DP mechanism. The injected noise level is selected adaptively, by considering the link density of the location and the functional category of the localized links. For path privacy, our mechanism samples waypoints and selects candidate paths between waypoints. DPMM provides privacy effectively with respect to link density instead of other trajectory samples in the database compared to other privacy mechanisms. Compared to the different baseline models our DP-based privacy model offers closer query responses to the raw data in terms of individual and aggregate trajectory-level statistics with an average at absolute deviation from the baseline for individual statistics on ϵ = 1.0. Beyond individual trajectory statistics, the DPMM outperforms the other benchmark DP-based mechanisms on different aggregate statistics with up to 8x improvement in utility.

Peisert, Sean [Lawrence Berkeley National Laborato↗

DOE OSTI2023

QA/QC-ed Groundwater Level Time Series in PLM-1 and PLM-6 Monitoring Wells, East River, Colorado (2016-2022)

This data set contains QA/QC-ed (Quality Assurance and Quality Control) water level data for the PLM1 and PLM6 wells. PLM1 and PLM6 are location identifiers used by the Watershed Function SFA project for two groundwater monitoring wells along an elevation gradient located along the lower montane life zone of a hillslope near the Pumphouse location at the East River Watershed, Colorado, USA. These wells are used to monitor subsurface water and carbon inventories and fluxes, and to determine the seasonally dependent flow of groundwater under the PLM hillslope. The downslope flow of groundwater in combination with data on groundwater chemistry (see related references) can be used to estimate rates of solute export from the hillslope to the floodplain and river. QA/QC analysis of measured groundwater levels in monitoring wells PLM-1 and PLM-6 included identification and flagging of duplicated values of timestamps, gap filling of missing timestamps and water levels, removal of abnormal/bad and outliers of measured water levels. The QA/QC analysis also tested the application of different QA/QC methods and the development of regular (5-minute, 1-hour, and 1-day) time series datasets, which can serve as a benchmark for testing other QA/QC techniques, and will be applicable for ecohydrological modeling. The package includes a Readme file, one R code file used to perform QA/QC, a series of 8 data csv files (six QA/QC-ed regular time series datasets of varying intervals (5-min, 1-hr, 1-day) and two files with QA/QC flagging of original data), and three files for the reporting format adoption of this dataset (InstallationMethods, file level metadata (flmd), and data dictionary (dd) files).QA/QC-ed data herein were derived from the original/raw data publication available at Williams et al., 2020 (DOI: 10.15485/1818367). For more information about running R code file (10.15485_1866836_QAQC_PLM1_PLM6.R) to reproduce QA/QC output files, see README (QAQC_PLM_readme.docx). This dataset replaces the previously published raw data time series, and is the final groundwater data product for the PLM wells in the East River. Complete metadata information on the PLM1 and PLM6 wells are available in a related dataset on ESS-DIVE: Varadharajan C, et al (2022). https://doi.org/10.15485/1660962. These data products are part of the Watershed Function Scientific Focus Area collection effort to further scientific understanding of biogeochemical dynamics from genome to watershed scales. 2022/09/09 Update: Converted data files using ESS-DIVE’s Hydrological Monitoring Reporting Format. With the adoption of this reporting format, the addition of three new files (v1_20220909_flmd.csv, V1_20220909_dd.csv, and InstallationMethods.csv) were added. The file-level metadata file (v1_20220909_flmd.csv) contains information specific to the files contained within the dataset. The data dictionary file (v1_20220909_dd.csv) contains definitions of column headers and other terms across the dataset. The installation methods file (InstallationMethods.csv) contains a description of methods associated with installation and deployment at PLM1 and PLM6 wells. Additionally, eight data files were re-formatted to follow the reporting format guidance (er_plm1_waterlevel_2016-2020.csv, er_plm1_waterlevel_1-hour_2016-2020.csv, er_plm1_waterlevel_daily_2016-2020.csv, QA_PLM1_Flagging.csv, er_plm6_waterlevel_2016-2020.csv, er_plm6_waterlevel_1-hour_2016-2020.csv, er_plm6_waterlevel_daily_2016-2020.csv, QA_PLM6_Flagging.csv). The major changes to the data files include the addition of header_rows above the data containing metadata about the particular well, units, and sensor description. 2023/01/18 Update: Dataset updated to include additional QA/QC-ed water level data up until 2022-10-12 for ER-PLM1 and 2022-10-13 for ER-PLM6. Reporting format specific files (v2_20230118_flmd.csv, v2_20230118_dd.csv, v2_20230118_InstallationMethods.csv) were updated to reflect the additional data. R code file (QAQC_PLM1_PLM6.R) was added to replace the previously uploaded HTML files to enable execution of the associated code. R code file (QAQC_PLM1_PLM6.R) and ReadMe file (QAQC_PLM_readme.docx) were revised to clarify where original data was retrieved from and to remove local file paths.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2020

Blockchain based Communication Architectures with Applications to Private Security Networks

Existing communication protocols in high consequence security networks are highly centralized. While this naively makes the controls easier to physically secure, external actors require fewer resources to disrupt the system because there are fewer points in the system can be destroyed or interrupted without the entire system failing. We present a solution to this problem using a proof-of-work-based blockchain implementation built on MultiChain. We construct a test-bed network containing two types of data input: visual imagers and microwave sensor information. These data types are ubiquitous in perimeter intrusion detection security systems and allow a realistic representation of a real-world network architecture. The cameras in this system use an object detection algorithm to nd important targets in the scene. The raw data from the camera and the outputs from the detection algorithm are then placed in a transaction on the distributed ledger. Similarly, microwave data is used to detect relevant events and are placed in a transaction. These transactions are then bundled into blocks and broadcast to the rest of the network using the Bitcoin-based MultiChain protocol. We develop five tests to examine the security metrics of our network. We performed the five security metric test using different sized networks from 7 to 39 nodes to determine how the metrics scale with respect to size. We nd that when compared to a centralized architecture our implementation provides a resiliency increase that is expected from a blockchain-based protocol without slowing the system so much that a human operator would notice. Furthermore, our approach is able to detect tampering in real time. Based on these results, we theorize that security networks in general could use a blockchain-based approach in a meaningful way.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

AI-Ready Data Pilot Project Report

The proliferation of artificial intelligence in scientific research has created an urgent need to define "AI-ready data" for researchers and, more importantly, provide resources to help them produce AI-ready data. At Pacific Northwest National Laboratory, we conducted a pilot study with three data scientists evaluating three CSV datasets from different scientific domains, followed by semi-structured interviews capturing assessment practices. Our findings reveal that AI-readiness evaluation is intuition-based, with practitioners asking "How fast can I go from raw data to my machine learning pipeline?" Data scientists consistently prioritized workflow efficiency, human interpretability, and quality stewardship signals. From these insights, we developed a practical evaluation framework comprising data requirements, metadata standards, and validation tests that provides actionable criteria for producing and curating AI-ready datasets, addressing the gap between theoretical understanding and practical implementation.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

fluxfinder: An R Package for Reproducible Calculation and Initial Processing of Greenhouse Gas Fluxes From Static Chamber Measurements

Fluxes of greenhouse gases are a critical component of the earth's natural climate, but anthropogenic emissions have created an imbalance and resulted in global climate change. Quantifying the emission of these gases is vital to our understanding of their sources and sinks, both natural and anthropogenic. The static chamber method, in which a system of interest is enclosed, and gas concentrations are measured over time, is widely used to estimate fluxes of greenhouse gases. With the development of instruments such as infrared gas analyzers (IRGAs) supporting high-frequency concentration data, there is a growing need for open-source workflows to calculate fluxes. Here we present fluxfinder, an R package designed to support reproducible calculations and processing of greenhouse gas fluxes measured with the static chamber method. The package includes raw data file parsing from widely used IRGAs, metadata matching, unit conversion, flux estimations, and initial quality assurance/quality control (QA/QC). Diagnostic graphical plots provide a transparent way to differentiate between measurement issues and nonlinear behavior. The package is also designed to be easily integrated with the gasfluxes package for further fitting of nonlinear concentration-time models, allowing alternative or additional flux QA/QC. The fluxfinder package offers a flexible workflow that is easily adaptable to promote open and reproducible greenhouse gas flux estimations.

Wilson, Stephanie J.↗

DOE OSTI2021

Quantitative analysis method for multi-color microscopy

We developed an analysis framework for quantitatively interpreting results from multi-color fluorescence microscopy data. When the emission signal from fluorescent probes (i.e. from biological microscopy or in single-particle microscopy) is spectrally separated before detection, the resulting signals contain information about the strengths of each probe. We built a model to deconvolve the raw data into quantitative descriptions of either of the wavelength of a single probe or the relative strengths among multiple probes. This approach requires only spectrally separating the image data into two or more channels, as opposed to fully resolving the spectral profile, thus allowing other microscopy techniques to be used simultaneously. Our analysis framework incorporates the response features of the optical and imaging elements used to produce the spectral separation, yielding highly quantitative interpretations of the underlying behaviors that give rise to the data.

Ryan, Duncan↗

DOE OSTI2023

Efforts to enhance reproducibility in a human performance research project

Background: Ensuring the validity of results from funded programs is a critical concern for agencies that sponsor biological research. In recent years, the open science movement has sought to promote reproducibility by encouraging sharing not only of finished manuscripts but also of data and code supporting their findings. While these innovations have lent support to third-party efforts to replicate calculations underlying key results in the scientific literature, fields of inquiry where privacy considerations or other sensitivities preclude the broad distribution of raw data or analysis may require a more targeted approach to promote the quality of research output. Methods: We describe efforts oriented toward this goal that were implemented in one human performance research program, Measuring Biological Aptitude, organized by the Defense Advanced Research Project Agency's Biological Technologies Office. Our team implemented a four-pronged independent verification and validation (IV&V) strategy including 1) a centralized data storage and exchange platform, 2) quality assurance and quality control (QA/QC) of data collection, 3) test and evaluation of performer models, and 4) an archival software and data repository. Results: Our IV&V plan was carried out with assistance from both the funding agency and participating teams of researchers. QA/QC of data acquisition aided in process improvement and the flagging of experimental errors. Holdout validation set tests provided an independent gauge of model performance. Conclusions: In circumstances that do not support a fully open approach to scientific criticism, standing up independent teams to cross-check and validate the results generated by primary investigators can be an important tool to promote reproducibility of results.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2023

SIF and Vegetation Indices in the US Midwestern Agroecosystems, 2016-2021, Carbon Monitoring System (CMS)

This dataset contains half-hourly ground solar-induced chlorophyll fluorescence (SIF) and vegetation indices including NDVI, EVI, Red edge chlorophyll index, green chlorophyll index, and photochemical reflectance index at seven crop sites in Nebraska and Illinois for the period 2016-2021. Four sites were located at Eddy Covariance (EC) tower sites (sites US-Ne2, US-Ne3, US-UiB, and US-UiC), and three sites were located on private farms (sites Reifsteck, Rund, and Reinhart). The sites were either miscanthus, corn-soybean rotation or corn-corn-soybean rotation. The spectral data for SIF retrieval and hyperspectral reflectance for vegetation index calculation were collected by the FluoSpec2 system, installed near planting, and uninstalled after harvest to collect whole growing-season data. Raw nadir SIF at 760 nm from different algorithms (sFLD, 3FLD, iFLD, SFM) are included. SFM_nonlinear and SFM_linear represent the Spectral fitting method (SFM) with the assumption that fluorescence and reflectance change with wavelength non-linearly and linearly, respectively. Additional data include two SIF correction factors including calibration coefficient adjustment factor (f_cal_corr_QEPRO) and upscaling nadir SIF to eddy covariance footprint factor (ratio_EC footprint, SIF pixel), and measured FPAR from quantum sensors and Rededge NDVI calculated FPAR. The data are provided in comma-separated values (CSV) format.

Ameriflux↗

DOE OSTI2022

Accelerating the discovery of battery electrode materials through data mining and deep learning models

The availability of crystalline materials databases allows for building accurate machine learning (ML) models that can accelerate the exploration of materials chemical space for energy storage applications. In this work, we screen all inorganic materials included in the Materials Project and AFLOW databases as potential metal-ion battery electrodes. We develop an efficient protocol to mine and screen raw data in current databases and provide a new database of electrode materials by considering pairs of charged and discharged electrodes. This effort leads to a new database with over 190,000 instances, in contrast to the original battery database which contains about 5000. The expanded battery data set is then used to build regression-based deep neural network models for predicting average voltages and percentage volume changes upon charging and discharging, which present improvements of at least 28% for target properties with respect to previous models, and are now able to predict anode electrodes (low voltage region) as well as electrodes that will not work in electrochemical cells (negative voltages), overcoming the challenges identified in previous ML models for battery electrodes. Additionally, a further screening of the expanded database itself allowed us to identify 35 novel electrode candidates with excellent battery performance metrics.

25 ENERGY STORAGE↗

DOE OSTI2024

Streaming Large-Scale Microscopy Data to a Supercomputing Facility

Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy generates data at a nominal rate of 480 Gbit s -1 (87,000 frames s -1 ⁠), producing a 700 GB dataset in 15 s. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera’s data acquisition system to supercomputing nodes at the National Energy Research Scientific Computing Center, bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a postmortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.

4D-STEM↗

DOE OSTI2021

Scaling Inference Using Triton to Accelerate Particle Physics at the LHC and DUNE

DUNE and the LHC experiments consist of unique and cutting-edge particle detectors that create massive, complex, and rich datasets with billions of events. They require sophisticated algorithms to reconstruct and interpret the data. Modern machine learning algorithms provide a powerful toolset to detect and classify particles, from familiar image processing convolutional neural networks to newer graph neural network architectures. A full reconstruction of these particle collisions requires novel approaches to handle the computing challenge of processing so much raw data. In a series of studies, physicists from Fermilab, CERN, and university groups explored how to accelerate their data processing using the Triton Inference Server.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2020

Review of thermophysical property methods applied to fueled and un-fueled molten salts

Recent investigation into designs of molten salt reactors (MSR) requires accurate data of candidate salts. Review of the literature shows lacking thermophysical (TP) property data for coolants and fuel-bearing salts of Advanced High Temperature Reactors (AHTR). Measurement uncertainties of current data are even more deficient, and this affects ongoing research where MSR modeling and testing are being performed. Melting temperature, viscosity, density, specific heat capacity, and thermal and electrical conductivity measurements are often not arranged together or discussed in great detail, especially for fuel-bearing salts. Raw data from key findings spanning over 70 years for LiF, LiF-BeF 2 , LiF-NaF-KF, KCl-MgCl 2 , KF-ZrF 4 , and NaF-KF-MgF 2 have been organized and presented here. Brief description of methods used to obtain these results, in addition to the TP data of some of these salts as fuel carriers, has also been discussed. Thermal conductivity data are particularly weak due to older, high-error measurement techniques. More recent measurement systems to evaluate salt thermal conductivity show promise for obtaining consistent absolute values. This work concludes that measurement systems such as transient hot-wire method, forced Rayleigh scattering, and laser flash analysis provide the most promise to fill these data gaps and reduce current uncertainty in thermal conductivity measurements of both fueled and un-fueled molten salts.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2021

Hybrid Machine Learning for Scanning Near-Field Optical Spectroscopy

The underlying physics behind an experimental observation often lacks a simple analytical description. This is especially the case for scanning probe microscopy techniques, where the interaction between the probe and the sample is nontrivial. Realistic modeling to include the details of the probe is always exponentially more difficult than its "spherical cow" counterparts. On the other hand, a well-trained artificial neural network based on real data can grasp the hidden correlation between the signal and sample properties. In this work, we show that, via a combination of model calculation and experimental data acquisition, a physics-infused hybrid neural network can predict the tip-sample interaction in the widely used scattering-type scanning near-field optical microscope. This hybrid network provides a long-sought solution for accurate extraction of material properties from tip-specific raw data. The methodology can be extended to other scanning probe microscopy techniques as well as other data-oriented physical problems in general.

36 MATERIALS SCIENCE↗

DOE OSTI2021

Data Arrays for Microearthquake (MEQ) Monitoring using Deep Learning for the Newberry EGS Sites

The 'Machine Learning Approaches to Predicting Induced Seismicity and Imaging Geothermal Reservoir Properties' project looks to apply machine learning (ML) methods to Microearthquake (MEQ) data for imaging geothermal reservoir properties and forecasting seismic events, in order to advance geothermal exploration and safe geothermal energy production. As part of the project, this submission provides data arrays for 149 microearthquakes between the year 2012 and 2013 at the Newberry EGS Site for use with the Deep Learning Algorithm that has been developed. The data provided includes raw waveform data, location data, normalized waveform data, and processed waveform data. Penn State Geothermal Team has shared the following files from the project: - 149 microearthquakes (MEQs) between 2012 and 2013 at Newberry EGS sites, 'Normalized Waveform Inputs.npz' are normalized waveforms. - labels of 149 MEQs: Processed Waveform Inputs.npz - location labels of 149 MEQs: Location Data.npz Note: .npz is the python file format by NumPy that provides storage of array data.

15 GEOTHERMAL ENERGY↗