Buoy - Lidar / Processed Data
This dataset contains standardized data from DOE Buoy 140 deployed during WFIP3. *.csv10m.zip files have been converted to netCDF.
17 WIND ENERGY↗
SEARCH · Engineering Papers
Results for “Data processing and standardization”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Rhode Island - Radar Wind Profiler / Processed Data
This dataset contains standardized data from the WFIP3 Wind Profiler located at the Rhode Island WFIP3 site.
17 WIND ENERGY↗
CACO Site - Radar Wind Profiler / Processed Data
This dataset contains standardized data from the WFIP3 Wind Profiler with Radio Acoustic Sounding System (RASS).
17 WIND ENERGY↗
Buoy 140 / Processed Data
These are the standardized buoy data collected during the WFIP3 project period, initially deployed near the Martha's Vineyard region for validation and later deployed at the WFIP3 location. The NetCDF files contain the data for most of the *.csv files for a given day.
17 WIND ENERGY↗
Lidar / Processed Data
This dataset contains standardized data from the WFIP3 NANT site UTD Halo XR Lidar.
17 WIND ENERGY↗
A Robust Data-Driven Approach for Mechanical Serial Sectioning
Mechanical serial sectioning (MSS) provides detailed microstructural information across large length scales. By repeatedly removing thin layers of material and imaging the exposed surface, a 3D representation of a specimen’s internal structure can be constructed, enabling failure analysis and feature identification that are otherwise inaccessible via conventional 2D or nondestructive evaluation techniques. Achieving consistent and accurate material removal can be challenging due to system variability, requiring an experienced operator to manually adjust parameters, prolonging data collection times and necessitating post-processing routines to standardize the data. Here, to address these challenges, this paper presents the employment of a one-step model predictive control (MPC) framework tailored to a run-to-run (R2R) controller. The R2R-MPC controller automates the parameter selection process, improving the consistency of material removal through iterative feedback for disturbance rejection and accurate tracking of the target removal rate. Using a data-driven approach, the controller robustly adapts to changing material characteristics. The effectiveness of the R2R-MPC controller is demonstrated through simulation and experimental results and compared to previous data collection procedures.
3D Materials Science↗
Improving Residential Building Simulations Through Large-Scale Empirical Validation
Residential building energy simulations are increasingly used for energy-efficient building design, codes and standards analysis, home certifications and ratings, utility programs, and technology assessments. Various software tools exist to perform residential building simulations, and these tools often use different models, inputs, and assumptions. This leads to inconsistencies that can undermine confidence in the predicted results. Validation of these tools can increase confidence by ensuring their accuracy and consistency. One way to validate simulation tools is through empirical testing, which compares predicted energy usage to measured utility billing data. This paper describes the process of data collection, data standardization, and empirical validation, and illustrates its use with our residential EnergyPlus (R)-based software. The data and process can be extended to other simulation tools and contribute to improving residential building simulations more broadly.
empirical validation↗
NANT Site - Lidar / Processed Data Reformatted
This dataset contains standardized data from the PNNL scanning Doppler lidar (S/N 184), consisting of range- and time-resolved measurements of radial velocity, attenuated backscatter, intensity, and spectral width. We note that the beam azimuth angles are NOT referenced to true north.
17 WIND ENERGY↗
VA EDH Advanced Software Pipeline Framework Report: Enhancing Automation and Scalability
The VA Environmental Determinants of Health (EDH) Advanced Software Pipeline Framework is designed to enhance the efficiency, scalability, and security of geospatial data processing workflows. This framework integrates modern data orchestration and containerization technologies, including Prefect for workflow automation, Docker for containerization, and PostgreSQL/PostGIS for geospatial data storage and analysis. It ensures standardized, reproducible, and automated data processing, supporting VA objectives related to substance use risk assessment and recovery research. The pipeline addresses key scalability and performance challenges through horizontal and vertical scaling, high-performance computing (HPC) integration, parallel processing, task caching, and dynamic resource allocation. These optimizations improve throughput and reduce latency, allowing the system to efficiently manage large and complex datasets. Additionally, security and compliance measures—such as data encryption (SSL), Role-Based Access Control (RBAC), and adherence to GDPR and HIPAA standards—safeguard sensitive information throughout data transmission and storage. A key implementation of this framework includes the automation of shelter list geolocation workflows, ensuring that up-to-date data is readily available for VA decision-making. Lessons learned from this project include the transition from in-memory processing to incremental storage writes, improving resource management and reliability. Future enhancements aim to expand automation, integrate AI-driven anomaly detection, and incorporate high-performance computing resources. This framework provides a scalable, secure, and adaptable solution for managing geospatial datasets, reinforcing the VA’s ability to support clinical and strategic initiatives through data-driven decision-making.
97 MATHEMATICS AND COMPUTING↗
Tidal Resource Data from Sequim Bay Inlet, WA, August 2020
Data from a Nortek Signature1000 deployed on a lander for 14 days in Aug 2020 in the entrance to Sequim Bay, WA. Raw data were processed using the DOLfYN python package and standardized using the ME Data Pipeline python package, tsdat version 0.2.12. Processed data were partitioned into 24 hour increments and saved in the NETCDF file format.
16 TIDAL AND WAVE POWER↗
Path Forward: Materials Data Modernization for ASME Codes and Standards in the Artificial Intelligence Era
Development of the ASME Materials Properties Database was initiated in the early 2010s to support the ASME Codes and Standards. As information technologies advance at an accelerated pace with the artificial intelligence era on the horizon, the ASME Materials Properties Database must be further modernized from a database to a knowledgebase to ride the wave of digital information revolution and effectively support the ASME Codes and Standards in the new era.This paper is intended to provide an overview of the ASME Materials Properties Database and discuss a roadmap for its future development to facilitate understanding of and participation from different sectors of the Codes and Standards community. It first reviews the basic concepts of data, information, knowledge, database, and database system; as well as the pros and cons in different types of data management, and then discusses the path forward for a desired evolution of the database into a self-explanatory and machine-readable knowledgebase that is consistent with human cognitive processes for the Codes and Standards development and furthermore provides resources for data processing and analysis to reach an eventual goal of streamlining the Codes and Standards development from the initial inquiry, throughout data submission, analysis, …, to Codes and Standards rule establishment for final publication.
Ren, Weiju↗
NASA GeneLab RNA-seq consensus pipeline: Standardized processing of short-read RNA-seq data
With the development of transcriptomic technologies, we are able to quantify precise changes in gene expression profiles from astronauts and other organisms exposed to spaceflight. Members of NASA GeneLab and GeneLab-associated analysis working groups (AWGs) have developed a consensus pipeline for analyzing short-read RNA-sequencing data from spaceflight-associated experiments. The pipeline includes quality control, read trimming, mapping, and gene quantification steps, culminating in the detection of differentially expressed genes. This data analysis pipeline and the results of its execution using data submitted to GeneLab are now all publicly available through the GeneLab database. We present here the full details and rationale for the construction of this pipeline in order to promote transparency, reproducibility, and reusability of pipeline data; to provide a template for data processing of future spaceflight-relevant datasets; and to encourage cross-analysis of data from other databases with the data available in GeneLab.
59 BASIC BIOLOGICAL SCIENCES↗
Path Forward: Materials Data Modernization for ASME Codes and Standards in the Artificial Intelligence Era
Development of the ASME Materials Properties Database was initiated in the early 2010s to support the ASME Codes and Standards. As information technologies advance at an accelerated pace with the artificial intelligence era on the horizon, the ASME Materials Properties Database must be further modernized from a database to a knowledgebase to ride the wave of digital information revolution and effectively support the ASME Codes and Standards in the new era. This paper is intended to provide an overview of the ASME Materials Properties Database and discuss a roadmap for its future development to facilitate understanding of and participation from different sectors of the Codes and Standards community. Further, it first reviews the basic concepts of data, information, knowledge, database, and database system as well as the pros and cons in different types of data management and then discusses the path forward for a desired evolution of the database into a self-explanatory and machine-readable knowledgebase that is consistent with human cognitive processes for the Codes and Standards development and, furthermore, provides resources for data processing and analysis to reach an eventual goal of streamlining the Codes and Standards development from the initial inquiry, throughout data submission, analysis, …, to Codes and Standards rule establishment for final publication.
36 MATERIALS SCIENCE↗
Physics-Informed Gaussian Process Regression for States Estimation and Forecasting in Power Grids
Real-time state estimation and forecasting are critical for the efficient operation of power grids. In this paper, a physics-informed Gaussian process regression (PhI-GPR) method is presented and used for forecasting and estimating the phase angle, angular speed, and wind mechanical power of a three-generator power grid system using sparse measurements. In standard data-driven Gaussian process regression (GPR), parameterized models for the prior statistics are fit by maximizing the marginal likelihood of observed data. In the PhI-GPR method, we propose to compute the prior statistics offline by solving stochastic differential equations (SDEs) governing the power grid dynamics. The short-term forecast of a power grid system dominated by wind generation is complicated by the stochastic nature of the wind and the resulting uncertainty in wind mechanical power. Here, we assume that the power grid dynamics are governed by swing equations, with the wind mechanical power fluctuating randomly in time. We solve these equations for the mean and covariances of the power grid states using the Monte Carlo simulation method. We demonstrate that the proposed PhI-GPR method can accurately forecast and estimate observed and unobserved states. For the considered problem, PhI-GPR has computational advantages over the ensemble Kalman filter (EnKF) method: In PhI-GPR, ensembles are computed offline and independently of the data acquisition process, whereas for EnFK, ensembles are computed online with data acquisition, rendering real-time forecast more challenging. We also demonstrate that the PhI-GPR forecast is more accurate than the EnKF forecast when the random mechanical wind power is non-Markovian. In contrast, the two methods produce similar forecasts for the Markovian mechanical wind power. For observed states, we show that PhI-GPR provides a forecast comparable to the standard data-driven GPR; both forecasts are significantly more accurate than the autoregressive integrated moving average (ARIMA) forecast. We also show that the ARIMA forecast is more sensitive to observation frequency and measurement errors than the PhI-GPR forecast.
24 POWER TRANSMISSION AND DISTRIBUTION↗
nmrrr : A Reproducible Workflow for Binning and Visualizing NMR Spectra From Environmental Samples
Nuclear magnetic resonance (NMR) spectroscopy is a useful tool for detection and identification of molecular structural information, with increasing applications in environmental sciences. NMR instrument outputs are however heterogeneous and require extensive post-processing, creating barriers to their use and application by non-specialists. Here, we report on a new open-source R package, nmrrr, that processes and visualizes spectral data obtained from one-dimensional solution-state and solid-state NMR experiments; the package also performs relevant calculations commonly applied in natural organic matter communities, such as computing the relative abundance of various functional groups. We document the package's installation, dependencies, and functions; and provide a standard workflow for processing NMR data. This package is currently available on CRAN and GitHub, and community contributions are welcome.
54 ENVIRONMENTAL SCIENCES↗
Enabling modern data discovery for atmospheric measurements
The Atmospheric Radiation Measurement (ARM) user facility is a US Department of Energy Office of Science user facility that is managed and operated through a collaborative effort led by nine US Department of Energy national laboratories. The ARM Data Center, located at Oak Ridge National Laboratory, is responsible for the timely collection, processing, and delivery of data products to the scientific community. The ARM Data Center holds more than 11,000 data products, including metadata collected from field campaigns, instruments, value-added products, and principal investigator–contributed data. These data sets are checked for successful transfer (for most data, this transfer is carried out automatically via the network; however, some of the largest data sets and some of the most remote sites require manual shipping of hard disks) and both the data and metadata are processed to a standard format, which is an ARM-standardized structure, via the Network Common Data Form. The Network Common Data Form is a self-describing binary format with many compatible software tools. Once processed, the data are cataloged, stored in the ARM Data Archive, and made discoverable through association with an array of metadata-characterizing information, such as location and measurement classification. These metadata enable powerful search capabilities through the ARM Data Center Data Discovery interface. This paper discusses the workflow of how the new discovery system has been redesigned from user requirements and how the data are distributed to the scientific community.
54 ENVIRONMENTAL SCIENCES↗
Accuracy of Vaisala RS41 and RS92 Upper Tropospheric Humidity Compared to Satellite Hyperspectral Infrared Measurements
Radiosondes are important for calibrating satellite sensors and assessing sounding retrievals. Vaisala RS41 radiosondes have mostly replaced RS92 in the Global Climate Observing System (GCOS) Reference Upper Air Network (GRUAN) and the conventional network. This study assesses RS41 and RS92 upper tropospheric humidity (UTH) accuracy by comparing with Infrared Atmospheric Sounding Interferometer (IASI) upper tropospheric water vapor absorption spectrum measurements. Using single RS41 and RS92 soundings at three GRUAN and DOE Atmospheric Radiation Measurement (ARM) sites and dual RS92/RS41 launches at three additional GRUAN sites, collocated with cloud-free IASI radiances (OBS), we compute Line-by-Line Radiative Transfer Model radiances for radiosonde profiles (CAL). We analyze OBS-CAL differences from 2015 to 2020, for daytime, nighttime, and dusk/dawn separately if data is available, for standard (STD) RS92 and RS41 processing, and RS92 GRUAN Data Processing (GDP; RS41 GDP is in development). We find that daytime RS41 (even without GDP) has ~1% smaller UTH errors than GDP RS92. RS41 may still have a dry bias of 1–1.5% for both daytime and nighttime, and a similar error for nighttime RS92 GDP, while standard RS92 may have a dry bias of 3–4%. These sonde humidity biases are probably upper limits since “cloud-free” scenes could still be cloud contaminated. Radiances computed from European Centre for Medium-Range Weather Forecasts (ECMWF) analyses match better than radiosondes with IASI measurements, perhaps because ECMWF assimilates IASI measurements. Relative differences between RS41 STD and RS92 GDP, or between radiosondes and ECMWF humidity profiles obtained from the radiance analysis, are consistent with their differences obtained directly from the RH measurements.
54 ENVIRONMENTAL SCIENCES↗
Big Data Processing for Power Grid Event Detection
In this paper we present the application of big data processing for the development of machine learning(ML) models to detect relevant events in power gridoperations. This is based on almost 20TB of phasormeasurement unit data corresponding to up to two years of operation of three grid interconnections which provide power to most of the United States. A significant aspect of the work consists in having all data processing performed on a single standard GPU server, from pre-processing to ML model training and testing. We describe the data and computational infrastructure, challenges faced and methods used in dataprocessing, main findings and results. The ML approach employed for best utilization of the big data is also discussed, including sample results.
Paes Leao, Bruno↗