Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Earth Science Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Sample Identifiers and Metadata to Support Data Management and Reuse in Multidisciplinary Ecosystem Sciences

Physical samples are foundational entities for research across biological, Earth, and environmental sciences. Data generated from sample-based analyses are not only the basis of individual studies, but can also be integrated with other data to answer new and broader-scale questions. Ecosystem studies increasingly rely on multidisciplinary team-science to study climate and environmental changes. While there are widely adopted conventions within certain domains to describe sample data, these have gaps when applied in a multidisciplinary context. In this study, we reviewed existing practices for identifying, characterizing, and linking related environmental samples. We then tested practicalities of assigning persistent identifiers to samples, with standardized metadata, in a pilot field test involving eight United States Department of Energy projects. Participants collected a variety of sample types, with analyses conducted across multiple facilities. We address terminology gaps for multidisciplinary research and make recommendations for assigning identifiers and metadata that supports sample tracking, integration, and reuse. Furthermore, our goal is to provide a practical approach to sample management, geared towards ecosystem scientists who contribute and reuse sample data.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Data from: "A guide to using GitHub for developing and versioning data standards and reporting formats"

These data are the results of a systematic review that investigated how data standards and reporting formats are documented on the version control platform GitHub. Our systematic review identified 32 data standards in earth science, environmental science, and ecology that use GitHub for version control of data standard documents. In our analysis, we characterized the documents and content within each of the 32 GitHub repositories to identify common practices for groups that version control their documents on GitHub.In this data package, there are 8 CSV files that contain data that we characterized from each repository, according to the location within the repository. For example, in 'readme_pages.csv' we characterize the content that appears across the 32 GitHub repositories included in our systematic review. Each of the 8 CSV files has an associated data dictionary file (names appended with '_dd.csv' and here we describe each content category within CSV files.There is one file-level metadata file (flmd.csv) that provides a description of each file within the data package.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

CMIP7 data request: Earth system priorities and opportunities

This paper presents a comprehensive overview of the Coupled Model Intercomparison Project Phase 7 (CMIP7) request for data pertaining to Earth systems science, and provides justification for the resources needed to produce this data. Topics within the CMIP7 Earth System (CMIP7-ES) theme centre around tracking of flows of energy, carbon, water and other fluxes across domains, and constraining feedbacks between these cycles and the climate system. These topics are summarized in this paper as scientific “opportunities” describing specific model intercomparison experiments and use cases for next-generation Earth System Model (ESM) output. These opportunities were submitted by modelling groups and scientific consortia following an extended public consultation process. Contained within each opportunity are requests for groups of Climate & Forecasting (CF) variables, which are bundled into variable groups representing all data required to address the opportunities' needs. Novel opportunities in CMIP7 compared with previous phases will include running `emissions-driven' simulations that integrate carbon emissions and removal scenarios with updated representations of the global carbon cycle, expanded variable groups needed to model marine trophic interactions and biogeochemistry, and data needed to understand the risk of global tipping points, among others. The production of these variables will close key gaps and uncertainties identified during previous rounds of CMIP, and support the 7th Intergovernmental Panel on Climate Change Assessment Report (AR7). We argue that CMIP7-ES data will be broadly used by scientific, policy, governmental, industry, and other communities that rely on climate model projections for research and decision making. As an author group we also reflect on the evolution of the CMIP7-ES data request as a part of a deliberative process in support of the global CMIP program.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

A Geo-Data Science Method for Assessing Unconventional Rare-Earth Element Resources in Sedimentary Systems

Abstract Rare-earth elements (REEs) supply raw materials that constitute many of our modern critical infrastructure, defense, technology, and electrification needs. Despite REE accumulations occurring in conventional bedrock and ion-adsorption deposits sourced from weathering of igneous rocks, unconventional host materials such as coal and related sedimentary strata have been identified as promising sources of REEs to meet growing demand. To maximize the potential of unconventional resources such as REE-coal systems, new approaches are needed overcome challenges from mineral systems with no known deposits and areas with sparse geochemical data. This article presents a systematic knowledge-data resource assessment method for predicting and identifying REE resource potential and occurrence in these unconventional systems. The method utilizes a geologic and geospatial knowledge-data approach informed and guided by REE accumulation mechanisms to systematically assess and identify areas of higher enrichment. An assessment of the Powder River Basin is presented as a test case to demonstrate the method workflow and results. The key output is a potential enrichment score map reported with varying confidence levels based on the amount of supporting evidence. Results from the test case indicate several locations with promising potential for different types of coal-REE deposits, demonstrating the viability of the method for exploration and assessment of unconventional REE resources. The method is flexible by design and, with sufficient applicable knowledge and data, can be adapted for assessing critical mineral systems in other sedimentary systems as well.

58 GEOSCIENCES↗
DOE OSTI2021

Connect the Dots: In Situ 4-D Seismic Monitoring of CO 2 Storage With Spatio-Temporal CNNs

4-D seismic imaging has been widely used in CO 2 sequestration projects to monitor the fluid flow in the volumetric subsurface region that is not sampled by wells. Ideally, real-time monitoring and near-future forecasting would provide site operators with great insights to understand the dynamics of the subsurface reservoir and assess any potential risks. However, due to obstacles such as high deployment cost, availability of acquisition equipment, exclusion zones around surface structures, only very sparse seismic imaging data can be obtained during monitoring. That leads to an unavoidable and growing knowledge gap over time. The operator needs to understand the fluid flow throughout the project lifetime and the seismic data are only available at a limited number of times. This is insufficient for understanding reservoir behavior. To overcome those challenges, we have developed spatio-temporal neural-network-based models that can produce high-fidelity interpolated or extrapolated images effectively and efficiently. Specifically, our models are built on an autoencoder, and incorporate the long short-term memory (LSTM) structure with a new loss function regularized by optical flow. We validate the performance of our models using real 4-D post-stack seismic imaging data acquired at the Sleipner CO 2 sequestration field. We employ two different strategies in evaluating our models. Numerically, we compare our models with different baseline approaches using classic pixel-based metrics. We also conduct a blind survey and collect a total of 20 responses from domain experts to evaluate the quality of data generated by our models. Finally, via both numerical and expert evaluation, we conclude that our models can produce high-quality 2-D/3-D seismic imaging data at a reasonable cost, offering the possibility of real-time monitoring or even near-future forecasting of the CO 2 storage reservoir.

4-D seismic imaging↗
DOE OSTI2025

The Pan-Arctic Vegetation Cover (PAVC) database v1.1

The Pan-Arctic Vegetation Cover (PAVC) database contains synthesized field-data observations of vegetation cover from 978 Arctic Alaska plots with observations from 2010 to 2021. The cover datasets contain plot data at both the plant functional type (PFT) and species-level resolution, with standardized PFT definitions and species names. We synthesized publicly available point-intercept and visual estimate plots from the Arctic Vegetation Archive of Alaska, the Alaska Vegetation Plots Database, the North Slope Science Catalog, and the National Ecological Observatory Network; as well as previously unpublished data from the Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic).Users will find four synthesized datasets, 4 associated data descriptor (dd) files, and 1 metadata file in the PAVC database:synthesized_species_fcover.csv contains fractional cover (fcover) for unique accepted species names, where names include vegetation identified at the family, genus, species, subspecies, and variety levels, as well as general functional types across all 5 data sources. The synthesized_species_fcover_dd.csv accompanies this dataset with header information.synthesized_pft_fcover.csv contains fcover for the following PFTs: non-vascular plants with lichen and bryophyte subcategories, trees with deciduous and evergreen subcategories, shrubs with deciduous and evergreen subcategories, graminoids (grasses), and forbs (herbaceous flowering plants) measured as total cover. Litter and “other” cover are also included as total cover. Additional “types” include water and bare ground, which were measured as top cover. The synthesized_pft_fcover_dd.csv accompanies this dataset with header information.species_pft_checklist.csv is a lookup table containing the translation from a dataset species name to an accepted species name and to a PFT. This table can be used to clarify our species to PFT adjudications, and to aid users in assigning their own PFTs. Any issues found in this checklist should be reported in the Issues tab of our github.survey_unit_information.csv contains auxiliary information about the plots synthesized in this database. It contains useful information for filtering plots of interest based on temporal, geospatial, and contextual information about the plot surveys.flmd.csv contains metadata information about each file in the database.This research was performed as a part of the NGEE Arctic project. The NGEE Arctic project was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Enhancing Regional Seismic Velocity Models With Higher‐Resolution Local Results Using Sparse Dictionary Learning

Abstract We use sparse dictionary learning to develop transformations between seismic velocity models of different resolution and spatial extent. Starting with data in the common region of both models, the method can enhance a regional lower‐resolution model to match the style and resolution of local higher‐resolution results while preserving its regional coverage. The method is demonstrated by applying it to two‐dimensional V S and three‐dimensional V P and V S regional and local velocity models in southern California. The enhanced reconstructed regional results exhibit clear visual improvements, especially in the reconstructed V P / V S ratios, and better correlations with geological features. Moreover, the reconstructed regional V P , V S models outperform the original ones in comparison of simulated earthquake waveforms to observations. The improved fitting to observed waveforms extends beyond the domain of the overlapping region. The developed dictionary learning approach provides physically interpretable results and offers a powerful tool for additional applications of data enhancement in earth sciences.

Geochemistry & Geophysics↗
DOE OSTI2021

Call to Action for Global Access to and Harmonization of Quality Information of Individual Earth Science Datasets

Knowledge about the quality of data and metadata is important to support informed decisions on the (re)use of individual datasets and is an essential part of the ecosystem that supports open science. Quality assessments reflect the reliability and usability of data. They need to be consistently curated, fully traceable, and adequately documented, as these are crucial for sound decision- and policy-making efforts that rely on data. Quality assessments also need to be consistently represented and readily integrated across systems and tools to allow for improved sharing of information on quality at the dataset level for individual quality attribute or dimension. Although the need for assessing the quality of data and associated information is well recognized, methodologies for an evaluation framework and presentation of resultant quality information to end users may not have been comprehensively addressed within and across disciplines. Global interdisciplinary domain experts have come together to systematically explore needs, challenges and impacts of consistently curating and representing quality information through the entire lifecycle of a dataset. This paper describes the findings of that effort, argues the importance of sharing dataset quality information, calls for community action to develop practical guidelines, and outlines community recommendations for developing such guidelines. Practical guidelines will allow for global access to and harmonization of quality information at the level of individual Earth science datasets, which in turn will support open science.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2024

A Unified Data Infrastructure for Biological and Environmental Research: A Report from the BER Advisory Committee

The Biological and Environmental Research (BER) program within the U.S. Department of Energy (DOE) Office of Science supports large-scale data generation efforts across its two divisions: Biological Systems Science and Earth and Environmental Systems Sciences. These efforts include user facilities in atmospheric radiation measurements, genomics, metabolomics, proteomics, compute, and imaging. In addition, BER supports the development of plant-based fuels; research in biosystems design, environmental microbiomes, and atmospheric systems; energy flux monitoring; climate-based ecosystem experiments; pathogen biopreparedness; and modeling of climate, urban interfaces, and interactions between people and energy resources. For data access, BER supports community data services at its user facilities, along with specialized data initiatives for Earth and environmental science, climate modeling, genomic and microbial analysis, and multisector dynamics modeling.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Data from: "Lab and Field Warming Similarly Advance Germination Date and Limit Germination Rate for High and Low Elevation Provenances of Two Widespread Subalpine Conifers"

This archive contains data used to draw conclusions in “Lab and Field Warming Similarly Advance Germination Date and Limit Germination Rate for High and Low Elevation Provenances of Two Widespread Subalpine Conifers”, by Kueppers et al. 2017. Cone collection and field experiments were conducted on Niwot Ridge, Colorado, with the latter completed within the Alpine Treeline Warming Experiment (ATWE). Laboratory germination experiments were conducted at the University of California Merced. Geospatial files are also provided for additional geographical context.There are four data file formats in this package: comma-separated values (.csv), Microsoft Excel (.xlsx), keyhole markup language (.kml), and ESRI shapefile (.shp). The latter two are geospatial file formats. .csv files can be opened using any simple text editor program such as TextEdit (Mac) and Notepad (Windows), and .xlsx files can be opened using Microsoft Excel or Google Sheets. The .kml file contains coordinate data, and is compatible with Google Earth and Google Maps. ESRI shapefiles (.shp) are compatible with any geospatial software able to read the file type, such as QGIS and ESRI’s ArcGIS Suite. This data user’s guide is available in as a .pdf, and can be opened using software such as Adobe Acrobat Reader.-----------------------------------------------------------------------------------------------------------------------------------Accurately predicting upslope shifts in subalpine tree ranges with warming requires understanding how future forest populations will be affected by climate change, as these are the seed sources for new tree line and alpine populations. Early life history stages are particularly sensitive to climate and are also influenced by genetic variation among populations. We tested the climate sensitivity of germination and initial development for two widely distributed subalpine conifers, using controlled-environment growth chambers with one temperature regime from subalpine forest in the Colorado Rocky Mountains and one 5 °C warmer, and two soil moisture levels. We tracked germination rate and timing, rate of seedling development, and seedling morphology for two seed provenances separated by ~300 m elevation. Warming advanced germination timing and initial seedling development by a total of ~2 weeks, advances comparable to mean differences between provenances. Advances were similar for both provenances and species; however, warming reduced the overall germination rate, as did low soil moisture, only for Picea engelmannii. A three-year field warming and watering experiment planted with the same species and provenances yielded responses qualitatively consistent with the lab trials. Together these experiments indicate that in a warmer, drier climate, P. engelmannii germination, and thus regeneration, could decline, which could lead to declining subalpine forest populations, while Pinus flexilis forest populations could remain robust as a seed source for upslope range shifts.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Dynamically Downscaled (WRF) 1km, Hourly Meteorological Conditions 1987-2020. East/Taylor Watersheds

This dataset contains meteorological output from the Weather Research and Forecasting (WRF) version 3.8.1. This dataset has been created to 1) investigate hydrometeorological processes impacting the East River and water-delivery to the Critical Zone, and 2) provide meteorological forcing data for distributed Earth-science modeling applications in the East River watershed. Variables have a 1 kilometer spatial resolution and hourly temporal resolution and encompass a rectangular region encompassing the East and Taylor River watersheds, Colorado, near the town of Crested Butte. WRF was forced using Climate Forecast System Reanalysis (CFSR) lateral boundary conditions. Each .zip file contains one "water year" of data (October 1 -- September 30; i.e. water year 2017 starts October 1, 2016 and ends September 30, 2017). Each zip folder contains 12 netcdf (.nc) files containing one month of hourly data each and are approximately 250mb. Model timestamps are in UTC time.The files contain the following data variables:EAST_MASK: binary mask of the watershed regionTAYLOR_MASK: binary mask of the watershed regionGLW downwelling longwave radiation (w/m2)HR_PRCP: Hourly Precipitation Rate (mm/hr). Includes all hydrometeors (solid+liquid). HFX NoahMP LSM total grid-cell modelled sensible heat flux (w/m2) [positive towards atmosphere]LH NoahMP LSM total grid-cell modelled latent heat flux (w/m2) [positive towards atmosphere; can be converted to ET]PSFC Surface Barometric Pressure (hPa)Q2 Two-meter specific humidity (kg/kg)SWDOWN Downwelling shortwave solar radiation (w/m2)SWNORM Terrain-normal downwelling shortwave radiation (w/m2)T2 Two-meter air temperature (deg K)U10 10-m U-component of wind velocity (m/s)V10 10m V-component of wind velocity (m/s)XLAT Latitude of grid-center point XLONG Longitude of grid-center point XTIME Model timestamp, **in UTC**

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

FAIR data infrastructure and tools for AI-assisted streamflow prediction

Focal Area(s) Areas: We discuss how the integration of AI into Earth Science models can impact streamflow predictions at both the science and data levels. Doing so, we address cross-cutting needs related to the goal of making data FAIR (Findable, Accessible, Interoperable, and Re-usable [1]) for seamless use with Artificial Intelligence/Machine Learning (AI/ML) in Earth System Science at DOE. A novel idea is that AI/ML itself can help with the FAIR data goal and address issues in targeted areas e.g. missing data, data quality and reduction. In addition, the interpretability of results obtained with new AI methods is poised to impact broader scientific challenges in hydrology..

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Simulating Atmospheric Processes in Earth System Models and Quantifying Uncertainties With Deep Learning Multi‐Member and Stochastic Parameterizations

Abstract Deep learning is a powerful tool to represent subgrid processes in climate models, but many application cases have so far used idealized settings and deterministic approaches. Here, we develop stochastic parameterizations with calibrated uncertainty quantification to learn subgrid convective and turbulent processes and surface radiative fluxes of a superparameterization embedded in an Earth System Model (ESM). We explore three methods to construct stochastic parameterizations: (a) a single Deep Neural Network (DNN) with Monte Carlo Dropout; (b) a multi‐member parameterization; and (c) a Variational Encoder Decoder with latent space perturbation. We show that the multi‐member parameterization improves the representation of convective processes, especially in the planetary boundary layer, compared to individual DNNs. The respective uncertainty quantification illustrates that methods (b) and (c) are advantageous compared to a dropout‐based DNN parameterization regarding the spread of convective processes. Hybrid simulations with our best‐performing multi‐member parameterizations remained challenging and crash within the first days. Therefore, we develop a pragmatic partial coupling strategy relying on the superparameterization for condensate emulation. Partial coupling reduces the computational efficiency of hybrid Earth‐like simulations but enables model stability over 5 months with our multi‐member parameterizations. However, our hybrid simulations exhibit biases in thermodynamic fields and differences in precipitation patterns. Despite this, the multi‐member parameterizations enable improvements in reproducing tropical extreme precipitation compared to a traditional convection parameterization. Despite these challenges, our results indicate the potential of a new generation of multi‐member machine learning parameterizations leveraging uncertainty quantification to improve the representation of stochasticity of subgrid effects.

Behrens, Gunnar [Deutsches Zentrum für Luft‐ und R↗
DOE OSTI2021

A Guide to Using GitHub for Developing and Versioning Data Standards and Reporting Formats

Abstract Data standardization combined with descriptive metadata facilitate data reuse, which is the ultimate goal of the Findable, Accessible, Interoperable, and Reusable (FAIR) principles. Community data or metadata standards are increasingly created through an approach that emphasizes collaboration between various stakeholders. Such an approach requires platforms for collaboration on the development process that centers on sharing information and receiving feedback. Our objective in this study was to conduct a systematic review to identify data standards and reporting formats that use version control for developing data standards and to summarize common practices, particularly in earth and environmental sciences. Out of 108 data standards and reporting formats identified in our review, 32 used GitHub as the version control platform, and no other platforms were used. We found no universally accepted methodology for developing and publishing data standards. Many GitHub repositories did not use key features that could help developers to gather user feedback, or to create and revise standards that build on previous work. We provide guidance for community‐driven standard development and associated documentation on GitHub based on a systematic review of existing practices.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Hyperspectral remote sensing-based plant community map for region around NGEE-Arctic intensive research watersheds at Seward Peninsula, Alaska, 2017-2019

Using airborne hyperspectral remote sensing data from NASA Airborne Visible-Infrared Imaging Spectrometer- Next Generation (AVIRIS-NG) platforms in a region near NGEE-Arctic intensive watersheds at Seward peninsula of Alaska, high resolution (5m) maps of plant community distribution were developed and included in this data collected. AVIRIS-NG data collected over 2017-2019 period were used to develop deep neural networks, trained using vegetation plot observations collected at NGEE-Arctic watersheds at Kougarok, Council and Teller. A hierarchical vegetation classification scheme consisting of six classes at Level I, and 16 classes at Level II contained in two .txt files were used to developed the plant community maps for the region. Two geospatial raster data files (.tif) at both thematic levels are shared in this data collection. Data files in this collection use Alaska Albers Equal Area projection. Readme files available in three formats (*.html, *.md, *.pdf) and one *.png visualization map.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Multiscale Data-Driven Seismic Full-Waveform Inversion With Field Data Study

Seismic full-waveform inversion (FWI), which uses iterative methods to estimate high-resolution subsurface models from seismograms, is a powerful imaging technique in exploration geophysics. In recent years, the computational cost of FWI has grown exponentially due to the increasing size and resolution of seismic data. Moreover, it is a nonconvex problem and can encounter local minima due to the limited accuracy of the initial velocity models or the absence of low frequencies in the measurements. To overcome these computational issues, we develop a multiscale data-driven FWI method based on fully convolutional networks (FCNs). In preparing the training data, we first develop a real-time style transform method to create a large set of synthetic subsurface velocity models from natural images. We then develop two convolutional neural networks with encoder-decoder structures to reconstruct the low- and high-frequency components of the subsurface velocity models, separately. To validate the performance of our data-driven inversion method and the effectiveness of the synthesized training set, we compare it with conventional physics-based waveform inversion approaches using both synthetic and field data. Finally, these numerical results demonstrate that, once our model is fully trained, it can significantly reduce the computation time and yield more accurate subsurface velocity models in comparison with conventional FWI.

58 GEOSCIENCES↗
DOE OSTI2024

On the anatomy of acoustic emission

Abrupt, local frictional fault failure comprises a displacement that is normally accompanied by acoustic emission (AE)—an impulsive elastic wave broadcast with an amplitude proportional to particle velocity. The aggregate of these displacements is the basic fault motion. In laboratory shear experiments, the examination of a sequence of laboratory earthquakes includes continuous measurements of fault motion and the associated AE that is broadcast. From these measurements, connections between the fault motion and cumulative sum of the AE amplitude can be identified. The composition of the AE broadcasts reveals inhomogeneity in the fault mechanical structure from which they arise. This inhomogeneity can be decomposed into a time invariant AE component and an articulated AE component. The articulated AE component serves as a “state of the fault diagnostic” that follows a distinctive pattern to fault failure. Thus, the articulated AE component can be used directly to monitor the state of the fault.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2025

Data and scripts from: “Denoising autoencoder for reconstructing sensor observation data and predicting evapotranspiration: noisy and missing values repair and uncertainty quantification”

This data package includes data and scripts from the manuscript “Denoising autoencoder for reconstructing sensor observation data and predicting evapotranspiration: noisy and missing values repair and uncertainty quantification”.The study addressed common challenges faced in environmental sensing and modeling, including uncertain input data, missing sensor observations, and high-dimensional datasets with interrelated but redundant variables. Point-scaled meteorological and soil sensor observations were perturbed with noises and missing values, and denoising autoencoder (DAE) neural networks were developed to reconstruct the perturbed data and further predict evapotranspiration. This study concluded that (1) the reconstruction quality of each variable depends on its cross-correlation and alignment to the underlying data structure, (2) uncertainties from the models were overall stronger than those from the data corruption, and (3) there was a tradeoff between reducing bias and reducing variance when evaluating the uncertainty of the machine learning models.This package includes:(1) Four ipython scripts (.ipynb): “DAE_train.ipynb” trains and evaluates DAE neural networks, “DAE_predict.ipynb” makes predictions from the trained DAE models, “ET_train.ipynb” trains and evaluates ET prediction neural networks, and “ET_predict.ipynb” makes predictions from trained ET models.(2) One python file (.py): “methods.py” includes all user-defined functions and python codes used in the ipython scripts.(3) A “sub_models” folder that includes five trained DAE neural networks (in pytorch format, .pt), which could be used to ingest input data before being fed to the downstream ET models in ‘ET_train.ipynb” or ‘ET_predict.ipynb’.(4) Two data files (.csv). Daily meteorological, vegetation, and soil data is in “df_data.csv”, where “df_meta.csv” contains the location and time information of “df_data.csv”. Each row (index) in “df_meta.csv” corresponds to each row in “df_data.csv”. These data files are formatted to follow the data structure requirements and be directly used in the ipython scripts, and they have been shuffled chronologically to train machine learning models. The meteorological and soil data was collected using point sensors between 2019-2023 at(4.a) Three shrub-dominated field sites in East River, Colorado (named “ph1”, “ph2” and “sg5” in “df_meta.csv”, where “ph1” and “ph2” were located at PumpHouse Hillslopes, and “sg5” was at Snodgrass Mountain meadow) and(4.b) One outdoor, mesoscale, and herbaceous-dominated experiment in Berkeley, California (named “tb” in “df_meta.csv”, short for Smartsoils Testbed at Lawrence Berkeley National Lab).- See "df_data_dd.csv" and "df_meta_dd.csv" for variable descriptions and the Methods section for additional data processing steps. See "flmd.csv" and "README.txt" for brief file descriptions.- All ipython scripts and python files are written in and require PYTHON language software.

54 ENVIRONMENTAL SCIENCES↗