Engineering Papers⌕ Search

Engineering topics

Agarwal, Deb

Publications and source records attributed to Agarwal, Deb.

At least 19 records

Advanced Research Directions on AI for Science, Energy, and Security: Report on Summer 2022 Workshops

Over the past decade, fundamental changes in artificial intelligence (AI)—from foundational to applied—have delivered dramatic insights across a wide breadth of U.S. Department of Energy (DOE) mission space. AI is helping to augment and improve scientific and engineering workflows (e.g., for control, design, and dramatic performance gains through surrogate models) in national security, the Office of Science, and DOE’s applied energy programs. The progress and potential for AI in DOE science was captured in the 2020 “AI for Science” report from the DOE laboratory community in collaboration with academia and industry. Specific scientific areas ready to further leverage the power of AI ranged from the scale and performance of computational models to data analysis to creating new classes of observations using computer vision. Since that report, the scale and scope of scientific AI have accelerated, revealing new, emergent properties that yield insights that go beyond enabling opportunities to being potentially transformative in the way that scientific problems are posed and solved. Thus, under the guidance of both the Office of Science (SC) and the National Nuclear Security Administration (NNSA), the DOE national laboratories organized a series of workshops in 2022 to gather input on new and rapidly emerging opportunities and challenges of scientific AI. This 2023 report is a synthesis of those workshops. The scientific community believes AI can have a foundational impact on a broad range of DOE missions, including science, energy, and national security. Further, DOE has unique capabilities that enable the community to drive progress in scientific use of AI, building on long-standing DOE strengths and investments in computation, data, and communications infrastructure, spanning the Energy Sciences Network (ESnet), the Exascale Computing Project (ECP), and integrative programs such as the NNSA Office of Defense Programs Advanced Simulation and Computing (ASC) and the SC Scientific Discovery through Advanced Computing (SciDAC) programs.

97 MATHEMATICS AND COMPUTING↗

ESS-DIVE Reporting Format for Location Metadata

The ESS-DIVE location metadata reporting format provides instructions and templates for reporting a minimum set of metadata for discrete point locations in geographic space represented by x, y, and z coordinates. This format was created based on a need for earth and environmental science researchers to more consistently provide metadata about locations where they conduct studies. To create the format, we incorporated elements from ESS-DIVE’s community reporting formats as well as 12 additional data standards or other data resources (e.g., databases, data systems, or repositories). In the template, we ask researchers to indicate unique locations using Location IDs and indicate hierarchies of locations through parent location IDs. We also provide additional optional fields for researchers to indicate how they measured the point location and the date and time that the location was first used as a research siteThis dataset contains support documentation for the reporting format (README.md and instructions.md), a terminology guide (guide.md), a crosswalk indicating how this reporting format relates to existing standards and data resources (Location_metadata_crosswalk.csv), a data dictionary (dd.csv), file-level metadata (flmd.csv), and the location metadata templates in both CSV (Location_metadata_template.csv) and Excel formats (Location_metadata_template.xlsx).

54 ENVIRONMENTAL SCIENCES↗

ESS-DIVE Reporting Format for Sample-based Water and Soil Chemistry Measurements

The ESS-DIVE (Environmental Systems Science Data Infrastructure for a Virtual Ecosystem) reporting format for sample-based water and soil chemistry measurements are written guidelines and spreadsheet templates, that facilitate archiving water/soil/sediment chemistry data. The templates and terminology lists provided as part of the reporting format can help organize data and potentially enable data reuse.The reporting format consists of a general instructions file (instructions.md) as well as more detailed instructions files for each template (files beginning “Detailed_Instructions_*.md). The “examples” folder includes an example of each data template with just the limited set of required fields filled out as well as other examples with both required and optional fields complete.The 'templates' folder contains CSV templates for data, methods, and terminology file templates. Similar to the examples folder, the blank templates are provided with versions for required fields only as well as required and optional fields. Lastly, the term list folder provides common terminology used in each of the templates as well as a definition and any constraints for the fields.

54 ENVIRONMENTAL SCIENCES↗

ESS-DIVE Reporting Format for Dataset Package Metadata

ESS-DIVE’s (Environmental Systems Science Data Infrastructure for a Virtual Ecosystem) dataset metadata reporting format is intended to compile information about a dataset (e.g., title, description, funding sources) that can enable reuse of data submitted to the ESS-DIVE data repository. The files contained in this dataset include instructions (dataset_metadata_guide.md and README.md) that can be used to understand the types of metadata ESS-DIVE collects. The data dictionary (dd.csv) follows ESS-DIVE’s file-level metadata reporting format and includes brief descriptions about each element of the dataset metadata reporting format. This dataset also includes a terminology crosswalk (dataset_metadata_crosswalk.csv) that shows how ESS-DIVE’s metadata reporting format maps onto other existing metadata standards and reporting formats.Data contributors to ESS-DIVE can provide this metadata by manual entry using a web form or programmatically via ESS-DIVE’s API (Application Programming Interface). A metadata template (dataset_metadata_template.docx or dataset_metadata_template.pdf) can be used to collaboratively compile metadata before providing it to ESS-DIVE.Since being incorporated into ESS-DIVE’s data submission user interface, ESS-DIVE’s dataset metadata reporting format, has enabled features like automated metadata quality checks, and dissemination of ESS-DIVE datasets onto other data platforms including Google Dataset Search and DataCite.

54 ENVIRONMENTAL SCIENCES↗

Data from: “Enabling FAIR data in Earth and environmental science with community-centric (meta)data reporting formats”

This dataset contains supplementary information for a manuscript describing the ESS-DIVE (Environmental Systems Science Data Infrastructure for a Virtual Ecosystem) data repository's community data and metadata reporting formats. The purpose of creating the ESS-DIVE reporting formats was to provide guidelines for formatting some of the diverse data types that can be found in the ESS-DIVE repository. The 6 teams of community partners who developed the reporting formats included scientists and engineers from across the Department of Energy National Lab network. Additionally, during the development process, 247 individuals representing 128 institutions provided input on the formats. The primary files in this dataset are 10 data and metadata crosswalk for ESS-DIVE’s reporting formats (all files ending in _crosswalk.csv). The crosswalks compare elements used in each of the reporting formats to other related standards and data resources (e.g., repositories, datasets, data systems). This dataset also contains additional files recommended by ESS-DIVE’s file-level metadata reporting format. Each data file has an associated dictionary (files ending in _dd.csv) which provide a brief description of each standard or data resource consulted in the data reporting format development process. The flmd.csv file describes each file contained within the dataset.

54 ENVIRONMENTAL SCIENCES↗

2019 Computing Sciences Strategic Plan

Computing has transformed nearly every aspect of scientific inquiry — across disciplines and across scales — from the behavior of subatomic particles to the formation of structures in the early universe, from the assembly of the human genome to the evolution of earth systems. Over the past two decades, computing has become an integral part of how Berkeley Lab is “Bringing Science Solutions to the World.” Advances in computing and mathematics have been key, with new mathematical models of complex physical phenomena, new methods for analyzing complex data, new algorithms for accuracy and scaling and sophisticated software systems that encapsulate these techniques into open, reusable tools. The performance of NERSC computers and the ESnet network have grown by several orders of magnitude, along with our understanding of how to map scientific computations and workflows onto these systems. From research to facility operations, the passion, talent and dedication of the Computing Sciences Area staff has been the cornerstone of our success. The plan outlined in this document describes the next step in a journey to expand the influence and impact of our efforts, building an increasingly connected global enterprise for science that places more powerful instruments in the hands of scientists, along with more powerful methods and tools for modeling, analysis and prediction.

97 MATHEMATICS AND COMPUTING↗

Quality Assurance and Quality Control (QA/QC) of Meteorological Time Series Data for Billy Barr, East River, Colorado USA

A comprehensive Quality Assurance (QA) and Quality Control (QC) statistical framework consists of three major phases: Phase 1—Preliminary raw data sets exploration, including time formatting and combining datasets of different lengths and different time intervals; Phase 2—QA of the datasets, including detecting and flagging of duplicates, outliers, and extreme values; and Phase 3—the development of time series of a desired frequency, imputation of missing values, visualization and a final statistical summary. The time series data collected at the Billy Barr meteorological station (East River Watershed, Colorado) were analyzed. The developed statistical framework is suitable for both real-time and post-data-collection QA/QC analysis of meteorological datasets.The files that are in this data package include one excel file, converted to CSV format (Billy_Barr_raw_qaqc.csv) that contains the raw meteorological data, i.e., input data used for the QA/QC analysis. The second CSV file (Billy_Barr_1hr.csv) is the QA/QC and flagged meteorological data, i.e., output data from the QA/QC analysis. The last file (QAQC_Billy_Barr_2021-03-22.R) is a script written in R that implements the QA/QC and flagging process. The purpose of the CSV data files included in this package is to provide input and output files implemented in the R script.

54 ENVIRONMENTAL SCIENCES↗

Sample Identifiers and Metadata Reporting Format for Environmental Systems Science

The ESS-DIVE sample identifiers and metadata reporting format primarily follows the System for Earth Sample Registration (SESAR) Global Sample Number (IGSN) guide and template, with modifications to address Environmental Systems Science (ESS) sample needs and practicalities (IGSN-ESS). IGSNs are associated with standardized metadata to characterize a variety of different sample types (e.g. object type, material) and describe sample collection details (e.g. latitude, longitude, environmental context, date, collection method). Globally unique sample identifiers, particularly IGSNs, facilitate sample discovery, tracking, and reuse; they are especially useful when sample data is shared with collaborators, sent to different laboratories or user facilities for analyses, or distributed in different data files, datasets, and/or publications. To develop recommendations for multidisciplinary ecosystem and environmental sciences, we first conducted research on related sample standards and templates. We provide a comparison of existing sample reporting conventions, which includes mapping metadata elements across existing standards and Environment Ontology (ENVO) terms for sample object types and environmental materials. We worked with eight U.S. Department of Energy (DOE) funded projects, including those from Terrestrial Ecosystem Science and Subsurface Biogeochemical Research Scientific Focus Areas. Project scientists tested the process of registering samples for IGSNs and associated metadata in workflows for multidisciplinary ecosystem sciences. We provide modified IGSN metadata guidelines to account for needs of a variety of related biological and environmental samples. While generally following the IGSN core descriptive metadata schema, we provide recommendations for extending sample type terms, and connecting to related templates geared towards biodiversity (Darwin Core) and genomic (Minimum Information about any Sequence, MIxS) samples and specimens. ESS-DIVE recommends registering samples for IGSNs through SESAR, and we include instructions for registration using the IGSN-ESS guidelines. Our resulting sample reporting guidelines, template (IGSN-ESS), and identifier approach can be used by any researcher with sample data for ecosystem sciences.

54 ENVIRONMENTAL SCIENCES↗

Calibrated soil moisture data from Manaus, Brazil, August 2018 - December 2019

This dataset contains calibrated soil moisture and rainfall data for ecohydrological studies in the Central Amazon (2.6099 ºS, 60.2093 ºW) taken at 30-min intervals. Soil moisture data was collected at depths of 0.025-, 0.05-, 0.15-, 0.30-, 0.50-,0.8-, 1.6-, 2.4-, 3.2-, 4.8-, 6.4-, 8.8-, and 14.20 m. Inconsistent data was removed during the QA/QC process. Further information including data collection methods are available in the associated publication, Negron-Juarez et al. 2020. Measurement and characterization of soil moisture dynamics in a Central Amazonian tropical forest (to be submitted).

54 ENVIRONMENTAL SCIENCES↗

San Lorenzo meteorological drivers

QA/QC-ed datasets of Temperature, Solar Radiation, Relative Humidity, Rainfall, Wind Speed, and Wind Direction for the period of 2008-2017.

54 ENVIRONMENTAL SCIENCES↗

Panama Land Model (CLM/ELM) Site Drivers

This data package contains meteorological and surface data at the Barro Colorado Island (BCI) and Bosque Protector San Lorenzo (Fort Sherman) NGEE Tropics sites in Panama. Data has been converted into formats usable by the land-models driving the Functionally-Assembled Terrestrial Ecosystem Simulator (FATES). Surface datasets are model release dependent and may have to be re-processed for later versions of the model. Current surface datasets based off of "<>_16pfts_Irrig_CMIP6_simyr2000_c170824.nc" See the README_ngeet_lm_driver text file in the attached zip file for detailed descriptions of the included data files.

54 ENVIRONMENTAL SCIENCES↗

The East River, Colorado, Watershed: A Mountainous Community Testbed for Improving Predictive Understanding of Multiscale Hydrological–Biogeochemical Dynamics

Core Ideas Development of a 300‐km 2 mountainous headwater testbed began in 2016 in the East River. The testbed can be used to explore how watershed changes impact downgradient water availability and quality. System‐of‐system, scale‐adaptive approaches can potentially improve watershed dynamics simulation. We have new approaches to monitor and simulate water partitioning and system responses. The East River watershed has been developed as a “community” testbed. Extreme weather, fires, and land use and climate change are significantly reshaping interactions within watersheds throughout the world. Although hydrological–biogeochemical interactions within watersheds can impact many services valued by society, uncertainty associated with predicting hydrology‐driven biogeochemical watershed dynamics remains high. With an aim to reduce this uncertainty, an approximately 300‐km 2 mountainous headwater observatory has been developed at the East River, CO, watershed of the Upper Colorado River Basin. The site is being used as a testbed for the Department of Energy supported Watershed Function Project and collaborative efforts. Building on insights gained from research at the “sister” Rifle, CO, site, coordinated studies are underway at the East River site to gain a predictive understanding of how the mountainous watershed retains and releases water, nutrients, carbon, and metals. In particular, the project is exploring how early snowmelt, drought, and other disturbances influence hydrological–biogeochemical watershed dynamics at seasonal to decadal timescales. A system‐of‐systems perspective and a scale‐adaptive simulation approach, involving the combined use of archetypal watershed subsystem “intensive sites” are being tested at the site to inform aggregated watershed predictions of downgradient exports. Complementing intensive site hydrological, geochemical, geophysical, microbiological, geological, and vegetation datasets are long‐term, distributed measurement stations and specialized experimental and observational campaigns. Several recent research advances provide insights about the intensive sites as well as aggregated watershed behavior. The East River “community testbed” is currently hosting scientists from more than 30 institutions to advance mountainous watershed methods and understanding.

54 ENVIRONMENTAL SCIENCES↗

Connecting AmeriFlux to the Globe, Extending the Partnership with Global Flux Network FLUXNET

We propose a project that will extend and expand the activities and contributions of AmeriFlux, to the global stage by working with our international partners to continue operating and expand the functioning of global FLUXNET database. One goal is to produce the next-generation FLUXNET dataset. This will be an integrated dataset on carbon, water and energy fluxes, on meteorological conditions and on the structure and function of the ecosystems; we have to potential to construct a new database, double the size of the La Thuile dataset, with over 2000 site-years of trace gas flux measurements, and supporting meteorological and site meta-data from over 400 sites world-wide. The production of the next FLUXNET database is critical to the activities of this ROSE’s solicitation because these data will drive many of the carbon cycle synthesis activities and the model parameterization and validation activities. A second goal is to build the next-generation, data-system that will house, distribute and query data and discover new information by the broader biogeoscience user community. The third goal is to use these data in synthesis activities that improve our understanding on how climate, weather, ecological, physiological and soil factors govern the exchange of carbon, water and energy between vegetation and the atmosphere at multiple time and space scales. The fourth goal is to act as a coordinator of the project to recruit data, to organize a workshop/conference and to build a community of data sharing and interaction across vast cultural spectrum of participating scientists. The rationale for proposing this work is based on the fact that the vast databases that are produced by flux networks cannot perpetuate and evolve without continued support of the intellectual activities of scientists who produce data, expert in databases, and scientists who use data for modeling. The actuality of a flux network takes human intervention to recruit data from different countries and cultures, to build trust to share data in open-access format, to support data users, and to collaborate. Specifically, we will develop an updated data portal that is easy to navigate and conducive for distributing, visualization, evaluation and discovery of the multiple types of flux and environmental data and information. We will foster collaboration and interactions with our national and international partners and the recruitment of data by: 1) continuing the periodic publication of the newsletter, FLUXLETTER; 2) by hosting an international conference; 3) by coordinating and launching multi-investigator data synthesis activities; and 4) by developing tools that coordinate the development of research papers resulting from the dataset. Data products include raw and gap-filled data at their native time steps (30 minute averages), data products that are integrated on daily and annual time scales and value added products like ecosystem respiration and canopy photosynthesis. Uncertainty analysis and quality control metrics will be produced in coordination with AmeriFlux and ICOS. We will also work with colleagues to expand the scope of FLUXNET beyond carbon dioxide and water vapor, and start archiving the growing methane flux databases that are being collected, world-wide. The interpretation of the flux data relies on good meta-data. We will work with data providers to populate and expand the meta-data archives. We will also work with collaborators to incorporate data from remote sensing products such as MODIS and Landsat, from corresponding climate and weather stations. Another goal is to incorporate new data on canopy structure, with recent LIDAR acquisitions, to populate the data archive with automated soil respiration and understory flux measurements, where available and local climate records for corresponding sites. And, we will collaborate with the TRY database, by merging flux data with plant trait information.

54 ENVIRONMENTAL SCIENCES↗

FRAMES Metadata Reporting Templates for Ecohydrological Observations, version 1.1

FRAMES is a a set of Excel metadata files and package-level descriptive metadata that are designed to facilitate and improve capture of desired metadata for ecohydrological observations. The metadata are bundled with data files into a data package and submitted to a data repository (e.g. the NGEE Tropics Data Repository) via a web form. FRAMES standardizes reporting of diverse ecohydrological and biogeochemical data for synthesis across a range of spatiotemporal scales and incorporates many best data science practices. This version of FRAMES supports observations for primarily automated measurements collected by permanently located sensors, including sap flow (tree water use), leaf surface temperature, soil water content, dendrometry (stem diameter growth increment), and solar radiation. Version 1.1 extend the controlled vocabulary and incorporates functionality to facilitate programmatic use of data and FRAMES metadata (R code available at NGEE Tropics Data Repository).

54 ENVIRONMENTAL SCIENCES↗

QA/QC-ed BCI meteorological drivers

Time series raw datasets of Temperature, Solar Radiation, Relative Humidity, Rainfall, Barometric Pressure, Wind Speed, and Wind Direction for the Barro Colorado (PA-BCI) site for the period from 2003-01-01 to 2016-12-31 were downloaded from the STRI website. The QA/QC protocol of raw datasets included the detection and removal of NAs, outliers and bad data points, imputation of missing data, and creation of the time series with equidistant time stamps. The time series for different meteorological parameters were aligned into a single database to be used for modeling. Statistical QA/QC analysis was performed using a series of R libraries in Rstudio. The changes in the dataset were flagged. The file BCI_met_drivers_2003-2016_QAQC_summary_report.docx provides the details of the QA/QC data analysis.

54 ENVIRONMENTAL SCIENCES↗

Hunting Data Rogues at Scale: Data Quality Control for Observational Data in Research Infrastructures

Data quality control is one of the most time consuming activities within Research Infrastructures (RIs), especially when involving observational data and multiple data providers. In this work we report on our ongoing development of data rogues, a scalable approach to manage data quality issues for observational data within RIs. The motivation for this work started with the creation of the FLUXNET2015 dataset, which includes carbon, water, and energy fluxes plus micrometeorological and ancillary data measured in over 200 sites around the world. To create an uniform dataset, including derived data products, extensive work on data quality control was needed. The unpredictable nature of observational data quality issues makes the automation of data quality control inherently difficult. Developed based on this experience, the data rogues methodology allows for increased automation of quality control activities by systematically identifying, cataloging, and documenting implementations of solutions to data issues. We believe this methodology can be extended and applied to others domains and types of data, making the automation of data quality control a more tractable problem.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Building a Cyberinfrastructure for Environmental System Science: Modeling Frameworks, Data Management, and Scientific Workflows, Workshop Report, Potomac, Maryland, April 30-May 1, 2015

The Environmental System Science (ESS) activity within the Department of Energy’s (DOE) Office of Biological and Environmental Research (BER) seeks to advance a robust, predictive understanding of terrestrial environments, extending from bedrock to the top of the vegetative canopy and from molecular to global scales, through an iterative cycle of model-driven experimentation and observation dubbed MODEX. Considerable progress has been made toward achieving this overarching goal, but widely recognized is the fragmentation across projects and disciplines of the relevant modeling and simulation capabilities, observational and experimental data, analysis algorithms, and workflow tools. This fragmentation creates significant challenges for studying impacts and feedbacks in these complex multiscale systems. These challenges are further exacerbated by ongoing disruptive changes in high-performance computational architectures and the exponential growth in the types and volume of data that render obsolete the conventional approaches to software development and data management. Overcoming these challenges will require the development of a BER Climate and Environmental Sciences Division (CESD)–wide enabling cyberinfrastructure to support data management, cross-domain modeling, data analysis, and collaborative research. To explore the potential for working groups to initiate and guide a more integrated and community-based cyberinfrastructure, BER held the ESS Workshop on Model-Data Integration: Modeling Frameworks, Data Management, and Scientific Workflows on April 30-May 1, 2015, following the ESS Principal Investigator Meeting in Potomac, Maryland. Participants included model developers, software engineers, and data management specialists from eight national laboratories, which represented a wide range of projects and programs from CESD as well as cross-cutting projects from DOE’s Office of Advanced Scientific Computing Research (ASCR; both BER and ASCR are operated from within DOE’s Office of Science). A series of plenary talks provided background information and clarified the workshop’s three objectives: (1) develop requirements for this community-based cyberinfrastructure to ensure enhanced scientific productivity of the community as a whole; (2) identify challenges associated with developing this new cyberinfrastructure using a phased approach guided by project-driven use cases; and, given these requirements and challenges, (3) chart a path forward for ESS working groups to lead the phased development of the new cyberinfrastructure.

54 ENVIRONMENTAL SCIENCES↗