Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Observation Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

Hunting Data Rogues at Scale: Data Quality Control for Observational Data in Research Infrastructures

Data quality control is one of the most time consuming activities within Research Infrastructures (RIs), especially when involving observational data and multiple data providers. In this work we report on our ongoing development of data rogues, a scalable approach to manage data quality issues for observational data within RIs. The motivation for this work started with the creation of the FLUXNET2015 dataset, which includes carbon, water, and energy fluxes plus micrometeorological and ancillary data measured in over 200 sites around the world. To create an uniform dataset, including derived data products, extensive work on data quality control was needed. The unpredictable nature of observational data quality issues makes the automation of data quality control inherently difficult. Developed based on this experience, the data rogues methodology allows for increased automation of quality control activities by systematically identifying, cataloging, and documenting implementations of solutions to data issues. We believe this methodology can be extended and applied to others domains and types of data, making the automation of data quality control a more tractable problem.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2024

Merged Observatory Data Files (MODFs): an integrated observational data product supporting process-oriented investigations and diagnostics

A large and ever-growing body of geophysical information is measured in campaigns and at specialized observatories as a part of scientific expeditions and experiments. These collections of observed data include many essential climate variables (as defined by the Global Climate Observing System) but are often distinguished by a wide range of additional non-routine measurements that are designed to not only document the state of the environment but also the drivers that contribute to that state. These field data are used not only to further understand environmental processes through observation-based studies but also to provide baseline data to test model performance and to codify understanding to improve predictive capabilities. To address the considerable barriers and difficulty in utilizing these diverse and complex data for observation–model research, the Merged Observatory Data File (MODF) concept has been developed. A MODF combines measurements from multiple instruments into a single file that complies with well-established data format and metadata practices and has been designed to parallel the development of corresponding Merged Model Data Files (MMDFs). Using the MODF and MMDF protocols will facilitate the evolution of model intercomparison projects into model intercomparison and improvement projects by putting observation and model data “on the same page” in a timely manner. The MODF concept was developed especially for weather forecast model studies in the Arctic. The surprisingly complex process of implementing MODFs in that context refined the concept itself. Thus, this article explains the concept of MODFs by providing details on the issues that were revealed and resolved during that first specific implementation. Detailed instructions are provided on how to make MODFs, and this article can be considered a MODF creation manual.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Data and scripts from: “Denoising autoencoder for reconstructing sensor observation data and predicting evapotranspiration: noisy and missing values repair and uncertainty quantification”

This data package includes data and scripts from the manuscript “Denoising autoencoder for reconstructing sensor observation data and predicting evapotranspiration: noisy and missing values repair and uncertainty quantification”.The study addressed common challenges faced in environmental sensing and modeling, including uncertain input data, missing sensor observations, and high-dimensional datasets with interrelated but redundant variables. Point-scaled meteorological and soil sensor observations were perturbed with noises and missing values, and denoising autoencoder (DAE) neural networks were developed to reconstruct the perturbed data and further predict evapotranspiration. This study concluded that (1) the reconstruction quality of each variable depends on its cross-correlation and alignment to the underlying data structure, (2) uncertainties from the models were overall stronger than those from the data corruption, and (3) there was a tradeoff between reducing bias and reducing variance when evaluating the uncertainty of the machine learning models.This package includes:(1) Four ipython scripts (.ipynb): “DAE_train.ipynb” trains and evaluates DAE neural networks, “DAE_predict.ipynb” makes predictions from the trained DAE models, “ET_train.ipynb” trains and evaluates ET prediction neural networks, and “ET_predict.ipynb” makes predictions from trained ET models.(2) One python file (.py): “methods.py” includes all user-defined functions and python codes used in the ipython scripts.(3) A “sub_models” folder that includes five trained DAE neural networks (in pytorch format, .pt), which could be used to ingest input data before being fed to the downstream ET models in ‘ET_train.ipynb” or ‘ET_predict.ipynb’.(4) Two data files (.csv). Daily meteorological, vegetation, and soil data is in “df_data.csv”, where “df_meta.csv” contains the location and time information of “df_data.csv”. Each row (index) in “df_meta.csv” corresponds to each row in “df_data.csv”. These data files are formatted to follow the data structure requirements and be directly used in the ipython scripts, and they have been shuffled chronologically to train machine learning models. The meteorological and soil data was collected using point sensors between 2019-2023 at(4.a) Three shrub-dominated field sites in East River, Colorado (named “ph1”, “ph2” and “sg5” in “df_meta.csv”, where “ph1” and “ph2” were located at PumpHouse Hillslopes, and “sg5” was at Snodgrass Mountain meadow) and(4.b) One outdoor, mesoscale, and herbaceous-dominated experiment in Berkeley, California (named “tb” in “df_meta.csv”, short for Smartsoils Testbed at Lawrence Berkeley National Lab).- See "df_data_dd.csv" and "df_meta_dd.csv" for variable descriptions and the Methods section for additional data processing steps. See "flmd.csv" and "README.txt" for brief file descriptions.- All ipython scripts and python files are written in and require PYTHON language software.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Comparison of Social Media, Syndromic Surveillance, and Microbiologic Acute Respiratory Infection Data: Observational Study

Internet data can be used to improve infectious disease models. However, the representativeness and individual-level validity of internet-derived measures are largely unexplored as this requires ground truth data for study. This study sought to identify relationships between Web-based behaviors and/or conversation topics and health status using a ground truth, survey-based dataset. This study leveraged a unique dataset of self-reported surveys, microbiological laboratory tests, and social media data from the same individuals toward understanding the validity of individual-level constructs pertaining to influenza-like illness in social media data. Logistic regression models were used to identify illness in Twitter posts using user posting behaviors and topic model features extracted from users’ tweets. Of 396 original study participants, only 81 met the inclusion criteria for this study. Of these participants’ tweets, we identified only two instances that were related to health and occurred within 2 weeks (before or after) of a survey indicating symptoms. It was not possible to predict when participants reported symptoms using features derived from topic models (area under the curve [AUC]=0.51; P=.38), though it was possible using behavior features, albeit with a very small effect size (AUC=0.53; P≤.001). Individual symptoms were also generally not predictable either. The study sample and a random sample from Twitter are predictably different on held-out data (AUC=0.67; P≤.001), meaning that the content posted by people who participated in this study was predictably different from that posted by random Twitter users. Individuals in the random sample and the GoViral sample used Twitter with similar frequencies (similar @ mentions, number of tweets, and number of retweets; AUC=0.50; P=.19). To our knowledge, this is the first instance of an attempt to use a ground truth dataset to validate infectious disease observations in social media data. The lack of signal, the lack of predictability among behaviors or topics, and the demonstrated volunteer bias in the study population are important findings for the large and growing body of disease surveillance using internet-sourced data.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2016

Confronting Weather and Climate Models with Observational Data from Soil Moisture Networks over the United States

Four land surface models in uncoupled and coupled configurations are compared to observations of daily soil moisture from 19 networks in the conterminous United States to determine the viability of such comparisons and explore the characteristics of model and observational data. First, observations are analyzed for error characteristics and representation of spatial and temporal variability. Some networks have multiple stations within an area comparable to model grid boxes; for those it is found that aggregation of stations before calculation of statistics has little effect on estimates of variance, but soil moisture memory is sensitive to aggregation. Statistics for some networks stand out as unlike those of their neighbors, likely because of differences in instrumentation, calibration, and maintenance. Buried sensors appear to have less random error than near-field remote sensing techniques, and heat-dissipation sensors show less temporal variability than other types. Model soil moistures are evaluated using three metrics: standard deviation in time, temporal correlation (memory), and spatial correlation (length scale). Models do relatively well in capturing large-scale variability of metrics across climate regimes, but they poorly reproduce observed patterns at scales of hundreds of kilometers and smaller. Uncoupled land models do no better than coupled model configurations, nor do reanalyses outperform free-running models. Spatial decorrelation scales are found to be difficult to diagnose. Using data for model validation, calibration, or data assimilation from multiple soil moisture networks with different types of sensors and measurement techniques requires great caution. Finally, data from models and observations should be put on the same spatial and temporal scales before comparison.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Towards Lightweight Data Integration Using Multi-Workflow Provenance and Data Observability

Modern large-scale scientific discovery requires multidisciplinary collaboration across diverse computing facilities, including High Performance Computing (HPC) machines and the Edge-to-Cloud continuum. Integrated data analysis plays a crucial role in scientific discovery, especially in the current AI era, by enabling Responsible AI development, FAIR, Reproducibility, and User Steering. However, the heterogeneous nature of science poses challenges such as dealing with multiple supporting tools, cross-facility environments, and efficient HPC execution. Building on data observability, adapter system design, and provenance, we propose MIDA: an approach for lightweight runtime Multi-workflow Integrated Data Analysis. MIDA defines data observability strategies and adaptability methods for various parallel systems and machine learning tools. With observability, it intercepts the dataflows in the background without requiring instrumentation while integrating domain, provenance, and telemetry data at runtime into a unified database ready for user steering queries. We conduct experiments showing end-to-end multi-workflow analysis integrating data from Dask and MLFlow in a real distributed deep learning use case for materials science that runs on multiple environments with up to 276 GPUs in parallel. We show near-zero overhead running up to 100,000 tasks on 1,680 CPU cores on the Summit supercomputer.

Santos Souza, Renan↗
DOE OSTI2020

Policy Considerations When Federating Facilities for Experimental and Observational Data Analysis

Today’s computational, experimental, and observational facilities afford us tremendous opportunities to couple theory and experiment at increasingly large scales. Empirical sensing capabilities are growing dramatically with beam line and detector improvements, and with advances in our ability to deploy large-scale data gathering observations of the natural world. The coupling of computational simulations and analysis to process the data from experimental and observational facilities is giving rise to cross-facility workflows. Such federations of facilities are in fact becoming an explicit requirement for large-scale scientific discovery. As we scale up these pipelines of scientific discovery, each participating facility needs to establish and align policies so that the federation can work seamlessly in an end-to-end manner. This chapter outlines specific policy considerations in enabling the federation of facilities for data analysis. Design choices and vital policy decisions cover the areas of data acquisition and storage, data transfer, computational resource allocation and co-scheduling, seamless federated user access, and cross-cutting governance. By highlighting the explicit and implicit interdependencies between facilities, we aim to provide facility designers and policymakers the information on policy issues to address early in a facility’s operations, thus enabling successful cross-facility federation and improved experimental and observational data analysis outcomes.

Shankar, Mallikarjun (Arjun)↗
DOE OSTI2022

Interaction between cloud–radiation, atmospheric dynamics and thermodynamics based on observational data from GoAmazon 2014/15 and a cloud-resolving model

Observational meteorological data from the field experiment GoAmazon 2014/15 and data from numerical simulations with the cloud-resolving model (CRM) called the System for Atmospheric Modeling (SAM) are used to study the interaction between the cloudiness–radiation as well as the atmospheric dynamics and thermodynamics variables for a site located in the central Amazon region (–3.2°S, –60.6°W) during the wet and dry periods. The main aims are to (a) analyze the temporal series of the integrated cloud fraction, precipitation rate and downward shortwave flux as well as (b) to determine the relationship between the integrated cloud fraction, radiative fluxes and large-scale variable anomalies as a function of the previous day's average. The temporal series of the integrated cloud fraction, precipitation rate and downward shortwave flux from SAM simulations showed physical consistency with the observations from GoAmazon 2014/15. Shallow and deep convection clouds show to have a meaningful impact on radiation fluxes in the Amazon region during wet and dry periods. Anomalies of large-scale variables (relative to the previous day's average) are physically associated with cloud formation, evolution and dissipation. SAM consistently simulated these results, where the cloud fraction vertical profile shows a pattern very close to the observed data (cloud type). Additionally, the integrated cloud fraction and large-scale variable anomalies, as a function of the previous day's average, have a good correlation. These results suggest that the memory of the large-scale dynamics from the previous day can be used to estimate the cloud fraction as well as the water content, which is a variable of the cloud itself. In general, the SAM satisfactorily simulated the interaction between cloud–radiation as well as dynamic and thermodynamic variables of the atmosphere during the periods of this study, being able to obtain atmospheric variables that are impossible to obtain in an observational way.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2016

Satellite Earth observation data to identify climate and anthropogenic pressures on Bucharest periurban forests

Satellite Earth observation data in the visible and near-infrared (VNIR) wavelengths represent a useful source of information for forest systems monitoring through derived biogeophysical parameters (vegetation index, leaf area index, canopy cover, fraction of absorbed photosynthetically active radiation, chlorophyll content, net primary production, canopy water stress, etc.). Use of satellite remote sensing data to assess forest spatio-temporal changes due to climatic or anthropogenic stressors is an excellent example of the value of multispectral and multitemporal observations. Fusion technique was applied to time-series multispectral and multitemporal satellite imagery (NOAA AVHRR, MODIS Terra/Aqua, Landsat ETM and IKONOS satellite data) for periurban forest areas Cernica-Branesti, placed in the neighboring of Bucharest town, Romania, over 2002-2014 period.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

Using the Tilted flat-ΛCDM and the Untilted Non-flat ΛCDM Inflation Models to Measure Cosmological Parameters from a Compilation of Observational Data

We use the physically consistent tilted spatially flat and untilted non-flat ΛCDM inflation models to constrain cosmological parameter values with the Planck 2015 cosmic microwave background (CMB) anisotropy data and recent SNe Ia measurements, baryonic acoustic oscillations (BAO) data, growth rate observations, and Hubble parameter measurements. The most dramatic consequence of including the four non-CMB data sets is the significant strengthening of the evidence for non-flatness in the non-flat ΛCDM model, from 1.8σ for the CMB data alone to 5.1σ for the full data combination. The BAO data is the most powerful of the non-CMB data sets in more tightly constraining model-parameter values and in favoring a spatially closed universe in which spatial curvature contributes about a percent to the current cosmological energy budget. The untilted non-flat ΛCDM model better fits the large-angle CMB temperature anisotropy angular spectrum and is more consistent with the Dark Energy Survey constraints on the current value of the rms amplitude of mass fluctuations (σ 8 ) as a function of the current value of the nonrelativistic matter-density parameter (Ω m ) but does not provide as good a fit to the smaller-angle CMB temperature anisotropy data, as does the tilted flat-ΛCDM model. Some measured cosmological parameter values differ significantly between the two models, including the reionization optical depth and the baryonic matter density parameter, both of whose 2σ ranges (in the two models) are disjointed or almost so.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

Investigating the contribution of grown new particles to cloud condensation nuclei with largely varying preexisting particles – Part 1: Observational data analysis

This study employed multiple techniques to investigate the contribution of grown new particles to the number concentration of cloud condensation nuclei (CCN) at various supersaturation (SS) levels at a rural mountain site in the North China Plain from 29 June to 14 July 2019. On 8 new particle formation (NPF) days, the total particle number concentrations (N cn ) were 8.4 ± 6.1×10 3 cm -3 , which was substantially higher compared to 4.7 ± 2.6×10 3 cm -3 on non-NPF days. However, the CCN concentration (Nccn) at 0.2 % SS and 0.4 % SS on the NPF days was significantly lower than those observed on non-NPF days (p<0.05). This was due to the lower cloud activation efficiency of preexisting particles resulting from organic vapor condensation and smaller number concentrations of preexisting particles on NPF days. A case-by-case examination showed that the grown new particles only yielded a detectable contribution to N ccn at 0.4 % SS and 1.0 % SS during the NPF event on 1 July 2019, accounting for 12 % ± 11 % and 23 % ± 12 % of N ccn , respectively. The increased N ccn during two other NPF events and at 0.2 % SS on 1 July 2019 were detectable but determined mainly by varying preexisting particles rather than grown new particles. In addition, the hygroscopicity parameter values, concentrations of inorganic and organic particulate components, and surface chemical composition of different sized particles were analyzed in terms of chemical drivers to grow new particles. The results showed that the grown new particles via organic vapor condensation generally had no detectable contribution to N ccn but incidentally did. However, this conclusion was drawn from a small size of observational data, leaving more observations, particularly long-term observations and the growth of preexisting particles to the CCN required size, needed for further investigation.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2016

Report of the DOE Workshop on Management, Analysis, and Visualization of Experimental and Observational data – The Convergence of Data and Computing

The Department of Energy (DOE) Office of Science (SC) operates dozens of national science user facilities that span many different disciplines. These facilities include accelerators, colliders, supercomputers, light sources and neutron sources, as well as facilities for studying the nanoworld, the environment, and the atmosphere. In Fiscal Year 2014 over 33,000 researchers from academia, industry, and government laboratories, spanning all fifty states and the District of Columbia, utilized these unique facilities to perform new scientific research. Each of these facilities generates vast amounts of scientific data, and the rate, size, and complexity of this data is rapidly increasing, thanks to advances in technology. A growing concern, which motivates this workshop, is the likely significant adverse impact on science programs that will result without significant advances in the capabilities needed to manage and gain knowledge from these collections of data. The purpose of this workshop, held 29 September 2015 through 1 October 2015 in Bethesda, MD, is to help the Advanced Scientific Computing Research (ASCR) and research community better understand needs related to the management, analysis, and visualization of experimental and observational data (EOD) collected and generated by experimental and observational science projects (EOS) at Office of Science user facilities.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Significant inconsistency of vegetation carbon density in CMIP5 Earth system models against observational data

Abstract Earth system models (ESMs) have been widely used for projecting global vegetation carbon dynamics, yet how well ESMs performed for simulating vegetation carbon density remains untested. We compiled observational data of vegetation carbon density from literature and existing data sets to evaluate nine ESMs at site, biome, latitude, and global scales. Three variables—root (including fine and coarse roots), total vegetation carbon density, and the root:total vegetation carbon ratios (R/T ratios), were chosen for ESM evaluation. ESM models performed well in simulating the spatial distribution of carbon densities in root ( r = 0.71) and total vegetation ( r = 0.62). However, ESM models had significant biases in simulating absolute carbon densities in root and total vegetation biomass across the majority of land ecosystems, especially in tropical and arctic ecosystems. Particularly, ESMs significantly overestimated carbon density in root (183%) and total vegetation biomass (167%) in climate zones of 10°S–10°N. Substantial discrepancies between modeled and observed R/T ratios were found: the R/T ratios from ESMs were relatively constant, approximately 0.2 across all ecosystems, along latitudinal gradients, and in tropic, temperate, and arctic climatic zones, which was significantly different from the observed large variations in the R/T ratios (0.1–0.8). There were substantial inconsistencies between ESM‐derived carbon density in root and total vegetation biomass and the R/T ratio at multiple scales, indicating urgent needs for model improvements on carbon allocation algorithms and more intensive field campaigns targeting carbon density in all key vegetation components.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2016

Zeldovich pancakes in observational data are cold

The present day universe consists of galaxies, galaxy clusters, one-dimensional filaments and two-dimensional sheets or pancakes, all of which combine to form the cosmic web. The so called ''Zeldovich pancakes' are very difficult to observe, because their overdensity is only slightly greater than the average density of the universe. Falco et al. [1] presented a method to identify Zeldovich pancakes in observational data, and these were used as a tool for estimating the mass of galaxy clusters. Here we expand and refine that observational detection method. We study two pancakes on scales of 10 Mpc, identified from spectroscopically observed galaxies near the Coma cluster, and compare with twenty numerical pancakes.We find that the observed structures have velocity dispersions of about 100 km/sec, which is relatively low compared to typical groups and filaments. These velocity dispersions are consistent with those found for the numerical pancakes. We also confirm that the identified structures are in fact two-dimensional structures. Finally, we estimate the stellar to total mass of the observational pancakes to be 2 · 10{sup −4}, within one order of magnitude, which is smaller than that of clusters of galaxies.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2016

OBSERVATION AND CONFIRMATION OF SIX STRONG-LENSING SYSTEMS IN THE DARK ENERGY SURVEY SCIENCE VERIFICATION DATA

We report the observation and confirmation of the first group- and cluster-scale strong gravitational lensing systems found in Dark Energy Survey data. Through visual inspection of data from the Science Verification season, we identified 53 candidate systems. We then obtained spectroscopic follow-up of 21 candidates using the Gemini Multi-object Spectrograph at the Gemini South telescope and the Inamori-Magellan Areal Camera and Spectrograph at the Magellan/Baade telescope. With this follow-up, we confirmed six candidates as gravitational lenses: three of the systems are newly discovered, and the remaining three were previously known. Of the 21 observed candidates, the remaining 15 either were not detected in spectroscopic observations, were observed and did not exhibit continuum emission (or spectral features), or were ruled out as lensing systems. The confirmed sample consists of one group-scale and five galaxy-cluster-scale lenses. The lensed sources range in redshift z ∼ 0.80–3.2 and in i -band surface brightness i {sub SB} ∼ 23–25 mag arcsec{sup −2} (2″ aperture). For each of the six systems, we estimate the Einstein radius θ {sub E} and the enclosed mass M {sub enc}, which have ranges θ {sub E} ∼ 5″–9″ and M {sub enc} ∼ 8 × 10{sup 12} to 6 × 10{sup 13} M {sub ⊙}, respectively.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Assessment of Storm-Associated Precipitation and Its Extremes Using Observational Data Sets and Climate Model Short-Range Hindcasts

Heavy precipitation, often associated with weather phenomena such as tropical cyclones, extratropical cyclones (ETCs), atmospheric rivers (ARs), and mesoscale convective systems (MCSs), can cause significant socio-economic loss. Here, in this study, we apply atmospheric feature trackers to quantify the contributions of these storm types in observational data sets and climate model short-range hindcasts. We generate a global hourly storm data set at 0.25° spatial resolution covering 2006–2020, based on the tracking results from TempestExtremes and Python FLEXible object TRacKeR. Our analyses show that these four storm types account for 67% of global annual mean precipitation and 82% of top 1% precipitation extremes, with MCSs mainly over the tropics, and ARs and ETCs over the midlatitudes. The percentage of precipitation contributions from these storms also show strong seasonality over many geographical locations. We further apply the tracking results to the Energy Exascale Earth System Model (E3SM) short-range hindcasts and evaluate how well these storms are simulated. The evaluation show that E3SM, with ∼1° resolution, significantly underestimates storm-associated precipitation totals and extremes, especially for MCSs in the tropics. Our analysis also suggests that model fails to capture the correct mean diurnal phases and amplitude of MCS precipitation. This phenomenon-based approach provides a better understanding of precipitation characteristics and can lead to enhanced model evaluation by revealing underlying problems in model physics related to precipitation processes associated with the heavy-precipitating storms.

54 ENVIRONMENTAL SCIENCES↗