Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data bias”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 577 records · Page 32
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

The influence of cloud cover on the reliability of satellite-based solar resource data

Satellite-based solar resource data are often developed and validated by using binary cloudiness categories: clear sky or overcast cloudy sky. To investigate the reliability of solar resource data in partially cloudy conditions, we estimate cloud fraction using two distinct algorithms: a physical retrieval model using surface observed global horizontal irradiance (GHI) and direct normal irradiance (DNI) and a temporal average of cloud mask data estimated by the observed DNI. Our analysis reveals a significant presence of scattered clouds, broken clouds, and mismatches between satellite- and surface-based cloud data at 17 surface sites across the contiguous United States, though confidently clear and cloudy conditions collectively account for more than 70 % of the data. Solar radiation is computed using the National Solar Radiation Database (NSRDB) algorithm and validated using surface observations. Here, our findings suggest that, in the presence of scattered clouds, NSRDB data for clear-sky conditions can be subject to significant overestimation. In cloudy-sky conditions classified by satellite data, DNI computed by the Fast All-sky Radiation Model for Solar applications with DNI (FARMS-DNI) can be underestimated when limited clouds are detected by surface observations. The bias observed in several cloudiness categories indicates that the NSRDB is exceptionally accurate in confidently clear conditions. However, clear-sky conditions with scattered clouds and mismatched cloud data contribute significantly to the overall uncertainties in the NSRDB. Therefore, future improvements in solar resource data should involve development and implementation of satellite-derived cloud fraction and should consider a novel radiative transfer model accounting for amplified cloud reflection. The evaluation within cloudiness categories also provides a physical rationale for the superior performance of FARMS-DNI compared to the Direct Insolation Simulation Code (DISC) in both cloudy-sky and all-sky conditions.

14 SOLAR ENERGY↗
DOE OSTI2020

Can machine learning improve the model representation of turbulent kinetic energy dissipation rate in the boundary layer for complex terrain?

Current turbulence parameterizations in numerical weather prediction models at the mesoscale assume a local equilibrium between production and dissipation of turbulence. As this assumption does not hold at fine horizontal resolutions, improved ways to represent turbulent kinetic energy (TKE) dissipation rate (ϵ) are needed. Here, we use a 6-week data set of turbulence measurements from 184 sonic anemometers in complex terrain at the Perdigão field campaign to suggest improved representations of dissipation rate. First, we demonstrate that the widely used Mellor, Yamada, Nakanishi, and Niino (MYNN) parameterization of TKE dissipation rate leads to a large inaccuracy and bias in the representation of ϵ. Next, we assess the potential of machine-learning techniques to predict TKE dissipation rate from a set of atmospheric and terrain-related features. We train and test several machine-learning algorithms using the data at Perdigão, and we find that the models eliminate the bias MYNN currently shows in representing ϵ, while also reducing the average error by up to almost 40 %. Of all the variables included in the algorithms, TKE is the variable responsible for most of the variability of ϵ, and a strong positive correlation exists between the two. These results suggest further consideration of machine-learning techniques to enhance parameterizations of turbulence in numerical weather prediction models.

58 GEOSCIENCES↗
DOE OSTI2021

Spatial biases of information influence global estimates of soil respiration: how can we improve global predictions?

Soil respiration (Rs), the efflux of CO2 from soils to the atmosphere, is a major component of the terrestrial carbon cycle, but is poorly constrained from regional to global scales. The global soil respiration database (SRDB) is a compilation of in-situ Rs observations from around the globe that has been consistently updated with new measurements over the past decade. It is unclear whether the addition of data to new versions of the SRDB has produced better-constrained global Rs estimates. We compared two versions of the SRDB (v3.0 n=5173 and v5.0 n=10366) to determine how additional data influenced the global sum and uncertainty of Rs as predicted by a quantile regression forest model. A model parameterized using SRDBv3 yielded a global Rs sum of 88.6 Pg C yr-1, and associated uncertainty of 29.9 (mean absolute error) and 57.9 (standard deviation) Pg C yr-1, whereas parameterization using SRDBv5 yielded 96.5 Pg C yr-1 and associated uncertainty of 30.2 (mean average error) and 73.4 (standard deviation) Pg C yr-1. Empirically estimated global heterotrophic respiration (Rh) from v3 and v5 were 49.9-50.2 (mean 50.1) and 53.3-53.5 (mean 53.4) Pg C yr-1, respectively. Interestingly, SRDBv5’s inclusion of new data from underrepresented regions (e.g., Asia, Africa, South America) thus resulted in overall higher model uncertainty. The SRDBv5 is still biased towards northern latitudes and temperate zones, so we tested that an optimized global distribution of Rs measurements resulted in a global sum of 96.4 ± 21.4 Pg C yr-1 and lower model uncertainty. These results support current global estimates of Rs but highlight spatial inconsistencies that influence model parameterization and interpretation that should be addressed to improve regional- to global-scale predictions.

Stell, Emma↗
DOE OSTI2023

Joint analysis of Dark Energy Survey Year 3 data and CMB lensing from SPT and P l a n c k . II. Cross-correlation measurements and cosmological constraints

Cross-correlations of galaxy positions and galaxy shears with maps of gravitational lensing of the cosmic microwave background (CMB) are sensitive to the distribution of large-scale structure in the Universe. Such cross-correlations are also expected to be immune to some of the systematic effects that complicate correlation measurements internal to galaxy surveys. We present measurements and modeling of the cross-correlations between galaxy positions and galaxy lensing measured in the first three years of data from the Dark Energy Survey with CMB lensing maps derived from a combination of data from the 2500 deg 2 SPT-SZ survey conducted with the South Pole Telescope and full-sky data from the Planck satellite. The CMB lensing maps used in this analysis have been constructed in a way that minimizes biases from the thermal Sunyaev Zel’dovich effect, making them well suited for cross-correlation studies. The total signal-to-noise of the cross-correlation measurements is 23.9 (25.7) when using a choice of angular scales optimized for a linear (nonlinear) galaxy bias model. Here we use the cross-correlation measurements to obtain constraints on cosmological parameters. For our fiducial galaxy sample, which consist of four bins of magnitude-selected galaxies, we find constraints of Ω m =0.272$^{+0.032}_{-0.052}$ and S 8 ≡σ8$\sqrt{Ω_m/0.3}$=0.736$^{+0.032}_{-0.028}$ (Ω m =0.245$^{+0.026}_{-0.044}$ and S 8 =0.734$^{+0.035}_{-0.028}$) when assuming linear (nonlinear) galaxy bias in our modeling. Considering only the cross-correlation of galaxy shear with CMB lensing, we find Ω m =0.270$^{+0.043}_{-0.061}$ and S 8 =0.740$^{+0.034}_{-0.029}$. Our constraints on S8 are consistent with recent cosmic shear measurements, but lower than the values preferred by primary CMB measurements from Planck.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Transformer-powered surrogates close the ICF simulation-experiment gap with extremely limited data

Abstract Recent advances in machine learning, specifically transformer architecture, have led to significant advancements in commercial domains. These powerful models have demonstrated superior capability to learn complex relationships and often generalize better to new data and problems. This paper presents a novel transformer-powered approach for enhancing prediction accuracy in multi-modal output scenarios, where sparse experimental data is supplemented with simulation data. The proposed approach integrates transformer-based architecture with a novel graph-based hyper-parameter optimization technique. The resulting system not only effectively reduces simulation bias, but also achieves superior prediction accuracy compared to the prior method. We demonstrate the efficacy of our approach on inertial confinement fusion experiments, where only 10 shots of real-world data are available, as well as synthetic versions of these experiments.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Application of the AI2 Climate Emulator to E3SMv2's Global Atmosphere Model, With a Focus on Precipitation Fidelity

Abstract Can the current successes of global machine learning‐based weather simulators be generalized beyond 2‐week forecasts to stable and accurate multiyear runs? The recently developed AI2 Climate Emulator (ACE) suggests this is feasible, based upon 10‐year simulations with a network trained on output from a physics‐based global atmosphere model using a grid spacing of approximately 110 km and forced by a repeating annual cycle of sea‐surface temperature. Here we show that ACE, without modification, can be trained to emulate another major atmospheric model, EAMv2, run at a comparable grid spacing for at least 10 years with similarly small climate biases—a prerequisite to wider applicability. With an analysis that combines multiple temporal, spatial, and frequency domain perspectives, we show that ACE faithfully represents the spatiotemporal structure of EAMv2 precipitation and related variables. Finally, we show that a pretrained ACE network is able to adapt to a new global climate model simulation data set with 10 fewer training steps than when starting from random initialization, all while still maintaining low levels of climate bias. Further analysis of these fine‐tuning experiments reveal ACE's intriguing ability to interpolate between distinct global climate models.

Duncan, James P. C.↗
DOE OSTI2020

Vertical Structure of Radiative Heating Rates of the MJO during DYNAMO

The vertical structure of radiative heating rates over the region of the tropi-cal Indian Ocean associated with the MJO during the DYNAMO/ARM MJO Investigation Experiment is presented. The mean and variability of heating rates during active, suppressed, and disturbed phases are determined from the Paci?c Northwest National Laboratory combined remote sensing retrieval (CombRet) from Gan Island, Maldives (0.69? S, 73.15? E). TOA and surface ?uxes from the CombRet product are compared with collocated three-hourly CERES SYN1deg Ed4A satellite retrievals. The ?uxes are correlated in time with correlation coef?cients around 0.9, yet CombRet time-mean OLR is 15 W m-2 larger. Previous work has suggested that CombRet under-samples high clouds, due to attenuation by low-level clouds and reduced instrument sensitivity with altitude. However, mean OLR differs between CombRet and CERES for all values of OLR, not just the lowest values corresponding to widespread high clouds. The discrepancy peaks for mid-range OLR, sugges-tive of precipitating, towering cumulus convective clouds, rather than strati-form cirrus clouds. Biases in the cloud-top height of thick clouds substantially contribute to the overestimate of OLR by CombRet. CombRet data is used to generate composite shortwave and longwave atmospheric heating rate pro-?les as a function of the local OLR. Though there is considerable variability in CombRet not directly related to OLR, the time-height structure of mean heating rate composites generated using OLR as the interpolant is broadly representative of tropical convective variability on intraseasonal time scales.

Shell, Karen M.↗
DOE OSTI2025

Selecting samples of galaxies with fewer Fingers-of-God

The radial positions of galaxies inferred from their measured redshift appear distorted due to their peculiar velocities. We argue that the contribution from stochastic velocities — which gives rise to `Fingers-of-God' (FoG) anisotropy in the inferred maps — does not lend itself to perturbative modelling already on scales targeted by current experiments. To get around this limitation, we propose to remove FoG using data-driven indicators of their abundance that are local in nature and thus avoid selection biases. In particular, we show that the scale where the measured power spectrum quadrupole changes sign is tightly anti-correlated with both the satellite fraction and the velocity dispersion, and can thus be used to select galaxy samples with fewer FoG. In addition, we show that excluding galaxies in haloes more massive than a given mass threshold can help to discard many of the most problematic galaxies. Such selection could be achieved in practice using maps of the thermal Sunyaev-Zel'dovich distortion of the cosmic microwave background frequency spectrum. These techniques could potentially improve reconstructions of the large-scale velocity and displacement fields from the redshift-space positions of galaxies. They may also extend the reach of perturbative models for galaxy clustering, though in practice we find only marginal gains when fitting one-loop EFTofLSS models to simulations with mitigated FoG due to the relevance of other effects entering at two-loop order.

cosmological parameters from LSS↗
DOE OSTI2024

Deep Learning-based Non-Stationary Bias Correction (NSBC)

This work develops the NSBC (non-stationary bias correction) methodology to correct temperature projection bias from E3SM. The NSBC deep learning framework consists of a three-part architecture: an auto-encoder for compressing the spatial information, an LSTM for predicting annual temperature mean, and a U-Net for capturing the residual bias in temperature. The non-stationary bias correction (NSBC) framework can correct the non-stationarity of the biases of the climate models, which significantly improves the accuracy of future temperature prediction and improves the overestimation of extreme high temperatures that many existing bias correction methods suffer from. Getting started 1. Obtain the historical climate simulation and observation data. The E3SM simulation data are available through https://aims2.llnl.gov/search/cmip6/. The pseudo observations, the Geophysical Fluid Dynamics Laboratory (GFDL)-ESM4 model (Krasting et al., 2018) are available through https://aims2.llnl.gov/search/cmip6/. The spatial resolution of E3SM and pseudo observation datasets are both regridded to a common 1° resolution grid using conservative interpolation. The regridded E3SM and pseudo observation with 1° resolution can be found throught ./data/. 2. Train the Auto-encoder model. Python 0-autoencoder.py 3. Train the LSTM Python 1-LSTM.py 4. Generate the annual mean temperature based on trained LSTM Python 2-generate_annual_mean_LSTM.py 5. Train the U-Net. Python 3-unet.py 6. Evaluation and compared with the baseline Python 4_evaluation.py Is there a deadline approaching that requires the release of yo

Lucas, Donald↗
DOE OSTI2021

The Amazon Water Cycle: Perspectives from Water Budget Closure and Ocean Salinity

Global and regional water cycle includes precipitation, water vapor divergence, and change of column water vapor in the atmosphere, and land surface evapotranspiration, terrestrial water storage change, and river discharge, which is linked to ocean salinity near the river mouth. The water cycle is a crucial component of the Earth system, and numerous studies have addressed its individual components (e.g., precipitation). Here we assess, for the first time, if remote sensing and reanalysis datasets can accurately and self-consistently portray the Amazon water cycle. This is further assisted with satellite ocean salinity measurements near the mouth of the Amazon River. The widely used practice of taking the mean of an ensemble of datasets to represent water cycle components (e.g., precipitation) can produce large biases in water cycle closure. Additionally, closure is achieved with only a small subset of data combinations (e.g., ERA5 precipitation and evapotranspiration plus GRACE satellite terrestrial water storage), which rules out the lower precipitation and higher evapotranspiration estimates, providing valuable constraints on assessments of precipitation, evapotranspiration, and their ratio. The common approach of using the Óbidos stream gauge (located hundreds of kilometers from the river mouth) multiplied by a constant (1.25) to represent the entire Amazon discharge is found to misrepresent the seasonal cycle, and this can affect the apparent influence of Amazon discharge on tropical Atlantic salinity.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

White Dwarf Binaries across the H-R Diagram

We created the APOGEE-GALEX-Gaia catalog to study white dwarf (WD) binaries. This database aims to create a minimally biased sample of WD binary systems identified from a combination of GALEX, Gaia, and APOGEE data to increase the number of WD binaries with orbital parameters and chemical compositions. We identify 3414 sources as WD binary candidates, with nondegenerate companions of spectral types between F and M, including main-sequence stars, main-sequence binaries, subgiants, sub-subgiants, red giants, and red clump stars. Among our findings are (a) a total of 1806 systems having inferred WD radii R < 25 R ⊕ , which constitute a more reliable group of WD binary candidates within the main sample; (b) a difference in the metallicity distribution function between WD binary candidates and the control sample of most luminous giants (M H < -3.0); (c) the existence of a population of sub-subgiants with WD companions; (d) evidence for shorter periods in binaries that contain WDs compared to those that do not, as shown by the cumulative distributions of APOGEE radial velocity shifts; (e) evidence for systemic orbital evolution in a sample of 252 WD binaries with orbital periods, based on differences in the period distribution between systems with red clump, main-sequence binary, and sub-subgiant companions and systems with main-sequence or red giant companions; and (f) evidence for chemical enrichment during common envelope (CE) evolution, shown by lower metallicities in wide WD binary candidates (P > 100 days) compared to post-CE (P < 100 days) WD binary candidates.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

CaloTrilogy: Toward a Breakthrough in One-Step, End-to-End, Physics-Guided Shower Generation for Modern Calorimeters

High-precision calorimeter simulation at current and future colliders imposes rapidly growing computational demands, motivating the development of machine-learning surrogates for traditional Monte Carlo tools such as Geant4. Flow matching and diffusion-based generative models have become leading approaches for high-dimensional fast simulation because of their sample quality, but typically require ${\cal O}(100)$ function evaluations at inference and often rely on auxiliary networks to constrain global observables, compromising streamlined end-to-end generation. We introduce a unified framework that improves the balance between speed, shower quality, and physics fidelity. The method combines: (i) an average velocity field integrator that enables sampling in one or a few evaluations; (ii) a learned generative prior in shower space, constructed from data rather than random noise; and (iii) physics-guided loss terms that impose inductive biases on key observables during training. These elements are training time regularizers, preserving end-to-end inference with no additional cost. With only one or a few evaluation steps, the model achieves shower quality competitive with state-of-the-art flow and diffusion approaches, tested on several public high granularity calorimeter datasets. The results demonstrate inter-layer shower structure consistent with the underlying physics, providing a strong candidate for future fast simulation workflows.

Jiang, Cheng [Edinburgh U.]↗
DOE OSTI2024

Cross Correlating Cosmological Probes for LSST & CMB-S4

The upcoming years will be populated with state-of-the-art Stage-IV cosmological surveys. This paper forecasts the cosmological information we expect to constrain using probes from the Rubin Observatory Legacy Survey of Space and Time (LSST) and CMB-S4. We explore the effectiveness of different large-scale-structure probes, including the power spectra of galaxy weak lensing, galaxy position, Cosmic Microwave Background (CMB) lensing, and their cross-correlations. We use correlated lognormal simulations with the expected redshift distributions, galaxy number densities, and noise levels of the LSST survey. For CMB weak lensing, we use CMB-S4 lensing simulations with anticipated noise levels to obtain the expected error bars of each probe. We investigate the constraining power of the cosmological parameters for the individual probes and their combinations in an idealized scenario. We do not take into account astrophysical and observational parameters such as galaxy bias variations and photometric redshift un- certainties. Overall, we find the auto-correlated probes hold stronger constraints on cosmological parameters than the cross-correlated probes due to kernel differences. However, the cross-correlation of galaxy clustering and CMB lensing is very comparable to the auto-correlation of galaxy clustering with only an 8% stronger constraint in S8. It will be important for future surveys to use these auto and cross probes in combination due to the different astrophysical and systematic effects involved in CMB and late-time galaxy data. When systematics are added to this analysis, the cross probes will serve as a check for systematic biases in individual probes. In addition, we find a significant increase in constraint using all four probes in combination. This can be illustrated by the increase in constraint of S8 by 93.2% comparing galaxy clustering auto-correlation to a combination of all probes. Our study is a first step toward forecasting the high-precision cosmological constraints we expect to obtain using the next generation of large-scale structure probes.

Gibbins, Grace↗
DOE OSTI2024

Analytical noise bias correction for precise weak lensing shear inference

Noise bias is a significant source of systematic error in weak gravitational lensing measurements that must be corrected to satisfy the stringent standards of modern imaging surveys in the era of precision cosmology. This paper reviews the analytical noise bias correction method and provides analytical derivations demonstrating that we can recover shear to its second order using the ‘renoising’ noise bias correction approach introduced by METACALIBRATION. We implement this analytical noise bias correction within the AnaCal shear estimation framework and propose several enhancements to the noise bias correction algorithm. We evaluate the improved AnaCal using simulations designed to replicate Rubin Legacy Survey of Space and Time (LSST) imaging data. These simulations feature semi-realistic galaxies and stars, complete with representative distributions of magnitudes and Galactic spatial density. We conduct tests under various observational challenges, including cosmic rays, defective CCD columns, bright star saturation, bleed trails, and spatially variable point spread functions. Our results indicate a multiplicative bias in weak lensing shear recovery of less than a few tenths of a per cent, meeting LSST Dark Energy Science Collaboration requirements without requiring calibration from external image simulations. Additionally, our algorithm achieves rapid processing, handling one galaxy in less than a millisecond.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

A bi-level data-driven framework for fault-detection and diagnosis of HVAC systems

Long-term operation of heating, ventilation, and air conditioning (HVAC) systems will eventually lead to a range of HVAC system failures, resulting in excessive energy consumption and maintenance costs. Here, to avoid HVAC malfunctioning, fault detection diagnostic (FDD) is utilized as a common practice. Machine learning methods have lately received considerable interest for FDD analysis of HVAC systems due to their high detection accuracy. Meanwhile, HVAC malfunctions are regarded as rare occurrences, hence normal operating data samples are much more accessible than data samples in faulty and malfunctioning conditions. The dominating frequency of normal operation in HVAC datasets has also led to heavily biased classification algorithms within the literature. Moreover, the focus of previous literature has been on increasing the accuracy of the models which leads to a high number of false positives (misleading alarms) in the system. In order to enhance the performance of diagnostic procedures and fill the mentioned gaps, this study proposes a novel data-driven framework. A bi-level machine learning framework is developed for diagnosing faults in air handling units (AHUs) and rooftop units (RTUs) based on principal component analysis (PCA), time series anomaly detection, and random forest (RF). It is shown that PCA can reduce the dataset dimension with one principal component accounting for 95% of data variance. Also, the random forest could classify the faults with 89% precision for single-zone AHU, 85% precision for RTU, and 79% for multi-zone AHU. By proposing this framework, three persistent challenges are addressed: (I) minimizing false positives; (II) accounting for data imbalance; and (III) normal condition monitoring of equipment.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2021

The value of human data annotation for machine learning based anomaly detection in environmental systems

Anomaly detection is the process of identifying unexpected data samples in datasets. Automated anomaly detection is either performed using supervised machine learning models, which require a labelled dataset for their calibration, or unsupervised models, which do not require labels. While academic research has produced a vast array of tools and machine learning models for automated anomaly detection, the research community focused on environmental systems still lacks a comparative analysis that is simultaneously comprehensive, objective, and systematic. This knowledge gap is addressed for the first time in this study, where 15 different supervised and unsupervised anomaly detection models are evaluated on 5 different environmental datasets from engineered and natural aquatic systems. To this end, anomaly detection performance, labelling efforts, as well as the impact of model and algorithm tuning are taken into account. As a result, our analysis reveals the relative strengths and weaknesses of the different approaches in an objective manner without bias for any particular paradigm in machine learning. Most importantly, our results show that expert-based data annotation is extremely valuable for anomaly detection based on machine learning.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Instability of an electron-plasma shear layer in an externally imposed strain flow

The E x B shear instability of a two-dimensional (2D) filament (i.e., a thin, rectangular strip perpendicular to the magnetic field) of magnetized pure electron plasma is investigated experimentally in the presence of an externally imposed strain flow. Data are acquired using a specialized Penning–Malmberg trap in which strain flows can be applied in 2D by biasing segmented electrodes surrounding the plasma. The E x B drift dynamics are well-described by the Drift-Poisson equations, which are isomorphic to the 2D Euler equations describing ideal fluids. Thus, the experimental results correspond to the Rayleigh instability of a shear layer in a 2D ideal fluid, where the electron density is analogous to the fluid vorticity. Shear layers are prepared by stretching initially axisymmetric electron vortices using a strong, applied strain flow. The data at early times are in quantitative agreement with a linear model which extends Rayleigh’s work to account for the influence of an external strain flow. In the presence of weak strain, the system approximately maintains a phase relationship that corresponds to an instantaneous Rayleigh eigenmode. The instability develops into the nonlinear regime later in time and at smaller spatial scales as the strain rate is increased. A secondary vortex pairing instability is observed, but it is suppressed when the strain-to-vorticity ratio exceeds roughly 0.025. In this way, vorticity transport perpendicular to the filament is diminished due to the applied strain.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗