Adapting quantile mapping to bias correct solar radiation data
Not Available
Bailey, Maggie D. (ORCID:0000000347017849)↗
SEARCH · Engineering Papers
Results for “bias correction”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
The National Climate Data Base (NCDB): A Bias-Corrected High-Resolution Climate Dataset
Assessing renewable energy resources under future climate scenarios has been highlighted in recent years to analyze and understand potential impacts of future change in renewable generation on the power sector. Solar energy is well-known as the most plentiful among various renewable resources and usually converted to electricity using photovoltaics (PV) technologies, and the global deployment of PV technology has increased rapidly in recent decades. In this study, we develop a statistical technique to downscale the future projection of solar irradiance for PV energy-related applications. A set of Regional Climate Model (RCM)-based projections obtained from the North American Coordinated Regional Climate Downscaling Experiment (NA-CORDEX) are used as inputs to statistical methods to generate high-resolution global horizontal irradiance (GHI) over the contiguous United States (CONUS). The main steps of the statistical downscaling method include (1) regridding RCM output (0.22 degree and daily resolutions) to handle the modeled-observed data sets on a common grid, (2) correcting bias of RCM GHI using satellite-derived observation, and (3) implementing temporal and spatial downscaling to generate GHI at 8-km and hourly resolution. Basically, complex physical processes and interactions between solar radiation and various atmospheric constituents lead solar irradiance to be highly variable and uncertain. Underrepresentation of clouds from the RCM parameterizations is the main source of error and uncertainty in modeling solar irradiance. Thus, we adapt and use the high-quality satellite-derived data from the National Solar Radiation Database (NSRDB) to analyze the bias and error of RCM GHI as well as estimate the statistical parameters for spatial and temporal downscaling. This presentation will summarize the comprehensive analysis conducted to produce and assess the results under two climate scenarios (RCP4.5 and RCP8.5). We will also present a detailed validation demonstrating the strengths of the proposed downscaling method and future extension of this research.
climate data↗
Evaluating Probabilistic Deep Learning Methods for Uncertainty Quantification of Precipitation Bias Correction
Climate models often exhibit biases in their precipitation predictions, particularly underestimating high-intensity events and overestimating low precipitation. Deep learning approaches offer promising solutions, but their epistemic uncertainty associated with a deep learning–based bias correction method has not previously been quantified for reliable downstream climate impact studies. While methods for capturing the epistemic uncertainty in deep learning frameworks exist, there is currently no consensus on the best method. In this work, we compare three uncertainty quantification (UQ) methods—Deep Ensembles (DEns), Monte Carlo Dropout (MCD), and Flipout—by assessing the reliability of their uncertainty estimates using standard measures such as sharpness and calibration. These UQ methods are applied to an existing deep learning precipitation bias correction model known as UFNet: a coupled U-Net and fully connected neural network. The methods utilized to assess the models’ uncertainties are 1) calibration, which ensures that the expected probabilities of the model align with reality and 2) sharpness, which is a measure of the precision of the model’s probabilistic predictions. Of the three UQ methods evaluated, the DEns and MCD methods demonstrated the best-calibrated performance (expected calibration error of 0.36 and 0.35, respectively), compared to Flipout (0.58). In contrast, Flipout had the sharpest predictions and the highest metric performance in bias correcting precipitation—especially for higher-order moments such as kurtosis with a spatial correlation of 72% compared to 32% and 55% spatial correlation for DEns and MCD, respectively. Of the three UQ methods, MCD was found to be the most suitable method for UQ purposes based on its calibration, sharpness, and computational requirements.
Bayesian methods↗
Deep Learning-based Non-Stationary Bias Correction (NSBC)
This work develops the NSBC (non-stationary bias correction) methodology to correct temperature projection bias from E3SM. The NSBC deep learning framework consists of a three-part architecture: an auto-encoder for compressing the spatial information, an LSTM for predicting annual temperature mean, and a U-Net for capturing the residual bias in temperature. The non-stationary bias correction (NSBC) framework can correct the non-stationarity of the biases of the climate models, which significantly improves the accuracy of future temperature prediction and improves the overestimation of extreme high temperatures that many existing bias correction methods suffer from. Getting started 1. Obtain the historical climate simulation and observation data. The E3SM simulation data are available through https://aims2.llnl.gov/search/cmip6/. The pseudo observations, the Geophysical Fluid Dynamics Laboratory (GFDL)-ESM4 model (Krasting et al., 2018) are available through https://aims2.llnl.gov/search/cmip6/. The spatial resolution of E3SM and pseudo observation datasets are both regridded to a common 1° resolution grid using conservative interpolation. The regridded E3SM and pseudo observation with 1° resolution can be found throught ./data/. 2. Train the Auto-encoder model. Python 0-autoencoder.py 3. Train the LSTM Python 1-LSTM.py 4. Generate the annual mean temperature based on trained LSTM Python 2-generate_annual_mean_LSTM.py 5. Train the U-Net. Python 3-unet.py 6. Evaluation and compared with the baseline Python 4_evaluation.py Is there a deadline approaching that requires the release of yo
Lucas, Donald↗
Bias Correction in Urban Building Energy Modeling for Chicago Using Machine Learning
Urban-scale building energy modeling (UBEM) holds promise for optimizing energy usage across extensive geographic regions. However, there is a recognized bias between simulated energy consumption and actual measured data. This study, based on building data from Chicago, delved into bias correction techniques for enhancing the accuracy of UBEM energy consumption estimates. Initially, the AutoBEM simulation yielded a normalized mean bias error (NMBE) of 1.1% and 51% of Coefficient of the Variation of the Root Mean Square Error (CVRMSE) after outlier exclusion. To address this, three bias correction methods were deployed: Average Mean Bias Error based bias correction, Quantile mapping bias correction, and Machine learning-based bias correction using Linear Regression and Random Forest models. Post-correction results exhibited marked improvement. The NMBE values were diminished to 0 for Average MBE-based, 0.36 for Quantile Mapping, and 0 for Machine Learning-based corrections. Concurrently, the CVRMSE values registered reductions from an original 51 to 50.8 for Quantile Mapping, and 38.56 for Machine Learning-based corrections, pointing towards the effectiveness of specific bias correction methods in refining the precision of UBEM energy predictions. Such accurate estimations are paramount for informed energy planning and urban policy-making.
Chowdhury, Shovan↗
Bias correcting regional scale Earth system model projections: novel approach using empirical mode decomposition
Bias correction is a crucial step in using Earth system model outputs for assessments, as it adjusts systematic errors by comparing the model to observations. However, standard methods – ranging from mean-based linear scaling to distribution-based quantile mapping typically treat bias correction as a single-scale process, overlooking the fact that biases can manifest differently across daily, seasonal, and annual timescales. In this study, we propose a novel, timescale-aware bias-correction approach built on Empirical Mode Decomposition. By decomposing the meteorological signal into multiple oscillatory components and aggregating them to represent distinct timescales, we apply targeted corrections to each component, thereby preserving both short- and long-term structure in the data. Experimental illustrations show that the timescale-aware EMDBC framework matches the performance of conventional quantile-delta mapping (QDM) at the native daily scale and achieves progressively larger bias reductions at bi-weekly, seasonal, and annual scales. As a result, the proposed approach offers a more robust path to accurate and reliable Earth system projections, strengthening their utility for resilience and adaptation planning.
Ganguli, Arkaprabha [Argonne National Laboratory (↗
Downscaling and bias-correction contribute considerable uncertainty to local climate projections in CMIP6
Abstract Efforts to diagnose the risks of a changing climate often rely on downscaled and bias-corrected climate information, making it important to understand the uncertainties and potential biases of this approach. Here, we perform a variance decomposition to partition uncertainty in global climate projections and quantify the relative importance of downscaling and bias-correction. We analyze simple climate metrics such as annual temperature and precipitation averages, as well as several indices of climate extremes. We find that downscaling and bias-correction often contribute substantial uncertainty to local decision-relevant climate outcomes, though our results are strongly heterogeneous across space, time, and climate metrics. Our results can provide guidance to impact modelers and decision-makers regarding the uncertainties associated with downscaling and bias-correction when performing local-scale analyses, as neglecting to account for these uncertainties may risk overconfidence relative to the full range of possible climate futures.
54 ENVIRONMENTAL SCIENCES↗
Global variance decomposition of downscaled and bias-corrected CMIP6 climate projections
This dataset provides the results of a global variance decomposition of downscaled and bias-corrected CMIP6 climate projections. The total projection variance for a set of climate metrics is partitioned into contributions from: scenario uncertainty, model/GCM uncertainty, downscaling and bias-correction uncertainty, and interannual variability. The contribution from each source is expressed as a percentage of the total variance. Seven climate metrics are analyzed: Annual average temperature (avg_tas.nc) Annual total precipitation (tot_pr.nc) Annual maximum of daily maximum temperature (max_tasmax.nc) Annual maximum 1-day precipitation (max_pr.nc) Annual number of extremely hot days (hot_days.nc) Annual number of extremely wet days (wet_days.nc) Annual number of dry days (dry_days.nc) Extremely hot/wet days are defined to occur when temperature/precipitation exceeds the local 99th percentile defined over 1980-2014. Dry days are defined to occur when daily precipitation is less than 1mm. all_metrics_timesliced.nc gives the results for all metrics averaged over three 20-year periods: 2020-2039, 2050-2069, 2080-2099. For more details on the methods, see: Lafferty & Sriver, Downscaling and bias-correction contribute considerable uncertainty to local climate projections in CMIP6, npj Climate & Atmospheric Science (2023) An interactive visualization of this data can be found at: https://lafferty-sriver-2023-downscaling-uncertainty.msdlive.org
Lafferty, David↗
Bias Corrected NOAA HRRR Wind Resource Data for Grid Integration Applications
To address the need for regularly updated wind resource data, NREL has processed the High-Resolution Rapid Refresh (HRRR) outputs for use in grid integration modeling. The HRRR is an hourly-updated operational forecast product produced by the National Oceanic and Atmospheric Administration (NOAA) (Dowell et al., 2022). Several barriers have prevented the HRRR's widespread proliferation in the wind energy industry: missing timesteps (prior to 2019), challenging file format for wind energy analysis, limited vertical height resolution, and negative bias versus legacy WIND Toolkit data (2007-2013). NREL has applied re-gridding, interpolation, and bias-correction to the native HRRR data to overcome these limitations. This results in the now-publicly-available bias corrected and interpolated HRRR (BC-HRRR) dataset for weather years 2015 to 2023. Bias correction is necessary for wind resource consistency across weather years to be used simultaneously in planning-focused grid integration studies alongside the original WIND Toolkit data. We show that quantile mapping with the WIND Toolkit as a historical baseline is an effective method for bias correcting the interpolated HRRR data: the BC-HRRR has reduced mean bias versus comparable gridded wind resource datasets (+0.12 m/s versus Vortex) and has very low mean bias versus ground measurement stations (+0.01 m/s) (Buster et al., 2024). BC-HRRR's consistency with the legacy WIND Toolkit allows NREL to extend grid integration analysis to 15+ weather years of wind data with low-overhead extensibility to future years as they are made available by NOAA. As with historical datasets like the WIND Toolkit, BC-HRRR is intended for use in grid integration modeling (e.g., capacity expansion, production cost, and resource adequacy modeling) both independently and alongside the legacy WIND Toolkit.
Array↗
Statistically bias-corrected and downscaled climate models underestimate the adverse effects of extreme heat on U.S. maize yields
Abstract Efforts to understand and quantify how a changing climate can impact agriculture often rely on bias-corrected and downscaled climate information, making it important to quantify potential biases of this approach. Here, we use a multi-model ensemble of statistically bias-corrected and downscaled climate models, as well as the corresponding parent models from the Coupled Model Intercomparison Project Phase 5 (CMIP5), to drive a statistical panel model of U.S. maize yields that incorporates season-wide measures of temperature and precipitation. We analyze uncertainty in annual yield hindcasts, finding that the CMIP5 models considerably overestimate historical yield variability while the bias-corrected and downscaled versions underestimate the largest weather-induced yield declines. We also find large differences in projected yields and other decision-relevant metrics throughout this century, leaving stakeholders with modeling choices that require navigating trade-offs in resolution, historical accuracy, and projection confidence.
54 ENVIRONMENTAL SCIENCES↗
A Machine Learning Bias Correction on Large–Scale Environment of High–Impact Weather Systems in E3SM Atmosphere Model
Large–scale dynamical and thermodynamical processes are common environmental drivers of high–impact weather systems causing extreme weather events. However, such large–scale environmental conditions often display systematic biases in climate simulations, posing challenges to evaluating high–impact weather systems and extreme weather events. In this paper, a machine learning (ML) approach was employed to bias correct the large–scale wind, temperature, and humidity simulated by the atmospheric component of the Energy Exascale Earth System Model (E3SM) at ~1° resolution. The usefulness of the ML approach for extreme weather analysis was demonstrated with a focus on three high–impact weather systems, including tropical cyclones (TCs), extratropical cyclones (ETCs), and atmospheric rivers (ARs). We show that the ML model can effectively reduce climate bias in large–scale wind, temperature, and humidity while preserving their responses to imposed climate change perturbations. The bias correction is found to directly improve water vapor transport associated with ARs, and representations of thermodynamical flows associated with ETCs. When the bias–corrected large–scale winds are used to drive a synthetic TC track forecast model over the Atlantic basin, the resulting TC track density agrees better with that of the TC track model driven by observed winds. In addition, the ML model insignificantly interferes with the mean climate change signals of large–scale storm environments as well as the occurrence and intensity of three weather systems. This study suggests that the proposed ML approach can be used to improve the downscaling of extreme weather events by providing more realistic large–scale storm environments simulated by low–resolution climate models.
54 ENVIRONMENTAL SCIENCES↗
Multivariate Drought Monitoring, Propagation, and Projection Using Bias-Corrected General Circulation Models
Understanding how droughts are characterized, propagated, and projected, particularly multivariate droughts, is necessary to explain the variability and changes in drought characteristics. This study aims to understand multimodel global drought monitoring, propagation, and projection by utilizing a multivariate standardized drought index (MSDI) during the historical (1959–2014) and future (2045–2100) periods under two socioeconomic pathways SSPs (370 and 585), derived from the bias-corrected Coupled Model Intercomparison Project Phase 6 (CMIP6). Based on the energy metrics, the multivariate bias correction method outperformed other techniques in correcting the biases in the CMIP6 drought representation. The drought indicators demonstrate distinct categories for meteorological, hydrological, and multivariate droughts. There were significant high cross correlations between Heatwave Total Length (HWTL) and MSDI in Africa and South America for all lagged times. Europe and North America generally saw the maximum MSDI drought duration (228 months) during the historical period. For future projections, Africa recorded the maximum drought duration (197 months), while Europe witnessed the minimum drought duration for SSP 370 (171 months), and North America (149 months) for SSP 585. Furthermore, during the historical period in tropical Africa, the propagation of meteorological to hydrological drought was slower during the wet months than during the dry months. Under the SSP 370 future projection, there was a shift in the long period of meteorological-hydrological propagation from the middle and late wet months to the beginning of the wet months in tropical Africa. Therefore, tracking and projecting drought characteristics is vital for understanding the risk of drought-related consequences.
54 ENVIRONMENTAL SCIENCES↗
Future Climate Projections for South Florida: Improving the Accuracy of Air Temperature and Precipitation Extremes With a Hybrid Statistical Bias Correction Technique
Projecting future climate variables is essential for comprehending the potential impacts on hydroclimatic hazards like floods and droughts. Evaluating these impacts is challenging due to the coarse spatial resolution of global climate models (GCMs); therefore, bias correction is widely used. Here, we applied two statistical methods—standard empirical quantile mapping (EQM) and a hybrid approach, EQM with linear correction (EQM-LIN)—to bias correct precipitation and air temperature simulated by nine GCMs. We used historical observations from 20 weather stations across South Florida to project future climate under three shared socioeconomic pathways (SSPs). Compared to the EQM, the hybrid EQM-LIN method improved R 2 of daily quantiles by up to 30% over the historical period and improved MAE up to 70% in months that contain most extreme values. Projected extreme precipitation at the weather stations showed that, compared to the EQM-LIN, the EQM method underestimates the high quantiles by up to 26% in SSP585. The projected changes in annual maximum precipitation from historical period (1985–2014) to near future (2040–2069) and far future (2070–2100) were between 2% and 16% across the study area. Projected future precipitation suggested a slight decrease during summer but an increase in fall. This, along with rising summer temperatures, suggested that South Florida can experience rapid oscillations from warmer summers and increased flooding in fall under future climate. Additionally, our comparative analyses with globally and nationally downscaled studies showed that such coarse scale studies do not represent the climatic extremes well, particularly for high quantile precipitation.
54 ENVIRONMENTAL SCIENCES↗
Analytical noise bias correction for precise weak lensing shear inference
Noise bias is a significant source of systematic error in weak gravitational lensing measurements that must be corrected to satisfy the stringent standards of modern imaging surveys in the era of precision cosmology. This paper reviews the analytical noise bias correction method and provides analytical derivations demonstrating that we can recover shear to its second order using the ‘renoising’ noise bias correction approach introduced by METACALIBRATION. We implement this analytical noise bias correction within the AnaCal shear estimation framework and propose several enhancements to the noise bias correction algorithm. We evaluate the improved AnaCal using simulations designed to replicate Rubin Legacy Survey of Space and Time (LSST) imaging data. These simulations feature semi-realistic galaxies and stars, complete with representative distributions of magnitudes and Galactic spatial density. We conduct tests under various observational challenges, including cosmic rays, defective CCD columns, bright star saturation, bleed trails, and spatially variable point spread functions. Our results indicate a multiplicative bias in weak lensing shear recovery of less than a few tenths of a per cent, meeting LSST Dark Energy Science Collaboration requirements without requiring calibration from external image simulations. Additionally, our algorithm achieves rapid processing, handling one galaxy in less than a millisecond.
79 ASTRONOMY AND ASTROPHYSICS↗
Future Projections of EURO‐CORDEX Raw and Bias‐Corrected Daily Maximum Wind Speeds Over Scandinavia
Twenty historical and future Coordinated Regional Climate Downscaling Experiment ensemble for Europe simulations are bias corrected to investigate the future changes in the daily maximum wind speed over Scandinavia. We use quantile mapping to adjust the wind for the historical period (1985–2014) and quantile delta mapping for two future periods (2041–2070, 2071–2100, RCP8.5) with the 3-km spatial resolution NORA3 hindcast as the reference data set. Decomposing the variance, we find that most of the inter-model spread in the bias and the response to climate change is due to the regional climate model over land and mainly to the general climate model over sea. On average, the mean daily maximum wind speed is projected to increase everywhere except along the western coast of Norway and Denmark. In summer, we see an opposite sign over Sweden and Finland. The Norwegian Sea experiences weaker mean and high wind whereas the Baltic Sea experiences a strengthening. Bias correction influences the amplitude of the response, not the response pattern. Wind speed distributions can have different shapes in the future, with, for example, a flattening of the distribution off the coast of Norway with more frequent weak and very strong winds. Apart from a few locations in Norway, we find an increase in the number of days in the local highest historical wind category. Overall, summer exhibits opposite signals compared to the three other seasons. At country scale, the sign of the change in the mean and 98th percentile varies greatly depending on the region and among the simulations, especially for Norway.
54 ENVIRONMENTAL SCIENCES↗
Bias Correcting NOAA's High-Resolution Rapid Refresh (HRRR) Wind Resource Data for Grid Integration Applications [Slides]
Many weather years of high-quality wind data are widely accepted in the grid integration community to be important for studying wind energy technical potential, energy system operations, and grid resilience. NREL makes high-quality wind and solar resource data available. NREL's Grid-Atmosphere workshop (March 2024) identified NREL National Solar Radiation Database as widely used in grid integration modeling, but there is less agreement on commonly used wind datasets. One important factor identified by ESIG's 2023 report 'Weather Dataset Needs for Planning and Analyzing Modern Power Systems' for gold standard wind data is regular updates. To address the need for regular updates, NREL's team can now process all currently available and regularly updated High-Resolution Rapid Refresh (HRRR) outputs. HRRR is an hourly-updated operational forecast product produced by the National Oceanic and Atmospheric Administration (NOAA) (Dowell et al., 2022). One barrier to NREL using HRRR is systematic bias and consistency with NREL's existing wind datasets (e.g. WIND Toolkit, 'WTK') across weather years. To address this barrier, we show that the HRRR can be interpolated and bias-corrected to be consistent with NRE's existing datasets. We call the new dataset BC-HRRR (bias-corrected HRRR). As with historical datasets like the WTK, BC-HRRR is intended for use in grid integration modeling (e.g., capacity expansion, production cost, and resource adequacy modeling). BC-HRRR's (2015-present) consistency with WTK (2007-2013) allows NREL to extend internal grid integration tooling with 15+ weather years of wind data with low-overhead extensibility to future years as they are made available by NOAA. The rest of this slide deck documents the BC-HRRR processing methods, validation, and its implications for intended use.
17 WIND ENERGY↗
Machine learning-based bias-corrected future projections of ozone concentrations from a chemistry-climate model
Reliable projection of future near-surface ozone is crucial for air quality management and health risk assessment. However, potential biases in spatial distribution, magnitude and trends in ozone concentrations simulated by global chemistry-climate models limit their applicability in regional-scale evaluations. In this study, LightGBM, a machine learning (ML) algorithm is applied to correct biases in CESM2-simulated ozone concentrations over China, the United States and Europe and calibrate future ozone projections under two diverse Shared Socioeconomic Pathways (SSP1-2.6 and SSP5-8.5) scenarios from 2020 to 2060. The ML-based correction significantly improves the spatial distribution and reduces the model bias by 40%–60%. It also reverses the potentially incorrect trend of ozone change under SSP1-2.6 in eastern China. When applying ML-based bias correction to CESM2 future projections, warm season mean ozone concentrations decrease across China, the United States, and Europe by –13.5, –17.9, and –13.7 µg/m³, respectively, between 2020 and 2060 in SSP1-2.6, while they increase by 9.4, 2.0, and 5.2 µg/m³ in SSP5-8.5. Decomposition analysis show that changes in anthropogenic emissions dominate future ozone changes in both scenarios, while strong climate penalty from ozone changes occurs in polluted eastern China and climate benefit is found in western China, the United States and Europe under SSP5-8.5. These findings demonstrate the value of combining ML with chemistry-climate models to produce more accurate air quality projections, thereby informing more effective and region-specific environmental protection strategies.
Chemistry Model↗
UFNet: Joint U-Net and Fully Connected Neural Network to Bias Correct Precipitation Predictions from
Paper information. Shuang Yu, Indrasis Chakraborty, Gemma J. Anderson, Donald D. Lucas, Yannic Lops, and Daniel Galea. UFNet: Joint U-Net and fully connected neural network to bias correct precipitation predictions from climate models. Artificial Intelligence for the Earth Systems, 2024. Overview. This work develops the UFNet methodology to correct E3SM historical precipitation projection bias. The UFNet deep learning framework consists of a two-part architecture: a U-Net convolutional network to capture the spatiotemporal distribution of precipitation and a fully connected network to capture the distribution of higher-order statistics. The joint network, termed UFNet, can simultaneously improve the spatial structure of the modeled precipitation and capture the distribution of extreme precipitation values. Below we provide guidance for applying UFNet to correct the Energy Exascale Earth System Model (E3SM; Golaz et al. 2019) daily precipitation projection over the contiguous United States (CONUS). Getting started 1. Obtain the historical climate simulation and observation data. The E3SM historical simulation data are available through https://aims2.llnl.gov/search/cmip6/. The CPC unified gauge-based analysis of daily precipitation can be found through https://psl.noaa.gov/data/gridded/data.cpc.globalprecip.html. The ECMWF atmospheric reanalysis of the 20th century (ERA-20C) data are available through https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era-20c. The spatial resolution of E3SM and observed datasets are both regridded to a common 1° resolution grid using conservative interpolation. The regridded E3SM, CPC and ERA-20C with 1° resolution can be found throught ./data/. 2. Train the fully connected network (DNN) Python train_dnn.py 3. Train the UFNet Python train_ufnet.py 4. Evaluation and compared with the baseline Python evaluation.py
Lucas, DonaldD↗