DOE OSTI · 2551914
A Practical Probabilistic Benchmark for AI Weather Models
Abstract
Since the weather is chaotic, it is necessary to forecast an ensemble of future states. Recently, multiple AI weather models have emerged claiming breakthroughs in deterministic skill. Unfortunately, it is hard to fairly compare ensembles of AI forecasts because variations in ensembling methodology become confounding and the baseline data volume is immense. We address this by scoring lagged initial condition ensembles—whereby an ensemble can be constructed from a library of deterministic hindcasts. This allows the first parameter‐free intercomparison of leading AI weather models' probabilistic skill against an operational baseline. Lagged ensembles of the two leading AI weather models, GraphCast and Pangu, perform similarly even though the former outperforms the latter in deterministic scoring. These results are elaborated upon by sensitivity tests showing that commonly used multiple time‐step loss functions damage ensemble calibration.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Brenowitz, Noah D. [NVIDIA Corporation, Santa Clara, CA (United States)] (ORCID:0000000186154921), Cohen, Yair [NVIDIA Corporation, Santa Clara, CA (United States)], Pathak, Jaideep [NVIDIA Corporation, Santa Clara, CA (United States)], Mahesh, Ankur [University of California, Berkeley, CA (United States)], Bonev, Boris [NVIDIA Corporation, Santa Clara, CA (United States)] (ORCID:000000024041238X), Kurth, Thorsten [NVIDIA Corporation, Santa Clara, CA (United States)], Durran, Dale R. [NVIDIA Corporation, Santa Clara, CA (United States)], Harrington, Peter [Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)], Pritchard, Michael S. [NVIDIA Corporation, Santa Clara, CA (United States)] (ORCID:0000000203406327). 2025-04-08. A Practical Probabilistic Benchmark for AI Weather Models. https://doi.org/10.1029/2024gl113656
Cite the original work for its findings. Save a collection to share your selection of sources.