How to derive skill from the fractions skill score
Monthly Weather Review American Meteorological Society 153:6 (2025) 1021-1033
Abstract:
The fractions skill score (FSS) is a widely used metric for assessing forecast skill, with applications ranging from precipitation to volcanic ash forecasts. By evaluating the fraction of grid squares exceeding a threshold in a neighborhood, the intuition is that it can avoid the pitfalls of pixelwise comparisons and identify length scales at which a forecast has skill. The FSS is typically interpreted relative to a “useful” criterion, where a forecast is considered skillful if its score exceeds a simple reference score. However, the typical reference score used is problematic, since it is not derived in a way that provides obvious meaning, does not scale with neighborhood size, and may not be exceeded by forecasts that have skill. We, therefore, provide a new method to determine forecast skill from the FSS, by deriving an expression for the FSS achieved by a random forecast, which provides a more robust and meaningful reference score to compare with. Through illustrative examples, we show that this new method considerably changes the length scales at which a forecast would be regarded as skillful and reveals subtleties in how the FSS should be interpreted.
Characterizing uncertainty in deep convection triggering using explainable machine learning
Journal of the Atmospheric Sciences American Meteorological Society 82:6 (2025) 1093-1111
Abstract:
Realistically representing deep atmospheric convection is important for accurate numerical weather and climate simulations. However, parameterizing where and when deep convection occurs (“triggering”) is a well-known source of model uncertainty. Most triggers parameterize convection deterministically, without considering the uncertainty in the convective state as a stochastic process. In this study, we develop a machine learning model, a random forest, that predicts the probability of deep convection, and then apply clustering of SHAP values, an explainable machine learning method, to characterize the uncertainty of convective events. The model uses observed large-scale atmospheric variables from the Atmospheric Radiation Measurement constrained variational analysis dataset over the Southern Great Plains, US. The analysis of feature importance shows which mechanisms driving convection are most important, with large-scale vertical velocity providing the highest predictive power for more certain, or easier to predict, convective events, followed by the dynamic generation rate of dilute convective available potential energy. Predictions of uncertain, or harder to predict, convective events instead rely more on other features such as precipitable water or low-level temperature. The model outperforms conventional convective triggers. This suggests that probabilistic machine learning models can be used as stochastic parameterizations to improve the occurrence of convection in weather and climate models in the future.
Deep Learning based reconstructions of the Atlantic Meridional Overturning Circulation confirm twenty-first century decline
Environmental Research Letters 20:6 (2025) 064036
Abstract:
Gaining knowledge of the past and present variations of the Atlantic Meridional Overturning Circulation (AMOC) is crucial for the development of accurate future climate projections. The short range covered by direct AMOC observations, inconsistent paleoclimate records, and scattered hydrographic data are insufficient to realistically reconstruct the AMOC strength since 1900. An AMOC proxy index based on sea surface temperatures suggests that the AMOC has declined by 15% since the late 19th century but this index received extensive scientific criticism. Here, we use a deep learning algorithm and climate model simulations to accurately reconstruct the AMOC strength between 20°N and 60°N since 1900. In contrast with the existing indices, our reconstructions are well in agreement with AMOC strength variations simulated by climate models and direct observations at 26.5°N. Our novel set of AMOC reconstructions contribute to a larger confidence in 21st century AMOC decline projections from climate models.
Potential for equation discovery with AI in the climate sciences
Earth Syst. Dynam., 16, 475–495, 2025
Abstract:
Climate change and artificial intelligence (AI) are increasingly linked sciences, with AI already showing capability in identifying early precursors to extreme weather events. There are many AI methods, and a selection of the most appropriate maximizes additional understanding extractable for any dataset. However, most AI algorithms are statistically based, so even with careful splitting between data for training and testing, they arguably remain emulators. Emulators may make unreliable predictions when driven by out-of-sample forcing, of which climate change is an example, requiring understanding responses to atmospheric greenhouse gas (GHG) concentrations potentially much higher than for the present or recent past. The emerging AI technique of “equation discovery” also does not automatically guarantee good performance for new forcing regimes. However, equations rather than statistical emulators guide better system understanding, as more interpretable variables and parameters may yield informed judgements as to whether models are trusted under extrapolation. Furthermore, for many climate system attributes, descriptive equations are not yet fully available or may be unreliable, hindering the important development of Earth system models (ESMs), which remain the main tool for projecting environmental change as GHGs rise. Here, we argue for AI-driven equation discovery in climate research, given that its outputs are more amenable to linking to processes. As the foundation of ESMs is the numerical discretization of equations that describe climate components, equation discovery from datasets provides a format capable of direct inclusion into such models where system component representation is poor. We present three illustrative examples of how AI-led equation discovery may help generate new equations related to atmospheric convection, parameter derivation for existing equations of the terrestrial carbon cycle, and (additional to ESM improvement) the creation of simplified models of large-scale oceanic features to assess tipping point risks.
Advancing organized convection representation in the unified model: implementing and enhancing multiscale coherent structure parameterization
Journal of Advances in Modelling Earth Systems Wiley 17:3 (2025) e2024MS004370