From Submesoscales to Global Impact: Oceananigans Powers Climate Predictions

:::infoAuthors:(1) Simone Silvestri, Massachusetts Institute of Technology, Cambridge, MA, USA;(2) Gregory Wagner, Massachusetts Institute of Technology, Cambridge, MA, USA;(3) Christopher Hill, Massachusetts Institute of Technology, Cambridge, MA, USA;(4) Matin Raayai Ardakani, Northeastern University, Boston, MA, USA;(5) Johannes Blaschke, Lawrence Berkeley National Laboratory, Berkeley, CA, USA;(6) Valentin Churavy, Massachusetts Institute of Technology, Cambridge, MA, USA;(7) Jean-Michel Campin, Massachusetts Institute of Technology, Cambridge, MA, USA;(8) Navid Constantinou, Australian National University, Canberra, ACT, Australia;(9) Alan Edelman, Massachusetts Institute of Technology, Cambridge, MA, USA;(10) John Marshall, Massachusetts Institute of Technology, Cambridge, MA, USA;(11) Ali Ramadhan, Massachusetts Institute of Technology, Cambridge, MA, USA;(12) Andre Souza, Massachusetts Institute of Technology, Cambridge, MA, USA;(13) Raffaele Ferrari, Massachusetts Institute of Technology, Cambridge, MA, USA.:::Table of LinksAbstract and 1 Justification2 Performance Attributes3 Overview of the Problem4 Current State of the Art5 Innovations5.1 Starting from scratch with Julia5.

2 New numerical methods for finite volume fluid dynamics on the sphere5.3 Optimization of ocean free surface dynamics for unprecedented GPU scalability6 How performance was measured7 Performance Results and 7.1 Scaling Results7.



2 Energy efficiency8 Implications9 Acknowledgments and References7.2 Energy efficiencyIn table 1 we summarize the energy metrics for our computations as well as the other investigated models. Figure 1 is derived from the data outlined in this table.

HadGEM3 and iHRES entries are estimated by including the whole coupled climate model (atmosphere and ocean). Unavailable data is marked with −. Our Oceananigans simulations are the highest in each of their columns.

This reflects our attention to memory and energy efficiency. 8 ImplicationsBy developing a new model from scratch specifically for GPUs, and wielding a handful of key ocean-model-specific innovations, Oceananigans achieves 9.9 SYPD at 10 km resolution using less than 1% of the resources of current state of the art supercomputers.

This achievement means that most climate model runs submitted to IPCC will be able use 10 km ocean models — precipitating a step change in the accuracy of climate prediction.At scales between 10–100 km, macroscale ocean turbulence exerts a key control on ocean carbon and heat uptake. However, attempts to accurately parameterize this key process in coarse resolution models have frustrated generations of oceanographers.

The inadequacies of macroscale parameterizations are associated with major biases and uncertainty in climate predictions [33, 39]. At resolutions of 10 km, the need for macroscale turbulence parameterization is eliminated, and ocean simulations capture key ocean features such as sharp sea surface temperature gradients supporting the formation of marine stratus clouds above narrow eastern boundary currents like the California and Benguela Current [28], and changes in the meridional overturning circulation due to the effect of Antarctic meltwater on deep convection in austral winter [26].Additionally, by achieving 0.

95 SYPD at 1.7 km resolution, we pave the way for decadal ocean simulations of the ocean “submesoscale” — the ocean analogue to atmospheric weather — which exhibits hourly fluctuations, high spatial and seasonal variability, and which exerts a strong control on ocean air-sea fluxes, biological productivity and fish stocks [46]. The granularity and accuracy provided by 1.

7 km resolution is further required to plan local mitigation strategies and predict local extreme events.Third, the unparalleled speed of execution and memory efficiency of Oceananigans allows global computations at never-before-seen sub-kilometer resolutions. The capacity for ultra-high-resolution simulations aligns with current advancements in resolution of ocean sampling platforms from satellites [32, 12] to fleets of floats and drones.

While this wealth of data is likely to provide new insights and scientific knowledge about the nature of small scale processes, global high-resolution ocean simulations will be needed to explore their impact on global climate scales.Finally, our results pave the way for marked increase in energy efficiency of climate simulations. The very reason to develop climate models, as stated by the Coupled Model Intercomparison Project (CMIP), for example, is to provide the necessary information to effectively reduce emissions and mitigate the effects of global warming — while, counterproductively, the carbon footprint of climate simulations that contribute to CMIP increases rapidly.

Oceananigans’ achievements represent a milestone towards decreased energy consumption by climate modeling efforts.9 AcknowledgmentsThis research used resources of the National Energy Research Scientific Computing Center (NERSC), a U.S.

Department of Energy Office of Science User Facility located at Lawrence Berkeley National Laboratory, operated under Contract No. DE-AC02-05CH11231 using NERSC award DDRERCAP0025591. This work is partly supported by the generosity of Eric and Wendy Schmidt by recommendation of the Schmidt Futures program and by NSF grant AGS-1835576.

N.C.C.

is supported by the Australian Research Council DECRA Fellowship DE210100749.References[1] HIP: C++ Heterogeneous-Compute Interface for Portability. https://github.

com/ ROCm-Developer-Tools/HIP.[2] MIT Satori User Documentation. https://mit-satori.

github.io/index.html.

[3] A. Arakawa and V.R.

Lamb. Computational design of the basic dynamical processes of the UCLA general circulation model. General Circulation Models of the Atmosphere, 17, 1977.

[4] V. Balaji, E. Maisonnave, N.

Zadeh, B.N. Lawrence, J.

Biercamp, U. Fladrich, G. Aloisio, R.

Benson, A. Caubel, J. Durachta, et al.

CPMIP: measurements of real computational performance of Earth system models in CMIP6. Geoscientific Model Development, 10(1):19–34, 2017.[5] T.

Besard, V. Churavy, A. Edelman, and B.

De Sutter. Rapid software prototyping for heterogeneous and distributed platforms. Advances in engineering software, 132:29–46, June 2019.

[6] T. Besard, C. Foket, and B.

De Sutter. Effective Extensible Programming: Unleashing Julia on GPUs. IEEE Trans.

Parallel Distrib. Syst., (4):827–841, December 2017.

[7] J. Bezanson, A. Edelman, S.

Karpinski, and V. B. Shah.

Julia: A fresh approach to numerical computing. SIAM Review, 59(1):65–98, January 2017.[8] E.

P. Chassignet, S.G.

Yeager, B. Fox-Kemper, A. Bozec, F.

Castruccio, G. Danabasoglu, C. Horvat, W.

M. Kim, N. Koldunov, Y.

Li, et al. Impact of horizontal resolution on global ocean–sea ice model simulations based on the experimental protocols of the Ocean Model Intercomparison Project phase 2 (OMIP-2). Geoscientific Model Development, 13(9):4595–4637, 2020.

[9] T.R. Christian, D.

Lebrun-Grandie, D. Arndt, J. Ciesko, V.

Dang, N. Ellingwood, R. Gayatri, E.

Harvey, D.S. Hollman, D.

Ibanez, et al. Kokkos 3: Programming model extensions for the exascale era. IEEE Transactions on Parallel and Distributed Systems, 33(4):805–817, 2021.

[10] V. Churavy, D. Aluthge, J.

Samaroo, A. Smirnov, J. Schloss, L.

C Wilcox, S. Byrne, M. Waruszewski, A.

Ramadhan, S. Schaub, N. C Constantinou, J.

Bolewski, M. Ng, T. Besard, B.

Arthur, C. Kawczynski, C. Hill, C.

Rackauckas, J. Cook, J. Liu, M.

Schanen, O. Schulz, P. Haraldsson, T.

Arakaki, and T. Chor. JuliaGPU/KernelAbstractions.

jl: v0.9.1, March 2023.

[11] M. Ding, H. Liu, P.

Lin, Y. Meng, W. Zheng, B.

An, Y. Luan, Y. Yu, Z.

Yu, Y. Li, J. Ma, J.

Chen, and K. Chen. A century-long eddy-resolving simulation of global oceanic large- and mesoscale state.

Sci. Data, 9:661, 2022.[12] C.

Donlon, B. Berruti, S. Mecklenberg, J.

Nieke, H. Rebhan, U. Klein, A.

Buongiorno, C. Mavrocordatos, J. Frerick, and B.

Seitz. The sentinel-3 mission: Overview and status. In 2012 IEEE International Geoscience and Remote Sensing Symposium, pages 1711–1714.

IEEE, 2012.[13] D. Ferreira, J.

Marshall, and J.-M. Campin.

Localization of deep water formation: Role of atmospheric moisture transport and geometrical constraints on ocean circulation. J. Climate, 23:1456–1476, 2010.

[14] T. Fiedler, A.J.

Pitman, K. Mackenzie, N. Wood, C.

Jakob, and S.E. Perkins-Kirkpatrick.

Business risk and the emergence of climate analytics. Nature Climate Change, 11:87–94, 2021.[15] O.

Fuhrer, T. Chadha, T. Hoefler, G.

Kwasniewski, X. Lapillonne, D. Leutwyler, D.

Lüth, C. Osuna, C. Schär, T.

C. Schulthess, and H. Vogt.

Near-global climate simulation at 1 km resolution: establishing a performance baseline on 4888 GPUs with COSMO 5.0. Geosci.

Model Dev., 14:2781–2799, 2021.[16] M.

Giordano, M. Klöwer, and V. Churavy.

Productivity meets performance: Julia on A64FX. In 2022 IEEE International Conference on Cluster Computing (CLUSTER), pages 549–555, September 2022.[17] W.

F. Godoy, P. Valero-Lara, E.

T. Dettling, C. Trefftz, I.

Jorquera, T. Sheehy, R. G.

Miller, M. Gonzalez-Tallada, J. S Vetter, and V.

Churavy. Evaluating performance and portability of high-level programming models: Julia, Python/Numba, and kokkos on exascale nodes. March 2023.

[18] J M. Gutiérrez and A.-M.

Tréguier. IPCC, 2021: Annex II: Models. Climate Change 2021: The Physical Science Basis, page 2087–2138, 2021.

[19] R. Haarsma, M. Acosta R.

, Bakhshi, P.-A. Bretonnière, L.

-P. Caron, M. Castrillo, S.

Corti, P. Davini, E. Exarchou, F.

Fabiano, U. Fladrich, R. Fuentes Franco, J.

García-Serrano, J. von Hardenberg, T. Koenig, X.

Levine, V.L. Meccia, T.

van Noije, G. van den Oord, F.M.

Palmeiro, M. Rodrigo, Y. Ruprich-Robert, P.

Le Sager, E. Tourigny, S. Wang, M.

van Weele, and K. Wyser. HighResMIP versions of EC-Earth: EC-Earth3P and EC-Earth3P-HR – description, model computational performance and basic validation.

Geoscientific Model Development, 13(8):3507–3527, 2020.[20] C. Hope.

The $10 trillion value of better information about the transient climate response. Phil. Trans.

R. Soc. A, 3:20140429, 2015.

[21] S. Hunold and S. Steiner.

Benchmarking Julia’s communication performance: Is Julia HPC ready or full HPC? In 2020 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), pages 20–25. IEEE, November 2020.[22] D.

Häfner, R. Nuterman, and M. Jochum.

Fast, Cheap, and Turbulent—Global Ocean Modeling With GPU Acceleration in Python. JAMES, 13(12):e2021MS002717, 2021.[23] E.

J. Kendo, N.M.

Roberts, H.J. Fowler, M.

J. Roberts, S.C.

Chan, and C.A. Senior.

Heavier summer downpours with climate change revealed by weather forecast resolution model. Nature Climate Change, 4(7):570–576, 2014.[24] C Lattner and V Adve.

LLVM: A compilation framework for lifelong program analysis & transformation. In International Symposium on Code Generation and Optimization, 2004. CGO 2004.

IEEE, 2004.[25] H. Lee and J.

Romero (eds.). IPCC, 2023: Climate Change 2023: Synthesis Report.

A Report of the Intergovernmental Panel on Climate Change. Contribution of Working Groups I, II and III to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change, 2023.[26] Q.

Li, M.H. England, A.

McC Hogg, S.R. Rintoul, and A.

K. Morrison. Abyssal ocean overturning slowdown and warming driven by Antarctic meltwater.

Nature, 615(7954):841–847, 2023.[27] W. C.

Lin and S. McIntosh-Smith. Comparing julia to performance portable parallel programming models for HPC.

In 2021 International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), pages 94–105. ieeexplore.ieee.

org, November 2021.[28] J. Ma, S.

Xu, and B. Wang. Warm bias of sea surface temperature in eastern boundary current regions—a study of effects of horizontal resolution in CESM.

Ocean Dynamics, 69:939–954, 2019.[29] J. Marshall, A.

Adcroft, C. Hill, L. Perelman, and C.

Heisey. A finite-volume, incompressible Navier Stokes model for studies of the ocean on parallel computers. Journal of Geophysical Research: Oceans, 102(C3):5753–5766, 1997.

[30] D.S. Medina, A.

St-Cyr, and T. Warburton. OCCA: A unified approach to multi-threading languages.

arXiv preprint arXiv:1403.0968, 2014.[31] Dimitris Menemenlis, Jean-Michel Campin, Patrick Heimbach, Chris Hill, Tong Lee, An Nguyen, Michael Schodlok, and Hong Zhang.

Ecco2: High resolution global ocean and sea ice data synthesis. Mercator Ocean Quarterly Newsletter, 31(October):13–21, 2008.[32] R.

Morrow, L-L. Fu, F. Ardhuin, M.

Benkiran, B. Chapron, E. Cosme, F.

d’Ovidio, J.T. Farrar, S.

T. Gille, G. Lapeyre, P.

-Y. Le Traon, A. Pascual, A.

Ponte, B. Qiu, Rascle N, C. Ubelmann, J.

Wang, and E.D. Zaron.

Global Observations of Fine-Scale Ocean Surface Topography with the Surface Water and Ocean topography (SWOT) Mission. 2019.[33] D.

R. Munday, H.L.

Johnson, and D.P. Marshall.

Impacts and effects of mesoscale ocean eddies on ocean carbon storage and atmospheric pCO2. Global Biogeochemical Cycles, 28(8):877–896, 2014.[34] T.

Palmer. Build high-resolution global climate models. Nature, 515:338–339, 2014.

[35] A. Ramadhan, G. Wagner, C.

Hill, J.-M. Campin, V.

Churavy, T. Besard, A. Souza, A.

Edelman, R. Ferrari, and J. Marshall.

Oceananigans.jl: Fast and friendly geophysical fluid dynamics on GPUs. Journal of Open Source Software, 5(53), 2020.

[36] T. Ringler, M. Petersen, R.

L. Higdon, D. Jacobsen, P.

W. Jones, and M. Maltrud.

A multiresolution approach to global ocean modeling. Ocean Modelling, 69:211–232, 2013.[37] M.

J. Roberts, A. Baker, E.

W. Blockley, D. Calvert, A.

Coward, H.T. Hewitt, L.

C. Jackson, T. Kuhlbrodt, P.

Mathiot, C.D. Roberts, R.

Schiemann, J. Seddon, B. Vannière, and P.

L. Vidale. Description of the resolution hierarchy of the global coupled hadgem3-gc3.

1 model as used in cmip6 highresmip experiments. Geosci. Model Dev.

, 12(12):4999–5028, 2019.[38] V. Churavy S.

Byrne, L.C. Wilcox.

MPI.jl: Julia bindings for the message passing interface. Proceedings of the JuliaCon Conferences, 2021.

[39] O.A. Saenko, D.

Yang, and J.M. Gregory.

Impact of mesoscale eddy transfer on heat uptake in an eddy-parameterizing ocean model. Journal of Climate, 31(20):8589 – 8606, 2018.[40] T.

Schneider, S. Behera, G. Boccaletti, C.

Deser, K. Emanuel, R. Ferrari, L.

R. Leung, N. Lin, T.

Müller, A. Navarra, O. Ndiaye, A.

Stuart, J. Tribbia, and T. Yamagata.

Harnessing AI, Data, and Computing to Advance Climate Modeling and Prediction. Nature Climate Change, submitted.[41] T.

Schneider, S. Lan, A. Stuart, and J.

Teixeira. Earth System Modeling 2.0: A Blueprint for Models That Learn From Observations and Targeted High-Resolution Simulations.

Geophysical Research Letters, 2017.[42] C. Shu.

Essentially non-oscillatory and weighted essentially non-oscillatory schemes for hyperbolic conservation laws. Icase report no. 97-65, Institute for Computer Applications in Science and Engineering, NASA Langley Research Center, 1997.

[43] S. Silvestri, G.L.

Wagner, A. Souza, N. Constantinou, C.

Hill, J.-M. Campin, and R.

Ferrari. A WENO-based Vector Invariant advection scheme for Implicit Ocean LES. In preparation.

[44] D. Struck. Meet the team shaking up climate models.

The Christian Science Monitor, 2021.[45] Z. Su, J.

Wang, P. Klein, A. Thompson, and D.

Menemenlis. Ocean submesoscales as a key component of the global heat budget. Nat.

Commun., 9:775, 2018.[46] J.

R. Taylor and A.F.

Thompson. Submesoscale Dynamics in the Upper Ocean. Annual Review of Fluid Mechanics, 55(1):103–127, 2023.

[47] MPI Team. MPI: A Message-Passing interface standard. Technical report, USA, 1994.

[48] P. Wang, J. Jiang, P.

Lin, M. Ding, J. Wei, F.

Zhang, L. Zhao, Y. Li, Z.

Yu, W. Zheng, Y. Yu, X.

Chi, and H. Liu. The GPU version of LASG/IAP climate system ocean model version 3 (LICOM3) under the heterogeneous-compute interface for portability (HIP) framework and its large-scale application.

Geosci. Model Dev., 14(5):2781–2799, 2021.

[49] N.P. Wedi, I.

Polichtchouk, P. Dueben, V.G.

Anantharaj, P. Bauer, S. Boussetta, P.

Browne, W. Deconinck, W. Gaudin, I.

Hadade, et al. A baseline for global weather and climate simulations at 1 km resolution. Journal of Advances in Modeling Earth Systems, 12(11):e2020MS002192, 2020.

[50] B. Xiao, F. Qiao, Q.

Shu, X. Yin, G. Wang, and S.

Wang. The development and validation of a global 1/32° surface-wave-tide-circulation coupled ocean model: FIO-COM32. Geosci.

Model Dev., 2022.[51] S.

Zhang, H. Fu, L. Wu, Y.

Li, H. Wang, Y. Zeng, X.

Duan, W. Wan, L. Wang, Y.

Zhuang, et al. Optimizing high-resolution community earth system model on a heterogeneous many-core supercomputing platform. Geoscientific Model Development, 13(10):4809–4829, 2020.

:::infoThis paper is available on arxiv under CC BY 4.0 DEED license.:::.