As it turns out, the PETSc preconditioner performance is quite dependent on the direction of the sun. We find that the normal SOR sweeps are okayisch for the south west quadrant but deteriorates quite strongly if the sun is in the north east. The suspicion is that the loop structure for solve sweeps should consider the sun direction. The current ordering of degrees of freedom in the mesh is

y(south to north)
  x (west to east)
    z(top to bottom)
      number of streams

Now, for the sun being low and in the north east, it could make sense to change the ordering for a SOR sweep to be

z(top to bottom)
  y(north to south)
    x (east to west)
      number of streams

To accomplish this rearrangement and to test if it helps I’ve setup a simple setup where we compute the I3RC cumulus cloud field for 10 timesteps consecutively, each time shifted by 5 pixels north- and eastward. The solar zenith angle is 40 degrees and we run the simulations on a Intel Xeon Silver 4114 CPU @ 2.20GHz with 10 ranks.

First, for a solar azimuth angle of 225 degrees (sun in the south west)

and for an azimuth angle of 45 degrees (sun in the north east)

Solve times are normalized by a twostream calculation.

• orig: the current default ordering
• matpermute: original KSP solvers but with local Matrix permutation (direct radiation along sun direction and diffuse part moving z loop to outer dimension)
• explicit: custom mpi solver, no KSP
• max_it5: explicit solve and limiting iterations to 5 and accepting incomplete solves