Change ORB to support arbitrary decomposition axes

[aliemen]

closes #559

This PR generalizes the ORB class to allow decomposing in arbitrary axis. Now you can e.g. pass a std::array<bool, Dim> allowedAxes = {false, false, true}; array to indicate that you only want to decompose in z. The actual algorithm is more or less untouched. The addition is mostly in the helpers like findAxis. Here an example in our SwissFEL-booster-sc input file from the OPALX side with decomposition only in z:

The decomposition works just fine only in z direction.

Other changes:

• Add allowed-axis cut selection:
  • findCutAxis(domain, allowedAxes) chooses the longest enabled axis,
  • returns failure if no enabled cut axis exists.
• Add validation before changing the layout:
  • reject selected cut axes with fewer than 4 cells,
  • clamp weighted median cuts to [1, cutAxisLength - 3],
  • reject one-cell domains,
  • reject boxes outside the global domain,
  • reject cuts along disabled axes,
  • reject overlapping boxes,
  • reject incomplete tiling of the global domain,
  • reject proposals whose domain count does not match the communicator size.
• Update FieldLayout and ORB weight field only after correct decomposition is found.

[aliemen]

cscs-ci run cscs-ci-gh200

[aliemen]

Jülich CI passes just fine. I don't know what cscs/cscs-ci-gh200 is doing, it says "no CUDA-capable device is detected" and doesn't re-start... Looks like it worked now

[aaadelmann]

General comment: before this can go into master we need to make sure it works with several k of GPUs. @srikrrish in ALPINE which of the tests are with ORB == ON?

[srikrrish]

General comment: before this can go into master we need to make sure it works with several k of GPUs. @srikrrish in ALPINE which of the tests are with ORB == ON?

Any of the Alpine example with load balancing threshold < 1 may invoke ORB (depending on the threshold). For example the usual value of 0.01 which means load balancing should be triggered when particle load imbalance in any rank exceeds 1% is a good start. I think Penning trap with more than 8 ranks might be the good stress test.

@aliemen yes the initial parallel/serial decomposition should be preserved during ORB also. If not that was a bug and if this PR fixes it that's a good thing I would say.

[aliemen]

Update: Instead of overloads with an allowedAxes mask, binaryRepartition(R, fl, isFirstRepartition, forbiddenAxes = {}) and findCutAxis(dom, forbiddenAxes = {}) take a forbiddenAxes mask, so all false is the default (legacy) behavior.

While running the unit test on 4 ranks I noticed that it only works <4 ranks, because ORB weights in the unit test were zero. Should be fixed now and passes on 1, 2, 4, 8 ranks.

[aliemen]

I activated "auto-merge". So once @srikrrish resolves the conversations, it should merge automatically (given that the tests pass)

[aliemen]

cscs-ci run cscs-ci-gh200, cscs-ci-mi300, cscs-ci-openmp