fix(orchestrator): use catalog-specified CUDA toolkit version per OS

[Nagachandan-P]

Description of the Solution

This PR fixes the CUDA toolkit installation so the version specified in the active catalog is installed instead of the latest available cuda-toolkit. The
orchestrator now dynamically resolves the CUDA toolkit package key from the functional layer's NVIDIA group components and passes the resolved version into
the rendered install_cuda_toolkit.sh cloud-init script.

Related Issue

• Fixes [Feature request] Omnia Modernization #4849

Changes

Orchestrator Metadata Resolution (src/orchestrator/roles/provision_common/tasks/)

• Added dynamic extraction of cuda_toolkit_version by traversing the catalog: functional layer → NVIDIA group → CUDA toolkit package.
• Regex ^cuda_toolkit supports both legacy cuda_toolkit and per-OS cuda_toolkit_<version> package keys.

CUDA Toolkit Install Script (src/orchestrator/roles/provision_common/templates/hpc_tools/)

• Updated install_cuda_toolkit.sh.j2 to use cuda-toolkit-${CUDA_TOOLKIT_VERSION} instead of unversioned cuda-toolkit.
• CUDA_TOOLKIT_VERSION defaults to 13.3.1 if not provided.

Metadata Service Templates (src/orchestrator/roles/provision_common/templates/metadata_svc/)

• All 4 ms-group-*.yaml.j2 templates now pass cuda_toolkit_version via lookup('template', ..., template_vars=...) to install_cuda_toolkit.sh.j2.

Files Changed

File	Change Type	Description
src/orchestrator/roles/provision_common/tasks/configure_metadata_svc.yml	Modified	Adds dynamic CUDA toolkit version extraction from catalog group r
eferences
src/orchestrator/roles/provision_common/templates/hpc_tools/install_cuda_toolkit.sh.j2	Modified	Uses catalog CUDA_TOOLKIT_VERSION in `dnf install
` command
src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_x86_64.yaml.j2	Modified	Passes cuda_toolkit_version to install
script
src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-slurm_node_aarch64.yaml.j2	Modified	Passes cuda_toolkit_version to instal
l script
src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_x86_64.yaml.j2	Modified	Passes cuda_toolkit_version t
o install script
src/orchestrator/roles/provision_common/templates/metadata_svc/ms-group-login_compiler_node_aarch64.yaml.j2	Modified	Passes cuda_toolkit_version
to install script

Testing

• Manual verification on RHEL 10.0 and 10.2 nodes:
  • Rendered script showed correct CUDA_TOOLKIT_VERSION per node (13.3.1 for 10.0, 13.4.2 for 10.2)
  • dnf repoquery cuda-toolkit-13.3.1 and cuda-toolkit-13.4.2 resolved correctly
  • Manual nvcc --version confirmed 13.3.73 on RHEL 10.0 and 13.4.92 on RHEL 10.2
• Jinja2 lookup test: Verified functional layers correctly resolve cuda_toolkit_10.0 and cuda_toolkit_10.2 package versions from catalog
• Catalog JSON validation: Active catalog validated successfully after updates

Backward Compatibility

• No breaking changes. The ^cuda_toolkit regex supports both single cuda_toolkit package and per-OS cuda_toolkit_<version> package keys.
• If cuda_toolkit_version is not resolvable, the script defaults to 13.3.1.