[Feature]: Merge custom MIG profiles with dynamically generated profiles

[botinca]

Is your feature request related to a specific problem?

GPU Operator currently supports either dynamically generated MIG profiles or a user-provided static MIG configuration.

When no custom ConfigMap is configured, MIG Manager generates hardware-aware profiles from the detected GPU capabilities. This keeps the profile set aligned with new GPU models, MIG profile changes, and upstream updates. The generated configuration also contains device-specific entries using device-filter.

However, when a custom MIG ConfigMap is configured, MIG Manager uses only that ConfigMap. The generated profiles are no longer available. Users therefore have to copy the complete upstream configuration, including all standard profiles and device-filter entries, and add their own profiles manually.

This creates an operational risk: the copied configuration will inevitably become stale as the upstream profile database evolves. Someone must periodically compare it with NVIDIA’s upstream 0400_configmap.yaml and update it, which is easy to miss across multiple clusters.

Describe the solution you would like

Please add an opt-in merge mode that combines the dynamically generated configuration with user-defined profiles.

For example:

migManager:
  config:
    name: custom-mig-parted-config
    mergeWithGenerated: true

The resulting profile set would contain both the generated profiles and the custom profiles. A custom ConfigMap would only need to define organization-specific profiles, for example:

version: v1
mig-configs:
  x100-80gb-10-20-20-20:
    - device-filter: ["0x233010DE", "0x233110DE"]
      devices: all
      mig-enabled: true
      mig-devices:
        "1g.10gb": 1
        "2g.20gb": 3

  x100-80gb-20-20-40:
    - device-filter: ["0x233010DE", "0x233110DE"]
      devices: all
      mig-enabled: true
      mig-devices:
        "2g.20gb": 2
        "3g.40gb": 1

On an H100 80GB node, users would then be able to select both generated profiles such as all-1g.10gb, all-2g.20gb, and all-balanced, and custom profiles such as x100-80gb-20-20-40.

Describe alternatives you have considered

The current workaround is to vendor the entire upstream 0400_configmap.yaml into every deployment and append custom profiles to it. This works, but creates a permanent maintenance burden and a risk of silently falling behind upstream.

Another workaround would be to patch the generated ConfigMap using a Kubernetes Job or hook. This is racy because the generated ConfigMap is node-specific and may be recreated or overwritten by MIG Manager.

Additional details

Suggested merge semantics:

• Generated profiles remain available by default.
• Custom profiles are appended.
• If a custom profile has the same name as a generated profile, the custom definition takes precedence.
• Duplicate names should produce a clear log message.
• device-filter must work for both generated and custom profiles.
• Invalid custom profiles should fail validation with an actionable error.
• Existing behavior must remain unchanged when merge mode is disabled.

This would preserve the convenience and freshness of dynamically generated profiles while allowing clusters to define a small set of stable, organization-specific combined layouts.

[botinca]

A related concern is the current single / mixed MIG strategy switch. With custom profiles that contain multiple MIG device types (for example, 2g.20gb: 2 plus 3g.40gb: 1), the distinction between the two modes may be outdated or at least unnecessarily manual.

Could the strategy be auto-detected from the selected MIG configuration? In particular, if the selected profile contains multiple MIG device types, the operator/device plugin could automatically use the required mixed behavior. Alternatively, the single / mixed setting could be deprecated or made optional when the selected profile unambiguously requires mixed mode.

This would avoid a second independent setting that can disagree with the selected profile and lead to confusing failures or missing nvidia.com/mig-* resources.

[rajathagasthya]

Thanks for feature request!

We don't plan to add a merge mode. The generated config is derived from each node's hardware, so a merged result would differ per node, and we don't want to have precedence rules between generated and custom entries.

One thing we can consider doing is have the mig-manager always publish the hardware-generated profiles to the per-node <node>-mig-config ConfigMap, even when a custom config is in use. That gives you a fresh configs to copy and lets you build your custom ConfigMap from it however you want.

On the single/mixed question: the strategy sets which resource name a node advertises (nvidia.com/gpu or nvidia.com/mig-*). Deriving it from the applied profile would change a node's resources whenever its mig.config label changes, breaking pods that request nvidia.com/gpu. FYI mixed works for uniform profiles too if you just want one cluster-wide setting.

[botinca]

Yes, thanks for the clarification (reg mig strategy). I carefully read through the documentation again this morning, so the question about the migration strategy is no longer relevant.

The "mixed" strategy seems more straightforward, though less compatible. And, in all likelihood, with the arrival of DRA, these naming distinctions will lose their significance entirely.

Reg the main topic:

That gives you a fresh configs to copy and lets you build your custom ConfigMap from it however you want.

Well, that might be okay, but I see a potential race condition here: if I were to modify a custom ConfigMap on the fly—say, using a job or a hook—the event triggering the read of that ConfigMap could occur before the update event. This doesn't strike me as an elegant or reliable way to handle customization.

What prevents us from describing all possible profiles in the dynamic version? The number of profile combinations is certainly significant. I realize few people need this, but https://github.com/NVIDIA/gpu-operator/blob/main/assets/state-mig-manager/0400_configmap.yaml wouldn't become much heavier with an extra 100 lines.