[Daniel McIlvaney (dmcilvaney)]

Summary

Return TMT artifact locations without introducing a second test-result model.

• Insert TMT's native JUnit reporter alongside existing reporters. Return flat rows with the test name, observed TMT exit code, retained artifact directory, and relative JUnit paths.
• Keep the shared azldev.TestRunResult as data only; TMT execution/capture/discovery lives in tmtrunner.go. Image-test adoption is a follow-up. Preserve existing command help and examples with just one reporting sentence added.
• Keep setup/runner transcripts and the native run directory; JSON stdout stays clean. Failures may have logs but no JUnit, and retain the existing nonzero command behavior.
• Bound capture-pipe waiting when a descendant outlives TMT. Keep a separate per-invocation workdir root for testcloud image-store isolation.
• Use one tmt E2E selector for local reporting and virtual provisioning. Only the known-broken virtual test is quarantined; when explicitly enabled it checks KVM before image construction and skips if the device is missing.

This is the reduced alternative to #395, retained for its previous discussion. It builds on #387 and uses the centralized dependency pins from #384. There is no schema version, lifecycle-state audit, recursive per-test log index, Python exporter, or custom JUnit template.

Example output

Table

Rendered through azldev's table formatter using illustrative values, without running TMT:

╭────────────┬──────────┬──────────────┬──────────────────────────────────────────╮
│ TEST       │ EXITCODE │ ARTIFACTDIR  │ JUNIT                                    │
├────────────┼──────────┼──────────────┼──────────────────────────────────────────┤
│ smoke      │        0 │ /tmp/run-123 │ smoke/report/azldev-junit/junit.xml       │
│ regression │        1 │ /tmp/run-123 │ regression/report/azldev-junit/junit.xml  │
╰────────────┴──────────┴──────────────┴──────────────────────────────────────────╯

Columns that are empty across all rows are hidden. A dry run therefore shows only TEST, since no process exit or artifacts exist.

JSON (-O json)

Saved output from a passing local-TMT run inside the disposable container:

[
  {
    "test": "azldev-tmt-probe",
    "exitCode": 0,
    "artifactDir": "/artifacts/work/azldev-test-840810260",
    "junit": [
      "001-azldev-tmt-probe/native/plans/pass/report/azldev-junit/junit.xml"
    ]
  }
]

JUnit paths are relative to artifactDir, which also retains setup/runner logs and native TMT artifacts. exitCode is the observed TMT process exit, or null when no normal exit was observed. Report discovery is best-effort and does not override the command's execution result.

Validation

• env -u TESTING_ENABLE_TMT_VIRTUAL_E2E mage e2e -target=tmt — passed, 1m34.088s. The combined selector runs real local TMT and skips the quarantined virtual test. Coverage includes native outcomes, root and multiple/empty plans, existing reporters, source preservation, dependency/RPM installation, relocatable artifacts, and bounded waits for surviving descendants. Local reporting also runs in the unfiltered E2E suite.
• mage unit — changed packages passed; aggregate failed in the same nine unchanged bumpspec/release tests.
• mage all — build and changed unit packages passed; stopped at those existing failures before full scenarios/quality stages.
• mage docs — passed, regenerated CLI docs and schema.
• MS_GOTOOLCHAIN_ALLOW_NON_LOCAL=1 GOTOOLCHAIN=go1.26.0 mage fix all — passed. The default system toolchain hits the existing linter export-data incompatibility.
• MS_GOTOOLCHAIN_ALLOW_NON_LOCAL=1 GOTOOLCHAIN=go1.26.0 mage check all — passed, including lint, static, license and Python checks.
• mage mutationDiff f850e76067c73dc631e444a702444ca8323f5b9a — blocked during baseline coverage by the same unchanged unit failures; no mutation result claimed.
• git diff --check — passed.

After the data/runner file split and help restoration, build/changed unit packages, docs generation, and compatible-toolchain quality checks were rerun with the outcomes above; the E2E result predates that behavior-preserving split. Full scenarios were not rerun. No host-local TMT/package installation, privileged containers, image builds, KVM, or broad E2E runs were used. Virtual provisioning remains quarantined; this change tests its workdir argument isolation but does not claim a successful VM run. This host has KVM, so the missing-device skip was not exercised. The combined selector is not added to the CI matrix.

[Copilot]

🔵 Needs a closer look

Report discovery needs correction, and process-lifecycle and virtual-provisioner changes still warrant human validation.

1 open issue

• Avoid glob traversal of unrelated test-data directories · New

What changed in this PR

Adds native TMT reports and retained artifact paths to component testing without introducing a separate test-result model.

Changes:

• Returns artifact locations, relative JUnit paths, and observed TMT exit codes.
• Retains command transcripts, isolates run directories, and bounds capture-pipe waiting.
• Adds local TMT end-to-end coverage and updates usage guidance.

File	Description
scenario/​tmt_local_e2e_test.go	Validates native reports, artifacts, and interruption behavior.
scenario/​testdata/​tmt-root/​main.fmf	Adds a root-plan fixture.
scenario/​testdata/​tmt-root/​.fmf/​version	Marks the root fixture as FMF metadata.
scenario/​testdata/​tmt-local/​tests.fmf	Defines test outcomes and package checks.
scenario/​testdata/​tmt-local/​run.sh	Exercises reporting and surviving descendants.
scenario/​testdata/​tmt-local/​plans.fmf	Defines reporting and failure scenarios.
scenario/​testdata/​tmt-local/​.fmf/​version	Marks the local fixture as FMF metadata.
magefiles/​magescenario/​magescenario.go	Adds the tmt-local selector.
internal/​app/​azldev/​command.go	Propagates JSON output-write failures.
internal/​app/​azldev/​command_test.go	Covers failed and short JSON writes.
internal/​app/​azldev/​cmds/​component/​testreport.go	Collects reports and captures command transcripts.
internal/​app/​azldev/​cmds/​component/​testreport_internal_test.go	Covers artifact handling and capture failures.
internal/​app/​azldev/​cmds/​component/​test.go	Integrates native reporting and isolated run paths.
internal/​app/​azldev/​cmds/​component/​test_internal_test.go	Updates runner and argument assertions.
internal/​app/​azldev/​agentskill/​content/​build-component.md.tmpl	Documents artifact retrieval and local-run precautions.
docs/​user/​reference/​cli/​azldev_component.md	Updates the generated command summary.
docs/​user/​reference/​cli/​azldev_component_test.md	Refreshes generated help and examples.
docs/​user/​explanation/​tmt-tests.md	Explains report fields and artifact retention.
docs/​developer/​how-to/​testing.md	Documents the focused local regression suite.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Virtual-provisioning work-directory changes lack successful end-to-end validation while the virtual regression remains quarantined.

1 open issue

• Avoid glob traversal of unrelated test-data directories

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The edited agent skill lacks required content assertions for its new reporting and safety guidance.

1 open issue

• Avoid glob traversal of unrelated test-data directories

1 previously missed issue

In code that hasn't changed since last review

Add content assertions for new TMT and local-host guidance

internal/​app/​azldev/​agentskill/​content/​build-component.md.tmpl:45

Add content assertions for the new TMT reporting guidance and local-host safety warning in internal/app/azldev/agentskill/agentskill_test.go. The existing build-component test checks bumpspec guidance, so removing this new content would go unnoticed. .github/instructions/agent-skills.instructions.md:30-42 requires a content test when editing a skill; extend the build-component coverage to assert distinctive phrases from both additions.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The new work-directory layout remains unvalidated through successful virtual provisioning, the command’s default execution path.

2 open issues

• Avoid glob traversal of unrelated test-data directories
• Add reporting and provisioner guidance to lock-free build template · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Shared process-capture and artifact-layout changes affect virtual provisioning, which remains quarantined without successful integration validation.

3 open issues

• Restrict report collection to actual TMT plan workdirs · New
• Avoid glob traversal of unrelated test-data directories
• Add reporting and provisioner guidance to lock-free build template

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Human validation remains warranted because the reported E2E run predates the final runner split and successful virtual provisioning remains unverified.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Align PR description with new E2E CI shard

.github/​workflows/​test.yml:78

This enables a fourth E2E CI shard, but the PR description explicitly states that the combined TMT selector is not added to the CI matrix. Please update the description and validation scope to reflect the CI rollout, or remove this matrix entry if enabling the job is intended as a follow-up.

3 resolved since last review

• Restrict report collection to actual TMT plan workdirs
• Avoid glob traversal of unrelated test-data directories
• Add reporting and provisioner guidance to lock-free build template

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Process capture and cancellation changes need human validation because the reported end-to-end run predates the final runner refactor.

1 open issue

• Update validation notes for newly added CI E2E job · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Subprocess capture and work-directory changes affect virtual provisioning, which remains quarantined without successful end-to-end validation.

1 open issue

• Update validation notes for newly added CI E2E job

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Process-capture and virtual work-directory changes need human review because successful virtual provisioning remains unverified and full scenario validation is incomplete.

1 open issue

• Update validation notes for newly added CI E2E job

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The virtual provisioning directory changes lack successful VM validation while the E2E test remains quarantined, requiring human assessment of testcloud image-store and guest artifact behavior.

No open issues 🎉

1 resolved since last review

• Update validation notes for newly added CI E2E job

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The CI matrix adds a fourth tmt shard despite the PR description explicitly excluding it, leaving the implementation and documented scope inconsistent.

No open issues 🎉

_{Review effort: Balanced}