Repository navigation
feat(component): report TMT results and diagnostic logs - #395
Closed
Daniel McIlvaney (dmcilvaney) wants to merge 2 commits into
Closed
Daniel McIlvaney (dmcilvaney) wants to merge 2 commits into
Daniel McIlvaney (dmcilvaney) wants to merge 2 commits into
Conversation
Add a zero-input virtual TMT harness with a generated Azure Linux image, test-only probe RPM, private session libvirt state, persistent diagnostics, and owned-resource cleanup. Keep the known-red regression compiled but skipped in the standard E2E suite, with mage e2eTMT as the explicit execution path until provisioning fixes land. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Emit versioned JSON manifests linking native JUnit reports, retained results, and diagnostic logs. Preserve partial evidence and report lifecycle failures independently of test verdicts. Add unprivileged local-TMT E2E coverage for reporting, artifact relocation, and cancellation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Descendant-held output pipes can block cancellation and manifest emission, and CI skips the real reporting integration regression.
2 open issues
What changed in this PR
Adds structured TMT reporting to component testing, building on the quarantined virtual regression from #382.
Changes:
- Retains versioned JSON manifests, JUnit reports, native results, and diagnostic logs.
- Adds local reporting regressions and container support for virtual testing.
- Documents reporting, retention, cancellation, and test workflows.
| File | Description |
|---|---|
| scenario/tmt_virtual_helpers_test.go | Validates external virtual-test inputs. |
| scenario/tmt_virtual_helpers_scenario_test.go | Tests input validation. |
| scenario/tmt_virtual_e2e_test.go | Adds quarantined virtual regression. |
| scenario/tmt_local_e2e_test.go | Validates reporting and retained artifacts. |
| scenario/testdata/tmt-virtual/tests/smoke/test.sh | Checks guest policy and installed RPM. |
| scenario/testdata/tmt-virtual/tests/smoke/main.fmf | Defines virtual smoke test. |
| scenario/testdata/tmt-virtual/plans/smoke.fmf | Configures virtual provisioning. |
| scenario/testdata/tmt-virtual/azldev.toml | Configures probe component. |
| scenario/testdata/tmt-virtual/azldev-tmt-probe.spec | Supplies component metadata. |
| scenario/testdata/tmt-virtual/.fmf/version | Declares virtual fixture root. |
| scenario/testdata/tmt-local/tests.fmf | Defines reporting outcome fixtures. |
| scenario/testdata/tmt-local/run.sh | Exercises reporting and cancellation. |
| scenario/testdata/tmt-local/plans.fmf | Defines lifecycle failure cases. |
| scenario/testdata/tmt-local/.fmf/version | Declares local fixture root. |
| scenario/internal/containertest/containers.go | Adds device options and timeout logs. |
| scenario/internal/containertest/containers_test.go | Checks timeout output retention. |
| scenario/internal/containertest/containers_internal_test.go | Checks container options. |
| scenario/internal/cmdtest/cmdtest.go | Exposes container execution options. |
| scenario/internal/cmdtest/cmdtest_test.go | Tests option configuration. |
| scenario/image_build_e2e_test.go | Shares image-building fixture. |
| scenario/docker/tmt-probe.spec | Packages probe executable. |
| scenario/docker/tmt-probe.sh | Provides deterministic RPM probe. |
| scenario/docker/run-tmt-e2e.sh | Runs virtual regression with diagnostics. |
| scenario/docker/Dockerfile.tmt | Supplies TMT test container. |
| magefiles/magescenario/magescenario.go | Adds focused TMT targets. |
| internal/app/azldev/command.go | Propagates JSON output failures. |
| internal/app/azldev/command_test.go | Tests failed and short writes. |
| internal/app/azldev/cmds/component/tmtartifacts.go | Collects and validates retained evidence. |
| internal/app/azldev/cmds/component/tmt_junit.xml.j2 | Defines JUnit outcome projection. |
| internal/app/azldev/cmds/component/tmt_junit.py | Exports persisted TMT results. |
| internal/app/azldev/cmds/component/testreport.go | Manages manifests and subprocess capture. |
| internal/app/azldev/cmds/component/testreport_internal_test.go | Tests reporting lifecycle and failures. |
| internal/app/azldev/cmds/component/test.go | Integrates reporting into component tests. |
| internal/app/azldev/cmds/component/test_internal_test.go | Updates execution expectations. |
| internal/app/azldev/agentskill/content/build-component.md.tmpl | Adds reporting and safety guidance. |
| docs/user/reference/cli/azldev_component.md | Updates command summary. |
| docs/user/reference/cli/azldev_component_test.md | Documents CLI reporting behavior. |
| docs/user/explanation/tmt-tests.md | Explains manifest and retention contracts. |
| docs/developer/how-to/testing.md | Documents focused regression workflows. |
Review effort: Balanced
Comment on lines
+263
to
+265
| command.Stderr = io.MultiWriter(log, os.Stderr) | ||
| if !captureOutput { | ||
| command.Stdout = io.MultiWriter(log, stdout) |
Comment on lines
+53
to
+54
| if testing.Short() || os.Getenv("TESTING_ENABLE_TMT_LOCAL_E2E") == "" { | ||
| t.Skip("opt in with 'mage e2e -target=tmt-local'") |
Contributor
Author
|
#396 supercedes this |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on top of #382