Conversation
vkarak
added
prio: normal
enhancement
reporting
vkarak
changed the title
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #3722 +/- ##
===========================================
- Coverage 91.92% 91.89% -0.03%
===========================================
Files 62 62
Lines 13808 13910 +102
===========================================
+ Hits 12693 12783 +90
- Misses 1115 1127 +12 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
3695390 to
e566487
Compare
Signed-off-by: Vasileios Karakasis <vkarak@gmail.com>
e566487 to
8e98874
Compare
Signed-off-by: Vasileios Karakasis <vkarak@gmail.com>
Review (Kimi)Overall this is a clean, long-overdue architectural cleanup of the run-report schema and the logging layer. The move to drive report fields from TestsI verified the unit tests claimed in the PR description in a fresh Python 3.13 venv: python test_reframe.py \
unittests/test_logging.py \
unittests/test_perflogging.py \
unittests/test_reporting.py \
unittests/test_pipeline.py \
unittests/test_meta.py \
unittests/test_config.py -qResult: 410 passed, 11 skipped. Suggestions / warnings
Things that look good
VerdictApprove with minor comments. The most worthwhile ones to address are (1) the RFC3339 naming mismatch and (2) the JUnit |
This branch has not been deployed
3 participants
Summary
Reworks the run report schema to give test cases proper identity fields and complete job/build-job reporting, adds output capture for failed tests (surfaced in both the JSON report and the JUnit XML report), and cleans up a number of inconsistencies uncovered along the way in the logging and storage layers.
DATA_VERSIONbumps from4.2to5.0— this is a breaking schema change; see below.TL;DR
Jobloggable attribute can now be logged and it doesn't need to be re-exposed in theRegressionTest. This makes the design cleaner and adds flexibility.ignore_keys).prebuild_cmds,prerun_cmdsetc.). The only exception ismodules, which can be useful standalone.slurmbackend it's derived from the job accounting data, so in combination with the completion time, the exact elapsed time can be calculated.RegressionTestorJobattribute is loggable.Report schema changes (
reframe/schemas/runreport.json)dependencies_actual,dependencies_conceptual,session_uuidanduuid(per-testcase),time_compile,time_performance,time_run,time_sanity,time_setup,time_total.A test case's identity can be reconstructed from
session_info.uuid+run_index+ the newtestcase_index.build_jobid→build_job_id,build_stdout/build_stderr→build_job_stdout/build_job_stderr.job_*/build_job_*prefixes (job_state,job_nodelist,job_scheduler,job_script_contents,build_job_exitcode, etc.), driven directly offJob's loggable attributes rather than a hand-picked subset.basename,sysenv,testcase_indexper test case;start_time_us/start_timestamp/end_time_us/end_timestamponsession_info(the oldtime_start/time_start_unix/time_end/time_end_unixare kept, but only for the SQLite storage backend).job_stdout_contents,job_stderr_contents,build_job_stdout_contents,build_job_stderr_contents— the tail of the captured build/run output for failed tests (size controlled by the existinggeneral/failure_inspect_linesoption).requiredon bothsession_infoandtestcase_typeis trimmed down to the minimal identifying fields (data_version/hostname/uuid, andname/resultrespectively) rather than an exhaustive list, since almost everything else can legitimately be absent depending on how far a test got before failing.JUnit XML report (
--report-junit)<system-out>/<system-err>(previously only emitted, empty, at the<testsuite>level) and afileattribute pointing at the test's source file.reframe/schemas/junit.xsdis extended to accept both additions — it's now a superset of the original Apache Ant JUnit XSD it started from. This matches what GitLab's Test Reports feature parses; stricter/differently-shaped JUnit consumers (e.g. Jenkins' own XSD) may not accept thefileattribute.Logging /
fileloghandlerfilelog'sprefix,format,format_perfvarsandignore_keysare now schema defaults (reframe/schemas/config.json) instead of being hardcoded in the builtinreframe/core/settings.py. A user defining their ownfileloghandler no longer has to redefine all of these from scratch to get sensible behavior —{'type': 'filelog'}alone now works.prefixis no longer a required option forfilelog.%(check_jobid)s,%(check_job_submit_time)s— silently resolved tonullafter thejob_*rework) are given a real backward-compatible alias to their replacements (%(check_job_id)s,%(check_job_submit_time_us)s), with a one-time deprecation warning instead of silent data loss.%(check_basename)s,%(check_sysenv)s,%(check_filename)s,%(check_fixture)s, the full%(check_job_NAME)s/%(check_build_job_NAME)sfamilies.Bug fixes found along the way
reframe/frontend/reporting/storage.py: session/testcase timestamps were being read from the new RFC3339 string fields (start_timestamp, etc.) and inserted intoREALSQLite columns compared against numeric epoch bounds — silently made every time-range query return zero rows. Fixed to use the numeric fields._job_contents()(report generation) resolved the test'sstdout/stderrinstead of the job's, so build failures reported the run job's (non-existent) output filename instead of the build job's._TCProxy._job_nodelistand a couple of report/printer code paths indexedjob_*fields unconditionally; any test case without a job (e.g. failing beforesetup()) wouldKeyError. Now uses.get()consistently for everything that's conditionally present.Documentation
docs/manpage.rst: new5.0entry in the run-report schema changelog;--report-junitupdated for thesystem-out/system-err/fileadditions and the JUnit-consumer compatibility note.docs/config_reference.rst:fileloghandler options updated (defaults, no-longer-requiredprefix); log placeholder table brought up to date; deprecation note for the renamed placeholders.reframe/core/schedulers/__init__.py(Job) andreframe/core/pipeline.py(RegressionTest): every variable/property that participates in the loggable-attribute mechanism now documents whether it's loggable, and under what name if different from the attribute (:loggable: Yes (logged as \`id``)`).Testing
unittests/test_logging.py,unittests/test_perflogging.py,unittests/test_reporting.py,unittests/test_pipeline.py,unittests/test_meta.py,unittests/test_config.pyall pass.--list-stored-sessions/--list-stored-testcases, and JUnit XML validated against the extended XSD.Example JSON report with failures
{ "session_info": { "data_version": "5.0", "hostname": "tresa.local", "uuid": "d19d9e8a-7464-429e-9b71-2777023f9f51", "start_time_us": 1788563638891562, "start_timestamp": "20260905T011358+0200", "end_time_us": 1788563640134085, "end_timestamp": "20260905T011400+0200", "time_start": "20260905T011358+0200", "time_start_unix": 1788563638.891562, "time_end": "20260905T011400+0200", "time_end_unix": 1788563640.1340852, "time_elapsed": 1.242523193359375, "cmdline": "/Users/user/reframe/.venv/bin/reframe -c unittests/resources/checks/hellocheck.py -r -S postbuild_cmds=false", "config_files": [ "<builtin>" ], "log_files": [ "/var/folders/h7/k7cgrdl13r996m4dmsvjq7v80000gp/T/rfm-2pq3oykm.log" ], "prefix_output": "/Users/user/reframe/output", "prefix_stage": "/Users/user/reframe/stage", "user": "karakasv", "version": "4.11.0-dev3+73d1c13e", "workdir": "/Users/user/reframe", "num_cases": 2, "num_failures": 2, "num_aborted": 0, "num_skipped": 0 }, "runs": [ { "num_cases": 2, "num_failures": 2, "num_aborted": 0, "num_skipped": 0, "run_index": 0, "testcases": [ { "environ": "builtin", "fail_phase": "compile_wait", "fail_reason": "build error: build job failed with exit code: 1", "partition": "default", "result": "fail", "run_index": 0, "testcase_index": 0, "scheduler": "local", "fail_info": { "exc_type": "BuildError", "exc_value": "build job failed with exit code: 1", "traceback": [ " File \"/Users/user/reframe/reframe/frontend/executors/__init__.py\", line 422, in _safe_call\n return fn(*args, **kwargs)\n", " File \"/Users/user/reframe/reframe/core/hooks.py\", line 109, in _fn\n func(obj, *args, **kwargs)\n ~~~~^^^^^^^^^^^^^^^^^^^^^^\n", " File \"/Users/user/reframe/reframe/core/pipeline.py\", line 2771, in compile_wait\n raise BuildError(\n f'build job failed with exit code: {self._build_job.exitcode}'\n )\n" ] }, "fail_severe": false, "basename": "HelloTest", "filename": "/Users/user/reframe/unittests/resources/checks/hellocheck.py", "fixture": false, "sysenv": "generic:default+builtin", "system": "generic", "build_locally": true, "build_time_limit": null, "descr": "C Hello World test", "display_name": "HelloTest", "env_vars": {}, "exclusive_access": false, "hashcode": "2b3e4546", "keep_files": [], "local": false, "maintainers": [ "VK" ], "max_pending_time": null, "modules": [], "name": "HelloTest", "num_cpus_per_task": null, "num_gpus_per_node": null, "num_tasks": 1, "num_tasks_per_core": null, "num_tasks_per_node": null, "num_tasks_per_socket": null, "outputdir": "/Users/user/reframe/output/generic/default/builtin/HelloTest", "perfvalues": {}, "prefix": "/Users/user/reframe/unittests/resources/checks", "readonly_files": [], "short_name": "HelloTest", "stagedir": "/Users/user/reframe/stage/generic/default/builtin/HelloTest", "strict_check": true, "tags": [ "bar", "foo" ], "time_limit": null, "unique_name": "HelloTest", "use_multithreading": null, "variables": {}, "build_job_scheduler": "local", "build_job_completion_time_us": 1788563639895499, "build_job_completion_timestamp": "20260905T011359+0200", "build_job_exitcode": 1, "build_job_id": "74734", "build_job_nodelist": [ "tresa.local" ], "build_job_nodelist_folded": "tresa.local", "build_job_script_contents": "#!/bin/bash\n\n_onerror()\n{\n exitcode=$?\n echo \"-reframe: command \\`$BASH_COMMAND' failed (exit code: $exitcode)\"\n exit $exitcode\n}\n\ntrap _onerror ERR\n\ncc hello.c -o ./HelloTest\nFalse\n", "build_job_script_filename": "rfm_build.sh", "build_job_start_time_us": 1788563639022853, "build_job_start_timestamp": "20260905T011359+0200", "build_job_state": "FAILURE", "build_job_stderr": "rfm_build.err", "build_job_stdout": "rfm_build.out", "build_job_submit_time_us": 1788563639022853, "build_job_submit_timestamp": "20260905T011359+0200", "job_scheduler": "local", "job_completion_time_us": null, "job_completion_timestamp": "20260905T011400+0200", "job_exitcode": null, "job_id": null, "job_nodelist": [], "job_nodelist_folded": "", "job_script_contents": "", "job_script_filename": "rfm_job.sh", "job_start_time_us": null, "job_start_timestamp": "20260905T011400+0200", "job_state": null, "job_stderr": "rfm_job.err", "job_stdout": "rfm_job.out", "job_submit_time_us": null, "job_submit_timestamp": "20260905T011400+0200", "build_job_stdout_contents": "-reframe: command `False' failed (exit code: 1)\n", "build_job_stderr_contents": "" }, { "environ": "builtin", "fail_phase": "compile_wait", "fail_reason": "build error: build job failed with exit code: 1", "partition": "default", "result": "fail", "run_index": 0, "testcase_index": 1, "scheduler": "local", "fail_info": { "exc_type": "BuildError", "exc_value": "build job failed with exit code: 1", "traceback": [ " File \"/Users/user/reframe/reframe/frontend/executors/__init__.py\", line 422, in _safe_call\n return fn(*args, **kwargs)\n", " File \"/Users/user/reframe/reframe/core/hooks.py\", line 109, in _fn\n func(obj, *args, **kwargs)\n ~~~~^^^^^^^^^^^^^^^^^^^^^^\n", " File \"/Users/user/reframe/reframe/core/pipeline.py\", line 2771, in compile_wait\n raise BuildError(\n f'build job failed with exit code: {self._build_job.exitcode}'\n )\n" ] }, "fail_severe": false, "basename": "CompileOnlyHelloTest", "filename": "/Users/user/reframe/unittests/resources/checks/hellocheck.py", "fixture": false, "sysenv": "generic:default+builtin", "system": "generic", "build_locally": true, "build_time_limit": null, "descr": "Compile-only C Hello World test", "display_name": "CompileOnlyHelloTest", "env_vars": {}, "exclusive_access": false, "hashcode": "ecfc0900", "keep_files": [], "local": false, "maintainers": [], "max_pending_time": null, "modules": [], "name": "CompileOnlyHelloTest", "num_cpus_per_task": null, "num_gpus_per_node": null, "num_tasks": 1, "num_tasks_per_core": null, "num_tasks_per_node": null, "num_tasks_per_socket": null, "outputdir": "/Users/user/reframe/output/generic/default/builtin/CompileOnlyHelloTest", "perfvalues": {}, "prefix": "/Users/user/reframe/unittests/resources/checks", "readonly_files": [], "short_name": "CompileOnlyHelloTest", "stagedir": "/Users/user/reframe/stage/generic/default/builtin/CompileOnlyHelloTest", "strict_check": true, "tags": [], "time_limit": null, "unique_name": "CompileOnlyHelloTest", "use_multithreading": null, "variables": {}, "build_job_scheduler": "local", "build_job_completion_time_us": 1788563640016950, "build_job_completion_timestamp": "20260905T011400+0200", "build_job_exitcode": 1, "build_job_id": "74736", "build_job_nodelist": [ "tresa.local" ], "build_job_nodelist_folded": "tresa.local", "build_job_script_contents": "#!/bin/bash\n\n_onerror()\n{\n exitcode=$?\n echo \"-reframe: command \\`$BASH_COMMAND' failed (exit code: $exitcode)\"\n exit $exitcode\n}\n\ntrap _onerror ERR\n\ncc hello.c -o ./CompileOnlyHelloTest\nFalse\n", "build_job_script_filename": "rfm_build.sh", "build_job_start_time_us": 1788563639075075, "build_job_start_timestamp": "20260905T011359+0200", "build_job_state": "FAILURE", "build_job_stderr": "rfm_build.err", "build_job_stdout": "rfm_build.out", "build_job_submit_time_us": 1788563639075075, "build_job_submit_timestamp": "20260905T011359+0200", "build_job_stdout_contents": "-reframe: command `False' failed (exit code: 1)\n", "build_job_stderr_contents": "" } ] } ], "restored_cases": [] }Supersedes #3702.