pgTAP-aware results validator, tolerant of suffixed/prefixed role names in assertion descriptions #106

Activity

1. jnasbyupgrade

   added

   enhancement

   on Sep 8, 2026
2. jnasbyupgrade commented on Oct 12, 2026

   

   jnasbyupgrade

   on Oct 12, 2026

   Last edited by jnasbyupgrade

   ContributorAuthor

   More actions

   🤖 (Claude)

   Suggestion: run the test command through a small wrapper that fails on either a non-zero exit status or a failure-looking line in the output. make test alone is not enough, because the pgxntool test target only fails on a non-empty regression.diffs and does not scan pgTAP output.

   The wrapper takes any command (not only make test), captures its combined stdout and stderr, and fails the overall run in two cases:

   1. The command exits non-zero. The wrapper passes the command's own status through unchanged; it does not replace it with 1 or any other value, and no one should force the exit code to 1.
   2. The captured output contains a line matching a failure pattern: not ok, FAILED, Looks like you planned, an N of M tests failed style summary, or the database error current transaction is aborted.

   Both checks are needed. The exit code catches hard failures. The output scan catches TAP-style failures that a command can still exit 0 on.

   Sketch:

   # bash; usage: run-and-scan <command> [args...]
   out=$(mktemp)
   "$@" 2>&1 | tee "$out"
   status=${PIPESTATUS[0]}
   if grep -E -q 'not ok|FAILED|Looks like you planned|[0-9]+ of [0-9]+ tests? failed|current transaction is aborted' "$out"; then
     echo "failure pattern found in output" >&2
     [ "$status" -ne 0 ] || status=1   # only when the command itself exited 0
   fi
   rm -f "$out"
   exit "$status"
3. jnasbyupgrade

   added 4 commits that reference this issue on Oct 14, 2026

   Add `PGXNTOOL_VERIFY_RESULTS_MODE=pgtap-only`

   ...

   This issue will close once commit 15e558b is merged into the 'master' branch.

   15e558b

   Test `PGXNTOOL_VERIFY_RESULTS_MODE=pgtap-only`

   ...

   This issue will close once commit 09cf607 is merged into the 'master' branch.

   09cf607

   Remove `PGXNTOOL_ENABLE_CHECK_TAP_OUTPUT`

   ...

   This issue will close once commit 7eec927 is merged into the 'master' branch.

   7eec927

   Follow the removal of `PGXNTOOL_ENABLE_CHECK_TAP_OUTPUT`

   ...

   This issue will close once commit 9e63c06 is merged into the 'master' branch.

   9e63c06
4. jnasbyupgrade commented on Oct 15, 2026

   

   jnasbyupgrade

   on Oct 15, 2026

   ContributorAuthor

   More actions

   🤖 (Claude)

   Update on the motivating case for this issue: we no longer need output that varies per run.

   The case was role names that get a per-build prefix or suffix and show up in pgTAP test descriptions. We removed it by replacing the real names in descriptions with fixed placeholders, so the expected output is byte-identical on every run. A content-tolerant validator is not needed for that case, and we have decided against using one as a general mechanism because it would also hide real changes in diagnostic text.

   What still seems worth pgxntool's attention from this issue, and does not depend on the naming case:

   1. A not ok line that gets accepted into an expected file still passes plain make test. Scanning the raw output for failure markers (not ok, FAILED, current transaction is aborted, and similar) catches this. We do that in a wrapper outside pgxntool.
   2. Nothing requires a 1..N plan line in the result file. A test that dies before planning can look like a pass.
5. jnasbyupgrade

   added a commit that references this issue on Oct 16, 2026

   Refer to the pgxntool issue by its full name in make-test.bats

   ...

   2e0615a