Fail make test on pgTap problems in actual output; add pgtap-only mode

[jnasbyupgrade]

make test now fails on pgTap failures in actual output, not just on regression.diffs: a not ok without # TODO, a missing or duplicate 1..N plan line, or a test count that differs from the plan. It also scans the test/install actual output, which never gets a real diff. A new PGXNTOOL_VERIFY_RESULTS_MODE=pgtap-only judges TAP tests by their TAP alone, ignoring description-only differences, which is the validator #106 asks for. This PR also fixes test/install under a separate TESTOUT.

Fixes #106

Merge order: merge this PR first, then the paired pgxntool-test PR (Postgres-Extensions/pgxntool-test#107). This is an addition, and the test PR exercises behavior that only exists here.

pgTap output check

• New test/bin/check-tap-output.sh FILE.... A file counts as TAP output if it has a top-level ok N/not ok N or 1..N line; all other files pass untouched. Nonexistent arguments are ignored, so an unmatched glob is harmless. 1..0 with no tests passes. --list prints which files count as TAP output and checks nothing.
• The test recipe runs it over $(TESTOUT)/results/*.out plus each test/install actual output, then still prints regression.diffs. If either check fails, the run fails, and both are reported.
• PGXNTOOL_VERIFY_RESULTS_MODE decides whether make test runs the check: it does in the pgtap (default) and pgtap-only modes, and skips it in diffs mode. The check lives in the test recipe rather than TEST_DEPS, so make PGXNTOOL_ENABLE_VERIFY_RESULTS=no results still works as an escape hatch.
• verify-results-pgtap.sh uses the same script for its TAP phase, taking the test/install outputs as extra arguments. make results therefore gains the plan checks and the test/install scan.

pgtap-only mode

• A test whose result file is TAP output (by check-tap-output.sh's definition) passes on two things: the pgTap output check, and no SQL error line (the (^|: )ERROR: match verify-results-pgtap.sh already uses). Its regression.diffs block is ignored. This applies to make test and make results alike; make test prints one NOTE line naming the TAP tests whose differences it ignored.
• A test without TAP output is still judged by its diff: any block fails make test, and make results applies the existing pgtap rules, including the allowance for a test with no expected output yet. test/install output gets the TAP check only.
• verify-results-pgtap.sh takes --pgtap-only, plus --for-test for the make test rule that every remaining block fails. One regression.diffs parser serves both targets.
• Apart from the above, pgtap and diffs modes behave as before.

For review: where the mode lives

The mode is a third value of the existing PGXNTOOL_VERIFY_RESULTS_MODE. That variable now governs make test's pass criterion too, although its name is about verify-results. The alternative is a separate variable that governs make test's pass criterion. This PR does not take that route; it is your call at review.

test/install with a separate TESTOUT

pg_regress writes test/install actual output to $(TESTOUT)/results/../install/<name>.out and diffs it against $(TESTDIR)/expected/../install/<name>.out. Those are the same file only when TESTOUT = TESTDIR, so make test TESTOUT=dir failed. When test/install is enabled and TESTOUT differs, a new _test-install-output-link rule (an order-only prerequisite of installcheck) makes $(TESTOUT)/install a symlink to $(TESTDIR)/install, on every PostgreSQL version. If something else is already at that path, the build fails rather than replace it. With TESTOUT = TESTDIR nothing changes. test/install output therefore always lands in $(TESTDIR)/install/ (gitignored).

Reproduced with make test TESTOUT=$(mktemp -d) on a repo built from the pgxntool-test template (PostgreSQL 17):

• before: exit 2, cannot create .../results/../install/create_install_marker.out: Directory nonexistent, then Bail out!
• after: exit 0, # All 4 tests passed., with $TESTOUT/install -> <repo>/test/install
• after, with a real directory already at $TESTOUT/install: exit 2, ERROR: <dir>/install exists and is not a symlink to <repo>/test/install

Rejected alternatives:

• a fail-fast error for TESTOUT != TESTDIR: leaves a documented variable broken whenever test/install is used
• --expecteddir: PostgreSQL 16+ only
• empty placeholder expected files under $(TESTOUT): a diff on every run

README.asc, CLAUDE.md and HISTORY.asc are updated. HISTORY lists the TAP check under breaking changes, pgtap-only under new features and the TESTOUT fix under bug fixes.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 4d599c9e-46bf-46d1-ae16-a87ae6903662

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}