Add verify-tap.sh, a standalone TAP output checker

HISTORY.asc

@@ -1,3 +1,39 @@
+STABLE
+------
+== Breaking changes
+
+=== `make results` refuses on more kinds of broken pgTAP output
+In `pgtap` mode, `make results` now also refuses when a `test/results/*.out`
+file holding TAP output has no plan line (`1..N`), more than one plan line,
+a plan line between test lines, a test count that differs from the plan, or
+a `Bail out!` line. A test that died before finishing (a `no_plan()` test
+that never reached `finish()`, or a `plan(N)` test cut short without pgTAP's
+own mismatch line) used to look like a pass. Files without any TAP line are
+still checked only through `regression.diffs`. See #106.
+
+=== `make test` fails on broken pgTAP output, even when blessed
+Under the default `pgtap` mode of `PGXNTOOL_VERIFY_RESULTS_MODE`, `make test`
+now runs `verify-tap.sh --skip-non-tap` on `test/results/*.out` and fails
+when a file holding TAP output has a `not ok` (without `# TODO`), a missing
+or duplicate plan line, a test count that differs from the plan, or a
+`Bail out!` line -- even if that output was blessed into its expected file,
+so it no longer shows up in `regression.diffs`. `regression.diffs` is still
+printed, and both failures are reported. `PGXNTOOL_VERIFY_RESULTS_MODE =
+diffs` turns the check off, though it also switches `make results` to
+`diffs` mode.
+
+== New features
+
+=== Add `verify-tap.sh`, a standalone TAP output checker
+`pgxntool/verify-tap.sh [--skip-non-tap] [FILE|-]...` fails if a TAP stream
+shows a failed run: a `not ok` without `# TODO`, `Bail out!`, pgTAP's "Looks
+like you planned" line, a missing or duplicate plan line, or a test count that
+differs from the plan. It reads stdin by default, judges each file argument
+as its own stream, and exits 0, 1 (a stream failed) or 2 (usage error or
+unreadable file); see README. `--skip-non-tap` skips input holding no TAP
+lines instead of failing it. `make test` and `make results` use it in
+`pgtap` mode.
+
 2.4.0
 -----
 == Breaking changes

README.asc

@@ -52,8 +52,9 @@ Warns if two `$(DOCS)` entries would install to the same filename, which would o
 === test
 Runs your extension's test suite: filesystem-installs the extension (unless `PGXNTOOL_ENABLE_FS_INSTALL=no`) and runs it through PGXS's `installcheck`, first pulling in anything you've hooked into <<_testdeps>> and, if enabled, sanity-checking your test SQL via <<_test_build>>.
 
-`make test` also fails on two safeguard checks:
+`make test` also fails on these safeguard checks:
 
+- In `pgtap` mode (the default; see <<_verify_results_safeguard>>), the TAP check: fails if <<_verify_tap,`verify-tap.sh`>> rejects any `test/results/*.out` file holding TAP output, even when that output matches its expected file. Files with no TAP output are skipped, and `test/install` output is never checked. This runs alongside the `regression.diffs` check, and both are always reported.
 - The stale-expected-file check, run after `pg_regress` finishes: fails if `test/expected/` (or `test/build/expected/`) has a `.out` file with no matching `.sql`, or a file that isn't `.out`. See <<_pgxntool_enable_check_stale_expected>>.
 - When <<_testinstall,test/install>> is in use, the `ON_ERROR_STOP` check: fails if any `test/install/*.sql` file neither includes `test/pgxntool/psql.sql` nor sets `ON_ERROR_STOP` itself. See <<_pgxntool_enable_test_install_error_stop_check>>.
 
@@ -68,7 +69,7 @@ Whether `test-build` runs is controlled by the `PGXNTOOL_ENABLE_TEST_BUILD` vari
 
 NOTE: `test` intentionally does *not* depend on `clean` — that caused problems with incremental/watch-based builds. If your tests need a clean build to pass, that's a sign of a missing dependency elsewhere rather than something to fix by adding `clean` back.
 
-NOTE: `test` exits non-zero (after printing `regression.diffs`) if any test fails, so CI and other automation can rely on its exit code.
+NOTE: `test` exits non-zero (after printing `regression.diffs` and any TAP check failures) if any test fails, so CI and other automation can rely on its exit code.
 
 NOTE: While you can still run `make installcheck` or any other valid PGXS make target directly, it's recommended to use `make test` when using pgxntool. The `test` target ensures proper test isolation and correct dependency installation.
 
@@ -272,10 +273,11 @@ IMPORTANT: *`make results` requires manual verification first*. The correct work
 Never run `make results` without first verifying the test changes are correct. The `results` target copies files from `test/results/` to `test/expected/`, so running it blindly will make incorrect output become the new expected behavior.
 
 ==== verify-results safeguard
-By default, `make results` will refuse to run if your tests are failing, so you can't accidentally promote incorrect output into the new expected results. Which failures it looks for is controlled by `PGXNTOOL_VERIFY_RESULTS_MODE`, which must be one of these two values (case-insensitive; anything else is a hard `make` error):
+By default, `make results` will refuse to run if your tests are failing, so you can't accidentally promote incorrect output into the new expected results. Which failures it looks for is controlled by `PGXNTOOL_VERIFY_RESULTS_MODE`, which also decides whether `make test` runs its TAP check (see <<_test>>). It must be one of these two values (case-insensitive; anything else is a hard `make` error):
 
-`pgtap` (the default):: Scans `test/results/*.out` for pgTAP `not ok` lines (ignoring `# TODO` items) and plan-count mismatches, then also checks `test/regression.diffs` for any output mismatch pgTAP itself wouldn't catch. A test that has no expected output file yet is the one exception: it always shows up in `regression.diffs` (pgxntool creates an empty expected file for it, because `pg_regress` aborts on a missing one), so `make results` is allowed to create its first expected output — provided its own pgTAP output is clean and free of SQL errors. If a new test legitimately needs output containing a SQL error blessed, use `PGXNTOOL_ENABLE_VERIFY_RESULTS=no` for that one run.
-`diffs`:: Only checks whether `test/regression.diffs` exists, so it can't create a new test's first expected output; use `pgtap` mode, or `PGXNTOOL_ENABLE_VERIFY_RESULTS=no` for that one run.
+`pgtap` (the default):: Runs <<_verify_tap,`verify-tap.sh --skip-non-tap`>> on every `test/results/*.out` file, so a `not ok` (ignoring `# TODO` items), a `Bail out!`, or a missing, duplicate or misplaced plan line or a test count that differs from the plan all block; then also checks `test/regression.diffs` for any output mismatch pgTAP itself wouldn't catch. A test that has no expected output file yet is the one exception: it always shows up in `regression.diffs` (pgxntool creates an empty expected file for it, because `pg_regress` aborts on a missing one), so `make results` is allowed to create its first expected output — provided its own pgTAP output is clean and free of SQL errors. If a new test legitimately needs output containing a SQL error blessed, use `PGXNTOOL_ENABLE_VERIFY_RESULTS=no` for that one run.
+`make test` runs the same `verify-tap.sh` check, so a TAP failure fails `make test` even after it has been blessed into an expected file.
+`diffs`:: Only checks whether `test/regression.diffs` exists, and `make test` skips its TAP check. It can't create a new test's first expected output; use `pgtap` mode, or `PGXNTOOL_ENABLE_VERIFY_RESULTS=no` for that one run.
 
 If tests are failing, `make results` prints an explanation of what failed and exits without touching `test/expected/`; fix the failures, then re-run `make results`.
 
@@ -424,6 +426,33 @@ $(MAKE) --no-print-directory -C ../deps/cat_tools print-pgtle >> pgtle-all.sql
 
 WARNING: Always pass `--no-print-directory` when invoking `print-pgtle` recursively from another Makefile's recipe (as above). GNU Make auto-prints `Entering directory`/`Leaving directory` announcements to stdout for recursive invocations, which silently corrupts a redirected combined SQL file with garbage lines. `--no-print-directory` suppresses them.
 
+[[_verify_tap]]
+== verify-tap.sh
+`pgxntool/verify-tap.sh` checks TAP output (such as pgTAP's) and fails if it shows a failed test run. It doesn't need `make` or a database, so you can run it on any TAP output, including outside of pgxntool; `make test` and `make results` use it in `pgtap` mode (see <<_verify_results_safeguard>>).
+
+----
+pgxntool/verify-tap.sh [--skip-non-tap] [FILE|-]...
+----
+
+With no arguments, or for a `-` argument, it reads stdin; otherwise each `FILE` is judged as its own TAP stream:
+
+----
+psql -XtA -f my_test.sql | pgxntool/verify-tap.sh
+pgxntool/verify-tap.sh test/results/*.out
+----
+
+Only unindented lines count, so subtest output is ignored. A stream passes only if:
+
+* no `not ok` line lacks `# TODO`
+* there is no `Bail out!` line
+* there is no pgTAP `# Looks like you planned` line
+* there is exactly one plan line (`1..N`), before every test line (`plan(N)`) or after every one (`no_plan()` plus `finish()`)
+* the number of test lines (`ok N`/`not ok N`) equals `N`; `1..0` with no tests passes
+
+Input with no plan line and no test lines fails, since it means nothing ran. With `--skip-non-tap`, such input (plain SQL output, say) is skipped silently instead, which lets you pass a mix of TAP and non-TAP files. Each problem is reported on stderr, naming its source (`stdin` or the file's path).
+
+Exit status is 0 if every input passes, 1 if any input fails, and 2 on a usage error or an unreadable file (in which case nothing is checked).
+
 == Version-Specific SQL Files
 
 PGXNtool automatically generates version-specific SQL files from your base SQL file. These files follow the pattern `sql/{extension}--{version}.sql` and are used by PostgreSQL's extension system to install specific versions of your extension.
@@ -786,15 +815,15 @@ Default: `$(TESTDIR)`. Directory `pg_regress` writes actual test output to -- `$
 
 === PGXNTOOL_VERIFY_RESULTS_MODE
 
-Default: `pgtap`. Controls how the <<_verify_results_safeguard,verify-results safeguard>> decides whether tests are failing. Must be `pgtap` or `diffs` (case-insensitive) when given explicitly; any other value, including an empty one, is a hard `make` error.
+Default: `pgtap`. Controls how the <<_verify_results_safeguard,verify-results safeguard>> decides whether tests are failing, and whether <<_test,`make test`>> runs its TAP check. Must be `pgtap` or `diffs` (case-insensitive) when given explicitly; any other value, including an empty one, is a hard `make` error.
 
 ==== pgtap
 
-Scans `test/results/*.out` for pgTap `not ok` lines and plan mismatches, falling back to checking `regression.diffs` too — except for a test that has no expected output yet, whose first expected file `make results` is allowed to create. Use this mode when your test suite uses pgTap.
+Runs <<_verify_tap,`verify-tap.sh --skip-non-tap`>> on each `test/results/*.out` file, falling back to checking `regression.diffs` too — except for a test that has no expected output yet, whose first expected file `make results` is allowed to create. `make test` runs the same `verify-tap.sh` check alongside its `regression.diffs` check, so blessed TAP failures still fail it. Use this mode when your test suite uses pgTap.
 
 ==== diffs
 
-Checks only for `regression.diffs`, matching classic pg_regress behavior. Use this mode when your tests use plain SQL expected-output comparison only.
+Checks only for `regression.diffs`, matching classic pg_regress behavior; `make test` doesn't run `verify-tap.sh`. Use this mode when your tests use plain SQL expected-output comparison only.
 
 === PGXNTOOL_ENABLE_TEST_BUILD *