diff --git a/CLAUDE.md b/CLAUDE.md index cca492c..d086fc4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -207,7 +207,7 @@ When tests fail, examine the diff output carefully. The actual test output in `t **Exceptions to the above** -- `test-build` and `test/install` (both optional, see `README.asc`) don't follow the `test/results` vs `test/expected` model: - **test-build** runs first, in its own separate `pg_regress` pass over `test/build/*.sql`, and gates the main suite: if it fails, `test/install`/`test/sql` never run at all. It does compare actual vs expected normally (`test/build/results/` vs `test/build/expected/`) -- use `make results-build` to refresh its expected output, not `make results`. -- **test/install** does NOT get a real diff at all: its actual output is written to the exact same file as its expected output, so a content difference can never fail the build, no matter what changed. The only thing that still fails the build is a hard SQL error, and only if the file has `ON_ERROR_STOP` set (directly, or via `\i test/pgxntool/psql.sql`) -- pgxntool checks for this by default, passing any file that includes `psql.sql` or has its own `\set`/`\unset ON_ERROR_STOP`, whatever the value. If a `test/install/*.sql` file is misbehaving, don't go looking for a diff; check whether it errored, and don't assume a stale-looking `.out` for it means anything. +- **test/install** only sets up the environment for the real tests; its output is not a test result (not compared against expected output, not validated as TAP), but a failure in it must abort the run. It does NOT get a real diff at all: its actual output is written to the exact same file as its expected output, so a content difference can never fail the build, no matter what changed. The only thing that still fails the build is a hard SQL error, and only if the file has `ON_ERROR_STOP` set (directly, or via `\i test/pgxntool/psql.sql`) -- pgxntool checks for this by default, passing any file that includes `psql.sql` or has its own `\set`/`\unset ON_ERROR_STOP`, whatever the value. If a `test/install/*.sql` file is misbehaving, don't go looking for a diff; check whether it errored, and don't assume a stale-looking `.out` for it means anything. ## Key Implementation Details diff --git a/README.asc b/README.asc index 92f132d..be587b7 100644 --- a/README.asc +++ b/README.asc @@ -166,7 +166,11 @@ cp test/build/results/.out test/build/expected/.out ---- === test/install -Runs setup files before the main test suite within the same `pg_regress` invocation. This allows expensive one-time operations (like extension installation) to set up state that persists into the regular test files. +`test/install` only prepares the environment for your real tests; its output is not a test result. If a file in it fails, the run aborts, but short of a failure nothing about its output is checked: it is not compared against expected output and not validated as TAP, so don't write tests there (put them in `test/sql`). + +Abort-on-failure relies on `ON_ERROR_STOP`: pgxntool's static check requires every `test/install/*.sql` file to include `test/pgxntool/psql.sql` (which sets it) or to `\set` or `\unset ON_ERROR_STOP` itself. An explicit setting means the author has chosen deliberately. + +It runs setup files before the main test suite within the same `pg_regress` invocation. This allows expensive one-time operations (like extension installation) to set up state that persists into the regular test files. **How it works:** @@ -202,11 +206,11 @@ PGXNTOOL_ENABLE_TEST_INSTALL = yes # or no **Why this is useful:** -Without `test/install`, each test file typically needs to run `CREATE EXTENSION` in its setup, which adds overhead and doesn't allow validating the installation step separately. With `test/install`, setup runs once before all tests, and any state it creates (tables, extensions, etc.) is available to every subsequent test file. +Without `test/install`, each test file typically needs to run `CREATE EXTENSION` in its setup, which adds overhead and repeats the installation step in every test. With `test/install`, setup runs once before all tests, and any state it creates (tables, extensions, etc.) is available to every subsequent test file. **Key detail:** Install files and regular tests run in a single `pg_regress` invocation. This means the database is NOT dropped between install and test phases — state created by install files persists into the main test suite. -WARNING: **`+test/install/*.out+` is never actually compared against anything.** Unlike every other test type pgxntool supports, the `test/install` directory's actual output is written to the exact same file as its expected output, so there is no diff — a wrong or changed result will never fail the build, no matter how much it changes. The *only* thing that still fails the build is a hard SQL error, and only if the file sets `ON_ERROR_STOP` (directly, or via `\i test/pgxntool/psql.sql`) — without it, psql prints the error, keeps going, and the file "passes" regardless. pgxntool enforces this by default (`_check-test-install-error-stop`, see `PGXNTOOL_ENABLE_TEST_INSTALL_ERROR_STOP_CHECK` to disable), but that only guarantees errors are caught — it does not give you real output validation. If you need actual output comparison, put that logic in `test/sql` instead (or use pgTap assertions from within the install file itself). +WARNING: **`+test/install/*.out+` is not a test result and is never compared against anything.** Unlike every other test type pgxntool supports, the `test/install` directory's actual output is written to the exact same file as its expected output, so there is no diff — a wrong or changed result will never fail the build, no matter how much it changes. The *only* thing that still fails the build is a hard SQL error, and only if the file sets `ON_ERROR_STOP` (directly, or via `\i test/pgxntool/psql.sql`) — without it, psql prints the error, keeps going, and the file "passes" regardless. pgxntool enforces this by default (`_check-test-install-error-stop`, see `PGXNTOOL_ENABLE_TEST_INSTALL_ERROR_STOP_CHECK` to disable), but that only guarantees errors are caught — it does not give you real output validation. If you need actual output comparison, put that logic in `test/sql` instead (or use pgTap assertions from within the install file itself). ==== Update & Upgrade (U&U) Testing diff --git a/README.html b/README.html index 10a92d7..c91f7dd 100644 --- a/README.html +++ b/README.html @@ -850,7 +850,13 @@

4.5. test/install

-

Runs setup files before the main test suite within the same pg_regress invocation. This allows expensive one-time operations (like extension installation) to set up state that persists into the regular test files.

+

test/install only prepares the environment for your real tests; its output is not a test result. If a file in it fails, the run aborts, but short of a failure nothing about its output is checked: it is not compared against expected output and not validated as TAP, so don’t write tests there (put them in test/sql).

+
+
+

Abort-on-failure relies on ON_ERROR_STOP: pgxntool’s static check requires every test/install/*.sql file to include test/pgxntool/psql.sql (which sets it) or to \set or \unset ON_ERROR_STOP itself. An explicit setting means the author has chosen deliberately.

+
+
+

It runs setup files before the main test suite within the same pg_regress invocation. This allows expensive one-time operations (like extension installation) to set up state that persists into the regular test files.

How it works:

@@ -914,7 +920,7 @@

Why this is useful:

-

Without test/install, each test file typically needs to run CREATE EXTENSION in its setup, which adds overhead and doesn’t allow validating the installation step separately. With test/install, setup runs once before all tests, and any state it creates (tables, extensions, etc.) is available to every subsequent test file.

+

Without test/install, each test file typically needs to run CREATE EXTENSION in its setup, which adds overhead and repeats the installation step in every test. With test/install, setup runs once before all tests, and any state it creates (tables, extensions, etc.) is available to every subsequent test file.

Key detail: Install files and regular tests run in a single pg_regress invocation. This means the database is NOT dropped between install and test phases — state created by install files persists into the main test suite.

@@ -926,7 +932,7 @@

Warning

-test/install/*.out is never actually compared against anything. Unlike every other test type pgxntool supports, the test/install directory’s actual output is written to the exact same file as its expected output, so there is no diff — a wrong or changed result will never fail the build, no matter how much it changes. The only thing that still fails the build is a hard SQL error, and only if the file sets ON_ERROR_STOP (directly, or via \i test/pgxntool/psql.sql) — without it, psql prints the error, keeps going, and the file "passes" regardless. pgxntool enforces this by default (_check-test-install-error-stop, see PGXNTOOL_ENABLE_TEST_INSTALL_ERROR_STOP_CHECK to disable), but that only guarantees errors are caught — it does not give you real output validation. If you need actual output comparison, put that logic in test/sql instead (or use pgTap assertions from within the install file itself). +test/install/*.out is not a test result and is never compared against anything. Unlike every other test type pgxntool supports, the test/install directory’s actual output is written to the exact same file as its expected output, so there is no diff — a wrong or changed result will never fail the build, no matter how much it changes. The only thing that still fails the build is a hard SQL error, and only if the file sets ON_ERROR_STOP (directly, or via \i test/pgxntool/psql.sql) — without it, psql prints the error, keeps going, and the file "passes" regardless. pgxntool enforces this by default (_check-test-install-error-stop, see PGXNTOOL_ENABLE_TEST_INSTALL_ERROR_STOP_CHECK to disable), but that only guarantees errors are caught — it does not give you real output validation. If you need actual output comparison, put that logic in test/sql instead (or use pgTap assertions from within the install file itself). @@ -2217,7 +2223,7 @@