diff --git a/.github/workflows/regression.yml b/.github/workflows/regression.yml new file mode 100644 index 0000000..08e18fb --- /dev/null +++ b/.github/workflows/regression.yml @@ -0,0 +1,39 @@ +# Report-only regression tests (not a required check; never gates merges). +# Harness: tests/regression, from Test Lab's Si5351 regression suite. +name: Regression tests (report-only) + +on: + pull_request: + push: + branches-ignore: [master] + workflow_dispatch: + +permissions: + contents: read + +jobs: + regression: + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.x' + - name: Run regression harness (native g++ against src/si5351.cpp) + working-directory: tests/regression + run: ./run_v231_checks.sh --src "$GITHUB_WORKSPACE" --out "$RUNNER_TEMP/regression" + - name: Summary + if: always() + run: | + if [ -f "$RUNNER_TEMP/regression/summary.md" ]; then + cat "$RUNNER_TEMP/regression/summary.md" | tee -a "$GITHUB_STEP_SUMMARY" + fi + - uses: actions/upload-artifact@v4 + if: always() + with: + name: regression-results + path: | + ${{ runner.temp }}/regression/summary.* + ${{ runner.temp }}/regression/results.tsv + ${{ runner.temp }}/regression/logs/ diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 0000000..754ca3a --- /dev/null +++ b/tests/README.md @@ -0,0 +1,11 @@ +# Tests + +`regression/` is the Si5351 regression harness written by Test Lab (an agent working for Jason Milldrum, NT7S), +ported unchanged from its working copy. It compiles `src/si5351.cpp` natively with g++ against host +Arduino/Wire stubs and checks register output against golden cases, AN619/datasheet oracles, and the +v2.3.1 acceptance scenarios. See `regression/README.md`. + +Run locally: `cd tests/regression && ./run_v231_checks.sh --src ../..` + +In CI (`.github/workflows/regression.yml`) it is **report-only**: not a required check, never gates merges. +In default mode, the 25 v2.3.1 acceptance tests are expected to XFAIL on master (they are bugs v2.3.1 fixes). diff --git a/tests/regression/README.md b/tests/regression/README.md new file mode 100644 index 0000000..9d2eaa3 --- /dev/null +++ b/tests/regression/README.md @@ -0,0 +1,53 @@ +# Si5351Arduino v2.3.1 pre-review checks (`run_v231_checks.sh`) + +One self-contained, repeatable entry point for the v2.3.1 acceptance suite and the regression locks. +Everything it needs is vendored here: no `/workspace` paths and no network access except the optional +read-only git fetch and arduino-cli core installs. It never commits, pushes, or writes to the library repo. +Library sources are always pinned with `git archive` (or copied, for a plain tree) into `/lib`. + +## Requirements +bash, git, tar, g++ (or `CXX=clang++`) with C++17, python3 (3.8+, standard library only). +Optional: `arduino-cli` (≥1.0) for `--cores`; it is looked up as `$ARDUINO_CLI`, then `~/bin/arduino-cli`, then `PATH`. +User-level install: `curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh | BINDIR=~/bin sh` + +## Usage +``` +./run_v231_checks.sh # etherkit/Si5351Arduino master HEAD (read-only shallow fetch) +./run_v231_checks.sh --ref abf9b2a # any branch/tag/SHA from upstream (GitHub serves SHAs) +./run_v231_checks.sh --repo https://github.com//Si5351Arduino.git --ref twowire-bus-select +./run_v231_checks.sh --src ~/src/Si5351Arduino --ref v2.3.1-fixes # a local clone; ref archived, clone untouched +./run_v231_checks.sh --src ~/src/Si5351Arduino # local clone at HEAD (committed state; uncommitted edits are flagged, not tested) +./run_v231_checks.sh --src /path/to/tree # plain source tree (src/si5351.{cpp,h} or si5351.{cpp,h}) +./run_v231_checks.sh --require-fixed ... # v2.3.1 gate: all 25 acceptance tests must PASS +./run_v231_checks.sh --cores # + arduino-cli compiles on all 9 boards (installs missing cores into ~/.arduino15) +./run_v231_checks.sh --cores uno,esp32 --out /tmp/r1 # subset of boards, explicit output dir +``` +Exit code: 0 = every check PASS/XFAIL/SKIP/INFO; 1 = at least one unexpected FAIL; 2 = usage/setup error. + +## What runs +| Check | What | Gating | +|---|---|---| +| `build_acceptance`, `run_acceptance` | `v231/acceptance.cpp` + `scenarios_v231.inc` (per-scenario fork, so SIGFPE is recorded) compiled with the host compiler against the pinned source and the Wire stub | yes | +| `acceptance_A01..A25` | `v231/test_v231.py` AN619/datasheet oracle. **Default mode: a failing test is XFAIL** (master is expected to fail all 25). `--require-fixed`: a failing test is FAIL | per mode | +| `must_not_break` | 776 register/state/return/reg177 locks from `v231/must_not_break.json` | yes | +| `t106_twowire` | issue #106 TwoWire bus-selection test (`issue106/t106.cpp`, multi-bus stub). SKIP when the source has no `TwoWire*` constructor | yes | +| `goldens27_vs_lib` | C++ equivalence harness (`equiv/harness.cpp`) on the pinned source vs the 27 original golden cases | yes (INFO under `--require-fixed`) | +| `equiv_vs_master_1ddd43e` | Full harness transcript diff against vendored master 1ddd43e (`reference/`). 0 means behaviourally identical | INFO | +| `goldens_newer_vs_lib` | Newer `lib_*` golden register maps | INFO | +| `py_test_golden`, `py_test_spec` | Python golden/spec suites: self-consistency of the AN619 oracle and goldens used above (library-independent) | yes | +| `cxx98_syntax` | `-std=c++98 -pedantic -fsyntax-only` warning count | INFO | +| `compile__` (`--cores`) | arduino-cli `--warnings all` compile of `si5351_example`, `si5351_vcxo` and (TwoWire sources only) `sketches/Ctor106` on `boards.txt`. A vcxo `get_correction` error in default mode is XFAIL (known v2.3.0 example bug, fixed in 2.4.0) | yes | + +## Notes +- Version-manager shims (mise, asdf, pyenv) on `PATH` can make cores that call `python3` during the build (esp8266, rp2040) fail with a "shim fallback recursion" error. When a real `python3` is still reachable without them, the script runs arduino-cli with those shim directories removed from `PATH`. +- Cores are installed per user into `~/.arduino15`. Nothing is installed system-wide and sudo is never used. + +## Output (`--out`, default `./runs/run__