Skip to content

ci: publish each release to PyPI, npm and crates.io, and open the Homebrew formula PR - #80

Merged
IsuraManchanayake merged 4 commits into
mainfrom
ci/publish-packages
Oct 26, 2026
Merged

IsuraManchanayake merged 4 commits into
mainfrom
ci/publish-packages

Conversation

@IsuraManchanayake

@IsuraManchanayake IsuraManchanayake commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Publishes each BaseRT release to PyPI, npm and crates.io and opens the Homebrew formula PR, from this repository.

What this adds

  • .github/workflows/publish.yml runs when a release is published, and by hand for a tag (tag, dry_run). It builds every package from this repository's checkout of the tag and the release's engine bundles:

    • PyPI: basert
    • npm: @basecompute/basert and its engine packages
    • crates.io: the basert-* CLI crates, baseRT-sys and baseRT
    • Homebrew: basert.rb for basecompute/homebrew-tap

    It then installs each package on macOS 15 and Linux arm64 and verifies the crates with cargo publish --workspace --dry-run.

  • packaging/ and tests/scripts/test_packaging.py: the scripts and their tests. Like bindings/, they arrive with the source sync, which keeps them current.

  • ci.yml: a packaging job (ubuntu-24.04-arm) runs those tests.

When it uploads

Only when all of these hold; otherwise the run is a dry run that still builds and tests everything:

  • the variable PUBLISH_ENABLED is "true" and dry_run is not ticked;
  • the release is published: not a draft, not a pre-release;
  • the run is at the commit the tag names;
  • every version file in that tree (base-convert, the Rust, Python and Node bindings, the header) states the tag's version.

Each upload job asks its registry first, so a rerun finishes an interrupted one and repeats nothing. The Linux wheel and npm engine package are built and tested every time but uploaded only with PUBLISH_LINUX (the engine licence covers Apple Silicon only for now). Uploads carry PyPI attestations and npm provenance. Crates go up with --no-verify, after the verify job has built them. Actions are pinned to commits. Tools are pinned to exact versions: Python 3.14.9, Node 24.21.0, Rust 1.99.0, build/wheel/twine.

Release order

From 0.4.2 on, a release's source sync PR is merged before the release is tagged. The engine's release pipeline refuses a tag whose source this repository doesn't hold, and creates the release at the commit that holds it. Publishing that release starts this workflow on that commit. Earlier tags were made before their sync merged: v0.4.1 here names a commit whose bindings still say 0.4.0, and the plan step refuses it. So 0.4.2 is the first release this can publish.

Setup before PUBLISH_ENABLED

  • Environments pypi, npm, crates, homebrew, each limited to v* tags.
  • PyPI: a pending trusted publisher for project basert: owner basecompute, repository baseRT, workflow publish.yml, environment pypi.
  • npm: secret NPM_TOKEN (environment npm), a token that can publish to @basecompute.
  • crates.io: secret CARGO_REGISTRY_TOKEN (environment crates). Its account must also be an owner of the existing baseRT-sys crate.
  • Homebrew: secret HOMEBREW_TAP_TOKEN (environment homebrew), a fine-grained token with Contents and Pull requests write on basecompute/homebrew-tap.
  • Then the variable PUBLISH_ENABLED=true, and PUBLISH_LINUX=true once the Linux licence allows.

Tested

Locally on macOS arm64, against the v0.4.1 release bundles, with this branch plus the 0.4.1 source and these packaging files applied:

  • The plan step's decisions: dry run for each unmet condition, upload at the tagged commit, refusal of the current v0.4.1 tag's tree, a missing release.
  • The registry probes, including their "already there" paths against real packages.
  • The packages, built from the checkout and checked with twine check --strict.
  • pip and npm installs running basert, plus a C program linked against each installed engine.
  • brew style, install and test from a throwaway tap.
  • cargo publish --workspace --dry-run for all 10 crates.

Not run: the workflow itself on Actions, the Linux installs, any upload.

…tests

packaging/ turns a release's engine bundles into the PyPI wheel (basert),
the npm packages (@basecompute/basert and its engine packages), the
crates.io crates and the Homebrew formula; tests/scripts/test_packaging.py
tests them on synthetic bundles. They are the engine tree's copy, written by
its source sync (tools/sync_public.py), which keeps them current from now on.
…ebrew formula PR

publish.yml runs when a release is published and by hand for a tag. It
builds every package from this repository's checkout and the release's
engine bundles, installs each on macOS and on Linux arm64, verifies the
crates with cargo publish --dry-run, and uploads only when PUBLISH_ENABLED is
"true", the release is published (not a draft or pre-release), and the run
is at the commit the tag names; every version file in that tree must say
the tag's version. Each upload job asks its registry first, so a rerun
repeats nothing. The Linux wheel and npm engine package go up only with
PUBLISH_LINUX. Being a public repository, it publishes with PyPI attestations
and npm provenance. Actions are pinned to commits, tools to exact versions.

ci.yml runs the packaging tests.
Comment thread .github/workflows/ci.yml Fixed
CodeQL (actions/missing-workflow-permissions) flagged the new packaging
job, and the three jobs before it on main. No job writes: each checks
out the repository, and rust-sys-abi downloads the latest release. The
repository's default token is read-only already; this says so in the
workflow.

@IsuraManchanayake IsuraManchanayake left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex Review

Reviewed e40be2c28e809ccf1a9fabe333c8f9792a9424f5, including the release trigger, source/tag/version checks, registry rerun logic, job dependencies, permissions, package scripts, and the companion source-sync/release handoff in internal #679.

One blocking finding: the Python 3.12.15 pin cannot be installed by setup-python on the required macOS runner. See the inline comment. All four publishing destinations are consequently blocked even though ordinary PR CI is green.

Validation:

  • All 17 packaging tests passed.
  • actionlint passed for publish.yml; all 19 extracted shell steps passed Bash syntax checking.
  • Replayed the actual plan/registry scripts with controlled command responses: 9 release/upload-gate cases and 5 registry/partial-upload cases passed, including annotated tags, mismatched commits, version mismatch, missing releases, and the Linux gate.
  • Verified the configured Node and Python packaging-tool versions exist. Reproduced the Python setup failure using the exact pinned action commit and checked its distribution manifest and the hosted macOS image inventory.
  • Compared the packaging helpers with the previously reviewed internal #679 implementation: differences are comments only.

The package/runtime integration still relies on the documented #679 source sync landing before the first release. No Actions dispatch, registry upload, Linux execution, or complete package installation was performed in this follow-up; these checks do not substitute for a full workflow dry run after fixing the pin.

Comment thread .github/workflows/publish.yml Outdated

env:
TAG: ${{ github.event.release.tag_name || inputs.tag }}
PYTHON_VERSION: "3.12.15"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex Review — [P1] Use a Python pin available on macOS

test-macos uses this pin at lines 267–269, but the official Python versions manifest provides 3.12.15 only for Linux/RHEL, with no Darwin build. The macOS 15 arm64 image has 3.12.10 cached, so the exact 3.12.15 request cannot fall back to it. I ran the pinned setup-python commit locally on macOS arm64 with the action's input defaults: it exits 1 with The version '3.12.15' with architecture 'arm64' was not found. This fails the job before any package install; PyPI, npm, crates.io and Homebrew all depend on that job, so no release can publish. Please select an exact version available on both platforms (for example, the manifest contains 3.13.15 for Darwin and Linux arm64/x64), then exercise the macOS setup step in the dry run.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 635eab8: PYTHON_VERSION is now 3.14.9. The manifest has no macOS build for any 3.12 release after 3.12.10. 3.13's bugfix releases are ending, so a later 3.13 bump would hit the same gap. 3.14.9 has builds for darwin arm64 and Linux arm64/x64, and build 1.6.1, wheel 0.48.0 and twine 7.0.0 all declare 3.14.

I replayed the Python steps locally on 3.14.7:

  • the 17 packaging tests pass;
  • both v0.4.1 bundles build wheels that pass twine check --strict;
  • the macOS wheel installs and its launcher runs on 3.14.7 and on 3.9.6. The wheel stays py3-none with Requires-Python: >=3.9.

The setup step first runs for real after merge, because workflow_dispatch needs the workflow on the default branch. A v0.4.1 dispatch exercises setup and the builds. Its bindings checks will still fail until the 0.4.2 source sync brings the _engine/ lookup to main. 0.4.2's own release run is the first that can pass end to end.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex Review — verified fixed in 635eab8. The new pin has builds for all workflow runner platforms. I also ran the exact official macOS Python 3.14.9 runtime: all 17 packaging tests pass, both wheels build and pass strict Twine checks, and the macOS wheel CLI works on 3.14.9 and 3.9.6. The previous P1 no longer applies; the full Actions dry run remains a separate validation step.

test-macos asked setup-python for 3.12.15, which actions/python-versions
builds for Linux only: the 3.12 line has had no macOS build since
3.12.10, when it went security-only. The step failed before anything was
installed, and every upload job needs test-macos.

3.14.9 has builds for macOS arm64 and for Linux arm64 and x64, and
build 1.6.1, wheel 0.48.0 and twine 7.0.0 all declare 3.14. 3.13's
bugfix releases end around now, so a later 3.13 bump would hit the same
gap. The pin only picks the interpreter these jobs run: the wheel is
py3-none and still requires Python >=3.9.

@IsuraManchanayake IsuraManchanayake left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex Review

Reviewed head 635eab89b0b458eaa910e267580af9c8b9e869f8 and the reply to the previous finding. The only code change since my last review is the Python pin plus its explanatory comment. The previous P1 is fixed; no new actionable findings.

Validation of the replacement pin:

  • Confirmed actions/python-versions supplies 3.14.9 for Darwin arm64 and Ubuntu 24.04 arm64/x64, covering the workflow runners.
  • Downloaded the official macOS 3.14.9 runtime used by the setup action, extracted it into an isolated directory, and ran that exact interpreter. No system-wide Python installation was needed.
  • Installed the exact workflow tools (build==1.6.1, wheel==0.48.0, twine==7.0.0), passed all 17 packaging tests, and rebuilt both v0.4.1 wheels from this checkout and the release bundles. Both pass twine check --strict.
  • The macOS wheel installs and runs basert --version, chat --help, and serve --help on both Python 3.14.9 and 3.9.6. Both wheels retain py3-none tags and Requires-Python: >=3.9.
  • actionlint and diff whitespace checks pass. Completed build/test checks are successful; Swift CodeQL is still running at review time.

This verifies the pin and Python packaging compatibility, not the whole release pipeline. The documented #679 source sync is still required for the packaged bindings' _engine/ discovery; I did not treat these CLI checks as validation of that pending change. A complete Actions dry run, Linux execution, and registry uploads remain untested here.

@IsuraManchanayake
IsuraManchanayake merged commit 6e3375d into main Oct 26, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants