Skip to content

Select the Ceph backend with CEPH_STACK - #2960

Merged
berendt merged 4 commits into
mainfrom
ceph-stack-selectable
Oct 2, 2026
Merged

berendt merged 4 commits into
mainfrom
ceph-stack-selectable

Conversation

@ideaship

@ideaship ideaship commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

The testbed has carried a deploy script for each Ceph backend for a while, scripts/deploy/100-ceph-with-ansible.sh and 100-ceph-with-cephadm.sh, and deploy-manager.sh links both into /usr/local/bin. Nothing chose between them: deploy-services.sh and deploy-ceph.sh named the ceph-ansible script outright, so a cephadm deployment could only be had by running the script by hand after the fact.

The variable to choose with already exists too. terraform's ceph_stack is written into /opt/manager-vars.sh as CEPH_STACK, which every script here sources, and terraform/Makefile passes it through from a make variable. It was inert because nothing read it.

Dispatch on it. deploy_ceph in include.sh picks the script for the configured backend and fails on an unknown value rather than silently deploying the default, and the two deploy wrappers call it. deploy.yml threads a ceph_stack job variable to make, so a Zuul job can select the backend the same way it selects ceph_version, defaulting to ceph-ansible where it says nothing.

The upgrade path deliberately does not follow. Upgrading Ceph under cephadm runs through ceph orch upgrade and there is no script for it, so upgrade-services.sh and upgrade-ceph.sh refuse on a cephadm cluster instead of running the ceph-ansible upgrade against one. In upgrade-services.sh the refusal is skipped when SKIP_CEPH_UPGRADE is already set, so the rest of an upgrade can still be exercised.

The Ceph check keeps no backend in its name. check/100-ceph-with-ansible.sh had nothing ceph-ansible specific left in it: every command is the plain Ceph CLI through cephclient, which both backends deploy, and the ceph-mons, ceph-mgrs and ceph-osds validators match either backend's daemon container names. It is renamed to check/100-ceph.sh rather than duplicated behind a dispatch, and gains an orchestrator section under cephadm: it fails when a service runs fewer daemons than its placement asks for — down to none, the failure a healthy-looking ceph -s hides. The backend is read from the cluster (ceph orch status), not from CEPH_STACK, which a nutshell testbed leaves unset. The dedicated-key scope check (101-ceph-ssh-key-scope.sh) runs after it as before.

A cephadm testbed runs no ceph-ansible container. On the latest track the manager's ceph-ansible image tag follows ceph_version, and ceph-ansible has no branch past squid, so a Tentacle testbed would ask for an image that does not exist. 000-manager.sh sets ceph_ansible_enable: false when CEPH_STACK is cephadm, the same way cfg-cookiecutter disables the container for Ceph releases ceph-ansible cannot deploy, and the manager waits for the container only when the manager configuration enables it, so a container disabled for any reason is not waited for. A nutshell testbed leaves CEPH_STACK unset because python-osism selects the backend there; the Ceph-release rule for that case is in #2961. Nothing on the cephadm path needs it: the Ceph version pins come from osism-ansible for numbered releases and from osism/defaults on the latest track, and the OSD LVM plays run in osism-ansible (osism/ansible-playbooks#576, used by #2959).

Every environment sees the selected Ceph release. set-ceph-version.sh also writes ceph_version into environments/configuration.yml. Without a ceph-ansible container, whose versions.yml used to carry the Ceph pins, cephclient_version and ceph_image_version fall back to ceph_version, which only the manager environment had. cfg-cookiecutter makes the same move for generated configurations (osism/cfg-cookiecutter#882).

Accepting the value also needs osism/terraform-base, which owns the ceph_stack variable and validates it.

Part of a cross-repo effort:

🤖 Generated with Claude Code

The testbed has carried a deploy script for each Ceph backend for a
while, scripts/deploy/100-ceph-with-ansible.sh and
100-ceph-with-cephadm.sh, and deploy-manager.sh links both into
/usr/local/bin. Nothing chose between them: deploy-services.sh and
deploy-ceph.sh named the ceph-ansible script outright, so a cephadm
deployment could only be had by running the script by hand after the
fact.

The variable to choose with already exists too. terraform's ceph_stack
is written into /opt/manager-vars.sh as CEPH_STACK, which every script
here sources, and terraform/Makefile passes it through from a make
variable. It was inert because nothing read it.

Dispatch on it. deploy_ceph in include.sh picks the script for the
configured backend and fails on an unknown value rather than silently
deploying the default, and the two deploy wrappers call it. deploy.yml
threads a ceph_stack job variable to make, so a Zuul job can select the
backend the same way it selects ceph_version, defaulting to
ceph-ansible where it says nothing.

The upgrade path deliberately does not follow. Upgrading Ceph under
cephadm runs through "ceph orch upgrade" and there is no script for it,
so upgrade-services.sh and upgrade-ceph.sh refuse on a cephadm cluster
instead of running the ceph-ansible upgrade against one. In
upgrade-services.sh the refusal is skipped when SKIP_CEPH_UPGRADE is
already set, so the rest of an upgrade can still be exercised.

Accepting the value also needs osism/terraform-base, which owns the
ceph_stack variable and validates it.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
ideaship force-pushed the ceph-stack-selectable branch 2 times, most recently from 25cf7b5 to 8ba1984 Compare October 1, 2026 09:37
@ideaship ideaship self-assigned this Oct 1, 2026
scripts/check/100-ceph-with-ansible.sh has nothing ceph-ansible
specific left in it. Every command it runs is the plain Ceph CLI
through the cephclient container, which both backends deploy, and the
three validators it calls -- osism validate ceph-mons, ceph-mgrs and
ceph-osds -- were taught to match cephadm's daemon container names and
to reach Ceph through cephclient. Only the name still said
ceph-ansible, and the two wrappers called it unconditionally, so a
cephadm testbed ran a check that claimed to be for the other backend.

Rename it to 100-ceph.sh rather than adding a second, near-identical
script and a dispatch in front of it. The deploy and upgrade steps keep
their per-backend names because those really are different procedures.
Nothing else referred to the old check name: deploy-manager.sh links
the deploy and upgrade scripts into /usr/local/bin, not this one.

Append the one thing that is backend specific. Under cephadm the
orchestrator holds state with no ceph-ansible equivalent -- the services
it has been asked to run and the daemons it placed for them. A service
running fewer daemons than its placement asks for, down to none at all,
is exactly the failure a healthy-looking "ceph -s" hides, so print
orch ls, orch ps and orch host ls, and fail when a service is short.

Read the backend from the cluster (ceph orch status) rather than from
CEPH_STACK: a nutshell testbed selects cephadm in python-osism and
leaves CEPH_STACK unset, so keying on it would skip the check exactly
on the lanes that deploy cephadm that way.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
On the latest track the manager's ceph-ansible image tag follows
ceph_version. A cephadm testbed on Tentacle therefore asks for
ceph-ansible:tentacle, an image that does not exist, because
ceph-ansible has no branch past squid.

Nothing on the cephadm path needs that container any more. The Ceph
version pins it supplied come from osism-ansible for numbered releases
and from osism/defaults on the latest track, and the OSD LVM plays run
in osism-ansible. So set ceph_ansible_enable: false in the manager
configuration when CEPH_STACK is cephadm, as cfg-cookiecutter does for
Ceph releases ceph-ansible cannot deploy.

The manager waits for the ceph-ansible container only when the manager
configuration enables it, rather than keying on CEPH_STACK, so a
container disabled for any reason is not waited for. A nutshell testbed
selects its Ceph backend in python-osism and leaves CEPH_STACK unset,
so CEPH_STACK alone cannot tell whether the container runs.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
set-ceph-version.sh wrote ceph_version into the manager environment
only. The plays of the other environments never load that file; they
got their Ceph pins from the ceph-ansible container's versions.yml,
which names the release its image tag selects. A cephadm testbed runs
no ceph-ansible container, so cephclient_version and
ceph_image_version fall back to ceph_version, which those plays do not
see, and `osism apply cephclient` fails on it.

Write ceph_version into environments/configuration.yml as well, which
every environment loads, next to ceph_cluster_fsid. The manager
environment keeps its copy, which the testbed's own manager deployment
reads. cfg-cookiecutter makes the same move for generated
configurations.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
ideaship force-pushed the ceph-stack-selectable branch from 8ba1984 to f6dd01c Compare October 1, 2026 13:32
@ideaship
ideaship marked this pull request as ready for review October 2, 2026 10:53
@berendt
berendt merged commit ec8a1bb into main Oct 2, 2026
2 checks passed
@berendt
berendt deleted the ceph-stack-selectable branch October 2, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

2 participants