Skip to content

upgrade guide: upgrade Ceph managed by cephadm - #1143

Open
ideaship wants to merge 1 commit into
mainfrom
docs/ceph-tentacle-upgrade
Open

ideaship wants to merge 1 commit into
mainfrom
docs/ceph-tentacle-upgrade

Conversation

@ideaship

@ideaship ideaship commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

The Ceph upgrade page only covers clusters managed by ceph-ansible. A cluster managed by cephadm, whether deployed new with OSISM 11 or migrated with the cephadm migration guide, is upgraded by cephadm itself, and nothing described how.

The page gains that part, from Reef to Tentacle with OSISM 11:

  • Preconditions, including Step 7 of the migration guide on a migrated cluster: Tentacle reports the cephx entities it removes as HEALTH_ERR.
  • Set ceph_version: tentacle, and take the target image from the release the manager runs.
  • Check, start and follow ceph orch upgrade, with what the cluster shows while it runs. That includes the key rotation cephadm does after the version upgrade, and a short HEALTH_ERR for AUTH_INSECURE_SERVICE_TICKETS that clears on its own.
  • Remove rgw_keystone_api_version only once RGW runs Tentacle, then redeploy the RGW daemons one at a time. On Reef, removing it breaks Keystone authentication.
  • Update the Ceph client.

After the upgrade the cluster is HEALTH_WARN for the cephx keys of the old type that cephadm does not rotate. The page says the cluster works with that, and warns against rotating the keys of the OpenStack services without distributing them. Rotating those keys will be covered separately.

The procedure was run end to end on test clusters (Reef 18.2.8 to Tentacle 20.2.4) and followed blind by an operator with only the text. It has since been checked against OSISM 11.0.3 (Ceph 20.2.5).

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

✅⚠️MegaLinter analysis: Success with warnings

Descriptor Linter Files Fixed Errors Max errors Warnings Elapsed time
✅ ACTION actionlint 5 0 0 0.08s
✅ JSON jsonlint 4 0 0 0.09s
✅ JSON prettier 4 0 0 0.41s
✅ JSON v8r 4 0 0 9.44s
✅ MARKDOWN markdownlint 184 0 0 3.35s
✅ MARKDOWN markdown-table-formatter 184 0 0 0.45s
✅ REPOSITORY betterleaks yes no no 0.83s
✅ REPOSITORY checkov yes no no 20.6s
✅ REPOSITORY git_diff yes no no 0.09s
✅ REPOSITORY secretlint yes no no 1.62s
✅ REPOSITORY trufflehog yes no no 4.39s
✅ SPELL codespell 194 0 0 0.68s
⚠️ SPELL lychee 194 5 0 22.55s
✅ YAML prettier 6 0 0 0.43s
✅ YAML v8r 6 0 0 8.25s
✅ YAML yamllint 6 0 0 0.6s

Detailed Issues

⚠️ SPELL / lychee - 5 errors
📝 Summary
---------------------
🔍 Total.........1306
🔗 Unique.........960
✅ Successful....1178
⏳ Timeouts.........3
🔀 Redirected.......9
👻 Excluded.......120
❓ Unknown..........0
🚫 Errors...........5
⛔ Unsupported......5

Errors in .claude/skills/security-advisory/reference/style.md
[ERROR] file://.claude/skills/guides/operations-guide/openstack/security-updates.md (at 189:3) | File not found. Check if file exists and path is correct

Errors in .claude/skills/security-advisory/SKILL.md
[400] https://cveawg.mitre.org/api/cve/ (at 16:18) | Rejected status code: 400 Bad Request
[404] https://raw.githubusercontent.com/osism/ (at 17:18) | Rejected status code: 404 Not Found
[400] https://registry.osism.tech/service/token (at 18:19) | Rejected status code: 400 Bad Request

Errors in docs/concepts/cluster-network.md
[TIMEOUT] https://stordis.com/basic-configuration-and-management-of-sonic-devices/ (at 196:3) | Request timed out
[TIMEOUT] https://stordis.com/bgp-unnumbered-in-enterprise-sonic/ (at 66:1) | Request timed out
[TIMEOUT] https://stordis.com/deploying-bgp-underlay-in-enterprise-sonic/ (at 197:3) | Request timed out

Errors in docs/guides/developer-guide/style-guide.md
[ERROR] https://wiki.openstack.org/wiki/GitCommitMessages (at 79:1) | Network unreachable. Check internet connection or VPN settings

Hint: Followed 9 redirects. You might want to consider replacing redirecting URLs with the resolved URLs. Use verbose mode (`-v`/`-vv`) to see redirection details.

See detailed reports in MegaLinter artifacts

Your project could benefit from a custom flavor, which would allow you to run only the linters you need, and thus improve runtime performances. (Skip this info by defining FLAVOR_SUGGESTIONS: false)

  • Documentation: Custom Flavors
  • Command: npx mega-linter-runner@10.2.0 --custom-flavor-setup --custom-flavor-linters ACTION_ACTIONLINT,JSON_JSONLINT,JSON_V8R,JSON_PRETTIER,MARKDOWN_MARKDOWNLINT,MARKDOWN_MARKDOWN_TABLE_FORMATTER,REPOSITORY_CHECKOV,REPOSITORY_GIT_DIFF,REPOSITORY_BETTERLEAKS,REPOSITORY_SECRETLINT,REPOSITORY_TRUFFLEHOG,SPELL_LYCHEE,SPELL_CODESPELL,YAML_PRETTIER,YAML_YAMLLINT,YAML_V8R

MegaLinter is provided by OX Security
Show us your support by starring ⭐ the repository

@ideaship
ideaship marked this pull request as ready for review November 13, 2026 16:15
@ideaship
ideaship marked this pull request as draft November 13, 2026 16:17
@ideaship ideaship self-assigned this Nov 13, 2026
@ideaship
ideaship force-pushed the docs/ceph-tentacle-upgrade branch from ec38160 to abcf0f4 Compare November 13, 2026 16:32
@ideaship ideaship changed the title upgrade guide: Ceph with cephadm and cephx keys upgrade guide: upgrade Ceph managed by cephadm Nov 13, 2026
@ideaship
ideaship marked this pull request as ready for review November 13, 2026 18:40
@ideaship
ideaship requested a review from berendt November 13, 2026 18:40
@ideaship
ideaship marked this pull request as draft November 13, 2026 18:40
@ideaship
ideaship marked this pull request as ready for review November 13, 2026 18:41
@ideaship
ideaship marked this pull request as draft November 16, 2026 07:26
The Ceph upgrade page shows only the ceph-ansible commands
(ceph-rolling_update, cephclient). A cluster managed by cephadm, new
with OSISM 11 or migrated with the migration guide, is upgraded by
cephadm itself with ceph orch upgrade, and nothing described how.

Add that part, Reef to Tentacle with OSISM 11:

- before starting: HEALTH_OK, OSISM 11 on the manager (with a check),
  and after a migration its Step 7 done, with a command that lists the
  legacy cephx entities it removes, since Tentacle reports them as
  HEALTH_ERR;
- add ceph_version: tentacle (a repository upgraded from OSISM 10
  usually has none) and drop rgw keystone api version from
  ceph_conf_overrides in the same commit, per the OSISM 11 notes;
- find the target image in the release (ceph_image_versions in the
  osism-ansible container), with the registry the cluster uses; the
  config store may show the current image by digest;
- check, start and follow the upgrade, and what it looks like. What a
  finished upgrade prints depends on the release: only a Tentacle
  manager prints "There are no upgrades in progress currently.", a
  Reef or Squid manager prints the status as JSON with "in_progress":
  false (orchestrator module, v18.2.7 and v19.2.3 against v20.2.0),
  which matters for an update within a release. A follow-up that was
  interrupted continues with ceph orch upgrade status, since the
  HEALTH_OK precondition does not hold during and right after the
  upgrade; a paused upgrade names its cause as an UPGRADE_* health
  check and continues with ceph orch upgrade resume, which clears it;
- remove rgw_keystone_api_version from the config store only once RGW
  runs Tentacle: on Reef its default 2 breaks Keystone authentication
  at once. The RGW daemons are then redeployed one at a time, each
  until ceph orch ps shows it running in a new container, which is the
  only reliable sign since ceph orch ps keeps listing the old container
  as running until it is replaced: ceph orch restart restarted all
  three within 20 seconds, and ceph orch daemon restart refuses an RGW
  daemon without --force. A Swift probe every 2 seconds saw no
  failure; each daemon took up to 80 seconds. The page gives the two
  commands per daemon rather than a script. If the step is interrupted
  after ceph config rm, the cluster no longer sets the option, so the
  page says to finish the step anyway: the daemons still in the
  container the first listing showed are left, and without that
  listing all of them are redeployed again, one at a time. ceph config
  rm of an option that is not set succeeds (ConfigMonitor.cc), so the
  step can be resumed;
- update cephclient;
- say that the cluster stays HEALTH_WARN for the remaining keys of the
  old type and works with that, and warn against rotating the keys of
  the OpenStack services without distributing them, since each service
  loses access to Ceph until its new key is in the configuration
  repository and applied. Rotating the remaining keys is left to a
  separate change.

The page ends by pointing to binding RGW daemons that a migration
deployed without --networks to their nodes' own addresses, as the
migration guide describes: the Tentacle image of OSISM 11 carries
OSISM's cephadm address-discovery fix (osism/container-images#999, in
every v20 build), and with it network-based binding worked on the
routed topology of a test cluster.

Measured in four runs, Reef 18.2.8 to Tentacle 20.2.4 directly,
which Tentacle supports: 33 to 40 minutes for 21 daemons plus crash;
cephadm mutes the AUTH_INSECURE_* checks; it upgrades the OSDs of a
host together (up to 88 PGs peering, a few inactive for up to a
minute); require_osd_release < tentacle shows for a moment; after the
version upgrade it rotates the keys of the mon, mgr, OSD and MDS
daemons and redeploys them, so the progress counter runs down again
(27/27 to 19/27), ceph orch upgrade status keeps saying "upgrading rgw
daemons", and the OSDs restart again two at a time (2 down, one or
two PGs inactive for seconds); the cluster log shows MDS_ALL_DOWN as
an error for some tens of milliseconds whenever the MDS daemons are
upgraded or redeployed. When the upgrade finishes, cephadm sets
auth_service_cipher and lifts its mutes in the same pass
(cephadm/upgrade.py), so AUTH_INSECURE_SERVICE_TICKETS, which
AuthMonitor.cc raises as HEALTH_ERR, shows for a few seconds until the
monitors have applied the change; the page says so, since an operator
who stops on HEALTH_ERR would stop there. Afterwards HEALTH_WARN from
the remaining client keys only. The upgrade command uses CEPH_IMAGE
from step 2, so the page says to run it in the same shell.
ceph_version in environments/configuration.yml takes precedence over
the reef that an upgraded manager's ceph-ansible inventory still
carries, so osism apply cephclient deploys the Tentacle client. The
text was also followed blind by an operator with only the page, three
times.

Rechecked against OSISM 11.0.3, which pins ceph-daemon:20.2.5-r2 and
cephclient 20.2.5: the key rotation in cephadm/upgrade.py is unchanged
from v20.2.4 apart from a guard for an empty mon cache, so the
observations above still describe the upgrade. The example image is
that of 11.0.3; the expected versions refer to the target image
instead of naming a release, since step 2 reads the image from the
release the manager runs.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
ideaship force-pushed the docs/ceph-tentacle-upgrade branch from abcf0f4 to b6bc8bb Compare November 16, 2026 07:46
@ideaship
ideaship marked this pull request as ready for review November 16, 2026 07:54

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New

Development

Successfully merging this pull request may close these issues.

1 participant