2024-07-10T14:27:03.997Z | <Raghu> Hello
Good morning, we upgraded our cluster using ceph orch upgrade, If we run in to any issues, is there a way to downgrade the cluster ?
I dont see any documentation online that would provide this information.
Can some one please let us know if it is possible or not ? if yes, can some one share documentation on how this can be done ? |
2024-07-10T14:27:29.462Z | <Raghu> Hello
Good morning, we are planning to upgrade our cluster using ceph orch upgrade, If we run in to any issues, is there a way to downgrade the cluster ?
I dont see any documentation online that would provide this information.
Can some one please let us know if it is possible or not ? if yes, can some one share documentation on how this can be done ? |
2024-07-10T14:43:46.208Z | <Eugen Block> A downgrade is not supported. I would recommend to do a staggered upgrade to minimize the risk. You start with MGR, see if that works. Then continue with MONs, if it fails at that point, you can cancel the upgrade and redeploy the failed MON in the worst case with the same version as the remaining MONs. If the MONs upgrade successfully, go ahead with OSDs, maybe try one host first, I think you can specify that with staggered upgrade. If the host upgrades successfully, you can do the rest. If it doesn’t, worst case is to redeploy those OSDs. So there’s no real fallback but a couple of safety checks. |
2024-07-10T15:03:38.917Z | <Raghu> thank you |
2024-07-10T15:44:57.811Z | <Joel Davidow> Note that staggered upgrade via `ceph orch upgrade` was introduced in 16.2.11 so if cluster is below that, staggered isn’t readily available - see [this procedure](https://docs.ceph.com/en/pacific/cephadm/upgrade/#upgrading-to-a-version-that-supports-staggered-upgrade-from-one-that-doesn-t) as a possible workaround. Also check out `ceph orch upgrade pause`, `ceph orch upgrade resume`, and `ceph orch upgrade stop` commands, which provide some ability to control the upgrade progression. I recommend running at least `ceph -W cephadm`, `watch ceph -s` , and `watch ceph versions`during the upgrade to keep an eye on the upgrade. |
2024-07-10T16:16:14.061Z | <Raghu> Thank you |