ceph - cephadm - 2024-11-12

Timestamp (UTC)Message
2024-11-12T15:23:09.353Z
<verdurin> We replaced an OSD HDD. However, the associated metadata device on NVMe was not zapped at the time. It has been removed now, with:

`ceph-volume lvm zap --osd-id 94`

However, both NVMes are shown as not available, so the new disk is not being provisioned by our OSD spec.
How do I persuade Cephadm that there is space to create a new WAL/DB LV, and add the new disk?
2024-11-12T15:31:12.984Z
<Ken Carlile> check the /var/lib/ceph/<clusteruuid>/removed directory on the node holding the defunct OSD--the block and the block.db symlinks should both be bad. If not, then you will need to do an lvremove on the extant symlink to clear the drive. This solution is presented without warranty or guarantee, be careful, use your best judgement, yadda yadda: <https://pastebin.com/gdtA6LTS>
2024-11-12T15:31:33.115Z
<Ken Carlile> there are even nastier ways to go (like using wipefs), but those are... reaaaaaally nasty.
2024-11-12T16:28:33.007Z
<verdurin> Thanks. Looks a bit different in our case - the symlinks are to device-mapper devices, not LVs.
2024-11-12T16:29:30.534Z
<verdurin> Thanks. Looks a bit different in our case - the symlinks are to device-mapper devices, not LVs.

The `block` one resolves, the `block.db` one does not resolve.
2024-11-12T17:12:31.784Z
<verdurin> Actually, there was an LV showing in the `lvs` output that didn't have an associated `dm-crypt` device, which was a bit of a giveaway.

Any issue? please create an issue here and use the infra label.