2024-09-13T09:51:02.700Z | <verdurin> Any idea how to fix the `STRAY_HOST` warning, now that the CRUSH part is fixed?
Or will it go away in time? |
2024-09-13T09:56:13.657Z | <Eugen Block> usually, those warnings go away after 10 or 15 minutes, not entirely sure which interval it is, after cephadm refreshes its invenory. |
2024-09-13T09:56:38.830Z | <verdurin> It's been several days... |
2024-09-13T09:56:39.596Z | <Eugen Block> you can trigger a refresh by failing the mgr |
2024-09-13T09:56:51.678Z | <Eugen Block> oh, okay, then it's still something else |
2024-09-13T09:57:18.855Z | <Eugen Block> can you share ceph health detail? |
2024-09-13T09:58:29.893Z | <verdurin> ```[WRN] CEPHADM_STRAY_HOST: 1 stray host(s) with 18 daemon(s) not managed by cephadm
stray host cepho-p008-ceph: ['osd.144', 'osd.145', 'osd.146', 'osd.147', 'osd.148', 'osd.149', 'osd.150', 'osd.151', 'osd.152', 'osd.153', 'osd.154', 'osd.155', 'osd.156', 'osd.157', 'osd.158', 'osd.159', 'osd.160', 'osd.161']``` |
2024-09-13T09:58:41.165Z | <verdurin> That's the incorrect hostname. |
2024-09-13T09:58:51.117Z | <verdurin> Maybe I need to restart the other containers on the node? |
2024-09-13T09:59:29.728Z | <verdurin> No, they've already been restarted. |
2024-09-13T10:00:37.844Z | <Eugen Block> So the correct hostname is in the ceph osd tree and in the orch host list, right? |
2024-09-13T10:01:09.357Z | <verdurin> Yes.
I haven't deleted the CRUSH bucket with the incorrect name yet - could that be it? |
2024-09-13T10:01:47.955Z | <verdurin> In CRUSH though it doesn't have any OSDs now. |
2024-09-13T10:04:30.454Z | <Eugen Block> no, I don't think deleting the crush bucket would fix it. I believe it's the config-key. Can you check `ceph config-key ls | grep cepho-p008`? There's probably only one entry for the host itself and one for the devices on it, correct? |
2024-09-13T10:05:38.125Z | <verdurin> ``` "config-history/116/-client.crash.cepho-p008/container_image",
"config-history/1160/+client.crash.cepho-p008/container_image",
"config-history/1181/-client.crash.cepho-p008/container_image",
"config-history/1725/+client.crash.cepho-p008/container_image",
"config-history/1747/-client.crash.cepho-p008/container_image",
"config-history/2281/+client.crash.cepho-p008/container_image",
"config-history/2301/-client.crash.cepho-p008/container_image",
"config-history/629/+client.crash.cepho-p008/container_image",
"config-history/650/-client.crash.cepho-p008/container_image",
"config-history/95/+client.crash.cepho-p008/container_image",
"mgr/cephadm/host.cepho-p008.<fqdn>",``` |
2024-09-13T10:06:29.786Z | <Eugen Block> I'm not sure if there's a way to trigger a refresh, maybe @Adam King can comment? I'll see if it could be resolved by removing the host config-key, maybe the mgr rebuilds it, I don't know yet. I have a couple of test clusters, I'll play around a bit |
2024-09-13T10:10:11.333Z | <Eugen Block> I just removed a host key entry "mgr/cephadm/host.<host>" as well as the corresponding devices key ( "mgr/cephadm/host.<host>.devices.0"), it was quickly rebuilt. |
2024-09-13T10:10:59.860Z | <Eugen Block> I believe it's triggered by the periodic `cephadm gather-facts`. |
2024-09-13T10:12:40.003Z | <verdurin> No sign of those device keys on any of the OSD nodes in this cluster.
Is that a problem? |
2024-09-13T10:23:19.325Z | <Eugen Block> I don’t think it’s a problem, which ceph version is this again? |
2024-09-13T10:25:27.701Z | <verdurin> `16.2.15` |
2024-09-13T10:31:10.740Z | <Eugen Block> okay, pacfic doesn't seem to have the devices key, I was looking at a newer cluster. |
2024-09-13T10:33:14.242Z | <Eugen Block> can you run `cephadm gather-facts | grep cepho-p008` locally on that host? Does that show the correct hostnames as well? |
2024-09-13T10:35:53.121Z | <Eugen Block> I think it's safe to remove the key and let gather-facts rebuild it. just did it on a pacific host as well. Maybe save the previous output to a flie, just in case. |
2024-09-13T10:36:58.161Z | <verdurin> Maybe your earlier suggestion of a reboot first - just noticed that the shell prompt still shows the incorrect hostname, even though `hostnamectl` shows the correct one. |
2024-09-13T10:37:23.563Z | <Eugen Block> oh, I thought you did reboot the node already |
2024-09-13T10:59:24.635Z | <verdurin> Reboot completed, and run that command now after installing the keyring.
It does show the correct hostnames. |
2024-09-13T11:06:12.205Z | <Eugen Block> so the warning is gone? |
2024-09-13T11:06:34.215Z | <verdurin> No, still there, so I'll try the config key removal next. |
2024-09-13T11:24:17.256Z | <verdurin> Deleted the key, and it's been rebuilt.
Warning still present, I'll give it a while to catch up. |
2024-09-13T11:25:34.211Z | <Eugen Block> but is the key correct now? |
2024-09-13T11:26:09.801Z | <verdurin> It's the same - it had the correct hostname before. |
2024-09-13T11:34:47.419Z | <Eugen Block> it's getting confusing 😄 do you by any chance have directories underneath `/var/lib/ceph/osd/ceph-{OSD_ID}`? So not underneath the {FSID} directory? |
2024-09-13T11:40:14.506Z | <verdurin> No, it only has `/var/lib/ceph/{OSD_ID}` and its subdirectories. |
2024-09-13T11:58:19.646Z | <Benard> What do people use to manage the `ceph.conf` configuration when using cephadm? I can see that cephadm can use `yaml` files to save certain configurations for deploying daemons and I was wondering if there is something similar for the actual internal ceph options |
2024-09-13T12:07:20.224Z | <Eugen Block> So the last resort would be to silence the warning, there's a cephadm config for that:
```pacific:~ # ceph config get mgr mgr/cephadm/warn_on_stray_hosts
true```
But it certainly doesn't fix anything. I'm running out of ideas at the moment and I'm heading into the weekend now. You could silence it until you have other approaches to test. |
2024-09-13T12:08:32.949Z | <verdurin> Yes, of course - many thanks again for puzzling through this.
Will certainly offer you a free beverage if I see you at Cephalocon... |
2024-09-13T13:19:05.003Z | <John Mulligan> Check out this link: <https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/#commands>
(the whole page is relevant to your question too, IMO) |
2024-09-13T13:20:05.358Z | <John Mulligan> In short, with cephadm you want to mainly use the centralized configuration db in the mons rather than files. |
2024-09-13T15:41:16.899Z | <Ken Carlile> woo! it worked! |
2024-09-13T16:22:02.150Z | <Benard> Thanks John. It indeed looks like you are meant to manage the config via the centralised database and make changes via `ceph config set ...` commands. However in the case that the cluster were destroyed and recreated for example I would have liked to have a centrally and easily viewable config file that can be used to restore all the settings |
2024-09-13T16:23:02.400Z | <Benard> I suppose I can achieve that via regularly using `config dump` and `config assimilate-conf` in case of a restore |
2024-09-13T16:26:01.861Z | <Ken Carlile> Using cephadm install of Reef, I've noticed that the cluster log on the dashboard fills up with DBG notices very quickly (and there's no paging). I've also enabled the daemon logs, and that also shows a ton of DBG messages, primarily from OSDs and MGR. I can't figure out where to decrease the log level on those--any tips? |
2024-09-13T16:49:16.473Z | <Benard> I tried to import an OSD from legacy config to cephadm using `cephadm adopt --style legacy --name osd.0`. However the disk fails to start and when I look in the logs I see:
```debug 2024-09-13T16:36:30.778+0000 7ff0b1e45380 1 bdev(0x5563e27f4400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
debug 2024-09-13T16:36:30.778+0000 7ff0b1e45380 -1 bdev(0x5563e27f4400 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied
debug 2024-09-13T16:36:30.778+0000 7ff0b1e45380 1 bdev(0x5563e27f4400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
debug 2024-09-13T16:36:30.778+0000 7ff0b1e45380 -1 bdev(0x5563e27f4400 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied
debug 2024-09-13T16:36:30.778+0000 7ff0b1e45380 0 osd.0:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cut
off=196)```
The dir is owned y `ceph:ceph` so I am not sure what is wrong with it. Anyone seen this before? |
2024-09-13T16:51:07.317Z | <drakonstein> the dir may be owned by `ceph:ceph`, but what about the block device behind it?
```ls -l $(readlink /var/lib/ceph/osd/ceph-0/block)``` |
2024-09-13T16:56:34.163Z | <Benard> oh, thats owned by a mixture of root:root and pid 167. Thats strange |
2024-09-13T16:56:55.245Z | <drakonstein> the user inside of the container is 167 |
2024-09-13T16:57:41.833Z | <Benard> Oh, well in that case all the osd dirs are owned by that user |
2024-09-13T16:57:43.179Z | <drakonstein> but before converting, that's not really helpful |
2024-09-13T16:58:57.170Z | <Benard> Should I just change those to something else? |
2024-09-13T17:01:43.965Z | <John Mulligan> That sounds reasonable. You could also file a enhancement request to have ceph automate that if you wish. No promises that it would be acted on quickly, but filing things in the tracker helps make them move visible to the team. |
2024-09-13T17:02:51.162Z | <drakonstein> before converting things should be owned by ceph:ceph, after converting they should be owned by 167:167. However the block devices like /dev/sdb aren't what needs to be owned. it's the dm devices like /dev/dm-0 |
2024-09-13T17:06:08.439Z | <drakonstein> pulling up my cluster really quick, the device that matters is dm-3 here. This is a cluster running cephadm. If I were running a legacy style osd it'd be owned by ceph:ceph
```root@ceph4:~# readlink /var/lib/ceph/a2f67bf4-47f9-4e95-9c2a-55e0dbb652a7/osd.0/block
/dev/mapper/ceph--a2f67bf4--47f9--4e95--9c2a--55e0dbb652a7-osd--block--231dd020--546b--490c--a754--b4bbba3494ef
root@ceph4:~# readlink "/dev/mapper/ceph--a2f67bf4--47f9--4e95--9c2a--55e0dbb652a7-osd--block--231dd020--546b--490c--a754--b4bbba3494ef"
../dm-3
root@ceph4:~# ls -l /dev/mapper/ceph--a2f67bf4--47f9--4e95--9c2a--55e0dbb652a7-osd--block--231dd020--546b--490c--a754--b4bbba3494ef
lrwxrwxrwx 1 root root 7 Sep 11 16:35 /dev/mapper/ceph--a2f67bf4--47f9--4e95--9c2a--55e0dbb652a7-osd--block--231dd020--546b--490c--a754--b4bbba3494ef -> ../dm-3
root@ceph4:~# ls -l /dev/dm-3
brw-rw---- 1 167 167 252, 3 Sep 13 13:04 /dev/dm-3``` |
2024-09-13T17:08:23.460Z | <Benard> ```root@strg0-bm:/var/lib/ceph/9f8709d6-61c8-11eb-be48-00163e247938/osd.0# readlink block
/dev/ceph-a5f4a6af-f18c-484d-b992-f1254f55fb66/osd-block-a5f4a6af-f18c-484d-b992-f1254f55fb66
root@strg0-bm:/var/lib/ceph/9f8709d6-61c8-11eb-be48-00163e247938/osd.0# readlink block.db
/dev/ceph-db-81335957-7220-43a9-8bef-c8d9790b6a50/osd-db-a5f4a6af-f18c-484d-b992-f1254f55fb66
root@strg0-bm:/var/lib/ceph/9f8709d6-61c8-11eb-be48-00163e247938/osd.0# readlink /dev/ceph-a5f4a6af-f18c-484d-b992-f1254f55fb66/osd-block-a5f4a6af-f18c-484d-b992-f1254f55fb66
../dm-5
root@strg0-bm:/var/lib/ceph/9f8709d6-61c8-11eb-be48-00163e247938/osd.0# ls -lh /dev/dm-5
brw-rw---- 1 167 167 253, 5 Sep 13 16:36 /dev/dm-5
root@strg0-bm:/var/lib/ceph/9f8709d6-61c8-11eb-be48-00163e247938/osd.0# readlink /dev/ceph-db-81335957-7220-43a9-8bef-c8d9790b6a50/osd-db-a5f4a6af-f18c-484d-b992-f1254f55fb66
../dm-3
root@strg0-bm:/var/lib/ceph/9f8709d6-61c8-11eb-be48-00163e247938/osd.0# ls -lh /dev/dm-3
brw-rw---- 1 167 167 253, 3 Sep 13 16:36 /dev/dm-3``` |
2024-09-13T17:08:52.876Z | <Benard> Seems the same to me. I thought perhaps becouse the WAL is on another device there may be an issue but its also owned by the correct user |
2024-09-13T17:09:09.094Z | <Benard> Could there be a setting in docker that needs to be enabled? |
2024-09-13T17:10:00.240Z | <drakonstein> those are owned perfectly if you already converted the OSD, but this is a legacy OSD, right? |
2024-09-13T17:11:05.735Z | <Benard> It was, but I ran the command to convert it |
2024-09-13T17:11:06.383Z | <drakonstein> cephadm daemons exist in `/var/lib/ceph/{{ fsid }}/osd.#` if you're looking at a daemon in `/var/lib/ceph/osd/ceph-#` then it should not be owned by 167:167, but ceph:ceph |
2024-09-13T17:11:31.475Z | <drakonstein> if you're expecting this to be a cephadm daemon, then your path is off |
2024-09-13T17:12:56.491Z | <drakonstein> your osd service is also different now. instead of `ceph-osd@#` it becomes `ceph-{{ fsid }}@osd.#` |
2024-09-13T17:14:10.396Z | <Benard> So to clarify there was an `osd.0` daemon in `/var/lib/ceph/osd/ceph-0` . I ran the adopt command and it seems to have wiped that dir and created one in `/var/lib/ceph/{{fsid}}/osd.0` . That dir is owned by the correct people from what I have just seen in the above cluster. However when I start the OSD I get that permissions issue |
2024-09-13T17:14:34.745Z | <Benard> So to clarify there was an `osd.0` daemon in `/var/lib/ceph/osd/ceph-0` . I ran the adopt command and it seems to have wiped that dir and created one in `/var/lib/ceph/{{fsid}}/osd.0` . That dir is owned by the correct user from what I have just seen in the above cluster. However when I start the OSD I get that permissions issue |
2024-09-13T17:15:28.870Z | <drakonstein> the output from your log before is showing the path to the wrong location |
2024-09-13T17:16:08.467Z | <drakonstein> after converting, did you delete the old service definition so it doesn't try to start as well? |
2024-09-13T17:16:26.955Z | <drakonstein> I'm not sure exactly what's happening, but your log is showing a daemon trying to run from the old location |
2024-09-13T17:16:56.779Z | <Benard> Hmm I didnt specifically delete it, I assumed that cephadm would take care of that |
2024-09-13T17:17:00.440Z | <Benard> Let me double check |
2024-09-13T17:17:26.108Z | <drakonstein> ```systemctl list-dependencies ceph.target``` |
2024-09-13T17:17:56.847Z | <drakonstein> hopefully your converted daemons show up under `ceph-{{ fsid }}` and not `ceph-osd` |
2024-09-13T17:19:26.272Z | <drakonstein> you can `systemctl disable ceph-osd@#` to try to stop it from starting up as well |
2024-09-13T17:20:18.018Z | <Benard> Hmm let me double check that |
2024-09-13T17:20:55.681Z | <Benard> I have the old ceph osd service still but its inactive and dead. It doesnt look like it started up again |
2024-09-13T17:23:20.181Z | <Benard> I checked the systemd service definition and the docker run command:
`/var/lib/ceph/9f8709d6-61c8-11eb-be48-00163e247938/osd.0:/var/lib/ceph/osd/ceph-0:z`
it looks like its mounting the fsid directory as `/var/lib/ceph/osd/` inside the container. Perhaps that is why the log looks like that |