2024-11-14T14:47:46.707Z | <Jose J Palacios-Perez> Unfortunately, it seems that ceph.conf forgets the first range, for example:
```/ceph/build/bin/ceph -c /ceph/build/ceph.conf config set osd.2 crimson_seastar_cpu_cores 3-4
/ceph/build/bin/ceph -c /ceph/build/ceph.conf config set osd.2 crimson_seastar_cpu_cores 31-31```
and so we see in the grid this:: https://files.slack.com/files-pri/T1HG3J90S-F080YQ9BB60/download/screenshot_2024-11-14_at_14.47.22.png |
2024-11-14T14:48:09.266Z | <Jose J Palacios-Perez> I'll see if it accepts comma sep values 🤞 |
2024-11-14T15:22:38.119Z | <Jose J Palacios-Perez> It works! this is a nice balanced small config:
```# MDS=0 MON=1 OSD=3 MGR=1 /ceph/src/vstart.sh --new -x --localhost --without-dashboard --cyanstore --redirect-output --crimson --crimson-smp 3 --no-restart --crimson-balance-cpu```: https://files.slack.com/files-pri/T1HG3J90S-F080WFU9V0S/download/screenshot_2024-11-14_at_15.21.24.png |
2024-11-14T15:36:00.005Z | <Matan Breizman> Great progress! Can you help me understand the last graph? No cores for Alien world since it's cyanstore. However, I can't understand whether the R cores are physical or not. |
2024-11-14T15:39:39.164Z | <Jose J Palacios-Perez> Precisely, the red R's stand for Seastar reactors,
``` 4944 pts/0 S 0:00 /bin/sh /ceph/src/ceph-run --no-restart /ceph/build/bin/crimson-osd -i 0 -c /ceph/build/ceph.conf -f
4945 pts/0 Sl 0:15 \_ /ceph/build/bin/crimson-osd -i 0 -c /ceph/build/ceph.conf -f
5024 pts/0 S 0:00 /bin/sh /ceph/src/ceph-run --no-restart /ceph/build/bin/crimson-osd -i 1 -c /ceph/build/ceph.conf -f
5025 pts/0 Sl 0:15 \_ /ceph/build/bin/crimson-osd -i 1 -c /ceph/build/ceph.conf -f
5103 pts/0 S 0:00 /bin/sh /ceph/src/ceph-run --no-restart /ceph/build/bin/crimson-osd -i 2 -c /ceph/build/ceph.conf -f
5106 pts/0 Sl 0:14 \_ /ceph/build/bin/crimson-osd -i 2 -c /ceph/build/ceph.conf -f```
for example, these are the details for osd.2:
```4238 4238 crimson-osd 31 pid 4238's current affinity list: 31
4238 4244 syscall-0 31 pid 4244's current affinity list: 31
4238 4245 crimson-osd 31 pid 4245's current affinity list: 31
5106 5106 crimson-osd 3 pid 5106's current affinity list: 3
5106 5112 reactor-1 4 pid 5112's current affinity list: 4
5106 5113 reactor-2 31 pid 5113's current affinity list: 31
5106 5114 syscall-0 3 pid 5114's current affinity list: 3
5106 5115 syscall-1 4 pid 5115's current affinity list: 4
5106 5116 syscall-2 31 pid 5116's current affinity list: 31
5106 5117 crimson-osd 3 pid 5117's current affinity list: 3
5106 5118 reactor-1 4 pid 5118's current affinity list: 4
5106 5119 reactor-2 31 pid 5119's current affinity list: 31``` |
2024-11-14T15:43:48.230Z | <Jose J Palacios-Perez> • the first two rows (0-27) in the grid correspond to the physical CPU cores for socket 0,
• the next two rows (28-55) are the physical CPU cores for socket 1
• the next two rows are the HT-siblings of the first physical range (0-27), socket 0
• finally, the last two rows are the HT-siblings for physical range (28-55), socket 1. |
2024-11-14T15:45:09.815Z | <Jose J Palacios-Perez> ```# lscpu | grep NUMA | tr -s ' '
NUMA node(s): 2
NUMA node0 CPU(s): 0-27,56-83 <-- physical range, HT-sibling range
NUMA node1 CPU(s): 28-55,84-111```
|
2024-11-14T15:47:35.452Z | <Matan Breizman> Ah, got it now thank you!
Should we mark R cores as R0 for OSD.0 R1 for OSD.1 and so on?
Should we try to separate the table for physical / siblings? |
2024-11-14T15:52:22.419Z | <Jose J Palacios-Perez> > Should we mark R cores as R0 for OSD.0 R1 for OSD.1 and so on?
I was considering precisely that, to include the OSD id, only for those processes since it might get too messy for alien/bluestore threads (and they can be jump around in the range, sort to speak).
> Should we try to separate the table for physical / siblings?
Yes, was also considering that, originally I had them side by side (2 grids, one per socket) but was getting very tricky (I'm still learning Python). I'll separate with a line to indicate physical vs HT-siblings 👍 |
2024-11-14T15:54:38.483Z | <Matan Breizman> This looks really good and could make allocation attempts slightly more friendlier! 🙂 |