2024-11-11T03:00:09.576Z

<Yingxin Cheng> Thanks for clarification!

For the enable_ht_threads case, it might worth taking the relationship of vCPUs and pCPUs into consideration. For example in your right graph (vCPUs) 0-29 OSDs, 30-53 BlueStores:
1. To make use of hyper-threading, BlueStore threads need to run on the siblings of the assigned pCPUs of vCPUs 30-53, meaning vCPUs 86-109 as well.
2. For seastar threads running on vCPUs 0-29, their sibling vCPUs 56-85 should not be shared for other purposes.
3. This implies that client threads (if locally deployed) should be limited to the rest of 2 pCPUs, meaning vCPUS 54-55, 110-111, which looks insufficient.

This is why I meant above^ that "starting with disabling hyperthreading might be simpler". The relationship between vCPUs and pCPUs can be found by checking "/proc/cpuinfo".

2024-11-11T10:36:17.073Z

<Jose J Palacios-Perez> Hi @Yingxin Cheng I think I see your point, let me clarify:
1. as part of the configuration we specify the number of alien threads and number of CPU cores for alien threads, and use these as parameters to `vstart.sh` that's why the allocation of CPUs for  Alien and Blustore threads only uses from CPU 30 to 53 (which are phys). I am not aware that vstart.sh will differentiate between types of CPU cores (whether they are physical or HT siblings) .
2. I agree, we can see in the grid that is precisely the case.
3. I was afraid that was the case, since in previous experiments, 8 CPU cores for the FIO client were enough, which means that unfortunately shares CPUs with the Alien/Bluestore threads.
4. Agree, the HT disabled scenario is simpler,  I'll start with that 👍  

2024-11-11T10:37:30.847Z

<Jose J Palacios-Perez> Snippet of the configuration parameters to vstart.sh:: https://files.slack.com/files-pri/T1HG3J90S-F080D7FANKE/download/screenshot_2024-11-11_at_10.27.07.png

2024-11-11T11:38:58.296Z

<Jose J Palacios-Perez> Sadly, I now realise the grid is not quite correct:  CPUs from 0-27 are on socket 0, then CPUs from 28-55 on socket 1. In other words, the first two rows are phys for socket 0, next 2 rows are phys for socket 1, next 2 rows HT siblings in socket 0, last two rows are HT siblings for socket 1, will need to double check from /proc/cpuinfo 😢

2024-11-11T11:41:46.358Z

<Jose J Palacios-Perez> https://files.slack.com/files-pri/T1HG3J90S-F080ASK84TD/download/screenshot_2024-11-11_at_11.41.27.png

2024-11-11T11:52:01.825Z

<Jose J Palacios-Perez> Hi @Yingxin Cheng I think I see your point, let me clarify:
1. as part of the configuration we specify the number of alien threads and number of CPU cores for alien threads, and use these as parameters to `vstart.sh` that's why the allocation of CPUs for  Alien and Blustore threads only uses from CPU 30 to 53 (which are phys). I am not aware that vstart.sh will differentiate between types of CPU cores (whether they are physical or HT siblings) . We started by reserving 8 CPU cores for the FIO client, so the figures in the table at the top of this conversation consider that caveat.
2. I agree, we can see in the grid that is precisely the case.
3. I was afraid that was the case, since in previous experiments, 8 CPU cores for the FIO client were enough, which means that unfortunately shares CPUs with the Alien/Bluestore threads.
4. Agree, the HT disabled scenario is simpler,  I'll start with that 👍  

2024-11-11T13:16:33.973Z

<Jose J Palacios-Perez> Fixed: might remove the header and add a column to indicate the socket number instead:: https://files.slack.com/files-pri/T1HG3J90S-F080BAWE5TL/download/crimson_3osd-10reactor_enable_ht.png

2024-11-11T13:45:57.236Z

<Jose J Palacios-Perez> There is also probably a bug in the expression to calculate the number of CPU cores for Alien/BS threads: we expect 22 CPU cores for the enable_ht (max_num_cpu is 52, minus 3 (num OSD )*10 (num_reactors) gives 52-30 = 22, but instead we see 24 CPU cores assigned by vstart to Alien/BS 🤔

2024-11-11T14:57:07.997Z

<Jose J Palacios-Perez> I guess another question would be to have the Seastar reactor threads balanced across sockets, since vstart allocates contiguously CPU uids, so we end up with most/all of the reactors on socket 0, rather than say, half and half (which might improve performance)