2024-11-07T14:08:14.592Z

<Igor Golikov> Hi I am getting `Link Error: RADOS library not found`  when building last main. Did anyone see it as well?

2024-11-07T17:08:18.494Z

<Ivveh> hey guys, i have a little wierd cephfs issue where i have a client causing slow requests for the mds, the mds points to osd_slow_ops and the counter is growing. the metadata pool is on nvme and decently fast. i have checked all involved osds and none of them have slow ops. so i turn to look at dump_ops_in_flight for the mds where i can find the slow ops, all of them start with `initiated` and then `throttled` and then some other stuff and they get `dispatched` , any hints?

2024-11-07T17:09:50.079Z

<Ivveh> i tried restarting the mds to no avail

2024-11-07T17:11:24.339Z

<Ivveh> so  its `initiated` `throttled` `header_read` `all_read` and then `dispatched` with locks being null

2024-11-07T17:15:57.766Z

<gregsfortytwo> Check the MDs’ objecter ops in flight and see what it’s got happening 

2024-11-07T17:16:14.920Z

<gregsfortytwo> OSDs only report slow ops at 30 seconds by default, which is extremely slow ;)

2024-11-07T17:16:24.348Z

<gregsfortytwo> Check the MDS’ objecter ops in flight and see what it’s got happening 

2024-11-07T17:16:26.914Z

<Ivveh> debug_objecter for mds?

2024-11-07T17:16:59.263Z

<Ivveh> yeah the mds too, stuff at 30sec+

2024-11-07T17:19:33.171Z

<Ivveh> yeah objecter logs dont say much except how many are in flight

2024-11-07T17:19:40.882Z

<Ivveh> at 5/5 anyways

2024-11-07T17:20:57.162Z

<Ivveh> anything in particular i'm looking for?

2024-11-07T17:24:49.395Z

<gregsfortytwo> Just how long they’re taking

2024-11-07T17:25:05.154Z

<Ivveh> 25-30ish seconds

2024-11-07T17:25:30.910Z

<Ivveh> ```   "age": 33.165397732999999,
            "duration": 33.165422638999999,```

2024-11-07T17:25:34.882Z

<gregsfortytwo> Well that’s a really long freaking time for OSD ops. You’ll have to dig in to the OSD performance

2024-11-07T17:26:18.198Z

<Ivveh> data osds or mds osds? mds seems ok, they report no slow ops anyways

2024-11-07T17:29:04.376Z

<Ivveh> i think its wierd i have no osd slow ops, but the mds is reporting it but not a single osd is "slow" ill check the data ones now

2024-11-07T18:49:37.146Z

<Patrick Donnelly> when building or starting a vstart cluster?

2024-11-07T19:28:56.997Z

<Igor Golikov> building, and for some reason i was missing the librados... Not sure how it worked for me before, but after running `sudo apt install librados`  it worked