2024-09-16T07:00:15.044Z | <Kotresh H R> @Venky Shankar @Xiubo Li PTAL at <https://tracker.ceph.com/issues/62951#note-14> |
2024-09-16T09:34:04.737Z | <Dhairya Parmar> @Venky Shankar found the root cause of this |
2024-09-16T09:40:22.128Z | <Dhairya Parmar> and my suspicion was correct for the buffer caps <https://ceph-storage.slack.com/archives/C04LVQMHM9B/p1726049636860949?thread_ts=1725969204.614829&cid=C04LVQMHM9B> |
2024-09-16T13:31:19.848Z | <Igor Golikov> priveleged worked ,thanks. now the last thing I am missing - does the teutology run archive include binaries that produced the core dump? |
2024-09-16T14:31:04.165Z | <Dhairya Parmar> what does `/@59` mean in `caps={4286=pAsLsXs/-@59}` ? |
2024-09-16T14:31:17.166Z | <Dhairya Parmar> what does `/-@59` mean in `caps={4286=pAsLsXs/-@59}` ? |
2024-09-16T15:53:36.880Z | <Dhairya Parmar> @Venky Shankar (re: <https://tracker.ceph.com/issues/66581>)
the mds doesn't seem to be sending revoke request for `Fb` :
```2024-09-16T19:47:43.770+0530 7f99556006c0 7 mds.0.locker sending MClientCaps to client.4286 seq 2 new pending pAsLsXsFcb was pAsxLsXsxFsxcrwb
2024-09-16T19:47:43.770+0530 7f99556006c0 20 mds.0.cache.ino(0x10000000000) encode_cap_message pfile 1 pauth 0 plink 0 pxattr 0 mtime 2024-09-16T19:47:41.877357+0530 ctime 2024-09-16T19:47:41.877357+0530 change_attr 3
2024-09-16T19:47:43.770+0530 7f99556006c0 10 mds.0.4 send_message_client_counted client.4286 seq 2 client_caps(revoke ino 0x10000000000 1 seq 2 caps=pAsLsXsFcb dirty=- wanted=pAsxXsxFxwb follows 0 size 123/4194304 ts 1/18446744073709551615 mtime 2024-09-16T19:47:41.877357+0530 ctime 2024-09-16T19:47:41.877357+0530 change_attr 3)
2024-09-16T19:47:43.770+0530 7f99556006c0 1 -- [v2:127.0.0.1:6830/2427047944,v1:127.0.0.1:6831/2427047944] --> 127.0.0.1:0/3870042328 -- client_caps(revoke ino 0x10000000000 1 seq 2 caps=pAsLsXsFcb dirty=- wanted=pAsxXsxFxwb follows 0 size 123/4194304 ts 1/18446744073709551615 mtime 2024-09-16T19:47:41.877357+0530 ctime 2024-09-16T19:47:41.877357+0530 change_attr 3) -- 0x557887d50e00 con 0x557887d65f80```
but the client seems to be dropping it:
```2024-09-16T19:47:43.771+0530 7fe46ea006c0 10 client.4286 send_cap 0x10000000000.head(faked_ino=0 nref=4 ll_ref=6 cap_refs={4=0,1024=0,4096=0,8192=0} open={2=0} mode=100644 size=164/4194304 nlink=1 btime=2024-09-16T19:47:39.825088+0530 mtime=2024-09-16T19:47:42.880406+0530 ctime=2024-09-16T19:47:42.880406+0530 change_attr=4 caps=pAsLsXsFcb(0=pAsLsXsFcb) flushing_caps=Fw objectset[0x10000000000 ts 0/0 objects 1 dirty_or_tx 0] parents=0x1.head["foo"] 0x7fe42c0072c0) mds.0 seq 2 used Fc want - flush Fw retain pAsLsXsFc held pAsxLsXsxFsxcrwb revoking AxXxFsxrw dropping Fb```
and the change is visible soon after it:
```2024-09-16T19:47:43.771+0530 7fe46ea006c0 10 client.4286 handle_cap_grant calling signal_caps_inode
2024-09-16T19:47:43.811+0530 7fe46ea006c0 10 client.4286 mds.0 seq now 3
2024-09-16T19:47:43.811+0530 7fe46ea006c0 5 client.4286 handle_cap_flush_ack mds.0 cleaned - on 0x10000000000.head(faked_ino=0 nref=4 ll_ref=6 cap_refs={4=0,1024=0,4096=0,8192=0} open={2=0} mode=100644 size=164/4194304 nlink=1 btime=2024-09-16T19:47:39.825088+0530 mtime=2024-09-16T19:47:42.880406+0530 ctime=2024-09-16T19:47:42.880406+0530 change_attr=4 caps=pAsLsXsFc(0=pAsLsXsFc) flushing_caps=Fw objectset[0x10000000000 ts 0/0 objects 1 dirty_or_tx 0] parents=0x1.head["foo"] 0x7fe42c0072c0) with Fw```
it looks like the write call is blocking the client from acquiring read caps, final caps - `pAsLsXsFc` . |
2024-09-16T16:37:18.192Z | <gregsfortytwo> Oh hmm if the client still has Fb then nobody else can get Fr @Dhairya Parmar, so that may also prevent the test case you and I discussed last week. Normally Fb would just get revoked but quiesce may prevent that |
2024-09-16T17:43:59.440Z | <Dhairya Parmar> if you see the the client is dropping it `retain pAsLsXsFc held pAsxLsXsxFsxcrwb revoking AxXxFsxrw dropping Fb` |
2024-09-16T17:44:36.690Z | <Dhairya Parmar> which is visible in the next log line `caps=pAsLsXsFc(0=pAsLsXsFc)` |
2024-09-16T17:45:55.745Z | <Dhairya Parmar> but if we say that quiesce is preventing it then isn't this a blocker/limitation? |
2024-09-16T17:46:09.609Z | <Dhairya Parmar> Theoretically this should go through |
2024-09-16T17:46:40.869Z | <Dhairya Parmar> Another thing that bugs me is how the cap management is different when running single-client/double-client setup |
2024-09-16T17:52:27.725Z | <Dhairya Parmar> a double-client setup does this `Fc want pAsxXsxFxwb flush - retain pAsxLsxXsxFsxcrwl held pAsLsXsFcb revoking Fb dropping -` while the single client setup drops `Fb` instead of revoking it `flush Fw retain pAsLsXsFc held pAsxLsXsxFsxcrwb revoking AxXxFsxrw dropping Fb` |
2024-09-16T17:52:39.375Z | <Dhairya Parmar> im still trying to figure out why this difference |
2024-09-16T18:10:19.511Z | <Dhairya Parmar> what i mean to ask is if mds can serve the getattr req and provide the required caps to client.b then what is stopping the the mds to do the same to client.a (single client setup), only probable culprit what i think is the `Fwb` caps |
2024-09-16T18:10:40.562Z | <Dhairya Parmar> what i mean to ask is if mds can serve the getattr req and provide the required caps to client.b then what is stopping the the mds to do the same to client.a (single client setup), only probable culprit what i think is the `Fwb` caps that are waiting |
2024-09-16T18:55:19.951Z | <gregsfortytwo> The MDs can’t satisfy certain getattr lookups (like size) while a client holds Fb, because it doesn’t know how much has been appended in the buffer. So when a client holding Fb needs to query the MDS for something requiring the client don’t hold Fb cap, it proactively drops it. This is a common pattern |
2024-09-16T18:56:07.933Z | <gregsfortytwo> When the request-that-requires-no-Fb comes from a different client, the MDS must revoke the Fb cap from whoever holds it prior to answering the request |
2024-09-16T18:56:32.712Z | <gregsfortytwo> The MDS can’t satisfy certain getattr lookups (like size) while a client holds Fb, because it doesn’t know how much has been appended in the buffer. So when a client holding Fb needs to query the MDS for something requiring the client don’t hold Fb cap, it proactively drops it. This is a common pattern |
2024-09-16T18:57:31.236Z | <gregsfortytwo> Since the caps are how we coordinate consistency, they will of course be different when we have multiple actors instead of a single actor |