2024-08-08T03:28:55.180Z | <Xiubo Li> @Rishabh Dave I also hit this in `quincy`, it seems <https://github.com/ceph/ceph/pull/41779> could resolve this ? |
2024-08-08T03:29:41.179Z | <Xiubo Li> @Rishabh Dave I also hit this in `quincy`, please see <https://pulpito.ceph.com/xiubli-2024-08-08_01:30:52-fs:functional:-wip-xiubli-testing-20240730.051531-quincy-distro-default-smithi/7843082/>.
It seems your <https://github.com/ceph/ceph/pull/41779> could resolve this ? |
2024-08-08T03:29:42.269Z | <Xiubo Li> @Rishabh Dave I also hit this in `quincy`, please see <https://pulpito.ceph.com/xiubli-2024-08-08_01:30:52-fs:functional:-wip-xiubli-testing-20240730.051531-quincy-distro-default-smithi/7843082/>.
It seems your <https://github.com/ceph/ceph/pull/41779> could resolve this ? |
2024-08-08T03:29:46.060Z | <Xiubo Li> @Rishabh Dave I also hit this in `quincy`, please see <https://pulpito.ceph.com/xiubli-2024-08-08_01:30:52-fs:functional:-wip-xiubli-testing-20240730.051531-quincy-distro-default-smithi/7843082/>.
It seems your <https://github.com/ceph/ceph/pull/41779> could resolve this ? |
2024-08-08T03:30:44.307Z | <Xiubo Li> The failure is that the test case set the caps in `cephfs` fs cluster first and then try to set the new caps in `cephfs2` fs cluster, then it failed |
2024-08-08T03:32:20.953Z | <Xiubo Li> @Rishabh Dave I also hit this in `quincy`, please see <https://pulpito.ceph.com/xiubli-2024-08-08_01:30:52-fs:functional:-wip-xiubli-testing-20240730.051531-quincy-distro-default-smithi/7843082/>.
It seems your <https://github.com/ceph/ceph/pull/41779>, which didn't do the backporting yet, could resolve this ? |
2024-08-08T06:15:08.500Z | <Rishabh Dave> this wasn't backported because there was no need for it on downstream's previous versions. we'll have to use `ceph auth add` instead. |
2024-08-08T06:15:26.898Z | <Rishabh Dave> this wasn't backported because there was no need for it in downstream's previous versions. we'll have to use `ceph auth add` instead. |
2024-08-08T07:13:20.055Z | <Xiubo Li> BTW, will this work ? |
2024-08-08T07:13:33.488Z | <Rishabh Dave> `ceph auth add`? |
2024-08-08T07:16:14.958Z | <Xiubo Li> yeah |
2024-08-08T07:18:05.469Z | <Rishabh Dave> it should. IIUC, in that testcase we want multiple caps to be assigned to the client. right? |
2024-08-08T07:18:19.781Z | <Rishabh Dave> same thing can be done by `ceph auth add` command in one go. |
2024-08-08T07:18:27.533Z | <Xiubo Li> yeah, but for different cephfs cluster |
2024-08-08T07:20:59.879Z | <Rishabh Dave> IIRC that PR only makes it possible to add new caps by re-running authorize command. |
2024-08-08T07:21:30.341Z | <Rishabh Dave> the PR that allowed having caps for multiple CephFSs was an older PR. |
2024-08-08T07:21:36.461Z | <Rishabh Dave> so it should be possible. |
2024-08-08T07:33:24.026Z | <Rishabh Dave> if you can tell me the caps you want in your client keyring, i can provide you the `auth add` command that will make it happen. |
2024-08-08T07:50:11.090Z | <Xiubo Li> I tried it and still the same
```Error EINVAL: entity client.uid_1000 exists but caps do not match``` |
2024-08-08T07:51:46.914Z | <Xiubo Li> @Rishabh Dave
It failed in line@996 as bellow:
``` 977 def test_multifs_rootsquash_nofeature(self):
978 """
979 That having root_squash on one fs doesn't prevent access to others.
980 """
981
982 if not isinstance(self.mount_a, FuseMount):
983 self.skipTest("only FUSE client has CEPHFS_FEATURE_MDS_AUTH_CAPS "
984 "needed to enforce root_squash MDS caps")
985
986 self.fs1 = self.fs
987 self.fs2 = self.mds_cluster.newfs('testcephfs2')
988
989 self.mount_a.umount_wait()
990
991 # Authorize client to fs1
992 FS_AUTH_CAPS = ('/', 'rw')
993 self.fs1.authorize(self.client_id, FS_AUTH_CAPS)
994
995 FS_AUTH_CAPS = ('/', 'rw', 'root_squash')
996 keyring = self.fs2.authorize(self.client_id, FS_AUTH_CAPS)
997
998 CEPHFS_FEATURE_MDS_AUTH_CAPS_CHECK = 21
999 # all but CEPHFS_FEATURE_MDS_AUTH_CAPS_CHECK
1000 features = ",".join([str(i) for i in range(CEPHFS_FEATURE_MDS_AUTH_CAPS_CHECK)])
1001 mntargs = [f"--client_debug_inject_features={features}"]
1002
1003 # should succeed
1004 with self.assert_cluster_log("report clients with broken root_squash", present=False):
1005 keyring_path = self.mount_a.client_remote.mktemp(data=keyring)
1006 self.mount_a.remount(client_id=self.client_id, client_keyring_path=keyring_path, mntargs=mntargs, cephfs_name=self.fs1.name)
1007
1008 self.conduct_pos_test_for_read_caps()
1009 self.conduct_pos_test_for_open_caps()
1010 ```
|
2024-08-08T07:52:32.294Z | <Xiubo Li> Locally I just tried the following two command:
```./bin/ceph auth get-or-create client.uid_1000 mon 'allow r' mds 'allow r, allow rw path=/testdir00 uid=1000' osd 'allow rw'
./bin/ceph auth add client.uid_1000 "allow rw fsname=a root_squash"``` |
2024-08-08T07:52:44.474Z | <Xiubo Li> Locally I just tried the following two commands:
```./bin/ceph auth get-or-create client.uid_1000 mon 'allow r' mds 'allow r, allow rw path=/testdir00 uid=1000' osd 'allow rw'
./bin/ceph auth add client.uid_1000 "allow rw fsname=a root_squash"``` |
2024-08-08T07:52:55.094Z | <Rishabh Dave> the test code is calling `fs authorize` 2nd time, which won't work AFAIK |
2024-08-08T07:53:21.551Z | <Xiubo Li> The second time it will use the `fs2` instead. |
2024-08-08T07:53:37.745Z | <Xiubo Li> It's running this in two different cephfs |
2024-08-08T07:54:03.396Z | <Xiubo Li> This works in main branch |
2024-08-08T07:54:08.586Z | <Rishabh Dave> right but running `fs authorize` won't update the caps in quincy |
2024-08-08T07:54:30.085Z | <Rishabh Dave> that is supported on squid onwards. |
2024-08-08T07:55:43.940Z | <Rishabh Dave> Instead of following piece of code -
```
991 # Authorize client to fs1
992 FS_AUTH_CAPS = ('/', 'rw')
993 self.fs1.authorize(self.client_id, FS_AUTH_CAPS)
994
995 FS_AUTH_CAPS = ('/', 'rw', 'root_squash')
996 keyring = self.fs2.authorize(self.client_id, FS_AUTH_CAPS)``` |
2024-08-08T07:55:55.145Z | <Rishabh Dave> you'll need to run auth add command as follows - |
2024-08-08T07:57:41.439Z | <Rishabh Dave> ```moncap = 'allow rw fsname=fs1, allow rw fsname=fs2'
osdcap = 'allow rw tag cephfs data=fs1, allow rw tag cephfs data=fs2'
mdscap = 'allow rw fsname=fs1, allow rw fsname=fs2 root_squash'
self.run_ceph_cmd('auth add mon "{moncap}" osd "{osdcap}" mds "{mdscap}"')``` |
2024-08-08T07:59:44.326Z | <Xiubo Li> Let me try |
2024-08-08T07:59:48.941Z | <Rishabh Dave> i just tried it locally -
```[client.x]
key = AQCserRmmvuwIhAAhg54A4y9rKJbirDRTXbT2g==
caps mds = "allow rw fsname=a, allow rw fsname=b root_squash"
caps mon = "allow r fsname=a, allow r fsname=b"
caps osd = "allow rw tag cephfs data=a, allow rw tag cephfs data=b"``` |
2024-08-08T08:00:07.708Z | <Rishabh Dave> i am edited the comment - <https://ceph-storage.slack.com/archives/C04LVQMHM9B/p1723103861377149?thread_ts=1721731428.347909&cid=C04LVQMHM9B> |
2024-08-08T08:00:19.427Z | <Rishabh Dave> ```moncap = 'allow r fsname=fs1, allow r fsname=fs2'
osdcap = 'allow rw tag cephfs data=fs1, allow rw tag cephfs data=fs2'
mdscap = 'allow rw fsname=fs1, allow rw fsname=fs2 root_squash'
self.run_ceph_cmd('auth add mon "{moncap}" osd "{osdcap}" mds "{mdscap}"')``` |
2024-08-08T08:00:38.114Z | <Rishabh Dave> > i am edited the comment - <https://ceph-storage.slack.com/archives/C04LVQMHM9B/p1723103861377149?thread_ts=1721731428.347909&cid=C04LVQMHM9B>
done, moncap only needs `r`, not `rw`. |
2024-08-08T08:01:22.223Z | <Xiubo Li> Okay |
2024-08-08T08:05:23.252Z | <Xiubo Li> @Rishabh Dave BTW, did you miss the client name in the `auth add` command ? |
2024-08-08T08:06:02.946Z | <Rishabh Dave> umm... |
2024-08-08T08:06:03.661Z | <Rishabh Dave> yes |
2024-08-08T08:06:19.623Z | <Xiubo Li> Okay |
2024-08-08T08:08:19.923Z | <Rishabh Dave> worked for me -
```$ ./bin/ceph auth add client.x mds 'allow rw fsname=a, allow rw fsname=b root_squash' mon 'allow r fsname=a, allow r fsname=b' osd 'allow rw tag cephfs data=a, allow rw tag cephfs data=b'
added key for client.x
$ ./bin/ceph auth get client.x
[client.x]
key = AQC8fLRmDx11GhAAPf9KhmdYVEWOPRU9mcR93g==
caps mds = "allow rw fsname=a, allow rw fsname=b root_squash"
caps mon = "allow r fsname=a, allow r fsname=b"
caps osd = "allow rw tag cephfs data=a, allow rw tag cephfs data=b"``` |
2024-08-08T08:16:12.567Z | <Xiubo Li> I need to get the `keyring` after switching to this |
2024-08-08T08:16:23.829Z | <Xiubo Li> Originally it will get this from `keyring = self.fs2.authorize(self.client_id, FS_AUTH_CAPS)` |
2024-08-08T08:16:37.238Z | <Rishabh Dave> in that case... |
2024-08-08T08:17:24.025Z | <Rishabh Dave> ```self.run_ceph_cmd(f'auth add mon "{moncap}" osd "{osdcap}" mds "{mdscap}"')
keyring = self.get_ceph_cmd_stdout('auth get {client}')``` |
2024-08-08T08:20:13.759Z | <Xiubo Li> Thanks, I am running the test |
2024-08-08T08:20:22.984Z | <Rishabh Dave> awesome. 🙂 |
2024-08-08T08:20:32.932Z | <Rishabh Dave> let me know if doesn't work fine. |
2024-08-08T08:46:53.395Z | <Xiubo Li> sure |
2024-08-08T09:34:27.750Z | <Xiubo Li> @Rishabh Dave
```2024-08-08T09:28:05.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-08-08T09:28:05.306+0000 7f14cee4c700 1 -- 172.21.15.12:0/3031236866 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "auth add", "entity": "testuser", "caps": ["mon", "allow r fsname=cephfs, allow r fsname=testcephfs2", "osd", "allow rw tag cephfs data=cephfs, allow rw tag cephfs data=testcephfs2", "mds", "allow rw fsname=cephfs, allow rw fsname=testcephfs2 root_squash"]} v 0) v1 -- 0x7f14c8155670 con 0x7f14c8155e90
2024-08-08T09:28:05.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-08-08T09:28:05.310+0000 7f14bdffb700 1 -- 172.21.15.12:0/3031236866 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "auth add", "entity": "testuser", "caps": ["mon", "allow r fsname=cephfs, allow r fsname=testcephfs2", "osd", "allow rw tag cephfs data=cephfs, allow rw tag cephfs data=testcephfs2", "mds", "allow rw fsname=cephfs, allow rw fsname=testcephfs2 root_squash"]}]=-22 bad entity name v38) v1 ==== 317+0+0 (secure 0 0 0) 0x7f14c0052c60 con 0x7f14c8155e90
2024-08-08T09:28:05.320 INFO:teuthology.orchestra.run.smithi012.stderr:Error EINVAL: bad entity name
2024-08-08T09:28:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 22``` |
2024-08-08T09:35:22.084Z | <Xiubo Li> I think we should `s/testuser/client.testuser/` , right ? |