2024-11-21T10:22:30.532Z | <Igor Golikov> Hi folks, i need some help with snapshots. I want to create snapshot, so I enabled it ( I see `flags 12 joinable allow_snaps allow_multimds_snap` in the `ceph fs get a` output). but when I run `ceph fs snapshot create...` it says Invalid command. I have the `.snap` folder in my fs. What do I miss? |
2024-11-21T10:23:24.092Z | <Venky Shankar> Can you share the full command you ran @Igor Golikov? |
2024-11-21T10:23:45.360Z | <Venky Shankar> `.snap` is virtual do its always iteratable even without snapshots. |
2024-11-21T10:27:11.571Z | <Skylar Kelty> Hi all,
I'm following the DR procedure at <https://docs.ceph.com/en/reef/cephfs/disaster-recovery-experts/> to recover a ~2PiB FS.
The problem is, scan_extents doesn't _*seem*_ to be doing anything.
strace is just showing a constant stream of:
```
futex(0x556dd24fa618, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7ffcaa19e278, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = 0
futex(0x7ffcaa19e220, FUTEX_WAKE_PRIVATE, 1) = 0
write(10, "c", 1) = 1
```
No data is being written to the new metadata pool.
Question: do all "scan_extents" workers need to be running on the same node or can you spread them out across multiple nodes as I have? |
2024-11-21T10:28:01.920Z | <Skylar Kelty> I didn't think "snapshot create" was a command, don't you just "mkdir .snap/<snap name>" to create them? |
2024-11-21T10:28:34.167Z | <Skylar Kelty> Hi all,
I'm following the DR procedure at <https://docs.ceph.com/en/reef/cephfs/disaster-recovery-experts/> to recover a ~2PiB FS.
The problem is, scan_extents doesn't _*seem*_ to be doing anything.
strace is just showing a constant stream of:
```futex(0x556dd24fa618, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7ffcaa19e278, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = 0
futex(0x7ffcaa19e220, FUTEX_WAKE_PRIVATE, 1) = 0
write(10, "c", 1) = 1```
No data is being written to the new metadata pool.
Question: do all "scan_extents" workers need to be running on the same node or can you spread them out across multiple nodes as I have? |
2024-11-21T10:28:55.158Z | <Igor Golikov> I tried the latter, but it says "readonly folder" |
2024-11-21T10:29:00.618Z | <Venky Shankar> oh, that's an interface to take snapshots on cephfs subvolumes. |
2024-11-21T10:29:12.679Z | <Igor Golikov> cannot create regular file '.snap/2': Read-only file system |
2024-11-21T10:29:28.627Z | <Igor Golikov> this is what I get when i try to mkdir under .snap |
2024-11-21T10:32:12.331Z | <Igor Golikov> OH i get it |
2024-11-21T10:32:28.535Z | <Igor Golikov> i was trying to mkdir when I am in the .snap folder |
2024-11-21T10:33:52.284Z | <Venky Shankar> ah kk |
2024-11-21T10:34:09.814Z | <Venky Shankar> what was the issue with `fs snapshots create` command? |
2024-11-21T10:34:21.131Z | <Venky Shankar> what was the issue with `fs snapshot create` command? |
2024-11-21T10:34:24.519Z | <Igor Golikov> jsut got "unknown command" |
2024-11-21T10:35:54.681Z | <Igor Golikov> but now I can't delete .snap...
rm -rf .snap/ gives me "rm: cannot remove '.snap/2/2': Read-only file system" |
2024-11-21T10:36:05.400Z | <Igor Golikov> do we have any docs on how to use snaps? |
2024-11-21T10:36:07.588Z | <Venky Shankar> ```./bin/ceph fs subvolume create a sub0
➜ build git:(main) ✗ ./bin/ceph fs subvolume snapshot create a sub0 snap0
➜ build git:(main) ✗ ./bin/ceph fs subvolume snapshot ls a sub0
[
{
"name": "snap0"
}
]``` |
2024-11-21T10:36:38.993Z | <Venky Shankar> > do we have any docs on how to use snaps?
There is a dev guide. @gregsfortytwo recently created a tracker to have admin docs for snapshots (which was lacking!). |
2024-11-21T10:36:52.413Z | <Skylar Kelty> I think you target the snap "rm .snap/<snap name>" |
2024-11-21T10:37:18.755Z | <Venky Shankar> @Igor Golikov <https://tracker.ceph.com/issues/68974> (nice timing btw 🙂) |
2024-11-21T10:38:47.545Z | <Igor Golikov> @Skylar Kelty I have now .snap/2/2 (the last 2 is a file) and I cant delete neither .snap nor .snap/2 |
2024-11-21T10:39:12.034Z | <Igor Golikov> is aways says `rm: cannot remove '.snap/2/2': Read-only file system` |
2024-11-21T10:39:45.231Z | <Skylar Kelty> try rmdir instead of rm |
2024-11-21T10:40:11.565Z | <Igor Golikov> oh... nice |
2024-11-21T10:40:13.080Z | <Igor Golikov> thanks!! |
2024-11-21T10:40:31.448Z | <Venky Shankar> @Igor Golikov `rmdir .snap/2` |
2024-11-21T10:41:12.519Z | <Igor Golikov> so if i want to create a snapshot now i can just copy stuff under .snap/<snapname> ? |
2024-11-21T10:41:41.724Z | <Venky Shankar> there is no need to copy |
2024-11-21T10:42:13.781Z | <Venky Shankar> once you take a snap under a dir, everything under that dir (snapped) is accessible under the snapshot dir (under .snap/<snapname>). |
2024-11-21T10:42:21.379Z | <Igor Golikov> ohhh |
2024-11-21T10:42:56.757Z | <Igor Golikov> got it, so running the `mkdir .snap/<snapname>` itself creates the snapshot |
2024-11-21T10:43:03.309Z | <Venky Shankar> yeh |
2024-11-21T10:43:06.540Z | <Igor Golikov> got it. |
2024-11-21T10:43:08.580Z | <Igor Golikov> 10 |
2024-11-21T10:43:11.872Z | <Igor Golikov> 10x |
2024-11-21T12:16:46.943Z | <Skylar Kelty> Hi all,
I'm following the DR procedure at <https://docs.ceph.com/en/reef/cephfs/disaster-recovery-experts/> to recover a ~2PiB FS.
The problem is, scan_extents doesn't _*seem*_ to be doing anything.
Question: ~~do all "scan_extents" workers need to be running on the same node or can you spread them out across multiple nodes as I have?~~
Read the [source code](https://github.com/ceph/ceph/blob/main/src/tools/cephfs/DataScan.cc#L655) and checked - they can run on multiple machines |
2024-11-21T12:16:48.607Z | <Skylar Kelty> Hi all,
I'm following the DR procedure at <https://docs.ceph.com/en/reef/cephfs/disaster-recovery-experts/> to recover a ~2PiB FS.
The problem is, scan_extents doesn't _*seem*_ to be doing anything.
Question: ~~do all "scan_extents" workers need to be running on the same node or can you spread them out across multiple nodes as I have?~~
Read the [source code](https://github.com/ceph/ceph/blob/main/src/tools/cephfs/DataScan.cc#L655) and checked - they can run on multiple machines |
2024-11-21T12:16:52.223Z | <Skylar Kelty> Hi all,
I'm following the DR procedure at <https://docs.ceph.com/en/reef/cephfs/disaster-recovery-experts/> to recover a ~2PiB FS.
The problem is, scan_extents doesn't _*seem*_ to be doing anything.
Question: ~~do all "scan_extents" workers need to be running on the same node or can you spread them out across multiple nodes as I have?~~
Read the [source code](https://github.com/ceph/ceph/blob/main/src/tools/cephfs/DataScan.cc#L655) and checked - they can run on multiple machines |
2024-11-21T12:40:41.291Z | <Venky Shankar> `sca_extents` is going to loop objects in tile file system data pool, so, the time taken is going to rely on that many number of objects. |
2024-11-21T12:41:02.313Z | <Venky Shankar> and yes, you can run multiple workers to speed it up 👍 |
2024-11-21T12:43:51.147Z | <Venky Shankar> `scan_extents` is going to loop objects in tile file system data pool, so, the time taken is going to rely on that many number of objects. |
2024-11-21T13:32:54.005Z | <Rishabh Dave> jitsi is too slow for me. i can't join it since last few minutes. anyone else facing same issues? |
2024-11-21T13:33:57.501Z | <jcollin> no, it's good here. |
2024-11-21T13:34:47.424Z | <Rishabh Dave> i can't join, it's stuck in loading mode. ping to jitsi are take >200 ms and to google it takes only few ms. |
2024-11-21T13:37:47.561Z | <Rishabh Dave> finally, was able to join now. |
2024-11-21T14:54:32.399Z | <gregsfortytwo> rmdir .snap/2 |
2024-11-21T14:54:47.815Z | <Igor Golikov> Hi folks, i need some help with snapshots. I want to create snapshot, so I enabled it ( I see `flags 12 joinable allow_snaps allow_multimds_snap` in the `ceph fs get a` output). but when I run `ceph fs snapshot create...` it says Invalid command. I have the `.snap` folder in my fs. What do I miss? |
2024-11-21T16:01:07.404Z | <Hutch> Hi All, im trying to use the cephfs root_squash option on mounting cephfs but it does not seem to be working. the client system can still remove and delete files as the root user |
2024-11-21T16:06:45.321Z | <Hutch> the cluster is running Quincy 17.2.7 and this is my keyring, tried both kernel and Fuse mounts
```client.home
key: XXXXXXXX
caps: [mds] allow rw fsname=cephfs root_squash path=/home
caps: [mgr] allow r
caps: [mon] allow r
caps: [osd] allow *``` |
2024-11-21T16:07:45.406Z | <Hutch> i also tried this keyring
```[client.home]
key = AXXXXX
caps mds = "allow rw fsname=cephfs root_squash, allow rw fsname=cephfs path=/home"
caps mgr = "allow r"
caps mon = "allow r"
caps osd = "allow *" ```
|
2024-11-21T16:09:06.051Z | <Hutch> i also tried this keyring
```client.home
key: XXXXXXXX
caps: [mds] allow rw fsname=cephfs root_squash, allow rw fsname=cephfs path=/home
caps: [mgr] allow r
caps: [mon] allow r
caps: [osd] allow *``` |