ceph - cephfs - 2024-07-24

Timestamp (UTC)Message
2024-07-24T05:59:04.512Z
<jcollin> @Rishabh Dave You could checkout that branch from remotes/ci/<branch>, update the sha and then push it again to shaman. When the build passes, re-run the test to reproduce the same failure.
2024-07-24T05:59:51.265Z
<Rishabh Dave> thanks! 🙂
2024-07-24T06:01:25.475Z
<jcollin> @Rishabh Dave You could checkout that branch from `remotes/ci/wip-vshankar-testing-20240705.161036-debug`, update the sha and then push it again to shaman. When the build passes, re-run the test to reproduce the same failure.
2024-07-24T06:02:16.519Z
<jcollin> @Rishabh Dave You could checkout that branch from `remotes/ci/wip-vshankar-testing-20240705.161036-debug`, update the sha and then push it again to shaman. When the build passes, re-run the test to reproduce the same failure, if necessary.
2024-07-24T16:16:29.785Z
<Erich Weiler> Hi All, quick question - we have some OSD servers in our cephfs filesystem, a few servers serve the metadata OSD pool and some other serve the data OSD pool.  Our filesystem is _*very*_ busy.  We are noticing on our data OSD servers that our memory is being used up extremely fast in the `buff/cache` category (as viewed via `free -g`).  This happens in only a couple minutes, then slowly, swap begins to fill up as well.  The OSDs have plenty of RAM for themselves, but the `buff/cache` filling up and causing the servers to swap is problematic.  If I run a script that flushes the caches every 60 seconds, this is mitigated, but that seems kinda hacky.  Does anyone know a way to get the caches to stop filling up on the OSD servers?  I’m running Reef.
2024-07-24T16:18:40.309Z
<Erich Weiler> Hi All, quick question - we have some OSD servers in our cephfs filesystem, a few servers serve the metadata OSD pool and some other serve the data OSD pool.  Our filesystem is _*very*_ busy.  We are noticing on our data OSD servers that our memory is being used up extremely fast in the `buff/cache` category (as viewed via `free -g`).  This happens in only a couple minutes, then slowly, swap begins to fill up as well.  The OSDs have plenty of RAM for themselves, but the `buff/cache` filling up and causing the servers to swap is problematic.  If I run a script that flushes the caches every 60 seconds, this is mitigated, but that seems kinda hacky.  The script kinda looks like this:

```#!/bin/bash

while :
do
    echo `date`
    sync ; sync ; echo 3 > /proc/sys/vm/drop_caches
    sleep 60
done```
Does anyone know a way to get the caches to stop filling up on the OSD servers, without constantly flushing the caches?  I’m running Reef.

Any issue? please create an issue here and use the infra label.