2024-07-24T05:59:04.512Z | <jcollin> @Rishabh Dave You could checkout that branch from remotes/ci/<branch>, update the sha and then push it again to shaman. When the build passes, re-run the test to reproduce the same failure. |
2024-07-24T05:59:51.265Z | <Rishabh Dave> thanks! 🙂 |
2024-07-24T06:01:25.475Z | <jcollin> @Rishabh Dave You could checkout that branch from `remotes/ci/wip-vshankar-testing-20240705.161036-debug`, update the sha and then push it again to shaman. When the build passes, re-run the test to reproduce the same failure. |
2024-07-24T06:02:16.519Z | <jcollin> @Rishabh Dave You could checkout that branch from `remotes/ci/wip-vshankar-testing-20240705.161036-debug`, update the sha and then push it again to shaman. When the build passes, re-run the test to reproduce the same failure, if necessary. |
2024-07-24T16:16:29.785Z | <Erich Weiler> Hi All, quick question - we have some OSD servers in our cephfs filesystem, a few servers serve the metadata OSD pool and some other serve the data OSD pool. Our filesystem is _*very*_ busy. We are noticing on our data OSD servers that our memory is being used up extremely fast in the `buff/cache` category (as viewed via `free -g`). This happens in only a couple minutes, then slowly, swap begins to fill up as well. The OSDs have plenty of RAM for themselves, but the `buff/cache` filling up and causing the servers to swap is problematic. If I run a script that flushes the caches every 60 seconds, this is mitigated, but that seems kinda hacky. Does anyone know a way to get the caches to stop filling up on the OSD servers? I’m running Reef. |
2024-07-24T16:18:40.309Z | <Erich Weiler> Hi All, quick question - we have some OSD servers in our cephfs filesystem, a few servers serve the metadata OSD pool and some other serve the data OSD pool. Our filesystem is _*very*_ busy. We are noticing on our data OSD servers that our memory is being used up extremely fast in the `buff/cache` category (as viewed via `free -g`). This happens in only a couple minutes, then slowly, swap begins to fill up as well. The OSDs have plenty of RAM for themselves, but the `buff/cache` filling up and causing the servers to swap is problematic. If I run a script that flushes the caches every 60 seconds, this is mitigated, but that seems kinda hacky. The script kinda looks like this:
```#!/bin/bash
while :
do
echo `date`
sync ; sync ; echo 3 > /proc/sys/vm/drop_caches
sleep 60
done```
Does anyone know a way to get the caches to stop filling up on the OSD servers, without constantly flushing the caches? I’m running Reef. |