2024-06-26T01:34:49.794Z

<Jos Collin> I've already scheduled jobs for the batches. You can tell me which batch to rerun when your fix is merged.

2024-06-26T06:46:17.672Z

<Adam Kraitman> I am seeing very low upload rates to [2.chacra.ceph.com](http://2.chacra.ceph.com) & [3.chacra.ceph.com](http://3.chacra.ceph.com) so I touched /tmp/fail_check so that shaman won't use them for now until I will migrate them to the new OVH instance type like I did with [4.chacra.ceph.com](http://4.chacra.ceph.com), That could be the reason why builds wore slow

2024-06-26T07:00:25.423Z

<Adam Kraitman> The upload speed to [2.chacra.ceph.com](http://2.chacra.ceph.com) was around 190.3KB/s and to [3.chacra.ceph.com](http://3.chacra.ceph.com) was around 157.1KB/s

2024-06-26T07:12:49.478Z

<Dan Mick> sigh.  I love it when our infrastructure just gets yanked out from under us)

2024-06-26T07:13:10.339Z

<Dan Mick> is 1. ok?

2024-06-26T07:23:38.295Z

<Adam Kraitman> I am seeing right now that it's not consistent when I am trying to upload a file from one of the builders(mira013) it looks good, I don't see those low upload rates, I mean we get around 29.0MB/s which is normal

2024-06-26T07:29:23.873Z

<Dan Mick> so it's not consistent, or it's good?

2024-06-26T07:32:03.119Z

<Dan Mick> so is it not consistent, or is it good, which one?

2024-06-26T07:35:56.511Z

<Adam Kraitman> I will need to look at build logs to see if builders get stuck when uploading to chacra, for now I will re-enable 2.chacra & 3.chacra and I will watch the logs

2024-06-26T07:41:23.522Z

<Adam Kraitman> But anyway I think I should migrate the chacra's cause I am seeing consistent rates on [4.chacra.ceph.com](http://4.chacra.ceph.com)(the one I migrated), around 54.9MB/s better then all the other chacra's

2024-06-26T07:41:38.457Z

<Adam Kraitman> But anyway I think I should migrate the chacra's because I am seeing consistent rates on [4.chacra.ceph.com](http://4.chacra.ceph.com)(the one I migrated), around 54.9MB/s better then all the other chacra's

2024-06-26T07:57:28.250Z

<Guillaume Abrioux> [https://docs.ceph.com/en/latest/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-debugging-tips/#debuggi[…]n-error](https://docs.ceph.com/en/latest/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-debugging-tips/#debugging-an-issue-using-interactive-on-error) is this documentation still valid (up to date) ?
```(virtualenv) gabrioux@teuthology:~$ ls /a
ls: cannot access '/a': No such file or directory
(virtualenv) gabrioux@teuthology:~$ ls /home/teuthworker/archive/gabrioux-2024-06-25_12\:15\:30-orch\:cephadm-wip-guits-squid-2024-06-24-0735-squid-distro-default-smithi/7771264/orig.config.yaml
ls: cannot access '/home/teuthworker/archive/gabrioux-2024-06-25_12:15:30-orch:cephadm-wip-guits-squid-2024-06-24-0735-squid-distro-default-smithi/7771264/orig.config.yaml': Permission denied
(virtualenv) gabrioux@teuthology:~$```

2024-06-26T07:58:51.214Z

<Guillaume Abrioux> @Laura Flores @Adam Kraitman @Dan Mick ^

2024-06-26T08:00:34.165Z

<Guillaume Abrioux> ```(virtualenv) gabrioux@teuthology:~$ ls /home/teuthworker/archive
ls: cannot access '/home/teuthworker/archive': Permission denied
(virtualenv) gabrioux@teuthology:~$```

2024-06-26T08:06:18.197Z

<Adam Kraitman> Can you try now? I added your user to the test-admins

2024-06-26T08:07:04.513Z

<Guillaume Abrioux> I suppose it relates to Patrick's email "Restricted access to /teuthology on [teuthology.front.sepia.ceph.com](http://teuthology.front.sepia.ceph.com)"

2024-06-26T08:07:37.951Z

<Guillaume Abrioux> it still doesn't work @Adam Kraitman

2024-06-26T08:07:39.334Z

<Guillaume Abrioux> ```(virtualenv) gabrioux@teuthology:~$ cd /home/teuthworker/archive
-bash: cd: /home/teuthworker/archive: Permission denied```

2024-06-26T08:10:46.943Z

<Guillaume Abrioux> I'm reading this <https://wiki.sepia.ceph.com/doku.php?id=services:cephfs>

2024-06-26T08:10:58.829Z

<Guillaume Abrioux> but I can't ssh [reesi001.front.sepia.ceph.com](http://reesi001.front.sepia.ceph.com)

2024-06-26T08:11:14.524Z

<Adam Kraitman> Yeah that related to that e-mail

2024-06-26T08:11:21.488Z

<Guillaume Abrioux> ```✗ ssh [reesi001.front.sepia.ceph.com](http://reesi001.front.sepia.ceph.com)
Received disconnect from 172.21.2.201 port 22:2: Too many authentication failures
Disconnected from 172.21.2.201 port 22```

2024-06-26T08:11:37.073Z

<Adam Kraitman> Yeah that's related to that e-mail

2024-06-26T08:15:27.290Z

<Guillaume Abrioux> is there anything else I can do for accessing that cephfs locally ?

2024-06-26T08:22:31.832Z

<Adam Kraitman> Please try again
```cd /home/teuthworker/archive ```

2024-06-26T08:23:14.355Z

<Guillaume Abrioux> still permission denied

2024-06-26T08:23:37.902Z

<Adam Kraitman> did you try to logout and login

2024-06-26T08:29:09.518Z

<Guillaume Abrioux> ```(virtualenv) gabrioux@teuthology:~$ logout
Connection to [teuthology.front.sepia.ceph.com](http://teuthology.front.sepia.ceph.com) closed.
➜  ~ ssh [gabrioux@teuthology.front.sepia.ceph.com](mailto:gabrioux@teuthology.front.sepia.ceph.com)
Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-166-generic x86_64)

 * Documentation:  <https://help.ubuntu.com>
 * Management:     <https://landscape.canonical.com>
 * Support:        <https://ubuntu.com/pro>
New release '22.04.3 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Fri Jun 14 08:21:58 2024 from 172.21.0.100
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

gabrioux@teuthology:~$ cd /home/teuthworker/archive
-bash: cd: /home/teuthworker/archive: Permission denied
gabrioux@teuthology:~$```

2024-06-26T08:33:08.735Z

<Adam Kraitman> You can use sudo for viewing those logs

2024-06-26T08:38:01.054Z

<Guillaume Abrioux> I don't want to view the log, just want to `cp` the yml file in order to rerun a job

2024-06-26T08:38:21.362Z

<Guillaume Abrioux> Patrick restricted all of this especially to prevent people viewing log from the machine

2024-06-26T08:39:16.875Z

<Guillaume Abrioux> i'm not going to oppose that 🙂

2024-06-26T08:44:54.148Z

<Guillaume Abrioux> anyway, i could copy the config file

2024-06-26T08:44:55.439Z

<Guillaume Abrioux> thanks!

2024-06-26T16:28:11.333Z

<Guillaume Abrioux> which project in [tracker.ceph.com](http://tracker.ceph.com) should I open a tracker for this <https://github.com/ceph/ceph/pull/58290> ?

2024-06-26T16:28:35.949Z

<Guillaume Abrioux> which project in [tracker.ceph.com](http://tracker.ceph.com) should I open a tracker in for this <https://github.com/ceph/ceph/pull/58290> ?

2024-06-26T17:15:46.783Z

<Dan Mick> my point was, you were talking about 2.chacra and 3.chacra, and 4.chacra was already upgraded, but you didn't mention 1.chacra.  The obvious question, to me, was "if 2 and 3 are slow, then what about 1?" [chacra.ceph.com](http://chacra.ceph.com) is different of course.

2024-06-26T17:23:30.681Z

<Adam Kraitman> You're right there is no reason for 1.chacra to be different but for some reason when I tested all nodes 1.chacra wasn't slow

2024-06-26T17:27:21.375Z

<Adam Kraitman> If I would have monitoring on the upload rates I could see if the rates are not stable like I saw yesterday, one time it was very low and another time it was normal

2024-06-26T17:30:24.056Z

<Dan Mick> ok.  that's all I wanted to know, was if 1.chacra was experiencing the same thing.  I agree there doesn't seem to be much point in not upgrading; upload speed on those hosts is critical.

2024-06-26T17:30:45.724Z

<Dan Mick> ok.  that's all I wanted to know, was if 1.chacra was experiencing the same thing.  I agree that upgrading makes sense; upload speed on those hosts is critical.

2024-06-26T17:31:05.279Z

<Dan Mick> Let me know if you need help with that or with [lists.ceph.io](http://lists.ceph.io)

2024-06-26T17:51:12.602Z

<Adam Kraitman> Thanks since I am not completely sure that the slowness is only caused because of slow upload rates on the chacra's, can you check if it's related to something else other then the chacra's?

2024-06-26T18:39:34.809Z

<Patrick Donnelly> probably just "Ceph"