2024-06-20T08:19:52.242Z | <Leonid Usov> Something’s wrong with sepia. 6k queue lentgh. My 10 jobs of priority 50 haven’t even started for over 2 hrs already.: https://files.slack.com/files-pri/T1HG3J90S-F078YF56KCK/download/image.png |
2024-06-20T08:21:02.820Z | <Leonid Usov> RGW enter is taking 7 hours: https://files.slack.com/files-pri/T1HG3J90S-F078YJRAAPL/download/image.png |
2024-06-20T14:59:30.563Z | <Kamoltat (Junior) Sirivadhna> Not sure if I’m late to the game but, I’ve been seeing [‘[ubuntu@smithi179.front.sepia.ceph.com](mailto:ubuntu@smithi179.front.sepia.ceph.com): Permission denied (publickey).’] failures lately in teuthology runs |
2024-06-20T15:04:40.764Z | <Vallari Agrawal> I saw it too in my last run: <https://pulpito.ceph.com/vallariag-2024-06-20_13:22:54-rbd:nvmeof-ceph-nvmeof-mon-distro-default-smithi/> |
2024-06-20T15:13:38.372Z | <Kamoltat (Junior) Sirivadhna> @Adam Kraitman @Dan Mick
something about mismatch keys:
```24-06-20T14:32:59.147 DEBUG:teuthology.task.kernel:Distro of this test job: centos
2024-06-20T14:33:00.148 INFO:teuthology.task.kernel:Enabling kdb on mon.e...
2024-06-20T14:33:00.148 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc
2024-06-20T14:33:00.235 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1
2024-06-20T14:33:00.265 DEBUG:teuthology.parallel:result is None
2024-06-20T14:33:05.226 ERROR:teuthology.orchestra.connection:Error authenticating with [smithi179.front.sepia.ceph.com](http://smithi179.front.sepia.ceph.com): BadHostKeyException('[smithi179.front.sepia.ceph.com](http://smithi179.front.sepia.ceph.com)', PKey(alg=ECDSA, bits=256, fp=SHA256:EHVTDBQ6UH82SJ7GiKkiiva1SLK8Tv51UtYTTD86IIY), PKey(alg=ED25519, bits=256, fp=SHA256:W+6Y/LgnRby4510vqEHFE/6fAReT8MdM2qNnjG5ecjQ)) (No SSH private key found!)
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_8e9714173de9e92c97e8ef1045d333e96b793454/teuthology/orchestra/connection.py", line 106, in connect
ssh.connect(**connect_args)
File "/home/teuthworker/src/git.ceph.com_teuthology_8e9714173de9e92c97e8ef1045d333e96b793454/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 468, in connect
raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server '[smithi179.front.sepia.ceph.com](http://smithi179.front.sepia.ceph.com)' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDCBevipXNFFmo/WmrXXCXtzB9CYK21DXaWbDg+mhxofk+5cjkD0KagOAcfpWXeE4VLt6gA4h0wo6FY+rFC6LvU=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIDnVB65LQ2ZrorCmRDXT/imYcIqttYXGdAR7PyxJCGep'
2024-06-20T14:33:33.245 ERROR:teuthology.orchestra.connection:Error authenticating with [smithi179.front.sepia.ceph.com](http://smithi179.front.sepia.ceph.com): BadHostKeyException('[smithi179.front.sepia.ceph.com](http://smithi179.front.sepia.ceph.com)', PKey(alg=ECDSA, bits=256, fp=SHA256:EHVTDBQ6UH82SJ7GiKkiiva1SLK8Tv51UtYTTD86IIY), PKey(alg=ED25519, bits=256, fp=SHA256:W+6Y/LgnRby4510vqEHFE/6fAReT8MdM2qNnjG5ecjQ)) (No SSH private key found!)
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_8e9714173de9e92c97e8ef1045d333e96b793454/teuthology/orchestra/connection.py", line 106, in connect
ssh.connect(**connect_args)
File "/home/teuthworker/src/git.ceph.com_teuthology_8e9714173de9e92c97e8ef1045d333e96b793454/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 468, in connect
raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server '[smithi179.front.sepia.ceph.com](http://smithi179.front.sepia.ceph.com)' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDCBevipXNFFmo/WmrXXCXtzB9CYK21DXaWbDg+mhxofk+5cjkD0KagOAcfpWXeE4VLt6gA4h0wo6FY+rFC6LvU=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIDnVB65LQ2ZrorCmRDXT/imYcIqttYXGdAR7PyxJCGep'
2024-06-20T14:33:33.245 DEBUG:teuthology.orchestra.remote:'connect to [smithi179.front.sepia.ceph.com](http://smithi179.front.sepia.ceph.com)' reached maximum tries (11) after waiting for 145 seconds
2024-06-20T14:33:42.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host '[ubuntu@smithi179.front.sepia.ceph.com](mailto:ubuntu@smithi179.front.sepia.ceph.com)'
2024-06-20T14:33:42.247 DEBUG:teuthology.orchestra.connection:{'hostname': '[smithi179.front.sepia.ceph.com](http://smithi179.front.sepia.ceph.com)', 'username': 'ubuntu', 'timeout': 60}
2024-06-20T14:33:42.566 DEBUG:teuthology.orchestra.run.smithi179:> true
2024-06-20T14:33:42.948 INFO:teuthology.orchestra.remote:Successfully reconnected to host '[ubuntu@smithi179.front.sepia.ceph.com](mailto:ubuntu@smithi179.front.sepia.ceph.com)'
2024-06-20T14:33:42.948 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-460.el9.x86_64"...
2024-06-20T14:33:42.948 DEBUG:teuthology.orchestra.run.smithi179:> uname -r
2024-06-20T14:33:42.965 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-460.el9.x86_64
2024-06-20T14:33:42.965 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-460.el9.x86_64 vs 5.14.0-460.el9.x86_64
2024-06-20T14:33:42.966 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install``` |
2024-06-20T15:14:36.231Z | <Kamoltat (Junior) Sirivadhna> @Adam Kraitman @Dan Mick
something about `NO SSH private key found`:
```24-06-20T14:32:59.147 DEBUG:teuthology.task.kernel:Distro of this test job: centos
2024-06-20T14:33:00.148 INFO:teuthology.task.kernel:Enabling kdb on mon.e...
2024-06-20T14:33:00.148 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc
2024-06-20T14:33:00.235 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1
2024-06-20T14:33:00.265 DEBUG:teuthology.parallel:result is None
2024-06-20T14:33:05.226 ERROR:teuthology.orchestra.connection:Error authenticating with smithi179.front.sepia.ceph.com: BadHostKeyException('smithi179.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:EHVTDBQ6UH82SJ7GiKkiiva1SLK8Tv51UtYTTD86IIY), PKey(alg=ED25519, bits=256, fp=SHA256:W+6Y/LgnRby4510vqEHFE/6fAReT8MdM2qNnjG5ecjQ)) (No SSH private key found!)
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_8e9714173de9e92c97e8ef1045d333e96b793454/teuthology/orchestra/connection.py", line 106, in connect
ssh.connect(**connect_args)
File "/home/teuthworker/src/git.ceph.com_teuthology_8e9714173de9e92c97e8ef1045d333e96b793454/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 468, in connect
raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi179.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDCBevipXNFFmo/WmrXXCXtzB9CYK21DXaWbDg+mhxofk+5cjkD0KagOAcfpWXeE4VLt6gA4h0wo6FY+rFC6LvU=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIDnVB65LQ2ZrorCmRDXT/imYcIqttYXGdAR7PyxJCGep'
2024-06-20T14:33:33.245 ERROR:teuthology.orchestra.connection:Error authenticating with smithi179.front.sepia.ceph.com: BadHostKeyException('smithi179.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:EHVTDBQ6UH82SJ7GiKkiiva1SLK8Tv51UtYTTD86IIY), PKey(alg=ED25519, bits=256, fp=SHA256:W+6Y/LgnRby4510vqEHFE/6fAReT8MdM2qNnjG5ecjQ)) (No SSH private key found!)
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_8e9714173de9e92c97e8ef1045d333e96b793454/teuthology/orchestra/connection.py", line 106, in connect
ssh.connect(**connect_args)
File "/home/teuthworker/src/git.ceph.com_teuthology_8e9714173de9e92c97e8ef1045d333e96b793454/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 468, in connect
raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi179.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDCBevipXNFFmo/WmrXXCXtzB9CYK21DXaWbDg+mhxofk+5cjkD0KagOAcfpWXeE4VLt6gA4h0wo6FY+rFC6LvU=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIDnVB65LQ2ZrorCmRDXT/imYcIqttYXGdAR7PyxJCGep'
2024-06-20T14:33:33.245 DEBUG:teuthology.orchestra.remote:'connect to smithi179.front.sepia.ceph.com' reached maximum tries (11) after waiting for 145 seconds
2024-06-20T14:33:42.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com'
2024-06-20T14:33:42.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-06-20T14:33:42.566 DEBUG:teuthology.orchestra.run.smithi179:> true
2024-06-20T14:33:42.948 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com'
2024-06-20T14:33:42.948 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-460.el9.x86_64"...
2024-06-20T14:33:42.948 DEBUG:teuthology.orchestra.run.smithi179:> uname -r
2024-06-20T14:33:42.965 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-460.el9.x86_64
2024-06-20T14:33:42.965 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-460.el9.x86_64 vs 5.14.0-460.el9.x86_64
2024-06-20T14:33:42.966 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install``` |
2024-06-20T15:53:29.705Z | <Zack Cerza> When did you start seeing this? |
2024-06-20T15:55:25.867Z | <Kamoltat (Junior) Sirivadhna> I started seeing this today |
2024-06-20T15:55:33.846Z | <Kamoltat (Junior) Sirivadhna> a 2-3 hours ago |
2024-06-20T16:00:08.823Z | <Vallari Agrawal> the above run is the first time I saw it (which failed around 14:21:24.399 UTC) |
2024-06-20T16:15:52.451Z | <Zack Cerza> Any theories yet? |
2024-06-20T16:16:02.391Z | <Zack Cerza> I'm seeing a number of "connection lost" type errors in sentry |
2024-06-20T16:16:57.269Z | <Zack Cerza> also very many instances of "hit max job timeout". probably not related, but not wonderful |
2024-06-20T16:26:23.144Z | <Dan Mick> Hm. Smells like network problems |
2024-06-20T19:54:03.028Z | <Kamoltat (Junior) Sirivadhna> issue seems to disappeared or at least doesn’t happen consistently. |
2024-06-20T19:54:17.743Z | <Kamoltat (Junior) Sirivadhna> thanks everyone for taking a look though |