2024-10-21T10:55:38.212Z

<Rost Khudov> Hi @Shilpa Jagannath
I think I might find an issue in squid release.
I am using squid release, which is [v19.2.0](https://github.com/ceph/ceph/tree/v19.2.0)
I am running [multisite rgw test cases](https://github.com/ceph/ceph/blob/v19.2.0/src/test/rgw/test_multi.py), but there are 2 tests cases that are failing:
• test_bucket_index_log_trim
• test_bucket_reshard_incremental
The error is the same for both:
```----------------------------------------------------------------------
 Traceback (most recent call last):
   File "/usr/local/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
     self.test(*self.arg)
   File "/usr/local/lib/python3.6/site-packages/nose/util.py", line 620, in newfunc
     return func(*arg, **kw)
   File "/ceph-src/src/test/rgw/rgw_multi/tests.py", line 1645, in test_bucket_reshard_incremental
     zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name)
   File "/ceph-src/src/test/rgw/rgw_multi/tests.py", line 398, in zonegroup_bucket_checkpoint
     zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name)
   File "/ceph-src/src/test/rgw/rgw_multi/tests.py", line 390, in zone_bucket_checkpoint
     target_zone.cluster.admin(cmd, debug_rgw=1)
   File "/ceph-src/src/test/rgw/test_multi.py", line 65, in admin
     return bash(cmd, **kwargs)
   File "/ceph-src/src/test/rgw/test_multi.py", line 46, in bash
     assert(process.returncode == 0)
 AssertionError:```
Is it expected?

2024-10-21T18:51:37.903Z

<yuriw> `ceph-pr-docs` seems broken => <https://github.com/ceph/ceph/pull/60314>
<https://github.com/ceph/ceph/pull/60314/checks?check_run_id=31836848521>

2024-10-21T18:51:45.860Z

<yuriw> Traceback (most recent call last):
  File "/home/jenkins-build/build/workspace/ceph-pr-docs/venv/bin/github-check", line 181, in <module>
    main()
  File "/home/jenkins-build/build/workspace/ceph-pr-docs/venv/bin/github-check", line 157, in main
    post_sphinx_check(check, args)
  File "/home/jenkins-build/build/workspace/ceph-pr-docs/venv/bin/github-check", line 63, in post_sphinx_check
    for m in parser.scan(args.sphinx_output):
  File "/home/jenkins-build/build/workspace/ceph-pr-docs/venv/lib/python3.10/site-packages/githubcheck/sphinx.py", line 85, in scan
    if re.match(self.succeed_re, last_line):
  File "/usr/lib/python3.10/re.py", line 190, in match
    return _compile(pattern, flags).match(string)
TypeError: expected string or bytes-like object
+ false
Build step 'Execute shell' marked build as failure

2024-10-21T18:51:54.397Z

<yuriw> Can somebody take a look pls?

2024-10-21T20:51:22.646Z

<yuriw> @Venky Shankar @aad @Zac Dover ^

2024-10-21T21:09:40.624Z

<Zac Dover> @yuriw, I'm watching the console output. If it's something I can help with, I'll let you know.

2024-10-21T21:17:23.222Z

<Zac Dover> I got one of these errors over the weekend, and running "ceph test docs" caused it to work as expected. Still, it's weird and it shouldn't be happening. I'm sorry to say that I don't have any insight into the Jenkins plumbing.

2024-10-21T21:21:09.206Z

<yuriw> Who usually can fix those?

2024-10-21T21:23:12.885Z

<Zac Dover> I don't know. It used to be Kefu.

2024-10-21T21:23:24.410Z

<Zac Dover> I'm willing to learn how to do it, but I need to be taught.

2024-10-21T23:08:28.820Z

<shilpa jagannath> do they fail consistently? if you are using just one zonegroup with two zones syncing with each other, they should pass. some of these tests are a bit flaky. you could try and increase 'checkpoint_retries'  and 'reconfigure_delay' in test_multi.conf to 90 and see if that helps.