2024-10-10T00:11:07.802Z | <Dan Mick> Will look into it. @Zack Cerza did something to the webserver proxy recently but I'm not certain what. |
2024-10-10T00:20:26.086Z | <badone> No movement on this yet Dan. I guess DW aren't treating this as much of a priority. |
2024-10-10T01:17:23.235Z | <Zack Cerza> uhoh. I can take a look at that |
2024-10-10T01:24:56.552Z | <Dan Mick> well |
2024-10-10T01:25:38.106Z | <Dan Mick> I did get some response on the ticket but haven't finished following up. They pointed to an SPF/DKIM rejection that might be from us and might be new; they said "rh" but I don't think rh's servers are involved |
2024-10-10T01:25:51.496Z | <Dan Mick> (well, unless you call sepia "rh") |
2024-10-10T01:52:51.785Z | <Zack Cerza> this should be resolved |
2024-10-10T02:07:40.387Z | <Dan Mick> @Zack Cerza figured it out, should be working now @Samuel Just |
2024-10-10T02:33:36.880Z | <badone> Ah, OK |
2024-10-10T06:58:23.794Z | <Adam Kraitman> It could be the case I will reach out to him about it |
2024-10-10T07:16:23.696Z | <Aviv Caro> Thx Adam, fixing it will be helpful, as currently our CI and other tests are failing because of that. |
2024-10-10T09:23:48.526Z | <Jose J Palacios-Perez> Hi <@U04JGS8QR9U> apologies if already know, but when convinient, could you please have a look at o05, its not pingable since the network issues, thanks in advance: https://files.slack.com/files-pri/T1HG3J90S-F07RMUVJM97/download/screenshot_2024-10-10_at_10.20.44.png |
2024-10-10T11:23:39.059Z | <Adam Kraitman> All good try now |
2024-10-10T11:26:32.411Z | <Jose J Palacios-Perez> Yes, its solved, thank you! |
2024-10-10T12:21:25.633Z | <Zac Dover> @Laura Flores, the mention of the "Stable Releases and Backports" team has been removed in the following two PRs: <https://github.com/ceph/ceph/pull/60242>
<https://github.com/ceph/ceph/pull/60243>.
Anthony D'Atri has already approved these, and as soon as they have run the Jenkins-test gauntlet, I'll backport them. This means that you can regard this matter as resolved (unless you find an error, of course). |
2024-10-10T14:06:03.072Z | <Adam Kraitman> 👍 |
2024-10-10T14:15:35.532Z | <yuriw> c9 still failed, retrying |
2024-10-10T14:15:39.845Z | <yuriw> <https://shaman.ceph.com/builds/ceph/wip-yuri-testing-2024-10-09-1524-squid/> |
2024-10-10T16:46:05.872Z | <Samuel Just> Yep, looks good. Thanks! |
2024-10-10T16:49:26.506Z | <yuriw> Unsure is this is the cause:
```[chacractl][ERROR ] RuntimeError: HEAD for <https://3.chacra.ceph.com/binaries/ceph/wip-yuri-testing-2024-10-09-1524-squid/913305eb5dc01a5e05f0ac5b0f77d73c679f3f43/centos/9/x86_64/flavors/default/librados2-19.2.0-458-g913305eb-0.el9.x86_64.rpm/> failed: 404 resource does not exist```
In [https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVA[…]entos9,DIST=centos9,MACHINE_SIZE=gigantic/83686//consoleFull](https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83686//consoleFull)
@Laura Flores @Dan Mick ^ |
2024-10-10T16:55:27.216Z | <Dan Mick> No, it's this: |
2024-10-10T16:55:47.340Z | <Dan Mick> STEP 9/35: RUN /bin/echo -e "FROM_IMAGE: ${FROM_IMAGE}\nCEPH_REF: ${CEPH_REF}\nGANESHA_REPO_BASEURL: ${GANESHA_REPO_BASEURL} \nOSD_FLAVOR: ${OSD_FLAVOR} \nCI_CONTAINER: ${CI_CONTAINER}"
/bin/sh: error while loading shared libraries: /lib64/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
Error: building at STEP "RUN /bin/echo -e "FROM_IMAGE: ${FROM_IMAGE}\nCEPH_REF: ${CEPH_REF}\nGANESHA_REPO_BASEURL: ${GANESHA_REPO_BASEURL} \nOSD_FLAVOR: ${OSD_FLAVOR} \nCI_CONTAINER: ${CI_CONTAINER}"": while running runtime: exit status 127
Thu Oct 10 12:59:39 AM UTC 2024 :: rm -fr /tmp/install-deps.2623743
Build step 'Execute shell' marked build as failure |
2024-10-10T16:56:53.853Z | <Dan Mick> Something's broken about podman or selinux on that host |
2024-10-10T16:57:19.088Z | <Dan Mick> I'll mark it down for investigation |
2024-10-10T16:57:34.199Z | <yuriw> • I did retrigger it as well, maybe it will take a new host |
2024-10-10T16:57:45.734Z | <yuriw> I did retrigger it as well, maybe it will take a new host |
2024-10-10T17:24:19.879Z | <Adam Kraitman> Zack fixed the issue you can docker pull again |
2024-10-10T18:16:14.699Z | <Dan Mick> it will now |
2024-10-10T18:16:36.090Z | <Dan Mick> (the chacra 404s are normal, btw...that's just the job checking to see if the package already exists before sending it) |
2024-10-10T19:35:56.803Z | <yuriw> I pushed 'wip-yuri-testing-2024-10-09-1524-squid' twice but still don't see it 🤷♂️ when searching jenkins |
2024-10-10T19:38:48.662Z | <yuriw> and main is failing as well ref: <https://tracker.ceph.com/issues/68375>
<https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83720//consoleFull>
/bin/sh: error while loading shared libraries: /lib64/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
Error: building at STEP "RUN /bin/echo -e "FROM_IMAGE: ${FROM_IMAGE}\nCEPH_REF: ${CEPH_REF}\nGANESHA_REPO_BASEURL: ${GANESHA_REPO_BASEURL} \nOSD_FLAVOR: ${OSD_FLAVOR} \nCI_CONTAINER: ${CI_CONTAINER}"": while running runtime: exit status 127
Thu Oct 10 06:59:30 PM UTC 2024 :: rm -fr /tmp/install-deps.2926809
Build step 'Execute shell' marked build as failure |
2024-10-10T20:16:24.758Z | <Dan Mick> same symptom. I will mark adami02 down as well |
2024-10-10T20:24:41.813Z | <yuriw> I will rebase and push it again , thx |
2024-10-10T21:54:34.603Z | <Laura Flores> @Dan Mick I'm seeing teuthology jobs fail from "image platform (linux/arm64/v8) does not match the expected platform (linux/amd64)":
<https://qa-proxy.ceph.com/teuthology/yuriw-2024-10-10_19:33:29-rados-wip-yuri13-testing-2024-10-09-1459-distro-default-smithi/7943443/teuthology.log>
```2024-10-10T19:50:56.866 INFO:tasks.cephadm:Pulling image [quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50](http://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50) on all hosts...
2024-10-10T19:50:56.867 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image [quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50](http://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50) pull
2024-10-10T19:50:56.870 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image [quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50](http://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50) pull
2024-10-10T19:50:56.872 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image [quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50](http://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50) pull
2024-10-10T19:50:58.499 INFO:teuthology.orchestra.run.smithi012.stderr:Pulling container image [quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50](http://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50)...
2024-10-10T19:50:58.549 INFO:teuthology.orchestra.run.smithi037.stderr:Pulling container image [quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50](http://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50)...
2024-10-10T19:50:58.617 INFO:teuthology.orchestra.run.smithi042.stderr:Pulling container image [quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50](http://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50)...
2024-10-10T19:51:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=[quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50](http://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50) -e NODE_NAME=smithi012 [quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50](http://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50) --version
2024-10-10T19:51:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:ceph: stderr WARNING: image platform (linux/arm64/v8) does not match the expected platform (linux/amd64)
2024-10-10T19:51:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:ceph: stderr ERROR (catatonit:2): failed to exec pid1: Exec format error
2024-10-10T19:51:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:Traceback (most recent call last):
2024-10-10T19:51:27.706 INFO:teuthology.orchestra.run.smithi012.stderr: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
2024-10-10T19:51:27.713 INFO:teuthology.orchestra.run.smithi012.stderr: return _run_code(code, main_globals, None,
2024-10-10T19:51:27.713 INFO:teuthology.orchestra.run.smithi012.stderr: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
2024-10-10T19:51:27.713 INFO:teuthology.orchestra.run.smithi012.stderr: exec(code, run_globals)
2024-10-10T19:51:27.713 INFO:teuthology.orchestra.run.smithi012.stderr: File "/tmp/tmp8gnbhoo5.cephadm.build/app/__main__.py", line 5628, in <module>
2024-10-10T19:51:27.713 INFO:teuthology.orchestra.run.smithi012.stderr: File "/tmp/tmp8gnbhoo5.cephadm.build/app/__main__.py", line 5616, in main
2024-10-10T19:51:27.713 INFO:teuthology.orchestra.run.smithi012.stderr: File "/tmp/tmp8gnbhoo5.cephadm.build/app/__main__.py", line 451, in _default_image
2024-10-10T19:51:27.713 INFO:teuthology.orchestra.run.smithi012.stderr: File "/tmp/tmp8gnbhoo5.cephadm.build/app/__main__.py", line 1917, in command_pull
2024-10-10T19:51:27.713 INFO:teuthology.orchestra.run.smithi012.stderr: File "/tmp/tmp8gnbhoo5.cephadm.build/app/cephadmlib/decorators.py", line 27, in _require_image
2024-10-10T19:51:27.713 INFO:teuthology.orchestra.run.smithi012.stderr: File "/tmp/tmp8gnbhoo5.cephadm.build/app/__main__.py", line 442, in _infer_image
2024-10-10T19:51:27.713 INFO:teuthology.orchestra.run.smithi012.stderr: File "/tmp/tmp8gnbhoo5.cephadm.build/app/__main__.py", line 1964, in command_inspect_image
2024-10-10T19:51:27.714 INFO:teuthology.orchestra.run.smithi012.stderr: File "/tmp/tmp8gnbhoo5.cephadm.build/app/cephadmlib/container_types.py", line 429, in run
2024-10-10T19:51:27.714 INFO:teuthology.orchestra.run.smithi012.stderr: File "/tmp/tmp8gnbhoo5.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws
2024-10-10T19:51:27.714 INFO:teuthology.orchestra.run.smithi012.stderr:RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=[quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50](http://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50) -e NODE_NAME=smithi012 [quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50](http://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50) --version: WARNING: image platform (linux/arm64/v8) does not match the expected platform (linux/amd64)
2024-10-10T19:51:27.714 INFO:teuthology.orchestra.run.smithi012.stderr:ERROR (catatonit:2): failed to exec pid1: Exec format error
2024-10-10T19:51:27.714 INFO:teuthology.orchestra.run.smithi012.stderr:
2024-10-10T19:51:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-10-10T19:51:27.745 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/contextutil.py", line 30, in nested
vars.append(enter())
File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/home/teuthworker/src/github.com_ceph_ceph-c_56b67e1ba5aa7927c9cd4d7a2884d694c672fe50/qa/tasks/cephadm.py", line 618, in pull_image
run.wait(ctx.cluster.run(args=cmd, wait=False))
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/orchestra/run.py", line 479, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image [quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50](http://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50) pull'``` |
2024-10-10T21:55:05.524Z | <Laura Flores> Seems like the issue with with "v8"? |
2024-10-10T21:55:36.976Z | <Laura Flores> I reproduced the issue locally:
```lflores:~$ podman image pull [quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50](http://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50)
Trying to pull [quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50](http://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50)...
Getting image source signatures
Copying blob e75b3b6cce54 done |
Copying blob 644ec3c9e9ce done |
Copying config 50609d8956 done |
Writing manifest to image destination
WARNING: image platform (linux/arm64/v8) does not match the expected platform (linux/amd64)
50609d8956eb76e4607816c344cb6c99bf3b439d3b7cbac0187fbc57e54b47b7``` |
2024-10-10T22:00:10.927Z | <Laura Flores> Or perhaps the amd64 arch is not getting created correctly |
2024-10-10T22:17:28.141Z | <Dan Mick> I...don't know what that is |
2024-10-10T23:23:40.312Z | <badone> @Laura Flores did you try passing the arch as an argument or looking at what the manifest lists as available arches? <https://www.redhat.com/sysadmin/specify-architecture-pulling-podman-images> |
2024-10-10T23:24:43.723Z | <badone> I'd try but apparently I have a login but I have no idea what the password is and can't seem to find a way to change it 😞 |
2024-10-10T23:28:03.128Z | <Dan Mick> skopeo inspect <docker://quay.ceph.io/ceph-ci/ceph:56b67e1ba5aa7927c9cd4d7a2884d694c672fe50> shows "Architecture: arm64" |
2024-10-10T23:28:32.689Z | <Dan Mick> so it looks like that image was from an arm build |
2024-10-10T23:29:05.717Z | <Dan Mick> hm. those would collide with amd64 builds if they pushed the sha1 tag. maybe that's my bug |
2024-10-10T23:29:36.298Z | <badone> Race? Last overwrite wins? |
2024-10-10T23:30:12.925Z | <Dan Mick> it's supposed to make tags for branch and sha1 only, and suffix them with -aarch64 |
2024-10-10T23:30:13.813Z | <badone> I'm very fuzzy on how multi-arch manifests work at that level |
2024-10-10T23:30:31.770Z | <Dan Mick> Ci doesn't do multi-arch |
2024-10-10T23:30:43.059Z | <badone> Ah |
2024-10-10T23:30:49.101Z | <Dan Mick> ah, yes, my bug, it's testing for aarch64, but the arch has been modified to arm64 by then. |
2024-10-10T23:30:50.903Z | <Dan Mick> PR coming. |
2024-10-10T23:31:32.003Z | <badone> Awesome. If anyone can point me to how I change my password I'd be grateful |
2024-10-10T23:31:50.174Z | <Dan Mick> what password |
2024-10-10T23:32:11.562Z | <badone> To log into [quay-quay-quay.apps.os.sepia.ceph.com](http://quay-quay-quay.apps.os.sepia.ceph.com) |
2024-10-10T23:32:59.167Z | <Dan Mick> you shouldn't need to log in to get at skopeo inspect or to pull, but |
2024-10-10T23:33:34.352Z | <Dan Mick> there's no "forgot password" dialog? |
2024-10-10T23:33:50.941Z | <badone> No, not that I could see (and I looked) |
2024-10-10T23:33:58.221Z | <badone> ```$ podman manifest inspect [quay-quay-quay.apps.os.sepia.ceph.com/repository/ceph-ci/ceph](http://quay-quay-quay.apps.os.sepia.ceph.com/repository/ceph-ci/ceph)
Error: reading image "<docker://quay-quay-quay.apps.os.sepia.ceph.com/repository/ceph-ci/ceph:latest>": reading manifest latest in [quay-quay-quay.apps.os.sepia.ceph.com/repository/ceph-ci/ceph](http://quay-quay-quay.apps.os.sepia.ceph.com/repository/ceph-ci/ceph): unauthorized: access to the requested resource is not authorized``` |
2024-10-10T23:34:16.207Z | <badone> I was trying to get around that |
2024-10-10T23:34:49.328Z | <badone> Neither important, nor urgent Dan |
2024-10-10T23:35:08.109Z | <Dan Mick> there's no latest tag. I really hate podman errors |
2024-10-10T23:35:20.764Z | <badone> Doh! |
2024-10-10T23:36:43.064Z | <Dan Mick> can you skopeo list-tags <docker://quay.ceph.io/ceph-ci/ceph> ? |
2024-10-10T23:37:16.827Z | <badone> Yes, works fine |
2024-10-10T23:37:26.440Z | <badone> Should have tried that first |
2024-10-10T23:37:27.803Z | <Dan Mick> ok. I wanted to be sure that didn't need login |
2024-10-10T23:38:21.805Z | <badone> You have bigger fish to fry mate 🙂 |
2024-10-10T23:47:08.142Z | <Dan Mick> @Laura Flores <https://github.com/ceph/ceph/pull/60255> |