2024-11-14T00:09:01.079Z

<Laura Flores> Thanks!

2024-11-14T02:05:44.432Z

<Dan Mick> @Adam Kraitman smithi056 (a 2.jenkins builder) was offline, "connection problems".  It's the only builder that can build containers at the moment.  I tried bringing it back, and it went away, and then came back, and then needed a new java.  I saw that it was trusty, and tried do-release-upgrade.  It went away mid-upgrade, so it may be in a bad state right now even given the connection-drop issues.  What's weird is that it's being dumb about connections to either ssh **or** ipmi, which ought to be completely different infrastructure...I don't have a clue.

2024-11-14T02:07:13.242Z

<Dan Mick> anyway, I had mira118 installed with jammy, so I hacked and ran the ansible on it to make it a builder, and then used the procedure in your PR to update to java 17, and it seems to be building now.

2024-11-14T02:07:41.281Z

<Dan Mick> still hoping to stop fighting fires and finish the release-build container work, which might have made this moot

2024-11-14T04:21:12.507Z

<gregsfortytwo> there’s a ceph-users mailing list question about the Octopus packages not being on [download.ceph.com](http://download.ceph.com). At first I just figured “that’s way past EOL”, but we have rpm folders going all the way back to Giant posted — just rpm-octopus is missing. Anybody know what’s going on there?

2024-11-14T04:23:52.829Z

<yuriw> All rpm-octopus of specific point?
I can take a look tomorrow at what was published last time and went thru QA, approvals etc. 

2024-11-14T04:56:09.838Z

<Dan Mick> I just moved them because the 3TB disk was out of space.  Ffs who is still relying on octopus?  I can symlink them back but we need to.have a conversation about how much we want to save....there's still pre-argonaut stuff there

2024-11-14T13:39:24.138Z

<Ronen Friedman> The latest 22 or so "shaman" entries have all failed, with no actual builds performed (for those I've checked)

2024-11-14T13:45:57.135Z

<Yuval Lifshitz> any idea about this failure i see in my tests:
```Command failed on smithi164 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1'```

2024-11-14T13:58:05.774Z

<Igor Golikov> I saw few of them in my tests... Unpredictable

2024-11-14T14:09:39.194Z

<Yuval Lifshitz> thanks

2024-11-14T16:25:24.726Z

<Ronen Friedman> No improvement yet. No branches are getting build.
... And I see some strange CI failures. e.g. inability to run the "were submodules changed?" check.
@Adam Kraitman?

2024-11-14T16:35:19.166Z

<Zack Cerza> I did a rebuild of your last build and it is - so far - running on an irvingi: <https://jenkins.ceph.com/job/ceph-dev-new/86157/>

2024-11-14T16:50:59.750Z

<Laura Flores> @Dan Mick @Adam Kraitman some jenkins jobs are failing from:
```Failed to evaluate name macro:org.jenkinsci.plugins.tokenmacro.MacroEvaluationException: Error processing tokens
ERROR: Unable to find a build for artifact copy from: ceph-dev-new-setup```
[https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVA[…]LE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic/84443/console](https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic/84443/console)

I'm trying retriggering the build, but wanted to message here to hopefully get ahead of the problem if it's not transient.

2024-11-14T16:52:34.719Z

<Laura Flores> @Dan Mick @Adam Kraitman some jenkins jobs are failing from:
```Failed to evaluate name macro:org.jenkinsci.plugins.tokenmacro.MacroEvaluationException: Error processing tokens
ERROR: Unable to find a build for artifact copy from: ceph-dev-new-setup```
[https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVA[…]LE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic/84443/console](https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic/84443/console)
[https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVA[…]IST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/84426/console](https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/84426/console)

I'm trying retriggering the build, but wanted to message here to hopefully get ahead of the problem if it's not transient.

2024-11-14T17:10:59.829Z

<Adam Kraitman> I re-imaged smithi056 with jammy and reconnected it to [2.jenkins.ceph.com](http://2.jenkins.ceph.com) you can use it again

2024-11-14T17:14:51.968Z

<Zack Cerza> That issue ^ should be resolved. There is an issue with podman affecting some other builds that I'm investigating now

2024-11-14T17:28:54.208Z

<Laura Flores> I see, thanks!

2024-11-14T17:28:57.770Z

<Laura Flores> @yuriw FYI

2024-11-14T17:30:48.147Z

<Laura Flores> Infra meeting starting now! <https://meet.jit.si/ceph-infra>

2024-11-14T18:02:01.540Z

<Dan Mick> [Lists.ceph.io](http://Lists.ceph.io) is a rhev vm that's running mailman in a docker container.  I'm pretty sure it's using iscsi to lrc but I'll check

2024-11-14T18:13:11.889Z

<Anthony D'Atri> Adam told me that the storage was backed by HDD but I'm not sure what's in between.

2024-11-14T19:51:17.424Z

<John Mulligan> <https://tracker.ceph.com/issues/62804>
@Casey Bodley FWIW, for the issue I mentioned in the infra meeting, I found my ubuntu 24.04 build issue with opentelemetry-cpp had been previously reported. I commented.

2024-11-14T23:15:18.804Z

<Laura Flores> @Dan Mick what are the steps to make arm64 required? Atp it looks like Rongqi has fixed all the arm-related failures.

2024-11-14T23:23:19.238Z

<Dan Mick> tweak something in github, I don't remember what. in branch protections maybe