2024-06-25T03:28:42.897Z | <Jos Collin> We have a problem scheduling QA runs for the quincy QA Testing. There are 4 batches of PRs ready for testing in quincy branch: <https://tracker.ceph.com/issues/66599>, <https://tracker.ceph.com/issues/66597>, <https://tracker.ceph.com/issues/66596>, <https://tracker.ceph.com/issues/66642>. The problem is that none of these batches are passing the shaman build for `centos` due to the lack of centos8 support. So the teuthology-suite command fails showing error: `At least one job needs packages that don't exist for hash <>`. I could still start a few jobs (1 or 2) by narrowing it down to use `-d ubuntu`. But that certainly won't do good for the QA. Using the option `--filter-out=centos8` doesn't help either. How can I get the shaman builds passed for quincy? |
2024-06-25T03:33:27.118Z | <Jos Collin> We have a problem scheduling QA runs for the quincy QA Testing. There are 4 batches of PRs ready for testing in quincy branch: <https://tracker.ceph.com/issues/66599>, <https://tracker.ceph.com/issues/66597>, <https://tracker.ceph.com/issues/66596>, <https://tracker.ceph.com/issues/66642>. The problem is that none of these batches are passing the shaman build for `centos` due to the lack of centos8 support. So the teuthology-suite command fails showing error: `At least one job needs packages that don't exist for hash <>`. I could still start a few jobs (1 or 2) by narrowing it down to use `-d ubuntu`. But that certainly won't do good for the QA. Using the option `--filter-out=centos8` doesn't help either. How can I get the shaman builds passed for quincy? What's the proposed strategy? |
2024-06-25T04:28:32.644Z | <Leonid Usov> For reef, I had to use this filter:
``` --filter-out centos_8,rhel_8,single-container-host``` |
2024-06-25T05:59:25.169Z | <Jos Collin> I had filter out `centos` too. |
2024-06-25T06:00:16.625Z | <Jos Collin> and got this scheduled <https://pulpito.ceph.com/jcollin-2024-06-25_05:57:40-fs-wip-jcollin-testing-20240621.040312-quincy-distro-default-smithi/> |
2024-06-25T06:01:26.360Z | <Jos Collin> I had filter out `centos` too to get rid of `jobs that are missing packages!` |
2024-06-25T09:11:51.553Z | <Kyrylo Shatskyy> I (username kshtsk) had an access to [wiki.sepia.ceph.com](http://wiki.sepia.ceph.com) but it appears it does not let me, who is currently maintaining the resource? |
2024-06-25T10:46:45.634Z | <Adam Kraitman> Hey @Kyrylo Shatskyy I will send you an e-mail with your new password |
2024-06-25T13:18:14.512Z | <Kyrylo Shatskyy> hey Adam, thanks that would be great |
2024-06-25T13:19:19.701Z | <Kyrylo Shatskyy> I hope you use my [gmail.com](http://gmail.com), not suse one |
2024-06-25T14:47:14.196Z | <yuriw> Builds are very slow and often require manual triggering (usually jammy and focal), just FYI, maybe we can rearrange some nodes...
cc: @Venky Shankar @nehaojha @Josh Durgin @Dan Mick @Adam Kraitman |
2024-06-25T15:17:19.607Z | <Rishabh Dave> how can jammy and focal builds be triggered manually? i am facing the same issue. |
2024-06-25T15:19:47.262Z | <yuriw> @Rishabh Dave via jenkins form
<https://jenkins.ceph.com/job/ceph-dev-new/build?delay=0sec> |
2024-06-25T15:21:04.329Z | <Adam Kraitman> Yeah I used this e-mail [kyrylo.shatskyy@gmail.com](mailto:kyrylo.shatskyy@gmail.com) did you got an e-mail? sometimes it goes to the spam folder |
2024-06-25T15:21:44.720Z | <Rishabh Dave> thanks, checking... |
2024-06-25T15:23:16.013Z | <Rishabh Dave> @yuriw git branch mentioned has to be a my ceph-ci branch, right? |
2024-06-25T15:26:33.236Z | <yuriw> yup |
2024-06-25T15:26:53.133Z | <yuriw> you just use distro you need (missing) |
2024-06-25T15:28:21.599Z | <Rishabh Dave> and how do i know that the build has been launched? i didn't seen any feedback on submitting the form and i don't seen anything on shaman yet. |
2024-06-25T15:29:04.734Z | <Adam Kraitman> @yuriw I started working on migrating more builders to jammy that should reduce some of the load <https://jenkins.ceph.com/label/gigantic&&x86_64&&jammy/load-statistics> |
2024-06-25T15:30:41.847Z | <Kyrylo Shatskyy> No, I cannot see anything like that, spam folder is also looks untouched |
2024-06-25T15:32:39.299Z | <yuriw> Thx @Adam Kraitman that's a welcome update. To build a single branch literally takes more than a day! |
2024-06-25T15:33:09.984Z | <yuriw> maybe some builders are down? |
2024-06-25T15:34:13.118Z | <Rishabh Dave> is that a reply to me? if so, there's no way to build jammy/focal for now? |
2024-06-25T15:34:37.882Z | <yuriw> no to @Adam Kraitman |
2024-06-25T15:35:32.148Z | <Rishabh Dave> @yuriw how do i know that the build has been launched? i didn't seen any feedback on submitting the form and i don't seen anything on shaman yet. |
2024-06-25T15:41:25.878Z | <Venky Shankar> Generally, you should see a new entry in the sidebar of the jenkins form. |
2024-06-25T15:41:34.452Z | <yuriw> @Rishabh Dave its not too intuitive, but ...
In jenkins you can search for your wip in Filter:
Say for my wip wip-yuri11-testing-2024-06-25-0817-quincy
It shows jobs 82016
Then in Parameters you will see what you are building |
2024-06-25T15:41:41.404Z | <Venky Shankar> There can be some delay though |
2024-06-25T15:41:53.163Z | <yuriw> Very convenient, yes ?! |
2024-06-25T15:41:55.201Z | <yuriw> ๐ |
2024-06-25T15:48:36.169Z | <Venky Shankar> I will backport the change that would allow fs:upgrade to rely on centos9 |
2024-06-25T15:48:41.928Z | <Venky Shankar> so you won't have to filter things out |
2024-06-25T15:48:51.572Z | <Venky Shankar> which reduces the number of test jobs ๐ |
2024-06-25T15:48:55.103Z | <Venky Shankar> on that soon! |
2024-06-25T18:17:53.023Z | <Dan Mick> @yuriw when you say "require manual triggering", is that because the distros aren't set correctly, or because certain distros seem to often fail to build, or?... |
2024-06-25T18:33:16.642Z | <yuriw> @Dan Mick I suspect (and it's not 100%) but some distro seem not to start building at all or very slow to start building. I see c9 built first and several hours after the rest get built, sometimes distro do not show up at all. But I have not traced it. |
2024-06-25T18:33:59.190Z | <yuriw> here is the latest I pushed <https://shaman.ceph.com/builds/ceph/wip-yuri11-testing-2024-06-25-0817-quincy/> |
2024-06-25T18:47:58Z | <Dan Mick> ok so that's 82016 |
2024-06-25T18:48:15.984Z | <Dan Mick> it's requested focal, jammy, centos9, leap15 |
2024-06-25T18:49:26.558Z | <Dan Mick> focal and jammy are waiting for nodes; manually triggering another build would only increase the congestion, since that build would eventually schedule |
2024-06-25T18:56:45.269Z | <yuriw> Whatโs leap15? |
2024-06-25T18:58:00.882Z | <Dan Mick> SuSE, which should not be build AFAIK (and it failed) |
2024-06-25T18:58:13.530Z | <Dan Mick> SuSE, which should not be built AFAIK (and it failed) |
2024-06-25T18:58:22.859Z | <Dan Mick> still waiting for clt to decide what we build |
2024-06-25T19:32:38.436Z | <Zack Cerza> could we add more focal/jammy nodes? |
2024-06-25T19:46:32.714Z | <Dan Mick> just looking at node stats to see what we have |