2024-07-31T00:28:39.008Z

<Dan Mick> it's an AWS host that was getting DoSed by a script kiddie.  They've mitigated for now, should be working

2024-07-31T08:16:36.282Z

<Hemanth Kumar> @Adam Kraitman - I have updated the tracker as the previous one is not working.
Can you restore my access ?

2024-07-31T13:32:12.913Z

<Adam Kraitman> Hey @Hemanth Kumar yes, but since there are several tickets in the queue it can take a few days

2024-07-31T16:02:21.950Z

<Adam Kraitman> Hey @Casey Bodley I have installed libstdc++-12-dev to all the jammy builders using Ansible it also installed gcc-12

2024-07-31T16:03:50.896Z

<Casey Bodley> thank you

2024-07-31T16:04:29.897Z

<Casey Bodley> this seems to be causing other 'make check' failures, there's some continued discussion in <https://github.com/ceph/ceph/pull/58960#issuecomment-2260507422>

2024-07-31T16:40:14.323Z

<Patrick Donnelly> @Dan Mick dev playgrounds are not assigned to anyone explicitly except two are dedicated to specific components (cephfs/rados)

2024-07-31T16:40:29.993Z

<Patrick Donnelly> i.e. devs should pick whichever box they like tow ork on, generally

2024-07-31T16:47:05.693Z

<Casey Bodley> still trying fix 'make check'. does anyone else have suggestions here? cc @Matan Breizman

2024-07-31T16:58:38.308Z

<Matan Breizman> The `cannot find -lstdc++` issue is fixed, right?
If you refer to the seastore unit tests failures, I'll try pushing a patch tomorrow (without upgrading fmt)

2024-07-31T16:59:36.433Z

<Casey Bodley> correct, Adam installed the packages to resolve that part

2024-07-31T17:00:17.866Z

<Casey Bodley> what do you have in mind there? anything i can do to help?

2024-07-31T17:01:06.855Z

<Matan Breizman> While upgrading fmt locally I got new compilation errors which weren't there in 9.0.0. I highly suspect them to be the cause.

2024-07-31T17:01:52.179Z

<Casey Bodley> interesting, okay

2024-07-31T17:02:41.011Z

<Casey Bodley> i don't want to leave ci broken until tomorrow though, so i think we should try to clean up gcc-12 stuff in the meantime

2024-07-31T17:09:15.043Z

<Casey Bodley> @Adam Kraitman @Dan Mick @Zack Cerza could one of you please help me clean up this mess by _uninstalling_ `gcc-12 g++-12 libstdc++-12-dev` from all of the ubuntu 22.04 builders?

2024-07-31T17:09:27.764Z

<Casey Bodley> it sounds like we'll have to wait for ubuntu 24.04 builders to try again with gcc-13

2024-07-31T17:33:08.515Z

<Adam Kraitman> Hey @Casey Bodley yes I will run it now

2024-07-31T18:01:25.336Z

<Dan Mick> I can get to it soon if need be too

2024-07-31T18:03:23.656Z

<Zack Cerza> the teuthology host has been slow for the last few years; I routinely see `git pull` on small repos take >30s whereas it takes <1s elsewhere

2024-07-31T19:14:44.920Z

<Laura Flores> @Zack Cerza do you know if anything has to be done to update the crontab beyond a PR change? (i.e. <https://github.com/ceph/ceph/pull/58800>)

2024-07-31T19:17:08.994Z

<Zack Cerza> hm I actually don't know

2024-07-31T19:18:59.940Z

<Zack Cerza> not finding answers in ceph-build.git or [wiki.sepia.ceph.com](http://wiki.sepia.ceph.com)

2024-07-31T19:22:28.759Z

<Zack Cerza> <https://github.com/ceph/ceph/pull/58800/files#diff-d3b4c4534d7543f57f0d7c80afebfe6aa092b5497936d86fd6998c69411bea61L24-L33>

2024-07-31T19:24:28.635Z

<Zack Cerza> looks like the changes in that PR are reflected:
```teuthology@teuthology:~$ crontab -l | grep rados
# 01 01 * * 0 CEPH_BRANCH=main; MACHINE_NAME=smithi; SUITE_NAME=crimson-rados; KERNEL=distro;  $CW $SCHEDULE 100000 $CEPH_BRANCH $MACHINE_NAME $SUITE_NAME $CEPH_QA_EMAIL $KERNEL
## suites rados and rbd use --subset arg and must be call with schedule_subset.sh
# rados is massive and difficult to bring down to less than 300 jobs, use one higher priority
00 20 * * 0        $CW $SS 100000 --ceph main --suite      rados -p 101
## suites rados and rbd use --subset arg and must be call with schedule_subset.sh
# rados is massive and difficult to bring down to less than 300 jobs, use one higher priority
00 21 * * 0        $CW $SS 100000 --ceph squid --suite      rados -p 700 --force-priority
## suites rados and rbd use --subset arg and must be call with schedule_subset.sh
# rados is massive and difficult to bring down to less than 300 jobs, use one higher priority
00 22 * * 0        $CW $SS 100000 --ceph reef --suite      rados -p 931```

2024-07-31T19:42:12.635Z

<Laura Flores> Ok cool. Looks like it's there. Thanks!

2024-07-31T19:53:07.624Z

<Zack Cerza> no problem!

2024-07-31T20:21:00.931Z

<Casey Bodley> hmm recent 'make check's are still hitting the same `/usr/bin/ld: cannot find -lstdc++: No such file or directory` failure from <https://tracker.ceph.com/issues/67233>

2024-07-31T20:22:34.735Z

<Casey Bodley> i see that gcc-12 is gone, as expected, from `-DCMAKE_CXX_COMPILER=g++-11 -DCMAKE_C_COMPILER=gcc-11 -DCMAKE_CXX_COMPILER=clang++-14 -DCMAKE_C_COMPILER=clang-14`

2024-07-31T20:25:22.937Z

<Casey Bodley> could there be other gcc-12-related packages left over causing clang to keep trying to use gcc-12's standard library?

2024-07-31T20:26:51.134Z

<Casey Bodley> is it possible to check the output of `clang --verbose`? this is what it looks like when both are installed:
```$ clang --verbose
Ubuntu clang version 14.0.0-1ubuntu1.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64```

2024-07-31T20:31:40.743Z

<Zack Cerza> ```zack@smithi015:~$ clang --verbose
Ubuntu clang version 14.0.0-1ubuntu1.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64```
from `/var/log/apt/history.log`:
```Start-Date: 2024-07-26  19:47:17
Commandline: apt-get install -y gcc-12
Requested-By: jenkins-build (10105)
Install: gcc-12:amd64 (12.3.0-1ubuntu1~22.04), libtsan2:amd64 (12.3.0-1ubuntu1~22.04, automatic), libasan8:amd64 (12.3.0-1ubuntu1~22.04, automatic), cpp-12:amd64 (12.3.0-1ubuntu1~22.04, automatic), libgcc-12-dev:amd64 (12.3.0-1ubuntu1~22.04, automatic)
End-Date: 2024-07-26  19:47:27```

2024-07-31T20:34:19.855Z

<Zack Cerza> ```zack@smithi015:~$ clang --verbose
Ubuntu clang version 14.0.0-1ubuntu1.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64```
from `/var/log/apt/history.log`:
```Start-Date: 2024-07-26  19:47:17
Commandline: apt-get install -y gcc-12
Requested-By: jenkins-build (10105)
Install: gcc-12:amd64 (12.3.0-1ubuntu1~22.04), libtsan2:amd64 (12.3.0-1ubuntu1~22.04, automatic), libasan8:amd64 (12.3.0-1ubuntu1~22.04, automatic), cpp-12:amd64 (12.3.0-1ubuntu1~22.04, automatic), libgcc-12-dev:amd64 (12.3.0-1ubuntu1~22.04, automatic)
End-Date: 2024-07-26  19:47:27
...
Start-Date: 2024-07-31  18:06:58
Commandline: /usr/bin/apt-get -q -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-con
fold remove gcc-12 libstdc++-12-dev
Requested-By: cm (1100)
Remove: gcc-12:amd64 (12.3.0-1ubuntu1~22.04), libstdc++-12-dev:amd64 (12.3.0-1ubuntu1~22.04)
End-Date: 2024-07-31  18:07:03```
so I don't know precisely what's doing the install / uninstall, but we're not removing everything we put into place

2024-07-31T20:37:50.387Z

<Casey Bodley> ok, i have a failed 'make check' [log](https://jenkins.ceph.com/job/ceph-pull-requests/140451/consoleFull#-16451420407641bc0b-bacc-4481-8290-0cf28df0ce68) from smithi015

2024-07-31T20:38:07.251Z

<Casey Bodley> > ```The following packages were automatically installed and are no longer required:
>   cpp-12 libasan8 libgcc-12-dev libleveldb-dev libleveldb1d libtsan2 lua-any
>   lua5.1 luarocks```
libgcc-12-dev may be the culprit

2024-07-31T20:40:14.059Z

<Casey Bodley> Adam recently removed `gcc-12 g++-12 libstdc++-12-dev` from those ubuntu builders, could we try `cpp-12 libgcc-12-dev` too?