2024-08-19T00:28:04.128Z | <Dan Mick> I'm not following. Which grafana instance are you thinking of adding data to, which collector do you want to add, and which one has the admin account you want to find? |
2024-08-19T09:13:06.819Z | <Dhairya Parmar> hey sorry brad, it was a recharge day for us on 16th.
I checked the trackers and I don't think they would be relevant, <https://tracker.ceph.com/issues/66771> is a very rare case that occurs if you try to write some data > 2GiB in one go.
I checked the coredump bt in <https://tracker.ceph.com/issues/67565> is pointing to `if (fh == NULL || !_ll_fh_exists(fh))` in `Client::ll_writev` but the provided `fh` does exist in the set `ll_unclosed_fh_set` . This is weird. Can you me what is the I/O size being tried here? |
2024-08-19T09:13:21.448Z | <Dhairya Parmar> hey sorry brad, it was a recharge day for us on 16th.
I checked the trackers and I don't think they would be relevant, <https://tracker.ceph.com/issues/66771> is a very rare case that occurs if you try to write some data > 2GiB in one go.
I checked the coredump bt in <https://tracker.ceph.com/issues/67565> is pointing to `if (fh == NULL || !_ll_fh_exists(fh))` in `Client::ll_writev` but the provided `fh` does exist in the set `ll_unclosed_fh_set` . This is weird. Can you tell me what is the I/O size being tried here? |
2024-08-19T09:17:50.459Z | <Jeroen Roodhart> Hmmm, <https://download.ceph.com/rpm-reef/el8/> seems to have cleaned out.
Maybe this has something to do with the (mid-release) change from el8 to el9 containers? If so, this is _really_ annoying for us that are in _production_ on el8. We need to test and accept this kind of change in prod! Why is this happening with a minor release update?! |
2024-08-19T09:59:41.528Z | <IcePic> Jeroen: There was something about that on the maillists too, some el8 thing missing |
2024-08-19T10:00:14.239Z | <IcePic> Jeroen: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/74RQYR7AEH2NIQZJ2P2NMAADQUOAWATA/?sort=date |
2024-08-19T10:58:34.038Z | <Dhairya Parmar> venky pushed a PR <https://github.com/ceph/ceph/pull/59300> 🙂 |
2024-08-19T12:13:14.909Z | <David Orman> @Jeroen Roodhart because centos stream is a poor choice for the underlying distribution to release against and it went eol mid cycle on the ceph releases |
2024-08-19T12:13:27.444Z | <David Orman> That’s the tldr for you |
2024-08-19T12:16:48.881Z | <David Orman> I think it is also important to note that the containerized version of ceph has been suggested for years at this point. And if you’re using rhel8 RedHat has a version of ceph being managed by an entirely different team that will likely still have el8 packages based on rhel8 |
2024-08-19T14:02:09.399Z | <Laura Flores> Join us at the Telemetry CDS! <https://meet.google.com/nbv-pxaz-fzt?authuser=0> |
2024-08-19T14:41:37.194Z | <Yuval Lifshitz> i get the following issue in `src/osd/SnapMapper.cc` when trying to compile "reef" with gcc14:
```/home/yuvalif/ceph3/src/osd/SnapMapper.cc:331:26: required from here
331 | dout(20) << fmt::format("{}: key string: {} oid:{}", __func__, keys, oid)
| ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/fmt/ranges.h:490:26: error: call of overloaded _format_to(fmt::v8::appender&, const char [9], uint32_t&)_ is ambiguous
490 | out = format_to(out, "\\x{:02x}", escape.cp);``` |
2024-08-19T14:42:00.034Z | <Yuval Lifshitz> i get the following issue in `src/osd/SnapMapper.cc` when trying to compile "reef" with gcc14:
```/home/yuvalif/ceph3/src/osd/SnapMapper.cc:331:26: required from here
331 | dout(20) << fmt::format("{}: key string: {} oid:{}", __func__, keys, oid)
| ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/fmt/ranges.h:490:26: error: call of overloaded _format_to(fmt::v8::appender&, const char [9], uint32_t&)_ is ambiguous
490 | out = format_to(out, "\\x{:02x}", escape.cp);```
any idea? |
2024-08-19T14:55:55.465Z | <Matan Breizman> Is `WITH_SYSTEM_FMT` off? |
2024-08-19T14:56:53.065Z | <Yuval Lifshitz> yes. it is OFF |
2024-08-19T14:59:32.319Z | <Jeroen Roodhart> Thanks, so it is exactly how I feared. Shows what a _bad_ idea it is to base of of CentOS. Most people in their right mind are running something stable and still are now in an undefined situation.
For a storage platform this is a proper wobbly foundation IMO. |
2024-08-19T15:03:18.534Z | <Matan Breizman> It might be that std (C++20) and fmt have their own format_to. Can you try using an older standard? |
2024-08-19T15:03:45.384Z | <Matan Breizman> Perhaps 20 is not supported w/ Reef |
2024-08-19T15:04:59.401Z | <Jeroen Roodhart> Which is why we _are_ running containerised on RHEL8 (well OK Alma8). We prefer to run vanilla over Redhat/IBM so we like to use the upstream repos.
We mostly use this for redeploying test environments and client. packages, and use Ansible wrapped cephadm commands to set up. Would be nice to actually be able to download the RPMs though. |
2024-08-19T15:05:19.997Z | <Matan Breizman> Either explicitly change to fmt::format_to, if that's the only issue |
2024-08-19T15:05:20.832Z | <Yuval Lifshitz> ok, trying with `-DCMAKE_CXX_FLAGS="-std=c++17"` |
2024-08-19T15:42:02.713Z | <David Orman> Totally understand, just an unfortunate choice of distro to base the packages on. |
2024-08-19T15:42:49.909Z | <David Orman> With 8 stream eol it’s just dead in the water, the build infrastructure apparently couldn’t cope with the multiple versions being built so with one distro EOL the choice was made |
2024-08-19T18:56:24.880Z | <Ben H> this change broke thecontainerized version too |
2024-08-19T18:56:41.621Z | <Ben H> becuase Rhel8 can't run Rhel9 containers |
2024-08-19T18:57:33.960Z | <Ben H> due to an obscure threading bug |
2024-08-19T18:57:57.534Z | <David Orman> Not sure how to help on the RHEL side, I don’t run it nor do I have any part of the decision making when it comes to what happened here. I believe the RHEL side of the house has builds under commercial support for RHEL |
2024-08-19T18:58:21.214Z | <Ben H> + centos8 as well afaik |
2024-08-19T18:58:44.618Z | <David Orman> I am very sorry to hear of this, though. I hope there is a shift away from centos entirely for the oss builds |
2024-08-19T18:58:59.855Z | <Ben H> ref: <https://tracker.ceph.com/issues/66989> |
2024-08-19T18:59:16.291Z | <David Orman> Centos made good sense when it was a RHEL rebuild but now it’s a big mess |
2024-08-19T18:59:39.440Z | <Ben H> also affects some ubuntus. |
2024-08-19T18:59:58.691Z | <Ben H> yeah we are movin everything off centos |
2024-08-19T19:01:01.579Z | <Ben H> <https://ceph-storage.slack.com/archives/C1HFZTW81/p1721859598857449> |
2024-08-19T19:01:40.180Z | <David Orman> Yeah I’ve read all of these threads |
2024-08-19T19:01:55.277Z | <David Orman> Centos8 stream is EOL, there are no more security updates for it, etc |
2024-08-19T19:02:17.720Z | <David Orman> There’s just no real “good” option there, and the community ceph build isn’t built on rhel8 |
2024-08-19T19:02:26.968Z | <David Orman> The RedHat storage thing is |
2024-08-19T20:37:33.181Z | <badone> Awesome, thank you both for getting involved. |