2024-10-08T05:23:17.633Z | <Sunil Angadi> sure will include that. |
2024-10-08T07:19:31.759Z | <Adam Kupczyk> Sepia VPN is not connecting now.
Do you guys have the same? |
2024-10-08T07:25:35.868Z | <Nitzan Mordechai> yes, i have it as well |
2024-10-08T07:49:37.269Z | <Ronen Friedman> Yes |
2024-10-08T07:50:24.280Z | <Ronen Friedman> Ah. See the messages from yesterday. |
2024-10-08T07:50:47.382Z | <Aishwarya Mathuria> Ah thanks for pointing it out |
2024-10-08T07:59:55.745Z | <Adam Kupczyk> Holiday! |
2024-10-08T08:38:31.911Z | <Yuval Lifshitz> same problem here ;-( |
2024-10-08T08:38:50.037Z | <Yuval Lifshitz> same problem here :-( |
2024-10-08T09:25:59.218Z | <Aishwarya Mathuria> It's working for me now |
2024-10-08T09:28:19.615Z | <Vallari Agrawal> It looks like I'm already added as a "Developer" in the list. |
2024-10-08T09:59:00.086Z | <Ronen Friedman> But resolving [api.github.com](http://api.github.com) - does not. |
2024-10-08T09:59:38.647Z | <Ronen Friedman> The problem is in the resolving. Pinging the IP works. |
2024-10-08T10:00:24.036Z | <Aishwarya Mathuria> Yup realised that now |
2024-10-08T10:20:06.434Z | <Ronen Friedman> Solved (for now) with
sudo resolvectl dns ens20f0np0 8.8.8.8
(help, as usual, by Mark) |
2024-10-08T10:36:25.180Z | <Nitzan Mordechai> <https://pulpito.ceph.com/> having some issue to start? |
2024-10-08T10:38:18.001Z | <Naveen Naidu> I am also already added in the developer list for the project :) |
2024-10-08T11:42:28.127Z | <Aviv Caro> Same question with regards to <https://quay.ceph.io/> |
2024-10-08T11:59:46.907Z | <rzarzynski> same about [o06.front.sepia.ceph.com](http://o06.front.sepia.ceph.com) |
2024-10-08T12:01:06.878Z | <rzarzynski> what is funny is that [o04.front.sepia.ceph.com](http://o04.front.sepia.ceph.com) resolves and works fine |
2024-10-08T12:02:48.662Z | <rzarzynski> looks the switch upgrade is still ongoing |
2024-10-08T12:03:32.816Z | <rzarzynski> (or there is a fallout) |
2024-10-08T12:18:13.579Z | <rzarzynski> oops, <https://wiki.sepia.ceph.com/doku.php?id=devplayground> is returning `50 Bad Gateway` . Looks like not everything survived the upgrade 😉 |
2024-10-08T12:22:15.603Z | <Shraddha Agrawal> I am still facing issues with pulpito. |
2024-10-08T12:24:04.070Z | <Shraddha Agrawal> I am still facing issues with pulpito and paddles (returns 502). |
2024-10-08T12:28:25.821Z | <John Mulligan> etherpad as well |
2024-10-08T13:28:33.239Z | <rzarzynski> @Adam Kraitman: ^^ |
2024-10-08T13:39:43.926Z | <Yaarit> same for [telemetry.front.sepia.ceph.com](http://telemetry.front.sepia.ceph.com) - I see the VM was rebooted and some services stopped working |
2024-10-08T13:40:45.838Z | <Yaarit> [telemetry-public.front.sepia.ceph.com](http://telemetry-public.front.sepia.ceph.com) was not rebooted |
2024-10-08T13:50:39.055Z | <Yaarit> looks like it's a DNS issue like @Ronen Friedman suggested |
2024-10-08T15:09:59.679Z | <yuriw> [pad.ceph.com](http://pad.ceph.com) is dead as well |
2024-10-08T16:10:50.453Z | <Zack Cerza> [pad.ceph.com](http://pad.ceph.com) is a CNAME for [gw.sepia.ceph.com](http://gw.sepia.ceph.com); I tried to restart nginx there but it dumped core. |
2024-10-08T16:11:21.451Z | <Zack Cerza> teuthology-dispatcher has failed because it can't access the cephfs mount. |
2024-10-08T16:24:21.342Z | <Zack Cerza> I keep seeing messages from sshd: `error: Unable to load host key: /etc/ssh/ssh_host_drop_ceph_com` |
2024-10-08T16:24:54.581Z | <Zack Cerza> I keep seeing messages from sshd: `error: Unable to load host key: /etc/ssh/ssh_host_drop_ceph_com`
In `/root/.bash_history` I see `rm ssh_host_drop_ceph_com ssh_host_drop_ceph_com.pub` |
2024-10-08T16:32:01.410Z | <Zack Cerza> ah, gw only had `172.21.0.1` & `172.21.0.2` configured as internal nameservers, and it can't reach them. Adding vpn-pub at least got it to resolve queries |
2024-10-08T16:56:06.983Z | <Adam Kraitman> Those services are down because rhev host hv02 is down I am still working on fixing it: https://files.slack.com/files-pri/T1HG3J90S-F07RM9M36L8/download/image.png |
2024-10-08T16:57:38.043Z | <Zack Cerza> ok. turns out nginx has been refusing to start because the hosts it proxies to are down; there is a way to configure it so that it will at least start up, I'm working on that |
2024-10-08T17:41:39.348Z | <Zack Cerza> ok, our reverse proxies are less fragile now |
2024-10-08T17:42:41.543Z | <Zack Cerza> ok, our reverse proxies are less fragile now (edit: the hosts they proxy _to_ are still down) |
2024-10-08T18:05:04.083Z | <Adam Kraitman> Services wore migrated to other rhev hosts all services are back online |
2024-10-08T19:30:05.889Z | <Vallari Agrawal> looks like <http://qa-proxy.ceph.com/> is still down? |
2024-10-08T19:56:02.235Z | <Zack Cerza> yeah, cephfs is still broken |
2024-10-08T19:57:46.659Z | <Vallari Agrawal> ah okay! |