2024-12-19T00:00:11.687Z | <Dan Mick> how about this: ls -l /etc/openvpn/client/sepia |
2024-12-19T00:01:05.946Z | <Slava Dubeyko> -rw-------. 1 root root 108 Dec 12 13:43 secret.old |
2024-12-19T00:01:17.122Z | <Dan Mick> ok. so you've lost your secret. |
2024-12-19T00:01:28.060Z | <Dan Mick> (I hope that's not the only file in sepia/) |
2024-12-19T00:01:40.744Z | <Dan Mick> (but you're asserting it's the only one taht could matter) |
2024-12-19T00:01:47.487Z | <Slava Dubeyko> I have several ones |
2024-12-19T00:02:00.480Z | <Dan Mick> show me the full ls -l just in case |
2024-12-19T00:02:23.619Z | <Slava Dubeyko> ls -l /etc/openvpn/client/sepia
total 64
-rw-r--r--. 1 root root 1220 Dec 2 11:23 ca.crt
-rw-r--r--. 1 1104 1104 290 Dec 11 16:09 client.conf
-rwxr-xr-x. 1 1104 1104 1951 Jun 7 2023 new-client
-rw-r--r-- 1 root root 33452 Dec 18 15:36 output.txt
-rw------- 1 vdubeyko1973 vdubeyko1973 109 Dec 18 14:23 secret
-rw-------. 1 root root 108 Dec 12 13:43 secret.old
-rw------- 1 vdubeyko1973 vdubeyko1973 108 Dec 17 18:14 secret.old.2
-r--r-----. 1 1104 1104 636 Mar 22 2015 tlsauth |
2024-12-19T00:03:38.790Z | <Dan Mick> I'm confused that your secret files seem to be 109/108 characters long, too; both of mine are 97 chars |
2024-12-19T00:03:52.336Z | <Dan Mick> what about wc -l secret* |
2024-12-19T00:04:36.866Z | <Slava Dubeyko> wc -l secret*
1 secret
2 secret.old
2 secret.old.2
5 total |
2024-12-19T00:05:06.459Z | <Dan Mick> so it's possible that secret.old.2, which is dated yesterday afternoon, is the right file |
2024-12-19T00:05:16.791Z | <Dan Mick> so mv secret secret.hash |
2024-12-19T00:05:20.803Z | <Slava Dubeyko> It has two lines |
2024-12-19T00:05:24.174Z | <Dan Mick> mv secret.old.2 secret |
2024-12-19T00:05:28.371Z | <Dan Mick> and try connecting again |
2024-12-19T00:05:33.222Z | <Slava Dubeyko> OK |
2024-12-19T00:07:42.532Z | <Slava Dubeyko> I connected |
2024-12-19T00:08:03.557Z | <Slava Dubeyko> ssh [vdubeyko1973@teuthology.front.sepia.ceph.com](mailto:vdubeyko1973@teuthology.front.sepia.ceph.com)
Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-200-generic x86_64)
* Documentation: <https://help.ubuntu.com>
* Management: <https://landscape.canonical.com>
* Support: <https://ubuntu.com/pro>
New release '22.04.5 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Wed Dec 18 02:32:01 2024 from 172.21.0.100 |
2024-12-19T00:08:13.478Z | <Dan Mick> ok. I've changed new-client to try to be clearer about the purpose of those files. |
2024-12-19T00:08:22.197Z | <Dan Mick> DO NOT TOUCH THE FILES IN /etc/openvpn |
2024-12-19T00:08:36.432Z | <Dan Mick> especially secret |
2024-12-19T00:08:42.804Z | <Slava Dubeyko> I have no attention to touch it :) |
2024-12-19T00:08:51.694Z | <Slava Dubeyko> But it didn't work |
2024-12-19T00:09:36.950Z | <Slava Dubeyko> Thanks again. π |
2024-12-19T00:10:09.740Z | <Dan Mick> I get it. Hopefully the new script will be clearer. it also saves the secret and the secret.hash inside a tarball so they're easier to track. |
2024-12-19T00:10:24.500Z | <Slava Dubeyko> Hope too π |
2024-12-19T12:44:11.028Z | <Vallari Agrawal> @David Galloway we do have teuthology weekly usually but it's been mostly cancelled last few times, everyone should be back in Jan :) |
2024-12-19T13:28:17.394Z | <Brett Niver> check with @Zack Cerza |
2024-12-19T13:28:38.397Z | <Brett Niver> Thanks, @Vallari Agrawal! |
2024-12-19T15:04:05.529Z | <Alexander Indenbaum> Hello π It seems the v19.2.0 CentOS 9 Shaman builds are currently broken. Here's an example: <https://shaman.ceph.com/builds/ceph/wip-baum-20241219-v19.2.0-base/59e7a529c4ecfa0738645a8b36ac66181f413664/>
The visible error points to a missing `librados2` RPM, and the issue appears persistent, as it has been reproduced across multiple builds:
```+ /tmp/venv.xVloWDfcV4/bin/chacractl exists binaries/ceph/wip-baum-20241219-v19.2.0-base/59e7a529c4ecfa0738645a8b36ac66181f413664/centos/9/x86_64/flavors/default/librados2-19.2.0-1-g59e7a529-0.el9.x86_64.rpm
[chacractl.api.exists][INFO ] HEAD: <https://2.chacra.ceph.com/binaries/ceph/wip-baum-20241219-v19.2.0-base/59e7a529c4ecfa0738645a8b36ac66181f413664/centos/9/x86_64/flavors/default/librados2-19.2.0-1-g59e7a529-0.el9.x86_64.rpm/>
[urllib3.connectionpool][DEBUG ] Starting new HTTPS connection (1): [2.chacra.ceph.com:443](http://2.chacra.ceph.com:443)
[urllib3.connectionpool][DEBUG ] <https://2.chacra.ceph.com:443> "HEAD /binaries/ceph/wip-baum-20241219-v19.2.0-base/59e7a529c4ecfa0738645a8b36ac66181f413664/centos/9/x86_64/flavors/default/librados2-19.2.0-1-g59e7a529-0.el9.x86_64.rpm/ HTTP/1.1" 404 0
[chacractl][ERROR ] RuntimeError: HEAD for <https://2.chacra.ceph.com/binaries/ceph/wip-baum-20241219-v19.2.0-base/59e7a529c4ecfa0738645a8b36ac66181f413664/centos/9/x86_64/flavors/default/librados2-19.2.0-1-g59e7a529-0.el9.x86_64.rpm/> failed: 404 resource does not exist```
Any idea? |
2024-12-19T16:06:10.208Z | <John Mulligan> I see a big error at the bottom of the jenkins log:
```/tmp/jenkins11583598016093059341.sh: line 2076: cd: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/dist/ceph-19.2.0-1-g59e7a529/container: No such file or directory``` |
2024-12-19T16:06:56.282Z | <John Mulligan> is your base branch (main,squid,etc) prehaps a bit old? Dan converted the container images to build from the ceph tree fairly recently, and if your base is old you may not have that dir |
2024-12-19T16:07:04.742Z | <John Mulligan> then ceph build scripts won't find it and may fail |
2024-12-19T16:08:42.401Z | <Alexander Indenbaum> @John Mulligan thank you for explanation. Any procedure to build v19.2.0 build on shaman? |
2024-12-19T16:08:46.504Z | <John Mulligan> if this is the correct branch <https://github.com/ceph/ceph-ci/tree/wip-baum-20241219-v19.2.0-base> then yeah, it doesn't have a container dir |
2024-12-19T16:09:08.710Z | <John Mulligan> oh, so you are trying to build that specific tag? |
2024-12-19T16:09:45.837Z | <Alexander Indenbaum> We need to generate v19.2.0 as a preparation for 8.1 downstream release. |
2024-12-19T16:10:02.300Z | <Alexander Indenbaum> We need to generate v19.2.0 based build as a preparation for 8.1 downstream release. |
2024-12-19T16:10:43.985Z | <John Mulligan> interesting. I think we need to ping @Dan Mick then |
2024-12-19T16:11:18.704Z | <John Mulligan> Dan might have a better idea on how to build an older branch/tag (maybe force it to use and older checkout of ceph-build?) |
2024-12-19T16:32:19.021Z | <Ronen Friedman> Thanks for the help in saving the data there! |
2024-12-19T17:05:42.133Z | <Laura Flores> @Dan Mick @David Galloway @Adam Kraitman @John Mulligan @yuriw I may be a bit late to the infrastructure meeting today. Feel free to start without me and I will join when I can! |
2024-12-19T17:13:15.186Z | <David Galloway> I don't have anything new compared to last week and I don't see any agenda items. Maybe we can cancel? |
2024-12-19T17:14:07.657Z | <yuriw> Cool with me |
2024-12-19T17:14:19.167Z | <Dan Mick> 1) builds happen on Jenkins. 2) every package gets a 404 from chacra, it's a part of the build to check for existence before potentially overwriting. 3) yes, ceph-build now requires the container subdir. |
2024-12-19T17:17:37.137Z | <Dan Mick> Has anyone resolved the issue with the LVM setup yet? I sent a detailed message to Patrick but he was on PTO at the time |
2024-12-19T17:18:26.990Z | <Adam Kraitman> Also fine by me, I just have one update, irvingi01 is now running noble ubuntu so we can use it to test unitests or anything we choose |
2024-12-19T17:19:56.415Z | <Dan Mick> @Adam Kraitman did you figure out an autoinstall or did you manually install? |
2024-12-19T17:26:11.739Z | <Adam Kraitman> I manually upgraded it, added a noble tag and removed the jammy tags so we will only use it for testing and so that other builds will not trigger on this builder until we finish testing |
2024-12-19T17:27:31.033Z | <Eric I> Our current fix is to consult <https://www.apache.org/dyn/closer.cgi> to get the closest mirror. These are versions of kafka and maven that we donβt update often. If this solution proves to be robust over time, I think we can leave it. But if we do hit other issues then we may want to explore using [download.ceph.com](http://download.ceph.com). I just wanted to know who weβd reach out to if we wanted to use this option. I appreciate the insight, Dan! |
2024-12-19T17:29:08.702Z | <Dan Mick> Ah. Makes sense. |
2024-12-19T17:29:45.629Z | <Dan Mick> Don't forget to.updateansible |
2024-12-19T17:34:23.732Z | <Slava Dubeyko> I can access teutology but I cannot access testnodes. |
2024-12-19T17:34:27.223Z | <Slava Dubeyko> vdubeyko1973@teuthology:~$ ssh [folio13.front.sepia.ceph.com](http://folio13.front.sepia.ceph.com)
The authenticity of host '[folio13.front.sepia.ceph.com](http://folio13.front.sepia.ceph.com) (172.21.5.163)' can't be established.
ECDSA key fingerprint is SHA256:emMdba3jiSHy18WZf1xai7z/F1U13CcRvknh9SFR4Q8.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[folio13.front.sepia.ceph.com](http://folio13.front.sepia.ceph.com),172.21.5.163' (ECDSA) to the list of known hosts.
[vdubeyko1973@folio13.front.sepia.ceph.com](mailto:vdubeyko1973@folio13.front.sepia.ceph.com)'s password: |
2024-12-19T17:34:48.378Z | <Slava Dubeyko> Is it something I need to do or something is wrong? |
2024-12-19T17:37:43.548Z | <Dan Mick> That's ssh StrictHostKeyChecking |
2024-12-19T17:38:05.653Z | <Dan Mick> Search the wiki for that term for advice |
2024-12-19T17:43:39.835Z | <Adam Kraitman> One build that ran on the noble version before I updated the tags <https://jenkins.ceph.com/job/ceph-pr-docs/110914/> |
2024-12-19T17:45:25.127Z | <Alexander Indenbaum> @Dan Mick, thank you! Will v19.2.1 include the `container` directory? I'll try merging the `container` directory commits into the branch based on v19.2.0. Do you think itβs expected to work? π
```git cherry-pick 5c40a5c1737cf0628c899c4e9f788017b8b4fca8
git cherry-pick 7e03ee798f4ed3aa4a0bb1a9e9d62df52e54406d
git cherry-pick fb4ec711176d5f87c3ab7d02d64d8e16dea58313
git cherry-pick 44b2fc4d5e1fd909370018ca9ec210686f98cefa
git cherry-pick 6c8e9984253a61c261cfe6e0981dda58895500ae
git cherry-pick 7374b9b1114da08f1fdb9dfcfe2ef37c2178c8f5
git cherry-pick 4f050d7811fa6ec347c3414d5749e0f86480d7f4
git cherry-pick a3a9f188cabf714de2c39e310bccbe848ed66671
git cherry-pick ce74f14de5042774b8386b0eff04497b517eaacc
git cherry-pick 08432ef54212aa2aba724102854082e76f0cd847
git cherry-pick 18d14b22f31c65ad3db79c7930a841157b68deee
git cherry-pick 4b3c0cb8f3f130563d9e1309beb7a8e0b59ea83e``` |
2024-12-19T17:49:29.077Z | <Alexander Indenbaum> ceph-ci branch: <https://github.com/ceph/ceph-ci/tree/wip-baum-20241219-v19.2.0-container>
shaman: <https://shaman.ceph.com/builds/ceph/wip-baum-20241219-v19.2.0-container/72f9ae00b44be68d05a8d0ead0f062780b19aa49/> |
2024-12-19T17:56:07.906Z | <John Mulligan> I would have missed (the start) anyway as I didn't know it was moved to weekly already. |
2024-12-19T18:01:25.573Z | <David Galloway> Just weekly for now during the migration planning |
2024-12-19T18:28:14.191Z | <Matan Breizman> I might also need 19.2.0 upstream images. @Alexander Indenbaum thanks for looking into this. |
2024-12-19T18:28:50.583Z | <Dan Mick> I would expect it to work, particularly if you get them from the squid branch |
2024-12-19T18:29:36.585Z | <Matan Breizman> then perhaps the name should include "squid" as well? |
2024-12-19T18:29:43.464Z | <Matan Breizman> then perhaps the branch name should include "squid" as well? |
2024-12-19T18:29:45.906Z | <Dan Mick> Container is completely separate from ceph mainline code and only affects the construction of the container |
2024-12-19T18:30:34.338Z | <John Mulligan> can I ask a silly question: why are you trying to build a new v19.2.0? What is missing from the builds at <https://quay.io/repository/ceph/ceph?tab=tags&tag=latest> Or <https://download.ceph.com/rpm-19.2.0/> ? |
2024-12-19T18:30:54.621Z | <Dan Mick> Afaik nothing changes on the branch name anymore...the differences are in the different ceph.git branch code itself |
2024-12-19T18:33:31.354Z | <Jesse Williamson> Hi, folks! I'm trying to set up Teuthology and when I try to get the VPN package from "<http://ceph.com/sage/sepia-vpn-client.tar.gz>" (as per "<https://ceph.github.io/sepia/adding_users/#setting-up-vpn-client>"), it looks like there's no file there. Has it moved? |
2024-12-19T18:37:36.699Z | <John Mulligan> those instructions may be out of date |
2024-12-19T18:37:39.229Z | <John Mulligan> maybe <https://wiki.sepia.ceph.com/doku.php?id=vpnaccess&s[]=vpn> ? |
2024-12-19T18:45:14.723Z | <Dan Mick> Yes. The tarball is on file dump now and the wiki has the correct instructions. We should update or purge those [GitHub.io](http://GitHub.io) docs |
2024-12-19T18:45:33.540Z | <Dan Mick> Yes. The tarball is on filedump now and the wiki has the correct instructions. We should update or purge those [GitHub.io](http://GitHub.io) docs |
2024-12-19T18:49:15.172Z | <Jesse Williamson> Thanks, appreciated! |
2024-12-19T19:02:52.636Z | <Slava Dubeyko> OK. Let me try. Thanks |
2024-12-19T19:17:54.496Z | <Dan Mick> ah. in fact, see <https://ceph.github.io/sepia/> |
2024-12-19T19:18:20.610Z | <Dan Mick> I expect you got a deep link from a search? |
2024-12-19T19:24:15.963Z | <Dan Mick> and yes, 19.2.1 should include the container directory, as it will be based on the current squid branch |
2024-12-19T19:42:18.894Z | <Slava Dubeyko> I have config file on my laptop |
2024-12-19T19:42:24.457Z | <Slava Dubeyko> cat ~/.ssh/config
host [teuthology.front.sepia.ceph.com](http://teuthology.front.sepia.ceph.com)
HostName [teuthology.front.sepia.ceph.com](http://teuthology.front.sepia.ceph.com)
User vdubeyko1973
IdentityFile ~/.ssh/id_ed25519
ForwardAgent yes
host folio13
HostName [folio13.front.sepia.ceph.com](http://folio13.front.sepia.ceph.com)
User vdubeyko1973
IdentityFile ~/.ssh/id_ed25519
ForwardAgent yes
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null |
2024-12-19T19:43:08.077Z | <Slava Dubeyko> But I still cannot ssh into folio13 because it asks for password |
2024-12-19T19:43:31.395Z | <Slava Dubeyko> Is it something I did wrong? |
2024-12-19T19:44:48.558Z | <Slava Dubeyko> vdubeyko1973@teuthology:~$ ssh [vdubeyko1973@folio13.front.sepia.ceph.com](mailto:vdubeyko1973@folio13.front.sepia.ceph.com)
[vdubeyko1973@folio13.front.sepia.ceph.com](mailto:vdubeyko1973@folio13.front.sepia.ceph.com)'s password: |
2024-12-19T20:11:23.329Z | <David Galloway> Yeah, where did you find that [github.io](http://github.io) link? We should really update that to point to the actual wiki. |
2024-12-19T20:49:44.051Z | <Zack Cerza> Yes I was out but I am back now. We could do a one- off meeting tomorrow if there is interest |
2024-12-19T20:53:08.654Z | <Zack Cerza> I've done a lot with containerized builds as well but have not integrated it into the new build pipeline yet. Happy to start coordinating sometime soon; the regular build meeting has been on Wednesdays but since the next two are holidays we could do a one-off |
2024-12-19T20:53:46.136Z | <Dan Mick> why would you think you have an account on folio13? |
2024-12-19T20:54:41.999Z | <Dan Mick> @David Galloway did you look at the link I posted? Honestly I think that subtree should just go away entirely |
2024-12-19T20:55:12.430Z | <Slava Dubeyko> I need access to testnodes too and I assumed that it was granted. Should I ask it additionally? |
2024-12-19T20:55:21.459Z | <Dan Mick> folio13 is not a testnode |
2024-12-19T20:56:08.062Z | <Dan Mick> again, why are you connecting to folio13? |
2024-12-19T20:56:42.150Z | <David Galloway> Agree. |
2024-12-19T20:57:47.733Z | <David Galloway> If I may, I would just ask to make the pipeline as tool-agnostic as possible. I'd love to stop using Jenkins at some point. |
2024-12-19T20:58:33.787Z | <David Galloway> (I understand that is literally the point of containers but would hate to see weeks of development work put into jamming Ceph builds into containers specifically so Jenkins looks happy) |
2024-12-19T20:58:44.972Z | <Slava Dubeyko> Because, my colleague Alex Markuze said to me that we are working on this machine. If it is not testnode, then what is it? π |
2024-12-19T20:59:34.370Z | <Dan Mick> <http://wiki.front.sepia.ceph.com/doku.php?id=devplayground> |
2024-12-19T20:59:40.134Z | <Dan Mick> let's start a conversation with Alex |
2024-12-19T20:59:49.556Z | <Slava Dubeyko> What is testnode then? |
2024-12-19T20:59:59.787Z | <Dan Mick> <http://wiki.front.sepia.ceph.com/doku.php?id=testnodeaccess> |
2024-12-19T21:00:28.734Z | <Dan Mick> where teuthology tests are run |
2024-12-19T21:00:40.610Z | <Slava Dubeyko> Maybe, I missed something. Let me double check with Alex |
2024-12-19T21:00:49.005Z | <Slava Dubeyko> Thanks π |
2024-12-19T21:01:06.434Z | <Dan Mick> I'm not even sure why [ceph.github.io](http://ceph.github.io) as a site exists |
2024-12-19T21:01:21.107Z | <Dan Mick> we have [docs.ceph.com](http://docs.ceph.com) and READMEs in ceph.git |
2024-12-19T21:01:30.217Z | <Zack Cerza> CI pipelines tend to have to use features of the CI systems to be able to e.g. split up different build phases, so i've been using a Jenkinsfile as the implementation |
2024-12-19T21:02:14.183Z | <Dan Mick> Let's start from the beginning: what are you working on? |
2024-12-19T21:02:38.858Z | <Slava Dubeyko> I am working on Linux kernel client of CephFS |
2024-12-19T21:03:08.648Z | <Zack Cerza> I would very much like to get away from Jenkins as well though, so I'm implementing it with migration in mind. trying to keep the real work in scripts that the pipeline calls out to, and modifying some of our existing `build_utils.sh` bits in the process |
2024-12-19T21:03:30.389Z | <Dan Mick> ok. so a 'playground' machine (like folio13), as the wiki suggests, would be used for building the Ceph code outside the normal CI builds |
2024-12-19T21:03:49.895Z | <Joseph Mundackal> βοΈ will also help people test things locally |
2024-12-19T21:04:14.239Z | <Dan Mick> do you need a build host (that is, do the hosts you use for your personal IBM work not support building Ceph)? |
2024-12-19T21:04:59.756Z | <Slava Dubeyko> I need to build as Ceph as Linux kernel for testing modifications in Linux kernel. |
2024-12-19T21:05:16.689Z | <Dan Mick> yes, obviously, I understand, but that wasn't the question |
2024-12-19T21:05:33.403Z | <David Galloway> Yeah, I didn't mean Jenkins was the only reason to containerize builds (if you're replying to my message). Just that doing the builds in a container means the process is more portable. |
2024-12-19T21:05:42.754Z | <Slava Dubeyko> Do you mean my laptop? |
2024-12-19T21:07:04.422Z | <Slava Dubeyko> I am trying to use my laptop for building and testing but it is very restrictive way |
2024-12-19T21:07:24.569Z | <Slava Dubeyko> Even 32GB of memory is not much |
2024-12-19T21:10:04.619Z | <Zack Cerza> yeah I understand. I would really like to arrive at a point where we have a pipeline we can run with different stages and sub-stages that are individually re-runnable, but using a Containerfile that can also yield artifacts that are functionally identical when run in one shot locally. don't want to end up re-implementing ceph-container |
2024-12-19T21:10:52.938Z | <Zack Cerza> or, if we can't achieve all that using a plain Containerfile we may want to consider something like <https://dagger.io/> |
2024-12-19T21:12:39.207Z | <Dan Mick> ok. so, people in your situation often need access to a build host, which is why we have a set of playground hosts to use. But yes, you're not given accounts on all of them by default; it's by arrangement with the infra team (me and my coworkers) |
2024-12-19T21:13:28.967Z | <Zack Cerza> a thing I get a little nervous about is that pretty much every open source CI system is developed by a company that has a managed cloud offering, and I don't want to end up in a situation where the self-hosted version gets hobbled over time. but probably just have to accept that risk and choose a replacement carefully |
2024-12-19T21:13:31.497Z | <Dan Mick> and, strictly speaking, these are provided for upstream development, although that's where all the initial development for Ceph is meant to happen |
2024-12-19T21:14:07.134Z | <Slava Dubeyko> So, what should I do? |
2024-12-19T21:15:13.619Z | <David Galloway> We might have to use a hosted version of $CI anyway. I can't get a clear answer from IBM about whether we'll have public IPs. |
2024-12-19T21:15:20.479Z | <David Galloway> We might have to use a hosted version of $CI anyway. I can't get a clear answer from IBM about whether we'll have public IPs in POK. |
2024-12-19T21:19:25.415Z | <Dan Mick> let me ask the larger question of my management about who and what we provide build hosts to. It could be that it's fine, but I want to understand what happens when we run out of resources |
2024-12-19T21:20:41.694Z | <Slava Dubeyko> OK. Sounds good. Thanks |