ceph - sepia - 2024-12-18

Timestamp (UTC)Message
2024-12-18T01:32:35.178Z
<Dan Mick> @Slava Dubeyko: it looks like @fernando.alcocer.ocho pushed your new credentials last Friday.  Are you still having trouble connecting, or have you not tried it since then?
2024-12-18T01:38:04.555Z
<Slava Dubeyko> As far as I can see, authentication is failed:
2024-12-18T01:38:08.169Z
<Dan Mick> I see the auth failing, yes.
2024-12-18T01:38:10.157Z
<Slava Dubeyko> sudo openvpn --config /etc/openvpn/client/sepia.conf --cd /etc/openvpn/client --verb 5
2024-12-17 17:36:54 WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.
2024-12-17 17:36:54 us=989116 Note: --cipher is not set. OpenVPN versions before 2.5 defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
2024-12-17 17:36:54 us=989178 Note: '--allow-compression' is not set to 'no', disabling data channel offload.
2024-12-17 17:36:54 us=989199 Current Parameter Settings:
2024-12-17 17:36:54 us=989203   config = '/etc/openvpn/client/sepia.conf'
2024-12-17 17:36:54 us=989207   mode = 0
2024-12-17 17:36:54 us=989212 NOTE: --mute triggered...
2024-12-17 17:36:54 us=989222 290 variation(s) on previous 10 message(s) suppressed by --mute
2024-12-17 17:36:54 us=989226 OpenVPN 2.6.12 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
2024-12-17 17:36:54 us=989234 library versions: OpenSSL 3.2.2 4 Jun 2024, LZO 2.10
2024-12-17 17:36:54 us=989241 DCO version: N/A
2024-12-17 17:36:54 us=990013 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
2024-12-17 17:36:54 us=990026 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
2024-12-17 17:36:54 us=990036 LZO compression initializing
2024-12-17 17:36:54 us=990122 Control Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1250 tun_max_mtu:0 headroom:126 payload:1600 tailroom:126 ET:0 ]
2024-12-17 17:36:54 us=998553 Data Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
2024-12-17 17:36:55 us=128 TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
2024-12-17 17:36:55 us=152 Socket Buffers: R=[212992->212992] S=[212992->212992]
2024-12-17 17:36:55 us=162 UDPv4 link local: (not bound)
2024-12-17 17:36:55 us=168 UDPv4 link remote: [AF_INET]8.43.84.129:1194
2024-12-17 17:36:55 us=209 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
WR2024-12-17 17:36:55 us=83763 TLS: Initial packet from [AF_INET]8.43.84.129:1194, sid=200b0cc4 59ff59f0
2024-12-17 17:36:55 us=83810 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
WRRWR2024-12-17 17:36:55 us=167706 VERIFY OK: depth=1, CN=openvpnca-sepia
2024-12-17 17:36:55 us=167846 VERIFY KU OK
2024-12-17 17:36:55 us=167856 Validating certificate extended key usage
2024-12-17 17:36:55 us=167862 NOTE: --mute triggered...
WRWRWWRR2024-12-17 17:36:58 us=587642 4 variation(s) on previous 10 message(s) suppressed by --mute
2024-12-17 17:36:58 us=587673 [[gw.sepia.ceph.com](http://gw.sepia.ceph.com)] Peer Connection Initiated with [AF_INET]8.43.84.129:1194
2024-12-17 17:36:58 us=587689 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1
2024-12-17 17:36:58 us=587746 TLS: tls_multi_process: initial untrusted session promoted to trusted
W2024-12-17 17:36:59 us=662261 SENT CONTROL [[gw.sepia.ceph.com](http://gw.sepia.ceph.com)]: 'PUSH_REQUEST' (status=1)
WRR2024-12-17 17:36:59 us=743959 AUTH: Received control message: AUTH_FAILED
2024-12-17 17:36:59 us=744151 TCP/UDP: Closing socket
2024-12-17 17:36:59 us=744192 SIGTERM[soft,auth-failure] received, process exiting
2024-12-18T01:38:24.344Z
<Dan Mick> You're using the secret that corresponds to the last thing in the issue?
2024-12-18T01:38:39.407Z
<Dan Mick> (to the last hash in the issue?)
2024-12-18T01:39:28.454Z
<Dan Mick> (when you run the script it generates a "secret" file and a "hashed secret"; the latter is what you send in for placement on the server)
2024-12-18T01:41:44.356Z
<Dan Mick> @Slava Dubeyko ^
2024-12-18T01:42:23.722Z
<Slava Dubeyko> My current secret corresponds to the latest one in the ticket
2024-12-18T01:43:54.285Z
<Dan Mick> ah, I see it, a space crept into the middle of the string. fixing.
2024-12-18T01:44:29.070Z
<Slava Dubeyko> OK. Thanks
2024-12-18T01:44:33.591Z
<Dan Mick> can you try again now?
2024-12-18T01:45:23.306Z
<Slava Dubeyko> WRR2024-12-17 17:45:11 us=111890 AUTH: Received control message: AUTH_FAILED
2024-12-17 17:45:11 us=112029 TCP/UDP: Closing socket
2024-12-17 17:45:11 us=112055 SIGTERM[soft,auth-failure] received, process exiting
2024-12-18T01:45:53.618Z
<Dan Mick> that's odd, I didn't see any attempt at all
2024-12-18T01:46:00.171Z
<Dan Mick> (I did before)
2024-12-18T01:46:05.711Z
<Dan Mick> are you completely restarting the service?
2024-12-18T01:46:33.522Z
<Slava Dubeyko> What is the command to completely restart?
2024-12-18T01:46:41.668Z
<Dan Mick> what is your OS?
2024-12-18T01:46:47.989Z
<Slava Dubeyko> Fedora 41
2024-12-18T01:47:14.858Z
<Dan Mick> systemctl restart <servicename> should.  systemctl stop/systemctl start definitely will
2024-12-18T01:47:35.770Z
<Dan Mick> servicename is probably openvpn-client@<something>
2024-12-18T01:47:44.523Z
<Slava Dubeyko> OK. Let me try
2024-12-18T01:50:26.757Z
<Dan Mick> watching, waiting
2024-12-18T01:51:44.717Z
<Dan Mick> if you're having trouble finding the service, it's possible you didn't create a service, maybe?...
2024-12-18T01:51:55.817Z
<Slava Dubeyko> sudo systemctl status openvpn-client@sepia
β—‹ openvpn-client@sepia.service - OpenVPN tunnel for sepia
     Loaded: loaded (/usr/lib/systemd/system/openvpn-client@.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf, 50-keep-warm.conf
     Active: inactive (dead) since Tue 2024-12-17 17:49:51 PST; 48s ago
   Duration: 2.665s
 Invocation: a736997fec18415fa099fd6fa2c2b29d
       Docs: man:openvpn(8)
             <https://openvpn.net/community-resources/reference-manual-for-openvpn-2-6/>
             <https://community.openvpn.net/openvpn/wiki/HOWTO>
    Process: 518946 ExecStart=/usr/sbin/openvpn --suppress-timestamps --nobind --config sepia.conf (code=exite>
   Main PID: 518946 (code=exited, status=0/SUCCESS)
     Status: "Pre-connection initialization successful"
   Mem peak: 2.1M
        CPU: 14ms

Dec 17 17:49:49 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[518946]: WARNING: this configuration m>
Dec 17 17:49:49 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[518946]: VERIFY OK: depth=1, CN=openvp>
Dec 17 17:49:49 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[518946]: VERIFY KU OK
Dec 17 17:49:49 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[518946]: Validating certificate extend>
Dec 17 17:49:49 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[518946]: NOTE: --mute triggered...
Dec 17 17:49:50 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[518946]: 4 variation(s) on previous 10>
Dec 17 17:49:50 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[518946]: [[gw.sepia.ceph.com](http://gw.sepia.ceph.com)] Peer Conn>
Dec 17 17:49:51 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[518946]: AUTH: Received control messag>
Dec 17 17:49:51 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[518946]: SIGTERM[soft,auth-failure] re>
Dec 17 17:49:51 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sepia.service: Deac>
lines 1-26/26 (END)
2024-12-18T01:52:17.645Z
<Slava Dubeyko> WRR2024-12-17 17:52:08 us=283012 AUTH: Received control message: AUTH_FAILED
2024-12-17 17:52:08 us=283106 TCP/UDP: Closing socket
2024-12-17 17:52:08 us=283119 SIGTERM[soft,auth-failure] received, process exiting
2024-12-18T01:54:10.061Z
<Dan Mick> I don't understand why I'm not even seeing a connection attempt.  What is your external IP (guessing 162.197.212.70)?
2024-12-18T01:55:29.846Z
<Slava Dubeyko> Can I see it by ifconfig?
2024-12-18T01:56:31.109Z
<Dan Mick> depends on if you're behind a home router or something else.  Probably easiest to use something like [whatsmyip.com](http://whatsmyip.com)
2024-12-18T01:57:53.213Z
<Slava Dubeyko> 2600:1700:6476:1430::23
2024-12-18T01:58:03.823Z
<Slava Dubeyko> I can see this πŸ™‚
2024-12-18T01:58:31.641Z
<Dan Mick> that's what [whatsmyip.com](http://whatsmyip.com) says?  it doesn't have an IPv4 address?
2024-12-18T01:59:15.965Z
<Slava Dubeyko> Your Public IPv4:
Your IPv4: Not Detected


IPv6:
2600:1700:6476:1430::23
2024-12-18T02:00:18.268Z
<Slava Dubeyko> Maybe, it's IPv6 issue?
2024-12-18T02:00:22.586Z
<Dan Mick> I...hm, that's new.
2024-12-18T02:01:35.228Z
<Dan Mick> can you try "ping 8.43.84.129" please
2024-12-18T02:02:07.304Z
<Slava Dubeyko> ping 8.43.84.129
PING 8.43.84.129 (8.43.84.129) 56(84) bytes of data.
64 bytes from 8.43.84.129: icmp_seq=1 ttl=48 time=86.7 ms
64 bytes from 8.43.84.129: icmp_seq=2 ttl=48 time=86.7 ms
64 bytes from 8.43.84.129: icmp_seq=3 ttl=48 time=85.2 ms
^C
--- 8.43.84.129 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 85.197/86.213/86.724/0.718 ms
2024-12-18T02:02:18.738Z
<Dan Mick> I got two pings, appaerntly from 162.197.212.70
2024-12-18T02:02:44.735Z
<Dan Mick> so I'm gonna say whatever it is you're passing through is using that v4 address.  Ok, let me watch for that while you try an openvpn connect
2024-12-18T02:02:52.679Z
<Dan Mick> try that now, please
2024-12-18T02:03:13.544Z
<Slava Dubeyko> OK
2024-12-18T02:04:04.065Z
<Slava Dubeyko> I did
2024-12-18T02:04:08.725Z
<Dan Mick> ok, definitely seeing your packets.  let me look at the openvpn log again
2024-12-18T02:04:42.317Z
<Slava Dubeyko> sudo openvpn --config /etc/openvpn/client/sepia.conf --cd /etc/openvpn/client --verb 5
2024-12-17 18:03:47 WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.
2024-12-17 18:03:47 us=369684 Note: --cipher is not set. OpenVPN versions before 2.5 defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
2024-12-17 18:03:47 us=369703 Note: '--allow-compression' is not set to 'no', disabling data channel offload.
2024-12-17 18:03:47 us=369713 Current Parameter Settings:
2024-12-17 18:03:47 us=369716   config = '/etc/openvpn/client/sepia.conf'
2024-12-17 18:03:47 us=369719   mode = 0
2024-12-17 18:03:47 us=369721 NOTE: --mute triggered...
2024-12-17 18:03:47 us=369728 290 variation(s) on previous 10 message(s) suppressed by --mute
2024-12-17 18:03:47 us=369734 OpenVPN 2.6.12 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
2024-12-17 18:03:47 us=369741 library versions: OpenSSL 3.2.2 4 Jun 2024, LZO 2.10
2024-12-17 18:03:47 us=369748 DCO version: N/A
2024-12-17 18:03:47 us=370281 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
2024-12-17 18:03:47 us=370290 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
2024-12-17 18:03:47 us=370297 LZO compression initializing
2024-12-17 18:03:47 us=370361 Control Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1250 tun_max_mtu:0 headroom:126 payload:1600 tailroom:126 ET:0 ]
2024-12-17 18:03:47 us=377158 Data Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
2024-12-17 18:03:47 us=377941 TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
2024-12-17 18:03:47 us=377952 Socket Buffers: R=[212992->212992] S=[212992->212992]
2024-12-17 18:03:47 us=377956 UDPv4 link local: (not bound)
2024-12-17 18:03:47 us=377959 UDPv4 link remote: [AF_INET]8.43.84.129:1194
2024-12-17 18:03:47 us=377969 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
WR2024-12-17 18:03:47 us=461851 TLS: Initial packet from [AF_INET]8.43.84.129:1194, sid=b00aa009 4f228c0d
2024-12-17 18:03:47 us=461909 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
WRRWR2024-12-17 18:03:47 us=547641 VERIFY OK: depth=1, CN=openvpnca-sepia
2024-12-17 18:03:47 us=547851 VERIFY KU OK
2024-12-17 18:03:47 us=547864 Validating certificate extended key usage
2024-12-17 18:03:47 us=547872 NOTE: --mute triggered...
WRWRWR2024-12-17 18:03:48 us=676116 4 variation(s) on previous 10 message(s) suppressed by --mute
2024-12-17 18:03:48 us=676151 [[gw.sepia.ceph.com](http://gw.sepia.ceph.com)] Peer Connection Initiated with [AF_INET]8.43.84.129:1194
2024-12-17 18:03:48 us=676168 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1
2024-12-17 18:03:48 us=676248 TLS: tls_multi_process: initial untrusted session promoted to trusted
W2024-12-17 18:03:49 us=721440 SENT CONTROL [[gw.sepia.ceph.com](http://gw.sepia.ceph.com)]: 'PUSH_REQUEST' (status=1)
WRR2024-12-17 18:03:49 us=808373 AUTH: Received control message: AUTH_FAILED
2024-12-17 18:03:49 us=808557 TCP/UDP: Closing socket
2024-12-17 18:03:49 us=808586 SIGTERM[soft,auth-failure] received, process exiting
2024-12-18T02:06:34.461Z
<Dan Mick> argh.  sorry, my error.  that space was supposed to be there and I broke it worse by adding it
2024-12-18T02:06:42.693Z
<Dan Mick> so let's put that back in and I'll watch again
2024-12-18T02:07:00.034Z
<Slava Dubeyko> OK
2024-12-18T02:08:25.377Z
<Dan Mick> ok sorry.  once more please.
2024-12-18T02:09:01.040Z
<Slava Dubeyko> Just a minute
2024-12-18T02:09:42.965Z
<Slava Dubeyko> sudo openvpn --config /etc/openvpn/client/sepia.conf --cd /etc/openvpn/client --verb 5
2024-12-17 18:09:22 WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.
2024-12-17 18:09:22 us=237206 Note: --cipher is not set. OpenVPN versions before 2.5 defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
2024-12-17 18:09:22 us=237227 Note: '--allow-compression' is not set to 'no', disabling data channel offload.
2024-12-17 18:09:22 us=237238 Current Parameter Settings:
2024-12-17 18:09:22 us=237241   config = '/etc/openvpn/client/sepia.conf'
2024-12-17 18:09:22 us=237243   mode = 0
2024-12-17 18:09:22 us=237245 NOTE: --mute triggered...
2024-12-17 18:09:22 us=237253 290 variation(s) on previous 10 message(s) suppressed by --mute
2024-12-17 18:09:22 us=237261 OpenVPN 2.6.12 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
2024-12-17 18:09:22 us=237269 library versions: OpenSSL 3.2.2 4 Jun 2024, LZO 2.10
2024-12-17 18:09:22 us=237276 DCO version: N/A
2024-12-17 18:09:22 us=237852 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
2024-12-17 18:09:22 us=237860 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
2024-12-17 18:09:22 us=237868 LZO compression initializing
2024-12-17 18:09:22 us=237934 Control Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1250 tun_max_mtu:0 headroom:126 payload:1600 tailroom:126 ET:0 ]
2024-12-17 18:09:22 us=240757 Data Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
2024-12-17 18:09:22 us=241500 TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
2024-12-17 18:09:22 us=241512 Socket Buffers: R=[212992->212992] S=[212992->212992]
2024-12-17 18:09:22 us=241517 UDPv4 link local: (not bound)
2024-12-17 18:09:22 us=241520 UDPv4 link remote: [AF_INET]8.43.84.129:1194
2024-12-17 18:09:22 us=241532 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
WR2024-12-17 18:09:22 us=324805 TLS: Initial packet from [AF_INET]8.43.84.129:1194, sid=98f43616 b0cb295a
2024-12-17 18:09:22 us=324872 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
WRRWR2024-12-17 18:09:22 us=409226 VERIFY OK: depth=1, CN=openvpnca-sepia
2024-12-17 18:09:22 us=409437 VERIFY KU OK
2024-12-17 18:09:22 us=409450 Validating certificate extended key usage
2024-12-17 18:09:22 us=409458 NOTE: --mute triggered...
WRWRWR2024-12-17 18:09:23 us=540186 4 variation(s) on previous 10 message(s) suppressed by --mute
2024-12-17 18:09:23 us=540253 [[gw.sepia.ceph.com](http://gw.sepia.ceph.com)] Peer Connection Initiated with [AF_INET]8.43.84.129:1194
2024-12-17 18:09:23 us=540278 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1
2024-12-17 18:09:23 us=540357 TLS: tls_multi_process: initial untrusted session promoted to trusted
W2024-12-17 18:09:24 us=595535 SENT CONTROL [[gw.sepia.ceph.com](http://gw.sepia.ceph.com)]: 'PUSH_REQUEST' (status=1)
WRR2024-12-17 18:09:24 us=678287 AUTH: Received control message: AUTH_FAILED
2024-12-17 18:09:24 us=678513 TCP/UDP: Closing socket
2024-12-17 18:09:24 us=678557 SIGTERM[soft,auth-failure] received, process exiting
2024-12-18T02:10:01.427Z
<Dan Mick> yeah, it really thinks that auth is invalid
2024-12-18T02:10:21.639Z
<Dan Mick> your secret file must mismatch.  we'll just have to generate it again.
2024-12-18T02:10:36.247Z
<Dan Mick> let me review the instructions briefly
2024-12-18T02:11:16.835Z
<Dan Mick> so make a completely new directory somewhere
2024-12-18T02:11:30.366Z
<Dan Mick> and just run the sepia/new-client script there
2024-12-18T02:12:03.360Z
<Slava Dubeyko> OK. Let me do it
2024-12-18T02:15:20.411Z
<Slava Dubeyko> OK. Done. Should I share a new secret?
2024-12-18T02:17:05.108Z
<Dan Mick> cut and paste the line that says "please paste" into the tracker ticket (<https://tracker.ceph.com/issues/69166>)
2024-12-18T02:19:27.769Z
<Slava Dubeyko> Done. Ticket contains the new secret.
2024-12-18T02:20:24.379Z
<Dan Mick> so then cd /etc/openvpn/sepia
2024-12-18T02:20:28.818Z
<Dan Mick> mv secret secret.old
2024-12-18T02:20:45.101Z
<Dan Mick> mv <the secret file you just created elsewhere> secret
2024-12-18T02:22:35.012Z
<Dan Mick> (we're saving the old actual secret, the file named "secret", and putting in the new one that we know for certain corresponds to the thing you just pasted in the ticket)
2024-12-18T02:23:22.903Z
<Slava Dubeyko> Done
2024-12-18T02:23:48.129Z
<Dan Mick> so ls -l /etc/openvpn/sepia/secret
2024-12-18T02:25:09.530Z
<Slava Dubeyko> ls -l /etc/openvpn/client/sepia/secret
-rw------- 1 vdubeyko1973 vdubeyko1973 108 Dec 17 18:14 /etc/openvpn/client/sepia/secret
2024-12-18T02:25:37.674Z
<Dan Mick> is your current time that far off?  I have 18:25
2024-12-18T02:25:52.073Z
<Slava Dubeyko> I have the same πŸ™‚
2024-12-18T02:26:07.327Z
<Dan Mick> oh, but you mv'ed the file.  never minde.
2024-12-18T02:26:56.516Z
<Dan Mick> so all of your files are in /etc/openvpn/client because Fedora.  Got it.
2024-12-18T02:27:07.395Z
<Slava Dubeyko> Yes
2024-12-18T02:27:13.862Z
<Dan Mick> ok.  let's try openvpn connecting again
2024-12-18T02:27:56.372Z
<Dan Mick> that looks better
2024-12-18T02:28:34.417Z
<Dan Mick> or...maybe, it seems to be connecting over and over.  does it seem like the service is up?
2024-12-18T02:28:35.157Z
<Slava Dubeyko> sudo openvpn --config /etc/openvpn/client/sepia.conf --cd /etc/openvpn/client --verb 5
2024-12-17 18:28:10 WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.
2024-12-17 18:28:10 us=688916 Note: --cipher is not set. OpenVPN versions before 2.5 defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
2024-12-17 18:28:10 us=688964 Note: '--allow-compression' is not set to 'no', disabling data channel offload.
2024-12-17 18:28:10 us=688984 Current Parameter Settings:
2024-12-17 18:28:10 us=688989   config = '/etc/openvpn/client/sepia.conf'
2024-12-17 18:28:10 us=688993   mode = 0
2024-12-17 18:28:10 us=688998 NOTE: --mute triggered...
2024-12-17 18:28:10 us=689010 290 variation(s) on previous 10 message(s) suppressed by --mute
2024-12-17 18:28:10 us=689040 OpenVPN 2.6.12 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
2024-12-17 18:28:10 us=689064 library versions: OpenSSL 3.2.2 4 Jun 2024, LZO 2.10
2024-12-17 18:28:10 us=689078 DCO version: N/A
2024-12-17 18:28:10 us=690200 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
2024-12-17 18:28:10 us=690216 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
2024-12-17 18:28:10 us=690230 LZO compression initializing
2024-12-17 18:28:10 us=690347 Control Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1250 tun_max_mtu:0 headroom:126 payload:1600 tailroom:126 ET:0 ]
2024-12-17 18:28:10 us=691308 Data Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
2024-12-17 18:28:10 us=692463 TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
2024-12-17 18:28:10 us=692492 Socket Buffers: R=[212992->212992] S=[212992->212992]
2024-12-17 18:28:10 us=692501 UDPv4 link local: (not bound)
2024-12-17 18:28:10 us=692508 UDPv4 link remote: [AF_INET]8.43.84.129:1194
2024-12-17 18:28:10 us=692530 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
WR2024-12-17 18:28:10 us=778944 TLS: Initial packet from [AF_INET]8.43.84.129:1194, sid=1556047e 555b5ad5
2024-12-17 18:28:10 us=778994 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
WRRWR2024-12-17 18:28:10 us=863539 VERIFY OK: depth=1, CN=openvpnca-sepia
2024-12-17 18:28:10 us=863684 VERIFY KU OK
2024-12-17 18:28:10 us=863696 Validating certificate extended key usage
2024-12-17 18:28:10 us=863703 NOTE: --mute triggered...
WRWRWR2024-12-17 18:28:12 us=39616 4 variation(s) on previous 10 message(s) suppressed by --mute
2024-12-17 18:28:12 us=39658 [[gw.sepia.ceph.com](http://gw.sepia.ceph.com)] Peer Connection Initiated with [AF_INET]8.43.84.129:1194
2024-12-17 18:28:12 us=39672 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1
2024-12-17 18:28:12 us=39746 TLS: tls_multi_process: initial untrusted session promoted to trusted
W2024-12-17 18:28:13 us=73256 SENT CONTROL [[gw.sepia.ceph.com](http://gw.sepia.ceph.com)]: 'PUSH_REQUEST' (status=1)
WRR2024-12-17 18:28:13 us=250323 PUSH: Received control message: 'PUSH_REPLY,dhcp-option DOMAIN [front.sepia.ceph.com](http://front.sepia.ceph.com),route 172.21.0.0 255.255.240.0 vpn_gateway,route 172.21.32.0 255.255.240.0 vpn_gateway,route 172.21.64.0 255.255.255.0 vpn_gateway,route 172.21.65.0 255.255.255.0 vpn_gateway,route 172.21.66.0 255.255.255.0 vpn_gateway,route 172.21.67.0 255.255.255.0 vpn_gateway,route 172.21.48.1,topology net30,ping 10,ping-restart 60,ifconfig 172.21.49.22 172.21.49.21,peer-id 20,cipher AES-256-GCM'
2024-12-17 18:28:13 us=250432 OPTIONS IMPORT: --ifconfig/up options modified
2024-12-17 18:28:13 us=250443 NOTE: --mute triggered...
2024-12-17 18:28:13 us=250463 2 variation(s) on previous 10 message(s) suppressed by --mute
2024-12-17 18:28:13 us=250472 net_route_v4_best_gw query: dst 0.0.0.0
2024-12-17 18:28:13 us=250641 net_route_v4_best_gw result: via 192.168.1.254 dev wlp2s0f0
2024-12-17 18:28:13 us=250675 ROUTE_GATEWAY 192.168.1.254/255.255.255.0 IFACE=wlp2s0f0 HWADDR=6c:2f:80:99:79:d3
2024-12-17 18:28:13 us=250749 ERROR: Cannot ioctl TUNSETIFF sepia0: Device or resource busy (errno=16)
2024-12-17 18:28:13 us=250754 Exiting due to fatal error
2024-12-18T02:29:24.381Z
<Slava Dubeyko> Is it something wrong on my side?
2024-12-18T02:29:44.538Z
<Dan Mick> how are you attempting connection?
2024-12-18T02:30:18.192Z
<Slava Dubeyko> sudo openvpn --config /etc/openvpn/client/sepia.conf --cd /etc/openvpn/client --verb 5
2024-12-18T02:30:30.485Z
<Dan Mick> oh.  well that's going to run outside of systemd
2024-12-18T02:30:45.132Z
<Dan Mick> and if you've got one running in systemd and one outside, they're going to conflict
2024-12-18T02:31:04.676Z
<Slava Dubeyko> What is the proper way to check the connection?
2024-12-18T02:31:19.775Z
<Dan Mick> the directions say
2024-12-18T02:31:33.929Z
<Dan Mick> ```# After you've been notified in your ticket that access has been granted,

sudo service openvpn restart
OR
sudo systemctl restart openvpn@sepia
OR
sudo systemctl restart openvpn-client@sepia

# Try all 3.  One of them should work.```
2024-12-18T02:32:16.997Z
<Slava Dubeyko> ssh [vdubeyko1973@teuthology.front.sepia.ceph.com](mailto:vdubeyko1973@teuthology.front.sepia.ceph.com)
The authenticity of host '[teuthology.front.sepia.ceph.com](http://teuthology.front.sepia.ceph.com) (172.21.0.51)' can't be established.
ED25519 key fingerprint is SHA256:/9BQuqfuumk1f5t02NAAJw+ecZE7+oGGCcby+gkMHe4.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[teuthology.front.sepia.ceph.com](http://teuthology.front.sepia.ceph.com)' (ED25519) to the list of known hosts.
Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-200-generic x86_64)

 * Documentation:  <https://help.ubuntu.com>
 * Management:     <https://landscape.canonical.com>
 * Support:        <https://ubuntu.com/pro>
New release '22.04.5 LTS' available.
Run 'do-release-upgrade' to upgrade to it.


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

vdubeyko1973@teuthology:~$
2024-12-18T02:32:34.785Z
<Dan Mick> yes, I was going to say, it seems to show you have a connection up
2024-12-18T02:32:41.382Z
<Slava Dubeyko> It looks like it works
2024-12-18T02:32:48.817Z
<Dan Mick> perhaps the service retried on its own, and the errors were from you running openvpn from the cli
2024-12-18T02:32:54.796Z
<Dan Mick> which you should not do.  so, yay.
2024-12-18T02:33:05.791Z
<Slava Dubeyko> Finally πŸ™‚
2024-12-18T02:33:08.409Z
<Dan Mick> you can remove that secret.old file if you wish
2024-12-18T02:33:20.618Z
<Slava Dubeyko> Thanks a lot
2024-12-18T02:33:27.192Z
<Dan Mick> cheers
2024-12-18T02:33:42.565Z
<Slava Dubeyko> Have a nice evening πŸ™‚
2024-12-18T14:20:23.107Z
<David Galloway> On it
2024-12-18T14:58:16.057Z
<Casey Bodley> thank you kindly
2024-12-18T17:03:38.470Z
<David Galloway> I guess the Teuthology Weekly - NA call isn't a thing anymore? πŸ™‚
2024-12-18T18:00:30.271Z
<John Mulligan> Dunno... I have not attended that one before, but it is close to the holidays and a few other weeklies were already canceled as folks are away already. Try again in Jan? :-)
2024-12-18T18:18:41.960Z
<Dan Mick> Almost certain the block is on their side
2024-12-18T18:19:58.934Z
<Dan Mick> There is a collection of bits and bobs for various purposes somewhere on [download.ceph.com](http://download.ceph.com) iirc, maybe mostly for test
2024-12-18T19:51:10.003Z
<Dan Mick> <https://download.ceph.com/qa/>
2024-12-18T19:51:37.372Z
<Dan Mick> see, for example, cmake/modules/BuildBoost.cmake, and others
2024-12-18T19:55:18.306Z
<Eric I> Thanks, @Dan Mick.
2024-12-18T19:55:27.553Z
<Eric I> What does it take to get something put there?
2024-12-18T19:55:43.742Z
<Eric I> We have merged a work-around where we grab the artifacts from a mirror.
2024-12-18T20:31:09.823Z
<Dan Mick> just talk to the infra team managed by Christina Meno, of which I am part.  What is the mirror you're using now, and what's the frequency of update, etc?  Is it an actual package repo mirror?
2024-12-18T22:33:21.304Z
<Slava Dubeyko> I don't know what is happened but I have the same issue today.
2024-12-18T22:33:34.919Z
<Slava Dubeyko> sudo openvpn --config /etc/openvpn/client/sepia.conf --cd /etc/openvpn/client --verb 5
2024-12-18 14:31:54 WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.
2024-12-18 14:31:54 us=283407 Note: --cipher is not set. OpenVPN versions before 2.5 defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
2024-12-18 14:31:54 us=283433 Note: '--allow-compression' is not set to 'no', disabling data channel offload.
2024-12-18 14:31:54 us=283448 Current Parameter Settings:
2024-12-18 14:31:54 us=283452   config = '/etc/openvpn/client/sepia.conf'
2024-12-18 14:31:54 us=283457   mode = 0
2024-12-18 14:31:54 us=283461 NOTE: --mute triggered...
2024-12-18 14:31:54 us=283471 290 variation(s) on previous 10 message(s) suppressed by --mute
2024-12-18 14:31:54 us=283477 OpenVPN 2.6.12 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
2024-12-18 14:31:54 us=283486 library versions: OpenSSL 3.2.2 4 Jun 2024, LZO 2.10
2024-12-18 14:31:54 us=283494 DCO version: N/A
Enter Auth Password: β€’β€’β€’β€’β€’β€’β€’β€’β€’β€’β€’β€’β€’β€’β€’β€’β€’β€’β€’β€’
2024-12-18 14:32:14 us=482993 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
2024-12-18 14:32:14 us=483024 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
2024-12-18 14:32:14 us=483043 LZO compression initializing
2024-12-18 14:32:14 us=483209 Control Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1250 tun_max_mtu:0 headroom:126 payload:1600 tailroom:126 ET:0 ]
2024-12-18 14:32:14 us=494882 Data Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
2024-12-18 14:32:14 us=496348 TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
2024-12-18 14:32:14 us=496376 Socket Buffers: R=[212992->212992] S=[212992->212992]
2024-12-18 14:32:14 us=496386 UDPv4 link local: (not bound)
2024-12-18 14:32:14 us=496393 UDPv4 link remote: [AF_INET]8.43.84.129:1194
2024-12-18 14:32:14 us=496416 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
WR2024-12-18 14:32:14 us=578429 TLS: Initial packet from [AF_INET]8.43.84.129:1194, sid=b4487c59 4723aa00
2024-12-18 14:32:14 us=578469 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
WRRWR2024-12-18 14:32:14 us=662603 VERIFY OK: depth=1, CN=openvpnca-sepia
2024-12-18 14:32:14 us=662714 VERIFY KU OK
2024-12-18 14:32:14 us=662721 Validating certificate extended key usage
2024-12-18 14:32:14 us=662726 NOTE: --mute triggered...
WRWRWR2024-12-18 14:32:15 us=846452 4 variation(s) on previous 10 message(s) suppressed by --mute
2024-12-18 14:32:15 us=846495 [[gw.sepia.ceph.com](http://gw.sepia.ceph.com)] Peer Connection Initiated with [AF_INET]8.43.84.129:1194
2024-12-18 14:32:15 us=846517 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1
2024-12-18 14:32:15 us=846599 TLS: tls_multi_process: initial untrusted session promoted to trusted
W2024-12-18 14:32:17 us=102117 SENT CONTROL [[gw.sepia.ceph.com](http://gw.sepia.ceph.com)]: 'PUSH_REQUEST' (status=1)
WRR2024-12-18 14:32:17 us=184075 AUTH: Received control message: AUTH_FAILED
2024-12-18 14:32:17 us=184291 TCP/UDP: Closing socket
2024-12-18 14:32:17 us=184326 SIGTERM[soft,auth-failure] received, process exiting
2024-12-18T22:42:18.811Z
<Dan Mick> why are you running openvpn directly?
2024-12-18T22:42:40.719Z
<Slava Dubeyko> ssh doesn't work too
2024-12-18T22:42:49.894Z
<Slava Dubeyko> What should I run?
2024-12-18T22:42:54.578Z
<Dan Mick> of course ssh can't work if the network isn't up
2024-12-18T22:43:29.430Z
<Dan Mick> <https://ceph-storage.slack.com/archives/C1HFJ4VTN/p1734489079428069?thread_ts=1734456379.464349&cid=C1HFJ4VTN>
2024-12-18T22:43:48.066Z
<Dan Mick> don't use "sudo openvpn"
2024-12-18T22:45:23.975Z
<Slava Dubeyko> sudo systemctl restart openvpn-client@sepia
πŸ” Enter Auth Password: (press TAB for no echo)
2024-12-18T22:45:29.999Z
<Slava Dubeyko> It asks password now
2024-12-18T22:45:49.949Z
<Slava Dubeyko> It didn't ask before
2024-12-18T22:47:37.661Z
<Dan Mick> the auth failure was because the source-code change to add your new new key hadn't been approved, and some other changes got in before (and reset the server users list).  Fixing that now
2024-12-18T22:48:22.809Z
<Slava Dubeyko> OK. Thanks. But how it worked yesterday? πŸ™‚
2024-12-18T22:48:53.175Z
<Dan Mick> I tested it by editing in your key directly on the server.  but we have source code control for that srtuff
2024-12-18T22:49:06.206Z
<Slava Dubeyko> I see
2024-12-18T23:02:18.549Z
<Dan Mick> sorry about this.  there's a Python incompatibility that's crept into Ansible
2024-12-18T23:02:29.762Z
<Slava Dubeyko> No problem
2024-12-18T23:02:46.064Z
<Slava Dubeyko> Can I try now?
2024-12-18T23:03:43.226Z
<Dan Mick> no, I'm rolling back ansible to see if I can find a version that works
2024-12-18T23:03:49.543Z
<Slava Dubeyko> OK
2024-12-18T23:04:06.611Z
<Dan Mick> the issue is it executes python code remotely, and the python on the openvpn server is older than it wants
2024-12-18T23:04:27.692Z
<Slava Dubeyko> Let me know when it will be done. Thanks
2024-12-18T23:08:51.985Z
<Dan Mick> ok, try it now
2024-12-18T23:10:56.963Z
<Slava Dubeyko> sudo systemctl restart openvpn-client@sepia
πŸ” Enter Auth Password: (press TAB for no echo)
2024-12-18T23:11:06.844Z
<Slava Dubeyko> Still requests password
2024-12-18T23:12:17.430Z
<Slava Dubeyko> I assume it's not correct behavior?
2024-12-18T23:12:29.310Z
<Dan Mick> no.
2024-12-18T23:12:51.683Z
<Dan Mick> you haven't change anything in your config, correct?
2024-12-18T23:13:42.846Z
<Dan Mick> you haven't changed anything in your config, correct?
2024-12-18T23:13:55.935Z
<Slava Dubeyko> Do you mean secret? It's the same like a latest one in the ticket. I didn't change anything
2024-12-18T23:15:11.387Z
<Slava Dubeyko> I double checked it today
2024-12-18T23:16:00.821Z
<Dan Mick> well, again, 'secret' is a secret, and wouldn't be in the ticket, so it shouldn't be the same as the one in the ticket.   But I don't think it's getting that far
2024-12-18T23:17:15.628Z
<Slava Dubeyko> OK. I confused. πŸ™‚ I believe that secret should the same on both sides.
2024-12-18T23:17:30.462Z
<Dan Mick> it's like a unix password
2024-12-18T23:17:34.287Z
<Dan Mick> your password is secret
2024-12-18T23:17:42.562Z
<Dan Mick> what's i the password file is a hash of your password
2024-12-18T23:18:00.053Z
<Dan Mick> when you log in, you type your password, and the login command hashes it, and compares it to what's in the password file
2024-12-18T23:18:05.362Z
<Dan Mick> that way your password remains secret
2024-12-18T23:18:36.001Z
<Slava Dubeyko> OK. What doesn't work right now? πŸ™‚
2024-12-18T23:18:41.187Z
<Dan Mick> I don't know.
2024-12-18T23:18:50.039Z
<Dan Mick> try sudo systemctl cat openvpn-client@sepia
2024-12-18T23:19:56.311Z
<Slava Dubeyko> # /usr/lib/systemd/system/openvpn-client@.service
[Unit]
Description=OpenVPN tunnel for %I
After=network-online.target
Wants=network-online.target
Documentation=man:openvpn(8)
Documentation=<https://openvpn.net/community-resources/reference-manual-for-openvpn-2-6/>
Documentation=<https://community.openvpn.net/openvpn/wiki/HOWTO>

[Service]
Type=notify
PrivateTmp=true
WorkingDirectory=/etc/openvpn/client
ExecStart=/usr/sbin/openvpn --suppress-timestamps --nobind --config %i.conf
CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SETPCAP>
LimitNPROC=10
DeviceAllow=/dev/null rw
DeviceAllow=/dev/net/tun rw
ProtectSystem=true
ProtectHome=true
KillMode=process

[Install]
WantedBy=multi-user.target

# /usr/lib/systemd/system/service.d/10-timeout-abort.conf
# This file is part of the systemd package.
# See <https://fedoraproject.org/wiki/Changes/Shorter_Shutdown_Timer>.
#
# To facilitate debugging when a service fails to stop cleanly,
# TimeoutStopFailureMode=abort is set to "crash" services that fail to stop in
# the time allotted. This will cause the service to be terminated with SIGABRT
# and a coredump to be generated.
#
# To undo this configuration change, create a mask file:
#   sudo mkdir -p /etc/systemd/system/service.d
#   sudo ln -sv /dev/null /etc/systemd/system/service.d/10-timeout-abort.conf

[Service]
TimeoutStopFailureMode=abort

# /usr/lib/systemd/system/service.d/50-keep-warm.conf
# Disable freezing of user sessions to work around kernel bugs.
# See <https://bugzilla.redhat.com/show_bug.cgi?id=2321268>
[Service]
Environment=SYSTEMD_SLEEP_FREEZE_USER_SESSIONS=0
2024-12-18T23:20:38.561Z
<Dan Mick> okay.  and so you should have an /etc/openvpn/client/sepia.conf file, I assume.  paste the contents of that.
2024-12-18T23:21:30.784Z
<Slava Dubeyko> cat ./sepia.conf
script-security 1
client
remote [vpn.sepia.ceph.com](http://vpn.sepia.ceph.com) 1194
dev sepia0
dev-type tun
remote-random
resolv-retry infinite
nobind
user openvpn
group openvpn
persist-tun
persist-key
comp-lzo
verb 2
mute 10
remote-cert-tls server
tls-auth sepia/tlsauth 1
ca sepia/ca.crt
auth-user-pass sepia/secret
2024-12-18T23:23:18.154Z
<Dan Mick> and /etc/openvpn/client/sepia/secret file is present, yes?  (don't paste it, it's your secret)
2024-12-18T23:23:50.528Z
<Slava Dubeyko> Yes
2024-12-18T23:25:17.805Z
<Dan Mick> what's journalctl -e | grep openvpn show for the 'last' connection attempt?
2024-12-18T23:25:53.581Z
<Dan Mick> (sorry, as root, so sudo journalctl...)
2024-12-18T23:26:44.619Z
<Slava Dubeyko> journalctl -e | grep openvpn
Dec 18 15:19:17 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) sudo[23193]: vdubeyko1973 : TTY=pts/3 ; PWD=/home/vdubeyko1973 ; USER=root ; COMMAND=/usr/bin/systemctl cat openvpn-client@sepia
2024-12-18T23:26:51.806Z
<Slava Dubeyko> Nothing else
2024-12-18T23:26:59.695Z
<Dan Mick> what the?..
2024-12-18T23:27:19.326Z
<Dan Mick> sudo systemctl status openvpn-client@sepia
2024-12-18T23:28:13.309Z
<Slava Dubeyko> sudo systemctl status openvpn-client@sepia
Γ— openvpn-client@sepia.service - OpenVPN tunnel for sepia
     Loaded: loaded (/usr/lib/systemd/system/openvpn-client@.service; enabled; preset: disable>
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf, 50-keep-warm.conf
     Active: failed (Result: timeout) since Wed 2024-12-18 15:11:23 PST; 16min ago
   Duration: 2.659s
 Invocation: cf156293d23941fb93a1fe1f1ad8f2c0
       Docs: man:openvpn(8)
             <https://openvpn.net/community-resources/reference-manual-for-openvpn-2-6/>
             <https://community.openvpn.net/openvpn/wiki/HOWTO>
    Process: 20641 ExecStart=/usr/sbin/openvpn --suppress-timestamps --nobind --config sepia.c>
   Main PID: 20641 (code=dumped, signal=ABRT)
   Mem peak: 2.6M
        CPU: 13ms

Dec 18 15:10:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20641]: library versio>
Dec 18 15:10:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20641]: DCO version: N>
Dec 18 15:11:13 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sep>
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sep>
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sep>
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sep>
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sep>
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sep
2024-12-18T23:28:57.877Z
<Dan Mick> sigh.  stupid default pager.  add --no-pager to that command
2024-12-18T23:29:48.556Z
<Slava Dubeyko> sudo systemctl status openvpn-client@sepia --no-pager
Γ— openvpn-client@sepia.service - OpenVPN tunnel for sepia
     Loaded: loaded (/usr/lib/systemd/system/openvpn-client@.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf, 50-keep-warm.conf
     Active: failed (Result: timeout) since Wed 2024-12-18 15:11:23 PST; 18min ago
   Duration: 2.659s
 Invocation: cf156293d23941fb93a1fe1f1ad8f2c0
       Docs: man:openvpn(8)
             <https://openvpn.net/community-resources/reference-manual-for-openvpn-2-6/>
             <https://community.openvpn.net/openvpn/wiki/HOWTO>
    Process: 20641 ExecStart=/usr/sbin/openvpn --suppress-timestamps --nobind --config sepia.conf (code=dumped, signal=ABRT)
   Main PID: 20641 (code=dumped, signal=ABRT)
   Mem peak: 2.6M
        CPU: 13ms

Dec 18 15:10:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20641]: library versi…0
Dec 18 15:10:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20641]: DCO version: …A
Dec 18 15:11:13 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@s…g.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@s…g.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@s…T.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@s…RT
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@s…'.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@s…d.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: Failed to start …a.
Dec 18 15:11:28 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20642]: Failed to que…d
Hint: Some lines were ellipsized, use -l to show in full.
2024-12-18T23:30:03.597Z
<Dan Mick> GAH and -l I guess
2024-12-18T23:30:22.812Z
<Slava Dubeyko> sudo systemctl status openvpn-client@sepia --no-pager -l
Γ— openvpn-client@sepia.service - OpenVPN tunnel for sepia
     Loaded: loaded (/usr/lib/systemd/system/openvpn-client@.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf, 50-keep-warm.conf
     Active: failed (Result: timeout) since Wed 2024-12-18 15:11:23 PST; 18min ago
   Duration: 2.659s
 Invocation: cf156293d23941fb93a1fe1f1ad8f2c0
       Docs: man:openvpn(8)
             <https://openvpn.net/community-resources/reference-manual-for-openvpn-2-6/>
             <https://community.openvpn.net/openvpn/wiki/HOWTO>
    Process: 20641 ExecStart=/usr/sbin/openvpn --suppress-timestamps --nobind --config sepia.conf (code=dumped, signal=ABRT)
   Main PID: 20641 (code=dumped, signal=ABRT)
   Mem peak: 2.6M
        CPU: 13ms

Dec 18 15:10:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20641]: library versions: OpenSSL 3.2.2 4 Jun 2024, LZO 2.10
Dec 18 15:10:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20641]: DCO version: N/A
Dec 18 15:11:13 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sepia.service: start operation timed out. Terminating.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sepia.service: State 'stop-sigterm' timed out. Aborting.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sepia.service: Killing process 20641 (openvpn) with signal SIGABRT.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sepia.service: Main process exited, code=dumped, status=6/ABRT
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sepia.service: Failed with result 'timeout'.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sepia.service: Unit process 20642 (systemd-ask-pas) remains running after unit stopped.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: Failed to start openvpn-client@sepia.service - OpenVPN tunnel for sepia.
Dec 18 15:11:28 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20642]: Failed to query password: Timer expired
2024-12-18T23:33:03.261Z
<Dan Mick> I'm not getting how this could have broken.  but to get better status, try, instead:

sudo journalctl -S today -l openvpn-client@sepia
2024-12-18T23:34:13.383Z
<Slava Dubeyko> sudo journalctl -S today -l openvpn-client@sepia
Failed to add match 'openvpn-client@sepia': Invalid argument
2024-12-18T23:34:20.103Z
<Slava Dubeyko> Something incorrect
2024-12-18T23:34:53.016Z
<Dan Mick> sudo journalctl -S today -l -u openvpn-client@sepia
2024-12-18T23:36:57.272Z
<Dan Mick> not working either?
2024-12-18T23:38:27.650Z
<Slava Dubeyko> I am trying to send output to you
2024-12-18T23:38:50.103Z
<Dan Mick> just the last 'burst' of similar timestamps is fine
2024-12-18T23:41:26.072Z
<Slava Dubeyko> Dec 18 15:09:37 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: Starting openvpn-client@sepia.service - OpenVPN tunnel for sepia...
Dec 18 15:09:37 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.
Dec 18 15:09:37 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: Note: --cipher is not set. OpenVPN versions before 2.5 defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
Dec 18 15:09:37 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: OpenVPN 2.6.12 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
Dec 18 15:09:37 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: library versions: OpenSSL 3.2.2 4 Jun 2024, LZO 2.10
Dec 18 15:09:37 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: DCO version: N/A
Dec 18 15:09:40 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: Started openvpn-client@sepia.service - OpenVPN tunnel for sepia.
Dec 18 15:09:40 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
Dec 18 15:09:40 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: UDPv4 link local: (not bound)
Dec 18 15:09:40 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: UDPv4 link remote: [AF_INET]8.43.84.129:1194
Dec 18 15:09:40 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Dec 18 15:09:40 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Dec 18 15:09:40 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: VERIFY OK: depth=1, CN=openvpnca-sepia
Dec 18 15:09:40 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: VERIFY KU OK
Dec 18 15:09:40 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: Validating certificate extended key usage
Dec 18 15:09:40 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: NOTE: --mute triggered...
Dec 18 15:09:42 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: 4 variation(s) on previous 10 message(s) suppressed by --mute
Dec 18 15:09:42 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: [[gw.sepia.ceph.com](http://gw.sepia.ceph.com)] Peer Connection Initiated with [AF_INET]8.43.84.129:1194
Dec 18 15:09:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: AUTH: Received control message: AUTH_FAILED
Dec 18 15:09:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20052]: SIGTERM[soft,auth-failure] received, process exiting
Dec 18 15:09:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sepia.service: Deactivated successfully.
Dec 18 15:10:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: Starting openvpn-client@sepia.service - OpenVPN tunnel for sepia...
Dec 18 15:10:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20641]: WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.
Dec 18 15:10:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20641]: Note: --cipher is not set. OpenVPN versions before 2.5 defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
Dec 18 15:10:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20641]: OpenVPN 2.6.12 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
Dec 18 15:10:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20641]: library versions: OpenSSL 3.2.2 4 Jun 2024, LZO 2.10
Dec 18 15:10:43 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20641]: DCO version: N/A
Dec 18 15:11:13 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sepia.service: start operation timed out. Terminating.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sepia.service: State 'stop-sigterm' timed out. Aborting.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sepia.service: Killing process 20641 (openvpn) with signal SIGABRT.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sepia.service: Main process exited, code=dumped, status=6/ABRT
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sepia.service: Failed with result 'timeout'.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: openvpn-client@sepia.service: Unit process 20642 (systemd-ask-pas) remains running after unit stopped.
Dec 18 15:11:23 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) systemd[1]: Failed to start openvpn-client@sepia.service - OpenVPN tunnel for sepia.
Dec 18 15:11:28 [li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com](http://li-4c4c4544-0032-4210-804c-c3c04f423534.ibm.com) openvpn[20642]: Failed to query password: Timer expired
2024-12-18T23:47:37.164Z
<Dan Mick> it's like it's just not properly reading your config file.  It never even tries to contact the server, it seems
2024-12-18T23:47:52.973Z
<Dan Mick> try this:  sudo systemctl stop openvpn-client*
2024-12-18T23:47:56.213Z
<Dan Mick> ps -ef | grep openvpn
2024-12-18T23:48:50.406Z
<Slava Dubeyko> ps -ef | grep openvpn
vdubeyk+   33856    7262  0 15:48 pts/3    00:00:00 grep --color=auto openvpn
2024-12-18T23:49:58.020Z
<Dan Mick> okay, so, I said not to try sudo openvpn, but, let's try it, but let's add some stracing.  Do you have strace installed?
2024-12-18T23:50:10.484Z
<Slava Dubeyko> Sure
2024-12-18T23:50:53.189Z
<Dan Mick> so add "strace -f -s 1024 -o /tmp/strace.out" after 'sudo' and before 'openvpn' in the sudo openvpn command you were using before
2024-12-18T23:51:59.256Z
<Slava Dubeyko> Do you mean this command: sudo openvpn --config /etc/openvpn/client/sepia.conf --cd /etc/openvpn/client --verb 5?
2024-12-18T23:52:10.911Z
<Dan Mick> yeah.  but wait, before we do that
2024-12-18T23:52:18.154Z
<Dan Mick> let's just verify the format of sepia/secret
2024-12-18T23:52:39.507Z
<Dan Mick> it should have two lines, the first one should be your openvpn username, and the second one your private secret.  Does it have two lines?
2024-12-18T23:54:37.580Z
<Slava Dubeyko> It's one line <login>@<system> <secret>
2024-12-18T23:54:52.656Z
<Slava Dubeyko> Should I add another line?
2024-12-18T23:55:12.710Z
<Dan Mick> no.  so are there three space-separated fields on that one line?
2024-12-18T23:56:05.903Z
<Slava Dubeyko> <login>@<system> space <secret>
2024-12-18T23:56:30.682Z
<Slava Dubeyko> <login>@<system> space <secret> \n
2024-12-18T23:56:31.131Z
<Dan Mick> what are the first three characters of the secret?
2024-12-18T23:56:54.385Z
<Slava Dubeyko> w+1
2024-12-18T23:57:10.165Z
<Dan Mick> ok.  do you see a space between Q and 5 on that line?
2024-12-18T23:57:23.369Z
<Slava Dubeyko> Yes
2024-12-18T23:57:34.287Z
<Dan Mick> ok.  so there are three fields on that line, separated by spaces, correct?
2024-12-18T23:57:58.702Z
<Slava Dubeyko> I think so πŸ™‚
2024-12-18T23:58:16.674Z
<Dan Mick> so that is your hashed secret, which is what we install on the openvpn server.  that is **not** your secret, which would be a two-line file
2024-12-18T23:58:35.163Z
<Dan Mick> sometime between "working" and "not working", you copied the output of new-client into sepia/secret
2024-12-18T23:59:02.748Z
<Dan Mick> that broke it.  Do you still have a file that has two lines in /etc/openvpn/client/sepia?
2024-12-18T23:59:38.128Z
<Slava Dubeyko> Yes, I have
2024-12-18T23:59:57.046Z
<Dan Mick> is it timestamped from yesterday afternoon?

Any issue? please create an issue here and use the infra label.