2024-08-12T07:06:52.023Z

<Hemanth Kumar> Hi
I m trying to connect to sepia lab using my new creds - VPN using Viscosity and seeing the following error :

2024-08-12 12:16:45: Viscosity Mac 1.10.4 (1611)
2024-08-12 12:16:45: Viscosity OpenVPN Engine Started
2024-08-12 12:16:45: Running on macOS 14.4.0
2024-08-12 12:16:45: ---------
2024-08-12 12:16:45: State changed to Connecting
2024-08-12 12:16:45: Checking reachability status of connection...
2024-08-12 12:16:45: DNS resolution failed for [vpn-pub.dhc.sepia.ceph.com](http://vpn-pub.dhc.sepia.ceph.com)
2024-08-12 12:16:45: Connection is reachable. Starting connection attempt.
2024-08-12 12:16:45: --cipher is not set. Previous OpenVPN version defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add ‘--data-ciphers-fallback BF-CBC’ to your configuration and/or add BF-CBC to --data-ciphers.
2024-08-12 12:16:45: OpenVPN 2.5.7 x86_64-apple-darwin [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [MH/RECVDA] [AEAD] built on Nov  2 2022
2024-08-12 12:16:45: library versions: OpenSSL 1.1.1s  1 Nov 2022, LZO 2.10
2024-08-12 12:16:45: Resolving address: [vpn-pub.dhc.sepia.ceph.com](http://vpn-pub.dhc.sepia.ceph.com)
2024-08-12 12:16:45: Address could not be resolved: [vpn-pub.dhc.sepia.ceph.com](http://vpn-pub.dhc.sepia.ceph.com)
2024-08-12 12:16:45: Valid endpoint found: 8.43.84.129:1194:udp
2024-08-12 12:16:45: TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
2024-08-12 12:16:45: UDP link local: (not bound)
2024-08-12 12:16:45: UDP link remote: [AF_INET]8.43.84.129:1194
2024-08-12 12:16:45: State changed to Authenticating
2024-08-12 12:16:47: WARNING: ‘link-mtu’ is used inconsistently, local=‘link-mtu 1542’, remote=‘link-mtu 1550’
2024-08-12 12:16:47: WARNING: ‘auth’ is used inconsistently, local=‘auth SHA1’, remote=‘auth [null-digest]’
2024-08-12 12:16:47: WARNING: ‘keysize’ is used inconsistently, local=‘keysize 128’, remote=‘keysize 256’
2024-08-12 12:16:47: [openvpn-sepia] Peer Connection Initiated with [AF_INET]8.43.84.129:1194
2024-08-12 12:16:47: AUTH: Received control message: AUTH_FAILED
2024-08-12 12:16:49: SIGUSR1[soft,auth-failure] received, process restarting
2024-08-12 12:16:49: Viscosity Mac 1.10.4 (1611)
2024-08-12 12:16:49: Viscosity OpenVPN Engine Started
2024-08-12 12:16:49: Running on macOS 14.4.0
2024-08-12 12:16:49: ---------
2024-08-12 12:16:49: State changed to Connecting
2024-08-12 12:16:49: Valid existing endpoint found... 8.43.84.129:1194:udp
2024-08-12 12:16:55: State changed to Disconnecting (Username/Password Cancelled)
2024-08-12 12:16:55: ERROR: could not read Auth username/password/ok/string from management interface
2024-08-12 12:16:55: Exiting due to fatal error
2024-08-12 12:16:55: State changed to Disconnected (Process Terminated)

I followed the steps mentioned to connect to VPN using Viscocity : <https://wiki.sepia.ceph.com/doku.php?id=vpnaccess>

2024-08-12T07:21:37.822Z

<Hemanth Kumar> @Adam Kraitman - Any help ?

2024-08-12T13:27:24.462Z

<Kyrylo Shatskyy> hey guys is maintaining shaman and chacra recently?

2024-08-12T13:28:28.460Z

<Kyrylo Shatskyy> are you sure you're using new creds?

2024-08-12T13:30:09.929Z

<Kyrylo Shatskyy> I haven't used Viscosity but for Tunnelblick I needed to reimport configuration from filesystem because just editing on a disk is not enough.

2024-08-12T15:12:02.366Z

<Sayalee Raut> Hello,
I am able to add the client.conf to my Tunnelblick successfully, but when I enter the username as USER@HOST and password as the secret contents of the file `/etc/openvpn/sepia/secret` ; the authentication fails with below error -
```2024-08-12 13:07:57.448839 *Tunnelblick: Disconnecting; user cancelled authorization or there was an error obtaining authorization
2024-08-12 13:07:57.594157 *Tunnelblick: Disconnecting using 'kill'
2024-08-12 13:07:57.740099 MANAGEMENT: CMD 'hold release'
2024-08-12 13:07:57.740420 MANAGEMENT: CMD 'hold release'
2024-08-12 13:07:57.774799 MANAGEMENT: Client disconnected
2024-08-12 13:07:57.774844 ERROR: could not read Auth username/password/ok/string from management interface
2024-08-12 13:07:57.774860 Exiting due to fatal error
2024-08-12 13:07:59.956908 *Tunnelblick: Expected disconnection occurred.```

2024-08-12T15:44:45.134Z

<Zack Cerza> @Adam Kraitman / @Dan Mick - I need to debug something on teuthology.front, and many times in the past I've used e.g. `journalctl -S '-2d'` - but no matter what I do I can't get more than ~13h of logs from it. I think I need help figuring this out.

2024-08-12T15:53:17.618Z

<Hemanth Kumar> @Kyrylo Shatskyy - Yes,  - <https://tracker.ceph.com/issues/67422>.

Tried tunnelblick also - the following errors are seen while connecting

2024-08-12 21:19:03.577334 MANAGEMENT: Client connected from [AF_INET]127.0.0.1:57540
2024-08-12 21:19:03.661373 MANAGEMENT: CMD ‘pid’
2024-08-12 21:19:03.661867 MANAGEMENT: CMD ‘auth-retry interact’
2024-08-12 21:19:03.661969 MANAGEMENT: CMD ‘state on’
2024-08-12 21:19:03.662268 NOTE: --mute triggered...
2024-08-12 21:19:03.663619 *Tunnelblick: Established communication with OpenVPN
2024-08-12 21:19:03.693840 *Tunnelblick: >INFO:OpenVPN Management Interface Version 5 -- type ‘help’ for more info
2024-08-12 21:19:03.700403 *Tunnelblick: Obtained VPN username and password from the Keychain
2024-08-12 21:19:03.702656 5 variation(s) on previous 10 message(s) suppressed by --mute
2024-08-12 21:19:03.702708 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2024-08-12 21:19:03.702755 PLUGIN_INIT: POST /Applications/Tunnelblick.app/Contents/Resources/openvpn/openvpn-2.6.9-openssl-3.0.13/openvpn-down-root.so ‘[/Applications/Tunnelblick.app/Contents/Resources/openvpn/openvpn-2.6.9-openssl-3.0.13/openvpn-down-root.so] [/Applications/Tunnelblick.app/Contents/Resources/client.down.tunnelblick.sh] [-9] [-d] [-f] [-m] [-w] [-ptADGNWradsgnw]’ intercepted=PLUGIN_UP|PLUGIN_DOWN
2024-08-12 21:19:03.705697 MANAGEMENT: >STATE:1723477743,RESOLVE,,,,,,
2024-08-12 21:19:05.252456 TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
2024-08-12 21:19:05.252597 Socket Buffers: R=[786896->786896] S=[9216->9216]
2024-08-12 21:19:05.252627 UDPv4 link local: (not bound)
2024-08-12 21:19:05.252648 UDPv4 link remote: [AF_INET]8.43.84.129:1194
2024-08-12 21:19:05.252740 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
2024-08-12 21:19:05.252808 MANAGEMENT: >STATE:1723477745,WAIT,,,,,,
2024-08-12 21:19:05.543952 MANAGEMENT: >STATE:1723477745,AUTH,,,,,,
2024-08-12 21:19:05.544115 TLS: Initial packet from [AF_INET]8.43.84.129:1194, sid=1c5701c1 c8184c7d
2024-08-12 21:19:05.544247 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
2024-08-12 21:19:05.838388 VERIFY OK: depth=1, O=Redhat, CN=openvpnca-sepia
2024-08-12 21:19:05.839145 VERIFY KU OK
2024-08-12 21:19:05.839185 Validating certificate extended key usage
2024-08-12 21:19:05.839206 NOTE: --mute triggered...
2024-08-12 21:19:07.172724 4 variation(s) on previous 10 message(s) suppressed by --mute
2024-08-12 21:19:07.172854 [openvpn-sepia] Peer Connection Initiated with [AF_INET]8.43.84.129:1194
2024-08-12 21:19:07.172964 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1
2024-08-12 21:19:07.173149 TLS: tls_multi_process: initial untrusted session promoted to trusted
2024-08-12 21:19:08.272465 MANAGEMENT: >STATE:1723477748,GET_CONFIG,,,,,,
2024-08-12 21:19:08.272955 SENT CONTROL [openvpn-sepia]: ‘PUSH_REQUEST’ (status=1)
2024-08-12 21:19:08.542672 AUTH: Received control message: AUTH_FAILED
2024-08-12 21:19:08.543466 SIGUSR1[soft,auth-failure] received, process restarting
2024-08-12 21:19:08.543614 MANAGEMENT: >STATE:1723477748,RECONNECTING,auth-failure,,,,,
2024-08-12 21:19:12.583503 MANAGEMENT: CMD ‘hold release’
2024-08-12 21:19:12.583812 MANAGEMENT: CMD ‘hold release’
2024-08-12 21:19:12.594298 *Tunnelblick: Obtained VPN username and password from the Keychain
2024-08-12 21:19:12.598579 MANAGEMENT: CMD ‘username “Auth” “hyelloji@Hemanths-Laptop”’
2024-08-12 21:19:12.598747 NOTE: --mute triggered...
2024-08-12 21:19:12.598827 1 variation(s) on previous 10 message(s) suppressed by --mute
2024-08-12 21:19:12.598855 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2024-08-12 21:19:12.599349 TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
2024-08-12 21:19:12.599499 Socket Buffers: R=[786896->786896] S=[9216->9216]
2024-08-12 21:19:12.599526 UDPv4 link local: (not bound)
2024-08-12 21:19:12.599548 UDPv4 link remote: [AF_INET]8.43.84.129:1194
2024-08-12 21:19:12.599647 MANAGEMENT: >STATE:1723477752,WAIT,,,,,,
2024-08-12 21:19:12.918928 MANAGEMENT: >STATE:1723477752,AUTH,,,,,,
2024-08-12 21:19:12.919062 TLS: Initial packet from [AF_INET]8.43.84.129:1194, sid=790b3a50 8b1c3131
2024-08-12 21:19:13.205958 VERIFY OK: depth=1, O=Redhat, CN=openvpnca-sepia
2024-08-12 21:19:13.206598 VERIFY KU OK
2024-08-12 21:19:13.206630 NOTE: --mute triggered...
2024-08-12 21:19:14.557360 5 variation(s) on previous 10 message(s) suppressed by --mute
2024-08-12 21:19:14.557472 [openvpn-sepia] Peer Connection Initiated with [AF_INET]8.43.84.129:1194
2024-08-12 21:19:14.557538 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1
2024-08-12 21:19:14.557706 TLS: tls_multi_process: initial untrusted session promoted to trusted
2024-08-12 21:19:15.703178 MANAGEMENT: >STATE:1723477755,GET_CONFIG,,,,,,
2024-08-12 21:19:15.703374 SENT CONTROL [openvpn-sepia]: ‘PUSH_REQUEST’ (status=1)
2024-08-12 21:19:16.002843 AUTH: Received control message: AUTH_FAILED
2024-08-12 21:19:16.003277 SIGUSR1[soft,auth-failure] received, process restarting
2024-08-12 21:19:16.003329 MANAGEMENT: >STATE:1723477756,RECONNECTING,auth-failure,,,,,
2024-08-12 21:19:17.935516 *Tunnelblick: Disconnecting; user cancelled authorization or there was an error obtaining authorization
2024-08-12 21:19:18.240531 *Tunnelblick: Disconnecting using ‘kill’
2024-08-12 21:19:18.546149 MANAGEMENT: CMD ‘hold release’
2024-08-12 21:19:18.546268 MANAGEMENT: CMD ‘hold release’
2024-08-12 21:19:18.586729 MANAGEMENT: Client disconnected
2024-08-12 21:19:18.586769 ERROR: could not read Auth username/password/ok/string from management interface
2024-08-12 21:19:18.586782 Exiting due to fatal error
2024-08-12 21:19:20.775088 *Tunnelblick: Expected disconnection occurred.

2024-08-12T16:58:21.026Z

<Laura Flores> @Dan Mick I noticed a protobuf failure on irvingi05. I know you were working on fixing that, so JYFI:

<https://jenkins.ceph.com/job/ceph-pull-requests/141077/consoleFull#-12742200307641bc0b-bacc-4481-8290-0cf28df0ce68>
```-- Found yaml-cpp: /usr/lib/x86_64-linux-gnu/libyaml-cpp.so (found suitable version "0.7.0", minimum required is "0.5.1") 
CMake Error at /usr/lib/x86_64-linux-gnu/cmake/protobuf/protobuf-targets.cmake:111 (message):
  The imported target "protobuf::protoc" references the file

     "/usr/bin/protoc-25.1.0"

  but this file does not exist.  Possible reasons include:

  * The file was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and contained

     "/usr/lib/x86_64-linux-gnu/cmake/protobuf/protobuf-targets.cmake"

  but not all the files it references.

Call Stack (most recent call first):
  /usr/lib/x86_64-linux-gnu/cmake/protobuf/protobuf-config.cmake:14 (include)
  src/CMakeLists.txt:390 (_find_package)
  src/seastar/cmake/SeastarDependencies.cmake:162 (find_package)
  src/seastar/CMakeLists.txt:399 (seastar_find_dependencies)


-- Configuring incomplete, errors occurred!
See also "/home/jenkins-build/build/workspace/ceph-pull-requests/build/CMakeFiles/CMakeOutput.log".
See also "/home/jenkins-build/build/workspace/ceph-pull-requests/build/CMakeFiles/CMakeError.log".
+ exit 1```

2024-08-12T17:05:11.738Z

<Casey Bodley> i commented about that in <https://tracker.ceph.com/issues/64907>, a couple other nodes have been failing there too

2024-08-12T17:36:58.680Z

<Hemanth Kumar> It worked. I am able to access the VPN

The wiki page needs an update - <https://wiki.sepia.ceph.com/doku.php?id=vpnaccess#vpn_client_access>

‘’ When connecting to the VPN for the first time,
- Enter your USER@HOST combination as the username
- Enter the second line of sepia/secret as the password
‘’
This needs to be updated as :
• username should be the first line in secret file
• password must be the second line in secret file

2024-08-12T17:50:04.114Z

<Zack Cerza> looks like we dropped ~2500 expired jobs over the weekend. not a miracle, but a start

2024-08-12T17:51:05.617Z

<Zack Cerza> hm, queue hasn't been this low in nearly two months

2024-08-12T17:52:23.555Z

<Dan Mick> journalctl on its own says "-- Logs begin at Mon 2024-08-12 02:15:02 UTC, end at Mon 2024-08-12 17:52:05 UT>"

2024-08-12T17:52:35.348Z

<Dan Mick> I assume that's because of some rotation setting.  Is that what you're asking about, where is that?

2024-08-12T17:55:31.156Z

<Zack Cerza> right, that's the problem - it should be storing logs for _far_ longer than that

2024-08-12T18:00:16.024Z

<Zack Cerza> seems pretty important to know when our disks are about to fail

2024-08-12T18:16:56.796Z

<Dan Mick> I got nothin.  Everything in /var/log/journal was apparently touched today at around 0727.  There are also very odd commands in the root cli history including "mailx/mutt [danmick@gmail.com](mailto:danmick@gmail.com)" which certainly were not me.  I can't find anything in the cli history that would indicate any sort of log resetting

2024-08-12T18:17:18.266Z

<Dan Mick> can't find any rotation settings changing the default; /, which has /var, has plenty of space.

2024-08-12T18:25:21.935Z

<Zack Cerza> um. _that_ is concerning

2024-08-12T18:26:10.129Z

<Zack Cerza> `auth.log` doesn't even go back 48h

2024-08-12T18:26:59.790Z

<Zack Cerza> @Adam Kraitman / @Dan Mick - I need to debug something on teuthology.front, and many times in the past I've used e.g. `journalctl -S '-2d'` - but no matter what I do I can't get more than ~13h of logs from it. I think I need help figuring this out.

2024-08-12T18:47:43.366Z

<kenan.al-shamie> hi there @Dan Mick , any update on finding me a new server? thanks 🙂 

2024-08-12T18:48:06.851Z

<kenan.al-shamie> hi there @Dan Mick , any update on finding me a playground host? thanks 🙂 

2024-08-12T18:57:54.293Z

<Dan Mick> I did some cleanup similar to the last time this happened, but the symptoms are different this time.  I could use some help turning that error message into an actual fault; cmake is a harsh mistress and I'm having to do basic research to get it.  I'll continue, but the issue isn't clear yet

2024-08-12T23:17:50.637Z

<Dan Mick> Just lost here.  The error comes, apparently, from /usr/lib/x86_64-linux-gnu/cmake/protobuf/protobuf-targets.cmake, but I can't find any package at all that installed that directory tree

2024-08-12T23:20:26.410Z

<Dan Mick> -rw-r--r-- 1 root root 7820 Feb 24 20:55 protobuf-targets.cmake

2024-08-12T23:20:45.245Z

<Dan Mick> now reduced to searching this builder for any file created between Feb23 and Feb25