[TCP] open fails, but curl works

Started by Loren Doornek, August 19, 2019, 04:15:29 PM

Previous topic - Next topic

Loren Doornek

We have a client where a certain program just quit working in the middle of the night last week.  A Secure TCP connection can no longer connect, and return MSG(-1) as 'cannot connect securely'.  This has been working fine for months, and there were no changes to any of our programs.  We aren't aware of any changes to the Linux server either, and have no insight into any network changes since it's a huge company with an extensive network. 

We updated everything on the OS, but still can't connect.

So, I tried using 'curl' instead of a PXPlus [TCP] channel.   I *CAN* connect using in PXPlus using a Linux input from curl (ie: OPEN (1)"< curl -X ....."), but trying to open the same site using PXPlus (ie: OPEN (1)"[TCP]some.server;443;SECURE;NODELAY;STREAM") fails ever time. 

PXPlus version is 12.50, and I've tried using NOSSLV2 and other options, but no luck.  I'm assuming that PXPlus is using the same SSL libraries as the Linux OS, so it makes no sense to me why curl can open the connection, but PXPlus cannot.

Any suggestions on what I can look for?

Allen Miglore

We've been hearing lately of some web services requiring tls 1.2.  From the message, it sounds like it might be that sort of restriction.  Don't remember if v12 supports 1.2.

Mike King


Two possibilities I can think of:

  • The Host is requiring TLS1.2 which means your version of OpenSSL has to be 1.0.1 (possibly 1.0.0 build 'h' or later might work).  If PxPlus is running on Windows then you need PxPlus 2014 (V12) which ships with OpenSSL 1.0.1.  If running on Unix/Aix/Linux, your server needs to upgraded as we use the OS supplied and installed version of OpenSSL.

  • The Host IP address may be using SNI which in simple terms means the same IP address is being serviced by multiple network names.  SSL matches the network name used to connect to the server with the name in the certificate on the host.  When you have multiple network names going to the same IP address you need to support SNI which supports multiple network names (multiple certificates) on the same physical IP address. SNI Support was added to PxPlus 2017 (v14)

Mike King
President - BBSysco Consulting
eMail: mike.king@bbsysco.com

Loren Doornek

We have a version 12 on our in-house dev system that is the exact same version as the client (tcb(29)=1250000), and it works fine from our dev system, so I suspect that v12 is not an issue.  We originally suspected that there is some network device that changed (since firewalls can block ssl/tls connections based on ciphers and versions).  But, the fact that we can connect every time via curl seems to rule out that possibility.  At this point, our only thought is that somehow PXPlus is using a different/out-dated SSL library, but I have no ideas on how to check that.

Loren Doornek

The server is running OpenSSL 1.0.1e, but I don't think the site requires TLS1.2.  We have an identical server, using the same RedHat version and OpenSSL library, and we can connect without a problem using OpenSSL 1.0.1e.  I'm starting to suspect that there is some device/firewall on their network that is blocking the ssl negotiation, either because it doesn't recognize or has removed a certain cipher, or only allows certain SSL/TLS versions. 

But really, I'm just grasping at straws at this point.   It works on every other server and PXPlus version we've tried outside of their network.  "Works for me" is the default programmer answer :-)

The big question for me is 'why does curl work'?  I think is uses the same libraries as PXPlus, but I'm not certain.  At this point, I might just need to switch the connections to use curl until we figure out the cause of the problem, but that's quite a bit of coding to change and test.

Allen Miglore

Are there symlinks pointing to older libssl/libcrypto .so files?

Loren Doornek

Allen, I was wondering about the symlinks myself, since I know this used to be a more common problem when updating the OS.  But, I can't remember or find any info on where to check for the symlinks used by PXPlus.  If you have some pointers, I'd appreciate it!

Allen Miglore

Here is an example CentOS system, where we created the libssl.so link.  There is also an equivalent libcrytpo.so link.

[root@demo lib64]# ls -l /usr/lib64/libssl.so*
lrwxrwxrwx 1 root root     16 May  2 15:48 /usr/lib64/libssl.so -> libssl.so.1.0.2k
-rwxr-xr-x 1 root root 470360 Mar 12 10:12 /usr/lib64/libssl.so.1.0.2k
lrwxrwxrwx 1 root root     16 May  2 15:47 /usr/lib64/libssl.so.10 -> libssl.so.1.0.2k

Loren Doornek

Well, that's interesting!  The server with a problem (which I'll call P1) doesn't even have the symlinks for libssl.so and libcrypto.so, nor the files that they would normally point to!

$ ldd /usr/pxp12/pxplus
        linux-vdso.so.1 =>  (0x00007ffeedffa000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00000034cc000000)
        libm.so.6 => /lib64/libm.so.6 (0x00000034cc800000)
        libnsl.so.1 => /lib64/libnsl.so.1 (0x00000034ce400000)
        libc.so.6 => /lib64/libc.so.6 (0x00000034cb800000)
        /lib64/ld-linux-x86-64.so.2 (0x00000034cb400000)
$ ls -l /lib64/libssl.so
   ls: cannot access /lib64/libssl.so: No such file or directory
$ ls -l /lib64/libcrypto.so
   ls: cannot access /lib64/libcrypto.so: No such file or directory
$ ll libssl*
   ls: cannot access libssl*: No such file or directory
$ ll libcrypto*
   ls: cannot access libcrypto*: No such file or directory

Now, we do have 3 other servers of various ages (which I'll call S1, S2, & S3, with S3 being the newest), which also run the same PXPlus version 12 (tcb(29)=1250000).  My test program fails on S1 & S2, but works on S3. So, I checked the files on all 3 servers.  The symlinks are exactly the same on all three, but the target files are newer and larger on S3 (the working server)

Logs from S1/S2 (where the test program fails) are below:

   lrwxrwxrwx. 1 root root 16 Sep 21  2016 /lib64/libssl.so -> libssl.so.1.0.1e
   lrwxrwxrwx. 1 root root 19 Sep 21  2016 /lib64/libcrypto.so -> libcrypto.so.1.0.1e
   -rwxr-xr-x. 1 root root 449880 May  9  2016 /lib64/libssl.so.1.0.1e
   -rwxr-xr-x. 1 root root 2017168 May  9  2016 /lib64/libcrypto.so.1.0.1e

Logs from S3 (where the test program works) are below.  Note that the target files are larger/newer:

   lrwxrwxrwx. 1 root root 16 Jun 19  2017 /lib64/libssl.so -> libssl.so.1.0.1e
   lrwxrwxrwx. 1 root root 19 Jun 19  2017 /lib64/libcrypto.so -> libcrypto.so.1.0.1e
   -rwxr-xr-x. 1 root root 454008 Feb 20  2017 /lib64/libssl.so.1.0.1e
   -rwxr-xr-x. 1 root root 2025760 Feb 20  2017 /lib64/libcrypto.so.1.0.1e

The fact that the problem server P1 doesn't even have the libssl.so and libcrypto.so links or targets makes me wonder if those files are even used any more since the program did work fine up until last week!  Mike or Allen, can you confirm whether those links are still needed in PXPlus 12?

I'm tempted to update the target files (libssl.so.1.0.1e and libcrypto.so.1.0.1e) on the S1 & S2 servers so that they match S3, but I have no idea how to update them.  Any suggestions on how or whether those files can be updated?

Again, I appreciate any feedback - it may seem minor or irrelevant, but it may trigger something I haven't looked at, and I'm really curious now to figure out what is causing this!

Allen Miglore

When we install UnForm, we generally have to create the generic libssl.so and libcrypto.so symlinks pointing to whatever the installed version is, if they aren't already there.

My guess would be there is a bug in the older 1.0.1e builds on S1/S2.  You could try 'yum update libssl' and see if that helps.  If they are the same levels of redhat as S3, you might be able to just copy the 1.0.1e files, though I'd sure back up the current ones first.

Loren Doornek

Thanks, Allen - I'll try the yum update later and post the results.

But, on the P1 (bad) server, why would ANY secure connections ever work if the libssl/libcrypto files don't even exist? 

Devon Austen

Are you sure they don't exist?

They may just be in a different folder i.e. /lib/, /lib64/oppenssl/, /usr/lib64, or /usr/lib etc

You may be able to figure out where by doing a "which openssl" and then a ldd on the openssl binary.
Principal Software Engineer for PVX Plus Technologies LTD.

Loren Doornek

Devon - Thanks for that.  I did the 'which openssl' and found "libssl.so.10 => /usr/lib64/libssl.so.10", so looked in the /usr/lib64/ directory (instead of /lib64/).  The libssl and libcrypto links were there.  So, that resolves my concern about how this could ever work in the first place.

Allen - We tried the 'yup update libssl', but it says that's not a valid package.  So we did 'yum update openssl', and it did update the libssl files. But, that didn't resolve the issue.  I suspect there may be something else blocking the connections from S1/S2.

I went back to the P1 (bad) server, and ran the test program in a loop, and found that it is actually connecting occasionally now! (about 25% of the time).  Over the last few days, we haven't been able to connect at all, but they did change some network settings/hardware yesterday since their network speeds were as low as 32kbps (ouch!) , and that seems to have had an effect.  At this point, I'm going to assume that the network issues are the cause, since the fact that we can occasionally connect indicates that the PXPlus/Linux installation is correct.

Alllen, Mike, and Devon - Thanks for all of your input on this!

Loren Doornek

FYI to all - the problem was the network speed.  Some email sync was killing the network.  They changed the routing on the network so that our PXPlus server used a different gateway, and the problems disappeared instantly.