Junction always marked as “not available”

Ran into an interesting challenge this week.  Created an SSL Junction to a server.  Had the usual warning about untrusted SSL appear – so I added the public key into the pdsrv keystore and that message disappeared.  But the junction was still marked as down.  And no-matter what I did – the junction would stay marked down.

There really wasn’t much in the logs.  The message log for the instance showed this:

2016-08-04-17:22:40.010+10:00I----- 0x38AD54BA webseald WARNING wiv ssl SSLConnection.cpp 2206 0x7f81cb57c700 -- DPWIV1210W   Function call, gsk_secure_soc_init, failed error: 00000196 GSK_ERROR_IO.

There’s not much in that.  I found an old technote that referenced multi-NIC Unix servers.  But I was running a 9.0.1.0 appliance.  I eventually tried a pdweb.snoop.jct trace:

2016-08-04-16:01:37.000+10:00I----- thread(3) trace.pdweb.snoop.jct:8 /home/webseal/20160415-0046/src/pdweb/webseald/junctions/remote/framework/JunctionConnectionPool.cpp:166: /junctionName CPool> Failed to find an existing connection in the pool!
2016-08-04-16:01:37.001+10:00I----- thread(3) trace.pdweb.snoop.jct:1 /home/webseal/20160415-0046/src/pdwebrte/webcore/amw_snoop.cpp:108:
----------------------------------------
Thread 140700332689152; fd 18; local 10.240.217.151:32896; remote xxx.xxx.xxx.xxx:443
Socket opened.
----------------------------------------
2016-08-04-16:01:37.015+10:00I----- thread(3) trace.pdweb.snoop.jct:1 /home/webseal/20160415-0046/src/pdwebrte/webcore/amw_snoop.cpp:108:
----------------------------------------
Thread 140700332689152; fd 18; local 10.240.217.151:32896; remote <error getting peer name for 18: Transport endpoint is not connected>
Closing socket.

Still not much help.  I tried changing the PING method (from HEAD to GET).  Still nothing.

In desperation — I was looking through the junctioned servers config to try and disable TLS altogether.  When I noticed the Client-Certificate authentication was enabled.  My browser (and curl with -k flag) would happily renegotiate the connection and ignore the Client Cert option.  But the ISAM Reverse Proxy healthcheck process couldn’t.  Disabling that – meant my junction went straight to “online”.