After rebuild openSSL 1.1 with some extra options, like enable-weak-ssl-ciphers, it is able to use with trinity and communication between server and client working fine.
Maybe it is a reason to enable openSSL 1.1 to be used to build new version of trinity?