Connection server SSL set credentials failed in OPMN log files

 Dear Folks,

I would like to bring this to your notice about below error. It was one we struggled recently in 12.2 environment.    - Follow ML - 2555355.1

Issue - Connection server SSL set credentials failed  found in OPMN log files  (12.2 Environment).

Reason - (OPMN) and Oracle HTTP Server (OHS) failing to start, as default wallets earlier generated by Oracle Fusion Middleware contain self-signed certificates using MD5withRSA alogrithm.

After applying the OSS patch (O27047184 (initially released July 2019) or its successor Patch 30332467 released in July 2020) , the use of certificates signed by MD5withRSA will fail to work. This is due to the Fusion Middleware (FMW) security library no longer accepting the usage of certificates signed by MD5withRSA and requiring a signature algorithm using SHA256withRSA or better.

Hence we will need to create a empty wallet and add acceptable self-signed certificate using SHA256withRSA algorthm.

Steps to over come above issue after apply the OSS patch and inorder to work OPMN and OHS.

-----------------------------------

1 ) Source Run file system & webtier home

EBSapps.env run

. $FMW_HOME/SetWebtier.env

2) alias orapki=$FMW_HOME/oracle_common/bin/orapki

3)cd $HOME

mkdir Wallet

cd Wallet

Create empty wallet -   orapki wallet create -wallet ./ -auto_login_only

add self-signed certificate  to wallet -   orapki wallet add -wallet . -dn "CN=FMWSmallCircleOfTrust" -asym_alg RSA -keysize 2048 -sign_alg sha256 -self_signed -validity 3652 -auto_login_only

orapki wallet display -wallet .    - It should display like below.

Oracle PKI Tool : Version 11.1.1.9.0

Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:

User Certificates:

Subject:        CN=FMWSmallCircleOfTrust

Subject:        CN=*.flyjacksonville.com,OU=IT,O=Jacksonville Aviati

4) grep -i s_ohs_instance $CONTEXT_FILE  -- note it down

5) cd $FMW_HOME/webtier/instances/EBS_web_OHS1   ( EBS_web_OHS1  is  the s_ohs_instance  variables  in CONTEX_FILE).

-bash-4.1$ find . -name cwallet.sso | fgrep -v /webgate/

./config/OHS/EBS_web/keystores/default/cwallet.sso

./config/OHS/EBS_web/proxy-wallet/cwallet.sso

./config/OPMN/opmn/wallet/cwallet.sso

6) Take the backup of defaul wallet in above locations.

cd $FMW_HOME/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web/keystores/default

-bash-4.1$ cp cwallet.sso.lck cwallet.sso.lck_bkp

-bash-4.1$ cp cwallet.sso cwallet.sso_bkp

cd $FMW_HOME/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web/proxy-wallet

-bash-4.1$ cp cwallet.sso.lck cwallet.sso.lck_bkp

-bash-4.1$ cp cwallet.sso cwallet.sso_bkp

-bash-4.1$

cd $FMW_HOME/webtier/instances/EBS_web_OHS1/config/OPMN/opmn/wallet

-bash-4.1$ cp cwallet.sso.lck cwallet.sso.lck_bkp

-bash-4.1$ cp cwallet.sso cwallet.sso_bkp

7)  Copy the new SHA-256 signed wallet to all default wallet locations

cd $FMW_HOME/webtier/instances/EBS_web_OHS1

-bash-4.1$ cp ~/home/**/Wallet/cwallet.sso ./config/OHS/EBS_web/keystores/default/cwallet.sso

-bash-4.1$ cp ~/home/**/Wallet/cwallet.sso ./config/OPMN/opmn/wallet/cwallet.sso

-bash-4.1$ cp ~/home/**/Wallet/cwallet.sso ./config/OHS/EBS_web/proxy-wallet/cwallet.sso

8) Obtain the host, port, and user name variables for WLS Admin from CONTEXT_FILE

aPort=$( tr < $CONTEXT_FILE '<>' ' ' | awk '/""s_wls_adminport""/ {print $(NF-1)}' )

-bash-4.1$ echo $aPort

7036

aUser=$( tr < $CONTEXT_FILE '<>' ' ' | awk '/""s_wls_admin_user""/ {print $(NF-1)}' )

-bash-4.1$ echo $aUser

weblogic

iName=$(tr < $CONTEXT_FILE '<>' ' ' | awk '/""s_ohs_instance""/ {print $(NF-1)}' )

-bash-4.1$ echo $iName

EBS_web_OHS1

9) Start Admin server

10) Re-register OHS using the following commands

cd $FMW_HOME/webtier/instances/$iName/bin

 ./opmnctl unregisterinstance -adminHost $aHost -adminPort $aPort -adminUsername $aUser -instanceName $iName

 ./opmnctl registerinstance -adminHost $aHost -adminPort $aPort -adminUsername $aUser

11) Start The application services and check opmn is up and have login page.

12) Then bring down the application services.

13) Apply OPMN patch 23716938  patch ( this patch should be applied to ensure that new self-signed wallets generated).

If you don't apply this  patch, FS_CLONE will eventually failed and it throws error when it is configuring the OHSfor patch file system.So be aware of this patch.

I could have mention this patch at first. But as i want you to aware the impact of this patch not apply.

Error from clone log file ( It is shown while patch file system OHS configuration in FS_CLONE).

==============

Loc - /u01/app/****/KKIT/fs1/inst/apps/*******/admin/log/clone/ohsT2PApply

Caused by: java.lang.Exception: Unable to start opmnserver of the instance /u01/app/****/KKIT/fs1/FMW_Home/webtier/instances/EBS_web_OHS1 . Check opmn log.

    at oracle.as.clone.provisioning.implementations.ASInstanceProvImpl.startInstance(ASInstanceProvImpl.java:380)

    at oracle.as.clone.provisioning.implementations.ASInstanceProvImpl.createInstanceAndStart(ASInstanceProvImpl.java:160)

    at oracle.as.clone.util.ASInstanceUtil.createStartAndregisterASInstance(ASInstanceUtil.java:318)

    ... 10 more

Caused by: oracle.as.management.opmn.optic.OpticException: Error in starting opmn server

Operation aborted because of a system call failure or internal error

From opmn logs

=========

2020-07-31T21:11:23-04:00] [opmn] [TRACE:32] [] [internal] ORACLE_HOME: /u01/app/****/KKIT/fs1/FMW_Home/webtier

[2020-07-31T21:11:23-04:00] [opmn] [TRACE:32] [] [internal] ORACLE_INSTANCE: /u01/app/****/KKIT/fs1/FMW_Home/webtier/instances/EBS_web_OHS1

[2020-07-31T21:11:23-04:00] [opmn] [ERROR:1] [] [ons-secure] Connection server SSL set credentials failed (43084)

[2020-07-31T21:11:23-04:00] [opmn] [ERROR:1] [222] [ons-secure] SSL initialization failed

Thanks.