Dear Folks,
I would like to bring this to your notice about below error. It was one we struggled recently in 12.2 environment. - Follow ML - 2555355.1
Issue - Connection server SSL set credentials failed found in OPMN log files (12.2 Environment).
Reason - (OPMN) and Oracle HTTP Server (OHS) failing to start, as default wallets earlier generated by Oracle Fusion Middleware contain self-signed certificates using MD5withRSA alogrithm.
After applying the OSS patch (O27047184 (initially released July 2019) or its successor Patch 30332467 released in July 2020) , the use of certificates signed by MD5withRSA will fail to work. This is due to the Fusion Middleware (FMW) security library no longer accepting the usage of certificates signed by MD5withRSA and requiring a signature algorithm using SHA256withRSA or better.
Hence we will need to create a empty wallet and add acceptable self-signed certificate using SHA256withRSA algorthm.
Steps to over come above issue after apply the OSS patch and inorder to work OPMN and OHS.
-----------------------------------
1 ) Source Run file system & webtier home
EBSapps.env run
. $FMW_HOME/SetWebtier.env
2) alias orapki=$FMW_HOME/oracle_common/bin/orapki
3)cd $HOME
mkdir Wallet
cd Wallet
Create empty wallet - orapki wallet create -wallet ./ -auto_login_only
add self-signed certificate to wallet - orapki wallet add -wallet . -dn "CN=FMWSmallCircleOfTrust" -asym_alg RSA -keysize 2048 -sign_alg sha256 -self_signed -validity 3652 -auto_login_only
orapki wallet display -wallet . - It should display like below.
Oracle PKI Tool : Version 11.1.1.9.0
Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.
Requested Certificates:
User Certificates:
Subject: CN=FMWSmallCircleOfTrust
Subject: CN=*.flyjacksonville.com,OU=IT,O=Jacksonville Aviati
4) grep -i s_ohs_instance $CONTEXT_FILE -- note it down
5) cd $FMW_HOME/webtier/instances/EBS_web_OHS1 ( EBS_web_OHS1 is the s_ohs_instance variables in CONTEX_FILE).
-bash-4.1$ find . -name cwallet.sso | fgrep -v /webgate/
./config/OHS/EBS_web/keystores/default/cwallet.sso
./config/OHS/EBS_web/proxy-wallet/cwallet.sso
./config/OPMN/opmn/wallet/cwallet.sso
6) Take the backup of defaul wallet in above locations.
cd $FMW_HOME/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web/keystores/default
-bash-4.1$ cp cwallet.sso.lck cwallet.sso.lck_bkp
-bash-4.1$ cp cwallet.sso cwallet.sso_bkp
cd $FMW_HOME/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web/proxy-wallet
-bash-4.1$ cp cwallet.sso.lck cwallet.sso.lck_bkp
-bash-4.1$ cp cwallet.sso cwallet.sso_bkp
-bash-4.1$
cd $FMW_HOME/webtier/instances/EBS_web_OHS1/config/OPMN/opmn/wallet
-bash-4.1$ cp cwallet.sso.lck cwallet.sso.lck_bkp
-bash-4.1$ cp cwallet.sso cwallet.sso_bkp
7) Copy the new SHA-256 signed wallet to all default wallet locations
cd $FMW_HOME/webtier/instances/EBS_web_OHS1
-bash-4.1$ cp ~/home/**/Wallet/cwallet.sso ./config/OHS/EBS_web/keystores/default/cwallet.sso
-bash-4.1$ cp ~/home/**/Wallet/cwallet.sso ./config/OPMN/opmn/wallet/cwallet.sso
-bash-4.1$ cp ~/home/**/Wallet/cwallet.sso ./config/OHS/EBS_web/proxy-wallet/cwallet.sso
8) Obtain the host, port, and user name variables for WLS Admin from CONTEXT_FILE
aPort=$( tr < $CONTEXT_FILE '<>' ' ' | awk '/""s_wls_adminport""/ {print $(NF-1)}' )
-bash-4.1$ echo $aPort
7036
aUser=$( tr < $CONTEXT_FILE '<>' ' ' | awk '/""s_wls_admin_user""/ {print $(NF-1)}' )
-bash-4.1$ echo $aUser
weblogic
iName=$(tr < $CONTEXT_FILE '<>' ' ' | awk '/""s_ohs_instance""/ {print $(NF-1)}' )
-bash-4.1$ echo $iName
EBS_web_OHS1
9) Start Admin server
10) Re-register OHS using the following commands
cd $FMW_HOME/webtier/instances/$iName/bin
./opmnctl unregisterinstance -adminHost $aHost -adminPort $aPort -adminUsername $aUser -instanceName $iName
./opmnctl registerinstance -adminHost $aHost -adminPort $aPort -adminUsername $aUser
11) Start The application services and check opmn is up and have login page.
12) Then bring down the application services.
13) Apply OPMN patch 23716938 patch ( this patch should be applied to ensure that new self-signed wallets generated).
If you don't apply this patch, FS_CLONE will eventually failed and it throws error when it is configuring the OHSfor patch file system.So be aware of this patch.
I could have mention this patch at first. But as i want you to aware the impact of this patch not apply.
Error from clone log file ( It is shown while patch file system OHS configuration in FS_CLONE).
==============
Loc - /u01/app/****/KKIT/fs1/inst/apps/*******/admin/log/clone/ohsT2PApply
Caused by: java.lang.Exception: Unable to start opmnserver of the instance /u01/app/****/KKIT/fs1/FMW_Home/webtier/instances/EBS_web_OHS1 . Check opmn log.
at oracle.as.clone.provisioning.implementations.ASInstanceProvImpl.startInstance(ASInstanceProvImpl.java:380)
at oracle.as.clone.provisioning.implementations.ASInstanceProvImpl.createInstanceAndStart(ASInstanceProvImpl.java:160)
at oracle.as.clone.util.ASInstanceUtil.createStartAndregisterASInstance(ASInstanceUtil.java:318)
... 10 more
Caused by: oracle.as.management.opmn.optic.OpticException: Error in starting opmn server
Operation aborted because of a system call failure or internal error
From opmn logs
=========
2020-07-31T21:11:23-04:00] [opmn] [TRACE:32] [] [internal] ORACLE_HOME: /u01/app/****/KKIT/fs1/FMW_Home/webtier
[2020-07-31T21:11:23-04:00] [opmn] [TRACE:32] [] [internal] ORACLE_INSTANCE: /u01/app/****/KKIT/fs1/FMW_Home/webtier/instances/EBS_web_OHS1
[2020-07-31T21:11:23-04:00] [opmn] [ERROR:1] [] [ons-secure] Connection server SSL set credentials failed (43084)
[2020-07-31T21:11:23-04:00] [opmn] [ERROR:1] [222] [ons-secure] SSL initialization failed
Thanks.