Unable To Generate Fnd Debug Logfiles And No Messages Are Writing To Fnd_log_messages

Dear Folks,

One of 12.2.4 customer facing issue several times to generate fnd log messages after the FND Diagnostics Profile is Set to 'Yes' at the Site Level.

It returns no rows in fnd_log_messages.

SQL> select count(*) from apps.fnd_log_messages;

 COUNT(*)
----------
     0

SQL> select max(LOG_SEQUENCE) from fnd_log_messages;

MAX(LOG_SEQUENCE)
-----------------


As per ML:: 2007068.1, We applied patch 22563940. Then issue resolved.

Expected Behavior
Expect FND Debugging to write to the fnd_log_messages tables.

The issue can be reproduced at will with the following steps:
1. Set FND Debugging

The issue has the following business impact:
Due to this issue, dba/appsdba cannot enable fnd debugging.

Now they are able to see rows in fnd_log_messages table.

Thanks.
Sreeharsha.

Relink of module "FEMCCE" failed after CPU applied.

Dear Folks,

Recently, we've observed relink error while applying CPU (Q1-2019) Patches on to one of non-prod instance.

Relinking module 'FEMCCE' in product fem ...

make -f

/u01/app/***/$TWO_TASK/apps/apps_st/appl/admin/$TWO_TASK/out/link_fem_28563.mk

/u01/app/*****/$TWO_TASK/apps/apps_st/appl/fem/12.0.0/bin/FEMCCE

Starting  link of fem executable 'FEMCCE' on Mon Feb  4 22:21:45 EST 2019

 ORDER BY 2;

make: *** [/u01/app/****/$TWO_TASK/apps/apps_st/appl/fem/12.0.0/bin/FEMCCE]

Error 1

Done with link of fem executable 'FEMCCE' on Mon Feb  4 22:21:45 EST 2019

Relink of module "FEMCCE" failed.

See error messages above (also recorded in log file) for possible

reasons for the failure.  Also, please check that the Unix userid

running adrelink has read, write, and execute permissions

on the directory /u01/app/****/$TWO_TASK/apps/apps_st/appl/fem/12.0.0/bin,

and that there is sufficient space remaining on the disk partition

containing your Oracle E-Business Suite installation.

Done with link of product 'fem' on Mon Feb  4 22:21:45 EST 2019

adrelink is exiting with status 1

Verified, We don't have file exists at OS level.

[****@Kouuda scripts]$ ls -lrt /u01/app/****/$TWO_TASK/apps/apps_st/appl/fem/12.0.0/bin/FEMCCE

ls: cannot access /u01/app/****/$TWO_TASK/apps/apps_st/appl/fem/12.0.0/bin/FEMCCE: No such file or directory

Hence, we copied binary file from production and relinked it manually.

[****@ scripts]$ make -f /u01/app/****/$TWO_TASK/apps/apps_st/appl/admin/$TWO_TASK/out/link_fem_28563.mk /u01/app/****/$TWO_TASK/apps/apps_st/appl/fem/12.0.0/bin/FEMCCE

make: `/u01/app/****/$TWO_TASK/apps/apps_st/appl/fem/12.0.0/bin/FEMCCE' is up to date.

Now Patch completed successfully.

Thanks.

Sreeharsha

APP-FND-01372 : Please register System printer with Application object library.

Dear Folks,

In one of our EBS customer running on 11i. Customer facing issue while switching responsibilities. Below pop-up error window does appear while doing.

APP-FND-01372 : Please register System printer with Application object library.

Solution
=======

Verified, The profile value for Printer is set to "Null"

Hence Change profile value of "Printer" to "noprint" and save changes.

Now,  customer able to switch responsibilities.

Thanks.
Sreeharsha

txkrun.pl (forms deplyoment) failed with TXK::Common::setError & TXK::Process::run

Dear Folks,

In one of our EBS customer, we came across an issue with forms deployment post 10.1.2 patches as part of CPU.

forms deployment failing with below error.

Command error: <rc> = 38400, <command> = /u01/app/****/apps/tech_st/10.1.3/opmn/bin/opmnctl startproc instancename=forms

STACK TRACE

    TXK::Error::abort('TXK::Error','HASH(0x830c890)') called at /u01/app/****/apps/apps_st/appl/au/12.0.0/perl/TXK/Common.pm line 299

    TXK::Common::doError('TXK::Process=HASH(0x8b112d0)','Command error: <rc> = 38400, <command> = /u01/app/****/apps/...','undef') called at /u01/app/****/apps/apps_st/appl/au/12.0.0/perl/TXK/Common.pm line 314

    TXK::Common::setError('TXK::Process=HASH(0x8b112d0)','Command error: <rc> = 38400, <command> = /u01/app/****/apps/...') called at /u01/app/****/apps/apps_st/appl/au/12.0.0/perl/TXK/Process.pm line 449

    TXK::Process::run('TXK::Process=HASH(0x8b112d0)','HASH(0x8add484)') called at /u01/app/****/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl line 1587

    TXK::RunScript::execOPMNControl('HASH(0x8ce7fb8)') called at /u01/app/****/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl line 606

    require /u01/app/****/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl called at /u01/app/****/apps/apps_st/appl/au/12.0.0/perl/TXK/RunScript.pm line 105

    TXK::RunScript::require('TXK::RunScript','/u01/app/****/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txk...') called at /u01/app/****/apps/apps_st/appl/au/12.0.0/perl/TXK/Script.pm line 177

Steps to deploy forms (397174.1).

==============

*** cp $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml.orig_11Aug2018

*** vi $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml

Add OC4J Administrator credentials.

<name>oc4jadmin</name>

    <display-name>OC4J Administrator</display-name>

    <description>OC4J Administrator</description>

    <credentials>!welcome123</credentials>

   </user>

***

[****@jauauaua ~]$ cp $IAS_ORACLE_HOME/opmn/bin/opmnctl $IAS_ORACLE_HOME/opmn/bin/opmnctl.org

[****@jauauaua ~]$ cp $INST_TOP/ora/10.1.3/opmn/bin/opmnctl $IAS_ORACLE_HOME/opmn/bin/opmnctl

[****@jauauaua ~]$ cp $IAS_ORACLE_HOME/opmn/conf/opmn.xml $IAS_ORACLE_HOME/opmn/conf/opmn.xml.org

[****@jauauaua ~]$ cp $INST_TOP/ora/10.1.3/opmn/conf/opmn.xml $IAS_ORACLE_HOME/opmn/conf/opmn.xml

*** Stop the application /adstpall.sh

4.[****@jauauaua ~]$ $FND_TOP/bin/txkrun.pl -script=CfgOC4JApp

*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS

*** Log File = /u01/app/k****/inst/apps/JAJAJ_jauauaua/logs/appl/rgf/TXK/txkCfgOC4JApp_Sat_Aug_11_16_55_58_2018.log

Program : /u01/app/****/k****/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl started @ Sat Aug 11 16:55:58 2018

*** Log File = /u01/app/****/inst/apps/JAJAJ_jauauaua/logs/appl/rgf/TXK/txkCfgOC4JApp_Sat_Aug_11_16_55_58_2018.log

Enter Application name for re-deployment ? forms

Enter Oc4j Instance password for re-deployment ?   <<Enter password welcome123>>

Run Autoconfig <Yes/No> ? No

6.Run AutoConfig on the instance by running the command:

$ $ADMIN_SCRIPTS_HOME/adautocfg.sh

7.Restart the application tier services by running the command:

$ $ADMIN_SCRIPTS_HOME/adstrtal.sh <apps user/apps password>

Solution when form deployment failing with above errors.

======================================

Add -Djdk.crypto.KeyAgreement.legacyKDF=true to oacore,forms,oafm start/stop properties in contextfile.

Ex::

****@jaidkalala ~]$ grep -i oacore $CONTEXT_FILE

                        <forms_jvm_start_options oa_var="s_forms_jvm_start_options">-server -verbose:gc -Xmx256M -Xms64M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Doracle.security.jazn.config=/u01/app/****/inst/apps/JUAUa_JAUAUAdkd/ora/10.1.3/j2ee/forms/config/jazn.xml -Djdk.crypto.KeyAgreement.legacyKDF=true</forms_jvm_start_options>

                        <forms_jvm_stop_options oa_var="s_forms_jvm_stop_options">-server -verbose:gc -Xmx256M -Xms64M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Djdk.crypto.KeyAgreement.legacyKDF=true</forms_jvm_stop_options>

                <oacore_server>

                        <oacore_jvm_start_options oa_var="s_oacore_jvm_start_options">-server -verbose:gc -Xmx1024M -Xms256M -XX:MaxPermSize=160M -XX:NewRatio=2  -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC  -XX:ParallelGCThreads=2  -Dcom.sun.management.jmxremote -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Doracle.security.jazn.config=/u01/app/***/inst/apps/JUAUa_JAUAUAdkd/ora/10.1.3/j2ee/oacore/config/jazn.xml -Doracle.mds.custcache.maxentries=1000 -Djdk.crypto.KeyAgreement.legacyKDF=true</oacore_jvm_start_options>

                        <oacore_oc4j_options oa_var="s_oacore_oc4j_options">-out /u01/app/***/inst/apps/JUAUa_JAUAUAdkd/logs/ora/10.1.3/opmn/oacorestd.out -err /u01/app/***/inst/apps/JUAUa_JAUAUAdkd/logs/ora/10.1.3/opmn/oacorestd.err</oacore_oc4j_options>

                        <oacore_jvm_stop_options oa_var="s_oacore_jvm_stop_options">-server -verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=160M -XX:NewRatio=2  -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC  -XX:ParallelGCThreads=2  -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Djdk.crypto.KeyAgreement.legacyKDF=true</oacore_jvm_stop_options>

                        <oafm_jvm_start_options oa_var="s_oafm_jvm_start_options">-server -verbose:gc -Xmx256M -Xms64M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2  -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy  -Djava.awt.headless=true  -Dhttp.webdir.enable=false -Doracle.security.jazn.config=/u01/app/***/inst/apps/JUAUa_JAUAUAdkd/ora/10.1.3/j2ee/oafm/config/jazn.xml -Djdk.crypto.KeyAgreement.legacyKDF=true</oafm_jvm_start_options>

                        <oafm_oc4j_options oa_var="s_oafm_oc4j_options">-out  /u01/app/***/inst/apps/JUAUa_JAUAUAdkd/logs/ora/10.1.3/opmn/oafmstd.out -err  /u01/app/***/inst/apps/JUAUa_JAUAUAdkd/logs/ora/10.1.3/opmn/oafmstd.err</oafm_oc4j_options>

                        <oafm_jvm_stop_options oa_var="s_oafm_jvm_stop_options">-server -verbose:gc -Xmx256M -Xms64M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2  -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy  -Djava.awt.headless=true  -Dhttp.webdir.enable=false -Djdk.crypto.KeyAgreement.legacyKDF=true</oafm_jvm_stop_options>

** And ran autoconfig

**Forms deployment completed successfully.

Program : /u01/app/****/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl completed @ Wed Nov 7 21:39:02 2018

End of /u01/app/****/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl : No Errors encountered

ML :: 2353710.1

Thanks

Sreeharsha

License A new product/Module in EBS using OAM.

Hello every one,

Let's see how we can do license a new module/product in EBS using Oracle Application Manager.

Here are the steps
==========

**Login as sysadmin user

**Go to System Administrator

**Go to Oracle Applications Manager

**Click on License Manager

**In license section select products.

**Click on Show more options and select License Applications Product

**Select required module and click next and submit.

Post steps

=======

3. Now using adadmin and choose first option 1.

            AD Administration Main Menu

   --------------------------------------------------

   1.    Generate Applications Files menu

   2.    Maintain Applications Files menu

   3.    Compile/Reload Applications Database Entities menu

   4.    Maintain Applications Database Entities menu

   5.    Change Maintenance Mode

   6.    Exit AD Administration

   

Select Generate Application Files menu

Enter your choice [6] : 1

         Generate Applications Files

   ----------------------------------------

   1.    Generate message files

   2.    Generate form files

   3.    Generate report files

   4.    Generate product JAR files

   5.    Return to Main Menu

Run below.

   ***Generate message files

   ***Generate form files

   ***Generate report files

Back to Main Menu

Select Compile/Reload Applications Database Entities menu

Enter your choice [6] : 3

        Compile/Reload Applications Database Entities

   ---------------------------------------------------

   1.    Compile APPS schema

   2.    Compile menu information

   3.    Compile flexfields

   4.    Reload JAR files to database

   5.    Return to Main Menu

Run following.

Compile APPS schema

Compile menu information

Compile flexfields

4. Run autoconfig 

5. Verify Product installed successfully or not using below query.

select fa.APPLICATION_SHORT_NAME, fpi.PATCH_LEVEL, DECODE(fpi.STATUS, 'I','Installed',

'S','Shared', 'N', 'Inactive', fpi.STATUS) Status , fpi.DB_STATUS

from fnd_product_installations fpi,FND_APPLICATION fa

where fpi.APPLICATION_ID in (

select APPLICATION_ID from FND_APPLICATION where APPLICATION_SHORT_NAME like 'AME')

and fa.APPLICATION_ID=fpi.APPLICATION_ID; 

6) Make sure we verify related TOPS available on File system level. 

Thanks.

Sreeharsha

UNNAMED data file in standby after adding new data file in primary – ORA-01111, ORA-01110, ORA-01157 , ORA-01274

Dear Folks,

We came across missing data file in standby after adding new data file in primary which causes MRP job terminate. This could be due to following reasons.

**Either we don't have sufficient  disk space/ASM available at standby host.
**standby_file_management parameter set to MANUAL.

i.e, if standby_file_management parameter is MANUAL on standby, MRP processes couldn't apply archives due to this wrong parameter settings, a UNNAMED data file being created at $ORACLE_HOME/dbs location.

From alert log
==========

File #59 added to control file as 'UNNAMED00059' because
the parameter STANDBY_FILE_MANAGEMENT is set to MANUAL
The file should be manually created to continue.
Errors with log +JDUAUA_FRA/xcmrc_stby/archivelog/2019_02_13/thread_1_seq_9619.319.1000088461
MRP0: Background Media Recovery terminated with error 1274
Errors in file /
ORA-01274: cannot add datafile '+JDUAUA_DATA/xcmrc/datafile/apps_ts_tx_data.299.1000076027' - file could not be created
Managed Standby Recovery not using Real Time Apply

Recovery stopped due to failure in applying recovery marker (opcode 17.30).

So from above log, we clearly identified missing data file=59

SQL> select file#,name from v$datafile where file#=59;

     FILE#

----------

NAME

--------------------------------------------------------------------------------

        59

+JDUAUA_DATA_DATA/xcmrc/datafile/apps_ts_tx_data.299.1000076027

Identify the missing datafile from standby

     FILE#

----------

NAME

--------------------------------------------------------------------------------

        59

/u01/app/oraxx/JDUAUA_DATA/db/tech_st/11.2.0/dbs/UNNAMED00059

Hence created missing data file in Standby.

SQL > alter database create datafile '/u01/app/oraxx/JDUAUA_DATAdb/tech_st/11.2.0/dbs/UNNAMED00059' as '+JDUAUA_DATA_DATA' size 5576m

Database altered.

Enabled standby_file_management to AUTO

SQL> alter system set standby_file_management=AUTO sid='*';

System altered.

Start MRP process.

===========

SQL> recover managed standby database disconnect from session;

Media recovery complete.

Thanks.

Sreeharsha

RMAN-6091: no channel allocated for maintenance (of an appropriate type)

Hello Every one,

Recently, In one of our EBS customer, we came across issue with RMAN backup post CPU Q3 patches  on to 12.1.0.2 instance. The backup failing with below error.

RMAN> delete noprompt archivelog until time 'sysdate -3' backed up 1 times to sbt;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of delete command at 02/03/2019 20:17:05
RMAN-06091: no channel allocated for maintenance (of an appropriate type)

Hence, we've implemented workaround  as per ML 2428682.1

allocate disk channel and delete the datafile copies and archivelog copies.


run {
allocate channel ch1 type 'sbt_tape''";
backup database
include current controlfile ;
}
allocate channel for maintenance device type disk;
crosscheck archivelog all;
delete force noprompt archivelog all;

Workaround::2

============

Apply patch  28432129.

Expected behavior
============

Event after applying the patch 28432129, RMAN may fail with an Error RMAN-6091.

Enhancement Request will be implemented in the future release:" Bug 28593955 : RMAN-6091 ERRORS SEEN EVEN AFTER APPLYING THE PATCH 28432129 "

Note:: This bug 28593955 still yet to implemented by oracle. So we need to apply patch 28432129

Bug fix 28432129 only to allow deletion of archive log and image copies (Not backupsets) , So having some maintenance command mail fail if appropriate channels are not allocated.

Rman never deleted disk backups from SBT channels.

Suggested changes to RMAN scripts to avoid an error.

1) Move all maintenance commands into separate RUN{} or outside RUN{} block as shown in above command and allocate required channels (DISK and TAPE, if RMAN needs to operate on backup files residing on both device types).

2) Have backup commands by allocating whatever device type channels the user wants to create backup files

We should still stick to requirement to have correct device types allocated for maintenance commands operations to work correctly.

Thanks.

Sreeharsha

Oracle XML Database invalid, could not find xdbconfig

Hello Guys,


In one of our EBS customer have many invalids in database and we try to compile them using utrp.sql and eventually failed with below error.


DECLARE
*
ERROR at line 1:
ORA-04063: package body "SYS.DBMS_SQLTUNE" has errors
ORA-06508: PL/SQL: could not find program unit being called: "SYS.DBMS_SQLTUNE"
ORA-06512: at "SYS.UTL_RECOMP", line 875
ORA-06512: at line 4


We've found few components in dba_registry shows INVALID.


COMP_NAME           COMP_ID  VERSION  STATUS
------------------------------ ---------- ---------- ----------
Oracle XML Database      XDB    12.1.0.2.0 INVALID
Oracle Database Packages and T CATPROC  12.1.0.2.0 INVALID
Oracle OLAP API        XOQ    12.1.0.2.0 INVALID


To over come the issue. Please follow below steps.


For CATPROC component:
If there has been no recent database upgrade taking place, the following set of commands may help validating them:
$ sqlplus / as sysdba
@$ORACLE_HOME/rdbms/admin/catalog.sql
@$ORACLE_HOME/rdbms/admin/catproc.sql
@$ORACLE_HOME/rdbms/admin/utlrp.sql
--
-- usually, <utlrp.sql> has to be executed couple of more times for dependency reasons
--
@$ORACLE_HOME/rdbms/admin/utlrp.sql
@$ORACLE_HOME/rdbms/admin/utlrp.sql


for Rest of component:
start up the database in UPGRADE mode and connect ‘/as sysdba’ to execute any of the following scripts.


Oracle XML Database
SQL> @?/rdbms/admin/xdbpatch.sql


OLAP API
SQL> @?/olap/admin/xoqpatch.sql




How to Diagnose Components with NON VALID Status in DBA_REGISTRY after an Upgrade (Doc ID 753041.1)
IF: Invalid Catalog and/or Catproc Database Registry Components (Doc ID 2358487.1)


Thanks.
Sreeharsha

MWA Startup Issue

Hello every one,


Recently we came across MWA bounce in which one of port couldn't able to start. This issue due to  MWA wrapper script which used to stop/start MWA. In our environment we have lower version of mwactlwrpr.sh script. Hence we had to apply 15868470 patch  as per ML DOC :: 1515211.1 to get latest version.


DOC ID:Mobile Web Applications MWA Server Processes Not Starting Using Wrapper Script mwactlwrpr.sh start (Doc ID 1515211.1)

From log file
==========

Stopping MWA Servers and the dispatcher ....
Stopping MWA Server on Port number: 10218
Stopping MWA Server on Port number: 10220
Stopping MWA Server on Port number: 10222
Stopping MWA Dispatcher on Port number: 10809
Killing mwa Dispatcher, if still running..........
Starting MWA Wrapper script. Checking if all ports are free to start the servers and dispatcher...
Starting MWA Server on Port number: 10218
Starting MWA Server on Port number: 10220
PortNumber: 10222 is not free to start the server. Please specify a different port in the configuration file (or) end the process using this port
Starting MWA Dispatcher on Port number: 10809


Before apply patch
===============
$Header mwactlwrpr.sh 120.6.12010000.2
After apply patch
============
$INST_TOP/admin/scripts/mwactlwrpr.sh 120.6.12010000.3

Thanks.
Sreeharsha

ORA-00600: internal error code, arguments: [krbvalmrange_metacnt_mismatch]

Hello every one,


Recently, one of our customer requested us to clone a test environment with mile stone backup which taken 2 weeks back in produciton. Hence we copied backup pieces from source to target using FTP.


But, when we start restoring, it fails with below error.


RMAN-03002: failure of restore command at 02/06/2019 22:19:58
ORA-19870: error while restoring backup piece /backup/JAJAUA/07022019/df_8dto8quo_1_12019%mJAJAJA
ORA-00600: internal error code, arguments: [krbvalmrange_metacnt_mismatch], [400], [13562], [34116], [], [], [], [], [], [], [], []


It is because we copied pieces using FTP and pieces are corrupted.


Solution
=====


Copy backup pieces using SCP command from source to Target.. Make sure not to use FTP command.

Thanks.
Sreeharsha

Javacache.log Flooded With Java.lang.OutOfMemoryError Error Messages

Hello every one,


In one of our customer, a file Javacache.log growing rapidly around 250GB and consuming most  space on mount. By looking into the log file, we've seen  it filled with "Java.lang.OutOfMemoryError Error Messages" . The issue is caused by JVM debug log level was not be set.

we came across one ML doc 1604874.1. However we had to check with oracle for further action plan.


Solution
====


1) Take backup of oc4j.properties


2) Add below parameter in below files.
IASCACHELOGLEVEL=0   >> Value is Zero.

$ORA_CONFIG_HOME/10.1.3/j2ee/oacore/config/oc4j.properties


[appdi@Jauaua1 config]$ diff oc4j.properties_1303470_bkp oc4j.properties
129a130,131
> #Parameter for JavaCachelog_TKT_1303470
> IASCACHELOGLEVEL=0


If autoconfig runs, it overwrite the file and value will be diminish. Hence we changed in template file which associated with the autoconfig generated file oc4j.properties.


To review which configuration file changed as part of next  autoconfig run. Please use below command. It will generate a HTML file and review it. It shows which files will be overwrite in next autoconfig run.


adchkcfg.sh contextfile=<CONTEXT_FILE>


Reviewed generated HTML file and found template file oc4j_properties_1013.tmp associated with oc4j.properties file. Hence we modified in template file. So next autoconfig run wouldn't diminish the value.


$FND_TOP/admin/template/custom/oc4j_properties_1013.tmp


[appdi@uauauacustom]$ cp oc4j_properties_1013.tmp oc4j_properties_1013.tmp_1303470


Add below parameter in template file.


IASCACHELOGLEVEL=0   >>Value is Zero.


[appdi@uauauauacustom]$ diff oc4j_properties_1013.tmp oc4j_properties_1013.tmp_1303470
122,123d121
< ##Parameter for JavaCachelog_TKT_1303470
< IASCACHELOGLEVEL=0
3) Take backup of javacache.log file and nullified it.
4) Bounce OPMN Services.


Thanks.
Sreeharsha

oracle.apps.fnd.framework.OAException: Application: FND, Message Name: FND_NO_DATABASE_CONNECTION

Hello Every one,


Recently one of our customer facing issue with PROD application login. They getting exception when they tried to login.


So further checked oacore logs (application.log)  and identify below error.


Location :: /prod/oracle/PROD/inst/apps/PROD_pchklpp01/logs/ora/10.1.3/j2ee/oacore/oacore_default_group_1


Errors from log file
=============


19/02/08 05:31:39.733 html: chain failed
javax.servlet.ServletException: Application: FND, Message Name: FND_NO_DATABASE_CONNECTION.
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.EvermindPageContext.handlePageThrowable(EvermindPageContext.java:899)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.EvermindPageContext.handlePageException(EvermindPageContext.java:816)
        at _OALogout._jspService(_OALogout.java:243)
        at com.orionserver[Oracle Containers for J2EE 10g (10.1.3.5.0) ].http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
        at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:473)
        at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:594)
        at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:518)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
        at oracle.apps.jtf.base.session.ReleaseResFilter.doFilter(ReleaseResFilter.java:26)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:15)
        at oracle.apps.fnd.security.AppsServletFilter.doFilter(AppsServletFilter.java:318)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:642)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:391)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:908)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:458)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)


For immediate fix, we bounced opmn and it sets every thing. However the issue still repeats. Hence we applied patch 11832737  as per ML DOC :: 1298103.1  for Permanant fix.


R12: Login Fails Intermittently With Error "oracle.apps.fnd.framework.OAException: Application: FND, Message Name: FND_NO_DATABASE_CONNECTION" (Doc ID 1298103.1)

This patch delivers below two files with higher version. Finally issue fixed.


fnd java/framework/webui OARESTUtil.java 120.0.12010000.12
fnd java/framework/webui/rest OABaseRESTHandler.java 120.0.12010000.13


Thanks,
Sreeharsha

ORA-04030: out of process memory when trying to allocate 123416 bytes (QERHJ hash-joi,kllcqas:kllsltba)

Hello Every one,


Recently we'd faced ORA-04030 errors in one of our customer alert log. Customer running on 11.2.0.4 database with 12.1.3 application. So it is expected few bugs in it. We've identified many ML Docs and didn't find exact hint. Hence we had to raise an SR with oracle and finally they came with solution.

There could be several reasons why these errors triggered. Generally  ORA-04030 can be caused by RAM  shortage on non shared server environment, that means oracle processes couldn't get enough amount of RAM to complete it operation  and more over instance having small PGA set , So we need to look at memory structure and set correct parameters as per ML DOC (396009.1) for 11g instance  and also it cause  by not setting enough kernel parameters  to allow enough RAM for processes when running.. 

From alert log
========


ORA-04030: out of process memory when trying to allocate 123416 bytes (QERHJ hash-joi,kllcqas:kllsltba)
ORA-04030: out of process memory when trying to allocate 96 bytes (kkqcs.c.kgght,kkqcs.c.kgght)
ORA-19202: Error occurred in XML processing
ORA-06503: PL/SQL: Function returned without value
ORA-06512: at "APPS.XXAS_COM_HIER_PKG", line 2767


By looking above alert log errors, we dig further and identified a custom package execution taking more PGA memory and finally failed with errors. We can involve developer to tune that package.


We checked same with oracle by providing them a trace and incident file and solutions given as


ACTION PLAN:
==============
1. On OS side edit /etc/sysctl.conf. Add or edit the following line:
vm.max_map_count = 262120
Save the file and load sysctl configuration
$ sysctl -p
Log out and log in again.
2. On database, set hidden parameter _realfree_heap_pagesize_hint:
alter system set "_realfree_heap_pagesize_hint" = 262144 scope=spfile;
Bounce database to reflect changes.


Thanks,
Sreeharsha

Users Receiving Larger Number of Emails For Already Closed Notifications

Hello All,

Recently we'd an issue with one of EBS user who receiving high volume of Emails from workflow around 4000 Emails. Hence we dig the issue deeper by logged in to workflow accout and identified bunch of Undeliverable emails in process folder. Below we could see the message.

Undeliverable: Closed: Aacksonville Aviation Sector- Standard Purchase Order 39535, 0"

So We opened email attachments and found few emails are invalid as below and and NID::442082 mail_status is INVALID and and status is CLOSED.The issue is caused by the mailer processing INVALID responses for CLOSED notifications.

Your message to samathar.b.kaakkt@ciecouldn't be delivered

Your message to mike.c.kakkat@cie.com couldn't be delivered.

Your message to pallu.p.deckler@cie.com couldn't be delivered.

SQL> select RECIPIENT_ROLE,STATUS,MAIL_STATUS,FROM_USER,TO_USER from wf_notifications where NOTIFICATION_ID =442082;

RECIPIENT_ROLE                   STATUS  MAIL_STATUS
-------------------------------------------------- -------- --------------------
FROM_USER           TO_USER
------------------------------ ----------------------------------------
ADHOC245759830598227883PO             CLOSED  NULL
fraier , monter Join Controver INC

The supplier "Join Controver INC" having 4-Emails to which notifications are being sent.

The EBS user who having valid Email and other 3 emails are invalid. So workflow is retrying 2-3 minutes to resend the undeliverable  emails and hence volume. Since one EBS user email is correct. He/She receiving bunch of undelierable email notifications every couple of minutes.

To over come the issue, we set mail_status to NULL. However we need to ask client those emails should be corrected by payable / Procurement team. Otherwise any further Purchase Orders that are communicated to that supplier without invalid emails to be corrected will cause this issue again.

Solutions 

=====

Mark Mail_status to NULL in wf_notifications for particular NID which having issue.

update wf_notifications set mail_status='NULL' where notification_id=442082

OR please follow solutions that refers in below document.

Users Receiving Larger Number of Emails For Already Closed Notifications (Doc ID 1293792.1) Thanks, Sreeharsha

Create scripts directory under appsutil in EBS environment (appsutil/scripts)

Create scripts directory in appsutil in EBS environment (appsutil/scrtips)

When we execute admkappsutil.pl on application server to generate appsutil.zip and unzip on DB node ,we don't find scripts directory under appsutil folder.( $ORACLE_HOME/appsutil)

In order to over come this issue, we need to run perl adbldxml.pl utility ($ORACLE_HOME/appsutil/bin) to  build XML file on database Tier.

This newly created XML file is a central repository for database which intact with application node.

By using newly generated XML file, we need to run adconfig.sh ($ORACLE_HOME/appsutil/bin) on DB node. This will create scripts directory under $ORACLE_HOME/appsutil

Steps

====

1 ) On Application Tier and source the env file.

**Go to $AD_TOP/bin

**perl admkappsutil.pl to create appsutil.zip file

** Log file created at $INST_TOP/admin/out

** Copy appsutil.zip to DB node $ORACLE_HOME

** unzip -o appsutil.zip

** Build new XML for Database using perl  adbldxml.pl ( $ORACLE_HOME/appsutil/bin )

** Run ./adconfig.sh ($ORACLE_HOME/appsutil/bin) using new XML file.

** This will create scripts directory at $ORACLE_HOME/appsutil

[@jajajja~]$ cd $AD_TOP/bin

[***@jajajbin]$ pwd

/u01/app/**/orti/apps/apps_st/appl/ad/12.0.0/bin

[***@jajajajbin]$ perl admkappsutil.pl

Starting the generation of appsutil.zip

Log file located at /u01/app/***/****/inst/apps/orat/admin/log/MakeAppsUtil_02011459.log

output located at /u01/app/***/DAJALA/inst/apps/DAJALA_lkjhga/admin/out/appsutil.zip

MakeAppsUtil completed successfully.

@jajajajbin]$ perl adbldxml.pl

Starting context file generation for db tier..

Using JVM from /u01/app/***/JddadaI/db/tech_st/12.1.0.2/jdk/jre/bin/java to execute java programs..

APPS Password: *****
The log file for this adbldxml session is located at:

/u01/app/*****/DAJALA/db/tech_st/12.1.0.2/appsutil/log/adbldxml_02011519.log

UnsatisfiedLinkError exception loading native library: njni12

java.lang.UnsatisfiedLinkError: no njni12 in java.library.path

Could not Connect to the Database with the above parameters, Please answer the Questions below

Enter Hostname of Database server: jajajajaa

Enter Port of Database server: 1522

Enter SID of Database server: DAJALA

The context file has been created at:

/u01/app/****/DAJALA/db/tech_st/12.1.0.2/appsutil/DAJALA_jaudauasdb01.xml

[@jaudauas bin]$ ./adconfig.sh

Enter the full path to the Context file: /u01/app/****/DAJALA/db/tech_st/12.1.0.2/appsutil/DAJALA_jaudauasxml

Enter the APPS user password:

The log file for this session is located at: /u01/app/****/DAJALA/db/tech_st/12.1.0.2/appsutil/log/DAJALA_jaudauas/02011522/adconfig.log

AutoConfig is configuring the Database environment...

AutoConfig will consider the custom templates if present.
l

Context Value Management will now update the Context file

        Updating Context file...COMPLETED

        Attempting upload of Context file and templates to database...COMPLETED

Updating rdbms version in Context file to db121

Updating rdbms type in Context file to 64 bits

Configuring templates from ORACLE_HOME ...

AutoConfig completed successfully.


Thanks.
Sreeharsha

Start_processes: failed to open STDOUT (1) , catconExec: catconInit has not been run datapatch error

Hi,

We experienced issue with  datapatch run and it had failed with below error.

start_processes: failed to open STDOUT (1)
Installing patches...
catconExec: catconInit has not been run
catconWrapUp: catconInit has not been runPatch installation complete.  Total patches installed: 2

Cause
====

It happens when we have multiple database instances running on a single machine and those OS users don't have privileges on each other's files (sqlpatch files) and as a result datapatch fails while trying to open catcon log files. 

Solution
====

Delete files under: /tmp/sqlpatch_* and re-run datapatch for the affected database.

Datapatch error "CATCONINIT FAILED" when applying patch in a multi-DB environment (Doc ID 1935401.1)

Thanks.
Sreeharsha