Remediation Action: [Note: No Downtime Required for this type] 1. alter user User_name identified by new_password
Category 3:
1. EDWREP 2. ODM
Remediation Action: [Note: Downtime Required,Shutdown the Applications Tier services on each of the Nodes] 1.FNDCPASS APPS/apps_pwd 0 Y SYSTEM/system_pwd ORACLE [EDWREP/ODM] new_pwd
Category 4:
APPS Base Product Accounts: ABM-ZX password:
ABM AHL AHM AK ALR AMF AMS AMV AMWAP AR ASF ASG ASL ASN ASO ASP AST AX AZ BEN BIC BIL BIM BIS BIV BIX BNE BOM BSC CCT CE CLN CN CRP CS CSC CSD CSE CSF CSI CSL CSM CSP CSR CSS CUA CUE CUF CUG CUI CUN CUP CUS CZ DDD DOM EAA EAM EC ECX EDR EGO ENG ENI EVM FA FEM FII FLM FPA FPT FRM FTE FUN FV GCS GL GMA GMD GME GMF GMI GML GMP GMS GR HR HRI HXC HXT IA IBA IBC IBE IBP IBU IBY ICX IEB IEC IEM IEO IES IEU IEX IGC IGF IGI IGS IGW IMC IMT INV IPA IPD ISC ITG JA JE JG JL JTF JTM JTS LNS ME MFG MRP MSC MSD MSO MSR MST MWA OE OKB OKC OKE OKI OKL OKO OKR OKS OKX ONT OPI OSM OTA OZF OZP OZS PA PJI PJM PMI PN PO POA POM PON POS PRP PSA PSB PSP PV QA QOT QP QRM RG RHX RLA RLM SSP VEA VEH WIP WMS WPS WSH WSM XDO XDP XLA XLE XNB XNC XNI XNM XNP XNS XTR ZFA ZPB ZSA ZX
Remediation Action: [Note: Downtime Required,Shutdown the Applications Tier services on each of the Nodes] 1. FNDCPASS APPS/apps_pwd 0 Y SYSTEM/system_pwd ALLORACLE new_pwd ----At a time all users password will change [ Please keep password same as apps password for these set of users]
FNDCPASS apps/xxx 0 Y system/xxx ORACLE ICX ICX ---For single user
Category 5:
1. PORTAL30 2. PORTAL30_SSO
Remediation Action: [Note: Downtime Required,Shutdown the Applications Tier services on each of the Nodes]
1.FNDCPASS APPS/apps_pwd 0 Y SYSTEM/system_pwd ORACLE [PORTAL30/PORTAL30_SSO] new_pwd
2.Modify APACHE modsql Password on All Middle Tiers: if $IAS_ORACLE_HOME/Apache/modplsql/cfg/wdbsvr.app exist then change APPS password in the file else if $ORACLE_HOME/listener/cfg/wdbsvr.app exist then change APPS password in the file else if $IAS_ORACLE_HOME/listener/cfg/wdbsvr.app exist then change APPS password in the file
Category 6: only in 11i we need to update the files, in R12 this are hardcoded.
Take the backup As a precaution, please back up the FND_ORACLE_ID and FND_USER tables before beginning.
1. APPLSYS 2. APPS 3. APPS_MRC
Remediation Action: [Note: Downtime Required,Shutdown the Applications Tier services on each of the Nodes] 1. FNDCPASS apps/apps_schema_password 0 Y system/system_schema_password SYSTEM APPLSYS new_password Ex: FNDCPASS apps/apps 0 Y system/xx SYSTEM APPLSYS New_password
To update the Apache configuration files with the new password execute a short autoconfig run for the specific templates.
The following command is to be executed on all nodes for the particular instance we are updating. Note the command must be executed on one line.
2.Modify Notification Mailer Password: On All Middle Tier Nodes open $FND_TOP/resource/wfmail.cfg and change APPS password if mentioned in file.
3.Modify Reports CGI Password: On All Middle Tier nodes open (806 OH) $ORACLE_HOME/reports60/server/CGIcmd.dat and change APPS password.
4.Modify APACHE modsql Password on All Middle Tiers: if $IAS_ORACLE_HOME/Apache/modplsql/cfg/wdbsvr.app exist then change APPS password in the file else if $ORACLE_HOME/listener/cfg/wdbsvr.app exist then change APPS password in the file else if $IAS_ORACLE_HOME/listener/cfg/wdbsvr.app exist then change APPS password in the file
5.Drop and Recreate the db links: · sqlplus -s apps/$NEW_PASSWORD · set feedback off; · set head off; · set echo off; · set pagesize 0; · set term off; · set trimspool on; · col owner format a15; · col db_link format a30; · col username format a15; · spool /tmp/db_links_tmp · select owner, db_link, username from dba_db_links where owner in ('APPLSYS','APPS') ; · spool off; · For each DB_LINKS found in upper select statement do the following: 1. sqlplus -s $OWNER/$OWNER_PASSWORD 2. drop database link ; 3. create database link connect to ( APPS OR APPLSYS) identified by "" using '’;
–If you have configured E-Business Suite with 10g Application Server SSO/OID then after changing APPS password, update provisioning profile with new apps password
SCRIPTS TO SEE APPLICATION PRODUCT VERSIONS AND PATCH LEVEL.
SQL> REM ===================================================== REM BEGIN "E-BUSINESS SUITE VERSION INFO" BEGIN REM ===================================================== SPOOL /tmp/apps_version_info.txt set linesize 132 set pagesize 132 col NAME format A20 col HOST format A20 col RELEASE format A12 col VERSION format A12 select i.instance_name NAME, i.host_name HOST, f.release_name RELEASE, i.version VERSION from v$instance i,fnd_product_groups f where upper(substr(i.instance_name,1,4)) = upper(substr(f.applications_system_name,1,4)); SPOOL OFF
SQL> REM ==================================== REM "BEGIN ATG PATCH VERSION CHECKING" REM ==================================== SPOOL /tmp/atg_ver_patches.txt set pagesize 1000; set linesize 141; col "BUG#" format A10; col "PATCHES APPLIED" format A70; select bug_number "BUG#", decode(bug_number, 3262159, 'Patch 11i.FND.H', 3262919, 'Patch 11i.FWK.H', 3126422, '11.5.9 CU1', 3171663, '11.5.9 CU2', 3140000, '11.5.10 Maintenance Pack', 3240000, '11.5.10 Consolidated Update 1 (CU1)', 3640000, '11.5.10.1 Maintenance Pack', 4017300, '11.5.10 Consolidated Update 1 (CU1) for ATG Product Family', 3460000, '11.5.10 Consolidated Update 2 (CU2)', 3480000, '11.5.10.2 Maintenance Pack', 4125550, '11.5.10 Consolidated Update (CU2) for ATG Product Family', 3438354, 'Patch 11i.ATG_PF.H ', 4334965, '11i.ATG_PF.H Rollup 3', 4676589, '11i.ATG_PF.H.RUP4', 5473858, '11i.ATG_PF.H.RUP5', 5903765, '11i.ATG_PF.H.RUP6' ) "PATCHES APPLIED", LAST_UPDATE_DATE "DATE APPLIED" from AD_BUGS where BUG_NUMBER in ('3262919','3262159','3126422','3171633','3140000', '3240000','3640000','4017300','3460000','3480000', '4125550','3438354','4334965','4676589','5473858', '5903765') order by BUG_NUMBER asc; SPOOL OFF
SQL> REM ==================================== REM "BEGIN CHECK AUTOCONFIG/TXK PATCHES" REM ==================================== SPOOL /tmp/txk_adx.patches.txt set pagesize 1000; set linesize 141; col "BUG#" format A10; col "PATCHES APPLIED" format A70; select bug_number "BUG#", decode(bug_number, 4175764, '11i.ADXE.1 FEB 2005 CUMULATIVE UPDATE (autoconfig)', 3453499, 'Patch 11i.ADX.F (autoconfig)', 2682863, 'TXK (FND) AutoConfig TEMPLATE ROLLUP Patch C (March 2003)', 2757379, 'TXK (FND) AutoConfig TEMPLATE ROLLUP Patch D (June 2003)', 2902755, 'TXK (FND) AutoConfig TEMPLATE ROLLUP Patch E (August 2003)', 3104607, 'TXK (FND) AutoConfig TEMPLATE ROLLUP Patch F (December 2003)', 3239694, 'TXK (FND) AutoConfig TEMPLATE ROLLUP Patch G (February 2004)', 3416234, 'TXK (FND) AUTOCONFIG TEMPLATE ROLLUP PATCH H (May 2004)', 3594604, 'TXK (FND) AUTOCONFIG TEMPLATE ROLLUP PATCH I (October 2004)', 3950067, 'TXK (FND) AUTOCONFIG TEMPLATE ROLLUP PATCH J (FEBRUARY 2005)', 4104924, 'TXK (FND) AUTOCONFIG TEMPLATE ROLLUP PATCH K (July 2005)', 4489303, 'TXK (FND) AUTOCONFIG TEMPLATE ROLLUP PATCH L (SEP 2005)', 4709948, 'TXK (FND) AUTOCONFIG TEMPLATE ROLLUP PATCH M (APRIL 2006)', 5107107, 'TXK (FND/ADX) AUTOCONFIG ROLLUP PATCH N (AUGUST 2006)', 5478710, 'TXK (FND/ADX) AUTOCONFIG ROLLUP PATCH O (December 2006)', 5759055, 'TXK (FND/ADX) AUTOCONFIG ROLLUP PATCH P (MAR/APR 2007)', 5985992, 'TXK (FND/ADX) AUTOCONFIG ROLLUP PATCH Q (JUL/AUG 2007)', 6323691, 'TXK AUTOCONFIG AND TEMPLATES ROLLUP PATCH R (OCT/NOV 2007)', 6372396, 'TXK AUTOCONFIG AND TEMPLATES ROLLUP PATCH S (APRIL/MAY 2008)' ) "PATCHES APPLIED", b.LAST_UPDATE_DATE "DATE APPLIED" from AD_BUGS b where b.BUG_NUMBER in ('4175764','3453499','2682863','2757379','2902755', '3104607','3239694','3416234','3594604','3950067', '4104924','4489303','4709948','5107107','5478710', '5759055','5985992','6323691','6372396') order by bug_number asc; SPOOL OFF
SQL> REM ======================================================= REM "BEGIN Extended fnd_nodes query to cover web_host,etc." REM ======================================================= spool /tmp/fndnodes_oss.txt set pagesize 50 col node_name format a15 col server_id format a8 col server_address format a15 col platform_code format a4 col webhost format a12 col domain format a20 col virtual_ip format a12 set linesize 132 select node_id, platform_code, support_db D, support_cp C, support_admin A, support_forms F, support_web W, node_name, server_id, server_address, domain, webhost, virtual_ip from fnd_nodes order by node_id; SPOOL OFF
SQL> REM ============================= REM "BEGIN LOGIN PROFILE OPTIONS" REM =============================
spool /tmp/login_profiles.txt
set pagesize 100; set linesize 133; set feedback off; set verify off;
col profile_option_name format a35 col profile_option_value format a60 col PROF_LEVEL format a30 break on profile_option_name
select p.profile_option_name, decode(v.level_id, 10001,'SITE', 10002, (select 'App:'||a.application_short_name from fnd_application a where a.application_id = v.level_value), 10003, (select 'Resp:'||f.RESPONSIBILITY_KEY from fnd_responsibility f where f.responsibility_id = v.level_value), 10004, (select 'User:'||u.user_name from fnd_user u where u.user_id = v.level_value), 10005, (select 'Server:'||n.node_name from fnd_nodes n where n.node_id = v.level_value), 10006, (select 'Org:'|| o.name from hr_operating_units o where o.organization_id = v.level_value), 10007, 'ServResp', 'NOT SET') PROF_LEVEL, nvl(v.profile_option_value,'NOT SET') profile_option_value from fnd_profile_options p, fnd_profile_option_values v where p.profile_option_id = v.profile_option_id (+) and p.application_id = v.application_id (+) and upper(p.profile_option_name) in ('APPLICATIONS_HOME_PAGE','APPS_SSO_LOCAL_LOGIN','APPS_SSO') order by 1, v.level_id; SPOOL OFF
SQL> REM =================================================== REM Shows us the values of all web like profile options REM =================================================== SPOOL /tmp/web_urls.txt set linesize 132 set pagesize 132 col NAME format A25 col LEVEL_SET format a15 col CONTEXT format a10 col VALUE format A20 wrap col Server format a10 col resp format a8 wrap col application format a10 break on NAME select n.user_profile_option_name NAME, decode(v.level_id, 10001, 'Site', 10002, 'Application', 10003, 'Responsibility', 10004, 'User', 10005, 'Server', 10006, 'Organization', 10007, 'ServResp', 'UnDef') LEVEL_SET, decode(to_char(v.level_id), '10001', '', '10002', app.application_short_name, '10003', rsp.responsibility_key, '10004', usr.user_name, '10005', svr.node_name, '10006', org.name, '10007', 'depends=', v.level_id) "CONTEXT", v.profile_option_value VALUE, (select n.node_name from fnd_nodes n where n.node_id=level_value2) Server, decode(v.LEVEL_VALUE, -1, 'Default', rsp.responsibility_key) Resp, decode(LEVEL_VALUE_APPLICATION_ID, -1, 'Default', app.application_short_name) Application from fnd_profile_options p, fnd_profile_option_values v, fnd_profile_options_tl n, fnd_user usr, fnd_application app, fnd_responsibility rsp, fnd_nodes svr, hr_operating_units org where p.profile_option_id = v.profile_option_id (+) and p.profile_option_name = n.profile_option_name and (upper(v.profile_option_value) like '%HTTP%') and usr.user_id (+) = v.level_value and rsp.application_id (+) = v.level_value_application_id and rsp.responsibility_id (+) = v.level_value and app.application_id (+) = v.level_value and svr.node_id (+) = v.level_value and org.organization_id (+) = v.level_value order by name, level_set; SPOOL OFF
3a. From an UNX shell that has the Apps environment sourced on your external middle tier
[ Part I ] - cut and paste the following commands in the shell
Please use the full path to $LOG_HOME e.g. OraLogDir /u01/inst/apps/JCB_atg/logs/ora/10.1.3/Apache/oracle
Warning: the log.xml file created by the http ODL log can get very large. Diligence must be taken to monitor this file and maintain its size under the 2GB limit which when exceeded can cause issues with login.
Make the following directory which will be where the Apache ODL log files are created
Increase OC4J logging for oacore
1/ edit j2ee-logging.xml, adjust the following in file: $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/config/j2ee-logging.xml 2/ edit orion-web.xml, adjust the following in file: $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/application-deployments/oacore/html/orion-web.xml
param-name>debug_mode true
Increase OC4J logging for forms 1/ edit j2ee-logging.xml, adjust the following in file: $ORA_CONFIG_HOME/10.1.3/j2ee/forms/config/j2ee-logging.xml
Increase OC4J logging for OAFM
1/ edit j2ee-logging.xml, adjust the following in file: $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/config/j2ee-logging.xml
2/ edit orion-web.xml, adjust the following in file: $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/application-deployments/oafm/webservices/orion-web.xml
param-name>debug_mode true
Increase OPMN Logging
edit opmn.xml adjust the following in file: $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml
a-1. $ cd $IAS_ORACLE_HOME/Apache/Apache/logs b-1. $ pwd (verify that you are in the "logs directory") c-1. $ rm -r *
a-2. $ cd $IAS_ORACLE_HOME/Apache/Jserv/logs/jvm b-2. $ pwd (verify that you are in the "jvm" directory) c-2. $ rm -r *
a-3. $ cd $IAS_ORACLE_HOME/Apache/Jserv/logs b-3. $ pwd (verify that you are in the "logs" directory) c-3. $ rm *
2. Edit both the $IAS_ORACLE_HOME/Apache/Apache/conf/httpd.conf file and $IAS_ORACLE_HOME/Apache/Apache/conf/httpd_pls.conf to enable debugging for apache.
LogLevel debug
3. Edit the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.conf to enable debug for the jserv module.
ApJServLogLevel debug ApJServLogFile
4. Edit the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties to enable debug for the java portion of the Apache jserv.
6. Start the apache server. UNIX users will typically use "adapcctl.sh start" while windows users will simply start the service from the services applet.
7. Reproduce the problem
8. Immediately shutdown the apache server. UNIX users will typically use "adapcctl.sh stop" while windows users will simply stop the service from the services applet.
9. Gather up all of the following logfiles and configuration files for analysis put these into the following zip file, which will be placed under /tmp directory...
General Information to Troubleshoot Apache Issues.
1)If you mistype any directive in Apache (httpd.conf or any conf file), Apache will not start & it will not write even in error_log. In these case it will report any error message to file $COMMON_TOP/admin/log/$CONTEXT_NAME/adapcctl.txt
So first check in the above file
2)Found *.pid file in Apache_Top/Apache/logs/ even after bounce of an Apache web server (When Apache starts it creates .pid file in logs directory and when you shutdown apache it cleans this .pid file). So if this file is already there then Apache will not start. Fix: Simply move this file
3)Could not bind port (Port already in Use) or Sometimes apache and concurrent Manager will interchange their port no.’s while starting while results error and both of them will not up Fix: To overcome this up the services in this manner 1)Listener 2)Apache web server 3)Concurrent manager 4)adstrtal.sh
4)Error 500 - Internal server error Fix: 1) Clear Cache (Application Server Cache)and start the services
5)If there are no error messages in error_log or error_log_pls then there might be issues in starting Jserv component of web server (which caters Java requests by mod_jserv)
6)Mobile is enabled in zone.properties disable it if you are not using it.
7)If you don't know port number of your web server you can check it in file $IAS_ORACLE_HOME/Apache/Apache/conf/httpd.conf with directive Listen & Port
8)Once you are able to access http://hostname.domainname:port and if you are getting any error message that means your web server (Core Apache) is working but there are various components called as modules and there may be issues with those modules which we will check these modules . few modules of our interest are -mod_jserv -mod_pls -mod_oprocmgr -mod_ossl (If you have ssl enabled on web server) -mod_osso (If single sign on server is enabled on web server)
9)Once in a while adapcctl.sh start will not work then ultimate you have to go and invoke apachectl script file in Apache top/apache/bin to start Apache webserver.
10)If we remove _pages from COMMON_TOP it will be created for the next time only when we bounce and up the Apache web server.
11)Jsp pages are complied with the file ojspcompile.pl executable. It is located at $JSP_TOP/bin/ and in R12 it is there in $FND_TOP/patch/115/bin/
12)Error : Your session is no longer valid while trying to loging to applications. The issue is resolved by changing the profile option session_cookie_domain to null from domain.com
13)HTTP-403 forbidden message after entering the username and password in the "Apps Logon Links" page. [Not authorised to view this page] Check for Apps password in $APACHE_TOP/modplsql/cfg/wdbsvr.app ,Comment out custom_auth entry in $APACHE_TOP/modplsql/cfg/wdbsvr.app and bounce apache.
14)Error message: You have insufficient privileges for the current operation. Verify and update SESSION_COOKIE_DOMAIN in ICX_parameters if it is wrong.
Example :
update icx_parameters set SESSION_COOKIE_DOMAIN='NEW_VALUE' where SESSION_COOKIE_DOMAIN='OLD_VALUE'
15)AppsLocalLogin?.jsp page error outs….
Edit the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties
Set wrapper.bin.parameters=-DLONG_RUNNING_JVM=false
add the line wrapper.bin.parameters=-DCACHEMODE=LOCAL
Restarted Apache for these changes to take effect and retested the issue
16) Login to the applications URL fails with 500 Internal server Error java.lang.NoClassDefFoundError at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:126) at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:170) at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.HttpApplication.loadServlet(HttpApplication.java:2231) at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:4617) at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:4541) at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0)
Performed the below steps, but it did not resolved the issue 1) Bounced Apache server, cleared cache and validated ports 2) Bounced all Middle tier services and killed defunct processes 3) Bounced Unix Server
Solution ======= Java object Caching Configuration parameter was Enabled in oc4j.properties $INST_TOP/ora/10.1.3/j2ee/oacore/config Disabled LONG_RUNNING_JVM=false from True
Bounced Apache server and could login to the applications URL without issues
ISSUES AND SOLUTIONS
ISSUE 1:
Description
When Apache script adapcctl.sh is issued.It shows that Apache is started.On checking the status Apache is stopped
Cause
The error log file in the folder cd $APACHE_TOP/Apache/logs
Showed
fopen: No such file or directory httpd: could not log pid to file /var/opt/_/Apache/logs/httpd_pls.pid
SOLUTION 1
Created the missing directories /_/Apache/logs under /var/opt folder Bounced Apache
ISSUE 2:
Description oacore was not comming up in the configurator node. opmnctl status
ias-component process-type pid status
OC4J oacore N/A Down OC4J oacore N/A Down OC4J oacore N/A Down OC4J oacore N/A Down
14:11:56 Error initializing server: //inst/apps/_/ora/10.1.3/j2ee/oacore/config/server.xml, Fatal error at line 70 offset 1 in file://inst/apps/_/ora/10.1.3/j2ee/oacore/config/server.xml: .: XML-20100: (Fatal Error) Expected 'EOF'.
SOLUTION 2
Edit the /sid/inst/apps/SID_hostname/ora/10.1.3/j2ee/oacore/configserver.xml file and remove the extra line in the file. Restart and check oacore will be starting. This sudden change in server.xml seems to be because of a bug. Bug - 6702510
Edited the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties a) Set wrapper.bin.parameters=-DLONG_RUNNING_JVM=false b) Add the line wrapper.bin.parameters=-DCACHEMODE=LOCAL c) Restarted Apache for these changes to take effect and retested the issue
ISSUE 4
New Message when starting Apache using adapcctl.sh start
Cause
It takes 7/10 minutes to clear this cache directory if present apache Web Server Listener is not running.
Starting Apache Web Server Listener (dedicated HTTP) ...
Removing gantt cache directory
Successfully removed the gantt cache directory:/tcge1i/applmgr/common/html/cabo/images/cache/gantt
Apache Web Server Listener (PLSQL) is not running.
Starting Apache Web Server Listener (dedicated PLSQL) ...
SOLUTION 4
Wait for atleast 7 to 10 minutes and this cache directory will be cleared and Apache will be started successfully
ISSUE 5
Apache server is up and running. But adapcctl.sh status shows adapcctl.sh version 115.54 Apache Web Server Listener is not running. Apache Web Server Listener (PLSQL) is not running.
adapcctl.sh: exiting with status 1
Cause
Incorrect Lock file path given in httpd.conf LockFile? $APACHE_TOP/Apache/log/PUPONI_httpd.lock and Incorrect path given in adapcctl.sh script.
SOLUTION 5:
Modified the path of lock file in $APACHE_TOP/Apache/conf/httpd.conf file. LockFile? /var/opt/SID_/httpd.lock and Modified the path of adapcctl.sh file by commenting the wrong path
ISSUE 6
After selecting the E-Business home page got internal server error Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, and inform them of the time the error occurred, and anything you might have done that may have caused the error. More information about this error may be available in the server error log.
Cause Found that all the properties files as below jserv.properties jserv_restrict.properties zone.properties zone_restrict.properties were pointing to a dbc file name _.dbc which was not physically present at the path /sid/applmgr/11510/fnd/11.5.0/secure/_ $ cd //applmgr/11510/fnd/11.5.0/secure/_ $ ls .dbc
SOLUTION 6:
copied the file twwyai.dbc which was physically present at path //applmgr/11510/fnd/11.5.0/secure/_ to the dbc file name which was being pointed by all the properties files $ cp .dbc _>sid>.dbc $ ls _>sid>.dbc .dbc This resolved the issue.
ISSUE 7 Instance is very slow / inaccessible with a warning on login page: Warning
Low-level logging is currently enabled. Your application will not perform as well while Low-level logging is on.
SOLUTION 7 1. Change these values to: FND: Debug Log Enabled -- No FND: Debug Log Level -- ( Blank ) FND: Debug Log Module -- ( Blank ) SLA: Enable Diagnostics -- No FND Validation Level -- None
2. Bounce Apache on all the MTs and clear Apache cache.
ISSUE 8
Login to the applications URL fails with 500 Internal Server Error java.lang.NoClassDefFoundError at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect?.java:126) at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect?.java:170) at com.evermind[Oracle Containers for J2EE? 10g (10.1.3.0.0) ].server.http.HttpApplication.loadServlet(HttpApplication?.java:2231) at com.evermind
Performed the below steps, but it did not resolved the issue 1) Bounced Apache server, cleared cache and validated ports 2) Bounced all Middle tier services and killed defunct processes 3) Bounced Unix Server
Cause No Cause Entered
SOLUTION 8
Java object Caching Configuration parameter was Enabled in oc4j.properties $INST_TOP/ora/10.1.3/j2ee/oacore/config Disabled LONG_RUNNING_JVM=false from True
Bounced Apache server and could login to the applications URL without issues
Java Caching is a mechanism used to improve performance by storing data in the Middle Tier JVM memory, instead of retrieving the data from the database every time it is requested.
ISSUE 9
When trying to access the URL (Either Big IP or the node level), it hangs for a long time and getting redirected to a different URL and gives a blank page. For example, When try to access the ipayments URL which is hosted on node2, it hangs for a long time and its getting redirected towards the Sales URL which is hosted on node3, and displays a blank page.
Cause All the host reference and the URL directives are pointing towards node3 and sales URL respectively, in the below files. Hence every attempt tries to launch the sales URL and get fails as the apache configuration files and parameters are pointing towards the correct one. jserv.conf forms.properties zone.properties jserv.properties
SOLUTION 9
Stop Apache. Edit the files jserv.conf, forms.properties, zone.properties, jserv.properties, jserv.conf to replace the host reference and the URL directives to the correct one (In our case, its node3 . Clear the cache and restart the Apache.
ISSUE 10
Apache login issues occurs in multi MT environment. Unable to login to oracle applications.
Cause Issue due to distributed cache.
SOLUTION 10
Set is_distributed to false in javacache.xml Bounce Apache.
ISSUE 11
Apache server is down. Apache logfile did not provide useful information. Started the Apache server, but the Apache Web listener did not started. 1) Verified no log files (Apache,Jserv) crossed 2GB limit. error_log, access.log, mod_jserv 2) Veified that the port is listening (netstat -an |grep ) Apache port(httpd.conf), pls listener(httpd_pls.conf) 3) Verified that the Apache and Jserv configuration files are not changed recently httpd.conf & Jserv.conf 4) Shutdown all the Middle tier services and check for ipcs (semaphores) did not resolve the outage. $ ipcs on application middle tier as Application user ap
Cause No Cause Entered
SOLUTION 11
Rebooted the server and started the services. All services came back normal.
ISSUE 12
Unable to login to the home page URL. Login fails with the error: Error Page You have encountered an unexpected error. Please contact the system administrator for assistance.
Click here for exception details
Checks:
Upon clicking the link, can observe the error: oracle.apps.fnd.framework.OAException: Application: FND, Message Name: SQL_PL SQL_ERROR. Tokens: ROUTINE = createSession(int)(userId=6,sessionMode='null','2D8 FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412?'); REASON = java.s ql.SQLException: ORA-01422: exact fetch returns more than requested number of ro ws ORA-06512: at "APPS.FND_SESSION_MANAGEMENT", line 295 ORA-06512: at "APPS.FND_SESSION_MANAGEMENT", line 455 ORA-06512: at line 1 ; ERRNO = 1422;
Cause FND_NODES table was updated with the same server_id for all Nodes.
SOLUTION 12
Update the table FND_NODES with correct server_ids for all nodes as in $FND_TOP/secure/sid_host/sid.dbf Bounced apache on all apache nodes.
and Run select node_name, server_id from fnd_nodes; to confirm the values.
Users getting '404 page not found errors' when tyring to access the URL
(OR)
Getting Exception in thread "main" java.lang.ClassFormatError: Truncated class file
(OR)
adstrtal.sh or adstpall.sh is throwing below error message
You are running adstrtal.sh version 115.16
Exception in thread "main" java.lang.ClassFormatError: oracle/apps/ad/context/AppsContext (Truncated class file) at java.lang.ClassLoader.defineClass0(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader?.java:539) at java.security.SecureClassLoader.defineClass(SecureClassLoader?.java:123) at java.net.URLClassLoader.defineClass(URLClassLoader?.java:251) at java.net.URLClassLoader.access$100(URLClassLoader?.java:55) at java.net.URLClassLoader$1.run(URLClassLoader?.java:194) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader?.java:187) at java.lang.ClassLoader.loadClass(ClassLoader?.java:289) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:274) at java.lang.ClassLoader.loadClass(ClassLoader?.java:235) at java.lang.ClassLoader.loadClassInternal(ClassLoader?.java:302) at oracle.apps.ad.autoconfig.ServiceControl.(ServiceControl?.java:319) at oracle.apps.ad.autoconfig.ServiceControl.main(ServiceControl?.java:662) Check logfile //applmgr/common/admin/log/_/08061104.log for details
Exiting with status 1
(OR)
[06/08/2007 11:03:12:181] (ERROR) an error returned handling request via protocol "ajpv12" [06/08/2007 11:03:12:181] (EMERGENCY) ajp12: can not connect to host 140.85.188.36:20021 [06/08/2007 11:03:13:184] (EMERGENCY) ajp12: can not connect to host 140.85.188.36:20021 [06/08/2007 11:03:15:038] (EMERGENCY) ajp12: auth did not receive challenge size [06/08/2007 11:03:15:038] (EMERGENCY) ajp12: auth fail
Cause
Extract from the JVM log file: //product/iAS/Apache/Jserv/logs/jvm/OACoreGroup.0.stderr Exception in thread "main" java.lang.ClassFormatError: Truncated class file at org.apache.jserv.JServServletManager.load_init(JServServletManager?.java:765) at org.apache.jserv.JServServletManager.loadServlet(JServServletManager?.java:659) at org.apache.jserv.JServServletManager.loadStartupServlets(JServServletManager?.java:789) at org.apache.jserv.JServServletManager.init(JServServletManager?.java:447) at org.apache.jserv.JServ.start(JServ.java:625)
Verified that there are no log files that exceed 2GB limit (OS limit). Clearing the Apache and Jserv cache did not resolve the issue. Unmount / mount product specific file system did not resolve
SOLUTION 13:
Try each of these solutions until the problem is resolved, in order to have the least impact on the system. 1. If you can determine the specific class file from the log files, use the 'touch' command.
e.g. touch classfile
If necessary, restart Apache and clear server cache.
2. Shut down all MT services, exit the Apps owner and ask the System Admin to unmount and remount the /sid/applmgr file system (mount point above $JAVA_TOP). Ask the System Admin to kill any processes that may be holding the file system open. Then log in to the Apps owner and restart MT services.
3. Shutdown all the MT services and ask System Admin to reboot the server if the actions above don't resolve the outage.
ISSUE 14:
Got login error - Internal Server Error after bounced all services on . find following message from Jserv log file:
$ tail -8 mod_jserv.log [07/05/2007 18:41:06:915] (ERROR) ajp12: Servlet Error: java.lang.NoClassDefFoundError: null [07/05/2007 18:41:06:916] (ERROR) an error returned handling request via protocol "ajpv12" [07/05/2007 18:41:06:916] (ERROR) balance: 11906 internal servlet error in server auohsnops03.oracleoutsourcing.com:20720 [07/05/2007 18:41:06:916] (ERROR) an error returned handling request via protocol "balance" [07/05/2007 20:07:52:732] (ERROR) ajp12: Servlet Error: java.lang.NoClassDefFoundError: null No runaway, no lock on port 20720.
SOLUTION 14:
Shutdown all services on MT. Removed all log file (standard error...) from Jserv/jvm. Started All services with successful login.
ISSUE 15
Customer reported Not able to login with “Internal Server Error” on .
We could not start some of services during bounced All Services with following error message: adapcctl.sh version 115.47 Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Apache Web Server Listener is not running (dedicated HTTP) ... Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error. Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Apache Web Server Listener is already stopped. Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error. adapcctl.sh: exiting with status 2
Unable to login the application URL, facing the error while login:
Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, ap@domain.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.
Cause:
SQL> select END_DATE from fnd_user where user_name='GUEST';
END_DATE --------- 05-OCT-10
SOLUTION 15
We found out the apps and applsys were not in sync. Just manual change the apps password from sqlplus. Restarted all services and login successfully.
or
Update the FND_USER table with END_DATE=null and bounced apache SQL> update fnd_user set END_DATE=null where user_name='GUEST';
1 row updated.
SQL> commit;
Commit complete.
SQL> select END_DATE from fnd_user where user_name='GUEST';
END_DATE -----------
ISSUE 16
Unable to login to oracle applications: error message: You have insufficient privileges for the current operation.
Checks
Unable to login to oracle applications: error message: You have insufficient privileges for the current operation.
SOLUTION 16
Verify and update SESSION_COOKIE_DOMAIN in ICX_parameters if it is wrong.
Example :
update icx_parameters set SESSION_COOKIE_DOMAIN='NEW_VALUE' where SESSION_COOKIE_DOMAIN='OLD_VALUE';
ISSUE 17
Apache home page is not coming up even after bounced Found in jvm log asking for java version
SOLUTION 17
//product/iAS/Apache/Apache/bin/java.sh: /usr/java/jdk1.3.1_15/bin/java: not found
Check the version in /usr/java and update that version in java.sh file and Bounced Apache.
ISSUE 18
Unable to access applications
cause All applications are up and running. "Solaris Error: 12: Not enough space" errors are encounterd in database listener logfile.
SOLUTION 18:
Check if the OS is SUN Solaris. Check if /tmp filesystem is 100% full. Check Swap space utilization. Check the application process that is hagging resources. (Check if any sql sessions are hagging resources) Clear the process in issue after getting required approval.
NOTE: The above is applicable only for environments on Solaris OS.
ISSUE 19
Not able to login to APPS. JSP Error: Unable to create an user session or ORA-01403 : no data found has been detected in FND.SIGNON.AUDIT_USER
Checks
Check number of rows in dual table. Select * from dual; DUAL table should always contain only one row. The table named DUAL is a small table in the data dictionary that Oracle and user-written programs can reference to guarantee a known result. This table has one column called DUMMY and one row containing the value X.
SOLUTION 19
DUAL table should always contain only one row and one column. Insert a row with value X into dual table if the dual table contains 0 rows. If the dual table contains more than one row then delete the rows by keeping one row.
The rows in dual table not equal to zero.
ISSUE 20
Not able to launch forms from PHP
Checks
Login to PHP.
Select system administrator responsibility and click on any menu option. The page shows "done" without launching forms (jinit) Check the apps version. If the apps version is 11.5.10 then check the Check the file version of $FND_TOP/admin/template/appsweb.cfg If the file version is 115.148, then verify that "!WindowsDPI= " exists in $FORMS60_WEB_CONFIG_FILE The windowsDPI value should be 96. If the value for WindowsDPI is not set in the $FORMS60_WEB_CONFIG_FILE then update PROFILE_OPTION_VALUE.
SOLUTION 20
select PROFILE_OPTION_VALUE from fnd_profile_option_values where PROFILE_OPTION_ID=3769; Update the tar with the output of the above command.
UPDATE fnd_profile_option_values set PROFILE_OPTION_VALUE='https://.domainname.com/dev60cgi/f60cgi?windowsDPI=96' where PROFILE_OPTION_ID=3769; This should update only one row. commit;
ISSUE 21
Signature for "https URL redirecting to http URL" Issue: https URL redirecting to http URL and the page cannot be displayed error occurs.
SOLUTION 21
Check the httpd.conf and url_fw.conf, in url_fw.conf the Rewrite rule was RewriteRule ^/$/OA_HTML/AppsLocalLogin.jsp. Changed into RewriteRule https://: /OA_HTML/AppsLocalLogin.jsp.
what is web server? A Web Server is a computer program that is responsible for accepting HTTP requests from clients, which are known as web browsers, and serving them HTTP responses along with optional data contents, which usually are web pages such as HTML documents or linked objects (images, etc.).
Basic process looks very simple. Your browser connects to the Web Server and requests a page and the server sends back the requested page.
Example:
When u type www.oracle.com .The Browser Breaks the URL into three Parts:
The protocol ("http")
The server name ("www.oracle.com")
The file name defined under Directory Index ("index.html")
The browser communicates with DNS to translate the server name www.oracle.com into an IP Address, which it uses to connect to the server machine.
The browser then forms a connection to the server at that IP address.
Using the HTTP protocol, the browser then sends a GET request to the server, asking for the file "http://www.oracle.com/index.html."
The server then sends the HTML text for the Web page to the browser.
The browser read the HTML tags and then formats the page onto your screen
What is Apache Web Server then?
For Oracle Applications, the Web Server is the Oracle Application Server, which is based on Apache technology.
Oracle people has Customized Apache by adding extra features/component/plugins like mod_plsql, mod_jserv, mod_osso … in to Apache Web server and released it as Oracle-Apache Server.
The Oracle Application server is also called as “iAS”
Oracle Apache Server, at startup, starts the parent process. This process loads the entire configuration and associated modules and spawns a preconfigured number of child processes. Apache Server, thus, actually runs several copies of itself to handle multiple connections simultaneously. The parent process never listens to any HTTP request. Its sole job is to ensure that the child processes are alive or to manage the child processes in conjunction with load on the system. Each child process at a given time deals with a single HTTP request
Apache is one of the most common Web Server Software. Apache is a freeware and it is highly customizable. In this context, Oracle Apache Server uses several standards as well as Oracle’s proprietary modules that extend the functionality of Apache Server. Oracle Apache server provides key infrastructure for serving the Internet’s HTTP protocol.
Main Components:
There are two main components. HTTP Listener (Web Listener) and Modules
Http Listener is in charge of receiving incoming http requests and servicing these requests by sending them to appropriate processing component. Modules includes some standard Apache modules as well as Oracle proprietary modules & Some add-on modules. Modules extend basic functionality of web server. Modules such as:
Mod_security : protects web server from external attacks Mod_perl : routes requests to PERL interpreter Mod_plsql : routes requests to pl/sql engine Mod_ossl : supports SSL
OHS Components
Apache mod_plsql mod_jserv oproc File System:
Its location on File System: As you know there are three ORACLE_HOME in Apps 8.0.6, iAS, 9.2.0. Web Server is under iAS ----- iAS stands for Internet Application server ------ The environment variable to go to this location in apps user is (cd $) IAS_ORACLE_HOME.
Version:-
To find out the version of Apache and iAS, go to this location and type this if environment file is not sourced. //ora/iAS/Apache/Apache/bin] $ httpd -v
Configuration Files:
httpd.conf - is a key configuration file for Oracle HTTP Server. httpds.conf - this is configuration file if we use SSL (Secure Socket layer)
How Oracle Apache Server Works?
Apache is a program that runs under a suitable multitasking operating system such as Windows, Netware, OS2 and various unix flavours. The Apache binary is called “httpd” under unix and normally runs in background. On unix OnDemand environments you can see “httpd” executable under $APACHE_TOP/Apache/bin.
Starting and Stopping Apache:
You can start or stop httpd processes using httpd command line however, File “apachectl’ , which is a shell script calls this “httpd” executable and you can thus start or stop Apache program using apachectl start or stop arguments. Generally, in EBSO application, we use “adapcctl.sh” script from $SCRIPT_TOP location. This file reads environment variables and again calls “apachectl” file to start, stop or display status of Apache processes. Thus, finally it is the httpd executable which is being started.
When Apache Web server up it uses the below configuration Files.
adstrtall.sh -> adapcctl.sh ->apachectl-> httpd.conf (which in turns apache web server i.e http server will up)
When httpd executable first starts, this file is processed. Thus httpd.conf is the main configuration file.
We have to keep some concentration on these directories in IAS_ORACLE_HOME SID_Hostname.env, Apache, and network.
Env file is to set environment variable to iAS_ORACLE_HOME (During iAS patching)
Apache directory: -
Apache is main directory where you’re most web server configuration Sit. The
Important Files / dir under this directory are
1) Apache
2) Jserv and
3) modplsql
Mod_jserv related stuff goes in Jserv directory (it caters requesters like Self Service, Servlets, jsp pages), see the below screen
Mod_plsql related stuff goes in modplsql directory (it caters requesters like anything after /pls/ i.e., executing packages & procedures in DB).
Under iAS/Apache/Apache important directory/files is
bin (apachectl, httpd),
conf (httpd.conf, httpd_pls.conf, oracle_apache.conf),
logs (access_log, access_log_pls, error_log, error_log_pls, httpd.pid, httpd_pls.pid)
Under iAS/Apache/Jserv files you want to know are
etc (all configuration files),
logs (jvm, mod_jserv.log)
Under iAS/Apache/modplsql its
cache (cookie, plsql),
cfg (wdbsvr.app, *.conf)
MOD_JSERV
Mod_jserv related stuff goes in Jserv directory (it caters requester like Self Service, Servlets, jsp pages), see the below screen
mod_jserv is controlled by directives in jserv.conf.
Mod_plsql(PHP) related stuff goes in modplsql directory (it caters requesters like anything after /pls/ i.e., executing packages & procedures in DB).
The following scenario provides overviews of what steps occur when a server receives a plsql related client request
Main Configurations Files and Directives (Apache PL/SQL (PHP)
1)The Oracle HTTP Server receives a PL/SQL Server Page request from a client browser.
2)The Oracle HTTP Server routes the request to mod_plsql.
3)The request is forwarded by mod_plsql to the Oracle Database. By using the configuration information stored in your Database Access Descriptor DAD(wbdsvr.app), mod_plsql connects to the database.
4)mod_plsql prepares the call parameters, and invokes the PL/SQL procedure in the application.
5)The PL/SQL procedure generates an HTML page using data and the PL/SQL Web Toolkit accessed from the database.
6)The response is returned to mod_plsql.
7)The Oracle HTTP Server sends the response to the client browser
EXPLANATION OF SOME IMPORTANT CONFIGURATION FILES in 11i:-
httpd.conf->This is main Apache configuration file. From this configuration file it identify port definitions, memory settings, loggin levels, log file locations and other configuration options and files.
Port=web_cache_port: Specifies the Oracle Application Server Web Cache listening ports
Listen=Oracle_HTTP_Server_port: Specifies the HTTP and HTTPS ports obtained by Oracle HTTP Server.
jserv.conf -> Oracle HTTP Server includes a Java Virtual Machine (JVM), which is called
JServ. jserv.conf and jserv.properties are the key configuration files for Jserv
All our Self Service servlets requests are via mod_jserv like discoverer view, xml services or OAM login (So you know now where to look if issue happens in these services) If your Form Server is in servlet Mode then Core Applications are also accessed via Web Server (Jserv Component)
This file calls few properties files like jserv.properties, zone properties, viewer4i.properties, forms.properties, xmlsvcs.properties.
This file calls other configuration files like plsql.conf, immeting.conf, apps.conf.
oprocmgr.conf ->(in Apache/conf) This Oracle module provides process management and load balancing services to JServ processes
This module starts, stops, and detects death of processes (starting new processes to replace them), and provides load balancing services to the processes
oracle_apache.conf -> This configuration file is used to configure Oracle built modules supplied with default Apache like mod_pls, mod_ossl, oem, imeeting. These files are used for mod_pls configuration.
Plsql.conf ->This file is used to configure pls (Plsql). This file defines to forward all requests like /pls/ to dedicated apache listener.
mod_plsql is an Oracle HTTP Server plug-in that communicates with the database. It maps browser requests into database stored procedure calls over a SQL*Net connection. It is often indicated by a /pls/ virtual path.
wdbsvr.app ->Which in my views is named so after Web Database Server for Applications. This file contains your dad (database Access Descriptor) information like database connection description & apps user name & password. If you are changing apps password you use utility FNDCPASS and after changing password you have to manually change apps password in this file.
Configuration Files Under R12.
R12 File System INST_TOP in R12
Remember there are no jserv.properties or jserv.conf or zone.properties in R12 (new techstack), Jserv is replaced by OACORE!
The following will cover the configuration files that will fall under INST TOP directory structure.
All the configuration Files are placed under $ORA_CONFIG_HOME
If you go inside this directory you will see 10.1.2 and 10.1.3
10.1.2 configuration files will be residing in 10.1.2. for e.g $FORMS_WEB_CONFIG_FILE is placed under $ORA_CONFIG_HOME/10.1.2 - in my instance it is in /slot/ems1636/appmgr/inst/apps/tkr12r3d_rws60029rems/ora/10.1.2/forms/server/appsweb.cfg
10.1.3 configuration files will be placed under $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/
httpd.conf is located at $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/
Same location you will see (all apache related configurations files)
