EM 12c: OMS Failed to start After restarting the Repository Database or reboot of the server

浏览数：38 / 时间：2015年06月12日

今天启动Oms发现了这个问题，启动不了：

[oracle@ora11g64 bin]$ emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server is Down

查看日志：

日志中记录1：
/u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/log

2014-09-19 13:57:06,329 [Thread-1] INFO wls.OMSController run.1744 - <OUT>FAILED_NOT_RESTARTABLE
2014-09-19 13:57:06,330 [Thread-1] INFO wls.OMSController run.1744 - <OUT>
2014-09-19 13:57:06,331 [Thread-1] INFO wls.OMSController run.1744 - <OUT>sleep for 10 seconds...
2014-09-19 13:57:16,334 [Thread-1] INFO wls.OMSController run.1744 - <OUT>status of EMGC_OMS1:
2014-09-19 13:57:16,336 [Thread-1] INFO wls.OMSController run.1744 - <OUT>
2014-09-19 13:57:16,337 [Thread-1] INFO wls.OMSController run.1744 - <OUT>FAILED_NOT_RESTARTABLE
2014-09-19 13:57:16,401 [Thread-1] INFO wls.OMSController run.1744 - <OUT>
2014-09-19 13:57:16,401 [Thread-1] INFO wls.OMSController run.1744 - <OUT>timed out waiting for EMGC_OMS1 to be RUNNING
2014-09-19 13:57:16,402 [Thread-1] INFO wls.OMSController run.1744 - <OUT>status of node manager:
2014-09-19 13:57:16,402 [Thread-1] INFO wls.OMSController run.1744 - <OUT>Currently connected to Node Manager to monitor the domain GCDomain.
2014-09-19 13:57:16,403 [Thread-1] INFO wls.OMSController run.1744 - <OUT>1
2014-09-19 13:57:16,413 [Thread-1] INFO wls.OMSController run.1744 - <OUT>Successfully disconnected from Node Manager.
2014-09-19 13:57:16,416 [Thread-1] INFO wls.OMSController run.1744 - <OUT>_END_
2014-09-19 13:57:16,416 [Thread-2] INFO wls.OMSController run.1744 - <ERR>_END_
2014-09-19 13:57:16,866 [main] INFO wls.OMSController processCmd.1392 - wlst process is finished
2014-09-19 13:57:16,867 [main] INFO wls.OMSController processCmd.1394 - wlst process exited with code 1
2014-09-19 13:57:16,867 [main] INFO wls.OMSController processCmd.1396 - wlst outstream joined
2014-09-19 13:57:16,868 [main] INFO wls.OMSController processCmd.1398 - wlst errstream joined
2014-09-19 13:57:16,868 [main] INFO wls.OMSController processCmd.1406 - process exited with code 1
2014-09-19 13:57:16,880 [main] DEBUG oms.StatusOMSCmd processStatusOMS.159 - HTTP port in emgc property file: 7203
2014-09-19 13:57:16,881 [main] DEBUG oms.StatusOMSCmd processStatusOMS.160 - HTTPS port in emgc property file: 7302
2014-09-19 13:57:16,881 [main] DEBUG oms.StatusOMSCmd processStatusOMS.166 - Webtier HTTP port in emgc property file: 7789
2014-09-19 13:57:16,882 [main] DEBUG oms.StatusOMSCmd processStatusOMS.167 - Webtier HTTPS port in emgc property file: 7801
2014-09-19 13:57:16,882 [main] DEBUG oms.StatusOMSCmd processStatusOMS.186 - Using https port.
2014-09-19 13:57:16,882 [main] DEBUG oms.StatusOMSCmd processStatusOMS.203 - Using webtier https port.
2014-09-19 13:57:17,074 [main] DEBUG oms.StatusOMSCmd pingOMSCheck.341 - Attempting to connect to https://ora11g64:7302/em/faces/logon/core-uifwk-console-login
2014-09-19 13:57:18,263 [main] DEBUG oms.StatusOMSCmd pingOMSCheck.341 - Attempting to connect to https://ora11g64:7801/empbs/check
2014-09-19 13:57:18,281 [main] DEBUG oms.StatusOMSCmd processStatusOMS.240 - console page status code is 404
2014-09-19 13:57:18,330 [main] DEBUG oms.StatusOMSCmd processStatusOMS.242 - pbs page status code is 200
2014-09-19 13:57:18,330 [main] DEBUG oms.StatusOMSCmd processStatusOMS.245 - console header is null
2014-09-19 13:57:18,330 [main] DEBUG oms.StatusOMSCmd processStatusOMS.247 - pbs header is null
2014-09-19 13:57:18,330 [main] DEBUG oms.StatusOMSCmd processStatusOMS.287 - Encountered error while hitting page. Status code is 404
2014-09-19 13:57:18,330 [main] INFO oms.StatusOMSCmd processStatusOMS.294 - Failed to connect to em login page
2014-09-19 13:57:18,330 [main] INFO oms.StatusOMSCmd processStatusOMS.295 - Encountered error while hitting page. Status codes are 404 for console and 200 for PBS
2014-09-19 13:57:18,331 [main] INFO oms.StatusOMSCmd processStatusOMS.296 - Connection exception when trying to hit OMS page. OMS is down
2014-09-19 13:57:18,331 [main] INFO wls.OMSController statusOMS.1354 - statusOMS finished with result: 8
2014-09-19 13:57:18,331 [main] INFO wls.OMSController startOMS.940 - startOMS finished with result: 1
2014-09-19 13:57:18,332 [main] DEBUG oms.StatusOMSCmd processStatusOMS.159 - HTTP port in emgc property file: 7203
2014-09-19 13:57:18,332 [main] DEBUG oms.StatusOMSCmd processStatusOMS.160 - HTTPS port in emgc property file: 7302
2014-09-19 13:57:18,332 [main] DEBUG oms.StatusOMSCmd processStatusOMS.166 - Webtier HTTP port in emgc property file: 7789
2014-09-19 13:57:18,332 [main] DEBUG oms.StatusOMSCmd processStatusOMS.167 - Webtier HTTPS port in emgc property file: 7801
2014-09-19 13:57:18,333 [main] DEBUG oms.StatusOMSCmd processStatusOMS.186 - Using https port.
2014-09-19 13:57:18,333 [main] DEBUG oms.StatusOMSCmd processStatusOMS.203 - Using webtier https port.
2014-09-19 13:57:18,333 [main] DEBUG oms.StatusOMSCmd pingOMSCheck.341 - Attempting to connect to https://ora11g64:7302/em/faces/logon/core-uifwk-console-login
2014-09-19 13:57:18,458 [main] DEBUG oms.StatusOMSCmd pingOMSCheck.341 - Attempting to connect to https://ora11g64:7801/empbs/check
2014-09-19 13:57:18,480 [main] DEBUG oms.StatusOMSCmd processStatusOMS.240 - console page status code is 404
2014-09-19 13:57:18,481 [main] DEBUG oms.StatusOMSCmd processStatusOMS.242 - pbs page status code is 200
2014-09-19 13:57:18,481 [main] DEBUG oms.StatusOMSCmd processStatusOMS.245 - console header is null
2014-09-19 13:57:18,481 [main] DEBUG oms.StatusOMSCmd processStatusOMS.247 - pbs header is null
2014-09-19 13:57:18,482 [main] DEBUG oms.StatusOMSCmd processStatusOMS.287 - Encountered error while hitting page. Status code is 404
2014-09-19 13:57:18,482 [main] INFO oms.StatusOMSCmd processStatusOMS.294 - Failed to connect to em login page
2014-09-19 13:57:18,482 [main] INFO oms.StatusOMSCmd processStatusOMS.295 - Encountered error while hitting page. Status codes are 404 for console and 200 for PBS
2014-09-19 13:57:18,484 [main] INFO oms.StatusOMSCmd processStatusOMS.296 - Connection exception when trying to hit OMS page. OMS is down
2014-09-19 13:57:18,499 [main] INFO wls.OMSController statusOMS.1354 - statusOMS finished with result: 8

日志文件中记:2：
2014-09-16 13:18:38,209 [Thread-1] INFO wls.OMSController run.1744 - <OUT>Successfully started server EMGC_OMS1 ...
2014-09-16 13:18:38,229 [Thread-1] INFO wls.OMSController run.1744 - <OUT>new status of EMGC_OMS1:
2014-09-16 13:18:38,229 [Thread-1] INFO wls.OMSController run.1744 - <OUT>
2014-09-16 13:18:38,229 [Thread-1] INFO wls.OMSController run.1744 - <OUT>RUNNING

日志中记录3：
[oracle@ora11g64 logs]$ pwd
/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs

Repos details fetched from credstore
Fetched repository credentials from Credential Store
Invalid Connection Pool. ERROR = User credentials doesn‘t match the existing ones
Failed to verify repository
java.lang.NullPointerException

metalink:文档 ID 1459936.1

CAUSE
OMS will not be able to make the connection to the database and restarting also fails, if the background processes are not stopped completely.

SOLUTION
As a workaround, follow the steps.

1. Stop the OMS using

/bin>./emctl stop oms -all

2. Check the following processes

$ps -ef | grep java
$ps -ef | grep opmn
$ps -ef | grep http

Please kill all the processes related to OMS user.

3. Restart the Repository Database.

4. Now start the OMS

OMS_HOME/bin>./emctl start oms

To prevent the issue in future:

Apply the Patch 13466904 to the 12.1.0.1.1 (With BP1) OMS home. After applying the patch, we dont need to restart the OMS if the reposiotry database is restarted.

This issue is fixed in 12.1.0.2

我的环境

[oracle@ora11g64 bin]$ ./emctl stop oms -all

Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Already Stopped
AdminServer Successfully Stopped

Oracle Management Server is Down

[oracle@ora11g64 bin]$ ps -ef |grep java
oracle 4753 4708 2 13:01 ? 00:02:40 /u01/app/oracle/Middleware/jdk16/jdk/bin/java -server -Xms256m -Xmx1740m -XX:MaxPermSize=768M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:CompileThreshold=8000 -XX:PermSize=128m -Dweblogic.Name=EMGC_OMS1 -Djava.security.policy=/u01/app/oracle/Middleware/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.system.BootIdentityFile=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.ReverseDNSAllowed=false -DINSTANCE_HOME=/u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1 -DORACLE_HOME=/u01/app/oracle/Middleware/oms -Ddomain.home=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain -Djava.awt.headless=true -Ddomain.name=GCDomain -Doracle.sysman.util.logging.mode=dual_mode -Djbo.doconnectionpooling=true -Djbo.txn.disconnect_level=1 -Docm.repeater.home=/u01/app/oracle/Middleware/oms -Djbo.ampool.minavailablesize=1 -Djbo.ampool.timetolive=-1 -Djbo.load.components.lazily=true -Djbo.recyclethreshold=50 -Djbo.ampool.maxavailablesize=50 -Djava.security.egd=file:///dev/./urandom -Dweblogic.debug.DebugSecurityAtn=true -Dweblogic.debug.DebugWebAppSecurity=true -Dweblogic.SSL.LoginTimeoutMillis=300000 -Djps.auth.debug=true -Djps.authz=ACC -Djps.combiner.optimize.lazyeval=true -Djps.combiner.optimize=true -Djps.policystore.hybrid.mode= -Djps.subject.cache.key=5 -Djps.subject.cache.ttl=600000 -Xbootclasspath/p:/u01/app/oracle/Middleware/oms/sysman/jlib/diagpatch_bug11725986.jar -Djdkpatchlog=/u01/app/oracle/Middleware/oms/sysman/log/diagpatch_bug11725986.log -Doracle.apm.home=/u01/app/oracle/Middleware/oms/apm/ -DAPM_HELP_FILENAME=oesohwconfig.xml -da -Dplatform.home=/u01/app/oracle/Middleware/wlserver_10.3 -Dwls.home=/u01/app/oracle/Middleware/wlserver_10.3/server -Dweblogic.home=/u01/app/oracle/Middleware/wlserver_10.3/server -Dcommon.components.home=/u01/app/oracle/Middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain -Djrockit.optfile=/u01/app/oracle/Middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.server.config.dir=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/servers/EMGC_OMS1 -Doracle.domain.config.dir=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig -Digf.arisidbeans.carmlloc=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/carml -Digf.arisidstack.home=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/u01/app/oracle/Middleware/oracle_common/modules/oracle.ossoiap_11.1.1,/u01/app/oracle/Middleware/oracle_common/modules/oracle.oamprovider_11.1.1 -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dweblogic.jdbc.remoteEnabled=false -Doracle.apm.home=/u01/app/oracle/Middleware/oms/apm/ -DAPM_HELP_FILENAME=oesohwconfig.xml -Dweblogic.management.discover=false -Dweblogic.management.server=https://ora11g64:7102 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/u01/app/oracle/Middleware/patch_wls1035/profiles/default/sysext_manifest_classpath weblogic.Server
oracle 10120 8047 0 14:53 pts/2 00:00:00 grep java
[oracle@ora11g64 bin]$ kill -9 4753
[oracle@ora11g64 bin]$ ps -ef |grep java
oracle 10147 8047 0 14:54 pts/2 00:00:00 grep java
[oracle@ora11g64 bin]$ ps -ef |grep opmn
oracle 10151 8047 0 14:54 pts/2 00:00:00 grep opmn
[oracle@ora11g64 bin]$ ps -ef |grep http
oracle 10155 8047 0 14:54 pts/2 00:00:00 grep http
[oracle@ora11g64 bin]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Fri Sep 19 14:54:56 2014

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SYS@ORA11G>shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SYS@ORA11G>startup
ORACLE instance started.

Total System Global Area 2137886720 bytes
Fixed Size 2215064 bytes
Variable Size 822084456 bytes
Database Buffers 1308622848 bytes
Redo Buffers 4964352 bytes
Database mounted.
Database opened.
SYS@ORA11G>exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@ora11g64 bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server Successfully Started
Oracle Management Server is Up

还有一个问题会导致：
就是归档路径下的文件多了空闲空间不足导致的，删除了后重启就行。

但是我的实验在非归档模式，所以不行。

结果果然可以，非常感谢 metalink！

2014 /9 /19

tyger beijing

郑重声明：本站内容如果来自互联网及其他传播媒体，其版权均属原媒体及文章作者所有。转载目的在于传递更多信息及用于网络分享，并不代表本站赞同其观点和对其真实性负责，也不构成任何其他建议。

EM 12c: OMS Failed to start After restarting the Repository Database or reboot of the server