VCS引起的oracle数据库异常重启一例

1. 环境描述

操作系统版本：SUSE Linux Enterprise Server 10 sp2 (x86_64)

数据库版本：Oracle 11.1.0.7.16

VCS版本：5.1

blog地址：http://blog.csdn.net/hw_libo/article/details/41171561

2. 问题现象及分析

凌晨3:46左右，手机短信收到数据库异常告警。

（1）查看数据库alert日志

Sun Nov 16 03:46:51 2014
Stopping background process SMCO
Stopping background process FBDA
Shutting down instance: further logons disabled
Sun Nov 16 03:46:54 2014
kkjcre1p: unable to spawn jobq slave process, slot 0, error 1089
Stopping background process CJQ0
Stopping background process QMNC
Stopping background process MMNL
Stopping background process MMON
Sun Nov 16 03:47:03 2014
Shutting down instance (immediate)
License high water mark = 366
Sun Nov 16 03:51:52 2014
License high water mark = 366
USER (ospid: 8654): terminating the instance
Sun Nov 16 03:51:52 2014
ORA-1092 : opiodr aborting process unknown ospid (28656_47367034883680)
Sun Nov 16 03:51:52 2014
LGWR waiting for instance termination

并未发现较为明显的异常原因。

同时也查看了那个时间段的报告，发现数据库压力不大，也没有什么异常的SQL存在。

（2）检查OS资源使用

系统资源使用正常，IO也不高。

（3）查看messages日志

Nov 16 03:42:46 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13027 Thread(4149214112) Resource(ora_vg) - monitor procedure did not complete within the expected time.
Nov 16 03:42:46 boscodb Had[10258]: VCS ERROR V-16-2-13027 (boscodb) Resource(ora_vg) - monitor procedure did not complete within the expected time.
Nov 16 03:43:01 boscodb su: (to root) root on none
……
Nov 16 03:45:39 boscodb su: (to oracle) root on none
Nov 16 03:45:47 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13210 Thread(4154813344) Agent is calling clean for resource(ora_vg) because 4 successive invocations of the monitor procedure did not complete within the expected time.
Nov 16 03:45:47 boscodb Had[10258]: VCS ERROR V-16-2-13210 (boscodb) Agent is calling clean for resource(ora_vg) because 4 successive invocations of the monitor procedure did not complete within the expected time.
……
Nov 16 03:46:48 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13006 Thread(4153760672) Resource(ora_vg): clean procedure did not complete within the expected time.
Nov 16 03:46:48 boscodb Had[10258]: VCS ERROR V-16-2-13006 (boscodb) Resource(ora_vg): clean procedure did not complete within the expected time.
Nov 16 03:46:49 boscodb AgentFramework[10899]: VCS ERROR V-16-2-13067 Thread(4155194272) Agent is calling clean for resource(ora_archive) because the resource became OFFLINE unexpectedly, on its own.
Nov 16 03:46:50 boscodb Had[10258]: VCS ERROR V-16-2-13067 (boscodb) Agent is calling clean for resource(ora_archive) because the resource became OFFLINE unexpectedly, on its own.
Nov 16 03:46:51 boscodb AgentFramework[10899]: VCS ERROR V-16-2-13068 Thread(4155194272) Resource(ora_archive) - clean completed successfully.

3. 解决办法

检查了监控vg的moniter脚本，发现也没有异常，而且脚本在其他库，压力都比较大的都不曾出问题。

当前的配置为：

        Application ora_vg (
                StartProgram = "/opt/BOSCO_ora/bin/vg_start.sh"
                StopProgram = "/opt/BOSCO_ora/bin/vg_stop.sh"
                CleanProgram = "/opt/BOSCO_ora/bin/vg_clean.sh"
                MonitorProgram = "/opt/BOSCO_ora/bin/vg_monitor.sh"
                MonitorInterval = 30
                MonitorTimeout = 30
                )

（1）可以修改main.cf，但需要重启VCS，无操作时间窗口

（2）命令动态修改

haconf -makerw

hares -modify ora_vg MonitorTimeout 90   ## 调整监控时长为90s，默认是60s

haconf -dump

blog地址：http://blog.csdn.net/hw_libo/article/details/41171561

-- Bosco  QQ：375612082

---- END ----
-------------------------------------------------------------------------------------------------------
版权所有，文章允许转载，但必须以链接方式注明源地址，否则追究法律责任!