记一次Oracle Clusterware成功安装后的故障处理

 

记一次Oracle Clusterware安装成功后的故障处理

 

1. 环境

[grid@rac1 rac1]$ cat /etc/issue
Red Hat Enterprise Linux Server release 5.8 (Tikanga)
Kernel \r on an \m

 

2. 问题描述在安装RAC的过程中, 成功安装好grid (clusterware) 后关闭了各节点. 在下次开启各节点后, 检查crs资源状态, 出现如下错误:

[grid@rac1 ~]$ crs_stat -t -v
CRS-0184: Cannot communicate with the CRS daemon.

 

3. 分析解决

检查crs状态:

[grid@rac1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services  # 无法与CRS通信
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

查看crsd对应日志:

2014-11-21 15:18:13.490: [GIPCXCPT][1002185440] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1732], ret gipcretSuccess (0)
2014-11-21 15:18:13.492: [GIPCXCPT][1002185440] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)
2014-11-21 15:18:13.498: [  OCRASM][1002185440]proprasmo: Error in open/create file in dg [DATA] # 打开磁盘组失败
[  OCRASM][1002185440]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
ORA-15077: could not locate ASM instance serving a required diskgroup  # 没有ASM实例

2014-11-21 15:18:13.498: [  OCRASM][1002185440]proprasmo: kgfoCheckMount returned [7]
2014-11-21 15:18:13.498: [  OCRASM][1002185440]proprasmo: The ASM instance is down # ASM实例处于关闭状态
2014-11-21 15:18:13.499: [  OCRRAW][1002185440]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2014-11-21 15:18:13.499: [  OCRRAW][1002185440]proprioo: No OCR/OLR devices are usable
2014-11-21 15:18:13.499: [  OCRASM][1002185440]proprasmcl: asmhandle is NULL
2014-11-21 15:18:13.499: [  OCRRAW][1002185440]proprinit: Could not open raw device 
2014-11-21 15:18:13.499: [  OCRASM][1002185440]proprasmcl: asmhandle is NULL
2014-11-21 15:18:13.499: [  OCRAPI][1002185440]a_init:16!: Backend init unsuccessful : [26]
2014-11-21 15:18:13.499: [  CRSOCR][1002185440] OCR context init failure.  Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
ORA-15077: could not locate ASM instance serving a required diskgroup
] [7]
2014-11-21 15:18:13.499: [    CRSD][1002185440][PANIC] CRSD exiting: Could not init OCR, code: 26
2014-11-21 15:18:13.499: [    CRSD][1002185440] Done.

 

日志信息表明, ASM实例未能启动, 导致crsd进程无法启动

尝试手动启动ASM实例:

[grid@rac1 ~]$ asmcmd
Connected to an idle instance.
ASMCMD> startup
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpsemsper
Connected to an idle instance.

上述信息表明, 失败的操作是semget.
semget的任务是获得信号量集(get set of semaphores), 这里的No space left on device并不是指存储空间,而是信号量资源.

检查系统中的信号量使用情况:

[grid@rac1 ~]$ ipcs

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x00000000 3407873    root      644        80         2                       
0x00000000 3440643    root      644        16384      2                       
0x00000000 3473412    root      644        280        2                       

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages 

未发现异常. 继续检查内核参数中的semmns:

root@rac1 ~]# sysctl -a|grep sem
kernel.sem = 250    100    32    128

四个参数分别为:
semmsl---每个信号集包含的信号数,该值应比ORACLE进程的最大数大10左右
semmns---系统中的信号数
semopm---每个信号操作呼叫的最大操作数
semmni---信号集标识符数,用于控制可随时创建的信号集数

加大系统中的信号量(/etc/sysctl.conf):

kernel.sem = 256 32768 100 228

 

重新启动ASM实例:

ASMCMD> startup
ORA-03113: end-of-file on communication channel
Connected to an idle instance.

因着急继续做实验, 此时直接对两个节点进行了重新启动, 重启后ASM实例正常启动, crs资源状态正常, 问题得到解决.

后来实验结束后查询ORA-03113, 导致该错误的可能原因有:

1) Unix核心参数设置不当
2) Oracle执行文件权限不正确/环境变量问题
3) 客户端通信不能正确处理
4) 数据库服务器崩溃/操作系统崩溃/进程被kill
5) Oracle 内部错误
6) 特定SQL、PL/SQL引起的错误
7) 空间不够
8) 防火墙的问题

但因错误环境已消失, 未能排查解决, 很是遗憾, 仅留做以后参考.

 

4. 参考
1) [Oracle 11g RAC CRS-4535/ORA-15077] http://blog.csdn.net/l106439814/article/details/8969060
2) [ASM启动报错ORA-27300, ORA-27301 and ORA-27302: failure occurred at: sskgpsemsper] http://www.51itstudy.com/33735.html
3) [DBA手记:共享内存无法正常释放的处理] http://www.eygle.com/archives/2011/03/ipcs_semaphore.html
4) [ORA-03113: end-of-file on communication channel 错误定位过程] http://www.51itstudy.com/6628.html

 

郑重声明:本站内容如果来自互联网及其他传播媒体,其版权均属原媒体及文章作者所有。转载目的在于传递更多信息及用于网络分享,并不代表本站赞同其观点和对其真实性负责,也不构成任何其他建议。