联系：手机/微信(+86 17813235971) QQ(107644445)
标题：12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
有朋友由于主机断电,导致数据库无法启动,需要我们提供恢复支持.通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)检查分析,发现存放在一个磁盘的数据文件scn过小,数据库无法正常open,需要历史redo(已经被覆盖,而且数据库未归档).
联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
对于19c在pdb情况下三种常见故障进行了模拟测试:
模拟19c数据库redo异常恢复
模拟19c数据库pdb undo异常恢复
模拟19c数据库root pdb undo异常恢复
模拟oracle 19c数据库redo丢失的恢复操作,模拟数据库有事务,在没有提交的情况下redo丢失故障
[oracle@localhost oradata]$ ss SQL*Plus: Release 19.0.0.0.0 - Production on Mon Nov 16 16:11:16 2020 Version 19.5.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.5.0.0.0 SQL> conn xff/oracle@127.0.0.1/pdb Connected. SQL> create table t_xifenfei as select * from dba_objects; Table created. SQL> insert into t_xifenfei select * from t_xifenfei; insert into t_xifenfei select * from t_xifenfei; insert into t_xifenfei select * from t_xifenfei; insert into t_xifenfei select * from t_xifenfei; insert into t_xifenfei select * from t_xifenfei; 72351 rows created. SQL> 144702 rows created. SQL> 289404 rows created. SQL> 578808 rows created. SQL> 1157616 rows created.
另外一个会话kill数据库并且删除redo
[root@localhost ~]# ps -ef|grep pmon oracle 38500 1 0 16:08 ? 00:00:00 ora_pmon_ora19c root 39030 39009 0 16:11 pts/2 00:00:00 grep --color=auto pmon [root@localhost ~]# kill -9 38500 [root@localhost ~]# ps -ef|grep pmon root 39042 39009 0 16:11 pts/2 00:00:00 grep --color=auto pmon [root@localhost ~]# ls -l /u01/app/oracle/oradata/ORA19C/redo*.log ls: cannot access /u01/app/oracle/oradata/ORA19C/redo*.log: No such file or directory
启动数据库报错ORA-00313 ORA-00312 ORA-27037
SQL> startup ORACLE instance started. Total System Global Area 4999609088 bytes Fixed Size 9145088 bytes Variable Size 905969664 bytes Database Buffers 4076863488 bytes Redo Buffers 7630848 bytes Database mounted. ORA-00313: open failed for members of log group 3 of thread 1 ORA-00312: online log 3 thread 1: '/u01/app/oracle/oradata/ORA19C/redo03.log' ORA-27037: unable to obtain file status Linux-x86_64 Error: 2: No such file or directory Additional information: 7
因为redo全部丢失只能尝试强制拉库
SQL> startup mount pfile='/tmp/pfile'
ORACLE instance started.
Total System Global Area 4999609088 bytes
Fixed Size 9145088 bytes
Variable Size 905969664 bytes
Database Buffers 4076863488 bytes
Redo Buffers 7630848 bytes
Database mounted.
SQL> recover database until cancel;
ORA-00279: change 2335666 generated at 11/16/2020 16:08:42 needed for thread 1
ORA-00289: suggestion :
/u01/app/oracle/product/19.2/db_1/dbs/arch1_12_1056620100.dbf
ORA-00280: change 2335666 for thread 1 is in sequence #12
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/u01/app/oracle/oradata/ORA19C/system01.dbf'
ORA-01112: media recovery not started
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [],
[], [], [], [], [], [], []
Process ID: 39588
Session ID: 9 Serial number: 32012
数据库报ORA-600 kcbzib_kcrsds_1错误是由于在强制拉库过程中文件异常导致,通过对异常文件进行处理数据库open成功
SQL> recover database;
Media recovery complete.
SQL> alter database open;
Database altered.
SQL>
SQL> alter session set container=pdb;
Session altered.
SQL> alter database open;
SQL> show pdbs;
CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 PDB READ WRITE NO
这个是模拟redo丢失或者损坏故障,在实际的生产故障中可能要比这个复杂很多.
联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
数据库版本
客户存储故障，修复之后，多套库增加_allow_resetlogs_corruption隐含参数强制拉库出现都类似错误ORA-600 kcbzib_kcrsds_1,出现这个错误,一般都是由于数据库不一致强制拉库导致
2019-02-23T01:25:43.125621+08:00 alter database open resetlogs 2019-02-23T01:25:43.231990+08:00 RESETLOGS is being done without consistancy checks. This may result in a corrupted database. The database should be recreated. RESETLOGS after incomplete recovery UNTIL CHANGE 149251865354 time Clearing online redo logfile 1 +DG_XFF/xifenfei/ONLINELOG/group_1.258.983824407 Clearing online redo logfile 2 +DG_XFF/xifenfei/ONLINELOG/group_2.259.983824409 Clearing online redo logfile 3 +DG_XFF/xifenfei/ONLINELOG/group_3.266.983825461 Clearing online redo logfile 4 +DG_XFF/xifenfei/ONLINELOG/group_4.267.983825461 Clearing online log 1 of thread 1 sequence number 20749 Clearing online log 2 of thread 1 sequence number 20750 Clearing online log 3 of thread 2 sequence number 1371 Clearing online log 4 of thread 2 sequence number 1372 2019-02-23T01:25:44.669890+08:00 ALTER SYSTEM SET remote_listener=' xifenfeidb-cluster-scan:1521' SCOPE=MEMORY SID='xifenfei2'; 2019-02-23T01:25:44.671436+08:00 ALTER SYSTEM SET listener_networks='' SCOPE=MEMORY SID='xifenfei2'; 2019-02-23T01:25:46.990077+08:00 Clearing online redo logfile 1 complete Clearing online redo logfile 2 complete Clearing online redo logfile 3 complete Clearing online redo logfile 4 complete Resetting resetlogs activation ID 3002369299 (0xb2f48513) Online log +DG_XFF/xifenfei/ONLINELOG/group_1.258.983824407: Thread 1 Group 1 was previously cleared Online log +DG_XFF/xifenfei/ONLINELOG/group_2.259.983824409: Thread 1 Group 2 was previously cleared Online log +DG_XFF/xifenfei/ONLINELOG/group_3.266.983825461: Thread 2 Group 3 was previously cleared Online log +DG_XFF/xifenfei/ONLINELOG/group_4.267.983825461: Thread 2 Group 4 was previously cleared 2019-02-23T01:25:47.137701+08:00 Setting recovery target incarnation to 2 2019-02-23T01:25:47.152393+08:00 This instance was first to open Ping without log force is disabled: not an Exadata system. Picked broadcast on commit scheme to generate SCNs Endian type of dictionary set to little 2019-02-23T01:25:47.597502+08:00 Assigning activation ID 3019587675 (0xb3fb405b) 2019-02-23T01:25:47.625734+08:00 TT00: Gap Manager starting (PID:22467) 2019-02-23T01:25:47.910026+08:00 Thread 2 opened at log sequence 1 Current log# 3 seq# 1 mem# 0: +DG_XFF/xifenfei/ONLINELOG/group_3.266.983825461 Successful open of redo thread 2 2019-02-23T01:25:47.911069+08:00 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set 2019-02-23T01:25:47.971709+08:00 Sleep 5 seconds and then try to clear SRLs in 2 time(s) 2019-02-23T01:25:48.065008+08:00 start recovery: pdb 0, passed in flags x10 (domain enable 0) 2019-02-23T01:25:48.065177+08:00 Instance recovery: looking for dead threads Instance recovery: lock domain invalid but no dead threads validate pdb 0, flags x10, valid 0, pdb flags x84 * validated domain 0, flags = 0x80 Instance recovery complete: valid 1 (flags x10, recovery domain flags x80) 2019-02-23T01:25:48.803746+08:00 Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei2/trace/xifenfei2_ora_21292.trc (incident=128552): ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], [] Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2019-02-23T01:25:49.947062+08:00 ***************************************************************** An internal routine has requested a dump of selected redo. This usually happens following a specific internal error, when analysis of the redo logs will help Oracle Support with the diagnosis. It is recommended that you retain all the redo logs generated (by all the instances) during the past 12 hours, in case additional redo dumps are required to help with the diagnosis. ***************************************************************** 2019-02-23T01:25:50.334684+08:00 Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei2/trace/xifenfei2_ora_21292.trc: ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], [] 2019-02-23T01:25:50.334880+08:00 Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei2/trace/xifenfei2_ora_21292.trc: ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], [] Error 600 happened during db open, shutting down database 2019-02-23T01:25:50.362808+08:00 Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei2/trace/xifenfei2_ora_21292.trc (incident=128553): ORA-00603: ORACLE server session terminated by fatal error ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], [] 2019-02-23T01:25:51.521133+08:00 opiodr aborting process unknown ospid (21292) as a result of ORA-603
另外出现该错误之后，数据库再次恢复会出现类似,这个是由于open库的过程中导致控制文件损坏而出现的错误.
2019-02-23T22:52:39.390966+08:00 ALTER DATABASE RECOVER database 2019-02-23T22:52:39.391125+08:00 Media Recovery Start Started logmerger process 2019-02-23T22:52:39.471904+08:00 Media Recovery failed with error 16433 2019-02-23T22:52:39.996235+08:00 Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_m000_1593.trc: ORA-01110: data file 11: '+DG_XFF/xifenfei/DATAFILE/ls_zh.274.984235699' 2019-02-23T22:52:40.224440+08:00 Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_m000_1593.trc: ORA-01110: data file 12: '+DG_XFF/xifenfei/DATAFILE/sf_zh.275.984235715' 2019-02-23T22:52:40.459606+08:00 Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_m000_1593.trc: ORA-01110: data file 13: '+DG_XFF/xifenfei/DATAFILE/tj_gl.276.984235729' 2019-02-23T22:52:40.574563+08:00 Recovery Slave PR00 previously exited with exception 283 ORA-283 signalled during: ALTER DATABASE RECOVER database ...
 |
|