ORA-600 kcratr_nab_less_than_odr和ORA-600 2662故障处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-600 kcratr_nab_less_than_odr和ORA-600 2662故障处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

异常断电之后,oracle启动报ORA-600 kcratr_nab_less_than_odr错误

Sun Aug 17 11:06:09 2025
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 11 processes
Started redo scan
Completed redo scan
 read 0 KB redo, 0 data blocks need recovery
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [40785], [33267], [40630]
Sun Aug 17 11:06:20 2025
Aborting crash recovery due to error 600
Sun Aug 17 11:06:20 2025
Trace dumping is performing id=[cdmp_20250817110620]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_2920.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [40785], [33267], [40630]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_2920.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [40785], [33267], [40630]
ORA-600 signalled during: ALTER DATABASE OPEN...

这个错误处理有多次处理经验
12c启动报kcratr_nab_less_than_odr
又一例ORA-600 kcratr_nab_less_than_odr
ORA-600 kcratr_nab_less_than_odr故障解决
差点被误操作的ORA-600 kcratr_nab_less_than_odr故障
11.2.0.4库中遇到ORA-600 kcratr_nab_less_than_odr报错
ORA-600 kcratr_nab_less_than_odr和ORA-600 4194故障处理
一般重建ctl或者using backup ctl方式恢复即可实现0丢失打开库,但是这个库尝试打开报ORA-600 2662错误

SQL> startup nomount;
ORACLE 例程已经启动。

Total System Global Area 6847938560 bytes
Fixed Size                  2188768 bytes
Variable Size            4680845856 bytes
Database Buffers         2147483648 bytes
Redo Buffers               17420288 bytes
SQL> @rectl.sql

控制文件已创建。

SQL>
SQL>
SQL> recover database;
完成介质恢复。
SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [4413], [1200914792],
[4413], [1201104184], [12583040], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [2662], [4413], [1200914791],
[4413], [1201104184], [12583040], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [4413], [1200914789],
[4413], [1201104184], [12583040], [], [], [], [], [], []
进程 ID: 3132
会话 ID: 3018 序列号: 1

对于这样的ORA-600 2662错误是比较常见的问题,直接通过Patch_SCN工具修改scn即可正常打开库
patch_scn-ora-600-2662

Mon Aug 18 03:30:41 2025
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 12 slaves
Mon Aug 18 03:30:41 2025
Recovery of Online Redo Log: Thread 1 Group 2 Seq 40787 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Completed: ALTER DATABASE RECOVER  database  
alter database open 
Beginning crash recovery of 1 threads
 parallel recovery started with 11 processes
Started redo scan
Completed redo scan
 read 1 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 40787, block 2, scn 18954891612040
Recovery of Online Redo Log: Thread 1 Group 2 Seq 40787 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 40787, block 4, scn 18954891632047
 0 data blocks read, 0 data blocks written, 1 redo k-bytes read
Mon Aug 18 03:30:47 2025
Thread 1 advanced to log sequence 40788 (thread open)
Thread 1 opened at log sequence 40788
  Current log# 3 seq# 40788 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Aug 18 03:30:47 2025
SMON: enabling cache recovery
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: alter database open

后续增加tempfile,导出数据完成本次恢复任务

win环境断电强制拉库报ORA-600 kcbzib_kcrsds_1故障处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:win环境断电强制拉库报ORA-600 kcbzib_kcrsds_1故障处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户环境异常断电,导致数据库无法正常启动,执行recover提示缺少归档(ORA-00279 ORA-00289 ORA-00280错误)

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on 星期四 8月 14 11:17:08 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


连接到:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0


SQL> select open_mode from v$database;

OPEN_MODE
----------------------------------------
MOUNTED

SQL> select group#,sequence#,status from v$log;

    GROUP#  SEQUENCE# STATUS
---------- ---------- --------------------------------
         1      15082 ACTIVE
         3      15084 CURRENT
         2      15083 ACTIVE

SQL> recover database;
ORA-00279: 更改 885427209 (在 08/11/2025 22:14:56 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001
ORA-00280: 更改 885427209 (用于线程 1) 在序列 #15080 中


指定日志: {<RET>=suggested | filename | AUTO | CANCEL}

ORA-00308: 无法打开归档日志
'D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件。

z这个库需要15080归档,但是由于非归档环境,redo最小为15082,所以这种情况,只能先恢复可以正常recover的数据文件,然后强制拉库

SQL> recover datafile 1;
完成介质恢复。
SQL> recover datafile 3,4,7;
完成介质恢复。
SQL> recover datafile 62;
完成介质恢复。

SQL> recover database until cancel;
ORA-00279: 更改 885427209 (在 08/11/2025 22:14:56 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001
ORA-00280: 更改 885427209 (用于线程 1) 在序列 #15080 中


指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: 警告: RECOVER 成功但 OPEN RESETLOGS 将出现如下错误
ORA-01152: 文件 1 没有从过旧的备份中还原
ORA-01110: 数据文件 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSTEM01.DBF'


ORA-01112: 未启动介质恢复


SQL> alter database open resetlogs ;
alter database open resetlogs
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [],
[], [], [], [], [], [], []
进程 ID: 6272
会话 ID: 3873 序列号: 46847

alert日志报错

2025-08-14T11:24:08.246798+08:00
alter database open resetlogs
2025-08-14T11:24:08.425971+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 28: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\PRESCRIPTION_P.DBF'
2025-08-14T11:24:08.643178+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 29: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\PRESCRIPTION_DETAIL_P.DBF'
2025-08-14T11:24:08.958486+08:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 885427209 time 
Resetting resetlogs activation ID 3572089731 (0xd4e9c383)
2025-08-14T11:24:08.988512+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 30: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\ORDER_LIST_P.DBF'
2025-08-14T11:24:12.029432+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 31: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\CHARGE_STATIS_P.DBF'
2025-08-14T11:24:13.687023+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 32: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\ECASETAB01.DBF'
2025-08-14T11:24:13.725060+08:00
Setting recovery target incarnation to 2
Checker run found 31 new persistent data failures
2025-08-14T11:24:14.701994+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
Endian type of dictionary set to little
2025-08-14T11:24:14.793084+08:00
Assigning activation ID 3729861866 (0xde512cea)
2025-08-14T11:24:14.848137+08:00
TT00 (PID:8068): Gap Manager starting
2025-08-14T11:24:14.870158+08:00
Redo log for group 1, sequence 1 is not located on DAX storage
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG
Successful open of redo thread 1
2025-08-14T11:24:14.912199+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2025-08-14T11:24:15.849098+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc(incident=717057):
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Please look for redo dump in pinned buffers history in incident trace file, 
if not dumped for what so ever reason, use the following command to dump it at the earliest.
ALTER SYSTEM DUMP REDO DBA MIN 4 128 DBA MAX 4 128 SCN MIN 1;
2025-08-14T11:24:17.402582+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Undo initialization recovery: err:600 start: 2392187 end: 2394046 diff: 1859 ms (1.9 seconds)
2025-08-14T11:24:17.528703+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2025-08-14T11:24:17.529703+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc(incident=717058):
ORA-00603: ORACLE 服务器会话因致命错误而终止
ORA-01092: ORACLE 实例终止。强制断开连接
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2025-08-14T11:24:18.995110+08:00
opiodr aborting process unknown ospid (6272) as a result of ORA-603
2025-08-14T11:24:18.997112+08:00
ORA-603 : opitsk aborting process
License high water mark = 42
USER (ospid: (prelim)): terminating the instance due to ORA error 
2025-08-14T11:24:23.749679+08:00
Instance terminated by USER(prelim), pid = 6272

遭遇到比较明显的ORA-600 kcbzib_kcrsds_1错误,通过自研的Patch_SCN小工具,以前有过类似恢复案例:
Patch SCN工具一键恢复ORA-600 kcbzib_kcrsds_1
patch_scn


然后直接open数据库

SQL> recover database;
完成介质恢复。
SQL> oradebug setmypid
已处理的语句
SQL> oradebug DUMPvar SGA kcsgscn_
kscn8 kcsgscn_ [7FF6623FF310, 7FF6623FF318) = 40B272E2 00000000
SQL>
SQL> alter database open;

数据库已更改。

然后安排导出数据,完成本次恢复任务.

ORA-600 kcratr_nab_less_than_odr和ORA-600 4194故障处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-600 kcratr_nab_less_than_odr和ORA-600 4194故障处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户一个11.2.0.1的库由于断电导致启动的时候报ORA-600 kcratr_nab_less_than_odr错误
ORA-600-kcratr_nab_less_than_odr


Wed Aug 06 15:55:07 2025
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 27 processes
Started redo scan
Completed redo scan
 read 11787 KB redo, 4129 data blocks need recovery
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_3520.trc  (incident=2624321):
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [111988], [42537], [43285], [], [], [], [], [], [], []
Incident details in: d:\app\xff\diag\rdbms\primary\orcl\incident\incdir_2624321\orcl_ora_3520_i2624321.trc
Aborting crash recovery due to error 600
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_3520.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [111988], [42537], [43285], [], [], [], [], [], [], []
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_3520.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [111988], [42537], [43285], [], [], [], [], [], [], []
ORA-600 signalled during: ALTER DATABASE OPEN...

类似的ORA-600 kcratr_nab_less_than_odr故障处理不少,但是应用redo完成之后,再open报错的不多:
12c启动报kcratr_nab_less_than_odr
又一例ORA-600 kcratr_nab_less_than_odr
ORA-600 kcratr_nab_less_than_odr故障解决
差点被误操作的ORA-600 kcratr_nab_less_than_odr故障
11.2.0.4库中遇到ORA-600 kcratr_nab_less_than_odr报错
但是这个客户比较不幸,经过一系列尝试打开数据库,但是报ORA-600 4194错误

Wed Aug 06 17:09:33 2025
ALTER DATABASE RECOVER  database until cancel using backup controlfile  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 28 slaves
ORA-279 signalled during: ALTER DATABASE RECOVER  database until cancel using backup controlfile  ...
Wed Aug 06 17:10:17 2025
ALTER DATABASE RECOVER    LOGFILE 'D:\app\xff\oradata\orcl\REDO05.LOG'  
Media Recovery Log D:\app\xff\oradata\orcl\REDO05.LOG
Wed Aug 06 17:10:17 2025
Incomplete recovery applied all redo ever generated.
Recovery completed through change 2118813100 time 08/06/2025 15:02:15
Completed: ALTER DATABASE RECOVER    LOGFILE 'D:\app\xff\oradata\orcl\REDO05.LOG'  
Wed Aug 06 17:14:05 2025
alter database open resetlogs
Archived Log entry 188143 added for thread 1 sequence 111985 ID 0x634438e6 dest 1:
Archived Log entry 188144 added for thread 1 sequence 111986 ID 0x634438e6 dest 1:
Archived Log entry 188145 added for thread 1 sequence 111987 ID 0x634438e6 dest 1:
Archived Log entry 188146 added for thread 1 sequence 111984 ID 0x634438e6 dest 1:
Archived Log entry 188147 added for thread 1 sequence 111988 ID 0x634438e6 dest 1:
RESETLOGS after complete recovery through change 2118813100
Resetting resetlogs activation ID 1665415398 (0x634438e6)
Wed Aug 06 17:15:03 2025
Archived Log entry 188148 added for thread 1 sequence 1 ID 0x677cfac1 dest 1:
Successfully onlined Undo Tablespace 2.
Dictionary check beginning
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_smon_2428.trc  (incident=3264274):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\app\xff\diag\rdbms\primary\orcl\incident\incdir_3264274\orcl_smon_2428_i3264274.trc
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Wed Aug 06 17:15:06 2025
QMNC started with pid=34, OS id=3052 
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: alter database open resetlogs

后续由于该undo异常报ORA-01595和ORA-600 4194等错误导致数据库crash

Wed Aug 06 17:15:49 2025
Doing block recovery for file 3 block 888
Resuming block recovery (PMON) for file 3 block 888
Block recovery from logseq 2, block 316 to scn 2118813732
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: D:\APP\xff\ORADATA\ORCL\REDO02.LOG
Block recovery stopped at EOT rba 2.447.16
Block recovery completed at rba 2.447.16, scn 0.2118813732
Doing block recovery for file 3 block 1960
Resuming block recovery (PMON) for file 3 block 1960
Block recovery from logseq 2, block 316 to scn 2118813622
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: D:\APP\xff\ORADATA\ORCL\REDO02.LOG
Block recovery completed at rba 2.317.16, scn 0.2118813626
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_smon_2428.trc:
ORA-01595: error freeing extent (2) of rollback segment (27))
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []

接手这个库之后,处理工作相对比较简单,直接对异常undo进行重建即可

Thu Aug 07 09:35:08 2025
create undo tablespace undotbs2 datafile 'D:\APP\xff\ORADATA\ORCL\undotbs02.dbf' size 128M autoextend on
Completed: create undo tablespace undotbs2 datafile 'D:\APP\xff\ORADATA\ORCL\undotbs02.dbf' size 128M autoextend on
Thu Aug 07 09:37:53 2025
drop tablespace UNDOTBS1 including contents and datafiles
Deleted file D:\APP\xff\ORADATA\ORCL\UNDOTBS01.DBF
Completed: drop tablespace UNDOTBS1 including contents and datafiles

然后数据库报大量的ORA-600 kdsgrp1错误

Thu Aug 07 10:03:53 2025
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_6156.trc  (incident=5121829):
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Thu Aug 07 10:04:00 2025
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_4564.trc  (incident=5121501):
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Thu Aug 07 10:04:06 2025
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_9160.trc  (incident=5121837):
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Thu Aug 07 10:04:16 2025
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_5868.trc  (incident=5121821):
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Thu Aug 07 10:04:28 2025
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_4448.trc  (incident=5122077):
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
* kdsgrp1-1: *************************************************
            row 0x013a9ae9.0 continuation at
            0x013a9ae9.0 file# 4 block# 3840745 slot 0 not found
KDSTABN_GET: 0 ..... ntab: 1
curSlot: 0 ..... nrows: 2
kdsgrp - dump CR block dba=0x013a9ae9
Block header dump:  0x013a9ae9
 Object id on Block? Y
 seg/obj: 0x31dd9  csc: 0x00.7dc0f246  itc: 2  flg: E  typ: 1 - DATA
     brn: 0  bdba: 0x13a9082 ver: 0x01 opc: 0
     inc: 2  exflg: 0
 
 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x0000.000.00000000  0x00000000.0000.00  ----    0  fsc 0x0000.00000000
0x02   0x0000.000.00000000  0x00000000.0000.00  ----    0  fsc 0x0000.00000000

通过分析是由于index和表记录不匹配导致,对index进行重新rebuild完成本次修复任务.

一次非常幸运的ORA-600 16703(tab$被清空)故障恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:一次非常幸运的ORA-600 16703(tab$被清空)故障恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

这次的ORA-600 16703的故障比较特殊,客户的一套rac运行了5年多没有重启,这次由于异常导致其中一个节点重启,然后触发了tab$被清空,异常节点启动报ORA-600 16703错误.朋友在故障之后,第一时间没有对在运行的节点进行重启(虽然也无法对外提供业务服务),使得恢复工作相对简单一些,恢复效果也是最完美的.这个是我在对于软件安装介质注入恶意脚本,300天之后重启触发tab$被清空的相关恢复case中,最完美的一次(以前遇到过一次客户是虚拟化环境通过cdp回退然后类似方法处理ORA-600 16703直接把orachk备份表插入到tab$恢复),凸显了这位朋友在故障发生之后对于问题的准确判断和果断的应对能力.
有朋友和我反馈,他们数据库突然报大量ORA-600错误,业务无法正常操作,我分析相关日志确认:节点2重启之后节点1开始报大量ORA-600错误,但是节点一直处于open状态

Fri Jul 25 15:28:53 2025
Decreasing number of real time LMS from 3 to 0
Fri Jul 25 15:29:18 2025
Reconfiguration started (old inc 13, new inc 15)
List of instances:
 1 2 (myinst: 1) 
 Global Resource Directory frozen
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Fri Jul 25 15:29:18 2025
 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Fri Jul 25 15:29:18 2025
 LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Fri Jul 25 15:29:18 2025
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info 
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
Fri Jul 25 15:29:20 2025
minact-scn: Master returning as live inst:2 has inc# mismatch instinc:0 cur:15 errcnt:0
Fri Jul 25 15:30:07 2025
Errors in file /u01/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_252634.trc  (incident=77234):
ORA-00600: internal error code, arguments: [ktsircinfo_num1], [0], [0], [0], [],[],[],[],[],[],[],[]
Incident details in: /u01/oracle/diag/rdbms/orcl/orcl1/incident/incdir_77234/orcl1_ora_252634_i77234.trc
Fri Jul 25 15:30:18 2025
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Fri Jul 25 15:30:19 2025
Sweep [inc][77234]: completed
Sweep [inc2][77234]: completed
Fri Jul 25 15:30:27 2025
Errors in file /u01/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_261587.trc  (incident=76487):
ORA-00600: internal error code, arguments: [ktsircinfo_num1],[0],[0],[0], [], [], [], [], [], [], [], []

通过grep筛选报错信息

[root@iZbp11c0qyuuo1gr7j98upZ tmp]# egrep "ORA-00600|ORA-07445" alert_1.txt |sort -u
ORA-00600: internal error code, arguments: [25027], [0], [0], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kkpo_rcinfo_defstg:delseg], [28941391], [], [], [], []
ORA-00600: internal error code, arguments: [ktsircinfo_num1], [0], [0], [0], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [16659], [kqldtu], [DEL], [0], [35038924], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [16659], [kqldtu], [INS], [0], [277736], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [16659], [kqldtu], [INS], [0], [28829570], [], [], [], [], [], [], []
ORA-07445: exception encountered: core dump [qknSetParent()+9] [SIGSEGV] [ADDR:0x10354] 
   [PC:0x1A48B9B] [Address not mapped to object] []
ORA-07445: exception encountered: core dump [qksxaMoveQbAnnotations()+168] [SIGSEGV]
   [ADDR:0x20304] [PC:0x1594954] [Address not mapped to object] []
ORA-07445: 出现异常错误: 核心转储 [qknExpRegIni_int()+87] [SIGSEGV] [ADDR:0x8C] 
   [PC:0x1A4D729] [Address not mapped to object] []
ORA-07445: 出现异常错误: 核心转储 [qksxaMoveQbAnnotations()+168] [SIGSEGV] [ADDR:0x0] 
   [PC:0x1594954] [SI_KERNEL(general_protection)] []

既然是由于节点2重启导致节点1报错,那分析节点2重启相关情况,第一次重启成功之后,数据库开始报ORA-600错误

Fri Jul 25 15:29:29 2025
QMNC started with pid=46, OS id=363757 
Fri Jul 25 15:29:31 2025
minact-scn: Inst 2 is a slave inc#:15 mmon proc-id:363622 status:0x2
minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.00000000 gcalc-scn:0x0000.00000000
Fri Jul 25 15:29:33 2025
Errors in file /u01/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_m003_363779.trc  (incident=248519):
ORA-00600: internal error code, arguments: [kgmfvmi#3], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle/diag/rdbms/orcl/orcl2/incident/incdir_248519/orcl2_m003_363779_i248519.trc
Starting background process SMCO
Fri Jul 25 15:29:35 2025
SMCO started with pid=57, OS id=363802 
Fri Jul 25 15:29:35 2025
Completed: ALTER DATABASE OPEN /* db agent *//* {2:23784:2} */
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_m003_363779.trc  (incident=248520):
ORA-00600: internal error code, arguments: [kgmfvmi#3], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle/diag/rdbms/orcl/orcl2/incident/incdir_248520/orcl2_m003_363779_i248520.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Exception [type: SIGSEGV, Address not mapped to object][ADDR:0x10] [PC:0x2FDA4BB,kgmdelsis()+219][flags:0x0,count:1]
Errors in file /u01/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_m003_363779.trc  (incident=248521):
ORA-07445: exception encountered: core dump [kgmdelsis()+219] [SIGSEGV] 
  [ADDR:0x10] [PC:0x2FDA4BB] [Address not mapped to object] []
ORA-00600: internal error code, arguments: [kgmfvmi#3], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle/diag/rdbms/orcl/orcl2/incident/incdir_248521/orcl2_m003_363779_i248521.trc
Use ADRCI or Support Workbench to package the incident.
Fri Jul 25 15:29:47 2025
Errors in file /u01/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_363808.trc  (incident=248559):
ORA-00600: internal error code, arguments: [kkposds2], [18446744073709551615], [18446744073709551615], 
  [18446744073709551615], [], [], [], [], [], [], [], []

然后第二次重启数据库无法open成功,而是报ORA-600 16703错误

ALTER DATABASE OPEN /* db agent *//* {2:21799:2} */
Picked broadcast on commit scheme to generate SCNs
ARCH: STARTING ARCH PROCESSES
Fri Jul 25 15:41:23 2025
ARC0 started with pid=39, OS id=369231 
ARC0: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Fri Jul 25 15:41:24 2025
ARC1 started with pid=40, OS id=369242 
Fri Jul 25 15:41:24 2025
ARC2 started with pid=41, OS id=369244 
Fri Jul 25 15:41:24 2025
ARC3 started with pid=42, OS id=369246 
ARC1: Archival started
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
Thread 2 opened at log sequence 33585
  Current log# 7 seq# 33585 mem# 0: +DATA/orcl/onlinelog/group_7.269.1011373611
Successful open of redo thread 2
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file /u01/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_369210.trc  (incident=260494):
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle/diag/rdbms/orcl/orcl2/incident/incdir_260494/orcl2_ora_369210_i260494.trc
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
SUCCESS: diskgroup FRA was mounted
Fri Jul 25 15:41:30 2025
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_369210.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Errors in file /u01/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_369210.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 369210): terminating the instance due to error 704
Instance terminated by USER, pid = 369210
ORA-1092 signalled during: ALTER DATABASE OPEN /* db agent *//* {2:21799:2} */...
opiodr aborting process unknown ospid (369210) as a result of ORA-1092
Fri Jul 25 15:41:31 2025
ORA-1092 : opitsk aborting process

到这一步基本上就清晰了,大概率是遭遇到以前恢复的类似case,tab$数据被清空导致,类似案例
ORA-600 16703故障解析—tab$表被清空
警告:互联网中有oracle介质被注入恶意程序导致—ORA-600 16703
通过在故障主机上找到安装介质,验证md5确认该程序是被注入恶意代码程序
md5


这个库由于还有一个节点处于open状态,相对处理比较简单,直接把备份的表数据反向插入回去即可

SYS@orcl1> select count(1) from ORACHK3C08C86E063530510ACD937;

  COUNT(1)
----------
     20696

SYS@orcl1> insert into tab$ select * from ORACHK3C08C86E063530510ACD937;

20696 rows created.

SYS@orcl1> commit;

Commit complete.

SYS@orcl1> select object_name,to_char(CREATED,'yyyy-mm-dd hh24:mi:ss') from dba_objects 
          2 where object_name in('DBMS_SUPPORT_DBMONITOR','DBMS_SUPPORT_DBMONITORP');

OBJECT_NAME                                  TO_CHAR(CREATED,'YY
-------------------------------------------- -------------------
DBMS_SUPPORT_DBMONITORP                      2019-06-19 17:06:46
DBMS_SUPPORT_DBMONITOR                       2019-06-19 17:06:46

然后清理掉恶意脚本,分别重启两个节点,完成数据恢复任务
2025-07-26_215903_578


这次故障能够快速顺利的恢复,和客户发现故障之后保留第一现场,没有把一个open的节点也重启有很大关系,open的节点也重启了,那后续恢复工作会麻烦很多,效果可能也没有这样的完美.

2025年的Oracle 8.0.5数据库恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:2025年的Oracle 8.0.5数据库恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

在10多年前恢复过几个Oracle 8.0版本的库
记录8.0.5数据库恢复过程
ORACLE 8.0.5 ORA-01207故障恢复
记录一次AIX 4.3.0+ORACLE 8.0.5恢复过程
没有想到在2025年的7月份还有朋友咨询8.0.5的库恢复case.心中一阵小激动,回想到当年的很多过往(在邮储的机房里面恢复从保险柜中拿出来的小带库恢复8.0.5的库,问领导bbed资料答复网上都有的失落,朋友给我发一个dul的激动,拿到oracle vpn畅游在oracle的internal资料库的爽快等等),感觉这个8.0.5的库不单是一个case,更是一种情怀,大环境的去o,也是一种大浪逝去留下的无奈,不过总的来说也算为Oracle已经奉献了最好的青春和精力,也挺自豪的.这次的库恢复本身不难,简单的总结下:
准备环境
把数据文件发给了我,准备win xp环境的虚拟机并安装8.0.5的库(安装版本要和数据库文件版本一致)
props


把数据文件,redo等拷贝到虚拟机中,并使用rename file方式重命名文件路径

SVRMGR> alter database rename file 'D:\ORANT\DATABASE\SYS1ORCL.ORA' to 'C:\805\SYS1ORCL.ORA';
语句已处理。
SVRMGR> alter database rename file 'D:\ORANT\DATABASE\USR1ORCL.ORA' to 'C:\805\USR1ORCL.ORA';
语句已处理。
SVRMGR> alter database rename file 'D:\ORANT\DATABASE\RBS1ORCL.ORA' to 'C:\805\RBS1ORCL.ORA';
语句已处理。
SVRMGR> alter database rename file 'D:\ORANT\DATABASE\TMP1ORCL.ORA' to 'C:\805\TMP1ORCL.ORA';
语句已处理。
SVRMGR> alter database rename file 'D:\DATA\OXFF01' to 'C:\805\OXFF01';
语句已处理。
………………
SVRMGR> alter database rename file 'D:\DATA\XFF15' to 'C:\805\XFF15';
语句已处理。
SVRMGR> alter database rename file 'D:\DATA\XFF16' to 'C:\805\XFF16';
语句已处理。

Thu Jul 10 00:05:41 2025
alter database rename file 'D:\ORANT\DATABASE\LOG4ORCL.ORA' to 'C:\805\LOG4ORCL.ORA'
Thu Jul 10 00:05:41 2025
Completed: alter database rename file 'D:\ORANT\DATABASE\LOG4
Thu Jul 10 00:05:41 2025
alter database rename file 'D:\ORANT\DATABASE\LOG3ORCL.ORA' to 'C:\805\LOG3ORCL.ORA'
Completed: alter database rename file 'D:\ORANT\DATABASE\LOG3
Thu Jul 10 00:05:41 2025
alter database rename file 'D:\ORANT\DATABASE\LOG2ORCL.ORA' to 'C:\805\LOG2ORCL.ORA'
Completed: alter database rename file 'D:\ORANT\DATABASE\LOG2
Thu Jul 10 00:05:43 2025
alter database rename file 'D:\ORANT\DATABASE\LOG1ORCL.ORA' to 'C:\805\LOG1ORCL.ORA'
Completed: alter database rename file 'D:\ORANT\DATABASE\LOG1

尝试recover数据库

SVRMGR> recover database;
ORA-00283: ??????????
ORA-01122: ?????29????
ORA-01110: ????29?'C:\805\XFF15'
ORA-01200: 974848?????????2048000??????

报ORA-01200错误,比较明显29号文件本身大小应该是2048000个block,但是现在只有974848个

2025-06-30  11:30     4,194,306,048 XFF14
2022-06-30  09:02     1,996,490,752 XFF15
2022-06-30  09:02     4,194,306,048 XFF16

明显该XFF15文件大小和文件头记录的不匹配,对文件头进行修改(或者修改文件大小)类似处理方法:
bbed处理ORA-01200故障
记录一次ORA-01200完美恢复
ORA-01122 ORA-01200故障处理
ORA-1200/ORA-1207数据库恢复

BBED> map
 File: XFF15 (0)
 Block: 1                                     Dba:0x00000000
------------------------------------------------------------
 Data File Header

 struct kcvfh, 360 bytes                    @0

 ub4 tailchk                                @2044


BBED> p kcvfhhdr.kccfhfsz
ub4 kccfhfsz                                @44       0x001f4000  为16进制===>>等同10进制的2048000

继续尝试恢复并打开数据库

SVRMGR> recover database;
完成介质的恢复。
SVRMGR> alter database open;
语句已处理。
SVRMGR>

由于29号文件部分丢失,导出数据遭遇ORA-08103错误
模拟普通ORA-08103并解决
模拟极端ORA-08103并解决
数据库启动ORA-08103故障恢复
数据库打开遭遇ORA-08103故障处理
ORA-01092 ORA-00604 ORA-08103故障处理
ORA-8103


对于这种错误,可以按照行的方式使用plsql进行逐行抽取,但是由于涉及的表比较多,比较麻烦,我这里直接使用dul对其进行抽取异常表
dul

然后把导出来的dmp,结合dul恢复出来的异常表数据,整合到一起,完成本次8.0.5的数据库恢复
下次遇到该版本不知道是什么时候,截个图纪念下
8.0.5