obet处理ORA-704 ORA-604 ORA-1578故障

Posted on 2026 年 02 月 09 日 by 惜分飞

有客户数据库启动报ORA-704 ORA-604 ORA-1578错误,导致启动失败

alert日志报错

Fri Feb 06 23:38:43 2026
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_239325.trc  (incident=1008437):
ORA-01578: ORACLE data block corrupted (file # 1, block # 241)
ORA-01110: data file 1: '+DATADG/orcl/datafile/system.270.1015415183'
Fri Feb 06 23:38:46 2026
Redo thread 2 internally disabled at seq 6 (CKPT)
Archived Log entry 182212 added for thread 2 sequence 5 ID 0xd68b4385 dest 1:
ARC0: Archiving disabled thread 2 sequence 6
Archived Log entry 182213 added for thread 2 sequence 6 ID 0xd68b4385 dest 1:
Thread 1 advanced to log sequence 9 (LGWR switch)
  Current log# 1 seq# 9 mem# 0: +DATADG/orcl/onlinelog/group_1.302.1223816761
  Current log# 1 seq# 9 mem# 1: +ARCHDG/orcl/onlinelog/group_1.257.1223816761
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_239325.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01578: ORACLE data block corrupted (file # 1, block # 241)
ORA-01110: data file 1: '+DATADG/orcl/datafile/system.270.1015415183'
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_239325.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01578: ORACLE data block corrupted (file # 1, block # 241)
ORA-01110: data file 1: '+DATADG/orcl/datafile/system.270.1015415183'
Error 704 happened during db open, shutting down database
USER (ospid: 239325): terminating the instance due to error 704
Fri Feb 06 23:38:48 2026
Instance terminated by USER, pid = 239325
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (239325) as a result of ORA-1092
Fri Feb 06 23:38:48 2026
ORA-1092 : opitsk aborting process

对system文件进行dbv检测(客户通过asmcmd cp命令拷贝出来system文件)

DBVERIFY - Verification starting : FILE = /asm_backup/orcl/datafile/SYSTEM.270.1015415183

DBV-00200: Block, DBA 4194545, already marked corrupt
csc(0x0006.d75a0ed7) higher than block scn(0x0000.00000000)
Page 241 failed with check code 6054

DBV-00200: Block, DBA 4306619, already marked corrupt
csc(0x0006.d75a14f4) higher than block scn(0x0000.00000000)
Page 112315 failed with check code 6054


DBVERIFY - Verification complete

Total Pages Examined         : 738560
Total Pages Processed (Data) : 424109
Total Pages Failing   (Data) : 1
Total Pages Processed (Index): 287403
Total Pages Failing   (Index): 1
Total Pages Processed (Other): 3408
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 23640
Total Pages Marked Corrupt   : 2
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3613010282 (6.3613010282)

这里可以看到有两个block被标记为坏块原因是由于:csc(0×0006.d75a0ed7) higher than block scn(0×0000.00000000),坏块错误代码为:Page 241 failed with check code 6054.这种问题的原因是由于block的scn小于数据块的csc(一般是由于写丢失导致),对于这个问题,最近在obet中刚刚增加了repair blkscn功能进行修复,具体参考:obet修复csc higher than block scn类型坏块,使用工具修复操作(其他block类似修改)

OBET> set filename /datapool/asm_backup/orcl/datafile/SYSTEM.270.1015415183
filename set to: /datapool/asm_backup/orcl/datafile/SYSTEM.270.1015415183

OBET> set block 241
block set to: 241

OBET> backup
Created backup directory: backup_blk
Successfully backed up current block from current file to 
    /tmp/backup_blk/SYSTEM.270.1015415183_241.20260209122822

OBET> repair blkscn

Repairing SCN for block 241...

Current block SCN information:
  scn_base: 0x00000000 (0)
  scn_wrap: 0x0000 (0)

Block 1 reference information:
  hdr_base: 0xD75FF65E (3613390430)
  hdr_wrap: 0x0006 (6)

Current block CSC information:
  csc_base: 0xD75A0ED7 (3613003479)
  csc_wrap: 0x0006 (6)

Calculated values:
  blkcsc: 29382807255 (0x00000006D75A0ED7)
  blkscn: 0 (0x0000000000000000)

blkscn < blkcsc, SCN repair required.

SCN repair requires edit mode.
Please enable edit mode with 'set mode edit' first.

OBET> set mode edit
mode set to: edit

OBET> repair blkscn

Repairing SCN for block 241...

Current block SCN information:
  scn_base: 0x00000000 (0)
  scn_wrap: 0x0000 (0)

Block 1 reference information:
  hdr_base: 0xD75FF65E (3613390430)
  hdr_wrap: 0x0006 (6)

Current block CSC information:
  csc_base: 0xD75A0ED7 (3613003479)
  csc_wrap: 0x0006 (6)

Calculated values:
  blkcsc: 29382807255 (0x00000006D75A0ED7)
  blkscn: 0 (0x0000000000000000)

blkscn < blkcsc, SCN repair required.

New SCN values to write:
  new_scn_base: 0xD75FF65D (3613390429)
  new_scn_wrap: 0x0006 (6)

Confirm SCN repair operations:
File: /datapool/asm_backup/orcl/datafile/SYSTEM.270.1015415183
Block: 241
Operations:
  1. Write new scn_base (0xD75FF65D)
  2. Write new scn_wrap (0x0006)
  3. Fix seq_kcbh (0xFF -> 0x01)

Confirm? (Y/YES to proceed): y

[Undo information saved - use 'undo' command to restore original values]

Verification after SCN repair:
1. scn_base: 5DF65FD7 (expected: 5DF65FD7)
2. scn_wrap: 0600 (expected: 0600)
3. seq_kcbh: 0x01 OK

Block 241 SCN repair completed successfully.
Execute tailchk apply and sum apply for this block.

OBET> tailchk apply

Confirm applying tailchk:
File: /datapool/asm_backup/orcl/datafile/SYSTEM.270.1015415183
Block: 241
Offset in block: 8188 (file offset: 0x001E3FFC)
Original value: 0xFF060000
New value:      0x01065DF6
Confirm? (Y/YES to proceed): y
Verification successful: Stored tailchk matches calculated value (0x01065DF6).
Tailchk applied successfully.

OBET> sum apply

Confirm applying checksum:
File: /datapool/asm_backup/orcl/datafile/SYSTEM.270.1015415183
Block: 241
Offset in block: 16 (file offset: 0x001E2010)
Original value: 0x7330
New value:      0x2AE7
Confirm? (Y/YES to proceed): y
Verification successful: Stored checksum matches calculated value (0x2AE7).
Checksum applied successfully.

然后dbv检查数据文件

[oracle@db4 tmp]$ dbv file=/datapool/asm_backup/orcl/datafile/SYSTEM.270.1015415183

DBVERIFY: Release 11.2.0.4.0 - Production on Mon Feb 9 20:37:54 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE =
         /datapool/asm_backup/orcl/datafile/SYSTEM.270.1015415183


DBVERIFY - Verification complete

Total Pages Examined         : 738560
Total Pages Processed (Data) : 424109
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 287403
Total Pages Failing   (Index): 1
Total Pages Processed (Other): 3408
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 23640
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3613451063 (6.3613451063)

然后直接顺利打开数据库

SQL> alter database open;

Database altered.

由于该库在我接手之前已经做了大量的强制拉库等各种恢复尝试,因此对该库做逻辑导出,导入新库完成本次恢复任务

obet修复csc higher than block scn类型坏块

Posted on 2026 年 02 月 08 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：obet修复csc higher than block scn类型坏块

最近有客户数据库报ORA-01092 ORA-01578错误导致数据库无法open

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01578: ORACLE data block corrupted (file # 1, block # 2123)
ORA-01110: data file 1: '+DATADG/orcl/datafile/system.270.1015415183'
Process ID: 239325
Session ID: 1599 Serial number: 7

通过dbv检查确认是csc higher than block scn故障

DBVERIFY: Release 11.2.0.4.0 - Production on Sat Feb 7 03:47:12 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = +DATADG/orcl/datafile/SYSTEM.270.1015415183

DBV-00200: Block, DBA 4306619, already marked corrupt
csc(0x0006.d75a14f4) higher than block scn(0x0000.00000000)
Page 2123 failed with check code 6054


DBVERIFY - Verification complete

Total Pages Examined         : 738560
Total Pages Processed (Data) : 424109
Total Pages Failing   (Data) : 1
Total Pages Processed (Index): 287403
Total Pages Failing   (Index): 1
Total Pages Processed (Other): 3408
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 23640
Total Pages Marked Corrupt   : 1
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3613010282 (6.3613010282)

对于这样的故障,最近把他整合到了obet工具中,执行命令为repair blkscn [block x]进行修复

C:\Users\XFF>dbv file=H:\TEMP\SYSTEM01.DBF

DBVERIFY: Release 11.2.0.4.0 - Production on 星期日 2月 8 11:07:51 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = H:\TEMP\SYSTEM01.DBF
csc(0x0012.a0d783c4) higher than block scn(0x0000.00000000)
页 241 失败, 校验代码为 6054

DBVERIFY - 验证完成

检查的页总数: 325120
处理的页总数 (数据): 99921
失败的页总数 (数据): 1
处理的页总数 (索引): 34013
失败的页总数 (索引): 0
处理的页总数 (其他): 174241
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 16945
标记为损坏的总页数: 0
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 2698836688 (18.2698836688)

使用obet修复csc higher than block scn 故障

OBET> set filename H:\TEMP\SYSTEM01.DBF
filename set to: H:\TEMP\SYSTEM01.DBF

OBET> set block 241
block set to: 241

OBET> set mode edit
mode set to: edit

OBET> repair blkscn

Repairing SCN for block 241...

Current block SCN information:
  scn_base: 0x00000000 (0)
  scn_wrap: 0x0000 (0)

Block 1 reference information:
  hdr_base: 0xA0DCE449 (2698830921)
  hdr_wrap: 0x0012 (18)

Current block CSC information:
  csc_base: 0xA0D783C4 (2698478532)
  csc_wrap: 0x0012 (18)

Calculated values:
  blkcsc: 80007889860 (0x00000012A0D783C4)
  blkscn: 0 (0x0000000000000000)

blkscn < blkcsc, SCN repair required.

New SCN values to write:
  new_scn_base: 0xA0DCE448 (2698830920)
  new_scn_wrap: 0x0012 (18)

Confirm SCN repair operations:
File: H:\TEMP\SYSTEM01.DBF
Block: 241
Operations:
  1. Write new scn_base (0xA0DCE448)
  2. Write new scn_wrap (0x0012)

Confirm? (Y/YES to proceed): y

[Undo information saved - use 'undo' command to restore original values]

Verification after SCN repair:
1. scn_base: 48E4DCA0 (expected: 48E4DCA0)
2. scn_wrap: 1200 (expected: 1200)
3. seq_kcbh: 0x01 OK

Block 241 SCN repair completed successfully.
Execute tailchk apply and sum apply for this block.

OBET> tailchk apply

Confirm applying tailchk:
File: H:\TEMP\SYSTEM01.DBF
Block: 241
Offset in block: 8188 (file offset: 0x001E3FFC)
Original value: 0x01060000
New value:      0x010648E4
Confirm? (Y/YES to proceed): y
Verification successful: Stored tailchk matches calculated value (0x010648E4).
Tailchk applied successfully.

OBET> sum apply

Confirm applying checksum:
File: H:\TEMP\SYSTEM01.DBF
Block: 241
Offset in block: 16 (file offset: 0x001E2010)
Original value: 0xAD0F
New value:      0x63AF
Confirm? (Y/YES to proceed): y
Verification successful: Stored checksum matches calculated value (0x63AF).
Checksum applied successfully.

dbv验证该错误已经修复

C:\Users\XFF>dbv file=H:\TEMP\SYSTEM01.DBF

DBVERIFY: Release 11.2.0.4.0 - Production on 星期日 2月 8 11:11:12 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = H:\TEMP\SYSTEM01.DBF


DBVERIFY - 验证完成

检查的页总数: 325120
处理的页总数 (数据): 99921
失败的页总数 (数据): 0
处理的页总数 (索引): 34013
失败的页总数 (索引): 0
处理的页总数 (其他): 174241
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 16945
标记为损坏的总页数: 0
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 2698836688 (18.2698836688)

obet工具下载地址:obet下载
obet工具使用说明:obet使用说明

ORA-600 kcratr_nab_less_than_odr和ORA-600 4193故障处理

Posted on 2026 年 02 月 06 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 kcratr_nab_less_than_odr和ORA-600 4193故障处理

数据库启动报ORA-600 kcratr_nab_less_than_odr

alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 2930 KB redo, 1357 data blocks need recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_19096.trc  (incident=576634):
ORA-00600: ??????, ??: [kcratr_nab_less_than_odr], [1], [7859], [62180], [63172], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_576634\orcl_ora_19096_i576634.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Aborting crash recovery due to error 600
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_19096.trc:
ORA-00600: ??????, ??: [kcratr_nab_less_than_odr], [1], [7859], [62180], [63172], [], [], [], [], [], [], []
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_19096.trc:
ORA-00600: ??????, ??: [kcratr_nab_less_than_odr], [1], [7859], [62180], [63172], [], [], [], [], [], [], []
ORA-600 signalled during: alter database open...

这个错误一般常见的是11.2.0.1的数据库异常关机了容易遇到
又一例ORA-600 kcratr_nab_less_than_odr
ORA-600 kcratr_nab_less_than_odr故障解决
这个客户是11.2.0.4的库(在这个版本中该错误相对较少,虽然也遇到过几次)
11.2.0.4库中遇到ORA-600 kcratr_nab_less_than_odr报错
分析日志发现是由于之前io比较慢导致写入异常导致

Fri Feb 06 00:32:59 2026
********************* ATTENTION: ******************** 
 The controlfile header block returned by the OS
 has a sequence number that is too old. 
 The controlfile might be corrupted.
 PLEASE DO NOT ATTEMPT TO START UP THE INSTANCE 
 without following the steps below.
 RE-STARTING THE INSTANCE CAN CAUSE SERIOUS DAMAGE 
 TO THE DATABASE, if the controlfile is truly corrupted.
 In order to re-start the instance safely, 
 please do the following:
 (1) Save all copies of the controlfile for later 
     analysis and contact your OS vendor and Oracle support.
 (2) Mount the instance and issue: 
     ALTER DATABASE BACKUP CONTROLFILE TO TRACE;
 (3) Unmount the instance. 
 (4) Use the script in the trace file to
     RE-CREATE THE CONTROLFILE and open the database. 
*****************************************************
CKPT (ospid: 2600): terminating the instance
Fri Feb 06 00:33:00 2026
System state dump requested by (instance=1, osid=2600 (CKPT)), summary=[abnormal instance termination].
System State dumped to trace file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_diag_2488_20260206003300.trc
Fri Feb 06 00:33:01 2026
opiodr aborting process unknown ospid (3420) as a result of ORA-1092

处理这个错误相对比较简单,重建控制文件即可

但是后续数据库出现ORA-600 4193错误

Fri Feb 06 02:57:34 2026
ALTER DATABASE OPEN
Thread 1 opened at log sequence 7860
  Current log# 1 seq# 7860 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\orcl\REDO01.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
[4044] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:5166812 end:5167140 diff:328 (3 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_smon_3868.trc  (incident=640589):
ORA-00600: 内部错误代码, 参数: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_640589\orcl_smon_3868_i640589.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Starting background process QMNC
Fri Feb 06 02:57:36 2026
QMNC started with pid=21, OS id=5008 
Completed: ALTER DATABASE OPEN
Fri Feb 06 02:57:37 2026
db_recovery_file_dest_size of 4182 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Block recovery from logseq 7860, block 132954 to scn 55142160294
Recovery of Online Redo Log: Thread 1 Group 1 Seq 7860 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\orcl\REDO01.LOG
Block recovery completed at rba 7860.132998.16, scn 12.3602552743
Block recovery from logseq 7860, block 132954 to scn 55142160257
Recovery of Online Redo Log: Thread 1 Group 1 Seq 7860 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\orcl\REDO01.LOG
Block recovery completed at rba 7860.132954.16, scn 12.3602552707
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_smon_3868.trc:
ORA-01595: 释放区 (5) 回退段 (21) 时出错
ORA-00600: 内部错误代码, 参数: [4193], [], [], [], [], [], [], [], [], [], [], []
Flush retried for xcb 0x96c82600, pmd 0x8c6a70b8
Block recovery from logseq 7860, block 132954 to scn 55142160294
Recovery of Online Redo Log: Thread 1 Group 1 Seq 7860 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\orcl\REDO01.LOG
Block recovery completed at rba 7860.132998.16, scn 12.3602552743
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_pmon_2728.trc:
ORA-00600: 内部错误代码, 参数: [4193], [], [], [], [], [], [], [], [], [], [], []
PMON (ospid: 2728): terminating the instance due to error 472
System state dump requested by (instance=1, osid=2728 (PMON)), summary=[abnormal instance termination].
Fri Feb 06 02:58:17 2026
Instance terminated by PMON, pid = 2728

这个错误相对比较简单,由于undo回滚段异常,处理掉异常undo之后,数据库正常,完成本次恢复任务

aix环境10g由于控制器异常导致ORA-600 4000故障处理

Posted on 2026 年 02 月 01 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：aix环境10g由于控制器异常导致ORA-600 4000故障处理

一台运行在aix环境下面的10.2.0.1的数据库

由于控制器异常导致数据库启动的时候报ORA-600 4000错误

Wed Jan 28 18:17:06 2026
Completed crash recovery at
 Thread 1: logseq 499321, block 14459, scn 17457591400427
 257 data blocks read, 41 data blocks written, 14457 redo blocks read
Wed Jan 28 18:17:06 2026
Thread 1 advanced to log sequence 499322
Thread 1 opened at log sequence 499322
  Current log# 2 seq# 499322 mem# 0: /dev/rrk_redo2
Successful open of redo thread 1
Wed Jan 28 18:17:06 2026
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Jan 28 18:17:06 2026
SMON: enabling cache recovery
Wed Jan 28 18:17:07 2026
Errors in file /u01/app/oracle/admin/orcl1/udump/orcl1_ora_16187632.trc:
ORA-00600: internal error code, arguments: [4000], [9], [], [], [], [], [], []
Wed Jan 28 18:17:08 2026
Errors in file /u01/app/oracle/admin/orcl1/udump/orcl1_ora_16187632.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [4000], [9], [], [], [], [], [], []
Wed Jan 28 18:17:08 2026
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Instance terminated by USER, pid = 16187632
ORA-1092 signalled during: alter database open...

ORA-600 4000这个是在10g版本中非常常见的一个错误,一般是由于对应的block上面有事务没有提交或者scn过大导致,跟踪数据库启动过程发现在以下sql语句报错,而且报错为file 1 block 27527

EXEC #2:c=0,e=62,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,tim=437860213380
WAIT #2: nam='db file sequential read' ela= 139 file#=1 block#=27527 blocks=1 obj#=-1 tim=437860213555
*** 2026-01-31 15:31:40.417
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [4000], [9], [], [], [], [], [], []
Current SQL statement for this session:
select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1

这里可以获取到两个有效信息:
1. 报错block为：file 1 block 27227
2. 报错sql为：select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1

进一步对报错数据块进行分析

Block header dump:  0x00406b87
 Object id on Block? Y
 seg/obj: 0x2  csc: 0xfe0.a99d3719  itc: 2  flg: -  typ: 1 - DATA
     fsl: 0  fnx: 0x0 ver: 0x01
 
 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x000b.014.00f1bb09  0x008027a5.eada.0e  C---    0  scn 0x0fda.f353b198
0x02   0x0009.01e.00212e2a  0x01400cdf.113e.01  --U-    1  fsc 0x0000.a99d371b

这里可以确认几个有效信息:
1. 该block的csc scn为:17457592743705
2. 一条已经提交的事务的scn为:17433059635608
3. 还有一条没有提交的事务,使用的回滚段为9,这个和我们报错的ORA-600[4000][9]这个回滚段名称匹配上

基于上述分析,我们需要确认两件事情:
1. 通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)脚本检查结果,确认当前是数据库的文件头scn为:17457591420434小于该block的csc scn

2. itl操上面有一个锁需要提交,通过bbed工具对其进行提交,然后得出dump block信息

Block header dump:  0x00406b87
 Object id on Block? Y
 seg/obj: 0x2  csc: 0xfe0.a99d3719  itc: 2  flg: -  typ: 1 - DATA
     fsl: 0  fnx: 0x0 ver: 0x01

 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x000b.014.00f1bb09  0x008027a5.eada.0e  C---    0  scn 0x0fda.f353b198
0x02   0x0009.01e.00212e2a  0x01400cdf.113e.01  C---    0  scn 0x0000.a99d371b

修改好itl之后,先尝试重启库,如果不出意外应该会报ORA-600 2662类似错误（因为前面分析了csc scn大于文件头scn的问题）

这里的ORA-600 2662中的4221831就是报错的rdba地址(10进制),通过dbms_utility.data_block_address_file转换

SQL> select dbms_utility.data_block_address_file(4221831) "file",
  2  dbms_utility.data_block_address_block(4221831) "block"
  3   from dual;

      file      block
---------- ----------
         1      27527

得出报错的ORA-600 2662的block就是我们之前分析和修复的itl块,通过修改该块scn或者修改数据库scn,该库均可open,后续就是安排导出数据导入新库的活

ORA-600 3716故障处理

Posted on 2026 年 01 月 31 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 3716故障处理

数据库由于ORA-600 kcvent_intcic_zero_half错误导致实例重启

Reconfiguration started (old inc 8, new inc 10)
List of instances:
 1 2 (myinst: 2) 
 Nested reconfiguration detected. 
 Global Resource Directory frozen
 Communication channels reestablished
 * domain 0 valid = 1 according to instance 1 
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info 
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
Fri Jan 30 22:07:08 2026
minact-scn: Master returning as live inst:1 has inc# mismatch instinc:0 cur:10 errcnt:0
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
Fri Jan 30 22:07:17 2026
Errors in file /u01/oracle11g/diag/rdbms/hisdb/hisdb2/trace/hisdb2_ckpt_1748.trc  (incident=228496):
ORA-00600: internal error code, arguments: [kcvent_intcic_zero_half], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle11g/diag/rdbms/hisdb/hisdb2/incident/incdir_228496/hisdb2_ckpt_1748_i228496.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/oracle11g/diag/rdbms/hisdb/hisdb2/trace/hisdb2_ckpt_1748.trc:
ORA-00600: internal error code, arguments: [kcvent_intcic_zero_half], [], [], [], [], [], [], [], [], [], [], []
CKPT (ospid: 1748): terminating the instance due to error 469
Fri Jan 30 22:07:18 2026
ORA-1092 : opitsk aborting process
Fri Jan 30 22:07:19 2026
System state dump requested by (instance=2, osid=1748 (CKPT)), summary=[abnormal instance termination].

初步分析该错误是由于oracle bug引起,在11.2.0.4版本中比较少见,主要常见于19c版本中

正常情况下,实例重启即可,这个客户比较不幸,数据库重启遭遇ORA-600 3716错误

ALTER DATABASE OPEN /* db agent *//* {0:11:28647} */
This instance was first to open
Beginning crash recovery of 2 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 1120 KB redo, 386 data blocks need recovery
Started redo application at
 Thread 2: logseq 847, block 641786
Recovery of Online Redo Log: Thread 2 Group 3 Seq 847 Reading mem 0
  Mem# 0: +DATADISK/hisdb/onlinelog/group_3.267.1109863469
  Mem# 1: +ARCHIVEDISK/hisdb/onlinelog/group_3.261.1109863469
Completed redo application of 0.38MB
Completed crash recovery at
 Thread 2: logseq 847, block 644027, scn 20674490933
 386 data blocks read, 0 data blocks written, 1120 redo k-bytes read
ARCH: STARTING ARCH PROCESSES
Fri Jan 30 22:07:47 2026
ARC0 started with pid=53, OS id=332 
ARC0: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Fri Jan 30 22:07:48 2026
ARC1 started with pid=54, OS id=364 
Fri Jan 30 22:07:48 2026
ARC2 started with pid=55, OS id=366 
Fri Jan 30 22:07:48 2026
ARC3 started with pid=56, OS id=368 
ARC1: Archival started
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
Errors in file /u01/oracle11g/diag/rdbms/hisdb/hisdb2/trace/hisdb2_ora_29981.trc  (incident=240679):
ORA-00600: ??????, ??: [3716], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle11g/diag/rdbms/hisdb/hisdb2/incident/incdir_240679/hisdb2_ora_29981_i240679.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Abort recovery for domain 0
Aborting crash recovery due to error 600
Errors in file /u01/oracle11g/diag/rdbms/hisdb/hisdb2/trace/hisdb2_ora_29981.trc:
ORA-00600: ??????, ??: [3716], [], [], [], [], [], [], [], [], [], [], []
Abort recovery for domain 0
Errors in file /u01/oracle11g/diag/rdbms/hisdb/hisdb2/trace/hisdb2_ora_29981.trc:
ORA-00600: ??????, ??: [3716], [], [], [], [], [], [], [], [], [], [], []
ORA-600 signalled during: ALTER DATABASE OPEN /* db agent *//* {0:11:28647} */...

接手客户故障尝试启动数据库也报ORA-600 3716错误

SQL> startup
ORACLE instance started.

Total System Global Area 2.0176E+10 bytes
Fixed Size                  2261928 bytes
Variable Size            3288337496 bytes
Database Buffers         1.6844E+10 bytes
Redo Buffers               41463808 bytes
Database mounted.
ORA-00600: internal error code, arguments: [3716], [], [], []

根据官方描述该问题

那解决该问题相对比较简单

SQL>@rectl.sql

Control file created.
SQL> recover database;
Media recovery complete.
SQL> alter database open;

Database altered.

完美恢复这次故障,实现数据0丢失,业务直接使用

13429648788

qq咨询

专业Oracle数据库恢复技术支持

专业Oracle数据库恢复，或许是您恢复数据的最后机会@phone:17813235971

Author Archives: 惜分飞

obet处理ORA-704 ORA-604 ORA-1578故障

obet修复csc higher than block scn类型坏块

ORA-600 kcratr_nab_less_than_odr和ORA-600 4193故障处理

aix环境10g由于控制器异常导致ORA-600 4000故障处理

ORA-600 3716故障处理