ORA-600 kddummy_blkchk 数据库循环重启

Posted on 2025 年 05 月 08 日 by 惜分飞

一个运行在hp平台的10g rac突然异常,之后运行一段时间就自动重启,客户让对其进行分析和解决

Thu May  8 06:23:21 2025
ALTER DATABASE OPEN
Picked broadcast on commit scheme to generate SCNs
Thu May  8 06:23:21 2025
Thread 1 opened at log sequence 74302
  Current log# 1 seq# 74302 mem# 0: /dev/vgdata/rrac_redo01
Successful open of redo thread 1
Thu May  8 06:23:21 2025
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu May  8 06:23:21 2025
SMON: enabling cache recovery
Thu May  8 06:23:22 2025
Successfully onlined Undo Tablespace 1.
Thu May  8 06:23:22 2025
SMON: enabling tx recovery
Thu May  8 06:23:22 2025
Database Characterset is ZHS16CGB231280
Opening with internal Resource Manager plan
where NUMA PG = 1, CPUs = 4
replication_dependency_tracking turned off (no async multimaster replication found)
Thu May  8 06:23:23 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
ORA-00600: internal error code, arguments: [kddummy_blkchk], [118], [333578], [18019], [], [], [], []
Starting background process QMNC
QMNC started with pid=22, OS id=15792
Thu May  8 06:23:25 2025
ORACLE Instance orcl1 (pid = 13) - Error 607 encountered while recovering transaction (9, 33) on object 775794.
Thu May  8 06:23:25 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kddummy_blkchk], [118], [333578], [18019], [], [], [], []
Thu May  8 06:23:26 2025
Completed: ALTER DATABASE OPEN
Thu May  8 06:23:26 2025
Doing block recovery for file 118 block 333578
Block recovery from logseq 74302, block 22 to scn 46740761996
Thu May  8 06:23:26 2025
Recovery of Online Redo Log: Thread 1 Group 1 Seq 74302 Reading mem 0
  Mem# 0: /dev/vgdata/rrac_redo01
Block recovery stopped at EOT rba 74302.33.16
Block recovery completed at rba 74302.33.16, scn 10.3791089036
Thu May  8 06:23:33 2025
Trace dumping is performing id=[cdmp_20250508062324]
Thu May  8 06:25:55 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
ORA-00600: internal error code, arguments: [kddummy_blkchk], [118], [333578], [18019], [], [], [], []
Thu May  8 06:25:58 2025
ORACLE Instance orcl1 (pid = 13) - Error 607 encountered while recovering transaction (9, 33) on object 775794.
Thu May  8 06:27:32 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
ORA-00600: internal error code, arguments: [kddummy_blkchk], [118], [333578], [18019], [], [], [], []
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
Doing block recovery for file 118 block 333578
Block recovery from logseq 74302, block 372 to scn 46740952565
Thu May  8 06:27:41 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
ORA-00600: internal error code, arguments: [kddummy_blkchk], [118], [333578], [18019], [], [], [], []
ORACLE Instance orcl1 (pid = 13) - Error 607 encountered while recovering transaction (9, 33) on object 775794.
Thu May  8 06:27:43 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kddummy_blkchk], [118], [333578], [18019], [], [], [], []
Doing block recovery for file 118 block 333578
Block recovery from logseq 74302, block 372 to scn 46740952565
Thu May  8 06:27:45 2025
Recovery of Online Redo Log: Thread 1 Group 1 Seq 74302 Reading mem 0
  Mem# 0: /dev/vgdata/rrac_redo01
Block recovery completed at rba 74302.394.16, scn 10.3791279606
Thu May  8 06:27:47 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
ORA-00600: internal error code, arguments: [kddummy_blkchk], [118], [333578], [18019], [], [], [], []
Thu May  8 06:28:07 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_pmon_15690.trc:
ORA-00474: SMON process terminated with error
Thu May  8 06:28:07 2025
PMON: terminating instance due to error 474

这个数据库重启是由于smon进程异常导致数据库关闭,然后rac自动拉起数据库,从而出现了循环重启启动,smon异常的主要原因是由于“ORACLE Instance orcl1 (pid = 13) – Error 607 encountered while recovering transaction (9, 33) on object 775794.”这个表示有实物要回滚,但是遇到了ORA-600 kddummy_blkchk异常,无法完成回滚从而使得smon进程异常进而使得数据实例crash.对于这个问题处理相对比较简单
1. 通过ORA-600 kddummy_blkchk找出来报错对象

ORA-00600: internal error code, arguments: [kddummy_blkchk], [a], [b], 1

ARGUMENTS:
Arg [a] Absolute file number
Arg [b] Block number
Arg 1 Internal error code returned from kcbchk() which indicates the problem encountered.

2. 屏蔽实物回滚,打开数据库,然后对通过dba_extents 查询出来的对象/或者根据事务报错的object信息查询出来对象进行重建,完成本次恢复任务

关于ORA-00600: internal error code, arguments: [kddummy_blkchk], [a], [b], 相关目前已知bug

Bug	Fixed	Description
12349316	11.2.0.4, 12.1.0.2, 12.2.0.1	DBMS_SPACE_ADMIN.TABLESPACE_FIX_BITMAPS fails with ORA-600 [kddummy_blkchk] / ORA-600 [kdBlkCheckError] / ORA-607
17325413	11.2.0.3.BP23, 11.2.0.4.2, 11.2.0.4.BP04, 12.1.0.1.3, 12.1.0.2, 12.2.0.1	Drop column with DEFAULT value and NOT NULL definition ends up with Dropped Column Data still on Disk leading to Corruption
13715932	11.2.0.4, 12.1.0.1	ORA-600 [kddummy_blkchk] [18038] while adding extents to a large datafile
12417369	11.2.0.2.5, 11.2.0.2.BP13, 11.2.0.2.GIPSU05, 11.2.0.3, 12.1.0.1	Block corruption from rollback on compressed table
10324526	10.2.0.5.4, 11.1.0.7.8, 11.2.0.2.3, 11.2.0.2.BP06, 11.2.0.3, 12.1.0.1	ORA-600 [kddummy_blkchk] [6106] / corruption on COMPRESS table in TTS
10113224	11.2.0.3, 12.1.0.1	Index coalesce may generate invalid redo if blocks in the buffer cache are invalid/corrupted
9726702	11.2.0.3, 12.1.0.1	DBMS_SPACE_ADMIN.assm_segment_verify reports HWM related inconsistencies
9724970	11.2.0.1.BP08, 11.2.0.2.2, 11.2.0.2.BP02, 11.2.0.3, 12.1.0.1	Block Corruption with PDML UPDATE. ORA_600 [4511] OERI[kdblkcheckerror] by block check
9711859	10.2.0.5.1, 11.1.0.7.6, 11.2.0.2, 12.1.0.1	ORA-600 [ktsptrn_fix-extmap] / ORA-600 [kdblkcheckerror] during extent allocation caused by bug 8198906
9581240	11.1.0.7.9, 11.2.0.2, 12.1.0.1	Corruption / ORA-600 [kddummy_blkchk] [6101] / ORA-600 [7999] after RENAME operation during ROLLBACK
9350204	11.2.0.3, 12.1.0.1	Spurious ORA-600 [kddummy_blkchk] .. [6145] during CR operations on tables with ROWDEPENDENCIES
9231605	11.1.0.7.4, 11.2.0.1.3, 11.2.0.1.BP02, 11.2.0.2, 12.1.0.1	Block corruption with missing row on a compressed table after DELETE
9119771	11.2.0.2, 12.1.0.1	OERI [kddummy_blkchk]…[6108] from ‘SHRINK SPACE CASCADE’
9019113	11.2.0.1.BP02, 11.2.0.2, 12.1.0.1	ORA-600 [17182] ORA-7445 [memcpy] ORA-600 [kdBlkCheckError] for OLTP COMPRESS table in OLTP Compression REDO during RECOVERY
8951812	11.2.0.2, 12.1.0.1	Corrupt index by rebuild online. Possible OERI [kddummy_blkchk] by SMON
8720802	10.2.0.5, 11.2.0.1.BP07, 11.2.0.2, 12.1.0.1	Add check for row piece pointing to itself (db_block_checking,dbv,rman,analyze)
8331063	11.2.0.3, 12.1.0.1	Corrupt Undo. ORA-600 [2015] in Undo Block During Rollback
6523037	11.2.0.1.BP07, 11.2.0.2.2, 11.2.0.2.BP01, 11.2.0.3, 12.1.0.1	Corruption / ORA-600 [kddummy_blkchk] [6110] on update
8277580	11.1.0.7.2, 11.2.0.1, 11.2.0.2, 12.1.0.1	Corruption on compressed tables during Recovery and Quick Multi Delete (QMD).
9964102	11.2.0.1	OERI:2015 / OERI:kddummy_blkchk / undo corruption from supplementat logging with compressed tables
8613137	11.1.0.7.2, 11.2.0.1	ORA-600 updating table with DEFERRED constraints
8360192	11.1.0.7.6, 11.2.0.1	ORA-600 [kdBlkCheckError] [6110] / corruption from insert
8239658	10.2.0.5, 11.2.0.1	Dump / corruption writing row to compressed table
8198906	10.2.0.5, 11.2.0.1	OERI [kddummy_blkchk] / OERI [5467] for an aborted transaction of allocating extents
7715244	11.1.0.7.2, 11.2.0.1	Corruption on compressed tables. Error codes 6103 / 6110
7662491	10.2.0.4.2, 10.2.0.5, 11.1.0.7.4, 11.2.0.1	Array Update can corrupt a row. Errors OERI[kghstack_free1] or OERI[kddummy_blkchk][6110]
7411865	10.2.0.4.2, 10.2.0.5, 11.1.0.7.1, 11.2.0.1	OERI:13030 / ORA-1407 / block corruption from UPDATE .. RETURNING DML with trigger
7331181	11.2.0.1	ORA-1555 or OERI [kddummy_blkchk] [file#] [block#] [6126] during CR Rollback in query
7293156	11.1.0.7, 11.2.0.1	ORA-600 [2023] by Parallel Transaction Rollback when applying Multi-block undo Head-piece / Tail-piece
7041254	11.1.0.7.5, 11.2.0.1	ORA-19661 during RMAN restore check logical of compressed backup / IOT dummy key
6760697	10.2.0.4.3, 10.2.0.5, 11.1.0.7, 11.2.0.1	DBMS_SPACE_ADMIN.ASSM_SEGMENT_VERIFY does not detect certain segment header block corruption
6647480	10.2.0.4.4, 10.2.0.5, 11.1.0.7.3, 11.2.0.1	Corruption / OERI [kddummy_blkchk] .. [18021] with ASSM
6134368	10.2.0.5, 11.2.0.1	ORA-1407 / block corruption from UPDATE .. RETURNING DML with trigger – SUPERCEDED
6057203	10.2.0.4, 11.1.0.7, 11.2.0.1	Corruption with zero length column (ZLC) / OERI [kcbchg1_6] from Parallel update
6653934	10.2.0.4.2, 10.2.0.5, 11.1.0.7	Dump / block corruption from ONLINE segment shrink with ROWDEPENDENCIES
6674196	10.2.0.4, 10.2.0.5, 11.1.0.6	OERI / buffer cache corruption using ASM, OCFS or any ksfd client like ODM
5599596	10.2.0.4, 11.1.0.6	Block corruption / OERI [kddummy_blkchk] on clustered or compressed tables
5496041	10.2.0.4, 11.1.0.6	OERI[6006] / index corruption on compressed index
5386204	10.2.0.4.1, 10.2.0.5, 11.1.0.6	Block corruption / OERI[kddummy_blkchk] after direct load of ASSM segment
5363584	10.2.0.4, 11.1.0.6	Array insert into table can corrupt redo
4602031	10.2.0.2, 11.1.0.6	Block corruption from UPDATE or MERGE into compressed table
4493447	11.1.0.6	Spurious ORA-600 [kddummy_blkchk] [file#] [block#] [6145] on rollback of array update
4329302	11.1.0.6	OERI [kddummy_blkchk] [file#] [block#] [6145] on rollback of update with logminer
6075487	10.2.0.4	OERI[kddummy_blkchk]..[18020/18026] for DDL on plugged ASSM tablespace with FLASHBACK
4054640	10.1.0.5, 10.2.0.1	Block corruption / OERI [kddummy_blkchk] at physical standby
4000840	10.1.0.4, 10.2.0.1, 9.2.0.7	Update of a row with more than 255 columns can cause block corruption
3772033	9.2.0.7, 10.1.0.4, 10.2.0.1	OERI[ktspfmb_create1] creating a LOB in ASSM using 2k blocksize

记录一次asm disk加入到vg通过恢复直接open库的案例

Posted on 2025 年 05 月 05 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：记录一次asm disk加入到vg通过恢复直接open库的案例

客户在不清楚磁盘被asm disk使用的情况下,直接分区做pv,加入到vg中并且分配给了lv,导致数据库异常

通过操作系统层面分析,确认客户把data磁盘组的一个磁盘给处理掉了,导致数据库报错

WARNING: ASMB force dismounting group 2 (DATA) due to failover
SUCCESS: diskgroup DATA was dismounted
2025-05-04T07:03:19.910082+08:00
KCF: read, write or open error, block=0x201544 online=1
        file=102 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_61.dbf'
        error=15078 txt: ''
2025-05-04T07:03:19.918972+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbwc_18507.trc:
2025-05-04T07:03:19.952045+08:00
KCF: read, write or open error, block=0x2013e7 online=1
        file=102 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_61.dbf'
        error=15078 txt: ''
2025-05-04T07:03:19.964538+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbw7_18486.trc:
2025-05-04T07:03:19.967133+08:00
KCF: read, write or open error, block=0x230e71 online=1
        file=105 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_64.dbf'
        error=15078 txt: ''
2025-05-04T07:03:19.973289+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbw2_18466.trc:
2025-05-04T07:03:19.978514+08:00
KCF: read, write or open error, block=0x1f6e91 online=1
        file=86 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_52.dbf'
        error=15078 txt: ''
2025-05-04T07:03:19.991060+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbwd_18511.trc:
2025-05-04T07:03:19.995762+08:00
KCF: read, write or open error, block=0x7f8 online=1
        file=15 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/undotbs01.dbf'
        error=15078 txt: ''
2025-05-04T07:03:20.006862+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbwa_18498.trc:
2025-05-04T07:03:20.020739+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_imr0_18937.trc:

这个客户比较幸运,处理该磁盘之后,没有往对应的lv中写入太多数据,导致覆盖部分很少

[root@rac01 rules.d]# df -h
文件系统               容量  已用  可用 已用% 挂载点
/dev/mapper/nlas-root  800G  272G  528G   34% /
devtmpfs               284G     0  284G    0% /dev
tmpfs                  284G  637M  283G    1% /dev/shm
tmpfs                  284G  4.0G  280G    2% /run
tmpfs                  284G     0  284G    0% /sys/fs/cgroup
/dev/mapper/nlas-home  200G   64M  200G    1% /home
/dev/sda1              197M  158M   40M   80% /boot
tmpfs                   57G   40K   57G    1% /run/user/0
tmpfs                   57G   48K   57G    1% /run/user/1000
[root@rac01 rules.d]# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda2  nlas lvm2 a--  564.00g    0 
  /dev/sdb1  nlas lvm2 a--   <2.00t 1.51t
[root@rac01 rules.d]# vgs
  VG   #PV #LV #SN Attr   VSize VFree
  nlas   2   3   0 wz--n- 2.55t 1.51t
[root@rac01 rules.d]# lvs
  LV   VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home nlas -wi-ao---- 200.00g                                                    
  root nlas -wi-ao---- 800.00g                                                    
  swap nlas -wi-ao----  64.00g

通过底层对磁盘进行分析,发现备份的磁盘头均以损坏,通过深入分析确认f1b1在sdb磁盘的第10个au上,通过相关信息,使用dul工具加载磁盘组,并分析元数据信息,发现恢复数据需要的元数据都可以正常加载

直接使用dul抽取数据到文件系统,然后open数据库成功

然后通过rman 检测坏块(3T多的库只有不到5000个坏块,相对来说效果非常好),对于坏块对象进行处理,完美完成本次恢复工作.对于这次能够有这样好的恢复效果有几个因素:
1)asm disk 加入到vg,并分配给lv之后,立刻停止写入操作,避免了因为写入数据而覆盖asm 磁盘的带来的风险
2)由于是19c库,默认au为4M,使得数据库文件数据相对比较靠后,覆盖几率小了一点
3)由于文件系统是xfs,相对覆盖比ext4会少很多
4)是云环境的ssd磁盘,没有触发trim功能
以前类似asm disk异常恢复的相关case汇总:
asm磁盘加入vg恢复
asm磁盘dd破坏恢复
asm磁盘分区丢失恢复
pvid=yes导致asm无法mount
win asm disk header 异常恢复
又一例asm disk 加入vg故障
pvcreate asm disk导致asm磁盘组异常恢复
asm disk被加入到另外一个磁盘组故障恢复
再一例asm disk被误加入vg并且扩容lv恢复
再一起asm disk被格式化成ext3文件系统故障恢复
一次完美的asm disk被格式化ntfs恢复
asm disk误设置pvid导致asm diskgroup无法mount恢复
asm disk被分区,格式化为ext4恢复
oracle asm disk格式化恢复—格式化为ext4文件系统
分享oracleasm createdisk重新创建asm disk后数据0丢失恢复案例

Oracle Recovery Tools修复ORA-00742、ORA-600 ktbair2: illegal inheritance故障

Posted on 2025 年 04 月 24 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle Recovery Tools修复ORA-00742、ORA-600 ktbair2: illegal inheritance故障

接到客户反馈,一套运行在虚拟化平台中的Oracle数据库,由于机房断电,导致数据库无法启动,最初启动报错

2025-04-22T16:59:48.922227+08:00
Completed: alter database mount exclusive
alter database open
2025-04-22T16:59:52.609726+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
2025-04-22T16:59:52.937852+08:00
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
 Thread 1: Recovery starting at checkpoint rba (logseq 11031 block 139863), scn 0
2025-04-22T16:59:53.094081+08:00
Started redo scan
2025-04-22T16:59:53.203464+08:00
Completed redo scan
 read 15533 KB redo, 1140 data blocks need recovery
2025-04-22T16:59:53.515972+08:00
Hex dump of (file 3,block 82642) in trace file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p001_6612.trc

Corrupt block relative dba: 0x00c142d2 (file 3,block 82642)
Fractured block found during crash/instance recovery
Data in bad block:
 type: 6 format: 2 rdba: 0x00c142d2
 last change scn: 0x0000.0001.7a0219e2 seq: 0x1 flg: 0x06
 spare3: 0x0
 consistency value in tail: 0x0fcb0601
 check value in block header: 0x82eb
 computed block checksum: 0x172a

2025-04-22T16:59:53.563004+08:00
Reading datafile 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF' for corrupt data at rdba: 0x00c142d2 (file 3,block 82642)
Reread (file 3,block 82642) found same corrupt data (no logical check)
2025-04-22T16:59:54.578467+08:00
Hex dump of (file 3,block 207498) in trace file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p000_6064.trc
2025-04-22T16:59:54.578467+08:00

Corrupt block relative dba: 0x00c32a8a (file 3,block 207498)
Fractured block found during crash/instance recovery
Data in bad block:
 type: 6 format: 2 rdba: 0x00c32a8a
 last change scn: 0x0000.0001.731ec74f seq: 0x2 flg: 0x06
 spare3: 0x0
 consistency value in tail: 0x10520601
 check value in block header: 0x61c3
 computed block checksum: 0xe27a

Reading datafile 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF' for corrupt data at rdba: 0x00c32a8a (file 3,block 207498)
Reread (file 3,block 207498) found same corrupt data (no logical check)
Hex dump of (file 4, block 3959) in trace file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p000_6064.trc

Corrupt block relative dba: 0x01000f77 (file 4, block 3959)
Fractured block found during crash/instance recovery
Data in bad block:
 type: 2 format: 2 rdba: 0x01000f77
 last change scn: 0x0000.0001.79fa3621 seq: 0x4 flg: 0x04
 spare3: 0x0
 consistency value in tail: 0xcf6c0201
 check value in block header: 0x99d2
 computed block checksum: 0x79e0

Reading datafile 'D:\APP\ADMINISTRATOR\ORADATA\HIS\UNDOTBS01.DBF' for corrupt data at rdba: 0x01000f77 (file 4,block 3959)
Reread (file 4, block 3959) found same corrupt data (no logical check)
2025-04-22T16:59:54.640991+08:00
Started redo application at
 Thread 1: logseq 11031, block 139863, offset 0
2025-04-22T16:59:54.672246+08:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 11031 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO03.LOG
2025-04-22T16:59:54.703494+08:00
Hex dump of (file 3,block 89738) in trace file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p006_5236.trc

Corrupt block relative dba: 0x00c15e8a (file 3,block 89738)
Fractured block found during crash/instance recovery
Data in bad block:
 type: 6 format: 2 rdba: 0x00c15e8a
 last change scn: 0x0000.0001.7a09108a seq: 0x2 flg: 0x06
 spare3: 0x0
 consistency value in tail: 0x278e0602
 check value in block header: 0xd721
 computed block checksum: 0x2705

Reading datafile 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF' for corrupt data at rdba: 0x00c15e8a (file 3,block 89738)
Reread (file 3,block 89738) found same corrupt data (no logical check)
2025-04-22T16:59:54.734756+08:00
Hex dump of (file 3,block 9504) in trace file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p005_3660.trc

Reading datafile 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF' for corrupt data at rdba: 0x00c02520 (file 3,block 9504)
Reread (file 3,block 9504) found same corrupt data (logically corrupt)
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2025-04-22T16:59:54.750364+08:00
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xC] [PC:0x7FF64221F2F8, kdxlin()+4824]
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p008_3964.trc  (incident=521460):
ORA-07445: 出现异常错误: 核心转储 [kdxlin()+4824][ACCESS_VIOLATION][ADDR:0xC][PC:0x7FF64221F2F8][UNABLE_TO_READ][]
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_521460\his_p008_3964_i521460.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-04-22T16:59:55.156605+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p005_3660.trc:
ORA-00742: 日志读取在线程 1 序列 11029 块 51218 中检测到写入丢失情况
ORA-00312: 联机日志 1 线程 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG'
2025-04-22T16:59:55.828516+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p00a_3428.trc  (incident=521476):
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_521476\his_p00a_3428_i521476.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-04-22T16:59:56.844113+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p006_5236.trc  (incident=521444):
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_521444\his_p006_5236_i521444.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-04-22T16:59:59.062944+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p005_3660.trc:
ORA-00742: 日志读取在线程 1 序列 11029 块 51218 中检测到写入丢失情况
ORA-00334: 归档日志: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG'
2025-04-22T16:59:59.156645+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p00a_3428.trc:
ORA-00742: 日志读取在线程 1 序列 11029 块 51218 中检测到写入丢失情况
ORA-00312: 联机日志 1 线程 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG'
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_521477\his_p00a_3428_i521477.trc
2025-04-22T17:00:04.250414+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p006_5236.trc:
ORA-00742: 日志读取在线程 1 序列 11029 块 51218 中检测到写入丢失情况
ORA-00334: 归档日志: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG'
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
2025-04-22T17:00:04.344200+08:00
Slave encountered ORA-10388 exception during crash recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p006_5236.trc  (incident=521445):
ORA-01578: ORACLE 数据块损坏 (文件号 3, 块号 299732)
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10564: tablespace SYSAUX
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 11351
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_521445\his_p006_5236_i521445.trc
2025-04-22T17:00:05.312967+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p00a_3428.trc:
ORA-01578: ORACLE 数据块损坏 (文件号 3, 块号 216445)
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10564: tablespace SYSAUX
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 11171
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
2025-04-22T17:00:05.359805+08:00
Slave encountered ORA-10388 exception during crash recovery
2025-04-22T17:00:06.172286+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p006_5236.trc:
ORA-01578: ORACLE 数据块损坏 (文件号 3, 块号 299732)
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10564: tablespace SYSAUX
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 11351
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
2025-04-22T17:00:06.172286+08:00
Slave encountered ORA-1578 exception during crash recovery
Slave exiting with ORA-1578 exception
2025-04-22T17:00:06.172286+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p006_5236.trc:
ORA-01578: ORACLE 数据块损坏 (文件号 3, 块号 299732)
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10564: tablespace SYSAUX
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 11351
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
ORA-1578 signalled during: alter database open...

这里面主要报错有几大类:
1.在需要实例恢复的block中有坏块,而且主要集中在3号文件(sysaux)上
2. ORA-00742: 日志读取在线程 1 序列 11029 块 51218 中检测到写入丢失情况
3. ORA-600 ktbair2: illegal inheritance错误
对于这些问题的本质都是数据库无法正常实例恢复,尝试按照数据文件级别进行恢复操作,结果只有datafile 3 不成功

2025-04-22T20:11:36.724880+08:00
ALTER DATABASE RECOVER  datafile 3  
2025-04-22T20:11:36.740523+08:00
Media Recovery Start
2025-04-22T20:11:36.771771+08:00
Serial Media Recovery started
2025-04-22T20:11:36.896781+08:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 11031 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO03.LOG
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xC] [PC:0x7FF64221F2F8, kdxlin()+4824]
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_2512.trc  (incident=721216):
ORA-07445: ??????: ???? [kdxlin()+4824] [ACCESS_VIOLATION] [ADDR:0xC] [PC:0x7FF64221F2F8] [UNABLE_TO_READ] []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_721216\his_ora_2512_i721216.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-04-22T20:12:56.585813+08:00
ALTER DATABASE RECOVER  datafile 3  
2025-04-22T20:12:56.601434+08:00
Media Recovery Start
2025-04-22T20:12:56.601434+08:00
Serial Media Recovery started
2025-04-22T20:12:57.351435+08:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 11031 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO03.LOG
2025-04-22T20:12:58.492071+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_5332.trc  (incident=721217):
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_721217\his_ora_5332_i721217.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-04-22T20:13:00.710878+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2025-04-22T20:13:01.398399+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_5332.trc:
ORA-00742: 日志读取在线程 1 序列 11029 块 51218 中检测到写入丢失情况
ORA-00312: 联机日志 1 线程 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG'
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
2025-04-22T20:13:03.820304+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_2200.trc:
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
2025-04-22T20:13:06.070340+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_5332.trc:
ORA-00742: 日志读取在线程 1 序列 11029 块 51218 中检测到写入丢失情况
ORA-00334: 归档日志: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG'
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
2025-04-22T20:13:06.085990+08:00
Media Recovery failed with error 10562
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 3  ...

对于这样的故障,前段时间通过Oracle Recovery Tools工具处理过类似异常:Oracle Recovery Tools修复ORA-600 6101/kdxlin:psno out of range故障

然后尝试recover datafile 3,并打开数据库成功

至此数据库open成功,由于该库有不少坏块(包括system的一些字典),后续使用逻辑方式迁移数据到新库中(其中由于C_OBJ#簇中有坏块,导致迁移过程中大量index丢失,通过一些方法使用obj$,inde$等基表人工生成index.sql对于丢失的index进行弥补),最终客户业务正常运行

11.2.0.4库中遇到ORA-600 kcratr_nab_less_than_odr报错

Posted on 2025 年 04 月 22 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：11.2.0.4库中遇到ORA-600 kcratr_nab_less_than_odr报错

今天有一个客户11.2.0.4的库由于断电报ORA-600 kcratr_nab_less_than_odr的故障

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on 星期二 4月 22 10:56:36 2025

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE    11.2.0.4.0      Production
TNS for 64-bit Windows: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 - Production

SQL> startup mount
ORACLE 例程已经启动。

Total System Global Area 1.9174E+10 bytes
Fixed Size                  2289320 bytes
Variable Size            3355443544 bytes
Database Buffers         1.5771E+10 bytes
Redo Buffers               45932544 bytes
数据库装载完毕。
SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [4653],
[50575], [50922], [], [], [], [], [], [], []

alert日志报错信息

Tue Apr 22 05:08:12 2025
ALTER DATABASE   MOUNT
Successful mount of redo thread 1, with mount id 3880792636
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Tue Apr 22 05:08:17 2025
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 2451 KB redo, 1442 data blocks need recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\ydhl\ydhl\trace\ydhl_ora_2164.trc  (incident=240429):
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [4653], [50575], [50922],[],[],[],[],[],[],[]
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\ydhl\ydhl\incident\incdir_240429\ydhl_ora_2164_i240429.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Aborting crash recovery due to error 600
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\ydhl\ydhl\trace\ydhl_ora_2164.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [4653], [50575], [50922],[],[],[],[],[],[],[]
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\ydhl\ydhl\trace\ydhl_ora_2164.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [4653], [50575], [50922],[],[],[],[],[],[],[]
ORA-600 signalled during: ALTER DATABASE OPEN...

这个故障在一般在11.2.0.1版本中最为常见,处理起来比较简单,一般数据也会0丢失
12c启动报kcratr_nab_less_than_odr
又一例ORA-600 kcratr_nab_less_than_odr
ORA-600 kcratr_nab_less_than_odr故障解决
 差点被误操作的ORA-600 kcratr_nab_less_than_odr故障
这里处理比较简单,重建控制文件就完成了恢复工作

Oracle Recovery Tools修复ORA-600 6101/kdxlin:psno out of range故障

Posted on 2025 年 04 月 14 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle Recovery Tools修复ORA-600 6101/kdxlin:psno out of range故障

数据库异常断电,然后启动异常,我接手该库,尝试recover恢复

SQL> recover database;
ORA-10562: Error occurred while applying redo to data block (file# 2, block#
63710)
ORA-10564: tablespace SYSAUX
ORA-01110: ???????? 2: 'H:\TEMP\GDLISNET\SYSAUX01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 179030
ORA-00600: ????????????, ????: [6101], [0], [700], [32], [], [], [], [], [],
[], [], []

SQL> recover datafile 1;
完成介质恢复。
SQL> recover datafile 3;
完成介质恢复。
SQL> recover datafile 4;
完成介质恢复。
SQL> recover datafile 5;
完成介质恢复。
SQL> recover datafile 6;
完成介质恢复。
SQL> recover datafile 2;
ORA-00283: ??????????
ORA-10562: Error occurred while applying redo to data block (file# 2, block#
63744)
ORA-10564: tablespace SYSAUX
ORA-01110: ???? 2: 'H:\TEMP\GDLISNET\SYSAUX01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 67482
ORA-00600: ??????, ??: [kdxlin:psno out of range], [], [], [], [], [], [], [],
[], [], [], []

alert日志信息

Mon Apr 14 12:31:55 2025
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 20 slaves
Mon Apr 14 12:31:56 2025
Recovery of Online Redo Log: Thread 1 Group 2 Seq 6845 Reading mem 0
  Mem# 0: H:\TEMP\GDLISNET\REDO02.LOG
Mon Apr 14 12:31:56 2025
Errors in file c:\app\xff\diag\rdbms\gdlisnet\gdlisnet\trace\gdlisnet_pr0e_9840.trc  (incident=1489):
ORA-00600: 内部错误代码, 参数: [6101], [0], [700], [32], [], [], [], [], [], [], [], []
Incident details in: c:\app\xff\diag\rdbms\gdlisnet\gdlisnet\incident\incdir_1489\gdlisnet_pr0e_9840_i1489.trc
Slave exiting with ORA-10562 exception
Errors in file c:\app\xff\diag\rdbms\gdlisnet\gdlisnet\trace\gdlisnet_pr0e_9840.trc:
ORA-10562: Error occurred while applying redo to data block (file# 2, block# 63710)
ORA-10564: tablespace SYSAUX
ORA-01110: 数据文件 2: 'H:\TEMP\GDLISNET\SYSAUX01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 179030
ORA-00600: 内部错误代码, 参数: [6101], [0], [700], [32], [], [], [], [], [], [], [], []
Mon Apr 14 12:31:57 2025
Checker run found 6 new persistent data failures
Recovery Slave PR0E previously exited with exception 10562
Media Recovery failed with error 448
Errors in file c:\app\xff\diag\rdbms\gdlisnet\gdlisnet\trace\gdlisnet_pr00_14232.trc:
ORA-00283: 恢复会话因错误而取消
ORA-00448: 后台进程正常结束
Slave exiting with ORA-283 exception
Errors in file c:\app\xff\diag\rdbms\gdlisnet\gdlisnet\trace\gdlisnet_pr00_14232.trc:
ORA-00283: 恢复会话因错误而取消
ORA-00448: 后台进程正常结束
ORA-10562 signalled during: ALTER DATABASE RECOVER  database  ...

ALTER DATABASE RECOVER  datafile 2  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 2 Seq 6845 Reading mem 0
  Mem# 0: H:\TEMP\GDLISNET\REDO02.LOG
Errors in file c:\app\xff\diag\rdbms\gdlisnet\gdlisnet\trace\gdlisnet_ora_22400.trc  (incident=1369):
ORA-00600: ??????, ??: [kdxlin:psno out of range], [], [], [], [], [], [], [], [], [], [], []
Incident details in: c:\app\xff\diag\rdbms\gdlisnet\gdlisnet\incident\incdir_1369\gdlisnet_ora_22400_i1369.trc
Media Recovery failed with error 10562
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 2  ...

发现2号数据文件异常,报ORA-600 6101和ORA-600 kdxlin:psno out of range错误,出现该错误的原因主要是由于数据文件的bock和redo中信息不匹配导致,对于这种情况,我这边选择使用Oracle Recovery Tools工具进行快速恢复

再次recover并且直接打开库

SQL> set numw 16
SQL> col CHECKPOINT_TIME for a40
SQL> set lines 150
SQL> set pages 1000
SQL> SELECT status,
  2  to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,FUZZY,checkpoint_change#,
  3  count(*) ROW_NUM
  4  FROM v$datafile_header
  5  GROUP BY status, checkpoint_change#, to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss'),fuzzy
  6  ORDER BY status, checkpoint_change#, checkpoint_time;

STATUS  CHECKPOINT_TIME                          FUZ CHECKPOINT_CHANGE#          ROW_NUM
------- ---------------------------------------- --- ------------------ ----------------
ONLINE  2025-03-28 22:00:09                      NO           118575875                6

SQL>
SQL>
SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-01113: 文件 2 需要介质恢复
ORA-01110: 数据文件 2: 'H:\TEMP\GDLISNET\SYSAUX01.DBF'


SQL> recover datafile 2;
完成介质恢复。
SQL> alter database open;

数据库已更改。

SQL>

逻辑方法导出该库,完成该库的恢复工作