ORA-600 kddummy_blkchk 数据库循环重启

Posted on 2025 年 05 月 08 日 by 惜分飞

一个运行在hp平台的10g rac突然异常,之后运行一段时间就自动重启,客户让对其进行分析和解决

Thu May  8 06:23:21 2025
ALTER DATABASE OPEN
Picked broadcast on commit scheme to generate SCNs
Thu May  8 06:23:21 2025
Thread 1 opened at log sequence 74302
  Current log# 1 seq# 74302 mem# 0: /dev/vgdata/rrac_redo01
Successful open of redo thread 1
Thu May  8 06:23:21 2025
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu May  8 06:23:21 2025
SMON: enabling cache recovery
Thu May  8 06:23:22 2025
Successfully onlined Undo Tablespace 1.
Thu May  8 06:23:22 2025
SMON: enabling tx recovery
Thu May  8 06:23:22 2025
Database Characterset is ZHS16CGB231280
Opening with internal Resource Manager plan
where NUMA PG = 1, CPUs = 4
replication_dependency_tracking turned off (no async multimaster replication found)
Thu May  8 06:23:23 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
ORA-00600: internal error code, arguments: [kddummy_blkchk], [118], [333578], [18019], [], [], [], []
Starting background process QMNC
QMNC started with pid=22, OS id=15792
Thu May  8 06:23:25 2025
ORACLE Instance orcl1 (pid = 13) - Error 607 encountered while recovering transaction (9, 33) on object 775794.
Thu May  8 06:23:25 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kddummy_blkchk], [118], [333578], [18019], [], [], [], []
Thu May  8 06:23:26 2025
Completed: ALTER DATABASE OPEN
Thu May  8 06:23:26 2025
Doing block recovery for file 118 block 333578
Block recovery from logseq 74302, block 22 to scn 46740761996
Thu May  8 06:23:26 2025
Recovery of Online Redo Log: Thread 1 Group 1 Seq 74302 Reading mem 0
  Mem# 0: /dev/vgdata/rrac_redo01
Block recovery stopped at EOT rba 74302.33.16
Block recovery completed at rba 74302.33.16, scn 10.3791089036
Thu May  8 06:23:33 2025
Trace dumping is performing id=[cdmp_20250508062324]
Thu May  8 06:25:55 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
ORA-00600: internal error code, arguments: [kddummy_blkchk], [118], [333578], [18019], [], [], [], []
Thu May  8 06:25:58 2025
ORACLE Instance orcl1 (pid = 13) - Error 607 encountered while recovering transaction (9, 33) on object 775794.
Thu May  8 06:27:32 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
ORA-00600: internal error code, arguments: [kddummy_blkchk], [118], [333578], [18019], [], [], [], []
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
Doing block recovery for file 118 block 333578
Block recovery from logseq 74302, block 372 to scn 46740952565
Thu May  8 06:27:41 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
ORA-00600: internal error code, arguments: [kddummy_blkchk], [118], [333578], [18019], [], [], [], []
ORACLE Instance orcl1 (pid = 13) - Error 607 encountered while recovering transaction (9, 33) on object 775794.
Thu May  8 06:27:43 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kddummy_blkchk], [118], [333578], [18019], [], [], [], []
Doing block recovery for file 118 block 333578
Block recovery from logseq 74302, block 372 to scn 46740952565
Thu May  8 06:27:45 2025
Recovery of Online Redo Log: Thread 1 Group 1 Seq 74302 Reading mem 0
  Mem# 0: /dev/vgdata/rrac_redo01
Block recovery completed at rba 74302.394.16, scn 10.3791279606
Thu May  8 06:27:47 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_smon_15721.trc:
ORA-00600: internal error code, arguments: [kddummy_blkchk], [118], [333578], [18019], [], [], [], []
Thu May  8 06:28:07 2025
Errors in file /users/oracle/admin/orcl/bdump/orcl1_pmon_15690.trc:
ORA-00474: SMON process terminated with error
Thu May  8 06:28:07 2025
PMON: terminating instance due to error 474

这个数据库重启是由于smon进程异常导致数据库关闭,然后rac自动拉起数据库,从而出现了循环重启启动,smon异常的主要原因是由于“ORACLE Instance orcl1 (pid = 13) – Error 607 encountered while recovering transaction (9, 33) on object 775794.”这个表示有实物要回滚,但是遇到了ORA-600 kddummy_blkchk异常,无法完成回滚从而使得smon进程异常进而使得数据实例crash.对于这个问题处理相对比较简单
1. 通过ORA-600 kddummy_blkchk找出来报错对象

ORA-00600: internal error code, arguments: [kddummy_blkchk], [a], [b], 1

ARGUMENTS:
Arg [a] Absolute file number
Arg [b] Block number
Arg 1 Internal error code returned from kcbchk() which indicates the problem encountered.

2. 屏蔽实物回滚,打开数据库,然后对通过dba_extents 查询出来的对象/或者根据事务报错的object信息查询出来对象进行重建,完成本次恢复任务

关于ORA-00600: internal error code, arguments: [kddummy_blkchk], [a], [b], 相关目前已知bug

Bug	Fixed	Description
12349316	11.2.0.4, 12.1.0.2, 12.2.0.1	DBMS_SPACE_ADMIN.TABLESPACE_FIX_BITMAPS fails with ORA-600 [kddummy_blkchk] / ORA-600 [kdBlkCheckError] / ORA-607
17325413	11.2.0.3.BP23, 11.2.0.4.2, 11.2.0.4.BP04, 12.1.0.1.3, 12.1.0.2, 12.2.0.1	Drop column with DEFAULT value and NOT NULL definition ends up with Dropped Column Data still on Disk leading to Corruption
13715932	11.2.0.4, 12.1.0.1	ORA-600 [kddummy_blkchk] [18038] while adding extents to a large datafile
12417369	11.2.0.2.5, 11.2.0.2.BP13, 11.2.0.2.GIPSU05, 11.2.0.3, 12.1.0.1	Block corruption from rollback on compressed table
10324526	10.2.0.5.4, 11.1.0.7.8, 11.2.0.2.3, 11.2.0.2.BP06, 11.2.0.3, 12.1.0.1	ORA-600 [kddummy_blkchk] [6106] / corruption on COMPRESS table in TTS
10113224	11.2.0.3, 12.1.0.1	Index coalesce may generate invalid redo if blocks in the buffer cache are invalid/corrupted
9726702	11.2.0.3, 12.1.0.1	DBMS_SPACE_ADMIN.assm_segment_verify reports HWM related inconsistencies
9724970	11.2.0.1.BP08, 11.2.0.2.2, 11.2.0.2.BP02, 11.2.0.3, 12.1.0.1	Block Corruption with PDML UPDATE. ORA_600 [4511] OERI[kdblkcheckerror] by block check
9711859	10.2.0.5.1, 11.1.0.7.6, 11.2.0.2, 12.1.0.1	ORA-600 [ktsptrn_fix-extmap] / ORA-600 [kdblkcheckerror] during extent allocation caused by bug 8198906
9581240	11.1.0.7.9, 11.2.0.2, 12.1.0.1	Corruption / ORA-600 [kddummy_blkchk] [6101] / ORA-600 [7999] after RENAME operation during ROLLBACK
9350204	11.2.0.3, 12.1.0.1	Spurious ORA-600 [kddummy_blkchk] .. [6145] during CR operations on tables with ROWDEPENDENCIES
9231605	11.1.0.7.4, 11.2.0.1.3, 11.2.0.1.BP02, 11.2.0.2, 12.1.0.1	Block corruption with missing row on a compressed table after DELETE
9119771	11.2.0.2, 12.1.0.1	OERI [kddummy_blkchk]…[6108] from ‘SHRINK SPACE CASCADE’
9019113	11.2.0.1.BP02, 11.2.0.2, 12.1.0.1	ORA-600 [17182] ORA-7445 [memcpy] ORA-600 [kdBlkCheckError] for OLTP COMPRESS table in OLTP Compression REDO during RECOVERY
8951812	11.2.0.2, 12.1.0.1	Corrupt index by rebuild online. Possible OERI [kddummy_blkchk] by SMON
8720802	10.2.0.5, 11.2.0.1.BP07, 11.2.0.2, 12.1.0.1	Add check for row piece pointing to itself (db_block_checking,dbv,rman,analyze)
8331063	11.2.0.3, 12.1.0.1	Corrupt Undo. ORA-600 [2015] in Undo Block During Rollback
6523037	11.2.0.1.BP07, 11.2.0.2.2, 11.2.0.2.BP01, 11.2.0.3, 12.1.0.1	Corruption / ORA-600 [kddummy_blkchk] [6110] on update
8277580	11.1.0.7.2, 11.2.0.1, 11.2.0.2, 12.1.0.1	Corruption on compressed tables during Recovery and Quick Multi Delete (QMD).
9964102	11.2.0.1	OERI:2015 / OERI:kddummy_blkchk / undo corruption from supplementat logging with compressed tables
8613137	11.1.0.7.2, 11.2.0.1	ORA-600 updating table with DEFERRED constraints
8360192	11.1.0.7.6, 11.2.0.1	ORA-600 [kdBlkCheckError] [6110] / corruption from insert
8239658	10.2.0.5, 11.2.0.1	Dump / corruption writing row to compressed table
8198906	10.2.0.5, 11.2.0.1	OERI [kddummy_blkchk] / OERI [5467] for an aborted transaction of allocating extents
7715244	11.1.0.7.2, 11.2.0.1	Corruption on compressed tables. Error codes 6103 / 6110
7662491	10.2.0.4.2, 10.2.0.5, 11.1.0.7.4, 11.2.0.1	Array Update can corrupt a row. Errors OERI[kghstack_free1] or OERI[kddummy_blkchk][6110]
7411865	10.2.0.4.2, 10.2.0.5, 11.1.0.7.1, 11.2.0.1	OERI:13030 / ORA-1407 / block corruption from UPDATE .. RETURNING DML with trigger
7331181	11.2.0.1	ORA-1555 or OERI [kddummy_blkchk] [file#] [block#] [6126] during CR Rollback in query
7293156	11.1.0.7, 11.2.0.1	ORA-600 [2023] by Parallel Transaction Rollback when applying Multi-block undo Head-piece / Tail-piece
7041254	11.1.0.7.5, 11.2.0.1	ORA-19661 during RMAN restore check logical of compressed backup / IOT dummy key
6760697	10.2.0.4.3, 10.2.0.5, 11.1.0.7, 11.2.0.1	DBMS_SPACE_ADMIN.ASSM_SEGMENT_VERIFY does not detect certain segment header block corruption
6647480	10.2.0.4.4, 10.2.0.5, 11.1.0.7.3, 11.2.0.1	Corruption / OERI [kddummy_blkchk] .. [18021] with ASSM
6134368	10.2.0.5, 11.2.0.1	ORA-1407 / block corruption from UPDATE .. RETURNING DML with trigger – SUPERCEDED
6057203	10.2.0.4, 11.1.0.7, 11.2.0.1	Corruption with zero length column (ZLC) / OERI [kcbchg1_6] from Parallel update
6653934	10.2.0.4.2, 10.2.0.5, 11.1.0.7	Dump / block corruption from ONLINE segment shrink with ROWDEPENDENCIES
6674196	10.2.0.4, 10.2.0.5, 11.1.0.6	OERI / buffer cache corruption using ASM, OCFS or any ksfd client like ODM
5599596	10.2.0.4, 11.1.0.6	Block corruption / OERI [kddummy_blkchk] on clustered or compressed tables
5496041	10.2.0.4, 11.1.0.6	OERI[6006] / index corruption on compressed index
5386204	10.2.0.4.1, 10.2.0.5, 11.1.0.6	Block corruption / OERI[kddummy_blkchk] after direct load of ASSM segment
5363584	10.2.0.4, 11.1.0.6	Array insert into table can corrupt redo
4602031	10.2.0.2, 11.1.0.6	Block corruption from UPDATE or MERGE into compressed table
4493447	11.1.0.6	Spurious ORA-600 [kddummy_blkchk] [file#] [block#] [6145] on rollback of array update
4329302	11.1.0.6	OERI [kddummy_blkchk] [file#] [block#] [6145] on rollback of update with logminer
6075487	10.2.0.4	OERI[kddummy_blkchk]..[18020/18026] for DDL on plugged ASSM tablespace with FLASHBACK
4054640	10.1.0.5, 10.2.0.1	Block corruption / OERI [kddummy_blkchk] at physical standby
4000840	10.1.0.4, 10.2.0.1, 9.2.0.7	Update of a row with more than 255 columns can cause block corruption
3772033	9.2.0.7, 10.1.0.4, 10.2.0.1	OERI[ktspfmb_create1] creating a LOB in ASSM using 2k blocksize

记录一次asm disk加入到vg通过恢复直接open库的案例

Posted on 2025 年 05 月 05 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：记录一次asm disk加入到vg通过恢复直接open库的案例

客户在不清楚磁盘被asm disk使用的情况下,直接分区做pv,加入到vg中并且分配给了lv,导致数据库异常

通过操作系统层面分析,确认客户把data磁盘组的一个磁盘给处理掉了,导致数据库报错

WARNING: ASMB force dismounting group 2 (DATA) due to failover
SUCCESS: diskgroup DATA was dismounted
2025-05-04T07:03:19.910082+08:00
KCF: read, write or open error, block=0x201544 online=1
        file=102 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_61.dbf'
        error=15078 txt: ''
2025-05-04T07:03:19.918972+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbwc_18507.trc:
2025-05-04T07:03:19.952045+08:00
KCF: read, write or open error, block=0x2013e7 online=1
        file=102 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_61.dbf'
        error=15078 txt: ''
2025-05-04T07:03:19.964538+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbw7_18486.trc:
2025-05-04T07:03:19.967133+08:00
KCF: read, write or open error, block=0x230e71 online=1
        file=105 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_64.dbf'
        error=15078 txt: ''
2025-05-04T07:03:19.973289+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbw2_18466.trc:
2025-05-04T07:03:19.978514+08:00
KCF: read, write or open error, block=0x1f6e91 online=1
        file=86 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_52.dbf'
        error=15078 txt: ''
2025-05-04T07:03:19.991060+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbwd_18511.trc:
2025-05-04T07:03:19.995762+08:00
KCF: read, write or open error, block=0x7f8 online=1
        file=15 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/undotbs01.dbf'
        error=15078 txt: ''
2025-05-04T07:03:20.006862+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbwa_18498.trc:
2025-05-04T07:03:20.020739+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_imr0_18937.trc:

这个客户比较幸运,处理该磁盘之后,没有往对应的lv中写入太多数据,导致覆盖部分很少

[root@rac01 rules.d]# df -h
文件系统               容量  已用  可用 已用% 挂载点
/dev/mapper/nlas-root  800G  272G  528G   34% /
devtmpfs               284G     0  284G    0% /dev
tmpfs                  284G  637M  283G    1% /dev/shm
tmpfs                  284G  4.0G  280G    2% /run
tmpfs                  284G     0  284G    0% /sys/fs/cgroup
/dev/mapper/nlas-home  200G   64M  200G    1% /home
/dev/sda1              197M  158M   40M   80% /boot
tmpfs                   57G   40K   57G    1% /run/user/0
tmpfs                   57G   48K   57G    1% /run/user/1000
[root@rac01 rules.d]# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda2  nlas lvm2 a--  564.00g    0 
  /dev/sdb1  nlas lvm2 a--   <2.00t 1.51t
[root@rac01 rules.d]# vgs
  VG   #PV #LV #SN Attr   VSize VFree
  nlas   2   3   0 wz--n- 2.55t 1.51t
[root@rac01 rules.d]# lvs
  LV   VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home nlas -wi-ao---- 200.00g                                                    
  root nlas -wi-ao---- 800.00g                                                    
  swap nlas -wi-ao----  64.00g

通过底层对磁盘进行分析,发现备份的磁盘头均以损坏,通过深入分析确认f1b1在sdb磁盘的第10个au上,通过相关信息,使用dul工具加载磁盘组,并分析元数据信息,发现恢复数据需要的元数据都可以正常加载

直接使用dul抽取数据到文件系统,然后open数据库成功

然后通过rman 检测坏块(3T多的库只有不到5000个坏块,相对来说效果非常好),对于坏块对象进行处理,完美完成本次恢复工作.对于这次能够有这样好的恢复效果有几个因素:
1)asm disk 加入到vg,并分配给lv之后,立刻停止写入操作,避免了因为写入数据而覆盖asm 磁盘的带来的风险
2)由于是19c库,默认au为4M,使得数据库文件数据相对比较靠后,覆盖几率小了一点
3)由于文件系统是xfs,相对覆盖比ext4会少很多
4)是云环境的ssd磁盘,没有触发trim功能
以前类似asm disk异常恢复的相关case汇总:
asm磁盘加入vg恢复
asm磁盘dd破坏恢复
asm磁盘分区丢失恢复
pvid=yes导致asm无法mount
win asm disk header 异常恢复
又一例asm disk 加入vg故障
pvcreate asm disk导致asm磁盘组异常恢复
asm disk被加入到另外一个磁盘组故障恢复
再一例asm disk被误加入vg并且扩容lv恢复
再一起asm disk被格式化成ext3文件系统故障恢复
一次完美的asm disk被格式化ntfs恢复
asm disk误设置pvid导致asm diskgroup无法mount恢复
asm disk被分区,格式化为ext4恢复
oracle asm disk格式化恢复—格式化为ext4文件系统
分享oracleasm createdisk重新创建asm disk后数据0丢失恢复案例

Oracle Recovery Tools修复ORA-00742、ORA-600 ktbair2: illegal inheritance故障

Posted on 2025 年 04 月 24 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle Recovery Tools修复ORA-00742、ORA-600 ktbair2: illegal inheritance故障

接到客户反馈,一套运行在虚拟化平台中的Oracle数据库,由于机房断电,导致数据库无法启动,最初启动报错

2025-04-22T16:59:48.922227+08:00
Completed: alter database mount exclusive
alter database open
2025-04-22T16:59:52.609726+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
2025-04-22T16:59:52.937852+08:00
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
 Thread 1: Recovery starting at checkpoint rba (logseq 11031 block 139863), scn 0
2025-04-22T16:59:53.094081+08:00
Started redo scan
2025-04-22T16:59:53.203464+08:00
Completed redo scan
 read 15533 KB redo, 1140 data blocks need recovery
2025-04-22T16:59:53.515972+08:00
Hex dump of (file 3,block 82642) in trace file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p001_6612.trc

Corrupt block relative dba: 0x00c142d2 (file 3,block 82642)
Fractured block found during crash/instance recovery
Data in bad block:
 type: 6 format: 2 rdba: 0x00c142d2
 last change scn: 0x0000.0001.7a0219e2 seq: 0x1 flg: 0x06
 spare3: 0x0
 consistency value in tail: 0x0fcb0601
 check value in block header: 0x82eb
 computed block checksum: 0x172a

2025-04-22T16:59:53.563004+08:00
Reading datafile 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF' for corrupt data at rdba: 0x00c142d2 (file 3,block 82642)
Reread (file 3,block 82642) found same corrupt data (no logical check)
2025-04-22T16:59:54.578467+08:00
Hex dump of (file 3,block 207498) in trace file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p000_6064.trc
2025-04-22T16:59:54.578467+08:00

Corrupt block relative dba: 0x00c32a8a (file 3,block 207498)
Fractured block found during crash/instance recovery
Data in bad block:
 type: 6 format: 2 rdba: 0x00c32a8a
 last change scn: 0x0000.0001.731ec74f seq: 0x2 flg: 0x06
 spare3: 0x0
 consistency value in tail: 0x10520601
 check value in block header: 0x61c3
 computed block checksum: 0xe27a

Reading datafile 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF' for corrupt data at rdba: 0x00c32a8a (file 3,block 207498)
Reread (file 3,block 207498) found same corrupt data (no logical check)
Hex dump of (file 4, block 3959) in trace file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p000_6064.trc

Corrupt block relative dba: 0x01000f77 (file 4, block 3959)
Fractured block found during crash/instance recovery
Data in bad block:
 type: 2 format: 2 rdba: 0x01000f77
 last change scn: 0x0000.0001.79fa3621 seq: 0x4 flg: 0x04
 spare3: 0x0
 consistency value in tail: 0xcf6c0201
 check value in block header: 0x99d2
 computed block checksum: 0x79e0

Reading datafile 'D:\APP\ADMINISTRATOR\ORADATA\HIS\UNDOTBS01.DBF' for corrupt data at rdba: 0x01000f77 (file 4,block 3959)
Reread (file 4, block 3959) found same corrupt data (no logical check)
2025-04-22T16:59:54.640991+08:00
Started redo application at
 Thread 1: logseq 11031, block 139863, offset 0
2025-04-22T16:59:54.672246+08:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 11031 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO03.LOG
2025-04-22T16:59:54.703494+08:00
Hex dump of (file 3,block 89738) in trace file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p006_5236.trc

Corrupt block relative dba: 0x00c15e8a (file 3,block 89738)
Fractured block found during crash/instance recovery
Data in bad block:
 type: 6 format: 2 rdba: 0x00c15e8a
 last change scn: 0x0000.0001.7a09108a seq: 0x2 flg: 0x06
 spare3: 0x0
 consistency value in tail: 0x278e0602
 check value in block header: 0xd721
 computed block checksum: 0x2705

Reading datafile 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF' for corrupt data at rdba: 0x00c15e8a (file 3,block 89738)
Reread (file 3,block 89738) found same corrupt data (no logical check)
2025-04-22T16:59:54.734756+08:00
Hex dump of (file 3,block 9504) in trace file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p005_3660.trc

Reading datafile 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF' for corrupt data at rdba: 0x00c02520 (file 3,block 9504)
Reread (file 3,block 9504) found same corrupt data (logically corrupt)
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2025-04-22T16:59:54.750364+08:00
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xC] [PC:0x7FF64221F2F8, kdxlin()+4824]
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p008_3964.trc  (incident=521460):
ORA-07445: 出现异常错误: 核心转储 [kdxlin()+4824][ACCESS_VIOLATION][ADDR:0xC][PC:0x7FF64221F2F8][UNABLE_TO_READ][]
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_521460\his_p008_3964_i521460.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-04-22T16:59:55.156605+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p005_3660.trc:
ORA-00742: 日志读取在线程 1 序列 11029 块 51218 中检测到写入丢失情况
ORA-00312: 联机日志 1 线程 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG'
2025-04-22T16:59:55.828516+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p00a_3428.trc  (incident=521476):
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_521476\his_p00a_3428_i521476.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-04-22T16:59:56.844113+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p006_5236.trc  (incident=521444):
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_521444\his_p006_5236_i521444.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-04-22T16:59:59.062944+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p005_3660.trc:
ORA-00742: 日志读取在线程 1 序列 11029 块 51218 中检测到写入丢失情况
ORA-00334: 归档日志: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG'
2025-04-22T16:59:59.156645+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p00a_3428.trc:
ORA-00742: 日志读取在线程 1 序列 11029 块 51218 中检测到写入丢失情况
ORA-00312: 联机日志 1 线程 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG'
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_521477\his_p00a_3428_i521477.trc
2025-04-22T17:00:04.250414+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p006_5236.trc:
ORA-00742: 日志读取在线程 1 序列 11029 块 51218 中检测到写入丢失情况
ORA-00334: 归档日志: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG'
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
2025-04-22T17:00:04.344200+08:00
Slave encountered ORA-10388 exception during crash recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p006_5236.trc  (incident=521445):
ORA-01578: ORACLE 数据块损坏 (文件号 3, 块号 299732)
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10564: tablespace SYSAUX
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 11351
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_521445\his_p006_5236_i521445.trc
2025-04-22T17:00:05.312967+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p00a_3428.trc:
ORA-01578: ORACLE 数据块损坏 (文件号 3, 块号 216445)
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10564: tablespace SYSAUX
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 11171
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
2025-04-22T17:00:05.359805+08:00
Slave encountered ORA-10388 exception during crash recovery
2025-04-22T17:00:06.172286+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p006_5236.trc:
ORA-01578: ORACLE 数据块损坏 (文件号 3, 块号 299732)
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10564: tablespace SYSAUX
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 11351
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
2025-04-22T17:00:06.172286+08:00
Slave encountered ORA-1578 exception during crash recovery
Slave exiting with ORA-1578 exception
2025-04-22T17:00:06.172286+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_p006_5236.trc:
ORA-01578: ORACLE 数据块损坏 (文件号 3, 块号 299732)
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10564: tablespace SYSAUX
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 11351
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
ORA-1578 signalled during: alter database open...

这里面主要报错有几大类:
1.在需要实例恢复的block中有坏块,而且主要集中在3号文件(sysaux)上
2. ORA-00742: 日志读取在线程 1 序列 11029 块 51218 中检测到写入丢失情况
3. ORA-600 ktbair2: illegal inheritance错误
对于这些问题的本质都是数据库无法正常实例恢复,尝试按照数据文件级别进行恢复操作,结果只有datafile 3 不成功

2025-04-22T20:11:36.724880+08:00
ALTER DATABASE RECOVER  datafile 3  
2025-04-22T20:11:36.740523+08:00
Media Recovery Start
2025-04-22T20:11:36.771771+08:00
Serial Media Recovery started
2025-04-22T20:11:36.896781+08:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 11031 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO03.LOG
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xC] [PC:0x7FF64221F2F8, kdxlin()+4824]
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_2512.trc  (incident=721216):
ORA-07445: ??????: ???? [kdxlin()+4824] [ACCESS_VIOLATION] [ADDR:0xC] [PC:0x7FF64221F2F8] [UNABLE_TO_READ] []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_721216\his_ora_2512_i721216.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-04-22T20:12:56.585813+08:00
ALTER DATABASE RECOVER  datafile 3  
2025-04-22T20:12:56.601434+08:00
Media Recovery Start
2025-04-22T20:12:56.601434+08:00
Serial Media Recovery started
2025-04-22T20:12:57.351435+08:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 11031 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO03.LOG
2025-04-22T20:12:58.492071+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_5332.trc  (incident=721217):
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_721217\his_ora_5332_i721217.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-04-22T20:13:00.710878+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2025-04-22T20:13:01.398399+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_5332.trc:
ORA-00742: 日志读取在线程 1 序列 11029 块 51218 中检测到写入丢失情况
ORA-00312: 联机日志 1 线程 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG'
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
2025-04-22T20:13:03.820304+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_2200.trc:
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
2025-04-22T20:13:06.070340+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_5332.trc:
ORA-00742: 日志读取在线程 1 序列 11029 块 51218 中检测到写入丢失情况
ORA-00334: 归档日志: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG'
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
2025-04-22T20:13:06.085990+08:00
Media Recovery failed with error 10562
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 3  ...

对于这样的故障,前段时间通过Oracle Recovery Tools工具处理过类似异常:Oracle Recovery Tools修复ORA-600 6101/kdxlin:psno out of range故障

然后尝试recover datafile 3,并打开数据库成功

至此数据库open成功,由于该库有不少坏块(包括system的一些字典),后续使用逻辑方式迁移数据到新库中(其中由于C_OBJ#簇中有坏块,导致迁移过程中大量index丢失,通过一些方法使用obj$,inde$等基表人工生成index.sql对于丢失的index进行弥补),最终客户业务正常运行

11.2.0.4库中遇到ORA-600 kcratr_nab_less_than_odr报错

Posted on 2025 年 04 月 22 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：11.2.0.4库中遇到ORA-600 kcratr_nab_less_than_odr报错

今天有一个客户11.2.0.4的库由于断电报ORA-600 kcratr_nab_less_than_odr的故障

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on 星期二 4月 22 10:56:36 2025

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE    11.2.0.4.0      Production
TNS for 64-bit Windows: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 - Production

SQL> startup mount
ORACLE 例程已经启动。

Total System Global Area 1.9174E+10 bytes
Fixed Size                  2289320 bytes
Variable Size            3355443544 bytes
Database Buffers         1.5771E+10 bytes
Redo Buffers               45932544 bytes
数据库装载完毕。
SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [4653],
[50575], [50922], [], [], [], [], [], [], []

alert日志报错信息

Tue Apr 22 05:08:12 2025
ALTER DATABASE   MOUNT
Successful mount of redo thread 1, with mount id 3880792636
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Tue Apr 22 05:08:17 2025
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 2451 KB redo, 1442 data blocks need recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\ydhl\ydhl\trace\ydhl_ora_2164.trc  (incident=240429):
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [4653], [50575], [50922],[],[],[],[],[],[],[]
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\ydhl\ydhl\incident\incdir_240429\ydhl_ora_2164_i240429.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Aborting crash recovery due to error 600
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\ydhl\ydhl\trace\ydhl_ora_2164.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [4653], [50575], [50922],[],[],[],[],[],[],[]
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\ydhl\ydhl\trace\ydhl_ora_2164.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [4653], [50575], [50922],[],[],[],[],[],[],[]
ORA-600 signalled during: ALTER DATABASE OPEN...

这个故障在一般在11.2.0.1版本中最为常见,处理起来比较简单,一般数据也会0丢失
12c启动报kcratr_nab_less_than_odr
又一例ORA-600 kcratr_nab_less_than_odr
ORA-600 kcratr_nab_less_than_odr故障解决
 差点被误操作的ORA-600 kcratr_nab_less_than_odr故障
这里处理比较简单,重建控制文件就完成了恢复工作

Oracle 19c 202504补丁（RUs+OJVM）-19.27

Posted on 2025 年 04 月 18 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle 19c 202504补丁（RUs+OJVM）-19.27

19.0.0.0
Description	Database Update	GI Update	Windows Bundle Patch
APR2025 (19.27.0.0.0)	37642901	37641958	37532350
JAN2025 (19.26.0.0.0)	37260974	37257886	37486199
OCT2024 (19.25.0.0.0)	36912597	36916690	36878821
JUL2024 (19.24.0.0.0)	36582781	36582629	36521936
APR2024 (19.23.0.0.0)	36233263	36233126	36219938
JAN2024 (19.22.0.0.0)	35943157	35940989	35962832
OCT2023 (19.21.0.0.0)	35643107	35642822	35681552
JUL2023 (19.20.0.0.0)	35320081	35319490	35348034
APR2023 (19.19.0.0.0)	35042068	35037840	35046439
JAN2023 (19.18.0.0.0)	34765931	34762026	34750795
Oct2022 (19.17.0.0.0)	34419443	34416665	34468114
JUL2022 (19.16.0.0.0)	34133642	34130714	34110685
APR2022 (19.15.0.0.0)	33806152	33803476	33829175
JAN2022 (19.14.0.0.0)	33515361	33509923	33575656
OCT2021(19.13.0.0.0)	33192793	33182768	33155330
JUL2021 (19.12.0.0.0)	32904851	32895426	32832237
APR2021 (19.11.0.0.0)	32545013	32545008	32409154
JAN2021 (19.10.0.0.0)	32218454	32226239	32062765
OCT2020 (19.9.0.0.0)	31771877	31750108	31719903
JUL2020 (19.8.0.0.0)	31281355	31305339	31247621
APR2020 (19.7.0.0.0)	30869156	30899722	30901317
JAN2020 (19.6.0.0.0)	30557433	30501910	30445947
OCT2019 (19.5.0.0.0)	30125133	30116789	30151705
JUL2019 (19.4.0.0.0)	29834717	29708769	NA
APR2019 (19.3.0.0.0)	29517242	29517302	NA

19.0.0.0
Description	OJVM Update	OJVM + DB Update	OJVM + GI Update
APR2025 (19.27.0.0.250415)	37499406	37591483	37591516
JAN2025 (19.26.0.0.250121)	37102264	37262172	37262208
OCT2024 (19.25.0.0.241015)	36878697	36866623	36866740
JUL2024 (19.24.0.0.240716)	36414915	36522340	36522439
APR2024 (19.23.0.0.240416)	36199232	36209492	36209493
JAN2024 (19.22.0.0.240116)	35926646	36031426	36031453
OCT2023 (19.21.0.0.231017)	35648110	35742413	35742441
JUL2023 (19.20.0.0.230718)	35354406	35370174	35370167
APR2023 (19.19.0.0.230418)	35050341	35058163	35058172
JAN2023 (19.18.0.0.230117)	34786990	34773489	34773504
OCT2022 (19.17.0.0.221018)	34411846	34449114	34449117
JUL2022 (19.16.0.0.220719)	34086870	34160831	34160854
APR2022 (19.15.0.0.220419)	33808367	33859194	33859214
JAN2022 (19.14.0.0.220118)	33561310	33567270	33567274
OCT2021 (19.13.0.0.211019)	33192694	33248420	33248471
JUL2021 (19.12.0.0.210720)	32876380	32900021	32900083
APR2021 (19.11.0.0.210420)	32399816	32578972	32578973
JAN2021 (19.10.0.0.210119)	32067171	32126828	32126842
OCT2020 (19.9.0.0.201020)	31668882	31720396	31720429
JUL2020 (19.8.0.0.200714)	31219897	31326362	31326369
APR2020 (19.7.0.0.200414)	30805684	30783543	30783556
JAN2020 (19.6.0.0.200114)	30484981	30463595	30463609
OCT2019 (19.5.0.0.191015)	30128191	30133124	30133178
JUL2019 (19.4.0.0.190716)	29774421	29699079	29699097
APR2019 (19.3.0.0.190416)	29548437	29621253	29621299

参考:Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases (Doc ID 2118136.2)