Oracle数据块编辑工具( Oracle Block Editor Tool)-obet

Posted on 2025 年 11 月 08 日 by 惜分飞

标题：Oracle数据块编辑工具( Oracle Block Editor Tool)-obet

由于oracle后续版本对于bbed的支持不是太友好(从10g之后无法直接编译使用,win版本偏移量错误等),基于这些情况,结合当前ai的便利,自己动手写了一个基础版的obet(Oracle Block Editor Tool),用来实现在某些情况下Oracle 数据块的编辑工作.目前(2025年11月7日)发布第一版主要功能有:
1. 通过16进制查看数据文件任何偏移量位置的数据(d/dump)
2. 通过16进制编辑数据文件任何偏移量位置的数据(m/modify)
3. 对oracle 数据块的tailchk自动修复(tailchk apply)
4. 对oracle 数据块的checksum自动修复(sum apply)
5. 数据文件之间部分数据拷贝功能(coopy)

启动软件界面

[root@iZbp11c0qyuuo1gr7j98upZ tmp]# ./obet
=============================================
Welcome to Oracle Block Editor Tool (OBET)
=============================================

***** !!! For Oracle Internal Use only !!! *****

[Software Function Description]
- View and edit data block in hexadecimal format
- Automatically repair tailchk and checksum
- Copy data between different data files
- Mark data block as corrupted block

[Developer Information]
- Name: XiFenFei
- Phone: +86-17813235971
- Email: dba@xifenfei
- Q Q: 107644445
- WeChat: 17813235971
- Website: https://www.xifenfei.com

[Version Details]
- Software Version: v2025.11.001
- Build Date: 2025.11.07

=============================================
Type 'help' for command list | 'exit' to quit
=============================================

==================================================
Software License Status: Authorized
==================================================

使用说明

OBET> help
OBET (Oracle Block Editor Tool) commands:
  open <config_file>    - Load file list from config file (format: <num> <path>)
  info                  - Show loaded file list (from open command)
  set filename <path>   - Set target file path (required)
  set file <num>        - Set filename using loaded file number (from open list)
  set blocksize <size>  - Set block size (2048,4096,[8192],16384,32768)
  set block <num>       - Set block number (starts from 0, default: 1)
  set offset <offset>   - Set offset within block (< blocksize, default: 0)
  set count <bytes>     - Set number of bytes to read (default: 32)
  set mode edit/browse  - Enable edit/browse mode
  d/dump [options]      - Display data (options: block X, offset Y, count N)
  m/modify <hex> [opts] - Modify data with hex (opts: block X, offset Y)
  undo                  - Undo last modification
  sum [block X]         - Calculate checksum for block (default: current block)
  sum apply [block X]   - Apply checksum: write calculated value to block
  tailchk [block X]     - Calculate tailchk for block (default: current block)
  tailchk apply [block X] - Apply tailchk: write calculated value to block
  copy <src> to <dest>  - Copy data between files
    <src> format: file,block,offset,count (e.g., 1,1,10,64)
    <dest> format: file[,block][,offset] (e.g., 3 or 3,1 or 3,1,128)
  corrupt [block X]     - Mark block as corrupted (default: current block)
  show                  - Display current settings (filename, blocksize, block, offset, count, mode)
  license               - Show/manage software license (registration code required)
  version               - Show software version and developer information
  help                  - Show this help message
  quit/exit             - Exit OBET

加载数据文件

--使用open打开数据文件列表(格式: 编号  路径)
OBET> open /tmp/3.txt
Loaded 4 files from config file '/tmp/3.txt'.

OBET> info

Loaded files (4 total):
----------------------------------------
Number  Path
----------------------------------------
     1  /u01/xifenfei/system01.dbf
     2  /u01/xifenfei/sysaux01.dbf
     3  /u01/xifenfei/undotbs01.dbf
     4  /u01/xifenfei/users01.dbf
----------------------------------------

OBET> set file 1
filename set to: /u01/xifenfei/system01.dbf (file#1)

--或者直接使用 set filename
OBET> set filename /tmp/system01.dbf
filename set to: /tmp/system01.dbf

进入数据文件特定位置

OBET> set file 2
filename set to: /u01/xifenfei/sysaux01.dbf (file#2)

OBET> set block 5
block set to: 5

OBET> set offset 128
offset set to: 128

16进制方式查看数据

OBET> d

File: /u01/xifenfei/system01.dbf
Block: 1                Offsets:     0 to    31
--------------------------------------------------------------------------------
00002000 0BA20000 01004000 00000000 00000104 224D0000 00000000 0004200B A2DB266A 

<32 bytes read>

OBET> dump block 521 offset 128

File: /u01/xifenfei/system01.dbf
Block: 521                Offsets:   128 to   159
--------------------------------------------------------------------------------
00412080 5E068D05 C6040000 00000000 00000000 00000000 00000000 00000000 00000000 

<32 bytes read>

OBET> set count 128
count set to: 128

OBET> d

File: /u01/xifenfei/system01.dbf
Block: 1                Offsets:     0 to   127
--------------------------------------------------------------------------------
00002000 0BA20000 01004000 00000000 00000104 224D0000 00000000 0004200B A2DB266A 
00002020 58494645 4E464549 AC020000 00720100 00200000 01000300 00000000 00000000 
00002040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00002060 08024000 07000000 00000000 7DC92131 64676345 06200E00 00000000 00000000 

<128 bytes read>

16进制方式修改数据块内容(一般修改数据块内容之后建议校验tailchk和sum)

OBET> d

File: /u01/xifenfei/system01.dbf
Block: 1                Offsets:     0 to   127
--------------------------------------------------------------------------------
00002000 0BA20000 01004000 00000000 00000104 224D0000 00000000 0004200B A2DB266A 
00002020 58494645 4E464549 AC020000 00720100 00200000 01000300 00000000 00000000 
00002040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00002060 08024000 07000000 00000000 7DC92131 64676345 06200E00 00000000 00000000 

<128 bytes read>

OBET> m 0123

Confirm modification:
File: /u01/xifenfei/system01.dbf
Block: 1
Offset: 0 (file offset: 0x00002000)
Original value: 0B
New value:      0123
Confirm? (Y/YES to proceed): y
Verification successful: Data written correctly.
Modified 2 bytes at offset 0x00002000 successfully.

OBET> d

File: /u01/xifenfei/system01.dbf
Block: 1                Offsets:     0 to   127
--------------------------------------------------------------------------------
00002000 01230000 01004000 00000000 00000104 224D0000 00000000 0004200B A2DB266A 
00002020 58494645 4E464549 AC020000 00720100 00200000 01000300 00000000 00000000 
00002040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00002060 08024000 07000000 00000000 7DC92131 64676345 06200E00 00000000 00000000 

<128 bytes read>

tail和checksum修改

OBET> tailchk
Check tailchk for File /u01/xifenfei/system01.dbf, Block 1:
current = 0x010B0000, required = 0x01010000

OBET> sum
Check value for File /u01/xifenfei/system01.dbf, Block 1:
current = 0x224D, required = 0x28CC

OBET> tailchk apply

Confirm applying tailchk:
File: /u01/xifenfei/system01.dbf
Block: 1
Offset in block: 8188 (file offset: 0x00003FFC)
Original value: 0x010B0000
New value:      0x01010000
Confirm? (Y/YES to proceed): y
Verification successful: Stored tailchk matches calculated value (0x01010000).
Tailchk applied successfully.

OBET> sum apply

Confirm applying checksum:
File: /u01/xifenfei/system01.dbf
Block: 1
Offset in block: 16 (file offset: 0x00002010)
Original value: 0x224D
New value:      0x28C6
Confirm? (Y/YES to proceed): y
Verification successful: Stored checksum matches calculated value (0x28C6).
Checksum applied successfully.

两个数据文件之前拷贝数据(一般copy数据之后建议校验tailchk和sum)
一般情况下文件之间的拷贝就是数据号不一样,比如修改checkpoint,resetlog信息等,这里支持不一样偏移量,不一样数据块的拷贝

OBET> copy 1,1,0,128 to 3,5,128

Confirm copy:
Source: file#1 (/u01/xifenfei/system01.dbf), block 1, offset 0, 128 bytes
Target: file#3 (/u01/xifenfei/undotbs01.dbf), block 5, offset 128
Proceed? (Y/YES to confirm): y
Copy successful: 128 bytes copied from file #1 to file #3.

OBET> set file 3
filename set to: /u01/xifenfei/undotbs01.dbf (file#3)

OBET> d block 5 offset 128

File: /u01/xifenfei/undotbs01.dbf
Block: 5                Offsets:   128 to   255
--------------------------------------------------------------------------------
0000A080 0BA20000 01004000 00000000 00000104 224D0000 00000000 0004200B A2DB266A 
0000A0A0 58494645 4E464549 AC020000 00720100 00200000 01000300 00000000 00000000 
0000A0C0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
0000A0E0 08024000 07000000 00000000 7DC92131 64676345 06200E00 00000000 00000000 

<128 bytes read>

OBET> set file 1
filename set to: /u01/xifenfei/system01.dbf (file#1)

OBET> dump block 1 offset 0

File: /u01/xifenfei/system01.dbf
Block: 1                Offsets:     0 to   127
--------------------------------------------------------------------------------
00002000 0BA20000 01004000 00000000 00000104 224D0000 00000000 0004200B A2DB266A 
00002020 58494645 4E464549 AC020000 00720100 00200000 01000300 00000000 00000000 
00002040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00002060 08024000 07000000 00000000 7DC92131 64676345 06200E00 00000000 00000000 

<128 bytes read>

标记数据块为坏块功能


OBET> show 

Current settings:
File: /u01/xifenfei/system01.dbf
Blocksize: 8192 bytes
Block: 1
Offset in block: 0 (file offset: 0x00002000)
Count: 128 bytes
Mode: edit
Loaded files: 4 (use 'info' to list)

OBET> corrupt

Confirm modification:
File: /u01/xifenfei/system01.dbf
Block: 1
Offset: 14 (file offset: 0x0000200E)
Original value: 01
New value:      FF
Are you sure to set this block corrupted? (Y/YES to proceed): y
Verification successful: Block 1 marked as corrupted (offset 14 set to 0xFF).
Modification successful.

由于该工具直接编辑Oracle 底层数据块操作具有一定的破坏性和风险性,所以在没有授权的情况下无法对数据块进行修改(只能查看),具体授权操作

OBET> license

========================================
           Software Registration        
========================================

Your Hardware ID: XXXXXXXX  ----->提供给我

Please send your Hardware ID to XiFenFei to register.
Website: https://www.xifenfei.com 
Tel/WX: +86-17813235971 

Enter Registration Code: XXXXXX-XXXXXXXX <-----输入注册码进行授权
Registration successful!

Oracle坏块修复工具:Patch_blk

Posted on 2025 年 11 月 04 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle坏块修复工具:Patch_blk

在win平台上开发了Oracle Recovery Tools小工具,可以实现坏块的快速恢复功能

具体参考相关文章:自研Oracle恢复小工具
由于之前工具使用c#开发不太方便实现跨平台(而且图形化在linux等操作系统上使用起来不方便),现在使用c语言写了小工具:Patch_blk(主要是seq_kcbh tailchk checksum类似坏块处理),模拟两个坏块

BBED> verify
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/xifenfei/system01.dbf
BLOCK = 521

Block 521 is corrupt
Corrupt block relative dba: 0x00400209 (file 0, block 521)
Fractured block found during verification
Data in bad block:
 type: 6 format: 2 rdba: 0x00400209
 last change scn: 0x0000.000001d7 seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x01d706fe
 check value in block header: 0x5205
 computed block checksum: 0x0


DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 0
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 1
Total Blocks Influx           : 2
Message 531 not found;  product=RDBMS; facility=BBED


BBED> verify block 522
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/xifenfei/system01.dbf
BLOCK = 522

Block 522 is corrupt
Corrupt block relative dba: 0x0040020a (file 0, block 522)
Fractured block found during verification
Data in bad block:
 type: 6 format: 2 rdba: 0x0040020a
 last change scn: 0x0000.000001d7 seq: 0xff flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x01d70601
 check value in block header: 0x1e16
 computed block checksum: 0x0


DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 0
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 1
Total Blocks Influx           : 2
Message 531 not found;  product=RDBMS; facility=BBED

配置坏块修复列表文件

[root@iZbp11c0qyuuo1gr7j98upZ tmp]# cat 1.txt
/u01/app/oracle/oradata/xifenfei/system01.dbf  521 8192 N
/u01/app/oracle/oradata/xifenfei/system01.dbf  522 8192 Y 

--列表文件说明
数据文件路径 文件号 数据块大小 是否人工干预(N表示不需要,Y表示需要)

执行修复操作

[root@iZbp11c0qyuuo1gr7j98upZ tmp]# ./Patch_blk 1.txt


===== Processing line 1 =====
=====================================================
Processing: File=/u01/app/oracle/oradata/xifenfei/system01.dbf, Block=521, Size=8192, Mode=N
=====================================================

=== Step 1: Check seq_kcbh ===
  Current value: 0x01
  [OK] seq_kcbh is normal

=== Step 2: Check tailchk ===
  Current (reversed): 0x01D706FE
  Expected (reversed): 0x01D70601
  [Auto-repair] tailchk mismatch, will fix
  [Backup] Block saved to: ./20251104_230616/system01.dbf_block521
  [Success] tailchk fixed
  [Verify] New tailchk (reversed): 0x01D70601

=== Step 3: Check checksum  ===
  Current checksum: 0x0552
  Computed checksum: 0xFA52
  [Auto-repair] Checksum mismatch, will fix
  [Success] Checksum fixed
  [Verify] New checksum: 0xFA52

=== Step 4: Final Verification ===
  seq_kcbh: 0x01 (not 0xFF: PASS)
  tailchk: 0x01D70601 (expected 0x01D70601: PASS)
  Checksum: 0xFA52 (expected 0xFA52: PASS)

[Result] Block repair completed successfully


===== Processing line 2 =====
=====================================================
Processing: File=/u01/app/oracle/oradata/xifenfei/system01.dbf, Block=522, Size=8192, Mode=Y
=====================================================

=== Step 1: Check seq_kcbh ===
  Current value: 0xFF
  [WARNING] Block is marked as BAD. Repair? (yes/no): y
  [Backup] Block saved to: ./20251104/system01.dbf_block522
  [Success] seq_kcbh updated to 0x01

=== Step 2: Check tailchk ===
  Current (reversed): 0x01D70601
  Expected (reversed): 0x01D70601
  [OK] tailchk is normal

=== Step 3: Check checksum ===
  Current checksum: 0x161E
  Computed checksum: 0xE81E
  [WARNING] Checksum mismatch. Repair? (yes/no): y
  [Success] Checksum fixed
  [Verify] New checksum: 0xE81E

=== Step 4: Final Verification ===
  seq_kcbh: 0x01 (not 0xFF: PASS)
  tailchk: 0x01D70601 (expected 0x01D70601: PASS)
  Checksum: 0xE81E (expected 0xE81E: PASS)

[Result] Block repair completed successfully


=====================================
Processing complete. Total: 2
  Modify Success: 2
  Modify None: 0
  Skipped/Failed: 0
  Blocks Backed Up: 2
=====================================

在修复坏块之前会对相关block进行备份

[root@iZbp11c0qyuuo1gr7j98upZ 20251104]# ls -ltra
total 24
-rw-r--r--   1 root root 8192 Nov  4 22:42 system01.dbf_block521
-rw-r--r--   1 root root 8192 Nov  4 22:42 system01.dbf_block522
drwxr-xr-x   2 root root 4096 Nov  4 22:42 .
drwxrwxrwt. 14 root root 4096 Nov  4 23:39 ..
[root@iZbp11c0qyuuo1gr7j98upZ 20251104]#

Patch_blk修复坏块之后,检查坏块正常

BBED> verify
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/xifenfei/system01.dbf
BLOCK = 521


DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED


BBED> verify block 522
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/xifenfei/system01.dbf
BLOCK = 522


DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED

对于不太熟悉bbed的客户,可以通过这个工具快速实现常见坏块类型恢复

ORA-01172 ORA-01151故障处理

Posted on 2025 年 11 月 01 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-01172 ORA-01151故障处理

接手客户一个云平台硬件故障恢复之后,数据库无法启动的case,通过分析alert日志,发现数据库在open过程中报ORA-01172: recovery of thread 1 stuck at block 2220167 of file 262,ORA-01151: use media recovery to recover block, restore backup if needed等相关错误（其实也就是在做实例恢复的过程中报了logically corrupt导致无法完成实例恢复

Sat Nov 01 14:29:10 2025
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 7034 KB redo, 937 data blocks need recovery
Started redo application at
 Thread 1: logseq 296553, block 389408
Recovery of Online Redo Log: Thread 1 Group 3 Seq 296553 Reading mem 0
  Mem# 0: /data/orcl/onlinelog/redo03a.log
Sat Nov 01 14:29:11 2025
Hex dump of (file 262, block 2220584) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p009_2648.trc
Sat Nov 01 14:29:11 2025
Hex dump of (file 262, block 2218886) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p007_2644.trc
Reading datafile '/data/orcl/datafile/xifenfei12.dbf' for corruption at rdba: 0x41a1db86 (file 262, block 2218886)
Reading datafile '/data/orcl/datafile/xifenfei12.dbf' for corruption at rdba: 0x41a1e228 (file 262, block 2220584)
Sat Nov 01 14:29:11 2025
Hex dump of (file 262, block 2219845) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p008_2646.trc
Reading datafile '/data/orcl/datafile/xifenfei12.dbf' for corruption at rdba: 0x41a1df45 (file 262, block 2219845)
Sat Nov 01 14:29:11 2025
Hex dump of (file 262, block 2220167) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p001_2632.trc
Reading datafile '/data/orcl/datafile/xifenfei12.dbf' for corruption at rdba: 0x41a1e087 (file 262, block 2220167)
Reread (file 262, block 2218886) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2218886 OF FILE 262
Reread (file 262, block 2220584) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2220584 OF FILE 262
Reread (file 262, block 2219845) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2219845 OF FILE 262
Reread (file 262, block 2220167) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2220167 OF FILE 262
Sat Nov 01 14:29:26 2025
Slave exiting with ORA-1172 exception
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p007_2644.trc:
ORA-01172: recovery of thread 1 stuck at block 2218886 of file 262
ORA-01151: use media recovery to recover block, restore backup if needed
Sat Nov 01 14:29:26 2025
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p008_2646.trc:
ORA-10388: parallel query server interrupt (failure)
Sat Nov 01 14:29:26 2025
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p009_2648.trc:
ORA-10388: parallel query server interrupt (failure)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p008_2646.trc:
ORA-10388: parallel query server interrupt (failure)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p009_2648.trc:
ORA-10388: parallel query server interrupt (failure)
Sat Nov 01 14:29:26 2025
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p001_2632.trc:
ORA-10388: parallel query server interrupt (failure)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p001_2632.trc:
ORA-10388: parallel query server interrupt (failure)
Sat Nov 01 14:29:26 2025
Aborting crash recovery due to slave death, attempting serial crash recovery
Beginning crash recovery of 1 threads
Started redo scan
Completed redo scan
 read 7034 KB redo, 937 data blocks need recovery
Started redo application at
 Thread 1: logseq 296553, block 389408
Recovery of Online Redo Log: Thread 1 Group 3 Seq 296553 Reading mem 0
  Mem# 0: /data/orcl/onlinelog/redo03a.log
Hex dump of (file 262,block 2220167) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2606.trc
Reading datafile '/data/orcl/datafile/xifenfei12.dbf'for corruption at rdba: 0x41a1e087 (file 262,block 2220167)
Reread (file 262, block 2220167) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2220167 OF FILE 262
Aborting crash recovery due to error 1172
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2606.trc:
ORA-01172: recovery of thread 1 stuck at block 2220167 of file 262
ORA-01151: use media recovery to recover block, restore backup if needed
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2606.trc:
ORA-01172: recovery of thread 1 stuck at block 2220167 of file 262
ORA-01151: use media recovery to recover block, restore backup if needed
ORA-1172 signalled during: ALTER DATABASE OPEN...

接手故障之后,尝试recover database恢复,结果报ORA-600 4552错误

SQL> recover database;
ORA-10562: Error occurred while applying redo to data block (file# 262, block#
2222153)
ORA-10564: tablespace XIFENFEI
ORA-01110: data file 262: '/data/orcl/datafile/xifenfei12.dbf'
ORA-10560: block type '0'
ORA-00600: internal error code, arguments: [4552], [1], [0], [], [], [], [],
[], [], [], [], []

关于ORA-600 4552对应的alert日志信息

Sat Nov 01 17:49:58 2025
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 16 slaves
Sat Nov 01 17:49:59 2025
Recovery of Online Redo Log: Thread 1 Group 3 Seq 296553 Reading mem 0
  Mem# 0: /data/orcl/onlinelog/redo03a.log
Sat Nov 01 17:49:59 2025
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr0c_28770.trc  (incident=1018821):
ORA-00600: internal error code, arguments: [4552], [1], [0], [], [], [], [], [], [], [], [], []
Incident details in:/u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_1018821/orcl_pr0c_28770_i1018821.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Sat Nov 01 17:50:03 2025
Sweep [inc][1018821]: completed
Sweep [inc2][1018821]: completed
Slave exiting with ORA-10562 exception
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr0c_28770.trc:
ORA-10562: Error occurred while applying redo to data block (file# 262, block# 2222153)
ORA-10564: tablespace xifenfei
ORA-01110: data file 262: '/data/orcl/datafile/xifenfei12.dbf'
ORA-10560: block type '0'
ORA-00600: internal error code, arguments: [4552], [1], [0], [], [], [], [], [], [], [], [], []
Recovery Slave PR0C previously exited with exception 10562
Media Recovery failed with error 448
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr00_28732.trc:
ORA-00283: recovery session canceled due to errors
ORA-00448: normal completion of background process
ORA-10562 signalled during: ALTER DATABASE RECOVER  database  ...

无法整个库级别recover,尝试数据文件recover操作

SQL> recover datafile 1;
Media recovery complete.
…………
SQL> recover datafile 22,23,24,26,25,27,28,29,30;
Media recovery complete.
…………
SQL>    recover datafile  251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261;
Media recovery complete.
SQL> recover datafile 262;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [3020], [262], [2215808],
[1101123456], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 262, block# 2215808,
file offset is 972029952 bytes)
ORA-10564: tablespace XIFENFEI
ORA-01110: data file 262: '/data/orcl/datafile/xifenfei12.dbf'
ORA-10560: block type '0'

SQL> recover datafile 263;
Media recovery complete.

出file# 262数据文件之外,其他文件全部recover成功,对应的ORA-600 3020错误相关alert日志信息

Sat Nov 01 17:53:37 2025
ALTER DATABASE RECOVER  datafile 262  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 3 Seq 296553 Reading mem 0
  Mem# 0: /data/orcl/onlinelog/redo03a.log
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_28561.trc  (incident=1018717):
ORA-00600: internal error code, arguments: [3020], [262], [2215808], [1101123456], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 262, block# 2215808, file offset is 972029952 bytes)
ORA-10564: tablespace xifenfei
ORA-01110: data file 262: '/data/orcl/datafile/xifenfei12.dbf'
ORA-10560: block type '0'
Incident details in:/u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_1018717/orcl_ora_28561_i1018717.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Media Recovery failed with error 600
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 262  ...

对于这种情况,有两种处理方式:
1)在recover过程中对于报错的block标记为坏块,然后继续恢复,这样正常应用日志成功,再把标记的坏块修复好
2)直接修改该文件头跳过该文件跳过这些block的应用日志,直接骗过数据库
在本case中由于客户急着恢复业务,需要尽快处理,所以采用了第一个方案,这里我使用自研的m_scn(modify_scn)工具快速修改相关数据文件信息

[oracle@host-172-18-50-10 tmp]$ cat 1.txt
1@/data/orcl/datafile/system01.dbf
262@/data/orcl/datafile/xifenfei12.dbf
[oracle@host-172-18-50-10 tmp]$ ./m_scn 1.txt
Please Enter Password: 

===== Starting Datafile Header modification program =====
Datafile list file: 1.txt
Operation Mode: Only Modify Datafile Header CheckPoint
Block Size: 8192
Log Path: /tmp/modify_scn
---------------------------------------------------------
Preparing Datafile list file...
Verifying Datafile existence...
Datafile verification passed
Initializing working directory...
Recovery script created: /tmp/modify_scn/backup/recover_datafile.sh
---------------------------------------------------------
Starting Datafile Header processing (total 2 files)...
[1/2] Processing Datafile Header: /data/orcl/datafile/system01.dbf (File number: 1)
  - Skipping file number 1 (control file)
---------------------------------------------------------
[2/2] Processing Datafile Header: /data/orcl/datafile/xifenfei12.dbf (File number: 262)
  - Backing up Datafile header...
  - Executing Datafile Header modification with block size 8192...
  - Datafile Header processing completed
---------------------------------------------------------
Cleaning up temporary files...
================= All operations completed =================

Note: Execute /tmp/modify_scn/backup/recover_datafile.sh operation for rollback

然后查询相关scn信息,确认修改文件信息没有问题并尝试recover 262号文件

[oracle@host-172-18-50-10 tmp]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sat Nov 1 18:02:18 2025

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 4.1823E+10 bytes
Fixed Size                  2262368 bytes
Variable Size            4294970016 bytes
Database Buffers         3.7447E+10 bytes
Redo Buffers               78614528 bytes
Database mounted.
SQL> set pages 10000
SQL> set numw 16
SQL> SELECT status,
  2  checkpoint_change#,
  3  to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,
  4  last_change#,
  5  count(*) ROW_NUM
FROM v$datafile
  6    7  GROUP BY status, checkpoint_change#, checkpoint_time,last_change#
ORDER BY status, checkpoint_change#, checkpoint_time;
  8  

set numw 16
col CHECKPOINT_TIME for a40
set lines 150
set pages 1000
SELECT status,
to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,FUZZY,checkpoint_change#,
count(*) ROW_NUM
FROM v$datafile_header
GROUP BY status, checkpoint_change#, to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss'),fuzzy
ORDER BY status, checkpoint_change#, checkpoint_time;

SELECT dd.FILE#,
dd.NAME,
dd.STATUS,
dd.checkpoint_change# dfile_chkp_change,
dh.checkpoint_change# dfile_hed_chkp_change,
dh.recover,
dh.fuzzy
FROM v$datafile dd,
v$datafile_header dh
WHERE dd.FILE#=dh.FILE#
AND dd.checkpoint_change#<>dh.checkpoint_change#;


STATUS  CHECKPOINT_CHANGE# CHECKPOINT_TIME         LAST_CHANGE#          ROW_NUM
------- ------------------ ------------------- ---------------- ----------------
ONLINE      16816934458875 2025-11-01 17:58:28   16816934458875              258
RECOVER     16816934368799 2025-11-01 05:29:39   16816934456943                1
SYSTEM      16816934458875 2025-11-01 17:58:28   16816934458875                4

SQL> SQL> SQL> SQL> SQL> SQL> SQL>   2    3    4    5    6  
STATUS  CHECKPOINT_TIME                          FUZ CHECKPOINT_CHANGE#          ROW_NUM
------- ---------------------------------------- --- ------------------ ----------------
OFFLINE 2025-11-01 17:58:28                      NO      16816934458875                1
ONLINE  2025-11-01 17:58:28                      NO      16816934458875              262

SQL> SQL>   2    3    4    5    6    7    8    9   10   11  
           FILE#
----------------
NAME
----------------------------------------------------------------------------------
STATUS  DFILE_CHKP_CHANGE DFILE_HED_CHKP_CHANGE REC FUZ
------- ----------------- --------------------- --- ---
             262
/data/orcl/datafile/xifenfei12.dbf
RECOVER    16816934368799        16816934458875 YES NO

SQL> recover datafile 262;
Media recovery complete.

open数据库成功

SQL> alter database open;

Database altered.

Sat Nov 01 18:06:00 2025
ALTER DATABASE OPEN
Thread 1 opened at log sequence 296554
  Current log# 1 seq# 296554 mem# 0: /data/orcl/onlinelog/redo01a.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
[33941] Successfully onlined Undo Tablespace 143.
Undo initialization finished serial:0 start:12793234 end:12793304 diff:70 (0 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Sat Nov 01 18:06:01 2025
QMNC started with pid=20, OS id=33973 
Completed: ALTER DATABASE OPEN

至此基本上完成本次恢复任务,后续根据alert日志,有个别表可能由于在file# 262中丢失一些数据导致不一致的问题进行单独,其他没有太大问题,最快帮客户恢复了业务

C_OBJ#_INTCOL#坏块导致数据库无法open故障处理

Posted on 2025 年 10 月 23 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：C_OBJ#_INTCOL#坏块导致数据库无法open故障处理

客户通过硬件恢复出来数据文件之后,尝试启动数据库报错,他们经过多轮尝试依旧无法open数据库,还原到恢复出来文件的初始状态,通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)脚本检测,发现file# 3 需要sequence#为3200的日志,其他文件需要sequence#为3205的日志

检查发现数据库为非归档模式,而且sequence#为3200的redo已经被覆盖

基于上述情况,可以确认除file# 3之外,其他文件可以正常recover

[oracle@oracledb check_db]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Oct 15 16:08:33 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> recover datafile 1;
Media recovery complete.
SQL> recover datafile 2,4,5,7,8,9,10,11,12,13,14,15,16,17;
Media recovery complete.
SQL> recover datafile 18,19,20,21;
Media recovery complete.

对于缺少日志的数据文件,直接使用自研的m_scn(modify scn)工具进行修改

[oracle@oracledb check_db]$ chmod +x m_scn
[oracle@oracledb check_db]$ ./m_scn 1.txt
Please Enter Password: 

===== Starting Datafile Header modification program =====
Datafile list file: 1.txt
Operation Mode: Only Modify Datafile Header CheckPoint
Block Size: 8192
Log Path: /tmp/modify_scn
---------------------------------------------------------
Preparing Datafile list file...
Verifying Datafile existence...
ERROR: Datafile  does not exist!
Datafile verification passed
Initializing working directory...
Recovery script created: /tmp/modify_scn/backup/recover_datafile.sh
---------------------------------------------------------
Starting Datafile Header processing (total 3 files)...
[1/3] Processing Datafile Header: /data/20251014HF/oradata/system01.dbf (File number: 1)
  - Skipping file number 1 (control file)
---------------------------------------------------------
[2/3] Processing Datafile Header: /data/20251014HF/oradata/sysaux01.dbf (File number: 3)
  - Backing up Datafile header...
  - Executing Datafile Header modification with block size 8192...
  - Datafile Header processing completed
---------------------------------------------------------
Cleaning up temporary files...
================= All operations completed =================

Note: Execute /tmp/modify_scn/backup/recover_datafile.sh operation for rollback

[oracle@oracledb check_db]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Oct 15 16:15:12 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> recover datafile 3;
Media recovery complete.

然后尝试打开数据库报ORA-01092 ORA-01578等错误

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00604: error occurred at recursive SQL level 2
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
Process ID: 1617
Session ID: 1 Serial number: 5

报错比较明显是由于坏块导致数据库无法打开,进一步检查alert日志

2025-10-14T21:38:22.889985+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2025-10-14T21:38:22.909799+08:00
Starting background process SMCO
2025-10-14T21:38:22.930114+08:00
SMCO started with pid=57, OS id=1630 
2025-10-14T21:38:23.000833+08:00
ARC0 (PID:1616): Archived Log entry 3 added for T-1.S-3205 ID 0x6539a8b7 LAD:1
2025-10-14T21:38:23.524409+08:00
Undo initialization recovery: err:0 start: 19911054 end: 19911123 diff: 69 ms (0.1 seconds)
2025-10-14T21:38:23.826920+08:00
[26765] Successfully onlined Undo Tablespace 2.
Undo initialization online undo segments: err:0 start: 19911123 end: 19911426 diff: 303 ms (0.3 seconds)
Undo initialization finished serial:0 start:19911054 end:19911454 diff:400 ms (0.4 seconds)
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Verifying minimum file header compatibility for tablespace encryption..
Verifying file header compatibility for tablespace encryption completed for pdb 0
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is AL32UTF8
No Resource Manager plan active
2025-10-14T21:38:24.645423+08:00
Hex dump of (file 1, block 132585) in trace file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc

Invalid temporary block relative dba: 0x004205e9 (file 1, block 132585)
Bad header found during buffer read
Data in bad block:
 type: 66 format: 2 rdba: 0x4c444242
 last change scn: 0x4c44.4242.4c444242 seq: 0x44 flg: 0x4c
 spare3: 0x4c44
 consistency value in tail: 0xfcc60601
 check value in block header: 0x4242
 computed block checksum: 0x9e0a

Reading datafile '/data/app/oracle/oradata/ORCL/system01.dbf' for corrupt data at rdba:0x004205e9(file 1,block 132585)
Reread (file 1, block 132585) found same corrupt data (no logical check)
2025-10-14T21:38:24.838291+08:00
Errors in file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc  (incident=51502):
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
Incident details in: /u01/app/diag/rdbms/orcl/ORCL/incident/incdir_51502/ORCL_ora_26765_i51502.trc
2025-10-14T21:38:24.861113+08:00
Corrupt Block Found
         TIME STAMP (GMT) = 10/14/2025 21:38:23
         CONT = 0, TSN = 0, TSNAME = SYSTEM
         RFN = 1, BLK = 132585, RDBA = 4326889
         OBJN = 66, OBJD = 64, OBJECT = C_OBJ#_INTCOL#, SUBOBJECT = 
         SEGMENT OWNER = SYS, SEGMENT TYPE = Cluster Segment
2025-10-14T21:38:26.554220+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2025-10-14T21:38:26.769742+08:00
Errors in file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc:
ORA-00604: error occurred at recursive SQL level 2
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
2025-10-14T21:38:26.769940+08:00
Errors in file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc:
ORA-00604: error occurred at recursive SQL level 2
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
Error 604 happened during db open, shutting down database
Errors in file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc  (incident=51503):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00604: error occurred at recursive SQL level 2
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
Incident details in: /u01/app/diag/rdbms/orcl/ORCL/incident/incdir_51503/ORCL_ora_26765_i51503.trc
Invalid temporary block relative dba: 0x004205e9 (file 1, block 132585)
Bad header found during validation
Data in bad block:
 type: 66 format: 2 rdba: 0x4c444242
 last change scn: 0x4c44.4242.4c444242 seq: 0x44 flg: 0x4c
 spare3: 0x4c44
 consistency value in tail: 0xfcc60601
 check value in block header: 0x4242
 computed block checksum: 0x9e0a

Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Checker run found 1 new persistent data failures
2025-10-14T21:38:27.515191+08:00
opiodr aborting process unknown ospid (26765) as a result of ORA-603
2025-10-14T21:38:27.528063+08:00
ORA-603 : opitsk aborting process
License high water mark = 18
USER (ospid: (prelim)): terminating the instance due to ORA error 
2025-10-14T21:38:28.576271+08:00
Instance terminated by USER(prelim), pid = 26765

通过分析alert日志可以确定是由于file 1, block 132585损坏,对应的对象为C_OBJ#_INTCOL#,该对象是histgrm$表的cluster,非数据库必须核心对象,可以在启动的时候跳过该对象,启动数据库,然后设置event对该对象进行处理

SQL> alter database open;

Database altered.

SQL>  create table good_histgrm$ as select * from histgrm$;

Table created.

SQL> truncate cluster c_obj#_intcol#;

Cluster truncated.

SQL> insert into histgrm$ select * from good_histgrm$ ;

286071 rows created.

SQL> commit;

Commit complete.

SQL> select /*+ full(a) */ count(*) from histgrm$ a ;

  COUNT(*)
----------
    286071

SQL> select /*+ full(a) */ count(*) from histgrm$ a ;

  COUNT(*)
----------
    286071

至此整体上完成该库的恢复任务

ORA-600 kkkicreatecgmap:!efn3

Posted on 2025 年 10 月 23 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 kkkicreatecgmap:!efn3

raid故障恢复之后,数据库recover成功,但是open报ORA-03113: end-of-file on communication channel错误

SQL> recover database;
Media recovery complete.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 20394
Session ID: 191 Serial number: 3

对应的alert日志错误为ORA-600 [kkkicreatecgmap:!efn3]错误

ALTER DATABASE RECOVER  database
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 4 slaves
Mon Oct 20 18:51:06 2025
Recovery of Online Redo Log: Thread 1 Group 1 Seq 32119 Reading mem 0
  Mem# 0: /u01/oradata/redo01.log
Media Recovery Complete (orcl)
Completed: ALTER DATABASE RECOVER  database
Mon Oct 20 18:51:16 2025
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 3 processes
Started redo scan
Completed redo scan
 read 41 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 32119, block 34978
Recovery of Online Redo Log: Thread 1 Group 1 Seq 32119 Reading mem 0
  Mem# 0: /u01/oradata/redo01.log
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 32119, block 35061, scn 17375938230308
 0 data blocks read, 0 data blocks written, 41 redo k-bytes read
Mon Oct 20 18:51:16 2025
Thread 1 advanced to log sequence 32120 (thread open)
Thread 1 opened at log sequence 32120
  Current log# 2 seq# 32120 mem# 0: /u01/oradata/redo02.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Oct 20 18:51:16 2025
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Re-creating tempfile /u01/oradata/temp01.dbf
Database Characterset is ZHS16GBK
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x2297750, kgegpa()+40][flags: 0x0, count: 1]
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x229597B, kgebse()+279][flags: 0x2, count: 2]
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x229597B, kgebse()+279][flags: 0x2, count: 2]
No Resource Manager plan active
Mon Oct 20 18:51:16 2025
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20286.trc  (incident=3649):
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_3649/orcl_dbrm_20286_i3649.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20286.trc:
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
DBRM (ospid: 20286): terminating the instance due to error 56710
Instance terminated by DBRM, pid = 20286

对应的trace文件内容


----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
skdstdst()+36        call     kgdsdst()            000000000 ? 000000000 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   7FFD08773E78 ? 000000000 ?
ksedst1()+98         call     skdstdst()           000000000 ? 000000000 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksedst()+34          call     ksedst1()            000000000 ? 000000001 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbkedDefDump()+2736  call     ksedst()             000000000 ? 000000001 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksedmp()+36          call     dbkedDefDump()       000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksfdmp()+64          call     ksedmp()             000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgexPhaseII()+1764  call     ksfdmp()             000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgexProcessError()  call     dbgexPhaseII()       7F16CCF3E6F0 ? 7F16CA6F2598 ?
+2279                                              7FFD0877BC68 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgeExecuteForError  call     dbgexProcessError()  7F16CCF3E6F0 ? 7F16CA6F2598 ?
()+83                                              000000001 ? 000000000 ?
                                                   7FFD00000000 ? 000000000 ?
dbgePostErrorKGE()+  call     dbgeExecuteForError  7F16CCF3E6F0 ? 7F16CA6F2598 ?
1615                          ()                   000000001 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbkePostKGE_kgsf()+  call     dbgePostErrorKGE()   000000000 ? 7F16CA560040 ?
63                                                 000000258 ? 7F16CA6F2598 ?
                                                   000000000 ? 000000000 ?
kgeadse()+383        call     dbkePostKGE_kgsf()   00A99D360 ? 7F16CA560040 ?
                                                   000000258 ? 7F16CA6F2598 ?
                                                   000000000 ? 000000000 ?
kgerinv_internal()+  call     kgeadse()            00A99D360 ? 7F16CA560040 ?
45                                                 000000258 ? 000000000 ?
                                                   000000000 ? 000000000 ?
kgerinv()+33         call     kgerinv_internal()   00A99D360 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000258 ?
                                                   000000000 ? 000000000 ?
kgeasnmierr()+143    call     kgerinv()            00A99D360 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
kkkicreatecgmap()+3  call     kgeasnmierr()        00A99D360 ? 7F16CA560040 ?
125                                                877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskirefreshcgmap()+  call     kkkicreatecgmap()    121DC3050 ? 7F16CA560040 ?
104                                                877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskreload()+1150     call     kskirefreshcgmap()   121DC3050 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskdbrmpa()+378      call     kskreload()          7FFD0877D428 ? 000000001 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
ksbabs()+465         call     kskdbrmpa()          7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
ksbrdp()+923         call     ksbabs()             7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opirip()+618         call     ksbrdp()             7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opidrv()+598         call     opirip()             000000032 ? 000000004 ?
                                                   7FFD0877E598 ? 000000001 ?
                                                   000000000 ? 00000020A ?
sou2o()+98           call     opidrv()             000000032 ? 000000004 ?
                                                   7FFD0877E598 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opimai_real()+261    call     sou2o()              7FFD0877E570 ? 000000032 ?
                                                   000000004 ? 7FFD0877E598 ?
                                                   000000000 ? 00000020A ?
ssthrdmain()+209     call     opimai_real()        000000000 ? 7FFD0877E760 ?
                                                   000000004 ? 7FFD0877E598 ?
                                                   000000000 ? 00000020A ?
main()+196           call     ssthrdmain()         000000003 ? 7FFD0877E760 ?
                                                   000000001 ? 000000000 ?
                                                   000000000 ? 00000020A ?
__libc_start_main()  call     main()               000000003 ? 7FFD0877E900 ?
+253                                               000000001 ? 000000000 ?
                                                   000000000 ? 00000020A ?
_start()+36          call     __libc_start_main()  0009D3D94 ? 000000001 ?
                                                   7FFD0877E8F8 ? 000000000 ?
                                                   000000000 ? 00000020A ?

虽然ORA-600 kkkicreatecgmap:!efn3没有见过但是数据库open过程中kgegpa、kgebse的故障还是遇到不少,大部分可能和undo有一定关系,处理undo问题之后,继续尝试open库,依旧报ORA-03113: end-of-file on communication channel

SQL> alter database Open;
ERROR:
ORA-03113: end-of-file on communication channel
Process ID: 20586
Session ID: 191 Serial number: 3

但是后台的alert日志已经改变ORA-600 4193,ORA-600 kkkicreatecgmap:!efn3错误

Mon Oct 20 18:54:39 2025
Thread 1 advanced to log sequence 32121 (thread open)
Thread 1 opened at log sequence 32121
  Current log# 3 seq# 32121 mem# 0: /u01/oradata/redo03.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Oct 20 18:54:39 2025
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc  (incident=4905):
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4905/orcl_smon_20492_i4905.trc
No Resource Manager plan active
Mon Oct 20 18:54:39 2025
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20478.trc  (incident=4849):
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4849/orcl_dbrm_20478_i4849.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20478.trc:
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
DBRM (ospid: 20478): terminating the instance due to error 56710
Some DDE async actions failed or were cancelled
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc  (incident=4906):
ORA-00353: log corruption near block 8 change 17375938190767 time 10/03/2025 00:20:34
ORA-00312: online log 1 thread 1: '/u01/oradata/redo01.log'
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4906/orcl_smon_20492_i4906.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 8 change 17375938190767 time 10/03/2025 00:20:34
ORA-00312: online log 1 thread 1: '/u01/oradata/redo01.log'
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Instance terminated by DBRM, pid = 20478

虽然ORA-600 kkkicreatecgmap:!efn3还在,但是看到了比较熟悉的ORA-600 4193错误,处理undo异常回滚段,数据库open成功,重建undo,尝试导出数据,完成数据恢复任务.

补充说明,对于ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403]网络上没有任何资料,查询了mos发现一个有一点类似的报错信息
Bug 28167557 – bigscn_dbim_tm_def – trc – kkkdchkcriticalobj – ORA-700 [kkkdchkcriticalob (Doc ID 28167557.8)

Description
Information about new symptoms:
 
- Signaling function: kkkdchkcriticalobj (kkkd.c)
- Symptom: ORA-700 [kkkdchkcriticalobj:fail]
- Owner: SUELEE
- Special Run:  BIGSCN_DBIM_TM_DEF (owner: WAI-SZE.TAM, sr_id: 2709)
- Release: 19.1
- Job id: 22488605
- Base label:  RDBMS_MAIN_LINUX.X64_180607
 
ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 
 REDISCOVERY INFORMATION:
 Symptoms are any of the following:
 ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 .
 WORKAROUND:
 None

由于这个是硬件故障恢复出来的数据文件（涉及磁盘坏道，磁盘顺序，磁盘在raid中均衡,raid的cache等因素可能会一起Oracle各种非常规问题）,分析原因意义不大,重点是快速解决问题,不做过多分析