ORA-600 krsm_mrp_rebuild_lf_list.lno_mismatch

Posted on 2026 年 09 月 16 日 by 惜分飞

标题：ORA-600 krsm_mrp_rebuild_lf_list.lno_mismatch

客户由于断电,导致一个19c数据库无法正常启动,接手时候报错为：ORA-600 krsm_mrp_rebuild_lf_list.lno_mismatch

2026-09-15T06:36:24.908317-07:00
ALTER DATABASE OPEN RESETLOGS
2026-09-15T06:36:24.912251-07:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 22297188690 time 
NET  (PID:36760): Unable to create archive log file '/u19/dbs/arch1_1_1243990641.dbf'
2026-09-15T06:36:24.919696-07:00
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_36760.trc:
ORA-19504: failed to create file "/u19/dbs/arch1_1_1243990641.dbf"
ORA-27038: created file already exists
Additional information: 1
NET  (PID:36760): Error 19504 Creating archive log file to '/u19/app/dbs/arch1_1_1243990641.dbf'
NET  (PID:36760): Archiving not possible: error count exceeded
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_36760.trc  (incident=119876):
ORA-00600: internal error code, arguments: [krsm_mrp_rebuild_lf_list.lno_mismatch], [3], [1], [1], [0], []
Incident details in: /u19/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_119876/orcl_ora_36760_i119876.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
NET  (PID:36760): Archive all ORLs failed, error=600
ORA-600 signalled during: ALTER DATABASE OPEN RESETLOGS...

通过alert日志大概回溯现场人员的恢复过程,最初数据库断电重启之后报ORA-00742错误

2026-09-14T20:52:37.294285-07:00
ALTER DATABASE OPEN
Ping without log force is disabled:
  instance mounted in exclusive mode.
2026-09-14T20:52:37.313266-07:00
Beginning crash recovery of 1 threads
 Thread 1: Recovery starting at checkpoint rba (logseq 63111 block 286483), scn 0
2026-09-14T20:52:37.370330-07:00
Started redo scan
2026-09-14T20:52:37.419944-07:00
Aborting crash recovery due to error 742
2026-09-14T20:52:37.420039-07:00
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_68055.trc:
ORA-00742: Log read detects lost write in thread 1 sequence 63111 block 326157
ORA-00312: online log 3 thread 1: '/u19/app/oracle/oradata/ORCL/redo03.log'
2026-09-14T20:52:37.420977-07:00
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_68055.trc:
ORA-00742: Log read detects lost write in thread 1 sequence 63111 block 326157
ORA-00312: online log 3 thread 1: '/u19/app/oracle/oradata/ORCL/redo03.log'
ORA-742 signalled during: ALTER DATABASE OPEN...

现场人员加上隐含参数强制拉库,然后数据库报ORA-600 kcbzib_kcrsds_1错误

2026-09-14T21:35:21.425310-07:00
ALTER DATABASE OPEN RESETLOGS
2026-09-14T21:35:21.429136-07:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 22297188666 time 
Resetting resetlogs activation ID 1680209845 (0x6425f7b5)
2026-09-14T21:35:34.017515-07:00
Setting recovery target incarnation to 3
2026-09-14T21:35:34.028360-07:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
Endian type of dictionary set to little
2026-09-14T21:35:34.043771-07:00
Assigning activation ID 1771732361 (0x699a7d89)
2026-09-14T21:35:34.125473-07:00
TT00 (PID:47466): Gap Manager starting
2026-09-14T21:35:34.127554-07:00
Redo log for group 1, sequence 1 is not located on DAX storage
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /u19/app/oracle/oradata/ORCL/redo01.log
Successful open of redo thread 1
2026-09-14T21:35:34.165332-07:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2026-09-14T21:35:35.507453-07:00
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1],  , [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
SYSTEM DUMP REDO DBA MIN 4 128 DBA MAX 4 128 SCN MIN 1;
2026-09-14T21:35:37.154962-07:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Undo initialization recovery: err:600 start: 3433752 end: 3435865 diff: 2113 ms (2.1 seconds)
2026-09-14T21:35:37.475752-07:00
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_46217.trc:
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1],  , [], [], [], [], []
2026-09-14T21:35:37.475831-07:00
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_46217.trc:
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1],  , [], [], [], [], []
Error 600 happened during db open, shutting down database
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_46217.trc  (incident=48128):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1],  , [], [], [], [], []

现场人员重建控制文件,然后启动库报ORA-600 krsm_mrp_rebuild_lf_list.lno_mismatch

2026-09-15T04:26:41.297637-07:00
Successful mount of redo thread 1, with mount id 1771762176
Completed: CREATE CONTROLFILE REUSE SET DATABASE "ORCL" RESETLOGS ARCHIVELOG
    MAXLOGFILES 16
    MAXLOGMEMBERS 3
    MAXDATAFILES 100
    MAXINSTANCES 1
    MAXLOGHISTORY 16
LOGFILE
  GROUP 1 '/u19/app/oracle/oradata/ORCL_NEW/redo01.log' SIZE 200M,
  GROUP 2 '/u19/app/oracle/oradata/ORCL_NEW/redo02.log' SIZE 200M
DATAFILE
  '/u19/app/oracle/oradata/ORCL_NEW/system01.dbf',
  '/u19/app/oracle/oradata/ORCL_NEW/sysaux01.dbf',
  '/u19/app/oracle/oradata/ORCL_NEW/undotbs01.dbf',
…………
2026-09-15T04:26:52.121793-07:00
ALTER DATABASE OPEN RESETLOGS
2026-09-15T04:26:52.126848-07:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 22297188686 time 
NET  (PID:27130): Unable to create archive log file '/u19/dbs/arch1_1_1243990641.dbf'
2026-09-15T04:26:52.233538-07:00
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_27130.trc:
ORA-19504: failed to create file "/u19/dbs/arch1_1_1243990641.dbf"
ORA-27038: created file already exists
Additional information: 1
NET  (PID:27130): Error 19504 Creating archive log file to '/u19/dbs/arch1_1_1243990641.dbf'
NET  (PID:27130): Archiving not possible: error count exceeded
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_27130.trc  (incident=102102):
ORA-00600: internal error code, arguments: [krsm_mrp_rebuild_lf_list.lno_mismatch], [2], [1], [1], [0], []
Incident details in: /u19/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_102102/orcl_ora_27130_i102102.trc

接手故障开始处理,尝试open库,重现ORA-600 krsm_mrp_rebuild_lf_list.lno_mismatch错误

SQL> recover database using backup controlfile;
ORA-00279: change 22297188689 generated at 09/15/2026 00:57:22 needed for thread 1
ORA-00289: suggestion : /u19/dbs/arch/1_1_1243990641.dbf
ORA-00280: change 22297188689 for thread 1 is in sequence #1


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/u19/app/oracle/oradata/ORCL/redo01.log
Log applied.
Media recovery complete.
SQL>        
SQL> 
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [krsm_mrp_rebuild_lf_list.lno_mismatch], [3], [1], [1], [0]

分析alert日志,是由于归档无法被正常创建导致该问题,对/u19/dbs/arch/进行创建并授权之后,再次尝试打开数据库,数据库报ORA-600 kcbzib_kcrsds_1错误

SQL> startup nomount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area       6442448984 bytes
Fixed Size			  8910936 bytes
Variable Size		       3858759680 bytes
Database Buffers	       2566914048 bytes
Redo Buffers			  7864320 bytes
SQL> alter database mount;

Database altered.

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], []
Process ID: 42259
Session ID: 1 Serial number: 19675

使用obet的patch scn功能修改scn之后（obet patch_scn使用说明）,顺利打开库

记录一次0丢失的ORA-00354: 损坏重做日志块标头故障恢复

Posted on 2026 年 09 月 10 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：记录一次0丢失的ORA-00354: 损坏重做日志块标头故障恢复

由于服务器断电导致数据库启动报错

Mon Sep 07 12:47:52 2026
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 32 processes
Started redo scan
ORA-00354: 损坏重做日志块标头
ORA-00353: 日志损坏接近块 41735 更改 2385313823 时间 09/07/2026 10:35:04
ORA-00312: 联机日志 5 线程 1: 'D:\APP\LIMSADMIN\ORADATA\ORCL\REDO05.LOG'
Mon Sep 07 12:47:58 2026
Trace dumping is performing id=[cdmp_20260907124758]
Errors in file d:\app\limsadmin\diag\rdbms\primary\orcl\trace\orcl_ora_3040.trc:
ORA-00354: 损坏重做日志块标头
ORA-00353: 日志损坏接近块 41735 更改 2385313823 时间 09/07/2026 10:35:04
ORA-00312: 联机日志 5 线程 1: 'D:\APP\LIMSADMIN\ORADATA\ORCL\REDO05.LOG'
ORA-354 signalled during: ALTER DATABASE OPEN...

从这里看,数据库在启动的时候实例恢复需要读取REDO05.LOG日志,但是发现该日志中有block损坏,而且损坏日志的时间点为09/07/2026 10:35:04,scn为2385313823（ORA-00353: 日志损坏接近块 41735 更改 2385313823 时间 09/07/2026 10:35:04）,导致实例恢复无法完成,从而数据库无法正常启动.通过Oracle Recovery Check脚本检查数据库当前状态,数据库的checkpoint time为:09/07/2026 10:37:37,checkpoint scn为:2385314357

这个就比较明显,数据库文件正在需要的日志是sequence 19001 REDO01.LOG(而不是报错的sequence 19000 REDO05.LOG),基于这样的情况,出现这种问题,是由于control中记录的Thread Checkpoint RBA不正确导致.解决这个问题相对比较简单,重建ctl即可

由于redo05.log有损坏,该库为归档模式,需要先clear redo05.log（不然会导致redo无法归档,数据库hang住）,然后对全库发起备份

不当数据库恢复操作导致一个月数据丢失

Posted on 2026 年 09 月 04 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：不当数据库恢复操作导致一个月数据丢失

今天分享一个本不该发生的故障,结果由于现场人员没有合适的判断导致客户丢失近一个月数据,事故经过大概是这样的
1. 客户数据库由于断电,启动之后报ORA-600 kcratr_scan_lastbwr错误

2026-09-03T11:22:58.998379+08:00
Beginning crash recovery of 2 threads
 parallel recovery started with 31 processes
 Thread 1: Recovery starting at checkpoint rba (logseq 37738 block 7724), scn 0
 Thread 2: Recovery starting at checkpoint rba (logseq 43821 block 37937), scn 0
2026-09-03T11:22:59.158232+08:00
Started redo scan
Hex dump of (file 20, block 2720) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_34049.trc

Reading datafile '+DATA/ORCL/DATAFILE/undotbs01.dbf' for corrupt data at rdba: 0x05000aa0 (file 20, block 2720)
Reread Not Encrypted (file 20, block 2720) found same data 
physically corrupt:No, logically corrupt:No, and is not expected version by upper layer. 
Write verification failed for File 20 Block 2720 (rdba 0x5000aa0)
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_34049.trc  (incident=430645):
ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_430645/orcl1_ora_34049_i430645.trc
2026-09-03T11:23:00.484486+08:00
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2026-09-03T11:23:00.510512+08:00
Slave encountered ORA-10388 exception during crash recovery

这个是一个比较常见的错误一般是由于redo写丢失导致(数据库open报ORA-600 kcratr_scan_lastbwr故障处理)

2. 现场操作的技术人员看了下这个错误,觉得数据库有备份，直接决定使用备份进行恢复

结果发现读取的是20260719的备份,而且原始的数据文件在asm里面，已经被覆盖。这个时候如果归档或者增量备份完整,还能够最大限度抢救数据,检查备份发现只有8月7日的增量备份,没有任何归档的备份(使用crontab 直接清理3天之前的归档)

基于这样的情况,应用到最后一个增量备份(也就是8月7日),后面的数据由于归档已经被清理(而且没有备份)

出现这个问题的原因是由于备份脚本本身备份归档,导致备份无法按照策略清理历史备份,导致磁盘空间满,从而后续备份无法继续(直接清理归档了)

恢复只能到此为止,查询最终的文件头恢复时间点

3. 基于这种情况只能尝试强制打开数据库,结果报ORA-600 kcbzibmlt_kcrsds_1错误

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzibmlt_kcrsds_1], [], [], []
Process ID: 30365
Session ID: 2180 Serial number: 43414

4. 这种错误类似ORA-600 kcbzib_kcrsds_1的处理思路,使用obet的patch_scn功能直接进行处理

OBET> patch_scn 89103 0x60017e98 3182707622

=== Oracle SCN Patch ===
PID:     89103
Address: 0x60017e98
New Value: 0xbdb443a6 (3182707622)
Original SCN: 0x0

Are you sure you want to modify Oracle SCN? (yes/no): y

SCN modified: 0x0 -> 0xbdb443a6
Oracle SCN successfully modified.

然后打开数据库

SQL> alter database open;

Database altered.

导出数据,导入新库完成本次恢复任务,可惜没有办法帮客户找回来丢失的那一个月数据.

案例总结:
1. rman备份脚本尽量不要有直接删除归档的操作,尽可能是先备份然后再删除(确认备份成功之后删除)
2. 备份还原操作之前,需要看备份文件和日志,大概评估下备份的可用性
3. 尽可能不要在损坏的现场环境上直接做rman还原操作(万一备份无法正常还原,比如中途有备份文件异常呢？)
4. 这个故障,如果不做还原操作,直接异常恢复,基本上也就丢失了当前redo中的数据,损失是最小的

obet快速修复oracle 位图损坏块

Posted on 2026 年 09 月 03 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：obet快速修复oracle 位图损坏块

近期有客户由于虚拟化平台系统负载过高,重启之后导致数据库出现一些坏块

其中被框起来的坏块比较好处理,属于数据库中间某个对象的block,对异常对象进行处理即可,但是有一个block 2，block 10 这种都属于数据文件中位图块(用来记录哪些block被使用,哪些block是没有使用),这种block如果损坏会导致该文件无法分配新的空间(也不能回收空间,无法统计表空间使用情况等)

alert日志有大量报错

对于这种情况,直接使用obet的坏块修复功能

OBET> set file 116
filename set to: /oradata/NNC_DATA207.dbf (file# 116)
endian auto-detected: little

OBET> set block 10
block set to: 10

OBET> backup
Backing up file #116, block 10
Successfully backed up block 10 from /oradata/NNC_DATA207.dbf to backup_blk/NNC_DATA207.dbf.10_20260902200547.blk

OBET> repair block
Warning: Missing value for 'block', using global setting: 10

Repairing block 10 in file /oradata/NNC_DATA207.dbf...

Repair analysis for block 10:
1. seq_kcbh check: 0xFF -> needs repair (0x01)
2. Tailchk check: 0x00001EFF -> needs repair (0x00001E01)
3. Checksum check: 0xCBC7 -> OK

Confirm repair operations:
File: /oradata/NNC_DATA207.dbf
Block: 10
Operations needed: fix offset14, fix tailchk
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x00001E01 OK
3. Checksum: 0xCBC7 OK

Block 10 repair completed successfully.

OBET>

OBET> set file 119
filename set to: /oradata/NNC_DATA210.dbf (file# 119)
endian auto-detected: little

OBET> set block 2
block set to: 2

OBET> backup
Backing up file #119, block 2
Successfully backed up block 2 from /oradata/NNC_DATA210.dbf to backup_blk/NNC_DATA210.dbf.2_20260902200645.blk

OBET> repair block
Warning: Missing value for 'block', using global setting: 2

Repairing block 2 in file /oradata/NNC_DATA210.dbf...

Repair analysis for block 2:
1. seq_kcbh check: 0xFF -> needs repair (0x01)
2. Tailchk check: 0x00001DFF -> needs repair (0x00001D01)
3. Checksum check: 0x80EC -> OK

Confirm repair operations:
File: /oradata/NNC_DATA210.dbf
Block: 2
Operations needed: fix offset14, fix tailchk
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x00001D01 OK
3. Checksum: 0x80EC OK

Block 2 repair completed successfully.

OBET> exit
Exiting OBET.

使用dbv检测,位图坏块全部修复

[oracle@nccdb ~]$ dbv file=/oradata/NNC_DATA210.dbf

DBVERIFY: Release 11.2.0.4.0 - Production on Wed Sep 2 20:07:15 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /oradata/NNC_DATA210.dbf


DBVERIFY - Verification complete

Total Pages Examined         : 4194302
Total Pages Processed (Data) : 3472440
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 205988
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 515867
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 7
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 751241252 (22.751241252)
[oracle@nccdb ~]$ dbv file=/oradata/NNC_DATA207.dbf

DBVERIFY: Release 11.2.0.4.0 - Production on Wed Sep 2 20:08:23 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /oradata/NNC_DATA207.dbf


DBVERIFY - Verification complete

Total Pages Examined         : 4194302
Total Pages Processed (Data) : 3471468
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 204185
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 518642
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 7
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 751241266 (22.751241266)

后续数据顺利导出,完成这次恢复

不太常见的10.2.0.1的oracle redo损坏恢复

Posted on 2026 年 08 月 27 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：不太常见的10.2.0.1的oracle redo损坏恢复

曾经恢复过大量10g的库,现在一年也恢复不了几个10g的了,而10.2.0.1的64位库更是少之又少了.近期有幸处理过一个这样的case,重温了当年的感觉

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bi
PL/SQL Release 10.2.0.1.0 - Production
CORE 10.2.0.1.0 Production
TNS for 64-bit Windows: Version 10.2.0.1.0 - Production
NLSRTL Version 10.2.0.1.0 - Production

由于系统断电导致该版本的erp系统的数据库无法启动,看alert日志由于redo损坏导致异常(ORA-00354 ORA-00353)

Thu Jun 04 10:27:40 2026
ALTER DATABASE RECOVER  datafile 5  
Thu Jun 04 10:27:40 2026
Media Recovery Start
WARNING! Recovering data file 5 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
 parallel recovery started with 7 processes
Thu Jun 04 10:27:40 2026
Recovery of Online Redo Log: Thread 1 Group 3 Seq 26215 Reading mem 0
  Mem# 0 errs 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG
Thu Jun 04 10:28:06 2026
Errors in file d:\oracle\product\10.2.0\admin\ORCL\udump\ORCL_ora_3856.trc:
ORA-00354: 损坏重做日志块头部
ORA-00353: 日志损坏接近块 93816 更改 1049070732 时间 05/27/2025 18:00:16
ORA-00334: 归档日志: 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG'

接手之后再次验证了该错误