不太常见的10.2.0.1的oracle redo损坏恢复

Posted on 2026 年 08 月 27 日 by 惜分飞

曾经恢复过大量10g的库,现在一年也恢复不了几个10g的了,而10.2.0.1的64位库更是少之又少了.近期有幸处理过一个这样的case,重温了当年的感觉

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bi
PL/SQL Release 10.2.0.1.0 - Production
CORE 10.2.0.1.0 Production
TNS for 64-bit Windows: Version 10.2.0.1.0 - Production
NLSRTL Version 10.2.0.1.0 - Production

由于系统断电导致该版本的erp系统的数据库无法启动,看alert日志由于redo损坏导致异常(ORA-00354 ORA-00353)

Thu Jun 04 10:27:40 2026
ALTER DATABASE RECOVER  datafile 5  
Thu Jun 04 10:27:40 2026
Media Recovery Start
WARNING! Recovering data file 5 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
 parallel recovery started with 7 processes
Thu Jun 04 10:27:40 2026
Recovery of Online Redo Log: Thread 1 Group 3 Seq 26215 Reading mem 0
  Mem# 0 errs 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG
Thu Jun 04 10:28:06 2026
Errors in file d:\oracle\product\10.2.0\admin\ORCL\udump\ORCL_ora_3856.trc:
ORA-00354: 损坏重做日志块头部
ORA-00353: 日志损坏接近块 93816 更改 1049070732 时间 05/27/2025 18:00:16
ORA-00334: 归档日志: 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG'

接手之后再次验证了该错误

这种错误基本上要不bbed/obet修改文件头,要不直接屏蔽一致性强制打开,我直接选择了强制拉库,先做不完全恢复

然后强制拉库报ORA-600 2662错误

直接使用_minimum_giga_scn修改scn,数据库open成功,并使用expdp成功导出数据,完成本次恢复工作

obet forcecopy功能抢救硬件故障中的数据文件

Posted on 2026 年 08 月 27 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：obet forcecopy功能抢救硬件故障中的数据文件

客户raid5发现有盘离线,并且其他盘有坏道,进行了多次换盘操作替换了大部分有坏道磁盘,但是部分文件访问依旧会报io错误

直接cp文件报

cp: error reading 'datafile': Input/output error

数据库层面操作报

ORA-01115: IO error reading block from file %s (block # %s)
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 21571487880

对于这样的情况,使用obet的forcecopy功能直接强制拷贝文件,可以最大限度求救数据obet forcecopy

把所有损坏文件拷贝出来之后传输到其他机器上,检查所有文件坏块情况

运气不错,整体坏块非常少,而且没有涉及数据库字典文件,然后尝试恢复

虽然recover database成功,但是不幸open 库报ORA-600 kcbzib_kcrsds_1错误

SQL> ALTER DATABASE OPEN ;
ALTER DATABASE OPEN 
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], []
Process ID: 15786
Session ID: 3054 Serial number: 46983

通过obet的patch_scn功能进行修改scn

顺利打开数据库,并导出数据

kcratr_nab_less_than_odr和system坏块故障处理

Posted on 2026 年 08 月 25 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：kcratr_nab_less_than_odr和system坏块故障处理

学校客户由于机房断电,导致数据库异常.现场工程师进行了现场备份,然后尝试强制拉库,结果没有成功,大概报错和操作过程如下:
断电之后启动数据库,数据库报ORA-600 kcratr_nab_less_than_odr故障解决错误,这个是一种非常常见的操作,一般是由于写丢失导致,以前有过很多类似恢复经历:
ORA-600 kcratr_nab_less_than_odr故障解决
 差点被误操作的ORA-600 kcratr_nab_less_than_odr故障
 ORA-600 kcratr_nab_less_than_odr和ORA-600 2662故障处理
 ORA-600 kcratr_nab_less_than_odr和ORA-600 4193故障处理

Tue Aug 18 23:59:19 2026
ALTER DATABASE OPEN
This instance was first to open
Beginning crash recovery of 2 threads
 parallel recovery started with 19 processes
Started redo scan
Completed redo scan
 read 21634 KB redo, 800 data blocks need recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_275950.trc  (incident=703037):
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6916], [3], [4], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_703037/orcl2_ora_275950_i703037.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Abort recovery for domain 0
Aborting crash recovery due to error 600
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_275950.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6916], [3], [4], [], [], [], []
Abort recovery for domain 0
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_275950.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6916], [3], [4], [], [], [], []
ORA-600 signalled during: ALTER DATABASE OPEN...

现场恢复人员上来之后,直接尝试做强制resetlogs操作,数据库报ORA-600 ORA-600 krsi_al_hdr_update.15错误,主要是由于redo写丢失导致无法resetlogs成功,具体参考：Alter Database Open Resetlogs returns error ORA-00600: [krsi_al_hdr_update.15], (Doc ID 2026541.1) Oracle断电故障处理

Wed Aug 19 00:28:47 2026
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 29721342127
Archived Log entry 38935 added for thread 1 sequence 6916 ID 0xceea62af dest 1:
ARCH: All Archive destinations made inactive due to error 742
ARCH: Closing local archive destination LOG_ARCHIVE_DEST_1: '+ARCHDG/2_19951_1224781365.arc' (error 742)(orcl2)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_287355.trc  (incident=727038):
ORA-00600: internal error code, arguments: [krsi_al_hdr_update.15], [4294967295], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_727038/orcl2_ora_287355_i727038.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Master archival failure: 600
Archive all online redo logfiles failed:600
ORA-600 signalled during: alter database open resetlogs...

通过 ALTER DATABASE RECOVER database using backup controlfile until cancel之后,继续尝试强制打开库，报ORA-600 2662错误.

Wed Aug 19 00:40:25 2026
Checker run found 32 new persistent data failures
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 29721342127
Archived Log entry 38936 added for thread 1 sequence 6915 ID 0xceea62af dest 1:
Archived Log entry 38937 added for thread 1 sequence 6916 ID 0xceea62af dest 1:
Archived Log entry 38938 added for thread 2 sequence 19951 ID 0xceea62af dest 1:
Archived Log entry 38939 added for thread 2 sequence 19950 ID 0xceea62af dest 1:
Clearing online redo logfile 1 +DATADG/orcl/onlinelog/group_1.319.1224781365
Clearing online log 1 of thread 1 sequence number 6915
Wed Aug 19 00:40:34 2026
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 +DATADG/orcl/onlinelog/group_2.320.1224781367
Clearing online log 2 of thread 1 sequence number 6916
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 +DATADG/orcl/onlinelog/group_3.323.1224781463
Clearing online log 3 of thread 2 sequence number 19951
Clearing online redo logfile 3 complete
Clearing online redo logfile 4 +DATADG/orcl/onlinelog/group_4.324.1224781465
Clearing online log 4 of thread 2 sequence number 19950
Clearing online redo logfile 4 complete
Resetting resetlogs activation ID 3471467183 (0xceea62af)
Online log +DATADG/orcl/onlinelog/group_1.319.1224781365: Thread 1 Group 1 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_1.4574.1224781367: Thread 1 Group 1 was previously cleared
Online log +DATADG/orcl/onlinelog/group_2.320.1224781367: Thread 1 Group 2 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_2.10729.1224781369: Thread 1 Group 2 was previously cleared
Online log +DATADG/orcl/onlinelog/group_3.323.1224781463: Thread 2 Group 3 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_3.12866.1224781463: Thread 2 Group 3 was previously cleared
Online log +DATADG/orcl/onlinelog/group_4.324.1224781465: Thread 2 Group 4 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_4.7527.1224781465: Thread 2 Group 4 was previously cleared
Wed Aug 19 00:40:43 2026
Setting recovery target incarnation to 3
Wed Aug 19 00:40:43 2026
Assigning activation ID 3488382552 (0xcfec7e58)
Thread 2 opened at log sequence 1
  Current log# 3 seq# 1 mem# 0: +DATADG/orcl/onlinelog/group_3.323.1224781463
  Current log# 3 seq# 1 mem# 1: +ARCHDG/orcl/onlinelog/group_3.12866.1224781463
Successful open of redo thread 2
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Aug 19 00:40:43 2026
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_291405.trc  (incident=739007):
ORA-00600: internal error code, arguments: [2662], [6], [3951555809], [6], [3951556555], [12583040], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_739007/orcl2_ora_291405_i739007.trc
Wed Aug 19 00:40:45 2026
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_291405.trc:
ORA-00600: internal error code, arguments: [2662], [6], [3951555809], [6], [3951556555], [12583040], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_291405.trc:
ORA-00600: internal error code, arguments: [2662], [6], [3951555809], [6], [3951556555], [12583040], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 291405): terminating the instance due to error 600
Instance terminated by USER, pid = 291405
ORA-1092 signalled during: alter database open resetlogs...

到这一步,现场停止了继续尝试,我接手故障处理.先dbv检测坏块,发现system有两个坏块

[oracle@db3 ~]$ dbv userid=sys/oracle file=/datapool/cold_backup_20260818/SYSTEM.314.1224781265

DBVERIFY: Release 11.2.0.4.0 - Production on Sat Aug 22 10:39:26 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /datapool/cold_backup_20260818/SYSTEM.314.1224781265
Page 94587 is marked corrupt
Corrupt block relative dba: 0x0041717b (file 1, block 94587)
Bad header found during dbv: 
Data in bad block:
 type: 11 format: 2 rdba: 0x00400001
 last change scn: 0x0000.00000000 seq: 0x1 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x00000b01
 check value in block header: 0xd49
 computed block checksum: 0x0

Page 95021 is marked corrupt
Corrupt block relative dba: 0x0041732d (file 1, block 95021)
Bad header found during dbv: 
Data in bad block:
 type: 11 format: 2 rdba: 0x00400001
 last change scn: 0x0000.00000000 seq: 0x1 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x00000b01
 check value in block header: 0xd49
 computed block checksum: 0x0



DBVERIFY - Verification complete

Total Pages Examined         : 157440
Total Pages Processed (Data) : 72079
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 21854
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 49484
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 14021
Total Pages Marked Corrupt   : 2
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 0 (0.0)

OBET> dbv file 1

===============================================
DBV (Data Block Verification)
Block Size: 8192 bytes
Endian:     little-endian (x86)
Target File: #1(only)
===============================================

Verifying file #1: /datapool/cold_backup_20260818/SYSTEM.314.1224781265 (157441 blocks) - Started: 2026-08-22 11:14:48
File #32: rfile=1 (0x00000001)  header_block_num=157440 (0x00026700)  filesize_status:OK
  Progress: 100000 / 157441 blocks checked...
  File #1completed: 0 all zero, 0 soft corrupted, 2 tailchk error, 0 checksum error, 0 rdba error

DBV completed at: 2026-08-22 11:14:59

===============================================
DBV Summary:
Total blocks checked: 157439
Total all zero blocks found: 0
Total all rdba error blocks found: 0
Total all tailchk error blocks found: 2
Total all soft corrupted blocks found: 0
Total all checksum error blocks found: 0
Total bad blocks found: 2
Execution time: 11.00 seconds
Throughput: 111.82 MB/s
===============================================

Detailed report saved to: dbv_file_32_20260822111448.log
Files processed: 1

OBET> list corrupt
file 1(/datapool/cold_backup_20260818/SYSTEM.314.1224781265) total bad blocks: 2
block#   bad block type
94587   tailchk
95021   tailchk

使用obet修复坏块
Oracle Block Editor Tool使用手册

OBET> set file 32
filename set to: /datapool/cold_backup_20260818/SYSTEM.314.1224781265 (file#1)

OBET> set block 94587
block set to: 94587

OBET> set mode edit
mode set to: edit

OBET> repair block
Warning: Missing value for 'block', using global setting: 94587

Repairing block 94587 in file /datapool/cold_backup_20260818/SYSTEM.314.1224781265...

Repair analysis for block 94587:
1. seq_kcbh check: 0x01 -> OK
2. Tailchk check: 0x0106C0E4 -> needs repair (0xE4C00601)
3. Checksum check: 0x5EC5 -> needs repair (0x7DE6)

Confirm repair operations:
File: /datapool/cold_backup_20260818/SYSTEM.314.1224781265
Block: 94587
Operations needed: fix tailchk, fix checksum
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x0106C0E4 OK
3. Checksum: 0x7DE6 OK

Block 94587 repair completed successfully.

OBET> set block 95021
block set to: 95021

OBET> repair block 
Warning: Missing value for 'block', using global setting: 95021

Repairing block 95021 in file /datapool/cold_backup_20260818/SYSTEM.314.1224781265...

Repair analysis for block 95021:
1. seq_kcbh check: 0x01 -> OK
2. Tailchk check: 0xF8FA0601 -> needs repair (0xE06C0601)
3. Checksum check: 0x81BA -> OK

Confirm repair operations:
File: /datapool/cold_backup_20260818/SYSTEM.314.1224781265
Block: 95021
Operations needed: fix tailchk
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x01066CE0 OK
3. Checksum: 0x81BA OK

Block 95021 repair completed successfully.

dbv检查确认坏块修复成功

[oracle@db3 ~]$ dbv file=/datapool/cold_backup_20260818/SYSTEM.314.1224781265

DBVERIFY: Release 11.2.0.4.0 - Production on Sat Aug 22 11:17:30 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /datapool/cold_backup_20260818/SYSTEM.314.1224781265


DBVERIFY - Verification complete

Total Pages Examined         : 157440
Total Pages Processed (Data) : 72080
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 21855
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 49484
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 14021
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3951556890 (6.3951556890)

后面的恢复比较简单,使用客户恢复之前的备份,直接重建ctl,然后打开库成功,并且做rman校验没有异常,直接把恢复之后的库备份还原到asm里面,完成本次恢复任务

通过obet 恢复system坏块,打开数据库

Posted on 2026 年 08 月 22 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：通过obet 恢复system坏块,打开数据库

客户由于误操作直接在虚拟化平台点击电源键,强制关闭了正在运行的数据库服务器虚机,导致数据库无法正常启动,检查发现ntfs文件系统损坏

通过obet工具检测数据文件坏块(obet dbv功能完整说明),发现核心的system文件上面有一些坏块

File #1: D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF (225281 blocks) - Started: 2026-08-21 16:40:23
File #1: rfile=1 (0x00000001)  header_block_num=225280 (0x00037000)  filesize_status:OK
file 1, block 212238: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 213507: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 213515: tailchk error (expected 0x01065E92, got 0x0106276F), bad block
file 1, block 213547: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 213555: tailchk error (expected 0x01065E92, got 0x0106276F), bad block
file 1, block 221387: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 221403: tailchk error (expected 0x01065E92, got 0x0106B19C), bad block
file 1, block 221419: tailchk error (expected 0x01065E92, got 0x0106C7E3), bad block
file 1, block 222117: tailchk error (expected 0x01065E92, got 0x0106276F), bad block
file 1, block 222133: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 222141: tailchk error (expected 0x01065E92, got 0x0106276F), bad block
file 1, block 222173: tailchk error (expected 0x0106C7E3, got 0x01065E92), bad block
  File #1 completed: 0 all zero, 0 soft corrupted, 12 tailchk error, 0 checksum error, 0 rdba error

这个12个坏块的数量和dbv检测结果一致


DBVERIFY: Release 19.0.0.0.0 - Production on 星期五 8月 21 17:37:02 2026

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.


DBVERIFY - 开始验证: FILE = D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF
页 212238 流入 - 很可能是介质损坏
Corrupt block relative dba: 0x00433d0e (file 1, block 212238)
Fractured block found during dbv: 
Data in bad block:
 type: 6 format: 2 rdba: 0x00433d0e
 last change scn: 0x0000.0002.d71a6f27 seq: 0x1 flg: 0x06
 spare3: 0x0
 consistency value in tail: 0x925e0601
 check value in block header: 0x93bb
 computed block checksum: 0xfd79

…………

页 222173 流入 - 很可能是介质损坏
Corrupt block relative dba: 0x004363dd (file 1, block 222173)
Fractured block found during dbv: 
Data in bad block:
 type: 6 format: 2 rdba: 0x004363dd
 last change scn: 0x0000.0002.d708e3c7 seq: 0x1 flg: 0x06
 spare3: 0x0
 consistency value in tail: 0x925e0601
 check value in block header: 0xe089
 computed block checksum: 0x7199


DBVERIFY - 验证完成

检查的页总数: 225280
处理的页总数 (数据): 125451
失败的页总数 (数据): 0
处理的页总数 (索引): 29984
失败的页总数 (索引): 0
处理的页总数 (其他): 48214
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 21619
标记为损坏的总页数: 12
流入的页总数: 12
加密的总页数        : 0
最高块 SCN            : 12199365950 (2.3609431358)

使用obet的reair block功能进行修复(obet repair block使用说明)

OBET> set file 1
filename set to: D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF (file#1)

OBET> set block 212238
block set to: 212238

OBET> d

File: D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF
Block: 212238                Offsets:     0 to    31
--------------------------------------------------------------------------------
67A1C000 06A20000 0E3D4300 276F1AD7 02000106 BB930000 02000000 D0020000 E46E1AD7

<32 bytes read>

OBET> tailchk
Check tailchk for File D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF, Block 212238:
current = 0x925E0601, required = 0x6F270601

OBET> backup
Backing up file #1, block 212238
Successfully backed up block 212238 from D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF 
   to backup_blk\SYSTEM01.DBF.212238_20260821173825.blk

OBET> set mode edit
mode set to: edit

OBET> repair
Usage: repair [subcommand]
  repair block [file N] [X]  - Repair tailchk/checksum (optional: file N, block X)
  repair blkscn [file N] [X] - Repair block SCN (optional: file N, block X)

OBET> repair block
Warning: Missing value for 'block', using global setting: 212238

Repairing block 212238 in file D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF...

Repair analysis for block 212238:
1. seq_kcbh check: 0x01 -> OK
2. Tailchk check: 0x925E0601 -> needs repair (0x6F270601)
3. Checksum check: 0xBB93 -> OK

Confirm repair operations:
File: D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF
Block: 212238
Operations needed: fix tailchk
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x6F270601 OK
3. Checksum: 0xBB93 OK

Block 212238 repair completed successfully.

所有坏块依次进行修复,然后再次使用dbv检测


DBVERIFY: Release 19.0.0.0.0 - Production on 星期五 8月 21 17:42:02 2026

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.


DBVERIFY - 开始验证: FILE = D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF
Block Checking: DBA = 4407811, Block Type = KTB-managed data block
**** actual rows locked by itl 2  = 0 != # in trans. header = 1
**** actual rows locked by itl 3  = 0 != # in trans. header = 1
---- end index block validation
页 213507 失败, 校验代码为 6401
Block Checking: DBA = 4407819, Block Type = KTB-managed data block
**** row 120: key out of order
**** actual rows locked by itl 2  = 0 != # in trans. header = 1
**** actual rows marked deleted = 1 != kdxlende = 0
---- end index block validation
页 213515 失败, 校验代码为 6401

DBVERIFY - 验证完成

检查的页总数: 225280
处理的页总数 (数据): 125451
失败的页总数 (数据): 0
处理的页总数 (索引): 29996
失败的页总数 (索引): 2
处理的页总数 (其他): 48214
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 21619
标记为损坏的总页数: 0
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 12199365950 (2.3609431358)

有两个block有少量逻辑错误,可以通过数据库级别设置进行跳过,基本上实现了这个12个坏块的自动修复.后续就是数据库的打开过程

SQL> startup mount pfile='d:/pfile.txt'
ORACLE 例程已经启动。

Total System Global Area 1.2885E+10 bytes
Fixed Size                 16120656 bytes
Variable Size            7751073792 bytes
Database Buffers         5100273664 bytes
Redo Buffers               17432576 bytes
数据库装载完毕。
SQL> recover database;
ORA-00283: 恢复会话因错误而取消
ORA-00742: 日志读取在线程 1 序列 19828 块 11511 中检测到写入丢失情况
ORA-00312: 联机日志 2 线程 1: 'D:\APP\ADMINISTRATOR\ORADATA\NHIS\REDO02.LOG'

SQL> RECOVER DATABASE UNTIL TIME '2026-08-21:13:28:56' USING BACKUP CONTROLFILE;
ORA-00279: change 12198905983 generated at 08/21/2026 12:43:18 needed for
thread 1
ORA-00289: suggestion :
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000019827_1199133733.0001
ORA-00280: change 12198905983 for thread 1 is in sequence #19827


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ADMINISTRATOR\ORADATA\NHIS\REDO01.LOG
ORA-00279: change 12199301614 generated at 08/21/2026 13:27:06 needed for
thread 1
ORA-00289: suggestion :
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000019828_1199133733.0001
ORA-00280: change 12199301614 for thread 1 is in sequence #19828
ORA-00278: log file 'D:\APP\ADMINISTRATOR\ORADATA\NHIS\REDO01.LOG' no longer
needed for this recovery


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ADMINISTRATOR\ORADATA\NHIS\REDO02.LOG
ORA-00756: 鎭㈠鎿嶄綔妫€娴嬪埌鏁版嵁鍧楀啓鍏ヤ涪澶?ORA-10567:
Redo is inconsistent with data block (file# 4, block# 31663, file offset is 259383296 bytes)
ORA-10564: tablespace UNDOTBS1
ORA-01110: 鏁版嵁鏂囦欢 4: 'D:\APP\ADMINISTRATOR\ORADATA\NHIS\UNDOTBS01.DBF'
ORA-10560: block type 'KTU UNDO BLOCK'

ORA-01112: media recovery not started

SQL> alter database open resetlogs;

Database altered.

使用expdp导出数据,完成本次恢复任务

分享一例运行在aix上的sap系统数据库恢复过程

Posted on 2026 年 08 月 14 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：分享一例运行在aix上的sap系统数据库恢复过程

情况描述
客户sap系统运行在aix系统,ibm v7000存储上,数据存放在三个1T的lun组成的vg的多个lv中.异常断电之后,但是给数据库使用的三个lun丢失了2个,从而使得所有vg/lv异常,通过硬件恢复出来异常的2个lun和剩余的1个lun一起,把所有的数据文件恢复出来.但是由于某种原因,出现部分block被覆盖(其中还包括两个文件头损坏).

坏块检测
对于恢复的所有文件,为了快速做一遍坏块检查,直接在恢复的win机器上使用过obet做了一次dbv检查obet实现对数据文件坏块检测功能(使用obet的dbv检查有几个好处:1>可以在win上面检测aix的数据文件;2>可以检测数据文件头损坏的数据文件其他block;3>检测速度比原生dbv快[每个文件内部加了并行检测]),检测结果如下

--其中两个文件头损坏
File #62: E:\sr3_54\sr3.data54 (1280000 blocks) - Started: 2026-07-28 23:57:53
File #62: rfile=0 (0x00000000)  header_block_num=0 (0x00000000)  filesize_status:NO
file 62, block 0: block all zero
file 62, block 1: block all zero

File #84: E:\sr3_76\sr3.data76 (4185601 blocks) - Started: 2026-07-29 00:55:32
File #84: rfile=638937491 (0x26156993)  header_block_num=77152556 (0x0499412C)  filesize_status:NO
file 84, block 1: rdba error (expected 1, got 69760), bad block
file 84, block 2: rdba error (expected 2, got 589836), bad block

--坏块汇总
DBV completed at: 2026-07-29 00:58:10
===============================================
DBV Summary:
Total blocks checked: 196287237
Total all zero blocks found: 337417
Total all rdba error blocks found: 1062468
Total all tailchk error blocks found: 1
Total all soft corrupted blocks found: 0
Total all checksum error blocks found: 0
Total bad blocks found: 1399886
Execution time: 10507.00 seconds
===============================================

这个统计下来好的block在99.3%左右,证明硬件层面的会效果已经非常好.

碎片工具进一步恢复
恢复公司文件系统层面恢复有数据块遗漏的可能,通过碎片工具(OraScan(Oracle 碎片扫描工具) 使用说明)进一步扫描

通过确认62号文件还有少量block可以进一步恢复（也就是说碎片扫描到的62号文件的block多于硬件公司恢复出来的62号文件里面好的block数量）,通过obet的merge功能进行填补

基于上述操作,对于lun里面的数据文件实现了最大效果恢复.

数据库恢复操作
1. 上次恢复文件到aix,offline掉异常文件头数据文件打开数据库

sapprd2:oraprd 8> sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Aug 3 18:41:28 2026

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select status from v$instance;

STATUS
------------------------------------
STARTED

SQL> alter database mount;
alter database mount
*
ERROR at line 1:
ORA-00214: control file '/oracle/PRD/origlogA/cntrl/cntrlPRD.dbf' version
9742943 inconsistent with file '/oracle/PRD/sapdata1/cntrl/cntrlPRD.dbf'
version 9742931

解决ctl不一致问题之后继续mount库恢复

sapprd2:oraprd 11> sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Aug 3 18:42:53 2026

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> alter database mount;

Database altered.

SQL> alter database datafile 62,84 offline;

Database altered.

SQL> recover database ;
Media recovery complete.
SQL> alter database open;

Database altered.

2.对于两个损坏的数据文件头进行修复
由于现在aix环境的客户比较少,obet没有在aix环境下进行编译,因此直接使用bbed工具进行文件头修复（列举了主要操作过程）

BBED> copy file 83 block 1 to file 84 block 1
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:    0 to   31           Dba:0x15000001
------------------------------------------------------------------------
 0ba20000 14c00001 00000000 00000104 22250000 00000000 0b200000 72bc991d

 <32 bytes per line>

BBED> set offset 368
        OFFSET          368

BBED> d
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:  368 to  399           Dba:0x15000001
------------------------------------------------------------------------
 00000053 00000000 00000000 495dc097 00000000 00000000 00000000 00000000

 <32 bytes per line>

BBED> m /x 00000054
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:  368 to  399           Dba:0x15000001
------------------------------------------------------------------------
 00000054 00000000 00000000 495dc097 00000000 00000000 00000000 00000000

 <32 bytes per line>

BBED> set offset 52
        OFFSET          52

BBED> d
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:   52 to   83           Dba:0x15000001
------------------------------------------------------------------------
 00530003 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 <32 bytes per line>

BBED> m /x 0054
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:   52 to   83           Dba:0x15000001
------------------------------------------------------------------------
 00540003 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 <32 bytes per line>

BBED> set offset 4
        OFFSET          4

BBED> d
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:    4 to   35           Dba:0x15000001
------------------------------------------------------------------------
 14c00001 00000000 00000104 22250000 00000000 0b200000 72bc991d 50524400

 <32 bytes per line>

BBED> m /x 15000001
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:    4 to   35           Dba:0x15000001
------------------------------------------------------------------------
 15000001 00000000 00000104 22250000 00000000 0b200000 72bc991d 50524400

 <32 bytes per line>


BBED> m /x E1B0
Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:  100 to  611           Dba:0x15000001
------------------------------------------------------------------------
 e1b0853f 00040000 4083fcfe 33606b63 02502452 0000ca28 49680fb3 93b9f6b9

 <32 bytes per line>

BBED> set offset +2
        OFFSET          102

BBED> m /x 8542
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:  102 to  613           Dba:0x15000001
------------------------------------------------------------------------
 85420004 00004083 fcfe3360 6b630250 24520000 ca284968 0fb393b9 f6b90005

 <32 bytes per line>


BBED> sum
Check value for File 84, Block 1:
current = 0x2225, required = 0x23e5

BBED> sum apply
Check value for File 84, Block 1:
current = 0x23e5, required = 0x23e5

BBED> verify
DBVERIFY - Verification starting
FILE = /oracle/PRD/sapdata3/sr3_76/sr3.data76
BLOCK = 1


DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 0
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED

修改完成之后,还出现过几个错误

SQL> alter database open ;
alter database open
*
ERROR at line 1:
ORA-01122: database file 62 failed verification check
ORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54'
ORA-01200: actual file size of 1279999 is smaller than correct size of 1280000

ORA-01200是由于数据文件比文件头记录信息小一个block,通过补上这个block解决

Read of datafile '/oracle/PRD/sapdata2/sr3_54/sr3.data54' (fno 62) header failed with ORA-01202
Rereading datafile 62 header failed with ORA-01202
Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_11272278.trc:
ORA-01122: database file 62 failed verification check
ORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54'
ORA-01202: wrong incarnation of this file - wrong creation time

ORA-01202是由于create time没有修改正确导致,重新修改解决

Rereading datafile 84 header failed with ORA-01203
Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_17105252.trc:
ORA-01122: database file 84 failed verification check
ORA-01110: data file 84: '/oracle/PRD/sapdata3/sr3_76/sr3.data76'
ORA-01203: wrong incarnation of this file - wrong creation SCN

ORA-01203是由于create scn没有修改正确导致,重新修改解决

Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_17105278.trc:
ORA-01122: database file 62 failed verification check
ORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54'
ORA-01207: file is more recent than control file - old control file
ORA-1122 signalled during: alter database open .

ORA-01207是由于数据文件的ckp信息比控制文件的新,重建ctl解决,解决这些问题之后,顺利打开数据库

SQL> startup mount;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 3.7548E+10 bytes
Fixed Size                  2254136 bytes
Variable Size            1.9193E+10 bytes
Database Buffers         1.8254E+10 bytes
Redo Buffers               98996224 bytes
Database mounted.
SQL> alter database open;

Database altered.

SQL> select status,count(1) from v$datafile_header group by status;

STATUS    COUNT(1)
------- ----------
ONLINE          84

然后跳过坏块,导出数据,对于无法导出的异常表进行特殊处理,完成本次恢复工作,最终恢复结果总结