联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
情况描述
客户sap系统运行在aix系统,ibm v7000存储上,数据存放在三个1T的lun组成的vg的多个lv中.异常断电之后,但是给数据库使用的三个lun丢失了2个,从而使得所有vg/lv异常,通过硬件恢复出来异常的2个lun和剩余的1个lun一起,把所有的数据文件恢复出来.但是由于某种原因,出现部分block被覆盖(其中还包括两个文件头损坏).
坏块检测
对于恢复的所有文件,为了快速做一遍坏块检查,直接在恢复的win机器上使用过obet做了一次dbv检查obet实现对数据文件坏块检测功能(使用obet的dbv检查有几个好处:1>可以在win上面检测aix的数据文件;2>可以检测数据文件头损坏的数据文件其他block;3>检测速度比原生dbv快[每个文件内部加了并行检测]),检测结果如下
--其中两个文件头损坏 File #62: E:\sr3_54\sr3.data54 (1280000 blocks) - Started: 2026-07-28 23:57:53 File #62: rfile=0 (0x00000000) header_block_num=0 (0x00000000) filesize_status:NO file 62, block 0: block all zero file 62, block 1: block all zero File #84: E:\sr3_76\sr3.data76 (4185601 blocks) - Started: 2026-07-29 00:55:32 File #84: rfile=638937491 (0x26156993) header_block_num=77152556 (0x0499412C) filesize_status:NO file 84, block 1: rdba error (expected 1, got 69760), bad block file 84, block 2: rdba error (expected 2, got 589836), bad block --坏块汇总 DBV completed at: 2026-07-29 00:58:10 =============================================== DBV Summary: Total blocks checked: 196287237 Total all zero blocks found: 337417 Total all rdba error blocks found: 1062468 Total all tailchk error blocks found: 1 Total all soft corrupted blocks found: 0 Total all checksum error blocks found: 0 Total bad blocks found: 1399886 Execution time: 10507.00 seconds ===============================================
这个统计下来好的block在99.3%左右,证明硬件层面的会效果已经非常好.
碎片工具进一步恢复
恢复公司文件系统层面恢复有数据块遗漏的可能,通过碎片工具(OraScan(Oracle 碎片扫描工具) 使用说明)进一步扫描
通过确认62号文件还有少量block可以进一步恢复（也就是说碎片扫描到的62号文件的block多于硬件公司恢复出来的62号文件里面好的block数量）,通过obet的merge功能进行填补
基于上述操作,对于lun里面的数据文件实现了最大效果恢复.
数据库恢复操作
1. 上次恢复文件到aix,offline掉异常文件头数据文件打开数据库
sapprd2:oraprd 8> sqlplus / as sysdba SQL*Plus: Release 11.2.0.4.0 Production on Mon Aug 3 18:41:28 2026 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL> select status from v$instance; STATUS ------------------------------------ STARTED SQL> alter database mount; alter database mount * ERROR at line 1: ORA-00214: control file '/oracle/PRD/origlogA/cntrl/cntrlPRD.dbf' version 9742943 inconsistent with file '/oracle/PRD/sapdata1/cntrl/cntrlPRD.dbf' version 9742931
解决ctl不一致问题之后继续mount库恢复
sapprd2:oraprd 11> sqlplus / as sysdba SQL*Plus: Release 11.2.0.4.0 Production on Mon Aug 3 18:42:53 2026 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL> alter database mount; Database altered. SQL> alter database datafile 62,84 offline; Database altered. SQL> recover database ; Media recovery complete. SQL> alter database open; Database altered.
2.对于两个损坏的数据文件头进行修复
由于现在aix环境的客户比较少,obet没有在aix环境下进行编译,因此直接使用bbed工具进行文件头修复（列举了主要操作过程）
BBED> copy file 83 block 1 to file 84 block 1
File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
Block: 1 Offsets: 0 to 31 Dba:0x15000001
------------------------------------------------------------------------
0ba20000 14c00001 00000000 00000104 22250000 00000000 0b200000 72bc991d
<32 bytes per line>
BBED> set offset 368
OFFSET 368
BBED> d
File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
Block: 1 Offsets: 368 to 399 Dba:0x15000001
------------------------------------------------------------------------
00000053 00000000 00000000 495dc097 00000000 00000000 00000000 00000000
<32 bytes per line>
BBED> m /x 00000054
File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
Block: 1 Offsets: 368 to 399 Dba:0x15000001
------------------------------------------------------------------------
00000054 00000000 00000000 495dc097 00000000 00000000 00000000 00000000
<32 bytes per line>
BBED> set offset 52
OFFSET 52
BBED> d
File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
Block: 1 Offsets: 52 to 83 Dba:0x15000001
------------------------------------------------------------------------
00530003 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<32 bytes per line>
BBED> m /x 0054
File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
Block: 1 Offsets: 52 to 83 Dba:0x15000001
------------------------------------------------------------------------
00540003 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<32 bytes per line>
BBED> set offset 4
OFFSET 4
BBED> d
File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
Block: 1 Offsets: 4 to 35 Dba:0x15000001
------------------------------------------------------------------------
14c00001 00000000 00000104 22250000 00000000 0b200000 72bc991d 50524400
<32 bytes per line>
BBED> m /x 15000001
File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
Block: 1 Offsets: 4 to 35 Dba:0x15000001
------------------------------------------------------------------------
15000001 00000000 00000104 22250000 00000000 0b200000 72bc991d 50524400
<32 bytes per line>
BBED> m /x E1B0
Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y
File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
Block: 1 Offsets: 100 to 611 Dba:0x15000001
------------------------------------------------------------------------
e1b0853f 00040000 4083fcfe 33606b63 02502452 0000ca28 49680fb3 93b9f6b9
<32 bytes per line>
BBED> set offset +2
OFFSET 102
BBED> m /x 8542
File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
Block: 1 Offsets: 102 to 613 Dba:0x15000001
------------------------------------------------------------------------
85420004 00004083 fcfe3360 6b630250 24520000 ca284968 0fb393b9 f6b90005
<32 bytes per line>
BBED> sum
Check value for File 84, Block 1:
current = 0x2225, required = 0x23e5
BBED> sum apply
Check value for File 84, Block 1:
current = 0x23e5, required = 0x23e5
BBED> verify
DBVERIFY - Verification starting
FILE = /oracle/PRD/sapdata3/sr3_76/sr3.data76
BLOCK = 1
DBVERIFY - Verification complete
Total Blocks Examined : 1
Total Blocks Processed (Data) : 0
Total Blocks Failing (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing (Index): 0
Total Blocks Empty : 0
Total Blocks Marked Corrupt : 0
Total Blocks Influx : 0
Message 531 not found; product=RDBMS; facility=BBED
修改完成之后,还出现过几个错误
SQL> alter database open ; alter database open * ERROR at line 1: ORA-01122: database file 62 failed verification check ORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54' ORA-01200: actual file size of 1279999 is smaller than correct size of 1280000
ORA-01200是由于数据文件比文件头记录信息小一个block,通过补上这个block解决
Read of datafile '/oracle/PRD/sapdata2/sr3_54/sr3.data54' (fno 62) header failed with ORA-01202 Rereading datafile 62 header failed with ORA-01202 Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_11272278.trc: ORA-01122: database file 62 failed verification check ORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54' ORA-01202: wrong incarnation of this file - wrong creation time
ORA-01202是由于create time没有修改正确导致,重新修改解决
Rereading datafile 84 header failed with ORA-01203 Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_17105252.trc: ORA-01122: database file 84 failed verification check ORA-01110: data file 84: '/oracle/PRD/sapdata3/sr3_76/sr3.data76' ORA-01203: wrong incarnation of this file - wrong creation SCN
ORA-01203是由于create scn没有修改正确导致,重新修改解决
Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_17105278.trc: ORA-01122: database file 62 failed verification check ORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54' ORA-01207: file is more recent than control file - old control file ORA-1122 signalled during: alter database open .
ORA-01207是由于数据文件的ckp信息比控制文件的新,重建ctl解决,解决这些问题之后,顺利打开数据库
SQL> startup mount; ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance ORACLE instance started. Total System Global Area 3.7548E+10 bytes Fixed Size 2254136 bytes Variable Size 1.9193E+10 bytes Database Buffers 1.8254E+10 bytes Redo Buffers 98996224 bytes Database mounted. SQL> alter database open; Database altered. SQL> select status,count(1) from v$datafile_header group by status; STATUS COUNT(1) ------- ---------- ONLINE 84
然后跳过坏块,导出数据,对于无法导出的异常表进行特殊处理,完成本次恢复工作,最终恢复结果总结
