记录一次比较棘手数据库恢复要点

在最近的一次数据库异常恢复过程中遇到不少问题,把重点记录下
ORA-00704/ORA-01555错误

Fri May  4 21:04:21 2012
select ctime, mtime, stime from obj$ where obj# = :1
Fri May  4 21:04:21 2012
Errors in file /oracle/admin/standdb/udump/perfdb_ora_1286288.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 40 with name "_SYSSMU40$" too small
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Instance terminated by USER, pid = 1286288
ORA-1092 signalled during: alter database open resetlogs...

这里的提示可以看出obj$基表中有事务存在,查询这个表的时候,要去找40号回滚段中相关数据;通过非常规方法,
查找到40号回滚段的状态是offliine了(这个查询出来的信息和是否使用隐含参数无关).

问题原因,为什么40号回滚段变得offline?
Fri May  4 17:36:26 2012
alter tablespace undotbs offline
Fri May  4 17:36:26 2012
ORA-1109 signalled during: alter tablespace undotbs offline...
Fri May  4 17:37:29 2012
alter database datafile  '/dev/rundodbs01' offline drop
Fri May  4 17:37:29 2012
Completed: alter database datafile  '/dev/rundodbs01' offline drop
因为强制offline 了file# 2文件导致(一个undo表空间文件)

解决方法:
1.bbed提交事务
因为现在生产的trace文件中未有关于obj$ 未提交事务的记录,做10046也为发现该记录,如果要使用bbed修改该事务,
那需要dump obj$相关的数据块(在mount状态下dump),然后找到相关事务,再修改

2.强制让file# 2 online
因为在resetlogs前file#2 已经offline掉了,所以要使得该文件能够成功online,需要先推进scn

ORA-00600[krhpfh_03-1209]

SQL> recover database until cancel;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [krhpfh_03-1209], [2], [782415504],
[782428968], [3987078030], [2379], [0], [0]
ORA-01110: data file 2: '/dev/rundodbs01'

问题原因:
数据库处于非归档模式下,连续三次resetlogs,引起该bug

解决办法:
重建控制文件

但是这里问题出现了,因为file# 2的resetlogs scn和其他数据文件不一致,导致在file# 2 online的前提下,无法重建.
这样就处在了一个循环中(需要online file# 2 又要重建控制文件),这样的问题,可以通过bbed修改file# 2的resetlogs scn完成
或者先让file# 2 offline(没有加drop)掉,重建控制文件(除掉file# 2的文件记录)

ORA-00600[25025]

SMON: enabling cache recovery
Fri May  4 22:36:36 2012
Errors in file /oracle/admin/standdb/udump/perfdb_ora_1167402.trc:
ORA-00600: internal error code, arguments: [25025], [2], [], [], [], [], [], []
Fri May  4 22:36:38 2012
Errors in file /oracle/admin/standdb/udump/perfdb_ora_1167402.trc:
ORA-00600: internal error code, arguments: [25025], [2], [], [], [], [], [], []
Fri May  4 22:36:38 2012
Error 600 happened during db open, shutting down database
USER: terminating instance due to error 600
Instance terminated by USER, pid = 1167402

错误原因:
因为有undo文件不在undo对应的表空间中,而我们的file# 2文件确实是undo文件,而且重建控制文件时候未加入进来

解决办法:
  undo_management          = AUTO
  undo_tablespace          = UNDODBS(file# 2属于该表空间)
修改为
  undo_management          = MANUAL
  undo_tablespace          = SYSTEM
或者bbed修改file# 2的header,然后重建控制文件

ORA-00600[4137]

Errors in file /oracle/admin/standdb/bdump/perfdb_smon_1290564.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
Fri May  4 23:20:52 2012
create undo tablespace undotbs3 datafile '/dev/rundodbs21' size 20400M
Fri May  4 23:23:47 2012
Errors in file /oracle/admin/standdb/bdump/perfdb_smon_1290564.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
Fri May  4 23:23:48 2012
Errors in file /oracle/admin/standdb/bdump/perfdb_pmon_1520126.trc:
ORA-00474: SMON process terminated with error
Fri May  4 23:23:48 2012
PMON: terminating instance due to error 474
Instance terminated by PMON, pid = 1520126

错误原因:
_smon_internal_errlimit(limit of SMON internal errors) SMON遇到了内部错误,最大允许100次,
不断计数增长,达到100的时候,数据库smon进程自动down掉,从而导致数据库down

解决办法:
1.临时解决办法:设置_smon_internal_errlimit一个较大值
3.根本解决办法:使用undo隐含参数,删除有问题undo 回滚段和undo表空间或者使用10513 事件

bbed 找回被删除数据

创建模拟表数据

SQL> create table t_xifenfei(id number,name varchar2(10));

Table created.

SQL> insert into t_xifenfei values(1,'xifenfei');

1 row created.

SQL> insert into t_xifenfei values(2,'XIFENFEI');

1 row created.

SQL> commit;

Commit complete.

dump数据块

SQL> alter system flush BUFFER_CACHE;

System altered.

SQL> select   rowid,id,name,
  2  dbms_rowid.rowid_relative_fno(rowid)rel_fno,
  3  dbms_rowid.rowid_block_number(rowid)blockno,
  4  dbms_rowid.rowid_row_number(rowid) rowno
  5  from chf.t_xifenfei;

ROWID                      ID NAME          REL_FNO    BLOCKNO      ROWNO
------------------ ---------- ---------- ---------- ---------- ----------
AAASdmAAEAAAACvAAA          1 xifenfei            4        175          0
AAASdmAAEAAAACvAAB          2 XIFENFEI            4        175          1

SQL> alter system dump datafile 4 block 175;

System altered.

dump文件内容

block_row_dump:
tab 0, row 0, @0x1f89
tl: 15 fb: --H-FL-- lb: 0x1  cc: 2
col  0: [ 2]  c1 02
col  1: [ 8]  78 69 66 65 6e 66 65 69
tab 0, row 1, @0x1f7a
tl: 15 fb: --H-FL-- lb: 0x1  cc: 2
col  0: [ 2]  c1 03
col  1: [ 8]  58 49 46 45 4e 46 45 49
end_of_block_dump
2012-05-01 05:09:29.287714 : kjbmbassert [0xaf.4]
End dump data blocks tsn: 4 file#: 4 minblk 175 maxblk 175

删除表数据

SQL> delete from t_xifenfei;

2 rows deleted.

SQL> commit;

Commit complete.

SQL> alter system flush BUFFER_CACHE;

System altered.

SQL> alter system dump datafile 4 block 175;

System altered.

dump文件内容

block_row_dump:
tab 0, row 0, @0x1f89
tl: 2 fb: --HDFL-- lb: 0x2 
tab 0, row 1, @0x1f7a
tl: 2 fb: --HDFL-- lb: 0x2 
end_of_block_dump
2012-05-01 05:13:35.214357 : kjbmbassert [0xaf.4]
End dump data blocks tsn: 4 file#: 4 minblk 175 maxblk 175

通过对比这两次的dump文件发现

1.数据内容被删除,并不是真正删除,而是给其增加了一个标识位(fd:---D----)
2.fb:--H-FL--(head of row piece+first data piece+last data piece )
  其有8个选项每个选项的值分别对应bitmask即32+8+4=44 or 0x2c
3.如果一个row被delete了,那么row flag就会更新,bitmask里的deleted被设置为16.
  此时row flag为:32+16+8+4 = 60 or 0x3c.
4.如果我们要找回来被删除的数据,只需要把3c改为2c即可

关闭数据库

SQL> select * from chf.t_xifenfei;

no rows selected

SQL> select name from v$datafile where file#=4;

NAME
------------------------------------------------
/tmp/user01.dbf

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

bbed修改数据

BBED> set filename '/tmp/user01.dbf'
        FILENAME        /tmp/user01.dbf

BBED> set block 175
        BLOCK#          175

BBED> set blocksize 8192
        BLOCKSIZE       8192

BBED> set mode edit
        MODE            Edit

BBED> map 
 File: /tmp/user01.dbf (0)
 Block: 175                                   Dba:0x00000000
------------------------------------------------------------
 KTB Data Block (Table/Cluster)

 struct kcbh, 20 bytes                      @0       

 struct ktbbh, 72 bytes                     @20      

 struct kdbh, 14 bytes                      @100     

 struct kdbt[1], 4 bytes                    @114     

 sb2 kdbr[2]                                @118     

 ub1 freespace[8036]                        @122     

 ub1 rowdata[30]                            @8158    

 ub4 tailchk                                @8188    

BBED> p *kdbr[0]
rowdata[15]
-----------
ub1 rowdata[15]                             @8173     0x3c

BBED> p *kdbr[1]
rowdata[0]
----------
ub1 rowdata[0]                              @8158     0x3c

BBED> m /x 2c offset 8158
 File: /tmp/user01.dbf (0)
 Block: 175              Offsets: 8158 to 8191           Dba:0x00000000
------------------------------------------------------------------------
 2c630202 c1030858 4946454e 46454932 630202c1 02087869 66656e66 65690106 
 b47e 

 <32 bytes per line>

BBED>  m /x 2c offset 8173
 File: /tmp/user01.dbf (0)
 Block: 175              Offsets: 8173 to 8191           Dba:0x00000000
------------------------------------------------------------------------
 2c630202 c1020878 6966656e 66656901 06b47e 

 <32 bytes per line>

BBED> sum apply
Check value for File 0, Block 175:
current = 0x4d13, required = 0x4d13

启动数据库验证

SQL> startup 
ORACLE instance started.

Total System Global Area  535662592 bytes
Fixed Size                  1346140 bytes
Variable Size             411043236 bytes
Database Buffers          117440512 bytes
Redo Buffers                5832704 bytes
Database mounted.
Database opened.
SQL> select * from chf.t_xifenfei;

        ID NAME
---------- ----------
         1 xifenfei
         2 XIFENFEI

exp导出数据报EXP-00056/ORA-01403错误

exp导出数据报EXP-00056/ORA-01403错误

[oracle@xifenfei ~]$ exp chf/xifenfei file=/tmp/chf.dmp log=/tmp/chf.log owner=chf

Export: Release 9.2.0.4.0 - Production on Sun Apr 29 03:11:31 2012

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.


Connected to: Oracle9i Enterprise Edition Release 9.2.0.4.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.4.0 - Production
Export done in ZHS16GBK character set and AL16UTF16 NCHAR character set
server uses WE8ISO8859P1 character set (possible charset conversion)

About to export specified users ...
. exporting pre-schema procedural objects and actions
. exporting foreign function library names for user CHF 
. exporting PUBLIC type synonyms
. exporting private type synonyms
. exporting object type definitions for user CHF 
About to export CHF's objects ...
. exporting database links
. exporting sequence numbers
. exporting cluster definitions
EXP-00056: ORACLE error 1403 encountered
ORA-01403: no data found
EXP-00000: Export terminated unsuccessfully

查看组件信息

SQL> select * from v$version;

BANNER
-------------------------------------------------------------------
Oracle9i Enterprise Edition Release 9.2.0.4.0 - Production
PL/SQL Release 9.2.0.4.0 - Production
CORE    9.2.0.3.0       Production
TNS for Linux: Version 9.2.0.4.0 - Production
NLSRTL Version 9.2.0.4.0 - Production

SQL> col comp_id for a15
SQL> col status for a7
SQL> col version for a10
SQL> col comp_name for a30
SQL> set pagesize 1000
SQL> SELECT substr(comp_id,1,15) comp_id, status, substr(version,1,10) version,
  2     substr(comp_name,1,30) comp_name FROM dba_registry ORDER BY 1;

COMP_ID         STATUS  VERSION    COMP_NAME
--------------- ------- ---------- ------------------------------
AMD             VALID   9.2.0.4.0  OLAP Catalog
APS             LOADED  9.2.0.4.0  OLAP Analytic Workspace
CATALOG         VALID   9.2.0.4.0  Oracle9i Catalog Views
CATJAVA         VALID   9.2.0.4.0  Oracle9i Java Packages
CATPROC         VALID   9.2.0.4.0  Oracle9i Packages and Types
CONTEXT         VALID   9.2.0.4.0  Oracle Text
JAVAVM          VALID   9.2.0.4.0  JServer JAVA Virtual Machine
ODM             LOADED  9.2.0.1.0  Oracle Data Mining
ORDIM           VALID   9.2.0.4.0  Oracle interMedia
OWM             VALID   9.2.0.1.0  Oracle Workspace Manager
SDO             LOADED  9.2.0.4.0  Spatial
WK              VALID   9.2.0.4.0  Oracle Ultra Search
XDB             VALID   9.2.0.4.0  Oracle XML Database
XML             VALID   9.2.0.6.0  Oracle XDK for Java
XOQ             LOADED  9.2.0.4.0  Oracle OLAP API

15 rows selected.

SQL> SELECT status, object_id, object_type, owner||'.'||object_name 
  2     "OWNER.OBJECT" FROM dba_objects WHERE owner='XDB' AND status != 'VALID' 
  3     ORDER BY 4,2;

no rows selected

做1403跟踪

SQL> ALTER SYSTEM SET EVENTS '1403 trace name errorstack level 3';

[oracle@xifenfei ~]$ exp chf/xifenfei file=/tmp/chf.dmp log=/tmp/chf.log owner=chf

SQL> ALTER SYSTEM SET EVENTS '1403 trace name errorstack off';

trace文件关键内容

*** SESSION ID:(11.17) 2012-04-29 03:17:13.555
*** 2012-04-29 03:17:13.555
ksedmp: internal or fatal error
ORA-01403: no data found
Current SQL statement for this session:
SELECT xdb_uid FROM SYS.EXU9XDBUID

问题原因
因为控制文件重建或者使用历史控制文件恢复,忘记添加临时文件

SQL> select name from v$tempfile;

no rows selected

解决方法
添加临时文件

SQL> alter tablespace TEMP add tempfile
  2 '/u01/oracle/oradata/xifenfei/temp01.dbf' size 10M reuse;

Tablespace altered.

SQL> select name from v$tempfile;

NAME
------------------------------------------------
/u01/oracle/oradata/xifenfei/temp01.dbf

验证exp导出

[oracle@xifenfei udump]$ exp chf/xifenfei file=/tmp/chf.dmp log=/tmp/chf.log owner=chf

Export: Release 9.2.0.4.0 - Production on Sun Apr 29 05:20:21 2012

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.


Connected to: Oracle9i Enterprise Edition Release 9.2.0.4.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.4.0 - Production
Export done in ZHS16GBK character set and AL16UTF16 NCHAR character set
server uses WE8ISO8859P1 character set (possible charset conversion)

About to export specified users ...
. exporting pre-schema procedural objects and actions
. exporting foreign function library names for user CHF 
. exporting PUBLIC type synonyms
. exporting private type synonyms
. exporting object type definitions for user CHF 
About to export CHF's objects ...
. exporting database links
. exporting sequence numbers
. exporting cluster definitions
. about to export CHF's tables via Conventional Path ...
. . exporting table ACC_OWE_TASK_LIST_HIS_07711202       4111 rows exported
. . exporting table                   CHF_XIFENFEI        868 rows exported
. . exporting table                     PLAN_TABLE          0 rows exported
. . exporting table                        T_XFF01          0 rows exported
. . exporting table                     T_XIFENFEI          1 rows exported
. exporting synonyms
. exporting views
. exporting stored procedures
. exporting operators
. exporting referential integrity constraints
. exporting triggers
. exporting indextypes
. exporting bitmap, functional and extensible indexes
. exporting posttables actions
. exporting materialized views
. exporting snapshot logs
. exporting job queues
. exporting refresh groups and children
. exporting dimensions
. exporting post-schema procedural objects and actions
. exporting statistics
Export terminated successfully no warnings.

bbed修改ASM中数据

本篇文章演示了如何从ASM中通过dd拷贝出某种表的记录,然后通过bbed修改相关记录,再拷贝到库中.说的简单点就是通过dd拷贝出最少的需要对象数据块,然后通过bbed绕过数据库级别对相关记录进行修改
模拟测试数据

--ORACLE数据库中执行
SQL> create tablespace xifenfei datafile '+xifenfei' size 30M autoextend on maxsize 10G;

Tablespace created.

SQL> create table t_xifenfei (id number,name varchar2(10)) tablespace xifenfei;

Table created.

SQL> insert into t_xifenfei values(1,'xifenfei');

1 row created.

SQL> insert into t_xifenfei values(2,'XFF');

1 row created.

SQL> SELECT * FROM T_XIFENFEI;

        ID NAME
---------- ----------
         1 xifenfei
         2 XFF

SQL> commit;

Commit complete.

SQL> select EXTENT_ID, BLOCK_ID, BLOCKS, FILE_ID from dba_extents 
  2  where SEGMENT_NAME='T_XIFENFEI' and OWNER='SYS';

 EXTENT_ID   BLOCK_ID     BLOCKS    FILE_ID
---------- ---------- ---------- ----------
         0        128          8          6

SQL> select name from v$datafile where file#=6;

NAME
----------------------------------------------------
+XIFENFEI/xff/datafile/xifenfei.268.781905429

SQL> select GROUP_NUMBER from V$ASM_DISKGROUP where NAME like '%XIFENFEI%';

GROUP_NUMBER
------------
           2

在ASM用户中查询相关数据

--ASM中执行
SQL> SELECT disk_kffxp, au_kffxp, xnum_kffxp
  2  FROM x$kffxp                      
  3   WHERE GROUP_KFFXP=2 
  4  AND NUMBER_KFFXP=268; 

DISK_KFFXP   AU_KFFXP XNUM_KFFXP
---------- ---------- ----------
         0        681          0
         1       1092          1
         1       1093          2
         0        682          3
         1       1094          4
         1       1095          5
         0        683          6
         1       1096          7
         0        684          8
         1       1097          9
         1       1098         10

DISK_KFFXP   AU_KFFXP XNUM_KFFXP
---------- ---------- ----------
         0        685         11
         1       1099         12
         0        686         13
         1       1100         14
         1       1101         15
         0        687         16
         1       1102         17
         1       1103         18
         0        688         19
         1       1104         20
         0        689         21

DISK_KFFXP   AU_KFFXP XNUM_KFFXP
---------- ---------- ----------
         1       1105         22
         1       1106         23
         0        690         24
         1       1107         25
         0        691         26
         1       1108         27
         1       1109         28
         0        692         29
         1       1110         30

31 rows selected.
--数据文件6的AU分配情况

SQL> select 128*8/1024 from dual;

128*8/1024
----------
         1


SQL> select 8*8/1024 from dual;

  8*8/1024
----------
     .0625
--可以得出该表T_XIFENFEI的数据分布在第二块AU中(DISK_KFFXP=1/AU_KFFXP=1092/XNUM_KFFXP=1)

SQL> select name, path from v$asm_disk where group_number=2     
  2  and disk_number=1;

NAME                           PATH
------------------------------ --------------------------
XIFENFEI_0001                  /dev/oracleasm/disks/VOL4

找出对应磁盘或者分区

[grid@rac1 ~]$ /etc/init.d/oracleasm querydisk -d VOL4
Disk "VOL4" is a valid ASM disk on device [8,18]
[grid@rac1 ~]$ cat /proc/partitions |grep "8       18"
   8       18    3879697 sdb2

因为这里的block_id=128,刚好是下一个AU的起点,所以dd操作的起点是第二个AU(DISK_KFFXP=1/AU_KFFXP=1092),而终点是8*8=64K(第二个AU中offset 64KB)

执行dd导出表数据

of=/dev/sdb2
sb=1K
skip=1092*1024=1118208
count=64

[root@rac1 ~]# dd if=/dev/sdb2 bs=1k count=64 skip=1118208|strings
64+0 records in
64+0 records out
65536 bytes (66 kB) copied, 0.000656471 seconds, 99.8 MB/s
XFF,
xifenfei
[root@rac1 ~]# dd if=/dev/sdb2 bs=1k count=64 skip=1118208 of=/tmp/t_xifenfe.tab
64+0 records in
64+0 records out
65536 bytes (66 kB) copied, 0.00226337 seconds, 29.0 MB/s
[root@rac1 ~]# chown oracle.oinstall /tmp/t_xifenfe.tab 
[root@rac1 ~]# ll /tmp/t_xifenfe.tab 
-rw-r--r-- 1 oracle oinstall 65536 Apr 29 21:54 /tmp/t_xifenfe.tab

bbed 修改数据内容

[oracle@rac1 ~]$ bbed
Password: 

BBED: Release 2.0.0.0.0 - Limited Production on Sun Apr 29 22:43:56 2012

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

************* !!! For Oracle Internal Use only !!! ***************

BBED> set filename '/tmp/t_xifenfe.tab'
        FILENAME        /tmp/t_xifenfe.tab

BBED> set block 4
        BLOCK#          4

BBED> set mode edit
        MODE            Edit

BBED> set blocksize 8192
        BLOCKSIZE       8192

BBED> map
 File: /tmp/t_xifenfe.tab (0)
 Block: 4                                     Dba:0x00000000
------------------------------------------------------------
 KTB Data Block (Table/Cluster)

 struct kcbh, 20 bytes                      @0       

 struct ktbbh, 72 bytes                     @20      

 struct kdbh, 14 bytes                      @100     

 struct kdbt[1], 4 bytes                    @114     

 sb2 kdbr[2]                                @118     

 ub1 freespace[8041]                        @122     

 ub1 rowdata[25]                            @8163    

 ub4 tailchk                                @8188    


BBED> p kdbr
sb2 kdbr[0]                                 @118      8073
sb2 kdbr[1]                                 @120      8063

BBED> find /c XFF
 File: /tmp/t_xifenfe.tab (0)
 Block: 4                Offsets: 8170 to 8191           Dba:0x00000000
------------------------------------------------------------------------
 5846462c 010202c1 02087869 66656e66 65690106 ba33 

 <32 bytes per line>

BBED> dump /v
 File: /tmp/t_xifenfe.tab (0)
 Block: 4       Offsets: 8170 to 8191  Dba:0x00000000
-------------------------------------------------------
 5846462c 010202c1 02087869 66656e66 l XFF,......xifenf
 65690106 ba33                       l ei...3

 <16 bytes per line>

BBED> m /c xff
 File: /tmp/t_xifenfe.tab (0)
 Block: 4                Offsets: 8170 to 8191           Dba:0x00000000
------------------------------------------------------------------------
 7866662c 010202c1 02087869 66656e66 65690106 ba33 

 <32 bytes per line>

BBED> dump /v
 File: /tmp/t_xifenfe.tab (0)
 Block: 4       Offsets: 8170 to 8191  Dba:0x00000000
-------------------------------------------------------
 7866662c 010202c1 02087869 66656e66 l xff,......xifenf
 65690106 ba33                       l ei...3

 <16 bytes per line>

BBED> find /c xifenfei
 File: /tmp/t_xifenfe.tab (0)
 Block: 4                Offsets: 8180 to 8191           Dba:0x00000000
------------------------------------------------------------------------
 78696665 6e666569 0106ba33 

 <32 bytes per line>

BBED> dump /v
 File: /tmp/t_xifenfe.tab (0)
 Block: 4       Offsets: 8180 to 8191  Dba:0x00000000
-------------------------------------------------------
 78696665 6e666569 0106ba33          l xifenfei...3

 <16 bytes per line>

BBED> m /c XIFENFEI
 File: /tmp/t_xifenfe.tab (0)
 Block: 4                Offsets: 8180 to 8191           Dba:0x00000000
------------------------------------------------------------------------
 58494645 4e464549 0106ba33 

 <32 bytes per line>

BBED> dump /v
 File: /tmp/t_xifenfe.tab (0)
 Block: 4       Offsets: 8180 to 8191  Dba:0x00000000
-------------------------------------------------------
 58494645 4e464549 0106ba33          l XIFENFEI...3

 <16 bytes per line>

BBED> sum
Check value for File 0, Block 4:
current = 0xd332, required = 0xf332

BBED> sum apply
Check value for File 0, Block 4:
current = 0xf332, required = 0xf332

BBED> set offset 8073
        OFFSET          8073

BBED> dump /v
 File: /tmp/t_xifenfe.tab (0)
 Block: 4       Offsets: 8073 to 8191  Dba:0x00000000
-------------------------------------------------------
 00000000 00000000 00000000 00000000 l ................
 00000000 00000000 00000000 00000000 l ................
 00000000 00000000 00000000 00000000 l ................
 00000000 00000000 00000000 00000000 l ................
 00000000 00000000 00000000 00000000 l ................
 00000000 00000000 00002c01 0202c103 l ..........,.....
 03786666 2c010202 c1020858 4946454e l .xff,......XIFEN
 46454901 06ba33                     l FEI...3

 <16 bytes per line>

BBED> exit

dd导入修改后数据验证

--会话1关闭数据库
[oracle@rac1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Sun Apr 29 22:48:51 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.


--会话2导入bbed修改后数据
[root@rac1 ~]# dd of=/dev/sdb2 bs=1k count=64 seek=1118208 if=/tmp/t_xifenfe.tab
64+0 records in
64+0 records out
65536 bytes (66 kB) copied, 0.0014908 seconds, 44.0 MB/s

--会话1启动数据库库查询
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
[oracle@rac1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Sun Apr 29 22:51:00 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area  535662592 bytes
Fixed Size                  1346140 bytes
Variable Size             411043236 bytes
Database Buffers          117440512 bytes
Redo Buffers                5832704 bytes
Database mounted.
Database opened.
SQL> select * from t_xifenfei;

        ID NAME
---------- ----------
         1 XIFENFEI
         2 xff

impdp报ORA-00904: “ORIGINAL_OBJECT_NAME”: invalid identifier

发现问题
impdp导入数据库不成功,一直在报ORA-00904: “ORIGINAL_OBJECT_NAME”: invalid identifier错误

[oracle@back1 backup]$ impdp username/password  schemas=center_admin dumpfile=center_admin20120427.dmp 
> logfile=center_admin20120427.log directory=impdir parallel=10 job_name=center_admin08;

Import: Release 11.1.0.6.0 - 64bit Production on Friday, 27 April, 2012 21:35:06

Copyright (c) 2003, 2007, Oracle.  All rights reserved.

Connected to: Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORA-39002: invalid operation
ORA-31694: master table "USERNAME"."CENTER_ADMIN08" failed to load/unload
ORA-02354: error in exporting/importing data
ORA-02373: Error parsing insert statement for table "USERNAME"."CENTER_ADMIN08".
ORA-00904: "ORIGINAL_OBJECT_NAME": invalid identifier

分析问题
看到这个错误,我第一个感觉根据ORA-31694,怀疑是没有创建相关用户,或者是该用户无权限权限CENTER_ADMIN08表.等我登陆目标数据库查看时候发现该用户存在,并且已经授予了DBA权限,所以不存在是用户相关问题导致.ORA-02354错误我怀疑是expdp导出来的文件在传输过程中发生意外(如使用ftp传输未使用二进制模式),当我使用md5sum命令检查发现两边一致,证明该文件传输正常.目标端不能检查明显故障,怀疑导出文件本身存在问题检查导出文件日志

[oracle@fcdb2 backup]$ more center_admin20120427.log
;;; 
Export: Release 11.1.0.7.0 - 64bit Production on Friday, 27 April, 2012 17:32:30

Copyright (c) 2003, 2007, Oracle.  All rights reserved.
;;; 
Connected to: Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
Starting "USERNAME"."CENTER_ADMIN08":USERNAME/**** schemas=center_admin directory=expdir dumpfile=center_admin20120427.dmp
er_admin20120427.log parallel=10 job_name=center_admin08 
Estimate in progress using BLOCKS method...
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 158.2 GB

发现新大陆,两边的数据库版本不一致,源端版本是11.1.0.7,目标端版本是11.1.0.6.这个时候我对问题的定位是可能版本兼用问题(毕竟是高版本到低版本)或者是bug.先查询datapump版本之间兼容性列表

     Version      Written by     Can be imported into Target:  
   Data Pump   database with      10gR1      10gR2      11gR1      11gR2  
Dumpfile Set   compatibility   10.1.0.x   10.2.0.x   11.1.0.x   11.2.0.x
------------ --------------- ---------- ---------- ---------- ----------
         0.1          10.1.x  supported  supported  supported  supported
         1.1          10.2.x         no  supported  supported  supported
         2.1          11.1.x         no         no  supported  supported 
         3.1          11.2.x         no         no         no  supported

这里可以看出11.1.0.7和11.1.0.6之间是相互兼容的,不应该会存在上述问题,那么现在对于该问题的解释很可能是bug导致,继续查询资料发现[ID 752374.1]描述的正是该问题.

解决问题

1. Apply 11.1.0.7 Patch:6890831 on the target database.

2. Workaround this issue by re-running an expdp from the 11.1.0.7 database with an additional 
parameter VERSION=10.2. This will create a new dump file compatible to be imported into 10gR2, 
which will also import successfully into 11.1.0.6.

Unfortunately, please be aware that the new 11g specific features will not be 
exported if expdp is run from 11.1.0.7 with the parameter VERSION=10.2. 
If only normal objects and features are involved, this could be a good workaround.