分享运气超级好的一次drop tablespace 数据恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:分享运气超级好的一次drop tablespace 数据恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

分享一次运气超级好的恢复,本身是一个测试库,应用厂商今天准备把应用正式上线,操作流程是:先删除用户,然后删除表空间,在创建表空间导入数据正式上线,不知何种原因最终客户在测试业务中做了一些正式数据,结果是无情的被删除了,通过alert日志找到应用厂商的一些操作记录
2021年8月份创建了业务表空间

Wed Aug 18 09:49:03 2021
create tablespace xifenfei datafile 'D:\app\Administrator\oradata\xifenfei\xifenfei.dbf' size 10g
Wed Aug 18 09:52:28 2021
Completed: create tablespace xifenfei datafile 'D:\app\Administrator\oradata\xifenfei\xifenfei.dbf' size 10g

今天删除表空间

Tue Apr 12 11:15:02 2022
drop tablespace xifenfei including contents and datafiles
WARNING: Cannot delete file D:\APP\ADMINISTRATOR\ORADATA\xifenfei\xifenfei.DBF
Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_ora_4296.trc:
ORA-01265: 鏃犳硶鍒犻櫎 DATA D:\APP\ADMINISTRATOR\ORADATA\xifenfei\xifenfei.DBF
ORA-27056: 鏃犳硶鍒犻櫎鏂囦欢
OSD-04024: 无法删除文件。
O/S-Error: (OS 32) 另一个程序正在使用此文件,进程无法访问。
Completed: drop tablespace xifenfei including contents and datafiles

然后客户创建新表空间提示ORA-01119,然后人工删除掉该数据文件

Tue Apr 12 11:49:02 2022
create tablespace xifenfei datafile'D:\oracle\oradata\xifenfei\xifenfei.dbf'size 20480m
ORA-1119 signalled during: create tablespace xifenfei datafile'D:\oracle\oradata\xifenfei\xifenfei.dbf'size 20480m...
Tue Apr 12 11:49:16 2022
create tablespace xifenfei datafile'D:\oracle\oradata\xifenfei\xifenfei.dbf'size 20480m
ORA-1119 signalled during: create tablespace xifenfei datafile'D:\oracle\oradata\xifenfei\xifenfei.dbf'size 20480m...

创建新表空间成功,并增加数据文件

Tue Apr 12 12:08:43 2022
create tablespace xifenfei datafile'D:\app\Administrator\oradata\xifenfei\xifenfei.dbf'size 5120m
Tue Apr 12 12:10:25 2022
Completed: create tablespace xifenfei datafile'D:\app\Administrator\oradata\xifenfei\xifenfei.dbf'size 5120m
Tue Apr 12 12:11:19 2022
alter tablespace xifenfei add datafile'D:\app\Administrator\oradata\xifenfei\xifenfei1.dbf'size 5120m
Tue Apr 12 12:13:02 2022
Completed: alter tablespace xifenfei add datafile'D:\app\Administrator\oradata\xifenfei\xifenfei1.dbf'size 5120m
alter tablespace xifenfei add datafile'D:\app\Administrator\oradata\xifenfei\xifenfei2.dbf'size 5120m
Tue Apr 12 12:14:52 2022
Completed: alter tablespace xifenfei add datafile'D:\app\Administrator\oradata\xifenfei\xifenfei2.dbf'size 5120m

基本情况就是客户删除了一个10G的业务数据文件,然后创建了3个5G的业务数据文件,现在要恢复被以前的两个表的核心数据,需要做的就是把以前的10G的数据文件找出来,但是由于删除10G文件之后又写入了15G的数据文件(而且这里面有文件的file#和删除的文件一致),理论上无法直接做block层面扫描恢复,对于此类情况,尝试文件系统层面直接反删除恢复,不过没有任何记录,文件目录被覆盖,这条路走不通.通过block扫描,发现2个file# 5文件的起始位置(分别是block 2和block 0),而且结束位置文件大小分别是10G和5G,根据经验这两个连续的磁盘分配空间很可能就是这两个file# 5的文件
20220415221719


通过winhex把数据拷贝出来,使用工具检测
20220407141224

除损坏的block 1之外(block 0 不统计在内),其他block都正常,也就是说这个10G的被删除的数据文件,只是丢失一个文件头,业务数据全部再,后续通过dul恢复客户需要数据,完成这次数据恢复,类似这种文件丢失,文件系统损坏,文件大小为0kb等类似恢复,参见以前类似blog:
win文件系统损坏oracle恢复
dbca删除库和rm删库恢复
文件系统重新分区oracle恢复
restore database误操作恢复
文件系统损坏导致数据文件异常恢复
Oracle 数据文件大小为0kb或者文件丢失恢复
rm -rf 删除数据文件恢复方法—文件系统反删除+oracle碎片重组

ORA-600 ktbsdp2 处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-600 ktbsdp2 处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户反馈数据库异常:两个节点rac,两个节点都启动,其中一个节点无法正常open,另外一个节点一段时间后也会挂。以下是无法正常open节点报错信息(正常open节点最终挂掉报错信息也是类似)

2022-03-30T19:19:12.870813+08:00
[84321] Successfully onlined Undo Tablespace 4.
Undo initialization finished serial:0 start:1728252021 end:1728256302 diff:4281 ms (4.3 seconds)
Verifying minimum file header compatibility for tablespace encryption..
Verifying file header compatibility for tablespace encryption completed for pdb 0
2022-03-30T19:19:13.953252+08:00
Database Characterset is ZHS16GBK
2022-03-30T19:19:14.538155+08:00
Errors in file /oracle/app/oracle/diag/rdbms/xff/xff2/trace/xff2_p02o_85718.trc  (incident=1093927):
ORA-00600: internal error code, arguments: [ktbsdp2], [18446744073709551615], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/xff/xff2/incident/incdir_1093927/xff2_p02o_85718_i1093927.trc
2022-03-30T19:19:15.536582+08:00
ORACLE Instance xff2 (pid = 57) - Error 607 encountered while recovering transaction (73, 12) on object 112841.
2022-03-30T19:19:33.699944+08:00
Errors in file /oracle/app/oracle/diag/rdbms/xff/xff2/trace/xff2_smon_84007.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [ktbsdp2], [18446744073709551615], [], [], [], [], [], [], [], [], [], []
2022-03-30T19:19:34.673840+08:00
Errors in file /oracle/app/oracle/diag/rdbms/xff/xff2/trace/xff2_smon_84007.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [ktbsdp2], [18446744073709551615], [], [], [], [], [], [], [], [], [], []
2022-03-30T19:19:34.673954+08:00
Errors in file /oracle/app/oracle/diag/rdbms/xff/xff2/trace/xff2_smon_84007.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [ktbsdp2], [18446744073709551615], [], [], [], [], [], [], [], [], [], []
Errors in file /oracle/app/oracle/diag/rdbms/xff/xff2/trace/xff2_smon_84007.trc  (incident=1092704):
ORA-607 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /oracle/app/oracle/diag/rdbms/xff/xff2/incident/incdir_1092704/xff2_smon_84007_i1092704.trc
2022-03-30T19:19:35.422779+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2022-03-30T19:19:36.154689+08:00
Starting background process GTX0
2022-03-30T19:19:36.169007+08:00
GTX0 started with pid=370, OS id=87409 
2022-03-30T19:19:36.645876+08:00
USER (ospid: 84007): terminating the instance due to error 607
2022-03-30T19:19:36.680109+08:00
opiodr aborting process unknown ospid (87439) as a result of ORA-1092
2022-03-30T19:19:36.681091+08:00
ORA-1092 : opitsk aborting process
2022-03-30T19:19:36.740357+08:00
System state dump requested by (instance=2, osid=84007 (SMON)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/app/oracle/diag/rdbms/xff/xff2/trace/xff2_diag_83895_20220330191936.trc
2022-03-30T19:19:40.135579+08:00
Instance terminated by USER, pid = 84007

对于上述报错信息分析,初步判断是由于事务异常导致,查询mos发现类似报错Bug 32208691 – After upgrade from 12.1 to 19.3 drop columns fails ORA-600[ktbsdp2] ORA-600[4512] (Doc ID 32208691.8),通过咨询客户,确认他们这边是通过plsql dev工具对id为112841表进行增加列的时候网络中断导致增加失败,后续我尝试对该表进行查询发现也报该错误,基本上可以确认由于该表事务异常导致,通过dul把该表数据恢复,然后drop 该表,数据库启动正常,未见其他报错,通过hcheck检查,数据库字典基本一致(除一些统计信息异常,原则上不影响数据库运行)

[oracle@xifenfei2 ~]$ sqlplus / as sysdba @hcheck.sql

SQL*Plus: Release 12.2.0.1.0 Production on Thu Mar 31 00:38:32 2022

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

HCheck Version 07MAY18 on 31-MAR-2022 00:38:34
----------------------------------------------
Catalog Version 12.2.0.1.0 (1202000100)
db_name: xff
Is CDB?: NO

                                   Catalog       Fixed
Procedure Name                     Version    Vs Release    Timestamp
Result
------------------------------ ... ---------- -- ---------- --------------
------
.- LobNotInObj                 ... 1202000100 <=  *All Rel* 03/31 00:38:34 PASS
.- MissingOIDOnObjCol          ... 1202000100 <=  *All Rel* 03/31 00:38:34 PASS
.- SourceNotInObj              ... 1202000100 <=  *All Rel* 03/31 00:38:34 PASS
.- OversizedFiles              ... 1202000100 <=  *All Rel* 03/31 00:38:38 PASS
.- PoorDefaultStorage          ... 1202000100 <=  *All Rel* 03/31 00:38:38 PASS
.- PoorStorage                 ... 1202000100 <=  *All Rel* 03/31 00:38:38 PASS
.- TabPartCountMismatch        ... 1202000100 <=  *All Rel* 03/31 00:38:38 PASS
.- OrphanedTabComPart          ... 1202000100 <=  *All Rel* 03/31 00:38:38 PASS
.- MissingSum$                 ... 1202000100 <=  *All Rel* 03/31 00:38:38 PASS
.- MissingDir$                 ... 1202000100 <=  *All Rel* 03/31 00:38:38 PASS
.- DuplicateDataobj            ... 1202000100 <=  *All Rel* 03/31 00:38:40 PASS
.- ObjSynMissing               ... 1202000100 <=  *All Rel* 03/31 00:38:42 PASS
.- ObjSeqMissing               ... 1202000100 <=  *All Rel* 03/31 00:38:42 PASS
.- OrphanedUndo                ... 1202000100 <=  *All Rel* 03/31 00:38:44 PASS
.- OrphanedIndex               ... 1202000100 <=  *All Rel* 03/31 00:38:44 PASS
.- OrphanedIndexPartition      ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- OrphanedIndexSubPartition   ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- OrphanedTable               ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- OrphanedTablePartition      ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- OrphanedTableSubPartition   ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- MissingPartCol              ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- OrphanedSeg$                ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- OrphanedIndPartObj#         ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- DuplicateBlockUse           ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- FetUet                      ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- Uet0Check                   ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- SeglessUET                  ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- BadInd$                     ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- BadTab$                     ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- BadIcolDepCnt               ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- ObjIndDobj                  ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- TrgAfterUpgrade             ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- ObjType0                    ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- BadOwner                    ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- StmtAuditOnCommit           ... 1202000100 <=  *All Rel* 03/31 00:38:45 PASS
.- BadPublicObjects            ... 1202000100 <=  *All Rel* 03/31 00:38:46 PASS
.- BadSegFreelist              ... 1202000100 <=  *All Rel* 03/31 00:38:50 PASS
.- BadDepends                  ... 1202000100 <=  *All Rel* 03/31 00:38:50 PASS
.- CheckDual                   ... 1202000100 <=  *All Rel* 03/31 00:38:57 PASS
.- ObjectNames                 ... 1202000100 <=  *All Rel* 03/31 00:38:57 WARN

HCKW-0018: OBJECT name clashes with SCHEMA name (Doc ID 2363142.1)
Schema=MHWZ PACKAGE=MHWZ.MHWZ
Schema=MHWZ PACKAGE BODY=MHWZ.MHWZ

.- BadCboHiLo                  ... 1202000100 <=  *All Rel* 03/31 00:39:01 WARN

HCKW-0019: HIST_HEAD$.LOWVAL > HIVAL (Doc ID 1361047.1)
OBJ# 324163 INTCOL#=22
OBJ# 482668 INTCOL#=4
OBJ# 442865 INTCOL#=31
OBJ# 436924 INTCOL#=31
OBJ# 580529 INTCOL#=8
OBJ# 459432 INTCOL#=31
OBJ# 451260 INTCOL#=31
OBJ# 530980 INTCOL#=21
OBJ# 498442 INTCOL#=5
OBJ# 652114 INTCOL#=8
OBJ# 701695 INTCOL#=21
OBJ# 831961 INTCOL#=31
OBJ# 831962 INTCOL#=31
OBJ# 831963 INTCOL#=31

.- ChkIotTs                    ... 1202000100 <=  *All Rel* 03/31 00:39:09 PASS
.- NoSegmentIndex              ... 1202000100 <=  *All Rel* 03/31 00:39:09 PASS
.- BadNextObject               ... 1202000100 <=  *All Rel* 03/31 00:39:09 PASS
.- DroppedROTS                 ... 1202000100 <=  *All Rel* 03/31 00:39:09 PASS
.- FilBlkZero                  ... 1202000100 <=  *All Rel* 03/31 00:39:09 PASS
.- DbmsSchemaCopy              ... 1202000100 <=  *All Rel* 03/31 00:39:09 PASS
.- OrphanedIdnseqObj           ... 1202000100 >  1201000000 03/31 00:39:09 PASS
.- OrphanedIdnseqSeq           ... 1202000100 >  1201000000 03/31 00:39:09 PASS
.- OrphanedObjError            ... 1202000100 >  1102000000 03/31 00:39:09 PASS
.- ObjNotLob                   ... 1202000100 <=  *All Rel* 03/31 00:39:09 PASS
.- MaxControlfSeq              ... 1202000100 <=  *All Rel* 03/31 00:39:09 PASS
.- SegNotInDeferredStg         ... 1202000100 >  1102000000 03/31 00:39:13 PASS
.- SystemNotRfile1             ... 1202000100 >   902000000 03/31 00:39:13 PASS
.- DictOwnNonDefaultSYSTEM     ... 1202000100 <=  *All Rel* 03/31 00:39:13 PASS
.- OrphanTrigger               ... 1202000100 <=  *All Rel* 03/31 00:39:13 PASS
.- ObjNotTrigger               ... 1202000100 <=  *All Rel* 03/31 00:39:13 PASS
---------------------------------------
31-MAR-2022 00:39:13  Elapsed: 39 secs
---------------------------------------
Found 0 potential problem(s) and 16 warning(s)
Contact Oracle Support with the output and trace file
to check if the above needs attention or not

PL/SQL procedure successfully completed.

Statement processed.

Complete output is in trace file:
/oracle/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_26887_HCHECK.trc

win文件系统损坏oracle恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:win文件系统损坏oracle恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户反馈数据文件无法拷贝也无法正常操作,通过查看系统日志发现ntfs文件系统异常
20220326210833


在文件系统中看数据文件状态正常,但是拷贝文件报错
20220404201143

通过恢复工具查看文件发现文件大小为0kb
20220326194440

通过文件系统层面扫描,然后进行相关数据恢复,依旧报错(恢复文件大小错误)
20220404201848

经过进一步人工修复文件系统目录,实现数据完整恢复,实现数据0丢失
20220404202616

基于这种情况如果通过文件系统层面无法恢复,对于此类oracle block级别进行处理,类似以前恢复case:
dbca删除库和rm删库恢复
文件系统损坏导致数据文件异常恢复
Oracle 数据文件大小为0kb或者文件丢失恢复

asm disk 磁盘部分被清空恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:asm disk 磁盘部分被清空恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

由于未知原因导致磁盘组的一个磁盘前面1G被置空(data磁盘组一共5个1T磁盘,第一个盘未知原因置空了1G数据)

[root@oradb1 ~]#kfed read /dev/mapper/asm-disk1
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
7F539088C400 00000000 00000000 00000000 00000000  [................]
  Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
…………
[root@oradb1 ~]#kfed read /dev/mapper/asm-disk1 aun=999
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
7F8FEB795400 00000000 00000000 00000000 00000000  [................]
  Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

通过分析disk 1和disk 2的前面1001MB数据,以及asm的数据分布特性,可以确认丢失的1000MB数据为39号文件的数据(和客户当时这个库是通过rman还原过来的操作有关系)
20220330224604
20220330225211


对于这样的情况,通过底层扫描,可以很好的恢复数据,参考以前类似文章
asm disk被加入vg恢复
asm disk header 彻底损坏恢复
asm磁盘dd破坏恢复
通过底层恢复,恢复结果如下
20220331133634

dbv检查数据文件
20220331134249

然后把数据库恢复出来数据文件中的数据恢复到新库中,至此完成该case恢复

File #xxx found in data dictionary but not in controlfile. Creating OFFLINE file ‘MISSING00XXX’ in the controlfile

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:File #xxx found in data dictionary but not in controlfile. Creating OFFLINE file ‘MISSING00XXX’ in the controlfile

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

接手客户的库,已经被强制resetlogs 报ORA-600 2662错误

Sat Mar 19 20:07:48 2022
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 26228222518
Clearing online redo logfile 1 /u2/oradb/oradata/xff/redo01.log
Clearing online log 1 of thread 1 sequence number 0
Clearing online redo logfile 1 complete
Online log /u2/oradb/oradata/xff/redo01.log: Thread 1 Group 1 was previously cleared
Online log /u2/oradb/oradata/xff/redo02.log: Thread 1 Group 2 was previously cleared
Online log /u2/oradb/oradata/xff/redo03.log: Thread 1 Group 3 was previously cleared
Sat Mar 19 20:08:02 2022
Setting recovery target incarnation to 2
Sat Mar 19 20:08:07 2022
Assigning activation ID 2327373166 (0x8ab8e56e)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /u2/oradb/oradata/xff/redo01.log
Successful open of redo thread 1
Sat Mar 19 20:08:14 2022
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sat Mar 19 20:08:14 2022
SMON: enabling cache recovery
ORA-00600: internal error code, arguments: [2662], [6], [458447781], [6], [458448180], [12583056]
ORA-00600: internal error code, arguments: [2662], [6], [458447780], [6], [458448180], [12583056]
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [6], [458447778], [6], [458448180], [12583056]
Incident details in: /u2/oradb/diag/rdbms/xifenfei/xifenfei/incident/incdir_1181122/xifenfei_ora_19893_i1181122.trc
Errors in file /u2/oradb/diag/rdbms/xifenfei/xifenfei/incident/incdir_1181122/xifenfei_ora_19893_i1181122.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [6], [458447781], [6], [458448180], [12583056]
ORA-00600: internal error code, arguments: [2662], [6], [458447780], [6], [458448180], [12583056]
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [6], [458447778], [6], [458448180], [12583056]

接手之后尝试获取创建控制文件脚本,报ORA-16433

Mon Mar 21 15:20:37 2022
alter database backup controlfile to trace as '/tmp/ctl'
ORA-16433 signalled during: alter database backup controlfile to trace as '/tmp/ctl'...

经过一些处理之后resetlogs 成功,但是悲剧产生了(客户之前自己重建过ctl,遗漏大量数据文件,然后我参照客户的ctl进行处理)使得新的ctl中遗漏的很多数据文件,库被resetlogs打开,导致部分文件的resetlog scn不一致,另外数据库还有ORA-600 4137错误需要处理

Mon Mar 21 15:35:01 2022
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 26228222522
Clearing online redo logfile 1 /u2/oradb/oradata/xff/redo01.log
Clearing online redo logfile 1 complete
Resetting resetlogs activation ID 2327373166 (0x8ab8e56e)
Errors in file /u2/oradb/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_4029.trc:
ORA-00367: checksum error in log file header
ORA-00322: log 1 of thread 1 is not current copy
ORA-00312: online log 1 thread 1: '/u2/oradb/oradata/xff/redo01.log'
Errors in file /u2/oradb/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_4029.trc:
ORA-00367: checksum error in log file header
ORA-00322: log 2 of thread 1 is not current copy
ORA-00312: online log 2 thread 1: '/u2/oradb/oradata/xff/redo02.log'
Errors in file /u2/oradb/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_4029.trc:
ORA-00367: checksum error in log file header
ORA-00322: log 3 of thread 1 is not current copy
ORA-00312: online log 3 thread 1: '/u2/oradb/oradata/xff/redo03.log'
Mon Mar 21 15:35:16 2022
Setting recovery target incarnation to 2
Mon Mar 21 15:35:23 2022
Assigning activation ID 2327514749 (0x8abb0e7d)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /u2/oradb/oradata/xff/redo01.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Mar 21 15:35:30 2022
SMON: enabling cache recovery
Undo initialization finished serial:0 start:191905344 end:191905754 diff:410 (4 seconds)
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Tablespace 'DS_POS' #9 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Mon Mar 21 15:36:00 2022
File #498 found in data dictionary but not in controlfile.
Creating OFFLINE file 'MISSING00498' in the controlfile.
…………
File #567 found in data dictionary but not in controlfile.
Creating OFFLINE file 'MISSING00567' in the controlfile.
This file can no longer be recovered so it must be dropped.
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
Mon Mar 21 15:36:08 2022
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is AL32UTF8
No Resource Manager plan active
Errors in file /u2/oradb/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_smon_4017.trc  (incident=1325274):
ORA-00600: internal error code, arguments: [4137], [28.27.4413199], [0], [0], [], [], [], [], [], [], [], []
Incident details in: /u2/oradb/diag/rdbms/xifenfei/xifenfei/incident/incdir_1325274/xifenfei_smon_4017_i1325274.trc
Use ADRCI or Support Workbench to package the incident.
Mon Mar 21 15:36:10 2022
db_recovery_file_dest_size of 49770 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Mon Mar 21 15:36:10 2022
Starting background process CJQ0
Mon Mar 21 15:36:10 2022
CJQ0 started with pid=27, OS id=4155 
Mon Mar 21 15:36:10 2022
Errors in file /u2/oradb/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_smon_4017.trc  (incident=1325275):
ORA-00600: internal error code, arguments: [4137], [36.20.1072031], [0], [0], [], [], [], [], [], [], [], []
Incident details in: /u2/oradb/diag/rdbms/xifenfei/xifenfei/incident/incdir_1325275/xifenfei_smon_4017_i1325275.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Mar 21 15:36:10 2022
Completed: alter database open resetlogs

对于类似:File #567 found in data dictionary but not in controlfile.Creating OFFLINE file ‘MISSING00567′ in the controlfile.
错误处理方法:(参照bbed解决ORA-01190
1. 从操作系统中找出来所有遗漏的文件
2. 通过bbed修改文件头信息
3. 重建ctl
4. 重新打开库

Mon Mar 21 16:06:39 2022
alter database open resetlogs
RESETLOGS after complete recovery through change 28991030095
Clearing online redo logfile 1 /u2/oradb/oradata/xff/redo01.log
Clearing online log 1 of thread 1 sequence number 0
Clearing online redo logfile 1 complete
Resetting resetlogs activation ID 2327514749 (0x8abb0e7d)
Online log /u2/oradb/oradata/xff/redo01.log: Thread 1 Group 1 was previously cleared
Online log /u2/oradb/oradata/xff/redo02.log: Thread 1 Group 2 was previously cleared
Online log /u2/oradb/oradata/xff/redo03.log: Thread 1 Group 3 was previously cleared
Mon Mar 21 16:06:53 2022
Setting recovery target incarnation to 2
Mon Mar 21 16:07:00 2022
Assigning activation ID 2327541328 (0x8abb7650)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /u2/oradb/oradata/xff/redo01.log
Successful open of redo thread 1
Mon Mar 21 16:07:07 2022
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Mar 21 16:07:07 2022
SMON: enabling cache recovery
Undo initialization finished serial:0 start:193802264 end:193802294 diff:30 (0 seconds)
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Mon Mar 21 16:07:38 2022
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
Mon Mar 21 16:07:38 2022
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is AL32UTF8
No Resource Manager plan active
**********************************************************
WARNING: Files may exists in db_recovery_file_dest
that are not known to the database. Use the RMAN command
CATALOG RECOVERY AREA to re-catalog any such files.
If files cannot be cataloged, then manually delete them
using OS command.
One of the following events caused this:
1. A backup controlfile was restored.
2. A standby controlfile was restored.
3. The controlfile was re-created.
4. db_recovery_file_dest had previously been enabled and
   then disabled.
**********************************************************
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Mon Mar 21 16:07:40 2022
QMNC started with pid=23, OS id=5476 
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Mon Mar 21 16:07:41 2022
db_recovery_file_dest_size of 49770 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Completed: alter database open resetlogs

至此数据库open成功,增加temp文件,然后逻辑迁移库