在数据库恢复遭遇ORA-07445 kgegpa错误

Posted on 2020 年 04 月 24 日 by 惜分飞

接到客户恢复请求,数据库启动报ORA-600 2662错误

Fri Apr 24 19:52:58 2020
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 15491509441794
Resetting resetlogs activation ID 1460987657 (0x5714e709)
Fri Apr 24 19:52:59 2020
Setting recovery target incarnation to 3
Fri Apr 24 19:52:59 2020
Assigning activation ID 1566342598 (0x5d5c7dc6)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: Y:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Apr 24 19:52:59 2020
SMON: enabling cache recovery
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_3860.trc  (incident=8561):
ORA-00600: 内部错误代码, 参数: [2662], [3606], [3857372426], [3606], [3857377059], [12583040], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_8561\orcl_ora_3860_i8561.trc
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_3860.trc:
ORA-00600: 内部错误代码, 参数: [2662], [3606], [3857372426], [3606], [3857377059], [12583040], [], [], [], [], [], []
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_3860.trc:
ORA-00600: 内部错误代码, 参数: [2662], [3606], [3857372426], [3606], [3857377059], [12583040], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 3860): terminating the instance due to error 600
Instance terminated by USER, pid = 3860
ORA-1092 signalled during: alter database open resetlogs...

这个错误比较常见,通过对数据库scn进行调整,顺利规避该错误,继续启动报如下错误

SQL> startup mount pfile='d:/pfile.txt';
ORACLE 例程已经启动。

Total System Global Area 1.3696E+10 bytes
Fixed Size                  2188768 bytes
Variable Size            6878661152 bytes
Database Buffers         6777995264 bytes
Redo Buffers               37044224 bytes
数据库装载完毕。
SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-03113: 通信通道的文件结尾
进程 ID: 5884
会话 ID: 66 序列号: 3

Fri Apr 24 20:57:49 2020
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Dictionary check beginning
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0x898ADE43] [PC:0x9287D88, kgegpa()+38]
Dump file d:\app\administrator\diag\rdbms\orcl\orcl\trace\alert_orcl.log
Fri Apr 24 20:57:49 2020
ORACLE V11.2.0.1.0 - 64bit Production vsnsta=0
vsnsql=16 vsnxtr=3
Windows NT Version V6.1  
CPU                 : 16 - type 8664, 16 Physical Cores
Process Affinity    : 0x0x0000000000000000
Memory (Avail/Total): Ph:21429M/32767M, Ph+PgF:54255M/65533M 
Fri Apr 24 20:57:49 2020
Errors in file 
ORA-07445: caught exception [ACCESS_VIOLATION] at [kgegpa()+38] [0x0000000009287D88]
Fri Apr 24 20:57:52 2020
PMON (ospid: 2496): terminating the instance due to error 397
Instance terminated by PMON, pid = 2496

这里的主要错误是由于ORA-07445 kgegpa,根据以前恢复经验,该问题很可能和undo有关,对undo进行处理之后启动库

SQL> startup mount pfile='d:/pfile.txt' ;
ORACLE 例程已经启动。

Total System Global Area 1.3696E+10 bytes
Fixed Size                  2188768 bytes
Variable Size            6878661152 bytes
Database Buffers         6777995264 bytes
Redo Buffers               37044224 bytes
数据库装载完毕。
SQL> recover database;
完成介质恢复。
SQL> alter database open;

数据库已更改。

SMON: enabling tx recovery
Database Characterset is ZHS16GBK
SMON: Restarting fast_start parallel rollback
Fri Apr 24 21:01:28 2020
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_p000_4360.trc  (incident=13377):
ORA-00600: internal error code, arguments: [4198], [], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_13377\orcl_p000_4360_i13377.trc
Stopping background process MMNL
Doing block recovery for file 3 block 296
Resuming block recovery (PMON) for file 3 block 296
Block recovery from logseq 3, block 25 to scn 15491947056761
Recovery of Online Redo Log: Thread 1 Group 3 Seq 3 Reading mem 0
  Mem# 0: Y:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Block recovery completed at rba 3.25.16, scn 3607.20090
Doing block recovery for file 6 block 165592
Resuming block recovery (PMON) for file 6 block 165592
Block recovery from logseq 3, block 33 to scn 15491947056769
Recovery of Online Redo Log: Thread 1 Group 3 Seq 3 Reading mem 0
  Mem# 0: Y:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Block recovery completed at rba 3.58.16, scn 3607.20098
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_4912.trc  (incident=13321):
ORA-00600: internal error code, arguments: [4198], [], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_13321\orcl_smon_4912_i13321.trc
SMON: Parallel transaction recovery slave got internal error
SMON: Downgrading transaction recovery to serial
Stopping background process MMON
Fri Apr 24 21:01:29 2020
Trace dumping is performing id=[cdmp_20200424210129]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_4912.trc  (incident=13322):
ORA-00600: internal error code, arguments: [4137], [12.30.1712324], [0], [0], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_13322\orcl_smon_4912_i13322.trc
ORACLE Instance orcl (pid = 14) - Error 600 encountered while recovering transaction (12, 30).
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_4912.trc:
ORA-00600: internal error code, arguments: [4137], [12.30.1712324], [0], [0], [], [], [], [], [], [], [], []
Completed: alter database open upgrade
Fri Apr 24 21:01:30 2020
MMON started with pid=16, OS id=4980 
Fri Apr 24 21:01:31 2020
Sweep [inc][13322]: completed
Corrupt block relative dba: 0x00c395ee (file 3, block 234990)
Fractured block found during buffer read
Data in bad block:
 type: 2 format: 2 rdba: 0x00c395ee
 last change scn: 0x0e16.e5ead38b seq: 0x2b flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0xdb720232
 check value in block header: 0xebe2
 computed block checksum: 0xb60b
Reading datafile'Y:\APP\ADMINISTRATOR\ORADATA\ORCL\UNDOTBS01.DBF'for corruption at rdba: 0x00c395ee (file 3,block 234990)
Reread (file 3, block 234990) found same corrupt data
Corrupt Block Found
         TSN = 2, TSNAME = UNDOTBS1
         RFN = 3, BLK = 234990, RDBA = 12817902
         OBJN = 0, OBJD = -1, OBJECT = , SUBOBJECT = 
         SEGMENT OWNER = , SEGMENT TYPE = 
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_m001_4852.trc  (incident=13641):
ORA-01578: ORACLE data block corrupted (file # 3, block # 234990)
ORA-01110: data file 3: 'Y:\APP\ADMINISTRATOR\ORADATA\ORCL\UNDOTBS01.DBF'
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_13641\orcl_m001_4852_i13641.trc

SQL> create undo tablespace undotbs2 datafile 
2   'Y:\APP\ADMINISTRATOR\ORADATA\ORCL\undo_xff02.dbf' size 128M autoextend on;

表空间已创建。

SQL> drop tablespace undotbs1 including contents and datafiles;

表空间已删除。

SQL> shutdown immediate;
数据库已经关闭。
已经卸载数据库。
ORACLE 例程已经关闭。
SQL> create spfile from pfile='d:/pfile.txt';

文件已创建。

SQL> startup mount
ORACLE 例程已经启动。

Total System Global Area 1.3696E+10 bytes
Fixed Size                  2188768 bytes
Variable Size            6878661152 bytes
Database Buffers         6777995264 bytes
Redo Buffers               37044224 bytes
数据库装载完毕。
SQL> alter database open;

数据库已更改。

数据库启动之后继续报出来的ORA-600 4198和ORA-600 4137以及undo坏块均证明是由于undo异常引起的问题,通过重建新undo,数据库open正常,安排客户进行数据导出导入到新库

aix平台tab$被删除可能出现ORA-600 [16703], [1403], [28]错误

Posted on 2020 年 04 月 04 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：aix平台tab$被删除可能出现ORA-600 [16703], [1403], [28]错误

以前在恢复过程中遇到过ORA-00600: internal error code, arguments: [16703], [1403], [28]错误(10g数据库遭遇ORA-600 16703)以为是因为10g版本的tab$记录被删除的原因导致报错和最常见的ORA-00600: internal error code, arguments: [16703], [1403], [20]不完全一致(警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703),最近又遇到一个ORA-600 16703 1403 28错误的case,而且数据库版本是11.2.0.4 for aix平台,进一步说明该问题不是由于10g和11g的tab$被删除的区别导致,更多可能是由于操作系统不一样,数据库启动基表访问顺序不一致导致,特此进行说明.
数据库启动成功后报错

Wed Apr 01 22:36:19 2020
Completed: ALTER DATABASE OPEN /* db agent *//* {2:54387:2} */
Wed Apr 01 22:36:19 2020
Starting background process CJQ0
Wed Apr 01 22:36:19 2020
CJQ0 started with pid=53, OS id=7078224 
Wed Apr 01 22:36:21 2020
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_7668220.trc  (incident=40337):
ORA-00600: internal error code, arguments: [kzrini:!uprofile], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_40337/orcl2_ora_7668220_i40337.trc
Wed Apr 01 22:36:21 2020
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_6881718.trc  (incident=40369):
ORA-00600: internal error code, arguments: [kzrini:!uprofile], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_40369/orcl2_ora_6881718_i40369.trc
Setting Resource Manager plan SCHEDULER[0x32DB]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter

Wed Apr 01 22:51:16 2020
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_smon_7078802.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00957: duplicate column name
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_smon_7078802.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00957: duplicate column name
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_smon_7078802.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00957: duplicate column name

Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_m000_4850312.trc  (incident=48045):
ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_48045/orcl2_m000_4850312_i48045.trc
Thu Apr 02 00:59:35 2020
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_smon_7078802.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00957: duplicate column name
Thu Apr 02 00:59:36 2020
DDE: Problem Key 'ORA 600 [kzrini:!uprofile]' was completely flood controlled (0x6)
Further messages for this problem key will be suppressed for up to 10 minutes
Thu Apr 02 00:59:37 2020
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_m000_4850312.trc  (incident=48046):
ORA-00600: internal error code, arguments: [kewrose_1], [600], 
[ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280], [], [], [], []

数据库再次重启报错

Completed: ALTER DATABASE MOUNT /* db agent *//* {1:25340:2} */
ALTER DATABASE OPEN /* db agent *//* {1:25340:2} */
This instance was first to open
Picked broadcast on commit scheme to generate SCNs
Thu Apr 02 02:17:59 2020
Thread 2 opened at log sequence 13485
  Current log# 3 seq# 13485 mem# 0: +DATA/orcl/onlinelog/group_3.265.1003137665
  Current log# 3 seq# 13485 mem# 1: +FLASH/orcl/onlinelog/group_3.259.1003137677
Successful open of redo thread 2
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu Apr 02 02:17:59 2020
SMON: enabling cache recovery
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_7799020.trc  (incident=48395):
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_48395/orcl2_ora_7799020_i48395.trc
Thu Apr 02 02:18:01 2020
Thu Apr 02 02:18:01 2020
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_7799020.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_7799020.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 7799020): terminating the instance due to error 704
Instance terminated by USER, pid = 7799020
ORA-1092 signalled during: ALTER DATABASE OPEN /* db agent *//* {1:25340:2} */...

对于此类问题,通过分析,确定也是由于DBMS_SUPPORT_DBMONITORP恶意脚本导致tab$记录被删除,导致数据库启动异常,处理方法基本上就是对tab$进行恢复,然后open数据库.

expdp dmp被加密破坏恢复

Posted on 2020 年 03 月 20 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：expdp dmp被加密破坏恢复

有朋友oracle数据库dmp备份被加密,后缀为:.DMP.voyager,通过分析发现文件加密2M左右

这里可以看出来dmp文件为expdp方式导出(expdp本质上xml方式存储,exp使用直接二进制方式存储),通过工具分析可恢复表情况.
通过工具对该dmp文件进行分析

CPFL> OPEN F:\BaiduNetdisk\KINGDEE85GH_2020-03-17.DMP.voyager
TABLE_NAME                                         START_POS       DATA_BYTE     
-------------------------------------------------- --------------- --------------- 
KINGDEE85GH.T_WFD_PROCESSDEF                       116300288       648396035       
KINGDEE85GH.T_DYN_DYNAMICCONFIGURE                 864710656       181453794       
KINGDEE85GH.T_RPTS_STORAGEFILEDATA                 1078767616      21548951        
KINGDEE85GH.T_BOT_RULESEGMENT                      1100324864      10372516        
KINGDEE85GH.T_LOG_APP                              1110712320      12603573        
KINGDEE85GH.T_PM_PERMITEM                          1123336192      7282412         
KINGDEE85GH.T_PM_USERORGPERM                       1130635264      6692320         
KINGDEE85GH.T_DYN_APPSOLUTION                      1137336320      801697          
KINGDEE85GH.T_PM_MAINMENUITEM                      1138155520      3573943         
KINGDEE85GH.T_PM_PERMUIGROUP                       1141751808      2159245         
KINGDEE85GH.T_SYS_ENTITYREF                        1143922688      4183869         
KINGDEE85GH.T_PM_ROLEPERM                          1148116992      2758960         
KINGDEE85GH.T_BAS_SYSMENUITEM                      1150885888      3304627         
KINGDEE85GH.T_JP_PAGE                              1154211840      3019174  
…………      
KINGDEE85GH.T_XT_CHECKTIME                         1212776448      41              
KINGDEE85GH.T_XT_SYNCHTIME                         1212784640      41              
SYSTEM.SYS_EXPORT_SCHEMA_02                        1212792832      215423380    
-------------------------------------------------- --------------- --------------- 
Scanned Find 895 segments.

通过这个基本上可以确定丢失了100多M数据，其他数据理论上可以恢复.
创建用户

SQL> create user KINGDEE85GHidentified by oracle;

User created.

SQL> grant dba to KINGDEE85GH;

Grant succeeded.

unexpdp数据(自动创建表和导入数据)

CPFL> unexpdp table KINGDEE85GH.T_WFD_PROCESSDEF

unexpdp table: KINGDEE85GH.T_WFD_PROCESSDEF storage(START_POSITION:116300288 DATA_BYTE:748396035)
824 rows unexpdp

查看恢复结果

如果你有oracle expdp dmp被加密或者破坏,无法正常导入数据库,可以联系我们对其进行恢复处理:提供(ORACLE数据库恢复技术支持)：
Phone:17813235971 Q Q:107644445 E-Mail:dba@xifenfei.com
如果你的oracle dmp是exp方式导出,也可以联系我们对其进行处理,参见:
exp dmp文件损坏恢复
oracle dmp被加密恢复

又一例system大量坏块恢复

Posted on 2020 年 03 月 12 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：又一例system大量坏块恢复

有朋友找到我们,说数据库服务可以启动,但是无法登陆,类似报错

C:\Users\XIFENFEI>D:\app\XIFENFEI\product\11.2.0.1\dbhome_2\bin\sqlplus / as sys
dba

SQL*Plus: Release 11.2.0.1.0 Production on 星期四 3月 12 15:04:32 2020

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

ERROR:
ORA-01075: 您现在已登录


请输入用户名:
ERROR:
ORA-01017: 用户名/口令无效; 登录被拒绝


请输入用户名:
ERROR:
ORA-01017: 用户名/口令无效; 登录被拒绝


SP2-0157: 在 3 次尝试之后无法连接到 ORACLE, 退出 SQL*Plus

通过分析发现数据库启动报错（未正常open成功）

C:\Users\XIFENFEI>D:\app\XIFENFEI\product\11.2.0.1\dbhome_2\bin\sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on 星期四 3月 12 14:58:28 2020

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

已连接到空闲例程。

SQL> startup mount pfile='F:\pfile.txt';
ORACLE 例程已经启动。

Total System Global Area 3307048960 bytes
Fixed Size                  2180264 bytes
Variable Size            1811942232 bytes
Database Buffers         1476395008 bytes
Redo Buffers               16531456 bytes
数据库装载完毕。

SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-01578: ORACLE 数据块损坏 (文件号 1, 块号 48396)
ORA-01110: 数据文件 1: 'F:\ORADATA\SYSTEM01.DBF'

数据库没有正常启动成功的原因是由于system文件有坏块导致,通过dbv检查system文件发现有大量连续坏块

DBVERIFY: Release 11.2.0.1.0 - Production on 星期四 3月 12 19:12:34 2020

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.


DBVERIFY - 开始验证: FILE = F:\ORADATA\SYSTEM01.DBF
页 48064 流入 - 很可能是介质损坏
Corrupt block relative dba: 0x0040bbc0 (file 1, block 48064)
Fractured block found during dbv: 
Data in bad block:
 type: 6 format: 2 rdba: 0x0040bbc0
 last change scn: 0x0000.0006f69c seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x43455474
 check value in block header: 0x3893
 computed block checksum: 0xadfa

…………

页 48412 标记为损坏
Corrupt block relative dba: 0x0040bd1c (file 1, block 48412)
Bad header found during dbv: 
Data in bad block:
 type: 36 format: 2 rdba: 0x6dce856d
 last change scn: 0xfc44.d24c936f seq: 0xdc flg: 0x3b
 spare1: 0x9c spare2: 0x92 spare3: 0xcf67
 consistency value in tail: 0x43455474
 check value in block header: 0x2598
 block checksum disabled


DBVERIFY - 验证完成

检查的页总数: 249600
处理的页总数 (数据): 209467
失败的页总数 (数据): 0
处理的页总数 (索引): 13616
失败的页总数 (索引): 0
处理的页总数 (其他): 3369
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 22799
标记为损坏的总页数: 349
流入的页总数: 1
加密的总页数        : 0
最高块 SCN            : 84123103 (0.84123103)

数据库alert日志信息

Thu Mar 12 14:52:20 2020
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Hex dump of (file 1, block 48403) in trace file d:\app\xifenfei\diag\rdbms\qdbdc\o11201gbk\trace\o11201gbk_ora_9480.trc
Corrupt block relative dba: 0x0040bd13 (file 1, block 48403)
Bad header found during multiblock buffer read
Data in bad block:
 type: 36 format: 2 rdba: 0x6dce856d
 last change scn: 0xfc44.d24c936f seq: 0xdc flg: 0x3b
 spare1: 0x9c spare2: 0x92 spare3: 0xcf67
 consistency value in tail: 0x43455474
 check value in block header: 0x2598
 block checksum disabled
Reading datafile 'F:\ORADATA\SYSTEM01.DBF' for corruption at rdba: 0x0040bd13 (file 1, block 48403)
Reread (file 1, block 48403) found same corrupt data
Hex dump of (file 1, block 48404) in trace file d:\app\xifenfei\diag\rdbms\qdbdc\o11201gbk\trace\o11201gbk_ora_9480.trc
Corrupt block relative dba: 0x0040bd14 (file 1, block 48404)
Corrupt Block Found
Bad header found during multiblock buffer read
         TSN = 0, TSNAME = SYSTEM
Data in bad block:
         RFN = 1, BLK = 48403, RDBA = 4242707
 type: 36 format: 2 rdba: 0x6dce856d
 last change scn: 0xfc44.d24c936f seq: 0xdc flg: 0x3b
 spare1: 0x9c spare2: 0x92 spare3: 0xcf67
 consistency value in tail: 0x43455474
 check value in block header: 0x2598
 block checksum disabled
Reading datafile 'F:\ORADATA\SYSTEM01.DBF' for corruption at rdba: 0x0040bd14 (file 1, block 48404)
Reread (file 1, block 48404) found same corrupt data
Errors in file d:\app\xifenfei\diag\rdbms\qdbdc\o11201gbk\trace\o11201gbk_ora_9480.trc  (incident=3784):
ORA-01578: ORACLE 数据块损坏 (文件号 1, 块号 48404)
ORA-01110: 数据文件 1: 'F:\ORADATA\SYSTEM01.DBF'
Incident details in: d:\app\xifenfei\diag\rdbms\qdbdc\o11201gbk\incident\incdir_3784\o11201gbk_ora_9480_i3784.trc
         OBJN = 36, OBJD = 36, OBJECT = I_OBJ1, SUBOBJECT = 
         SEGMENT OWNER = SYS, SEGMENT TYPE = Index Segment
Corrupt Block Found
         TSN = 0, TSNAME = SYSTEM
         RFN = 1, BLK = 48404, RDBA = 4242708
         OBJN = 36, OBJD = 36, OBJECT = I_OBJ1, SUBOBJECT = 
         SEGMENT OWNER = SYS, SEGMENT TYPE = Index Segment
Errors in file d:\app\xifenfei\diag\rdbms\qdbdc\o11201gbk\trace\o11201gbk_ora_9480.trc:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-01578: ORACLE 数据块损坏 (文件号 1, 块号 48404)
ORA-01110: 数据文件 1: 'F:\ORADATA\SYSTEM01.DBF'
Errors in file d:\app\xifenfei\diag\rdbms\qdbdc\o11201gbk\trace\o11201gbk_ora_9480.trc:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-01578: ORACLE 数据块损坏 (文件号 1, 块号 48404)
ORA-01110: 数据文件 1: 'F:\ORADATA\SYSTEM01.DBF'
Error 604 happened during db open, shutting down database
USER (ospid: 9480): terminating the instance due to error 604

这里可以看出来数据库不能正常启动的原因,主要是由于I_OBJ1(obj$表的index)刚好被损坏,导致数据库无法,通过分析定位确定是如下sql导致启动失败

Dump continued from file: d:\app\xifenfei\diag\rdbms\qdbdc\o11201gbk\trace\o11201gbk_ora_9480.trc
ORA-01578: ORACLE 数据块损坏 (文件号 1, 块号 48404)
ORA-01110: 数据文件 1: 'F:\ORADATA\SYSTEM01.DBF'

========= Dump for incident 3784 (ORA 1578) ========

*** 2020-03-12 14:52:20.614
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=cq514nkrp38hv) -----
select distinct d.p_obj#,d.p_timestamp from sys.dependency$ d, obj$ o where d.p_obj#>=:1 and 
d.d_obj#=o.obj# and o.status not in (5,6)

通过对其i_obj1损坏block进行修复,数据库正启动成功

C:\Users\XIFENFEI>D:\app\XIFENFEI\product\11.2.0.1\dbhome_2\bin\sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on 星期四 3月 12 15:05:48 2020

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

已连接到空闲例程。

SQL> startup pfile='f:/pfile.txt' mount;
ORACLE 例程已经启动。

Total System Global Area 3307048960 bytes
Fixed Size                  2180264 bytes
Variable Size            1811942232 bytes
Database Buffers         1476395008 bytes
Redo Buffers               16531456 bytes
数据库装载完毕。
SQL> alter database open;

数据库已更改。

尝试导出数据

C:\Windows\system32>exp system/oracle owner=gis_sys file=f:/gis_sys.dmp FEEDBACK
=10000  COMPRESS=NO BUFFER=102400000 STATISTICS=none

Export: Release 11.2.0.1.0 - Production on 星期四 3月 12 15:46:19 2020

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.


连接到: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
已导出 ZHS16GBK 字符集和 AL16UTF16 NCHAR 字符集

即将导出指定的用户...
. 正在导出 pre-schema 过程对象和操作
. 正在导出用户 GIS_SYS 的外部函数库名
. 导出 PUBLIC 类型同义词
EXP-00008: 遇到 ORACLE 错误 604
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-01410: 无效的 ROWID
EXP-00000: 导出终止失败

分析trace文件

*** SESSION ID:(192.3) 2020-03-12 15:05:36.132
OBJD MISMATCH typ=6, seg.obj=18, diskobj=224, dsflg=100100, dsobj=18, tid=18, cls=1
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-01410: 无效的 ROWID
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-01410: 无效的 ROWID

由于obj id为18(obj$)的对象和对应的数据库中实际block存储的表的block为224(aud$)不匹配,从而出现该错误,通过分析是由于i_obj1记录错误导致该问题,通过修复该记录之后,数据实现完美导出.

类似system文件坏块案例有:
通过拷贝block实现system文件大量坏块恢复
记录一次system表空间坏块(ORA-01578)数据库恢复
SYSTEM表空间坏块恢复—C_TS#对象坏块恢复(file 1 block 60)
file 1 block 128 corrupted/坏块恢复—system rollback坏块修复

Oracle 19c故障恢复

Posted on 2020 年 03 月 10 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle 19c故障恢复

有客户找到我们,他们的oracle 19c数据库由于异常断电,导致启动异常,经过一系列恢复之后,依旧无法解决问题,请求我们给予支持.通过我们的Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check),获取数据库当前信息如下:
数据库版本为19C并且安装了19.5.0.0.191015 (30125133)补丁

数据库使用pdb

数据库启动成功后,一会就crash掉

2020-03-10T01:44:41.018032+08:00
Pluggable database RACBAK opened read write
2020-03-10T01:44:41.018996+08:00
Pluggable database RAC opened read write
2020-03-10T01:44:51.244050+08:00
Completed: ALTER PLUGGABLE DATABASE ALL OPEN
Starting background process CJQ0
Completed: ALTER DATABASE OPEN
2020-03-10T01:44:51.317085+08:00
CJQ0 started with pid=224, OS id=32581 
2020-03-10T01:44:56.067043+08:00
Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_j001_32588.trc  (incident=1095281) (PDBNAME=RAC):
ORA-00600: internal error code, arguments: [4193], [27733], [27754], [], [], [], [], [], [], [], [], []
RAC(4):Incident details in: /opt/oracle/diag/rdbms/XFF/XFF/incident/incdir_1095281/XFF_j001_32588_i1095281.trc
RAC(4):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2020-03-10T01:44:56.073112+08:00
RAC(4):*****************************************************************
RAC(4):An internal routine has requested a dump of selected redo.
RAC(4):This usually happens following a specific internal error, when
RAC(4):analysis of the redo logs will help Oracle Support with the
RAC(4):diagnosis.
RAC(4):It is recommended that you retain all the redo logs generated (by
RAC(4):all the instances) during the past 12 hours, in case additional
RAC(4):redo dumps are required to help with the diagnosis.
RAC(4):*****************************************************************
2020-03-10T01:44:56.079228+08:00
Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_j002_32590.trc  (incident=1095289) (PDBNAME=RAC):
ORA-00600: internal error code, arguments: [4193], [2633], [2638], [], [], [], [], [], [], [], [], []
RAC(4):Incident details in: /opt/oracle/diag/rdbms/XFF/XFF/incident/incdir_1095289/XFF_j002_32590_i1095289.trc
RAC(4):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2020-03-10T01:44:56.085068+08:00
RAC(4):*****************************************************************
RAC(4):An internal routine has requested a dump of selected redo.
RAC(4):This usually happens following a specific internal error, when
RAC(4):analysis of the redo logs will help Oracle Support with the
RAC(4):diagnosis.
RAC(4):It is recommended that you retain all the redo logs generated (by
RAC(4):all the instances) during the past 12 hours, in case additional
RAC(4):redo dumps are required to help with the diagnosis.
RAC(4):*****************************************************************
2020-03-10T01:44:56.115765+08:00
Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_j004_32594.trc  (incident=1095305) (PDBNAME=RAC):
ORA-00600: internal error code, arguments: [4193], [63532], [63537], [], [], [], [], [], [], [], [], []
RAC(4):Incident details in: /opt/oracle/diag/rdbms/XFF/XFF/incident/incdir_1095305/XFF_j004_32594_i1095305.trc
RAC(4):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2020-03-10T01:46:48.202213+08:00
RAC(4):Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
RAC(4):  Mem# 0: /opt/oracle/oradata/XFF/redo02.log
RAC(4):Block recovery completed at rba 0.0.0, scn 0x0000000d3675e48e
RAC(4):DDE: Problem Key 'ORA 600 [4193]' was completely flood controlled (0x6)
Further messages for this problem key will be suppressed for up to 10 minutes
2020-03-10T01:46:48.384040+08:00
Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_clmn_31741.trc:
ORA-00600: internal error code, arguments: [4193], [27733], [27754], [], [], [], [], [], [], [], [], []
Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_clmn_31741.trc  (incident=1093505) (PDBNAME=CDB$ROOT):
ORA-501 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /opt/oracle/diag/rdbms/XFF/XFF/incident/incdir_1093505/XFF_clmn_31741_i1093505.trc
2020-03-10T01:46:49.264624+08:00
USER (ospid: 31741): terminating the instance due to ORA error 501
2020-03-10T01:46:49.280664+08:00
System state dump requested by (instance=1, osid=31741 (CLMN)), summary=[abnormal instance termination].
System State dumped to trace file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_diag_31759.trc
2020-03-10T01:46:53.156926+08:00
ORA-00501: CLMN process terminated with error
2020-03-10T01:46:53.157103+08:00
Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_diag_31759.trc:
ORA-00501: CLMN process terminated with error
2020-03-10T01:46:53.157211+08:00
Dumping diagnostic data in directory=[cdmp_20200310014649], requested by (instance=1, osid=31741 (CLMN)), 
summary=[abnormal instance termination].

通过报错信息判断,数据库open之后(特别是pdb 4 open之后),开始报ORA-600 4193错误.然后由于CLMN进程异常,最后数据库crash.对于这类故障,因为使用的pdb,而且是由于pdb的undo异常导致数据库启动之后crash,可以通过对于pdb进行特殊处理,从而实现数据库启动之后不再crash.