Category Archives: Oracle备份恢复

非归档数据库异常恢复一例

Posted on by

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：非归档数据库异常恢复一例

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

由于存储故障，数据库为非归档模式，通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)收集信息确认数据库redo异常
2

Thu Mar 28 11:36:13 2019
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log /u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397869_%u_.arc
Thu Mar 28 11:36:13 2019
Errors with log /u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397869_%u_.arc
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr00_17611.trc:
ORA-00308:cannot open archived log
    '/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397869_%u_.arc'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log /u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397869_%u_.arc
Errors with log /u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397869_%u_.arc
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr00_17611.trc:
ORA-00308:cannot open archived log
   '/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397869_%u_.arc'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...

Thu Mar 28 11:38:44 2019
ALTER DATABASE RECOVER  datafile 5,6  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 3 Seq 5397870 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/orcl/redo03.log
ORA-279 signalled during: ALTER DATABASE RECOVER  datafile 5,6  ...
Thu Mar 28 11:39:08 2019
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log /u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397870_%u_.arc
Errors with log /u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397870_%u_.arc
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER CANCEL 
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_17605.trc  (incident=365041):
ORA-00600: internal error code, arguments: [3051], [82], [], [], [], [], [], [], [], [], [], []
ORA-600 signalled during: ALTER DATABASE RECOVER CANCEL ...

通过屏蔽一致性,强制打开库,报kgegpa错误,实例启动失败

Database Characterset is ZHS16GBK
No Resource Manager plan active
Exception[type:SIGSEGV, Address not mapped to object][ADDR:0x319C0CF3] [PC:0x2297740, kgegpa()+40] [flags: 0x0, count:1]
Exception[type:SIGSEGV, Address not mapped to object][ADDR:0x319C0CF3] [PC:0x229596B, kgebse()+279][flags: 0x2, count:2]
Exception[type:SIGSEGV, Address not mapped to object][ADDR:0x319C0CF3] [PC:0x229596B, kgebse()+279][flags: 0x2, count:2]
Thu Mar 28 11:43:15 2019
PMON (ospid: 17939): terminating the instance due to error 397
Instance terminated by PMON, pid = 17939

处理上述错误相关undo,启动数据库报ORA-00600 4193,ORA-00600 4137, ORA-00600 6006

Thu Mar 28 11:50:37 2019
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p001_18267.trc  (incident=373059):
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_373059/orcl_p001_18267_i373059.trc
Stopping background process MMON
Trace dumping is performing id=[cdmp_20190328115038]
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_18247.trc  (incident=372995):
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_372995/orcl_smon_18247_i372995.trc
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_18247.trc  (incident=372996):
ORA-00600: internal error code, arguments: [4137], [34.22.4206895], [0], [0], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_372996/orcl_smon_18247_i372996.trc
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_18263.trc  (incident=373044):
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_373044/orcl_ora_18263_i373044.trc
ORACLE Instance orcl (pid = 16) - Error 600 encountered while recovering transaction (34, 22).
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_18247.trc:
ORA-00600: internal error code, arguments: [4137], [34.22.4206895], [0], [0], [], [], [], [], [], [], [], []

通过重建undo,相关报错消失,安排数据导出重建库

ORA-00322 ORA-00312恢复

Posted on by

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：ORA-00322 ORA-00312恢复

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库启动ORA-00322 ORA-00312错误，无法正常启动

Fri Mar 29 17:44:20 2019
ALTER DATABASE RECOVER  datafile 1  
Media Recovery Start
Serial Media Recovery started
Media Recovery failed with error 19909
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 1  ...
Fri Mar 29 17:44:20 2019
Errors in file c:\app\administrator\diag\rdbms\xff\xff\trace\xff_m000_5392.trc:
ORA-00322: log 1 of thread 1 is not current copy
ORA-00312: online log 1 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\xff\REDO01.LOG'
Errors in file c:\app\administrator\diag\rdbms\xff\xff\trace\xff_m000_5392.trc:
ORA-00322: log 2 of thread 1 is not current copy
ORA-00312: online log 2 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\xff\REDO02.LOG'
Errors in file c:\app\administrator\diag\rdbms\xff\xff\trace\xff_m000_5392.trc:
ORA-00322: log 3 of thread 1 is not current copy
ORA-00312: online log 3 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\xff\REDO03.LOG'

人工指定redo应用，报ORA-00600 3051错误

Fri Mar 29 17:56:33 2019
ALTER DATABASE RECOVER  datafile 1  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 2 Seq 27542 Reading mem 0
  Mem# 0: D:\XIFENFEI\REDO02.LOG
ORA-279 signalled during: ALTER DATABASE RECOVER  datafile 1  ...
Fri Mar 29 17:56:49 2019
ALTER DATABASE RECOVER    LOGFILE 'D:\xifenfei\REDO02.log'  
Media Recovery Log D:\xifenfei\REDO02.log
Errors with log D:\xifenfei\REDO02.log
ORA-363 signalled during: ALTER DATABASE RECOVER    LOGFILE 'D:\xifenfei\REDO02.log'  ...
ALTER DATABASE RECOVER CANCEL 
Errors in file c:\app\administrator\diag\rdbms\xff\xff\trace\xff_ora_8532.trc  (incident=147928):
ORA-00600: ??????, ??: [3051], [82], [], [], [], [], [], [], [], [], [], []
Incident details in: c:\app\administrator\diag\rdbms\xff\xff\incident\incdir_147928\xff_ora_8532_i147928.trc

比较明显redo无法正常应用,通过屏蔽数据库一致性,强制拉库

Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
  Mem# 0: D:\XIFENFEI\REDO01.LOG
Block recovery stopped at EOT rba 1.76.16
Block recovery completed at rba 1.76.16, scn 0.1073742057
Doing block recovery for file 3 block 272
Resuming block recovery (PMON) for file 3 block 272
Block recovery from logseq 1, block 72 to scn 1073742051
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
  Mem# 0: D:\XIFENFEI\REDO01.LOG
Block recovery completed at rba 1.72.16, scn 0.1073742052
Errors in file c:\app\administrator\diag\rdbms\xff\xff\trace\xff_smon_5144.trc:
ORA-01595: error freeing extent (16) of rollback segment (10))
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Fri Mar 29 17:59:12 2019
Errors in file c:\app\administrator\diag\rdbms\xff\xff\trace\xff_mmon_13928.trc  (incident=149097):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: c:\app\administrator\diag\rdbms\xff\xff\incident\incdir_149097\xff_mmon_13928_i149097.trc
Fri Mar 29 17:59:12 2019
Trace dumping is performing id=[cdmp_20190329175912]
Completed: alter database open resetlogs

通过重建undo,数据库open正常,安排导出数据导入数据,恢复完成

dul支持Oracle 19C

Posted on by

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：dul支持Oracle 19C

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

Oracle 19C

[oracle@localhost ~]$ ss

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Mar 2 07:02:18 2019
Version 19.2.0.0.0

Copyright (c) 1982, 2018, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.2.0.0.0

SQL> select name from v$datafile where rownum<3;

NAME
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/ORA19C/system01.dbf
/u01/app/oracle/oradata/ORA19C/sysaux01.dbf

dul支持19C

[root@localhost dul]# ./dul

Data UnLoader: 11.3.0.0.2 - Internal Only - on Sat Mar  2 08:26:11 2019
with 64-bit io functions and the decompression option

Copyright (c) 1994 2018 Bernard van Duijnen All rights reserved.

 Strictly Oracle Internal Use Only


DUL: Warning: Recreating file "dul.log"
Found db_id = 1054612630
Found db_name = ORA19C
DUL>  bootstrap;

DUL: Warning: Recreating file "dict.ddl"
Generating dict.ddl for version 12
 OBJ$: segobjno 18, file 1 block 240
 TAB$: segobjno 2, tabno 1, file 1  block 144
 COL$: segobjno 2, tabno 5, file 1  block 144
 USER$: segobjno 10, tabno 1, file 1  block 208
 TABPART$: segobjno 814, file 1 block 5424
 INDPART$: segobjno 819, file 1 block 5464
 TABCOMPART$: segobjno 836, file 1 block 5600
 INDCOMPART$: segobjno 841, file 1 block 5640
 TABSUBPART$: segobjno 826, file 1 block 5520
 INDSUBPART$: segobjno 831, file 1 block 5560
 IND$: segobjno 2, tabno 3, file 1  block 144
 ICOL$: segobjno 2, tabno 4, file 1  block 144
 LOB$: segobjno 2, tabno 6, file 1  block 144
 COLTYPE$: segobjno 2, tabno 7, file 1  block 144
 TYPE$: segobjno 740, tabno 1, file 1  block 4888
 COLLECTION$: segobjno 740, tabno 2, file 1  block 4888
 ATTRIBUTE$: segobjno 740, tabno 3, file 1  block 4888
 LOBFRAG$: segobjno 847, file 1 block 5688
 LOBCOMPPART$: segobjno 850, file 1 block 5720
 UNDO$: segobjno 15, file 1 block 224
 TS$: segobjno 6, tabno 2, file 1  block 176
 PROPS$: segobjno 127, file 1 block 1320
Running generated file "@dict.ddl" to unload the dictionary tables
. unloading table                      OBJ$
DUL: Warning: Recreating file "OBJ.ctl"
   72388 rows unloaded
. unloading table                      TAB$
DUL: Warning: Recreating file "TAB.ctl"
    2218 rows unloaded
. unloading table                      COL$
DUL: Warning: Recreating file "COL.ctl"
  123175 rows unloaded
. unloading table                     USER$
DUL: Warning: Recreating file "USER.ctl"
     127 rows unloaded
. unloading table                  TABPART$
DUL: Warning: Recreating file "TABPART.ctl"
     294 rows unloaded
. unloading table                  INDPART$
DUL: Warning: Recreating file "INDPART.ctl"
     195 rows unloaded
. unloading table               TABCOMPART$
DUL: Warning: Recreating file "TABCOMPART.ctl"
       1 row  unloaded
. unloading table               INDCOMPART$
DUL: Warning: Recreating file "INDCOMPART.ctl"
       0 rows unloaded
. unloading table               TABSUBPART$
DUL: Warning: Recreating file "TABSUBPART.ctl"
      32 rows unloaded
. unloading table               INDSUBPART$
DUL: Warning: Recreating file "INDSUBPART.ctl"
       0 rows unloaded
. unloading table                      IND$
DUL: Warning: Recreating file "IND.ctl"
    2878 rows unloaded
. unloading table                     ICOL$
DUL: Warning: Recreating file "ICOL.ctl"
    4958 rows unloaded
. unloading table                      LOB$
DUL: Warning: Recreating file "LOB.ctl"
     678 rows unloaded
. unloading table                  COLTYPE$
DUL: Warning: Recreating file "COLTYPE.ctl"
    2999 rows unloaded
. unloading table                     TYPE$
DUL: Warning: Recreating file "TYPE.ctl"
    5895 rows unloaded
. unloading table               COLLECTION$
DUL: Warning: Recreating file "COLLECTION.ctl"
    1384 rows unloaded
. unloading table                ATTRIBUTE$
DUL: Warning: Recreating file "ATTRIBUTE.ctl"
   15365 rows unloaded
. unloading table                  LOBFRAG$
DUL: Warning: Recreating file "LOBFRAG.ctl"
      14 rows unloaded
. unloading table              LOBCOMPPART$
DUL: Warning: Recreating file "LOBCOMPPART.ctl"
       0 rows unloaded
. unloading table                     UNDO$
DUL: Warning: Recreating file "UNDO.ctl"
      21 rows unloaded
. unloading table                       TS$
DUL: Warning: Recreating file "TS.ctl"
       6 rows unloaded
. unloading table                    PROPS$
DUL: Warning: Recreating file "PROPS.ctl"
      42 rows unloaded
Reading USER.dat 127 entries loaded
Reading OBJ.dat 72388 entries loaded and sorted 72388 entries
Reading TAB.dat 2201 entries loaded
Reading COL.dat
DUL: Notice: Increased the size of DC_COLUMNS from 100000 to 132768 entries
 123148 entries loaded and sorted 123148 entries
Reading TABPART.dat 294 entries loaded and sorted 294 entries
Reading TABCOMPART.dat 1 entries loaded and sorted 1 entries
Reading TABSUBPART.dat 32 entries loaded and sorted 32 entries
Reading INDPART.dat 195 entries loaded and sorted 195 entries
Reading INDCOMPART.dat 0 entries loaded and sorted 0 entries
Reading INDSUBPART.dat 0 entries loaded and sorted 0 entries
Reading IND.dat 2878 entries loaded
Reading LOB.dat 678 entries loaded
Reading ICOL.dat 4958 entries loaded
Reading COLTYPE.dat 2999 entries loaded
Reading TYPE.dat 5895 entries loaded
Reading ATTRIBUTE.dat 15365 entries loaded
Reading COLLECTION.dat 1384 entries loaded
Reading BOOTSTRAP.dat 60 entries loaded
Reading LOBFRAG.dat 14 entries loaded and sorted 14 entries
Reading LOBCOMPPART.dat 0 entries loaded and sorted 0 entries
Reading UNDO.dat 21 entries loaded
Reading TS.dat 6 entries loaded
Reading PROPS.dat 42 entries loaded
DUL> desc sys.tab$;
Table SYS.TAB$
obj#= 4, dataobj#= 2, ts#= 0, file#= 1, block#=144
      tab#= 1, segcols= 45, clucols= 1
Column information:
icol# 01 segcol# 01         OBJ# len   22 type  2 NUMBER(0)
icol# 02 segcol# 02     DATAOBJ# len   22 type  2 NUMBER(0)
icol# 03 segcol# 03          TS# len   22 type  2 NUMBER(0)
icol# 04 segcol# 04        FILE# len   22 type  2 NUMBER(0)
icol# 05 segcol# 05       BLOCK# len   22 type  2 NUMBER(0)
icol# 06 segcol# 06        BOBJ# len   22 type  2 NUMBER(0)
icol# 07 segcol# 07         TAB# len   22 type  2 NUMBER(0)
icol# 08 segcol# 08         COLS len   22 type  2 NUMBER(0)
icol# 09 segcol# 09      CLUCOLS len   22 type  2 NUMBER(0)
icol# 10 segcol# 10     PCTFREE$ len   22 type  2 NUMBER(0)
icol# 11 segcol# 11     PCTUSED$ len   22 type  2 NUMBER(0)
icol# 12 segcol# 12     INITRANS len   22 type  2 NUMBER(0)
icol# 13 segcol# 13     MAXTRANS len   22 type  2 NUMBER(0)
icol# 14 segcol# 14        FLAGS len   22 type  2 NUMBER(0)
icol# 15 segcol# 15       AUDIT$ len   38 type  1 VARCHAR2 cs 852(ZHS16GBK)
icol# 16 segcol# 16       ROWCNT len   22 type  2 NUMBER(0)
icol# 17 segcol# 17       BLKCNT len   22 type  2 NUMBER(0)
icol# 18 segcol# 18       EMPCNT len   22 type  2 NUMBER(0)
icol# 19 segcol# 19       AVGSPC len   22 type  2 NUMBER(0)
icol# 20 segcol# 20       CHNCNT len   22 type  2 NUMBER(0)
icol# 21 segcol# 21       AVGRLN len   22 type  2 NUMBER(0)
icol# 22 segcol# 22   AVGSPC_FLB len   22 type  2 NUMBER(0)
icol# 23 segcol# 23       FLBCNT len   22 type  2 NUMBER(0)
icol# 24 segcol# 24  ANALYZETIME len    7 type 12 DATE
icol# 25 segcol# 25   SAMPLESIZE len   22 type  2 NUMBER(0)
icol# 26 segcol# 26       DEGREE len   22 type  2 NUMBER(0)
icol# 27 segcol# 27    INSTANCES len   22 type  2 NUMBER(0)
icol# 28 segcol# 28      INTCOLS len   22 type  2 NUMBER(0)
icol# 29 segcol# 29   KERNELCOLS len   22 type  2 NUMBER(0)
icol# 30 segcol# 30     PROPERTY len   22 type  2 NUMBER(0)
icol# 31 segcol# 31     TRIGFLAG len   22 type  2 NUMBER(0)
icol# 32 segcol# 32       SPARE1 len   22 type  2 NUMBER(0)
icol# 33 segcol# 33       SPARE2 len   22 type  2 NUMBER(0)
icol# 34 segcol# 34       SPARE3 len   22 type  2 NUMBER(0)
icol# 35 segcol# 35       SPARE4 len 1000 type  1 VARCHAR2 cs 852(ZHS16GBK)
icol# 36 segcol# 36       SPARE5 len 1000 type  1 VARCHAR2 cs 852(ZHS16GBK)
icol# 37 segcol# 37       SPARE6 len    7 type 12 DATE
icol# 38 segcol# 38       SPARE7 len   22 type  2 NUMBER(0)
icol# 39 segcol# 39       SPARE8 len   22 type  2 NUMBER(0)
icol# 40 segcol# 40       SPARE9 len 1000 type  1 VARCHAR2 cs 852(ZHS16GBK)
icol# 41 segcol# 41      SPARE10 len 1000 type  1 VARCHAR2 cs 852(ZHS16GBK)
icol# 42 segcol# 42    ACDRFLAGS len   22 type  2 NUMBER(0)
icol# 43 segcol# 43   ACDRTSOBJ# len   22 type  2 NUMBER(0)
icol# 44 segcol# 44 ACDRDEFAULTTIME len   11 type 180 TIMESTAMP(9)
icol# 45 segcol# 45 ACDRROWTSINTCOL# len   22 type  2 NUMBER(0)

可能遇到错误

Reading TAB.dat
DUL: Error: string2ub8(618970019642690137449563136), Conversion to number (ub8) overflowed
DUL: Error: Number conversion error in file TAB.dat, line 22
DUL: Warning: Ignoring file TAB.dat cache
Reading COL.dat
DUL: Error: string2ub8(73786976294838206464), Conversion to number (ub8) overflowed
DUL: Error: Number conversion error in file COL.dat, line 114321
DUL: Warning: Ignoring file COL.dat cache

ORA-600 kcbzib_kcrsds_1报错

Posted on by

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：ORA-600 kcbzib_kcrsds_1报错

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库版本
12.2
客户存储故障，修复之后，多套库增加_allow_resetlogs_corruption隐含参数强制拉库出现都类似错误ORA-600 kcbzib_kcrsds_1,出现这个错误,一般都是由于数据库不一致强制拉库导致

2019-02-23T01:25:43.125621+08:00
 alter database open resetlogs
2019-02-23T01:25:43.231990+08:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 149251865354 time 
Clearing online redo logfile 1 +DG_XFF/xifenfei/ONLINELOG/group_1.258.983824407
Clearing online redo logfile 2 +DG_XFF/xifenfei/ONLINELOG/group_2.259.983824409
Clearing online redo logfile 3 +DG_XFF/xifenfei/ONLINELOG/group_3.266.983825461
Clearing online redo logfile 4 +DG_XFF/xifenfei/ONLINELOG/group_4.267.983825461
Clearing online log 1 of thread 1 sequence number 20749
Clearing online log 2 of thread 1 sequence number 20750
Clearing online log 3 of thread 2 sequence number 1371
Clearing online log 4 of thread 2 sequence number 1372
2019-02-23T01:25:44.669890+08:00
ALTER SYSTEM SET remote_listener=' xifenfeidb-cluster-scan:1521' SCOPE=MEMORY SID='xifenfei2';
2019-02-23T01:25:44.671436+08:00
ALTER SYSTEM SET listener_networks='' SCOPE=MEMORY SID='xifenfei2';
2019-02-23T01:25:46.990077+08:00
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 complete
Clearing online redo logfile 4 complete
Resetting resetlogs activation ID 3002369299 (0xb2f48513)
Online log +DG_XFF/xifenfei/ONLINELOG/group_1.258.983824407: Thread 1 Group 1 was previously cleared
Online log +DG_XFF/xifenfei/ONLINELOG/group_2.259.983824409: Thread 1 Group 2 was previously cleared
Online log +DG_XFF/xifenfei/ONLINELOG/group_3.266.983825461: Thread 2 Group 3 was previously cleared
Online log +DG_XFF/xifenfei/ONLINELOG/group_4.267.983825461: Thread 2 Group 4 was previously cleared
2019-02-23T01:25:47.137701+08:00
Setting recovery target incarnation to 2
2019-02-23T01:25:47.152393+08:00
This instance was first to open
Ping without log force is disabled:
  not an Exadata system.
Picked broadcast on commit scheme to generate SCNs
Endian type of dictionary set to little
2019-02-23T01:25:47.597502+08:00
Assigning activation ID 3019587675 (0xb3fb405b)
2019-02-23T01:25:47.625734+08:00
TT00: Gap Manager starting (PID:22467)
2019-02-23T01:25:47.910026+08:00
Thread 2 opened at log sequence 1
  Current log# 3 seq# 1 mem# 0: +DG_XFF/xifenfei/ONLINELOG/group_3.266.983825461
Successful open of redo thread 2
2019-02-23T01:25:47.911069+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
2019-02-23T01:25:47.971709+08:00
Sleep 5 seconds and then try to clear SRLs in 2 time(s)
2019-02-23T01:25:48.065008+08:00
start recovery: pdb 0, passed in flags x10 (domain enable 0) 
2019-02-23T01:25:48.065177+08:00
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
validate pdb 0, flags x10, valid 0, pdb flags x84 
* validated domain 0, flags = 0x80
Instance recovery complete: valid 1 (flags x10, recovery domain flags x80) 
2019-02-23T01:25:48.803746+08:00
Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei2/trace/xifenfei2_ora_21292.trc  (incident=128552):
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2019-02-23T01:25:49.947062+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2019-02-23T01:25:50.334684+08:00
Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei2/trace/xifenfei2_ora_21292.trc:
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2019-02-23T01:25:50.334880+08:00
Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei2/trace/xifenfei2_ora_21292.trc:
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
2019-02-23T01:25:50.362808+08:00
Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei2/trace/xifenfei2_ora_21292.trc  (incident=128553):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2019-02-23T01:25:51.521133+08:00
opiodr aborting process unknown ospid (21292) as a result of ORA-603

另外出现该错误之后，数据库再次恢复会出现类似,这个是由于open库的过程中导致控制文件损坏而出现的错误.

2019-02-23T22:52:39.390966+08:00
ALTER DATABASE RECOVER  database  
2019-02-23T22:52:39.391125+08:00
Media Recovery Start
 Started logmerger process
2019-02-23T22:52:39.471904+08:00
Media Recovery failed with error 16433
2019-02-23T22:52:39.996235+08:00
Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_m000_1593.trc:
ORA-01110: data file 11: '+DG_XFF/xifenfei/DATAFILE/ls_zh.274.984235699'
2019-02-23T22:52:40.224440+08:00
Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_m000_1593.trc:
ORA-01110: data file 12: '+DG_XFF/xifenfei/DATAFILE/sf_zh.275.984235715'
2019-02-23T22:52:40.459606+08:00
Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_m000_1593.trc:
ORA-01110: data file 13: '+DG_XFF/xifenfei/DATAFILE/tj_gl.276.984235729'
2019-02-23T22:52:40.574563+08:00
Recovery Slave PR00 previously exited with exception 283
ORA-283 signalled during: ALTER DATABASE RECOVER  database  ...

ORA-704 ORA-604 ORA-942 恢复

Posted on by

接到客户请求,oracle数据库停机重启维护之后,无法正常启动,请求我们给予协助
数据库启动报ORA-00704 ORA-00604 ORA-00942错误

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
PL/SQL Release 11.2.0.1.0 - Production
CORE    11.2.0.1.0      Production
TNS for 32-bit Windows: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production

SQL> startup open
ORACLE 例程已经启动。

Total System Global Area 1288949760 bytes
Fixed Size                  1376520 bytes
Variable Size             377491192 bytes
Database Buffers          897581056 bytes
Redo Buffers               12500992 bytes
数据库装载完毕。
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00942: table or view does not exist
进程 ID: 2756
会话 ID: 5 序列号: 9

alert日志报错

SMON: enabling cache recovery
Errors in file d:\app\administrator\diag\rdbms\xff\xff\trace\xff_ora_2756.trc:
ORA-00704: 引导程序进程失败
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-00942: 表或视图不存在
Errors in file d:\app\administrator\diag\rdbms\xff\xff\trace\xff_ora_2756.trc:
ORA-00704: 引导程序进程失败
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-00942: 表或视图不存在
Error 704 happened during db open, shutting down database
USER (ospid: 2756): terminating the instance due to error 704
Instance terminated by USER, pid = 2756
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (2756) as a result of ORA-1092
Fri Nov 30 12:51:26 2018
ORA-1092 : opitsk aborting process

根据这些年的恢复经验,恢复过相关错误的主要有:
ORA-01092 ORA-00704 ORA-00942
Oracle 11g丢失access$恢复方法
11.1.0.7版本也会出现access$表丢失导致数据库无法启动
总结主要两类:1. 由于某种bug导致access$表丢失的故障,另外一种是由于坏块导致数据库核心基表损坏引起,对于这个库进行分析

dbv 检查坏块

D:\APP\ADMINISTRATOR\ORADATA\ORCL> dbv file=system01.dbf

DBVERIFY: Release 10.2.0.3.0 - Production on 星期五 11月 30 15:11:24 2018

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

DBVERIFY - 开始验证: FILE = system01.dbf


DBVERIFY - 验证完成

检查的页总数: 93440
处理的页总数 (数据): 61979
失败的页总数 (数据): 0
处理的页总数 (索引): 13560
失败的页总数 (索引): 0
处理的页总数 (其它): 3067
处理的总页数 (段)  : 0
失败的总页数 (段)  : 0
空的页总数: 14834
标记为损坏的总页数: 0
流入的页总数: 0
最高块 SCN            : 659587683 (0.659587683)

通过dbv检查数据库,确定system没有坏块
10046跟踪数据库启动

SQL> startup mount;
ORACLE 例程已经启动。

Total System Global Area 1288949760 bytes
Fixed Size                  1376520 bytes
Variable Size             377491192 bytes
Database Buffers          897581056 bytes
Redo Buffers               12500992 bytes
数据库装载完毕。

SQL> oradebug setmypid
已处理的语句
SQL> alter session set db_file_multiblocK_read_count=1;

会话已更改。

SQL> ALTER SESSION SET EVENTS '704 trace name errorstack level 3';

会话已更改。

SQL> oradebug EVENT 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12
已处理的语句
SQL> oradebug TRACEFILE_NAME
d:\app\administrator\diag\rdbms\xff\xff\trace\xff_ora_1132.trc

--trace信息
PARSE ERROR #3:len=208 dep=1 uid=0 oct=9 lid=0 tim=5838844893 err=942
CREATE UNIQUE INDEX I_OBJ1 ON OBJ$(OBJ#,OWNER#,TYPE#) PCTFREE 10 INITRANS 2 MAXTRANS 255 
STORAGE (  INITIAL 64K NEXT 1024K MINEXTENTS 1 MAXEXTENTS 2147483645
 PCTINCREASE 0 OBJNO 36 EXTENTS (FILE 1 BLOCK 336))

*** 2018-11-30 13:18:07.593
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=3, mask=0x0)
----- Error Stack Dump -----
ORA-00704: 引导程序进程失败
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-00942: 表或视图不存在

通过这一步基本上可以判断是由于obj$表丢失导致数据库创建I_OBJ1 index不成功,从而使得数据库无法正常启动。通过一些技巧修复出来obj$表,尝试启动数据库

SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [4000], [6], [], [], [], [], [], [],
[], [], [], []
进程 ID: 836
会话 ID: 355 序列号: 6027

alert日志报错

Fri Nov 30 15:33:47 2018
SMON: enabling cache recovery
Errors in file d:\app\administrator\diag\rdbms\xff\xff\trace\xff_ora_836.trc  (incident=648694):
ORA-00600: 内部错误代码, 参数: [4000], [6], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\xff\xff\incident\incdir_648694\xff_ora_836_i648694.trc
Errors in file d:\app\administrator\diag\rdbms\xff\xff\trace\xff_ora_836.trc:
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [4000], [6], [], [], [], [], [], [], [], [], [], []
Errors in file d:\app\administrator\diag\rdbms\xff\xff\trace\xff_ora_836.trc:
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [4000], [6], [], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 836): terminating the instance due to error 704
Fri Nov 30 15:33:57 2018
Instance terminated by USER, pid = 836
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (836) as a result of ORA-1092

发现数据库启动报ORA-00704和ORA-00600: internal error code, arguments: [4000], [6], [], [], [], [], [], [],错误,根据以往经验,该问题是由于回滚段异常导致
分析trace信息

Dump continued from file: d:\app\administrator\diag\rdbms\xff\xff\trace\xff_ora_2124.trc
ORA-00600: 内部错误代码, 参数: [4000], [6], [], [], [], [], [], [], [], [], [], []

========= Dump for incident 651102 (ORA 600 [4000]) ========

*** 2018-11-30 15:46:32.125
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- Current SQL Statement for txff session (sql_id=6apq2rjyxmxpj) -----
select line#, sql_text from bootstrap$ where obj# != :1

----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
_skdstdst()+121      CALLrel  _kgdsdst()           E6B1614 2
_ksedst1()+93        CALLrel  _skdstdst()          E6B1614 0 1 436646 435BE2
                                                   436646
_ksedst()+49         CALLrel  _ksedst1()           0 1
_dbkedDefDump()+367  CALLrel  _ksedst()            0
2                                                  
_ksedmp()+44         CALLrel  _dbkedDefDump()      3 2
_ksfdmp()+56         CALLrel  _ksedmp()            3EB
_dbgexPhaseII()+164  CALLreg  00000000             C378A28 3EB
0                                                  
_dbgexProcessError(  CALLrel  _dbgexPhaseII()      E2B0454 E2B6E50 E6B6078
)+2061                                             
_dbgeExecuteForErro  CALLrel  _dbgexProcessError(  E2B0454 E2B6E50 1 0 E2B0454
r()+43                        )                    E2B6E50
__VInfreq__dbgePost  CALLrel  _dbgeExecuteForErro  E2B0454 E2B6E50 0 1 0
ErrorKGE()+260                r()                  
_dbkePostKGE_kgsf()  CALLrel  _dbgePostErrorKGE()  C378A28 E2BD274 258
+56                                                
_kgeade()+299        CALLreg  00000000             C378A28 E2BD274 258
_kgeriv_int()+79     CALLrel  _kgeade()            C378A28 C378B50 E2BD274 258 0
                                                   FA0 0 0 0 1 E6B68A8
_kgeriv()+22         CALLrel  _kgeriv_int()        C378A28 E2BD274 FA0 0 1
                                                   E6B68A8
_kgeasi()+107        CALLrel  _kgeriv()            C378A28 E2BD274 FA0 1 E6B68A8
__VInfreq__ktuGetUs  CALLrel  _kgeasi()            C378A28 E2BD274 FA0 2 1 0 6 0
egDba()+123                                        
_ktrgcm()+5147       CALLrel  _ktuGetUsegDba()     6 E6B6E78 0 0 E6B6F48 0
_ktrget2()+596       CALLrel  _ktrgcm()            F9FCEAC
_kdst_fetch()+816    CALLrel  _ktrget2()           F9FCEAC F9FCE24 303 0
_kdstf11001010000km  CALLrel  _kdst_fetch()        1 F9FCEA8 E6B71C8
()+2806                                            
_kdsttgr()+5944      CALLrel  _kdstf11001010000km  F9FCEA8 0 557B044C F9FCDF8
                              ()                   2F0C3A6 E6B7894
_qertbFetch()+767    CALLrel  _kdsttgr()           F9FCEA8 0 557B044C F9FCDF8
                                                   557B0498 2F0C3A6 E6B7894 1
_opifch2()+2729      CALLptr  00000000             E6E7228 0 0 2 26180001
_opifch()+53         CALLrel  _opifch2()           89 5 E6B7A04
_opiodr()+1248       CALLreg  00000000             5 2 E6B81FC
_rpidrus()+186       CALLrel  _opiodr()            5 2 E6B81FC 2
_rpidru()+90         CALLrel  _rpidrus()           E6B7D58
_rpiswu2()+557       CALLrel  _rpidru()            E6B813C
_rpidrv()+1242       CALLrel  _rpiswu2()           6D9A295C 0 6D9A29A8 2 E6B8184
                                                   0 6D9A2A28 0 0 544632 5448D6
                                                   E6B813C 8
_rpifch()+43         CALLrel  _rpidrv()            2 5 E6B81FC 8
_kqlbebs()+1213      CALLrel  _rpifch()            2 2 2 F013232 FA0 1 0 E6B8634
                                                   0 0 0 0 0
_kqlblfc()+175       CALLrel  _kqlbebs()           0 E6BBBD4
_adbdrv()+16992      CALLrel  _kqlblfc()           0 E6BBBD4
_opiexe()+13594      CALLrel  _adbdrv()            4A 6E6CBC48 6DDEDB6C E6BBD68
                                                   6D60697 6E6CBC48
_opiosq0()+6248      CALLrel  _opiexe()            4 0 E6BC734
_kpooprx()+277       CALLrel  _opiosq0()           3 E E6BC9A0 A4 0
_kpoal8()+632        CALLrel  _kpooprx()           E6BF0A4 E6BD420 1B 1 0 A4
_opiodr()+1248       CALLreg  00000000             5E 1C E6BF0A0
_ttcpip()+1051       CALLreg  00000000             5E 1C E6BF0A0 0
_opitsk()+1404       CALL???  00000000             C3832A8 5E E6BF0A0 0 E6BED30
                                                   E6BF1CC 53E52E 0 E6BF1F8
_opiino()+980        CALLrel  _opitsk()            0 0
_opiodr()+1248       CALLreg  00000000             3C 4 E6BFBF4
_opidrv()+1201       CALLrel  _opiodr()            3C 4 E6BFBF4 0
_sou2o()+55          CALLrel  _opidrv()            3C 4 E6BFBF4
_opimai_real()+124   CALLrel  _sou2o()             E6BFC04 3C 4 E6BFBF4
_opimai()+125        CALLrel  _opimai_real()       2 E6BFC2C
_OracleThreadStart@  CALLrel  _opimai()            2 E6BFF6C 7C9BA7F4 E6BFC34 0
4()+830                                            E6BFD04
7C82484C             CALLreg  00000000             E5BFF9C 0 0 E5BFF9C 0 E6BFFC4
00000000             CALL???  00000000             
 

--------------------- Binary Stack Dump ---------------------


Block header dump:  0x0040020b
 Object id on Block? Y
 seg/obj: 0x3b  csc: 0x00.27508136  itc: 1  flg: O  typ: 1 - DATA
     fsl: 0  fnx: 0x0 ver: 0x01
 
 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x0006.015.0005e9e9  0x00c0052c.916f.10  --U-    1  fsc 0x0000.27508263

报错比较明显该block比较异常,通过bbed修改相关block信息,相关处理参考:
重现ORA-600 4000异常
通过bbed解决ORA-00600[4000]案例
记录一次ORA-600 4000数据库故障恢复
ORACLE 8.1.7 数据库ORA-600 4000故障恢复

处理之后数据库正常open

Fri Nov 30 15:57:34 2018
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Fri Nov 30 15:57:38 2018
Starting background process QMNC
Fri Nov 30 15:57:38 2018
QMNC started with pid=23, OS id=1152 
Completed: alter database open