又一例asm disk 加入vg故障

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:又一例asm disk 加入vg故障

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

又一客户把asm disk加入到vg,并且扩容到lv中
asm-disk-vg


通过asm层面查看
20220914191351
20220914191418

ASMCMD> lsdsk
Path
/dev/asmdisk/asm-arch01
/dev/asmdisk/asm-data01
/dev/asmdisk/asm-ocr01
/dev/asmdisk/asm-ocr02
/dev/asmdisk/asm-ocr03
dbsrv2-> ls -l /dev/asm*
/dev/asm:
total 0

/dev/asmdisk:
total 0
lrwxrwxrwx 1 root root 6 Sep 14 17:39 asm-arch01 -> ../sdb
lrwxrwxrwx 1 root root 6 Sep 14 17:35 asm-data01 -> ../sda
lrwxrwxrwx 1 root root 6 Sep 11 09:11 asm-ocr01 -> ../sdc
lrwxrwxrwx 1 root root 6 Sep 11 09:11 asm-ocr02 -> ../sdd
lrwxrwxrwx 1 root root 6 Sep 11 09:11 asm-ocr03 -> ../sde

对于这类情况,由于客户的系统是ext4,根据这个文件系统特性,每隔2G会有一点破坏,最终数据恢复效果看运气,运气好直接通过元数据恢复出来所有数据文件,然后open库,然后不好可能需要底层碎片等,参见类似恢复:
asm disk被加入vg恢复
asm disk 磁盘部分被清空恢复
文件系统重新分区oracle恢复
删除分区 oracle asm disk 恢复
pvcreate asm disk导致asm磁盘组异常恢复
对于使用asm的客户,在对文件系统进行操作时,一定要注意asm disk,别弄错磁盘(把asm disk磁盘给误操作掉了),适当情况下linux平台可以考虑AFD(ASM FILTER DRIVER)

oracle启动报ORA-600 kdBlkCheckError故障解决

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:oracle启动报ORA-600 kdBlkCheckError故障解决

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库启动报ORA-600 kdBlkCheckError错误

SQL> alter database open ;
alter database open 
*
第 1 行出现错误:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [3], [144], [38504]
进程 ID: 17516
会话 ID: 14 序列号: 5

根据ORA-600 kdBlkCheckError的经验,这个错误是3号文件的144号block逻辑不一致导致.通过dbv检查该文件

Microsoft Windows [版本 10.0.19044.1949]
(c) Microsoft Corporation。保留所有权利。

C:\Users\XFF>dbv file=H:\BaiduNetdisk\oradata\XFF\UNDOTBS1.DBF

DBVERIFY: Release 11.2.0.4.0 - Production on 星期六 9月 17 10:51:32 2022

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = H:\BAIDUNETDISK\ORADATA\SPECTRA\UNDOTBS1.DBF
页 112 失败, 校验代码为 18018
Block Checking: DBA = 12583056, Block Type = System Managed Segment Header Block
ERROR: SMU Segment Header Corrupted.  Error Code = 38504
ktu4smck: SCN commited txn list is not sorted.
  previous txn slot=23, scn=0x0000.ee917d05
  offending txn slot=18, scn=0x0000.ee916272
  TRN CTL:: seq: 0x0c3f chd: 0x0017 ctl: 0x0018 inc: 0x00000000 nfb: 0x0001
            mgc: 0xb000 xts: 0x0068 flg: 0x0001 opt: 2147483646 (0x7ffffffe)
            uba: 0x00c087c8.0c3f.05 scn: 0x0000.ee9160d2
            Version: 0x01
  FREE BLOCK POOL::
    uba: 0x00c087c8.0c3f.05  ext: 0xe  spc: 0x1594
    uba: 0x00000000.0bfe.03  ext: 0x12 spc: 0x1eb8
    uba: 0x00000000.0b20.04  ext: 0x4  spc: 0x1d2e
    uba: 0x00000000.c6e5.01  ext: 0x2  spc: 0x1f84
    uba: 0x00000000.0000.00  ext: 0x0  spc: 0x0
  TRN TBL::
  index  state cflags  wrap#    uel         scn            dba            parent-xid    nub       bcl     cmt
  -----------------------------------------------------------------------------------------
   0x00  9 0x00  0x62878b  0xffe4  0x0000.ee917f13  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031369
   0x01 10 0x80  0x62887e  0x001f  0x0000.ee917efe  0x00c087d6  0x0000.000.00000000  0x00000001 0x00000000  38
   0x02  9 0x00  0x628871  0xffde  0x0000.ee917cce  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031373
   0x03  9 0x00  0x62865f  0x0000  0x0000.ee916279  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x04  9 0x00  0x628823  0x0011  0x0000.ee916083  0x00c087d4  0x0000.000.00000000  0x00000001 0x00000000  1663031371
   0x05  9 0x00  0x6287b3  0x0000  0x0000.ee917e56  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x06  9 0x00  0x628893  0x001c  0x0000.ee916288  0x00002338  0x0000.000.00000000  0x00000000 0x00000000  1663031371
   0x07  9 0x00  0x628820  0x0011  0x0000.ee917f66  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031373
   0x08  9 0x00  0x628833  0x000c  0x0000.ee917eaf  0x00c087d4  0x0000.000.00000000  0x00000001 0x00000000  1663031371
   0x09  9 0x00  0x628815  0x0000  0x0000.ee917e72  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x0a  9 0x00  0x628863  0x0002  0x0000.ee917cc1  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031373
   0x0b  9 0x00  0x62870c  0x0008  0x0000.ee916085  0x00c087d4  0x0000.000.00000000  0x00000001 0x00000000  1663031372
   0x0c  9 0x00  0x62881a  0x0005  0x0000.ee917ff2  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x0d  9 0x00  0x6289fb  0x000f  0x0000.ee917d1d  0x00c087d4  0x0000.000.00000000  0x00000001 0x00000000  1663031375
   0x0e  9 0x00  0x6287d8  0x000a  0x0000.ee91638a  0x00c087d6  0x0000.000.00000000  0x00000001 0x00000000  1663031371
   0x0f  9 0x00  0x62880c  0x001b  0x0000.ee91619e  0x00003003  0x0000.000.00000000  0x00000000 0x00000000  1663031370
   0x10  9 0x00  0x6287e6  0x0013  0x0000.ee9161fc  0x00c087d5  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x11  9 0x00  0x62863b  0x0019  0x0000.ee916354  0x00c087d5  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x12  9 0x00  0x6287d4  0x0010  0x0000.ee916272  0x00c087d6  0x0000.000.00000000  0x00000001 0x00000000  1663031371
   0x13  9 0x00  0x628470  0x0007  0x0000.ee9160b2  0x00c087d5  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x14  9 0x00  0x6287a4  0x001e  0x0000.ee91627f  0x00c087d6  0x0000.000.00000000  0x00000001 0x00000000  1663031372
   0x15  9 0x00  0x628797  0x000a  0x0000.ee9162bb  0x00c087c9  0x0000.000.00000000  0x00000001 0x00000000  1663031368
   0x16  9 0x00  0x6287ad  0x0005  0x0000.ee917f6c  0x00c087d4  0x0000.000.00000000  0x00000001 0x00000000  1663031371
   0x17  9 0x00  0x6287b5  0x0012  0x0000.ee917d05  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031373
   0x18  9 0x00  0x628719  0x000b  0x0000.ee916136  0x00c087d4  0x0000.000.00000000  0x00000001 0x00000000  1663031373
   0x19  9 0x00  0x628783  0x0006  0x0000.ee916363  0x00c087d5  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x1a  9 0x00  0x6287d8  0xffff  0x0000.ee917d97  0x00c087cb  0x0000.000.00000000  0x00000001 0x00000000  1663031375
   0x1b  9 0x00  0x6287d7  0x0022  0x0000.ee916043  0x00c087d5  0x0000.000.00000000  0x00000001 0x00000000  1663031373
   0x1c  9 0x00  0x62880e  0x0005  0x0000.ee917db7  0x00002338  0x0000.000.00000000  0x00000000 0x00000000  1663031373
   0x1d  9 0x00  0x6287b7  0x0003  0x0000.ee9161e1  0x00c087d4  0x0000.000.00000000  0x00000001 0x00000000  1663031373
   0x1e  9 0x00  0x6287f6  0x0015  0x0000.ee9162e6  0x00002338  0x0000.000.00000000  0x00000000 0x00000000  1663031368
   0x1f  9 0x00  0x6287ad  0x0003  0x0000.ee917eae  0x00003003  0x0000.000.00000000  0x00000000 0x00000000  1663031372
   0x20  9 0x00  0x6287b0  0x0003  0x0000.ee9163a5  0x0000133b  0x0000.000.00000000  0x00000000 0x00000000  1663031368
   0x21  9 0x00  0x62886a  0x0001  0x0000.ee916056  0x00c087d5  0x0000.000.00000000  0x00000001 0x00000000  1663031373
  EXT TRN CTL::
  usn: 2
  sp1:0x00000000 sp2:0x00000000 sp3:0x00000000 sp4:0x00000000
  sp5:0x00000000 sp6:0x00000000 sp7:0x00000000 sp8:0x00000000
  EXT TRN TBL::
index extflag  extHash  extSpare1  extSpare2
---------------------------------------------
   0x00  0x00000000 0x00000000 0x00000000  0x00000000
   0x01  0x00000000 0x00000000 0x00000000  0x00000000
   0x02  0x00000000 0x00000000 0x00000000  0x00000000
   0x03  0x00000000 0x00000000 0x00000000  0x00000000
   0x04  0x00000000 0x00000000 0x00000000  0x00000000
   0x05  0x00000000 0x00000000 0x00000000  0x00000000
   0x06  0x00000000 0x00000000 0x00000000  0x00000000
   0x07  0x00000000 0x00000000 0x00000000  0x00000000
   0x08  0x00000000 0x00000000 0x00000000  0x00000000
   0x09  0x00000000 0x00000000 0x00000000  0x00000000
   0x0a  0x00000000 0x00000000 0x00000000  0x00000000
   0x0b  0x00000000 0x00000000 0x00000000  0x00000000
   0x0c  0x00000000 0x00000000 0x00000000  0x00000000
   0x0d  0x00000000 0x00000000 0x00000000  0x00000000
   0x0e  0x00000000 0x00000000 0x00000000  0x00000000
   0x0f  0x00000000 0x00000000 0x00000000  0x00000000
   0x10  0x00000000 0x00000000 0x00000000  0x00000000
   0x11  0x00000000 0x00000000 0x00000000  0x00000000
   0x12  0x00000000 0x00000000 0x00000000  0x00000000
   0x13  0x00000000 0x00000000 0x00000000  0x00000000
   0x14  0x00000000 0x00000000 0x00000000  0x00000000
   0x15  0x00000000 0x00000000 0x00000000  0x00000000
   0x16  0x00000000 0x00000000 0x00000000  0x00000000
   0x17  0x00000000 0x00000000 0x00000000  0x00000000
   0x18  0x00000000 0x00000000 0x00000000  0x00000000
   0x19  0x00000000 0x00000000 0x00000000  0x00000000
   0x1a  0x00000000 0x00000000 0x00000000  0x00000000
   0x1b  0x00000000 0x00000000 0x00000000  0x00000000
   0x1c  0x00000000 0x00000000 0x00000000  0x00000000
   0x1d  0x00000000 0x00000000 0x00000000  0x00000000
   0x1e  0x00000000 0x00000000 0x00000000  0x00000000
   0x1f  0x00000000 0x00000000 0x00000000  0x00000000
   0x20  0x00000000 0x00000000 0x00000000  0x00000000
   0x21  0x00000000 0x00000000 0x00000000  0x00000000
页 144 失败, 校验代码为 38504

…………

DBVERIFY - 验证完成

检查的页总数: 161280
处理的页总数 (数据): 0
失败的页总数 (数据): 0
处理的页总数 (索引): 0
失败的页总数 (索引): 0
处理的页总数 (其他): 161277
处理的总页数 (段)  : 9
失败的总页数 (段)  : 0
空的页总数: 1
标记为损坏的总页数: 4
流入的页总数: 2
加密的总页数        : 0
最高块 SCN            : 4002695098 (0.4002695098)

C:\Users\XFF>

可以确认是由于SMU Segment Header异常,导致数据库无法正常启动,通过数据库层面设置,规避数据库启动访问该block,数据库正常启动正常,并顺利导出数据

Thu Sep 15 11:02:23 2022
ALTER DATABASE   MOUNT
Successful mount of redo thread 1, with mount id 2863639551
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Thu Sep 15 11:02:31 2022
alter database open upgrade
Thread 1 opened at log sequence 660107
  Current log# 2 seq# 660107 mem# 0: H:\BAIDUNETDISK\ORADATA\XFF\REDO02.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Undo initialization finished serial:0 start:74439375 end:74439375 diff:0 (0 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Completed: alter database open 

Oracle Recovery Tools 解决ORA-600 3020故障

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:Oracle Recovery Tools 解决ORA-600 3020故障

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

尝试recover datafile,部分文件报ORA-600 3020,其他文件recover成功

ALTER DATABASE RECOVER  datafile 1  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 3 Seq 24972 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_72232.trc  (incident=749532):
ORA-00600: 内部错误代码, 参数: [3020], [1], [272255], [4466559], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 1, block# 272255, file offset is 2230312960 bytes)
ORA-10564: tablespace SYSTEM
ORA-01110: 数据文件 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 383
Media Recovery failed with error 600
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 1  ...
Tue Aug 02 10:28:24 2022
Trace dumping is performing id=[cdmp_20220802102824]
Tue Aug 02 10:28:31 2022
ALTER DATABASE RECOVER  datafile 2  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 3 Seq 24972 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_72232.trc  (incident=749533):
ORA-00600: 内部错误代码, 参数: [3020], [2], [92323], [8480931], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 2, block# 92323, file offset is 756310016 bytes)
ORA-10564: tablespace SYSAUX
ORA-01110: 数据文件 2: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSAUX01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 12330
Media Recovery failed with error 600
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 2  ...

利用Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)检查文件头相关信息,发现recover 失败的两个文件异常
20220802163502


通过Oracle Recovery Tools工具进行修复
20220802105543

数据库recover 成功,并顺利open
20220802105622

Tue Aug 02 10:56:13 2022
ALTER DATABASE RECOVER  datafile 1  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 3 Seq 24972 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Completed: ALTER DATABASE RECOVER  datafile 1  
ALTER DATABASE RECOVER  datafile 2  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 3 Seq 24972 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Completed: ALTER DATABASE RECOVER  datafile 2  
Tue Aug 02 10:56:34 2022
alter database open 
Beginning crash recovery of 1 threads
 parallel recovery started with 7 processes
Started redo scan
Completed redo scan
 read 8504 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 24972, block 2, scn 177712270
Recovery of Online Redo Log: Thread 1 Group 3 Seq 24972 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 24972, block 17011, scn 177734679
 0 data blocks read, 0 data blocks written, 8504 redo k-bytes read
Tue Aug 02 10:56:35 2022
Thread 1 advanced to log sequence 24973 (thread open)
Thread 1 opened at log sequence 24973
  Current log# 1 seq# 24973 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Aug 02 10:56:35 2022
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
**********************************************************
WARNING: Files may exists in db_recovery_file_dest
that are not known to the database. Use the RMAN command
CATALOG RECOVERY AREA to re-catalog any such files.
If files cannot be cataloged, then manually delete them
using OS command.
One of the following events caused this:
1. A backup controlfile was restored.
2. A standby controlfile was restored.
3. The controlfile was re-created.
4. db_recovery_file_dest had previously been enabled and
   then disabled.
**********************************************************
replication_dependency_tracking turned off (no async multimaster replication found)
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: alter database open 

增加tempfile,导出数据该库恢复完成

又一例ORA-600 kcratr_nab_less_than_odr

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:又一例ORA-600 kcratr_nab_less_than_odr

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库启动报错ORA-600 kcratr_nab_less_than_odr

alter database open
Sat Jul 23 21:38:32 2022
Beginning crash recovery of 1 threads
 parallel recovery started with 19 processes
Sat Jul 23 21:38:33 2022
Started redo scan
Sat Jul 23 21:38:33 2022
Completed redo scan
 read 244 KB redo, 64 data blocks need recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl12c\orcl12c\trace\orcl12c_ora_5748.trc  (incident=309845):
ORA-00600: ??????, ??: [kcratr_nab_less_than_odr], [1], [10343], [67442], [67454], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:34 2022
Slave encountered ORA-10388 exception during crash recovery
Sat Jul 23 21:38:38 2022
Aborting crash recovery due to error 600
Sat Jul 23 21:38:38 2022
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl12c\orcl12c\trace\orcl12c_ora_5748.trc:
ORA-00600: ??????, ??: [kcratr_nab_less_than_odr], [1], [10343], [67442], [67454], [], [], [], [], [], [], []
Sat Jul 23 21:38:39 2022
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl12c\orcl12c\trace\orcl12c_ora_5748.trc:
ORA-00600: ??????, ??: [kcratr_nab_less_than_odr], [1], [10343], [67442], [67454], [], [], [], [], [], [], []
ORA-600 signalled during: alter database open...

这个错误比较简单,参考:
12c启动报kcratr_nab_less_than_odr
ORA-600 kcratr_nab_less_than_odr故障解决
解决该问题之后,数据库启动报ORA-600 4194错误

Mon Jul 25 12:18:04 2022
SMON: enabling tx recovery
Starting background process SMCO
Mon Jul 25 12:18:05 2022
SMCO started with pid=26, OS id=8164 
Mon Jul 25 12:18:06 2022
Database Characterset is ZHS16GBK
ORA-00600: ??????, ??: [4194], [46], [44], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Jul 25 12:18:14 2022
Doing block recovery for file 5 block 1267
Mon Jul 25 12:18:14 2022
Resuming block recovery (PMON) for file 5 block 1267
Block recovery from logseq 1, block 67 to scn 217083444
Mon Jul 25 12:18:15 2022
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL12C\REDO01.LOG
Block recovery stopped at EOT rba 1.68.16
Block recovery completed at rba 1.68.16, scn 0.217083444
Doing block recovery for file 5 block 272
Resuming block recovery (PMON) for file 5 block 272
Block recovery from logseq 1, block 67 to scn 217083443
Mon Jul 25 12:18:18 2022
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL12C\REDO01.LOG
Block recovery completed at rba 1.68.16, scn 0.217083444
Mon Jul 25 12:18:19 2022
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl12c\orcl12c\trace\orcl12c_smon_7100.trc:
ORA-01595: 释放区 (5) 回退段 (10) 时出错
ORA-00600: ??????, ??: [4194], [46], [44], [], [], [], [], [], [], [], [], []
Mon Jul 25 12:18:19 2022
No Resource Manager plan active
Mon Jul 25 12:18:23 2022
Sweep [inc][317000]: completed
Sweep [inc2][317000]: completed
Starting background process FBDA
Mon Jul 25 12:18:40 2022
FBDA started with pid=28, OS id=7828 
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl12c\orcl12c\trace\orcl12c_ora_5804.trc  (incident=317056):
ORA-00600: 内部错误代码, 参数: [4194], [46], [44], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Jul 25 12:18:53 2022
Doing block recovery for file 5 block 1267
Resuming block recovery (PMON) for file 5 block 1267
Block recovery from logseq 1, block 67 to scn 217083444
Mon Jul 25 12:18:53 2022
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL12C\REDO01.LOG
Block recovery completed at rba 1.68.16, scn 0.217083454
Doing block recovery for file 5 block 272
Resuming block recovery (PMON) for file 5 block 272
Block recovery from logseq 1, block 67 to scn 217083454
Mon Jul 25 12:18:54 2022
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL12C\REDO01.LOG
Block recovery completed at rba 1.69.16, scn 0.217083455
Mon Jul 25 12:18:55 2022
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl12c\orcl12c\trace\orcl12c_ora_5804.trc:
ORA-00600: 内部错误代码, 参数: [4194], [46], [44], [], [], [], [], [], [], [], [], []
Mon Jul 25 12:18:55 2022
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl12c\orcl12c\trace\orcl12c_ora_5804.trc:
ORA-00600: 内部错误代码, 参数: [4194], [46], [44], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 5804): terminating the instance due to error 600
Mon Jul 25 12:19:07 2022
Instance terminated by USER, pid = 5804
ORA-1092 signalled during: alter database open resetlogs...

该错误也比较简单,对异常undo段进行处理即可,参考类似操作:How to resolve ORA-600 [4194] errors

存储重启,oracle无法启动故障处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:存储重启,oracle无法启动故障处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户由于机房要停电,正常关闭两个节点数据库,通过数据库alert日志均可看到类似如下记录,证明数据库确实是正常shutdown immediate
1
2
然后关闭存储,启动存储之后发现数据库无法正常启动(数据scn不一致).相关信息如下:
20220707175525


最初报ORA-214错

that ORACLE_BASE be set in the environment
Wed Jul 06 00:50:02 2022
ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=10.10.10.10)(PORT=1521))' SCOPE=MEMORY SID='xffdb2';
ALTER DATABASE MOUNT /* db agent *//* {1:42392:203} */
This instance was first to mount
NOTE: Loaded library: System 
SUCCESS: diskgroup DATA1 was mounted
SUCCESS: diskgroup DATA2 was mounted
ORA-214 signalled during: ALTER DATABASE MOUNT /* db agent *//* {1:42392:203} */...
NOTE: dependency between database xffdb and diskgroup resource ora.DATA1.dg is established
NOTE: dependency between database xffdb and diskgroup resource ora.DATA2.dg is established

提示ctl不存在,通过处理之后报ORA-600 2131错误

Wed Jul 06 01:55:45 2022
ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=10.10.10.10)(PORT=1521))' SCOPE=MEMORY SID='xffdb2';
ALTER DATABASE MOUNT /* db agent *//* {1:42392:663} */
This instance was first to mount
NOTE: Loaded library: System 
SUCCESS: diskgroup DATA1 was mounted
SUCCESS: diskgroup DATA2 was mounted
NOTE: dependency between database xffdb and diskgroup resource ora.DATA1.dg is established
NOTE: dependency between database xffdb and diskgroup resource ora.DATA2.dg is established
Errors in file /u01/app/oracle/diag/rdbms/xffdb/xffdb2/trace/xffdb2_ora_47746.trc  (incident=576488):
ORA-00600: internal error code, arguments: [2131], [33], [32], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xffdb/xffdb2/incident/incdir_576488/xffdb2_ora_47746_i576488.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ORA-600 signalled during: ALTER DATABASE MOUNT /* db agent *//* {1:42392:663} */...

重建控制文件后恢复报错

Parallel Media Recovery started with 127 slaves
ORA-279 signalled during: ALTER DATABASE RECOVER  database using BACKUP CONTROLFILE  ...
Wed Jul 06 02:41:04 2022
ALTER DATABASE RECOVER    LOGFILE '+DATA3/xffdb/archivelog/2022_07_05/thread_2_seq_40889.18030.1109269215'  
Media Recovery Log +DATA3/xffdb/archivelog/2022_07_05/thread_2_seq_40889.18030.1109269215
Wed Jul 06 02:41:04 2022
Errors with log +DATA3/xffdb/archivelog/2022_07_05/thread_2_seq_40889.18030.1109269215
Errors in file /u01/app/oracle/diag/rdbms/xffdb/xffdb1/trace/xffdb1_pr00_96503.trc:
ORA-00325: archived log for thread 1, wrong thread # 2 in header
ORA-00334: archived log: '+DATA3/xffdb/archivelog/2022_07_05/thread_2_seq_40889.18030.1109269215'
ORA-325 signalled during: ALTER DATABASE RECOVER    LOGFILE '+DATA3/thread_2_seq_40889.18030.1109269215'  ...
ALTER DATABASE RECOVER CANCEL 
Media Recovery Canceled
Completed: ALTER DATABASE RECOVER CANCEL 
…………
Wed Jul 06 02:22:25 2022
ALTER DATABASE RECOVER  DATABASE  
Media Recovery Start
 started logmerger process
Only allocated 127 recovery slaves (requested 128)
Parallel Media Recovery started with 127 slaves
Wed Jul 06 02:22:28 2022
Errors in file /u01/app/oracle/diag/rdbms/xffdb/xffdb1/trace/xffdb1_pr00_77044.trc:
ORA-00313: open failed for members of log group 7 of thread 1
Media Recovery failed with error 313
Errors in file /u01/app/oracle/diag/rdbms/xffdb/xffdb1/trace/xffdb1_pr00_77044.trc:
ORA-00283: recovery session canceled due to errors
ORA-00313: open failed for members of log group 7 of thread 1
Wed Jul 06 02:22:28 2022
Errors in file /u01/app/oracle/diag/rdbms/xffdb/xffdb1/trace/xffdb1_m000_77318.trc:
ORA-00322: log 4 of thread 2 is not current copy
ORA-00312: online log 4 thread 2: '+DATA3/xffdb/onlinelog/group_4.16148.1107795635'
Errors in file /u01/app/oracle/diag/rdbms/xffdb/xffdb1/trace/xffdb1_m000_77318.trc:
ORA-00322: log 7 of thread 1 is not current copy
ORA-00312: online log 7 thread 1: '+DATA3/xffdb/onlinelog/group_7.18959.1107796013'
Errors in file /u01/app/oracle/diag/rdbms/xffdb/xffdb1/trace/xffdb1_m000_77318.trc:
ORA-00314: log 9 of thread 1, expected sequence# 133495 doesn't match 133490
ORA-00312: online log 9 thread 1: '+DATA3/xffdb/onlinelog/group_9.3142.1107796071'
Checker run found 208 new persistent data failures
ORA-10877 signalled during: ALTER DATABASE RECOVER  DATABASE  ...
…………
Only allocated 127 recovery slaves (requested 128)
Parallel Media Recovery started with 127 slaves
ORA-279 signalled during: ALTER DATABASE RECOVER  database using backup controlfile  ...
Wed Jul 06 06:15:26 2022
ALTER DATABASE RECOVER    LOGFILE '+DATA3/xffdb/onlinelog/group_4.16442.1107795653'  
Media Recovery Log +DATA3/xffdb/onlinelog/group_4.16442.1107795653
ORA-279 signalled during: ALTER DATABASE RECOVER    LOGFILE '+DATA3/xffdb/onlinelog/group_4.16442.1107795653'  ...
Wed Jul 06 06:15:43 2022
ALTER DATABASE RECOVER    LOGFILE '+DATA3/xffdb/onlinelog/group_7.18959.1107796013'  
Media Recovery Log +DATA3/xffdb/onlinelog/group_7.18959.1107796013
Wed Jul 06 06:15:50 2022
Errors with log +DATA3/xffdb/onlinelog/group_7.18959.1107796013
Wed Jul 06 06:15:50 2022
Errors in file /u01/app/oracle/diag/rdbms/xffdb/xffdb1/trace/xffdb1_pr29_306479.trc  (incident=961030):
ORA-00600: internal error code, arguments: [6102], [13], [17], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Wed Jul 06 06:15:54 2022
Sweep [inc][961030]: completed
Sweep [inc2][961030]: completed
Slave exiting with ORA-10562 exception
Errors in file /u01/app/oracle/diag/rdbms/xffdb/xffdb1/trace/xffdb1_pr29_306479.trc:
ORA-10562: Error occurred while applying redo to data block (file# 159, block# 3591756)
ORA-10564: tablespace LIS
ORA-01110: data file 159: '+DATA1/xffdb/datafile/lis.379.1080445903'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 138875
ORA-00600: internal error code, arguments: [6102], [13], [17], [], [], [], [], [], [], [], [], []
Wed Jul 06 06:15:59 2022
Recovery Slave PR29 previously exited with exception 10562

基于上述情况,很可能是由于存储重启之后,cache或者某些数据没有写入到数据文件和redo中,数据库重启之后redo不是最新的[ORA-00322错误可以证明,],数据文件也需要进行恢复(不是数据库正常关闭之后该有的情况),而且redo和数据文件还不一致[ORA-00600 6102可以证明],对于类似这样的情况,只能尝试强制打开数据库,报ORA-600 2663

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2663], [1393], [4159455578],
[1393], [4160374753], [], [], [], [], [], [], []
Process ID: 357910
Session ID: 1585 Serial number: 7
Wed Jul 06 06:57:25 2022
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/xffdb/xffdb1/trace/xffdb1_ora_357910.trc  (incident=1056360):
ORA-00600: internal error code, arguments: [2663], [1393],[4159455578],[1393],[4160374753],[], [], [], []
Redo thread 2 internally disabled at seq 1 (CKPT)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/xffdb/xffdb1/trace/xffdb1_arc3_360348.trc:
ORA-00600: internal error code, arguments: [ORA_NPI_ERROR],[600], 
  [ORA-00600: internal error code, arguments: [kffbAddBlk04]
Unable to create archive log file '+DATA3'
ARC3: Error 19504 Creating archive log file to '+DATA3'
ARCH: Archival error occurred on a closed thread. Archiver continuing
ORACLE Instance xffdb1 - Archival Error. Archiver continuing.
ARCH: Archival error occurred on a closed thread. Archiver continuing
ORACLE Instance xffdb1 - Archival Error. Archiver continuing.
Wed Jul 06 06:57:34 2022
Errors in file /u01/app/oracle/diag/rdbms/xffdb/xffdb1/incident/incdir_1056360/xffdb1_ora_357910_i1056360.trc:
ORA-00339: archived log does not contain any redo
ORA-00334: archived log: '+DATA1/xffdb/onlinelog/group_4.424.1109314453'
ORA-00600: internal error code, arguments: [2663], [1393], [4159455578], [1393], [4160374753], [], [],
Wed Jul 06 06:57:34 2022
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/xffdb/xffdb1/trace/xffdb1_ora_357910.trc:
ORA-00600: internal error code, arguments: [2663], [1393], [4159455578], [1393], [4160374753], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/xffdb/xffdb1/trace/xffdb1_ora_357910.trc:
ORA-00600: internal error code, arguments: [2663], [1393], [4159455578], [1393], [4160374753], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 357910): terminating the instance due to error 600
Instance terminated by USER, pid = 357910
ORA-1092 signalled during: alter database open resetlogs...
opiodr aborting process unknown ospid (357910) as a result of ORA-1092
Wed Jul 06 06:57:35 2022
ORA-1092 : opitsk aborting process

该错误比较常见,参考:ORA-600 2663,也可以利用我的Patch_SCN小工具快速解决,后续数据库报ORA-03113错

SQL> alter database open ;
alter database open 
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 369324
Session ID: 1585 Serial number: 1

查看alert日志,确认具体报错为kgegpa

Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Jul 06 07:17:08 2022
SMON: enabling cache recovery
ARC1: Archiving disabled thread 2 sequence 1
Archived Log entry 1 added for thread 1 sequence 1 ID 0x36317f52 dest 1:
Archived Log entry 2 added for thread 1 sequence 2 ID 0x36317f52 dest 1:
Archived Log entry 3 added for thread 2 sequence 1 ID 0x0 dest 1:
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x4D562123][PC:0x983CDD6,kgegpa()+40][flags: 0x0,count:1]
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x4D562123][PC:0x983B84A, kgebse()+776][flags: 0x2,count:2]
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x4D562123][PC:0x983B84A, kgebse()+776][flags: 0x2,count:2]
Wed Jul 06 07:17:11 2022
PMON (ospid: 377647): terminating the instance due to error 397

该问题有过类似的案例通过处理数据库open成功:
在数据库恢复遭遇ORA-07445 kgegpa错误
Exception [type: SIGSEGV, Address not mapped to object] [] [ kgegpa()+36]