联系：手机/微信(+86 17813235971) QQ(107644445)
标题：ORA-15335: ASM metadata corruption detected in disk group ‘DATA’
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
asm磁盘组增加磁盘进行扩容之后报ORA-15335: ASM metadata corruption detected in disk group ‘DATA’和ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479],磁盘组dismount,然后mount之后立马dismount掉.
Tue Jun 29 09:19:09 2021 SQL> ALTER DISKGROUP DATA ADD DISK '/dev/raw/raw5' SIZE 102400M /* ASMCA */ NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (2,1) to disk (/dev/raw/raw5) NOTE: requesting all-instance membership refresh for group=2 NOTE: initializing header on grp 2 disk DATA_0001 NOTE: requesting all-instance disk validation for group=2 Tue Jun 29 09:19:11 2021 NOTE: skipping rediscovery for group 2/0xb0c845ce (DATA) on local instance. NOTE: requesting all-instance disk validation for group=2 NOTE: skipping rediscovery for group 2/0xb0c845ce (DATA) on local instance. NOTE: initiating PST update: grp = 2 Tue Jun 29 09:19:16 2021 GMON updating group 2 at 7 for pid 27, osid 25020 NOTE: PST update grp = 2 completed successfully NOTE: membership refresh pending for group 2/0xb0c845ce (DATA) GMON querying group 2 at 8 for pid 18, osid 3852 NOTE: cache opening disk 1 of grp 2: DATA_0001 path:/dev/raw/raw5 NOTE: Attempting voting file refresh on diskgroup DATA NOTE: Refresh completed on diskgroup DATA. No voting file found. GMON querying group 2 at 9 for pid 18, osid 3852 SUCCESS: refreshed membership for 2/0xb0c845ce (DATA) Tue Jun 29 09:19:20 2021 SUCCESS: ALTER DISKGROUP DATA ADD DISK '/dev/raw/raw5' SIZE 102400M /* ASMCA */ NOTE: starting rebalance of group 2/0xb0c845ce (DATA) at power 1 Starting background process ARB0 Tue Jun 29 09:19:21 2021 ARB0 started with pid=33, OS id=25176 NOTE: assigning ARB0 to group 2/0xb0c845ce (DATA) with 1 parallel I/O cellip.ora not found. Tue Jun 29 09:19:24 2021 NOTE: Attempting voting file refresh on diskgroup DATA NOTE: Refresh completed on diskgroup DATA. No voting file found. Tue Jun 29 09:19:46 2021 WARNING: cache read a corrupt block: group=2(DATA) dsk=0 blk=7 disk=0 (DATA_0000) incarn=3915953476 au=0 blk=7 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25176.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] NOTE: a corrupted block from group DATA was dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25176.trc Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25176.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] ERROR: cache failed to read group=2(DATA) dsk=0 blk=7 from disk(s): 0(DATA_0000) ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] NOTE: cache initiating offline of disk 0 group DATA NOTE: process _arb0_+asm1 (25176) initiating offline of disk 0.3915953476 (DATA_0000) with mask 0x7e in group 2 NOTE: initiating PST update: grp = 2, dsk = 0/0xe968b544, mask = 0x6a, op = clear Tue Jun 29 09:19:46 2021 GMON updating disk modes for group 2 at 10 for pid 33, osid 25176 ERROR: Disk 0 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 2) Tue Jun 29 09:19:46 2021 NOTE: cache dismounting (not clean) group 2/0xB0C845CE (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 25395, image: oracle@frsrac1 (B000) Tue Jun 29 09:19:46 2021 NOTE: halting all I/Os to diskgroup 2 (DATA) Tue Jun 29 09:19:46 2021 NOTE: LGWR doing non-clean dismount of group 2 (DATA) NOTE: LGWR sync ABA=11.10715 last written ABA 11.10715 WARNING: Offline for disk DATA_0000 in mode 0x7f failed. Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25176.trc (incident=54665): ORA-15335: ASM metadata corruption detected in disk group 'DATA' ORA-15130: diskgroup "DATA" is being dismounted ORA-15066: offlining disk "DATA_0000" in group "DATA" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_54665/+ASM1_arb0_25176_i54665.trc Tue Jun 29 09:19:46 2021 kjbdomdet send to inst 2 detach from dom 2, sending detach message to inst 2 Tue Jun 29 09:19:46 2021 List of instances: 1 2 Dirty detach reconfiguration started (new ddet inc 1, cluster inc 24) Global Resource Directory partially frozen for dirty detach * dirty detach - domain 2 invalid = TRUE 796 GCS resources traversed, 0 cancelled Dirty Detach Reconfiguration complete Tue Jun 29 09:19:46 2021 WARNING: dirty detached from domain 2 NOTE: cache dismounted group 2/0xB0C845CE (DATA) SQL> alter diskgroup DATA dismount force /* ASM SERVER:2965915086 */ Tue Jun 29 09:19:47 2021 ERROR: ORA-15130 thrown in ARB0 for group number 2 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25176.trc: ORA-15130: diskgroup "DATA" is being dismounted ORA-15335: ASM metadata corruption detected in disk group 'DATA' ORA-15130: diskgroup "DATA" is being dismounted ORA-15066: offlining disk "DATA_0000" in group "DATA" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] Tue Jun 29 09:19:47 2021 NOTE: stopping process ARB0 Tue Jun 29 09:19:47 2021 Sweep [inc][54665]: completed Tue Jun 29 09:19:47 2021 Sweep [inc2][54665]: completed NOTE: cache deleting context for group DATA 2/0xb0c845ce Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_3852.trc: ORA-15130: diskgroup "DATA" is being dismounted GMON dismounting group 2 at 11 for pid 27, osid 25395 NOTE: Disk DATA_0000 in mode 0x7f marked for de-assignment NOTE: Disk DATA_0001 in mode 0x7f marked for de-assignment SUCCESS: diskgroup DATA was dismounted SUCCESS: alter diskgroup DATA dismount force /* ASM SERVER:2965915086 */
通过kfed分析报错block,确认错误
kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL kfbh.datfmt: 2 ; 0x003: 0x02 kfbh.block.blk: 7 ; 0x004: blk=7 kfbh.block.obj: 2147483648 ; 0x008: disk=0 kfbh.check: 2183628676 ; 0x00c: 0x82278784 <<======该值错误,应该为:686982479 kfbh.fcn.base: 3430 ; 0x010: 0x00000d66 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdatb.aunum: 2240 ; 0x000: 0x000008c0 kfdatb.shrink: 448 ; 0x004: 0x01c0 kfdatb.ub2pad: 0 ; 0x006: 0x0000
通过修复该错误,并且禁止reblance操作[增加磁盘数据需要重新分布],mount磁盘组,然后open库,发现redo已经被覆盖(非归档),强制打开库报错
SQL> alter database open resetlogs; alter database open resetlogs * ERROR at line 1: ORA-00603: ORACLE server session terminated by fatal error ORA-00600: internal error code, arguments: [2662], [0], [2691201882], [0], [2691227745], [12583040], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [2662], [0], [2691201881], [0], [2691227745], [12583040], [], [], [], [], [], [] ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00600: internal error code, arguments: [2662], [0], [2691201879], [0], [2691227745], [12583040], [], [], [], [], [], [] Process ID: 25110 Session ID: 287 Serial number: 3
通过对scn进行处理,数据库顺利open
SQL> startup mount pfile='/tmp/pfile'; ORACLE instance started. Total System Global Area 5044088832 bytes Fixed Size 2261928 bytes Variable Size 1442843736 bytes Database Buffers 3590324224 bytes Redo Buffers 8658944 bytes Database mounted. SQL> alter database open; Database altered.
