时过境迁,以前恢复大量oracle 8/9版本的库,现在一套oracle 9i的库都比较稀奇了.今天恢复客户一套9.2.0.6的aix环境rac库,通过分析确认主要问题:
1. 重建控制文件，resetlogs库遗漏数据文件
2. 数据库启动主要报错ORA-600 2663和ORA-600 kclchkblk_4
Tue Nov 8 09:10:05 2022
Successfully onlined Undo Tablespace 1.
Dictionary check beginning
Tablespace 'TEMP' #2 found in data dictionary,
but not in the controlfile. Adding to controlfile.
File #84 found in data dictionary but not in controlfile.
Creating OFFLINE file 'MISSING00084' in the controlfile.
This file can no longer be recovered so it must be dropped.
Dictionary check complete
Tue Nov 8 09:10:05 2022
SMON: enabling tx recovery
Tue Nov 8 09:10:05 2022
Database Characterset is ZHS16GBK
Tue Nov 8 09:10:05 2022
Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_536662.trc:
ORA-00600: internal error code, arguments: [2663], [3301], [2638369768], [3301], [2640322622], [], [], []
Tue Nov 8 09:10:06 2022
Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_647352.trc:
ORA-00600: internal error code, arguments: [kclchkblk_4], [3301], [18446744072061740072],[3301],[18446744072052954088]
Tue Nov 8 09:10:06 2022
Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_536662.trc:
ORA-00600: internal error code, arguments: [2663], [3301], [2638369768], [3301], [2640322622], [], [], []
Error 600 happened during db open, shutting down database
USER: terminating instance due to error 600
Instance terminated by USER, pid = 536662
ORA-1092 signalled during: alter database open...
根据客户文件名称的规则,推算出来84号文件实际的文件名(因为使用的是lv[aix的hacmp管理的lv的裸设备方式]),通过dbv确认文件无坏块
DBVERIFY: Release 9.2.0.6.0 - Production on Sat May 13 16:44:09 2023
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
DBVERIFY - Verification starting : FILE = /dev/ra_txn_ind12.dbf
DBVERIFY - Verification complete
Total Pages Examined : 256000
Total Pages Processed (Data) : 0
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 299
Total Pages Failing (Index): 0
Total Pages Processed (Other): 13
Total Pages Processed (Seg) : 0
Total Pages Failing (Seg) : 0
Total Pages Empty : 255688
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Highest block SCN : 11177081099136 (2602.1576194944)
bbed验证文件该文件是否是84号文件
$ bbed blocksize=8192 filename='/dev/ra_txn_ind12.dbf'
Password:
BBED: Release 2.0.0.0.0 - Limited Production on Mon May 15 09:45:44 2023
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
************* !!! For Oracle Internal Use only !!! ***************
BBED> map
File: /dev/ra_txn_ind12.dbf (0)
Block: 1 Dba:0x00000000
------------------------------------------------------------
Data File Header
struct kcvfh, 608 bytes @0
ub4 tailchk @8188
BBED> p kcvfh
struct kcvfh, 608 bytes @0
struct kcvfhbfh, 20 bytes @0
ub1 type_kcbh @0 0x0b
ub1 frmt_kcbh @1 0x02
ub1 spare1_kcbh @2 0x00
ub1 spare2_kcbh @3 0x00
ub4 rdba_kcbh @4 0x15000001
ub4 bas_kcbh @8 0x00000000
ub2 wrp_kcbh @12 0x0000
ub1 seq_kcbh @14 0x01
ub1 flg_kcbh @15 0x04 (KCBHFCKV)
ub2 chkval_kcbh @16 0x1b4a
ub2 spare3_kcbh @18 0x0000
struct kcvfhhdr, 76 bytes @20
ub4 kccfhswv @20 0x09200000
ub4 kccfhcvn @24 0x08000000
ub4 kccfhdbi @28 0x05d15ccf
……
ub4 kccfhcsq @40 0x00525a20
ub4 kccfhfsz @44 0x0003e800
s_blkz kccfhbsz @48 0x00
ub2 kccfhfno @52 0x0054
ub2 kccfhtyp @54 0x0003
……
ub4 kcvfhrfn @528 0x00000054 ---确认是84号文件
……
通过bbed修改文件相关信息,然后尝试rename文件,但是recover datafile 84报错
Mon May 15 09:49:44 2023
alter database rename file '/u01/prod/proddb/9.2.0/dbs/MISSING00084' to '/dev/ra_txn_ind12.dbf'
Mon May 15 09:49:44 2023
Completed: alter database rename file '/u01/prod/proddb/9.2.0
Mon May 15 09:51:15 2023
ALTER DATABASE RECOVER datafile 84
Media Recovery Start
Mon May 15 09:51:15 2023
Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_467190.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
通过处理之后,数据库recover 正常,但是open报ORA-600 4193错误
Mon May 15 09:57:53 2023
ALTER DATABASE RECOVER DATABASE
Media Recovery Start
Mon May 15 09:57:53 2023
Recovery of Online Redo Log: Thread 1 Group 1 Seq 4 Reading mem 0
Mem# 0 errs 0: /dev/rlog01a.dbf
Mem# 1 errs 0: /dev/rlog01b.dbf
Media Recovery Complete
Completed: ALTER DATABASE RECOVER DATABASE
Mon May 15 09:59:24 2023
alter database open
Mon May 15 09:59:24 2023
Beginning crash recovery of 1 threads
Mon May 15 09:59:24 2023
Started redo scan
Mon May 15 09:59:24 2023
Completed redo scan
75 redo blocks read, 0 data blocks need recovery
Mon May 15 09:59:24 2023
Started recovery at
Thread 1: logseq 4, block 2, scn 3301.2638369687
Mon May 15 09:59:24 2023
Recovery of Online Redo Log: Thread 1 Group 1 Seq 4 Reading mem 0
Mem# 0 errs 0: /dev/rlog01a.dbf
Mem# 1 errs 0: /dev/rlog01b.dbf
Mon May 15 09:59:24 2023
Completed redo application
Mon May 15 09:59:24 2023
Ended recovery at
Thread 1: logseq 4, block 77, scn 3301.2638389765
0 data blocks read, 0 data blocks written, 75 redo blocks read
Crash recovery completed successfully
Mon May 15 09:59:25 2023
Thread 1 advanced to log sequence 5
Thread 1 opened at log sequence 5
Current log# 2 seq# 5 mem# 0: /dev/rlog02a.dbf
Current log# 2 seq# 5 mem# 1: /dev/rlog02b.dbf
Successful open of redo thread 1
Mon May 15 09:59:25 2023
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon May 15 09:59:25 2023
SMON: enabling cache recovery
Mon May 15 09:59:25 2023
ARC0: Media recovery disabled
Mon May 15 09:59:25 2023
Successfully onlined Undo Tablespace 1.
Dictionary check beginning
Tablespace 'TEMP' #2 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Mon May 15 09:59:25 2023
SMON: enabling tx recovery
Mon May 15 09:59:25 2023
Database Characterset is ZHS16GBK
Mon May 15 09:59:25 2023
Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_413872.trc:
ORA-00600: internal error code, arguments: [4193], [781], [6399], [], [], [], [], []
Mon May 15 09:59:25 2023
Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_844004.trc:
ORA-00600: internal error code, arguments: [4193], [56042], [1895], [], [], [], [], []
Mon May 15 09:59:26 2023
Doing block recovery for fno: 12 blk: 153
Mon May 15 09:59:26 2023
Doing block recovery for fno: 12 blk: 2893
Mon May 15 09:59:26 2023
Recovery of Online Redo Log: Thread 1 Group 2 Seq 5 Reading mem 0
Mem# 0 errs 0: /dev/rlog02a.dbf
Mon May 15 09:59:26 2023
Recovery of Online Redo Log: Thread 1 Group 2 Seq 5 Reading mem 0
Mon May 15 09:59:26 2023
Mem# 1 errs 0: /dev/rlog02b.dbf
Mon May 15 09:59:26 2023
Mem# 0 errs 0: /dev/rlog02a.dbf
Mem# 1 errs 0: /dev/rlog02b.dbf
Doing block recovery for fno: 12 blk: 3009
Mon May 15 09:59:26 2023
Recovery of Online Redo Log: Thread 1 Group 2 Seq 5 Reading mem 0
Mem# 0 errs 0: /dev/rlog02a.dbf
Mem# 1 errs 0: /dev/rlog02b.dbf
Mon May 15 09:59:26 2023
Doing block recovery for fno: 12 blk: 89
Mon May 15 09:59:26 2023
Recovery of Online Redo Log: Thread 1 Group 2 Seq 5 Reading mem 0
Mem# 0 errs 0: /dev/rlog02a.dbf
Mem# 1 errs 0: /dev/rlog02b.dbf
Mon May 15 09:59:26 2023
Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_844004.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4193], [56042], [1895], [], [], [], [], []
Error 607 happened during db open, shutting down database
USER: terminating instance due to error 607
Instance terminated by USER, pid = 844004
ORA-1092 signalled during: alter database open...
绕过该错误之后,数据库启动报ORA-600 2662错误
$ sqlplus "/ as sysdba"
SQL*Plus: Release 9.2.0.6.0 - Production on Mon May 15 10:04:44 2023
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
Connected to an idle instance.
SQL> startup mount pfile='/tmp/pfile'
ORACLE instance started.
Total System Global Area 1102023336 bytes
Fixed Size 744104 bytes
Variable Size 922746880 bytes
Database Buffers 167772160 bytes
Redo Buffers 10760192 bytes
Database mounted.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Mon May 15 10:05:03 2023
SMON: enabling cache recovery
Mon May 15 10:05:03 2023
ARC0: Media recovery disabled
Mon May 15 10:05:03 2023
SMON: enabling tx recovery
Mon May 15 10:05:03 2023
Database Characterset is ZHS16GBK
Mon May 15 10:05:03 2023
Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_413880.trc:
ORA-00600: internal error code, arguments: [2662], [3301], [2638409995], [3301], [2644132966], [4195678]
Mon May 15 10:05:04 2023
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
Mon May 15 10:05:04 2023
Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_413880.trc:
ORA-00600: internal error code, arguments: [2662], [3301], [2638409998], [3301], [2644132966], [4195678]
Mon May 15 10:05:04 2023
Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_413880.trc:
ORA-00600: internal error code, arguments: [2662], [3301], [2638409998], [3301], [2644132966], [4195678]
SMON: terminating instance due to error 600
Instance terminated by SMON, pid = 413880
解决该错误之后,数据库open正常
$ sqlplus "/ as sysdba"
SQL*Plus: Release 9.2.0.6.0 - Production on Mon May 15 10:10:30 2023
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
Connected to an idle instance.
SQL> startup mount pfile='/tmp/pfile'
ORACLE instance started.
Total System Global Area 1102023336 bytes
Fixed Size 744104 bytes
Variable Size 922746880 bytes
Database Buffers 167772160 bytes
Redo Buffers 10760192 bytes
Database mounted.
SQL> alter database open;
Database altered.
逻辑方式导出数据,本次恢复任务基本完成.
以前有过的类似恢复案例(类似较多选择典型几个):
ORA-600 2663
ORA-600 2663 故障恢复
ORA-600 2662
ora-600 2662和ora-600 kclchkblk_4恢复
redo异常 ORA-600 kclchkblk_4 故障恢复
ORA-600 4193 错误说明和解决
ORA-00600 [2662]和ORA-00600 [4194]恢复
