存储掉电强制拉库引起ORA-01555和ORA-01189/ORA-01190故障处理

Posted on 2025 年 08 月 30 日 by 惜分飞

标题：存储掉电强制拉库引起ORA-01555和ORA-01189/ORA-01190故障处理

机房存储突然掉电导致Oracle数据库访问存储异常,数据库报出大量的ORA-27072: File I/O error,Linux-x86_64 Error: 5: Input/output error,ORA-15081: failed to submit an I/O operation to a disk等错误,实例直接crash

Wed Aug 27 07:11:53 2025
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_m000_17596.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 6297632
Additional information: -1
WARNING: Read Failed. group:1 disk:0 AU:3075 offset:16384 size:16384
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-00202: control file: '+DG/xff/controlfile/current.284.918834897'
ORA-15081: failed to submit an I/O operation to a disk
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of 
  file 284 in group [1.2747812198] from disk DG_0000  allocation unit 3075 reason error; 
  if possible, will try another mirror side
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_m000_17596.trc:
ORA-00202: control file: '+DG/xff/controlfile/current.284.918834897'
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 16384
WARNING: Write Failed. group:1 disk:0 AU:3080 offset:49152 size:16384
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 16384
WARNING: Write Failed. group:1 disk:0 AU:3075 offset:49152 size:16384
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 0 logical 
 extent 0 of file 284 in group 1 on disk 0 allocation unit 3075 
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0
  of file 283 in group 1 on disk 0 allocation unit 3080 
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG/xff/controlfile/current.283.918834897'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG/xff/controlfile/current.284.918834897'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG/xff/controlfile/current.283.918834897'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG/xff/controlfile/current.284.918834897'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
CKPT (ospid: 6165): terminating the instance due to error 221
Wed Aug 27 07:11:53 2025
ORA-1092 : opitsk aborting process

存储恢复之后,尝试open数据库报ORA-00333错误（该错误一般是由于redo写丢失导致)

Wed Aug 27 16:36:32 2025
ALTER DATABASE OPEN
This instance was first to open
Beginning crash recovery of 2 threads
 parallel recovery started with 31 processes
Started redo scan
Incomplete read from log member '+DG/xff/onlinelog/group_2.287.918834905'. Trying next member.
Incomplete read from log member '+DG/xff/onlinelog/group_2.288.918834911'. Trying next member.
Incomplete read from log member '+DG/xff/onlinelog/group_2.287.918834905'. Trying next member.
Abort recovery for domain 0
Aborting crash recovery due to error 333
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_10257.trc:
ORA-00333: redo log read error block 1275904 count 5721
Abort recovery for domain 0
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_10257.trc:
ORA-00333: redo log read error block 1275904 count 5721
ORA-333 signalled during: ALTER DATABASE OPEN...
1
现场人员使用隐含参数,尝试直接拉库操作报ORA-00704 ORA-01555错误,导致拉库失败
1
Wed Aug 27 16:47:11 2025
ALTER DATABASE RECOVER  database until cancel  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 32 slaves
ORA-279 signalled during: ALTER DATABASE RECOVER  database until cancel  ...
Wed Aug 27 16:47:56 2025
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log +DG
Wed Aug 27 16:47:56 2025
Errors with log +DG
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_pr00_24154.trc:
ORA-00308: cannot open archived log '+DG'
ORA-17503: ksfdopn:2 Failed to open file +DG
ORA-15045: ASM file name '+DG' is not in reference form
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER CANCEL 
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_pr00_24154.trc:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '+DG/xff/datafile/system.279.918834827'
Slave exiting with ORA-1547 exception
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_pr00_24154.trc:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '+DG/xff/datafile/system.279.918834827'
ORA-1547 signalled during: ALTER DATABASE RECOVER CANCEL ...
Wed Aug 27 16:48:09 2025
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 25330611827626
Resetting resetlogs activation ID 3307041102 (0xc51d714e)
Deleted Oracle managed file +DG/xff/onlinelog/group_1.285.918834899
Deleted Oracle managed file +DG/xff/onlinelog/group_1.286.918834901
Deleted Oracle managed file +DG/xff/onlinelog/group_2.287.918834905
Deleted Oracle managed file +DG/xff/onlinelog/group_2.288.918834911
Wed Aug 27 16:48:28 2025
Deleted Oracle managed file +DG/xff/onlinelog/group_3.289.918834917
Deleted Oracle managed file +DG/xff/onlinelog/group_3.290.918834923
Deleted Oracle managed file +DG/xff/onlinelog/group_4.293.918835035
Deleted Oracle managed file +DG/xff/onlinelog/group_4.294.918835037
Wed Aug 27 16:48:48 2025
Deleted Oracle managed file +DG/xff/onlinelog/group_5.295.918835041
Deleted Oracle managed file +DG/xff/onlinelog/group_5.296.918835047
Deleted Oracle managed file +DG/xff/onlinelog/group_6.297.918835055
Wed Aug 27 16:48:58 2025
Deleted Oracle managed file +DG/xff/onlinelog/group_6.298.918835061
Wed Aug 27 16:49:10 2025
Setting recovery target incarnation to 3
Wed Aug 27 16:49:10 2025
This instance was first to open
Picked broadcast on commit scheme to generate SCNs
Wed Aug 27 16:49:10 2025
Assigning activation ID 3598492411 (0xd67ca2fb)
Thread 2 opened at log sequence 1
  Current log# 4 seq# 1 mem# 0: +DG/xff/onlinelog/group_4.294.1210265317
  Current log# 4 seq# 1 mem# 1: +DG/xff/onlinelog/group_4.293.1210265323
Successful open of redo thread 2
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Aug 27 16:49:10 2025
SMON: enabling cache recovery
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
ORA-01555 caused by SQL statement below (SQL ID: 4krwuz0ctqxdt, SCN: 0x1709.be1eb3b1):
select ctime, mtime, stime from obj$ where obj# = :1
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_23787.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 20 with name "_SYSSMU20_1295954159$" too small
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_23787.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 20 with name "_SYSSMU20_1295954159$" too small
Error 704 happened during db open, shutting down database
USER (ospid: 23787): terminating the instance due to error 704
Instance terminated by USER, pid = 23787
ORA-1092 signalled during: alter database open resetlogs...
opiodr aborting process unknown ospid (23787) as a result of ORA-1092

现场进行了一系列尝试操作,最后我接手数据库之时报错为:ORA-01190 ORA-01110,无法recover,也无法重建controlfile,陷入了死局

Completed: ALTER DATABASE   MOUNT
Sat Aug 30 10:03:20 2025
ALTER DATABASE OPEN
This instance was first to open
Abort recovery for domain 0
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_6435.trc:
ORA-01190: control file or data file 1 is from before the last RESETLOGS
ORA-01110: data file 1: '+DG/xff/datafile/system0829.dbf'
ORA-1190 signalled during: ALTER DATABASE OPEN...

Sat Aug 30 00:56:32 2025
NOTE: Loaded library: System 
SUCCESS: diskgroup DG was mounted
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_17302.trc:
ORA-01189: file is from a different RESETLOGS than previous files
ORA-01110: data file 2: '+DG/xff/datafile/sysaux.280.918834827'
ORA-1503 signalled during: create controlfile reuse database xff noarchivelog noresetlogs

对于这种情况,通过Oracle recovery check脚本可以直接发现异常(WRONG RESETLOGS)

使用Oracle Recovery Tools小工具实现快速恢复

再尝试重建ctl成功

然后修改数据库scn信息,顺利open数据库

后续建议客户逻辑迁移该库

ORA-600 kcratr_nab_less_than_odr和ORA-600 2662故障处理

Posted on 2025 年 08 月 18 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 kcratr_nab_less_than_odr和ORA-600 2662故障处理

异常断电之后,oracle启动报ORA-600 kcratr_nab_less_than_odr错误

Sun Aug 17 11:06:09 2025
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 11 processes
Started redo scan
Completed redo scan
 read 0 KB redo, 0 data blocks need recovery
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [40785], [33267], [40630]
Sun Aug 17 11:06:20 2025
Aborting crash recovery due to error 600
Sun Aug 17 11:06:20 2025
Trace dumping is performing id=[cdmp_20250817110620]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_2920.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [40785], [33267], [40630]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_2920.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [40785], [33267], [40630]
ORA-600 signalled during: ALTER DATABASE OPEN...

这个错误处理有多次处理经验
12c启动报kcratr_nab_less_than_odr
又一例ORA-600 kcratr_nab_less_than_odr
ORA-600 kcratr_nab_less_than_odr故障解决
 差点被误操作的ORA-600 kcratr_nab_less_than_odr故障
 11.2.0.4库中遇到ORA-600 kcratr_nab_less_than_odr报错
 ORA-600 kcratr_nab_less_than_odr和ORA-600 4194故障处理
一般重建ctl或者using backup ctl方式恢复即可实现0丢失打开库,但是这个库尝试打开报ORA-600 2662错误

SQL> startup nomount;
ORACLE 例程已经启动。

Total System Global Area 6847938560 bytes
Fixed Size                  2188768 bytes
Variable Size            4680845856 bytes
Database Buffers         2147483648 bytes
Redo Buffers               17420288 bytes
SQL> @rectl.sql

控制文件已创建。

SQL>
SQL>
SQL> recover database;
完成介质恢复。
SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [4413], [1200914792],
[4413], [1201104184], [12583040], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [2662], [4413], [1200914791],
[4413], [1201104184], [12583040], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [4413], [1200914789],
[4413], [1201104184], [12583040], [], [], [], [], [], []
进程 ID: 3132
会话 ID: 3018 序列号: 1

对于这样的ORA-600 2662错误是比较常见的问题,直接通过Patch_SCN工具修改scn即可正常打开库

Mon Aug 18 03:30:41 2025
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 12 slaves
Mon Aug 18 03:30:41 2025
Recovery of Online Redo Log: Thread 1 Group 2 Seq 40787 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Completed: ALTER DATABASE RECOVER  database  
alter database open 
Beginning crash recovery of 1 threads
 parallel recovery started with 11 processes
Started redo scan
Completed redo scan
 read 1 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 40787, block 2, scn 18954891612040
Recovery of Online Redo Log: Thread 1 Group 2 Seq 40787 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 40787, block 4, scn 18954891632047
 0 data blocks read, 0 data blocks written, 1 redo k-bytes read
Mon Aug 18 03:30:47 2025
Thread 1 advanced to log sequence 40788 (thread open)
Thread 1 opened at log sequence 40788
  Current log# 3 seq# 40788 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Aug 18 03:30:47 2025
SMON: enabling cache recovery
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: alter database open

后续增加tempfile,导出数据完成本次恢复任务

joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin)

Posted on 2025 年 08 月 15 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin)

expdp导出数据报ORA-39127 ORA-29548错误

ORA-39127: 调用 BEGIN :1 := SYS.DBMS_JAVA.START_EXPORT(:2, :3, :4, :5, :6, :7, :8, :9 ,:10); END; 时发生意外错误
ORA-29548: Java 系统类报告: release of Java system classes in the database (19.0.0.0.0 1.8) does not match that of the o
racle executable (19.3.0.0.0 1.8)
ORA-06512: 在 "SYS.DBMS_JAVA", line 520
ORA-06512: 在 line 1
ORA-06512: 在 "SYS.DBMS_JAVA", line 520
ORA-06512: 在 line 1
ORA-06512: 在 "SYS.DBMS_METADATA", line 11827
ORA-06512: 在 "SYS.DBMS_SYS_ERROR", line 95

对应的alert日志报错

2025-08-14T11:37:10.886745+08:00
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
2025-08-14T11:37:11.379211+08:00
Resize operation completed for file# 68, old size 83776K, new size 84800K
Resize operation completed for file# 68, old size 84800K, new size 85824K
2025-08-14T11:37:12.333138+08:00
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0

数据库版本（19.3版本,没有打任何其他patch）

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on 星期四 8月 14 11:17:08 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


连接到:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

出现这个错误一般是由于Oracle JVM组件和Oracle JVM库不同步导致,对于这个初始版本（没有打任何patch）,理论上不应该发生该问题,既然已经发生了,那就解决问题

SQL> startup upgrade;
ORACLE 例程已经启动。

Total System Global Area 1.5435E+10 bytes
Fixed Size                 16124312 bytes
Variable Size            4563402752 bytes
Database Buffers         1.0838E+10 bytes
Redo Buffers               17428480 bytes
数据库装载完毕。
数据库已经打开。
SQL> alter system set java_jit_enabled = FALSE;

系统已更改。

SQL> alter system set "_system_trig_enabled"=FALSE;

系统已更改。

SQL> alter system set JOB_QUEUE_PROCESSES=0;

系统已更改。

SQL> create or replace java system
  2          /

Java 已创建。

SQL> alter system set java_jit_enabled = true;

系统已更改。

SQL> alter system set "_system_trig_enabled"=TRUE;

系统已更改。

SQL> alter system set JOB_QUEUE_PROCESSES=100;

系统已更改。

SQL>  @?/rdbms/admin/utlrp.sql

会话已更改。

TIMESTAMP
--------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_BGN              2025-08-14 12:42:59

……………………

PL/SQL 过程已成功完成。

SQL> SELECT dbms_java.longname('TEST') long_name FROM dual;

LONG_NAME
--------------------------------------------------------------------------------
TEST

SQL> shutdown immediate;
数据库已经关闭。
已经卸载数据库。
ORACLE 例程已经关闭。
SQL> startup
ORACLE 例程已经启动。

Total System Global Area 1.5435E+10 bytes
Fixed Size                 16124312 bytes
Variable Size            4563402752 bytes
Database Buffers         1.0838E+10 bytes
Redo Buffers               17428480 bytes
数据库装载完毕。
数据库已经打开。

如果有其他情况,具体参考:ORA-29548 Error While Running Java Code or Datapatch In Database (Doc ID 2934546.1)文档

win环境断电强制拉库报ORA-600 kcbzib_kcrsds_1故障处理

Posted on 2025 年 08 月 14 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：win环境断电强制拉库报ORA-600 kcbzib_kcrsds_1故障处理

客户环境异常断电,导致数据库无法正常启动,执行recover提示缺少归档（ORA-00279 ORA-00289 ORA-00280错误)

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on 星期四 8月 14 11:17:08 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


连接到:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0


SQL> select open_mode from v$database;

OPEN_MODE
----------------------------------------
MOUNTED

SQL> select group#，sequence#,status from v$log;

    GROUP#  SEQUENCE# STATUS
---------- ---------- --------------------------------
         1      15082 ACTIVE
         3      15084 CURRENT
         2      15083 ACTIVE

SQL> recover database;
ORA-00279: 更改 885427209 (在 08/11/2025 22:14:56 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001
ORA-00280: 更改 885427209 (用于线程 1) 在序列 #15080 中


指定日志: {<RET>=suggested | filename | AUTO | CANCEL}

ORA-00308: 无法打开归档日志
'D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件。

z这个库需要15080归档,但是由于非归档环境,redo最小为15082,所以这种情况,只能先恢复可以正常recover的数据文件，然后强制拉库

SQL> recover datafile 1;
完成介质恢复。
SQL> recover datafile 3,4,7;
完成介质恢复。
SQL> recover datafile 62;
完成介质恢复。

SQL> recover database until cancel;
ORA-00279: 更改 885427209 (在 08/11/2025 22:14:56 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001
ORA-00280: 更改 885427209 (用于线程 1) 在序列 #15080 中


指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: 警告: RECOVER 成功但 OPEN RESETLOGS 将出现如下错误
ORA-01152: 文件 1 没有从过旧的备份中还原
ORA-01110: 数据文件 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSTEM01.DBF'


ORA-01112: 未启动介质恢复


SQL> alter database open resetlogs ;
alter database open resetlogs
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [],
[], [], [], [], [], [], []
进程 ID: 6272
会话 ID: 3873 序列号: 46847

alert日志报错

2025-08-14T11:24:08.246798+08:00
alter database open resetlogs
2025-08-14T11:24:08.425971+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 28: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\PRESCRIPTION_P.DBF'
2025-08-14T11:24:08.643178+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 29: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\PRESCRIPTION_DETAIL_P.DBF'
2025-08-14T11:24:08.958486+08:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 885427209 time 
Resetting resetlogs activation ID 3572089731 (0xd4e9c383)
2025-08-14T11:24:08.988512+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 30: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\ORDER_LIST_P.DBF'
2025-08-14T11:24:12.029432+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 31: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\CHARGE_STATIS_P.DBF'
2025-08-14T11:24:13.687023+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 32: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\ECASETAB01.DBF'
2025-08-14T11:24:13.725060+08:00
Setting recovery target incarnation to 2
Checker run found 31 new persistent data failures
2025-08-14T11:24:14.701994+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
Endian type of dictionary set to little
2025-08-14T11:24:14.793084+08:00
Assigning activation ID 3729861866 (0xde512cea)
2025-08-14T11:24:14.848137+08:00
TT00 (PID:8068): Gap Manager starting
2025-08-14T11:24:14.870158+08:00
Redo log for group 1, sequence 1 is not located on DAX storage
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG
Successful open of redo thread 1
2025-08-14T11:24:14.912199+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2025-08-14T11:24:15.849098+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc(incident=717057):
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Please look for redo dump in pinned buffers history in incident trace file, 
if not dumped for what so ever reason, use the following command to dump it at the earliest.
ALTER SYSTEM DUMP REDO DBA MIN 4 128 DBA MAX 4 128 SCN MIN 1;
2025-08-14T11:24:17.402582+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Undo initialization recovery: err:600 start: 2392187 end: 2394046 diff: 1859 ms (1.9 seconds)
2025-08-14T11:24:17.528703+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2025-08-14T11:24:17.529703+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc(incident=717058):
ORA-00603: ORACLE 服务器会话因致命错误而终止
ORA-01092: ORACLE 实例终止。强制断开连接
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2025-08-14T11:24:18.995110+08:00
opiodr aborting process unknown ospid (6272) as a result of ORA-603
2025-08-14T11:24:18.997112+08:00
ORA-603 : opitsk aborting process
License high water mark = 42
USER (ospid: (prelim)): terminating the instance due to ORA error 
2025-08-14T11:24:23.749679+08:00
Instance terminated by USER(prelim), pid = 6272

遭遇到比较明显的ORA-600 kcbzib_kcrsds_1错误,通过自研的Patch_SCN小工具,以前有过类似恢复案例：
Patch SCN工具一键恢复ORA-600 kcbzib_kcrsds_1

然后直接open数据库

SQL> recover database;
完成介质恢复。
SQL> oradebug setmypid
已处理的语句
SQL> oradebug DUMPvar SGA kcsgscn_
kscn8 kcsgscn_ [7FF6623FF310, 7FF6623FF318) = 40B272E2 00000000
SQL>
SQL> alter database open;

数据库已更改。

然后安排导出数据,完成本次恢复任务.

log_archive_dest_1配置不当,当时standby redo无法归档引起adg不能实时同步

Posted on 2025 年 08 月 13 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：log_archive_dest_1配置不当,当时standby redo无法归档引起adg不能实时同步

oracle database dataguard无法利用standby redo实时同步

2025-08-12T18:41:57.474031+08:00
PR00 (PID:245113): Media Recovery Log /u01/app/oracle/fast_recovery_area/……/2025_08_12/o1_mf_1_34206_n9p6mmkb_.arc
PR00 (PID:245113): Media Recovery Waiting for T-1.S-34207 (in transit)

clear standby redo之后，实时同步一会儿，又无法实时同步

2025-08-12T18:43:28.183908+08:00
PR00 (PID:245113): MRP0: Background Media Recovery cancelled with status 16037
2025-08-12T18:43:28.184131+08:00
Errors in file /u01/app/oracle/diag/rdbms/dclhrdg2/DCLHR/trace/DCLHR_pr00_245113.trc:
ORA-16037: user requested cancel of managed recovery operation
PR00 (PID:245113): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Stopping change tracking
2025-08-12T18:43:28.356862+08:00
Errors in file /u01/app/oracle/diag/rdbms/dclhrdg2/DCLHR/trace/DCLHR_pr00_245113.trc:
ORA-16037: user requested cancel of managed recovery operation
2025-08-12T18:43:28.482234+08:00
Background Media Recovery process shutdown (DCLHR)
2025-08-12T18:43:29.178708+08:00
Managed Standby Recovery Canceled (DCLHR)
2025-08-12T18:43:44.724806+08:00
Clearing online log 11 of thread 1 sequence number 34111
2025-08-12T18:44:00.545234+08:00
Clearing online log 12 of thread 1 sequence number 34112
2025-08-12T18:44:10.770626+08:00
Clearing online log 13 of thread 1 sequence number 34113
2025-08-12T18:44:31.911565+08:00
Clearing online log 14 of thread 1 sequence number 34114
2025-08-12T18:44:42.111444+08:00
Clearing online log 15 of thread 1 sequence number 34108
2025-08-12T18:45:06.031673+08:00
Warning: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE has been deprecated.
Warning: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE has been deprecated.
2025-08-12T18:45:06.047722+08:00
Attempt to start background Managed Standby Recovery process (DCLHR)
Starting background process MRP0
2025-08-12T18:45:06.062512+08:00
MRP0 started with pid=219, OS id=245503
2025-08-12T18:45:06.064300+08:00
Background Managed Standby Recovery process started (DCLHR)
2025-08-12T18:45:11.086271+08:00
 Started logmerger process
2025-08-12T18:45:11.100434+08:00

IM on ADG: Start of Empty Journal

IM on ADG: End of Empty Journal
PR00 (PID:245509): Managed Standby Recovery starting Real Time Apply
max_pdb is 3
2025-08-12T18:45:13.275444+08:00
Parallel Media Recovery started with 80 slaves
2025-08-12T18:45:13.392342+08:00
Stopping change tracking
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34207 (in transit)
2025-08-12T18:46:22.717320+08:00
 rfs (PID:245728): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is Foreground (PID:15566)
2025-08-12T18:46:31.950359+08:00
 rfs (PID:244324): Archived Log entry 99 added for B-1149239150.T-1.S-34207 ID 0x57544f44 LAD:32
2025-08-12T18:46:32.185391+08:00
 rfs (PID:245735): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is ASYNC (PID:3334853)
 rfs (PID:245735): Primary database is in MAXIMUM PERFORMANCE mode
2025-08-12T18:46:32.331928+08:00
 rfs (PID:245735): Selected LNO:11 for T-1.S-34208 dbid 1299764411 branch 1149239150
2025-08-12T18:46:32.885400+08:00
PR00 (PID:245509): Media Recovery Log /u01/app/oracle/fast_recovery_area/……/2025_08_12/o1_mf_1_34207_n9p6mo8f_.arc
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34208 (in transit)
2025-08-12T18:46:33.669769+08:00
Recovery of Online Redo Log: Thread 1 Group 11 Seq 34208 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo11.log
2025-08-12T18:50:03.011923+08:00
Expanded controlfile section 28 from 141 to 282 records
Requested to grow by 141 records; added 2 blocks of records
2025-08-12T19:16:29.515866+08:00
 rfs (PID:245735): Selected LNO:12 for T-1.S-34209 dbid 1299764411 branch 1149239150
2025-08-12T19:16:29.515943+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34209 (in transit)
2025-08-12T19:16:30.522048+08:00
Recovery of Online Redo Log: Thread 1 Group 12 Seq 34209 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo12.log
2025-08-12T19:46:29.852369+08:00
 rfs (PID:245735): Selected LNO:13 for T-1.S-34210 dbid 1299764411 branch 1149239150
2025-08-12T19:46:29.852443+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34210 (in transit)
2025-08-12T19:46:30.858423+08:00
Recovery of Online Redo Log: Thread 1 Group 13 Seq 34210 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo13.log
2025-08-12T20:16:29.978563+08:00
 rfs (PID:245735): Selected LNO:14 for T-1.S-34211 dbid 1299764411 branch 1149239150
2025-08-12T20:16:30.012723+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34211 (in transit)
2025-08-12T20:16:30.013045+08:00
Recovery of Online Redo Log: Thread 1 Group 14 Seq 34211 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo14.log
2025-08-12T20:46:30.307116+08:00
 rfs (PID:245735): Selected LNO:15 for T-1.S-34212 dbid 1299764411 branch 1149239150
2025-08-12T20:46:30.339601+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34212 (in transit)
2025-08-12T20:46:30.339893+08:00
Recovery of Online Redo Log: Thread 1 Group 15 Seq 34212 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo15.log
2025-08-12T21:16:27.517565+08:00
 rfs (PID:245735): No SRLs available for T-1
2025-08-12T21:16:27.560282+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34213 (in transit)
2025-08-12T21:16:27.615513+08:00
 rfs (PID:245735): Opened log for T-1.S-34213 dbid 1299764411 branch 1149239150
2025-08-12T21:46:27.633777+08:00
 rfs (PID:245735): Archived Log entry 100 added for B-1149239150.T-1.S-34213 ID 0x57544f44 LAD:32
 rfs (PID:245735): No SRLs available for T-1
2025-08-12T21:46:27.787746+08:00
 rfs (PID:245735): Opened log for T-1.S-34214 dbid 1299764411 branch 1149239150
2025-08-12T21:46:28.439910+08:00
PR00 (PID:245509): Media Recovery Log /u01/app/oracle/fast_recovery_area/……/2025_08_12/o1_mf_1_34213_n9phockd_.arc
2025-08-12T21:46:32.804540+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34214 (in transit)
2025-08-12T22:16:27.968668+08:00
 rfs (PID:245735): Archived Log entry 101 added for B-1149239150.T-1.S-34214 ID 0x57544f44 LAD:32
 rfs (PID:245735): No SRLs available for T-1

从报错信息看是由于无可用的standby redo导致（No SRLs available for T-1）,查询standby redo状态

[oracle@db03 trace]$ ss

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Aug 13 15:58:46 2025
Version 19.20.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

s
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.20.0.0.0

SQL> elect group#,status from v$standby_log;

    GROUP# STATUS
---------- ----------
        11 ACTIVE
        12 ACTIVE
        13 ACTIVE
        14 ACTIVE
        15 ACTIVE

所有的standby redo都是active状态,导致无可用standby redo,出现该问题的原因,初步怀疑是standby redo无法归档导致,引起该问题的原因怀疑是由于该备库的log_archive_dest_N参数设置不当导致,检查相关参数

*.db_unique_name='DCLHRDG2'
*.fal_client='DCLHRDG2'
*.fal_server='DCLHRDG','DCLHR'
*.log_archive_config='dg_config=(DCLHR,DCLHRDG,DCLHRDG2)'
*.log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST valid_for=(all_logfiles,all_roles) db_unique_name=DCLHRDG'
*.log_archive_dest_2='service=DCLHR lgwr async valid_for=(online_logfiles,primary_role) db_unique_name=DCLHR'
*.log_archive_dest_3='service=DCLHRDG2 lgwr async valid_for=(online_logfiles,primary_role) db_unique_name=DCLHRDG2'

比较明显由于log_archive_dest_1和log_archive_dest_3的相关参数配置不对导致standby redo无法通过log_archive_dest_1进行归档导致,正确的配置方式为

*.log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST valid_for=(all_logfiles,all_roles) db_unique_name=DCLHRDG'
*.log_archive_dest_3='service=DCLHRDG lgwr async valid_for=(online_logfiles,primary_role) db_unique_name=DCLHRDG'

这样配置之后,该dg库同步日志正常.