RMAN-06214: Archivelog错误

Posted on 2023 年 06 月 26 日 by 惜分飞

有一个客户他是linux到win环境dg,alert日志报清除fra中日志失败

un Jun 25 10:50:14 2023
Media Recovery Waiting for thread 1 sequence 196437 (in transit)
Sun Jun 25 10:50:26 2023
WARNING: Cannot delete Oracle managed file /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_28/o1_mf_1_192078_l74s4m2l_.arc
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\XFFwin\XFF\trace\XFF_rfs_1100.trc:
ORA-01265: 无法删除 ARCHIVED LOG /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_28/o1_mf_1_192078_l74s4m2l_.arc
ORA-27056: 无法删除文件
OSD-04029: 无法获取文件属性
O/S-Error: (OS 3) 系统找不到指定的路径。

尝试人工rman删除日志,报RMAN-06214错误

RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192575_l78zobv0_.arc
RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192576_l791fo3j_.arc
RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192577_l7935w3d_.arc
RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192578_l794y5bc_.arc
RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192579_l795cngq_.arc
RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192580_l795con4_.arc
RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192581_l795jtxk_.arc
RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192582_l795k97z_.arc
RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192583_l795noy1_.arc
RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192584_l795vvjg_.arc
RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192585_l796y9o2_.arc
RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192586_l798pk99_.arc
RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192587_l79bgx33_.arc
RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192588_l79bm1wf_.arc
RMAN-06214: Archivelog      /u01/oracle/fast_recovery_area/XFFDG/archivelog/2023_05_29/o1_mf_1_192589_l79bm2tn_.arc

crosscheck报ORA-19633错

RMAN> CROSSCHECK ARCHIVELOG ALL;

释放的通道: ORA_DISK_1
释放的通道: ORA_DISK_2
释放的通道: ORA_DISK_3
释放的通道: ORA_DISK_4
分配的通道: ORA_DISK_1
通道 ORA_DISK_1: SID=1717 设备类型=DISK
分配的通道: ORA_DISK_2
通道 ORA_DISK_2: SID=13 设备类型=DISK
分配的通道: ORA_DISK_3
通道 ORA_DISK_3: SID=579 设备类型=DISK
分配的通道: ORA_DISK_4
通道 ORA_DISK_4: SID=1148 设备类型=DISK
对归档日志的验证成功
归档日志文件名=D:\APP\ADMINISTRATOR\FAST_RECOVERY_AREA\XFFWIN\ARCHIVELOG\2023_06_25\O1_MF_1_196451_L9HC90MS_.ARC REC
ID=35113 STAMP=1140433431
对归档日志的验证成功
归档日志文件名=D:\APP\ADMINISTRATOR\FAST_RECOVERY_AREA\XFFWIN\ARCHIVELOG\2023_06_25\O1_MF_1_196452_L9HC9271_.ARC REC
ID=35112 STAMP=1140433425
已交叉检验的 2 对象

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: crosscheck 命令 (ORA_DISK_4 通道上, 在 06/25/2023 11:04:30 上) 失败
ORA-19633: 控制文件记录 30322 与恢复目录不同步

常规方法无法删除归档日志,只能通过dbms包强制删除归档日志

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on 星期日 6月 25 11:05:15 2023

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> execute sys.dbms_backup_restore.resetCfileSection( 11);

PL/SQL 过程已成功完成。

asm磁盘加入vg恢复

Posted on 2023 年 06 月 25 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：asm磁盘加入vg恢复

又一个客户把asm disk做成pv,加到vg中,并且对lv进行了扩展(ext4的文件系统)

这个客户做了上述操作之后,没有对lv进行写入其他数据,所以破坏较少(主要的破坏就是ext4的每个一段就会置空一部分block预留给文件系统写入元数据使用),通过winhex查看被破坏磁盘发现lvm信息

对于这种情况,通过对文件头进行修复,结合工具直接拷贝出来数据文件（个别文件元数据损坏通过基于block的方式恢复dbf）

然后直接恢复dbf中数据文件（对于异常的主要是segment header被置空的tab使用dul单独扫描处理）,实现客户数据的最大限度恢复
以前类似文章:
asm disk被加入vg恢复
asm disk被分区,格式化为ext4恢复
pvcreate asm disk导致asm磁盘组异常恢复
再一起asm disk被格式化成ext3文件系统故障恢复
一次完美的asm disk被格式化ntfs恢复
oracle asm disk格式化恢复—格式化为ext4文件系统

ORA-600 kcbr_apply_change_11

Posted on 2023 年 06 月 25 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 kcbr_apply_change_11

客户active dataguard应用日志报ORA-600 kcbr_apply_change_11错误

Sun Jun 25 10:31:25 2023
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION
Sun Jun 25 10:31:25 2023
MRP0 started with pid=32, OS id=6380 
 started logmerger process
Sun Jun 25 10:31:30 2023
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 4 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log D:\APP\ADMINISTRATOR\FAST_RECOVERY_AREA\xffWIN\ARCHIVELOG\2023_06_23\O1_MF_1_196319_L99NQQSN_.ARC
Sun Jun 25 10:31:31 2023
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr03_1540.trc  (incident=180300):
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\incident\incdir_180300\xff_pr03_1540_i180300.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION
Sun Jun 25 10:31:32 2023
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr02_752.trc  (incident=180292):
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\incident\incdir_180292\xff_pr02_752_i180292.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_rfs_6032.trc:
ORA-01265: 无法删除 ARCHIVED LOG /u01/oracle/fast_recovery_area/xffDG/archivelog/2023_05_28/o1_mf_1_192064_l74rj3jz_.arc
ORA-27056: 无法删除文件
OSD-04029: 无法获取文件属性
O/S-Error: (OS 3) 系统找不到指定的路径。
Archived Log entry 35084 added for thread 1 sequence 196424 rlc 1013082900 ID 0xe513780f dest 3:
RFS[7]: Opened log for thread 1 sequence 196430 dbid -451738353 branch 1013082900
Slave exiting with ORA-600 exception
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr02_752.trc:
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Sun Jun 25 10:31:41 2023
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr04_3696.trc  (incident=180308):
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\incident\incdir_180308\xff_pr04_3696_i180308.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Sun Jun 25 10:31:41 2023
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_mrp0_6380.trc  (incident=180268):
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\incident\incdir_180268\xff_mrp0_6380_i180268.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Recovery Slave PR02 previously exited with exception 600
Sun Jun 25 10:31:41 2023
MRP0: Background Media Recovery terminated with error 448
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr00_7812.trc:
ORA-00448: 后台进程正常结束
Managed Standby Recovery not using Real Time Apply
Sun Jun 25 10:31:41 2023
Slave exiting with ORA-600 exception
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr03_1540.trc:
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Sun Jun 25 10:31:43 2023
Sweep [inc][180308]: completed
Sweep [inc][180300]: completed
Sweep [inc][180292]: completed
Sweep [inc][180268]: completed
Sweep [inc2][180300]: completed
Sweep [inc2][180292]: completed
Sweep [inc2][180268]: completed
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr04_3696.trc:
ORA-00448: 后台进程正常结束
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr04_3696.trc:
ORA-00339: 归档日志未包含任何重做
ORA-00334: 归档日志: 'D:\APP\ADMINISTRATOR\ORADATA\xff\REDO02.LOG'
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Slave exiting with ORA-600 exception
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr04_3696.trc:
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []

通过分析可能为：Bug 31104809 – ORA-600: [kcbr_apply_change_11] during adg recovery (Doc ID 31104809.8)
官方WORKAROUND:
In ADG, mount the database (do not open it) and restart media recovery; once
the affected redo log sequence is applied, open the ADG in read only mode.

ORA-16038 ORA-00354故障处理

Posted on 2023 年 05 月 24 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-16038 ORA-00354故障处理

遇到一个案例,数据库open报ORA-16038,ORA-00354等错误

查询该redo状态(使用Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)脚本收集),确认为inactive

由于inactive 状态的redo损坏,无法被arch进程归档导致数据库无法正常open,尝试强制clear联机日志

由于25号文件属于offline状态,导致联机日志无法正常被clear,报ORA-00393 ORA-00312等错误.通过试验重现该问题.

SQL> alter database datafile 5 offline;

Database altered.

--使用一些技巧让数据库无法归档

SQL> select group#,status,archived from v$log;

    GROUP# STATUS           ARC
---------- ---------------- ---
         1 INACTIVE         NO
         2 ACTIVE           NO
         3 CURRENT          NO

SQL> shutdown abort;
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area  626327552 bytes
Fixed Size                  2255832 bytes
Variable Size             234882088 bytes
Database Buffers          385875968 bytes
Redo Buffers                3313664 bytes
Database mounted.
SQL> alter database clear unarchived logfile group 1;
alter database clear unarchived logfile group 1
*
ERROR at line 1:
ORA-00393: log 1 of thread 1 is needed for recovery of offline datafiles
ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log'
ORA-01110: data file 5: '/u01/app/oracle/oradata/orcl/t_xifenfei01.dbf'

SQL> !oerr ora 393
00393, 00000, "log %s of thread %s is needed for recovery of offline datafiles"
// *Cause:  Log cannot be cleared because the redo in it is needed to recover
//          offline datafiles. It has not been archived so there is no
//          other copy available. If the log is cleared the tablespaces
//          containing the files will have to be dropped.
// *Action: Archive the log then repeat the clear command. If archiving is not
//          possible, and dropping the tablespaces is acceptible, then add the
//          clause UNRECOVERABLE DATAFILE at the end of the clear command.


SQL>  alter database clear unarchived logfile group 1 unrecoverable datafile;

Database altered.

SQL> select group#,status,archived from v$log;

    GROUP# STATUS           ARC
---------- ---------------- ---
         1 UNUSED           YES
         3 CURRENT          NO
         2 ACTIVE           NO

客户的问题也是通过unrecoverable datafile 方式强制clear联机日志成功,数据库open成功

ORA-600 16703故障,客户找人恢复数据库,数据库被进一步恶意破坏—ORA-00704 ORA-00922

Posted on 2023 年 05 月 17 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 16703故障,客户找人恢复数据库,数据库被进一步恶意破坏—ORA-00704 ORA-00922

有朋友找到我,数据库报ORA-600 16703错误,这个本来是一个比较常见的破坏故障(警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703数据库启动报错如下:

修复tab$启动库报ORA-00704 ORA-00922错误

SQL> alter database Open;
alter database Open
*
第 1 行出现错误:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00922: missing or invalid option
进程 ID: 1340
会话 ID: 191 序列号: 3

ORA-00704 ORA-00922是比较少见的错误,第一感觉bootstrap$损坏了,对数据库启动过程进行跟踪

PARSING IN CURSOR #11700472 len=600 dep=1 uid=0 oct=1 lid=0 tim=338738406773 hv=4034608779 
ad='7ffdef83f360' sqlid='asgjp8bs7qgnb'
CREATE TABLE UNDO$("US#" 
END OF STMT
PARSE #11700472:c=0,e=361,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=338738406773
EXEC #11700472:c=0,e=73,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=0,tim=338738406917
CLOSE #11700472:c=0,e=3,dep=1,type=0,tim=338738406997
=====================
PARSE ERROR #635423520:len=841 dep=1 uid=0 oct=1 lid=0 tim=338738407066 err=922
CREATE TABLE TS$<"TS#" NU ...
ORA-00704: 引导程序进程失败
ORA-00922: 选项缺失或无效
ORA-00704: 引导程序进程失败
ORA-00922: 选项缺失或无效

*** 2023-05-17 19:27:51.813
USER (ospid: 1340): terminating the instance due to error 704

*** 2023-05-17 19:27:54.050
EXEC #11710688:c=0,e=2481834,p=16,cr=62,cu=0,mis=0,r=0,dep=0,og=1,plh=0,tim=338740646732
ERROR #11710688:err=1092 tim=338740646777

进一步分析bootstrap$表记录

通过上述分析,可以确认原库的CREATE TABLE TS$(“TS#”被人修改为CREATE TABLE TS$<“TS#”,通过观察客户机器以及和客户确认,客户找的技术人员上传了bbed工具,并进行了一些操作.基于上述信息,大概率是被人通过bbed工具把TS$(修改为了TS$<,从而使得数据库修复tab$之后也无法正常启动.