Patch_SCN快速解决ORA-600 2663故障

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:Patch_SCN快速解决ORA-600 2663故障

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

一个维保客户和我说他们测试库删除了日志文件导致库无法启动,让我帮忙看看
客户现场现况
1. 磁盘空间使用100%

[oracle@We1-db_Test ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  880K  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vda1        40G   38G   92M 100% /
tmpfs           783M     0  783M   0% /run/user/0

2. 数据库redo被删除了部分,而且是active状态的被删除

[oracle@We1-db_Test ~]$ ls -l /opt/app/oracle/oradata/orcl/redo0*
-rw-r----- 1 oracle oinstall 52429312 Jan 15 15:29 /opt/app/oracle/oradata/orcl/redo04.log
-rw-r----- 1 oracle oinstall 52429312 Jan 15 16:26 /opt/app/oracle/oradata/orcl/redo05.log

SQL> select group#,SEQUENCE#,STATUS FROM V$lOG;

    GROUP#  SEQUENCE# STATUS
---------- ---------- ----------------
         1       8989 CURRENT
         2          0 UNUSED
         5          0 UNUSED
         4          0 UNUSED
         3       8988 ACTIVE

SQL> select member from v$logfile;

MEMBER
-----------------------------------------------------
/opt/app/oracle/oradata/orcl/redo03.log
/opt/app/oracle/oradata/orcl/redo02.log
/opt/app/oracle/oradata/orcl/redo01.log
/opt/app/oracle/oradata/orcl/redo04.log
/opt/app/oracle/oradata/orcl/redo05.log

基于当前情况,直接open库无望,但是空间不足问题需要先解决,不然恢复过程中创建redo空间不足依旧会报错卡死,所以先清理了监听和alert等日志,系统空闲了3G多空间,可以进行恢复操作

[oracle@We1-db_Test trace]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  880K  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vda1        40G   34G  3.9G  90% /
tmpfs           783M     0  783M   0% /run/user/0

恢复数据库
1. 由于active redo丢失,毫无疑问,直接强制拉库,使用_allow_resetlogs_corruption参数开干

SQL> startup mount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area 2455228416 bytes
Fixed Size                  2255712 bytes
Variable Size             905970848 bytes
Database Buffers         1526726656 bytes
Redo Buffers               20275200 bytes
Database mounted.
SQL> recover database until cancel;
ORA-00283: recovery session canceled due to errors
ORA-01610: recovery using the BACKUP CONTROLFILE option must be done


SQL> recover database using backup controlfile;
ORA-00279: change 311982775 generated at 12/31/2025 17:35:11 needed for thread
1
ORA-00289: suggestion :
/opt/app/oracle/fast_recovery_area/ORCL/archivelog/2026_01_16/o1_mf_1_8988_%u_.a
rc
ORA-00280: change 311982775 for thread 1 is in sequence #8988


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2663], [0], [311982792], [0],
[311982833], [], [], [], [], [], [], []
Process ID: 11917
Session ID: 576 Serial number: 3

alert日志报错

Fri Jan 16 21:25:31 2026
Assigning activation ID 1750515127 (0x6856bdb7)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /opt/app/oracle/oradata/orcl/redo01.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan 16 21:25:31 2026
SMON: enabling cache recovery
Errors in file /opt/app/oracle/diag/rdbms/orcl/we1db/trace/we1db_ora_11917.trc  (incident=81753):
ORA-00600: internal error code, arguments: [2663], [0], [311982792], [0], [311982833], [], []
Incident details in: /opt/app/oracle/diag/rdbms/orcl/we1db/incident/incdir_81753/we1db_ora_11917_i81753.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /opt/app/oracle/diag/rdbms/orcl/we1db/trace/we1db_ora_11917.trc:
ORA-00600: internal error code, arguments: [2663], [0], [311982792], [0], [311982833], [], []
Errors in file /opt/app/oracle/diag/rdbms/orcl/we1db/trace/we1db_ora_11917.trc:
ORA-00600: internal error code, arguments: [2663], [0], [311982792], [0], [311982833], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 11917): terminating the instance due to error 600
Instance terminated by USER, pid = 11917
ORA-1092 signalled during: alter database open resetlogs...
opiodr aborting process unknown ospid (11917) as a result of ORA-1092
Fri Jan 16 21:25:33 2026
ORA-1092 : opitsk aborting process

不幸数据库遇到ORA-600 2663错误,这个故障在以前的文章中描述过,基本上和ORA-600 2662的处理思路类似,这里直接使用:Patch_SCN for Linux进行恢复
2. 使用Patch_SCN处理数据库SCN

SQL> startup nomount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area 2455228416 bytes
Fixed Size                  2255712 bytes
Variable Size             905970848 bytes
Database Buffers         1526726656 bytes
Redo Buffers               20275200 bytes
SQL>@rectl

Control file created.

SQL> recover database;
Media recovery complete.

patch_scn


SQL> alter database open;

Database altered.
SQL> SELECT CURRENT_SCN FROM V$DATABASE;

     CURRENT_SCN
----------------
       322002903

到这里完成数据库open操作,后续逻辑导出完成恢复任务

在生产环境错误执行dd命令破坏asm磁盘故障恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:在生产环境错误执行dd命令破坏asm磁盘故障恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

由于ssh登录错误,客户对生产环境进行了误操作把系统的一块磁盘dd到另外两个磁盘上,由于及时发现立马进行了终止操作,但是还是分别破坏了一点数据(一块盘破坏了2G多,另外一块盘破坏了1G多)
dd


通过分析udev的绑定关系确认被破坏的asm disk名称
udev

再通过asm alert日志确认破坏磁盘在asm disk中情况
asmdisk

通过上述信息基本上可以确认,asmdisk13被分别dd到了asmdisk11和asmdisk26中了部分11
26

基于这种情况,由于asm disk被破坏了1-2G多,直接修复然后正常mount磁盘组基本上没有希望,经过分析以及与客户沟通,确认他们改系统是4节点组成的集群,1/2节点上面跑2套库,3/4节点上跑2套库,数据整体放在data_dg磁盘组中,需要恢复的库是第二个顺序创建的1套库(4套库中只需恢复一套即可),由于破坏的数据本身不多,而且需要恢复的数据不是最初写入asm磁盘组,基于这样的情况,需要恢复的数据库机会比较大.

由于现在三个磁盘头信息一致(一个磁盘被dd到另外两个磁盘上),因此第一步先把损坏的两个磁盘头进行简单修复,为了便于amdu(找回ASM中数据文件)等之类数据可以识别到正确的磁盘头信息,然后进行后续的数据文件提取恢复.使用工具对数据文件进行了批量提取,提取数据完成之后,尝试recover和open库
open

虽然数据库正常打开了,不过很不幸,后台还是有一些坏块报错,通过dbv检查发现有文件有一部分坏块,类似dbv报错信息
1_1

通过分析该文件在磁盘组中各个磁盘的分布情况
map

确认该文件确实有部分block分布在被dd磁盘的破坏的范围内这个部分的数据丢失无法挽回,只能是定位到具体对象然后由业务想办法处理.相对以往的各种dd破坏的案例恢复而言,这个应该是效果比较好的一个,而且也是恢复比较容易的一个,没有使用到asm disk 基于asm au/oracle block 扫描的级别,而且system表空间没有任何损坏,数据库甚至直接open成功了,以往的一些dd案例列举:
asm磁盘dd破坏恢复
dd破坏asm磁盘头恢复
asm disk 磁盘部分被清空恢复