ORA-600 2662快速恢复之Patch scn工具

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-600 2662快速恢复之Patch scn工具

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户数据库启动报ORA-600 2662错误

SQL> recover database;
完成介质恢复。
SQL> alter database open ;
alter database open 
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [0], [272845978], [0], [272853062], [12583040]
ORA-00600: internal error code, arguments: [2662], [0], [272845977], [0], [272853062], [12583040]
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [0], [272845973], [0], [272853062], [12583040]
进程 ID: 3528
会话 ID: 33 序列号: 1

通过自研开发的patch scn工具,修改数据库scn值
20240313180657


然后open数据库成功
20240313181402

对于这类故障,patch scn工具是最快速的解决方案

断电引起文件scn异常数据库恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:断电引起文件scn异常数据库恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

由于异常断电,数据库最初启动报错

Fri Mar 01 08:41:17 2024
ALTER DATABASE   MOUNT
Successful mount of redo thread 1, with mount id 1865809648
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Fri Mar 01 08:41:24 2024
ALTER DATABASE OPEN
Errors in file /home/oracle/app/diag/rdbms/XFF/XFF/trace/XFF_ora_25243.trc:
ORA-01113: file 13 needs media recovery
ORA-01110: data file 13: '/data2/oracle/oradata/data/data00.dbf'
ORA-1113 signalled during: ALTER DATABASE OPEN...

经过应用厂商一系列操作,主要是如下操作

Fri Mar 01 11:10:56 2024
ALTER DATABASE RECOVER  datafile 13  
Media Recovery Start
Serial Media Recovery started
WARNING! Recovering data file 13 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
ORA-279 signalled during: ALTER DATABASE RECOVER  datafile 13  ...
Fri Mar 01 11:11:09 2024
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
Errors with log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
Errors with log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER CANCEL 
Media Recovery Canceled
Completed: ALTER DATABASE RECOVER CANCEL 
Fri Mar 01 11:16:50 2024
db_recovery_file_dest_size of 10240 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Fri Mar 01 11:51:56 2024
Starting ORACLE instance (normal)
Fri Mar 01 12:11:35 2024
alter database datafile 13 offline
ORA-1145 signalled during: alter database datafile 13 offline...
Fri Mar 01 12:12:29 2024
alter database recover cancel
ORA-1112 signalled during: alter database recover cancel...
Fri Mar 01 12:13:24 2024
ALTER DATABASE RECOVER  database until cancel  
Media Recovery Start
 started logmerger process
Fri Mar 01 12:13:24 2024
WARNING! Recovering data file 13 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 14 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 15 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 16 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 17 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 18 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 19 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 20 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 21 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 22 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
Parallel Media Recovery started with 48 slaves
ORA-279 signalled during: ALTER DATABASE RECOVER  database until cancel  ...
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
Errors with log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
Errors in file /home/oracle/app/diag/rdbms/XFF/XFF/trace/XFF_pr00_38097.trc:
ORA-00308:cannot open archived log '/home/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
Errors with log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
Errors in file /home/oracle/app/diag/rdbms/XFF/XFF/trace/XFF_pr00_38097.trc:
ORA-00308:cannot open archived log '/home/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER CANCEL 
Signalling error 1152 for datafile 1!
Errors in file /home/oracle/app/diag/rdbms/XFF/XFF/trace/XFF_pr00_38097.trc:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup 
ORA-01110: data file 1: '/data1/oracle/oradata/XFF/system01.dbf'
Slave exiting with ORA-1547 exception
Errors in file /home/oracle/app/diag/rdbms/XFF/XFF/trace/XFF_pr00_38097.trc:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup 
ORA-01110: data file 1: '/data1/oracle/oradata/XFF/system01.dbf'
ORA-10879 signalled during: ALTER DATABASE RECOVER CANCEL ...
Fri Mar 01 13:23:05 2024
ALTER DATABASE DATAFILE '/data2/oracle/oradata/data/data00.dbf' OFFLINE DROP
Completed: ALTER DATABASE DATAFILE '/data2/oracle/oradata/data/data00.dbf' OFFLINE DROP

接手现场之后,尝试单个文件recover操作

SQL> recover datafile 1;
ORA-00283: recovery session canceled due to errors
ORA-00264: no recovery required
SQL> recover datafile 2;
Media recovery complete.
SQL> recover datafile 3;
Media recovery complete.
SQL> recover datafile 4;
Media recovery complete.
SQL> recover datafile 5;
Media recovery complete.
SQL> recover datafile 6,7,8,9,10;
Media recovery complete.
SQL> recover datafile 11;
Media recovery complete.
SQL> recover datafile 12;
Media recovery complete.
SQL> recover datafile 13;
ORA-00279: change 1474236715 generated at 02/29/2024 17:13:00 needed for thread 1
ORA-00289: suggestion : /home/oracle/app/program/11g/dbs/arch1_153563_1136037378.dbf
ORA-00280: change 1474236715 for thread 1 is in sequence #153563


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}

ORA-00308: cannot open archived log '/home/oracle/app/program/11g/dbs/arch1_153563_1136037378.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3


SQL> recover datafile 14;
ORA-00279: change 1474236715 generated at 02/29/2024 17:13:00 needed for thread 1
ORA-00289: suggestion : /home/oracle/app/program/11g/dbs/arch1_153563_1136037378.dbf
ORA-00280: change 1474236715 for thread 1 is in sequence #153563


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}

ORA-00308: cannot open archived log '/home/oracle/app/program/11g/dbs/arch1_153563_1136037378.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

基于这样的情况,通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)检查发现/data2挂载点所有数据文件异常,由于以前的操作日志已经被清空无法判断原因,初步怀疑和这个挂载点本身有关系
20240303160404
20240303160524
这种情况直接使用bbed修改文件头,然后open库,再逻辑导出数据,完成本次数据恢复工作,参考类似文档
bbed 修改datafile header
使用bbed让rac中的sysaux数据文件online
当然这类故障也可以通过自研的Oracle Recovery Tools工具进行修复处理,类似文档:
Oracle Recovery Tools解决ORA-00279 ORA-00289 ORA-00280故障

模拟带库实现rman远程备份

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:模拟带库实现rman远程备份

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

rman通过模拟带库备份到远程机器,需要先配置用户ssh到目标机器
linux平台oracle备份

RMAN> RUN {
2> ALLOCATE CHANNEL S1 DEVICE TYPE sbt
3> PARMS='SBT_LIBRARY=/tmp/rman_sync.so,ENV=(
4> OB_SERVER=www.xifenfei.com,
5> OB_USER=root,
6> OB_PORT=22123,
7> OB_PATH=/tmp)';
8> BACKUP CURRENT CONTROLFILE FORMAT 'ctl_%U';
9> }

allocated channel: S1
channel S1: SID=125 device type=SBT_TAPE
channel S1: SBT/SSH2-SFTP

Starting backup at 22-FEB-24
channel S1: starting full datafile backup set
channel S1: specifying datafile(s) in backup set
including current control file in backup set
channel S1: starting piece 1 at 22-FEB-24
channel S1: finished piece 1 at 22-FEB-24
piece handle=ctl_012jq4gg_1_1 tag=TAG20240222T190344 comment=API Version 2.0,MMS Version 1.0.9.0
channel S1: backup set complete, elapsed time: 00:00:25
Finished backup at 22-FEB-24
released channel: S1

RMAN> list backup;

List of Backup Sets
===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
1       Full    9.50M      SBT_TAPE    00:00:24     22-FEB-24
        BP Key: 1   Status: AVAILABLE  Compressed: NO  Tag: TAG20240222T190344
        Handle: ctl_012jq4gg_1_1   Media: 0B2B9255
  Control File Included: Ckp SCN: 4438494      Ckp time: 22-FEB-24

win平台oracle备份

RMAN> RUN {
2>  ALLOCATE CHANNEL S1 DEVICE TYPE sbt
3>  PARMS='SBT_LIBRARY=D:\TEMP\rman_sync.dll,
4>  ENV=(
5>  OB_SERVER=www.xifenfei.com,
6>  OB_USER=root,
7>  OB_PORT=22123,
8>  OB_PATH=/tmp/,
9>  OB_PUBLIC_KEY=C:\Users\XIFENFEI\.ssh\id_rsa.pub,
10>  OB_SECRET_KEY=C:\Users\XIFENFEI\.ssh\id_rsa,
11>  OB_LOGFILE=d:\xifenfei.log)';
12>  BACKUP CURRENT CONTROLFILE FORMAT 'ctl_%U';
13>  }

分配的通道: S1
通道 S1: SID=156 设备类型=SBT_TAPE
通道S1: SBT/SSH2-SFTP

启动 backup 于 23-2月 -24
通道 S1: 正在启动全部数据文件备份集
通道 S1: 正在指定备份集内的数据文件
备份集内包括当前控制文件
通道 S1: 正在启动段 1 于 23-2月 -24
通道 S1: 已完成段 1 于 23-2月 -24
段句柄=ctl_012jt7mr_1_1 标记=TAG20240223T231642 注释=API Version 2.0,MMS Version 1.0.9.0
通道 S1: 备份集已完成, 经过时间:00:00:07
完成 backup 于 23-2月 -24
释放的通道: S1

RMAN> list backup;


备份集列表
===================


BS 关键字  类型 LV 大小       设备类型 经过时间 完成时间
------- ---- -- ---------- ----------- ------------ ----------
1       Full    9.50M      SBT_TAPE    00:00:05     23-2月 -24
        BP 关键字: 1   状态: AVAILABLE  已压缩: NO  标记: TAG20240223T231642
        句柄: ctl_012jt7mr_1_1   介质: 0B2B9255
  包括的控制文件: Ckp SCN: 14544437     Ckp 时间: 23-2月 -24

又一例:ORA-600 kclchkblk_4和2662故障

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:又一例:ORA-600 kclchkblk_4和2662故障

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户恢复请求:由于未知原因导致aix环境的rac两台主机同时重启之后数据库无法正常启动,初步判断是由于写丢失导致故障(ORA-00742 ORA-00353)

Wed Feb 21 09:23:06 2024
ALTER DATABASE OPEN
This instance was first to open
Abort recovery for domain 0
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_5767246.trc:
ORA-01113: file 32 needs media recovery
ORA-01110: data file 32: '+DATA/xff/datafile/x5sys_cs.dbf'
ORA-1113 signalled during: ALTER DATABASE OPEN...
Wed Feb 21 09:23:27 2024
ALTER DATABASE RECOVER  datafile '+DATA/xff/datafile/x5sys_cs.dbf'  
Media Recovery Start
Serial Media Recovery started
WARNING! Recovering data file 32 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
Recovery of Online Redo Log: Thread 2 Group 14 Seq 48490 Reading mem 0
  Mem# 0: +DATA/xff/onlinelog/group_14.313.1060528521
Recovery of Online Redo Log: Thread 1 Group 7 Seq 64195 Reading mem 0
  Mem# 0: +DATA/xff/onlinelog/group_7.306.1060527979
Wed Feb 21 09:24:10 2024
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_5767246.trc:
ORA-00742: Log read detects lost write in thread %d sequence %d block %d
ORA-00334: archived log: '+DATA/xff/onlinelog/group_14.313.1060528521'
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_5767246.trc  (incident=336478):
ORA-00353: log corruption near block 139727 change 26346459680 time 02/20/2024 20:13:50
ORA-00312: online log 14 thread 2: '+DATA/xff/onlinelog/group_14.313.1060528521'

尝试屏蔽一致性强制拉库后数据库报ORA-600 kclchkblk_4
参考:ora-600 2662和ora-600 kclchkblk_4恢复redo异常 ORA-600 kclchkblk_4 故障恢复

Wed Feb 21 09:55:26 2024
SMON: enabling cache recovery
Wed Feb 21 09:55:26 2024
Redo thread 2 internally disabled at seq 5 (CKPT)
Archived Log entry 112707 added for thread 2 sequence 4 ID 0xffffffffe144183b dest 1:
ARC0: Archiving disabled thread 2 sequence 5
Archived Log entry 112708 added for thread 2 sequence 5 ID 0xffffffffe144183b dest 1:
Wed Feb 21 09:55:28 2024
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6423264.trc  (incident=360479):
ORA-00600: internal error code, arguments: [kclchkblk_4], [6], [576721660], [6], [576702892]
Incident details in: /oracle/db/diag/rdbms/xff/xff1/incident/incdir_360479/xff1_ora_6423264_i360479.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Thread 1 advanced to log sequence 3 (LGWR switch)
  Current log# 7 seq# 3 mem# 0: +DATA/xff/onlinelog/group_7.306.1161510375
Archived Log entry 112709 added for thread 1 sequence 2 ID 0xffffffffe144183b dest 1:
Wed Feb 21 09:55:31 2024
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6423264.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kclchkblk_4], [6], [576721660], [6], [576702892]
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6423264.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kclchkblk_4], [6], [576721660], [6], [576702892]

后续处理中出现和这个错误类似的ORA-600 2662错误

Wed Feb 21 15:37:35 2024
SMON: enabling cache recovery
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6357664.trc  (incident=432423):
ORA-00600: internal error code, arguments: [2662], [6], [576742938], [6], [576834283], [12583104]
Incident details in: /oracle/db/diag/rdbms/xff/xff1/incident/incdir_432423/xff1_ora_6357664_i432423.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6357664.trc:
ORA-00600: internal error code, arguments: [2662], [6], [576742938], [6], [576834283], [12583104], [], [], [], [], [], []
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6357664.trc:
ORA-00600: internal error code, arguments: [2662], [6], [576742938], [6], [576834283], [12583104], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 6357664): terminating the instance due to error 600
Instance terminated by USER, pid = 6357664

通过对oracle scn进行修改,数据库open成功

SQL> recover database;
Media recovery complete.
SQL> 
SQL> 
SQL> 
SQL> oradebug setmypid
Statement processed.
SQL> oradebug DUMPvar SGA kcsgscn_
kcslf kcsgscn_ [700000000019B70, 700000000019BA0) = 00000000 00000000 00000000 00000000 
SQL> oradebug poke 
BEFORE: [700000000019B70, 700000000019B78) = 00000000 00000000
AFTER:  [700000000019B70, 700000000019B78) = 00000006 22710D2B
SQL> oradebug DUMPvar SGA kcsgscn_ 
kcslf kcsgscn_ [700000000019B70, 700000000019BA0) = 00000006 22710D2B 00000000 00000000 
SQL> alter database open;

Database altered.

后续检查发现obj$中的index异常(ORA-08102: index key not found, obj# 39)
类似文章:通过bbed修改obj$中dataobj$重现I_OBJ4索引报ORA-08102错误

SQL> truncate CLUSTER "SYS"."SMON_SCN_TO_TIME_AUX";

truncate CLUSTER "SYS"."SMON_SCN_TO_TIME_AUX"
                       *
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-08102: index key not found, obj# 39, file 1, block 967206 (2)

SQL> SQL> select object_name,object_type from dba_objects where object_id=39;

OBJECT_NAME                    OBJECT_TYPE
------------------------------ -------------------
I_OBJ4                         INDEX

对于此类问题使用非常规方法把obj$字典表进行重建(需要注意undo需要为自动管理方式,temp不能为空),参考:
bootstrap$核心index(I_OBJ1,I_USER1,I_FILE#_BLOCK#,I_IND1,I_TS#,I_CDEF1等)异常恢复—ORA-00701错误解决

Oracle误删除数据文件恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:Oracle误删除数据文件恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户通过sftp误删除oracle数据文件,咨询我们是否可以恢复,通过远程上去检查,发现运气不错,数据库还没有crash,通过句柄找到被删除文件

oracle@cwgstestdb[testwctdb]/proc/20611/fd$ls -ltr
total 0
lr-x------ 1 oracle oinstall 64 Feb 20 14:03 9 -> /oracle/db19c/rdbms/mesg/oraus.msb
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 8 -> /oracle/db19c/dbs/lkTESTWCTDB
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 7 -> /oracle/db19c/dbs/hc_testwctdb.dat
lr-x------ 1 oracle oinstall 64 Feb 20 14:03 6 -> /var/lib/sss/mc/passwd
lr-x------ 1 oracle oinstall 64 Feb 20 14:03 5 -> /proc/20611/fd
lr-x------ 1 oracle oinstall 64 Feb 20 14:03 4 -> /oracle/db19c/rdbms/mesg/oraus.msb
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 305 -> /oradata/ftms_zx_test01_data8.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 304 -> /oradata/ftms_zx_test01_data7.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 303 -> /oradata/ftms_zx_test01_data6.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 302 -> '/oradata/ftms_zx_test01_data5.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 301 -> '/oradata/ftms_zx_test01_data4.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 300 -> '/oradata/ftms_zx_test01_data3.dbf (deleted)'
lr-x------ 1 oracle oinstall 64 Feb 20 14:03 3 -> /dev/null
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 299 -> '/oradata/ftms_zx_test01_data2.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 298 -> '/oradata/ftms_zx_test01_data1.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 297 -> '/oradata/ftms_zx_test01_data.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 296 -> /oradata/ftms_zx_test_data.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 295 -> '/oradata/TESTWCTDB/sd.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 294 -> /oradata/TESTWCTDB/ftms_cs3_jiamiceshi
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 293 -> /langchao/dumpdata/FTMS_CS_TDE.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 292 -> /oradata/ftms_zx_test01.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 291 -> /langchao/dumpdata/FTMS_CS_DATA4.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 290 -> '/oradata/ftms_zx_data5.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 289 -> /langchao/dumpdata/FTMS_CS_DATA3.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 288 -> /langchao/dumpdata/FTMS_CS_DATA2.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 287 -> /langchao/dumpdata/FTMS_JD_DATA2.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 286 -> '/oradata/LCBIPECDS _TEMP_DAT.DBF'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 285 -> '/oradata/rTB_MBFE_TEMP (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 284 -> '/oradata/TESTWCTDB/temp01.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 283 -> '/oradata/ftms_credit_data5.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 282 -> /oradata/ftmshtdata.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 281 -> '/oradata/dump_data/FTMS_CSBF_DATA.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 280 -> /langchao/dumpdata/FTMS_NEWBL2_DATA.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 279 -> /langchao/dumpdata/FTMS_CS_DATA.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 278 -> /oradata/LCBIPECDS_DAT.DBF
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 277 -> /oradata/rTB_MBFE
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 276 -> /oradata/udpcount_02.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 275 -> /oradata/udpcount_01.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 274 -> '/oradata/ftms_credit_data_6.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 273 -> /langchao/dumpdata/FTMS_JD_DATA.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 272 -> '/oradata/ftms_old.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 271 -> '/oradata/ftms_credit_data2.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 270 -> /langchao/dumpdata/PJDIP_DATA.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 269 -> '/oradata/ftms_credit_data.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 268 -> /langchao/dumpdata/FTMS_NEWBL_DATA.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 267 -> '/oradata/ftms_zx_data4.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 266 -> /langchao/dumpdata/QIANZHANG_DATA.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 265 -> '/oradata/ftms_zx_data3.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 264 -> '/oradata/ftms_zx_data2.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 263 -> '/oradata/ftms_zx_data.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 262 -> /langchao/dumpdata/FTMSDIP_DATA.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 261 -> /oradata/TESTWCTDB/users01.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 260 -> '/oradata/TESTWCTDB/undotbs01.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 259 -> '/oradata/TESTWCTDB/sysaux01.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 258 -> '/oradata/TESTWCTDB/system01.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 257 -> /oradata/TESTWCTDB/control02.ctl
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 256 -> /oradata/TESTWCTDB/control01.ctl
l-wx------ 1 oracle oinstall 64 Feb 20 14:03 2 -> /dev/null
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 10 -> 'socket:[823411]'
l-wx------ 1 oracle oinstall 64 Feb 20 14:03 1 -> /dev/null
lr-x------ 1 oracle oinstall 64 Feb 20 14:03 0 -> /dev/null

查询数据文件大小(被删除的文件文件大小通过v$datafile查询为0)

SQL> select name,bytes/1024/1024/1024 from v$datafile;

NAME                                                                             BYTES/1024/1024/1024
-------------------------------------------------------------------------------- --------------------
/oradata/TESTWCTDB/system01.dbf                                                                     0
/oradata/TESTWCTDB/sysaux01.dbf                                                                     0
/oradata/TESTWCTDB/undotbs01.dbf                                                                    0
/oradata/TESTWCTDB/users01.dbf                                                             .004882813
/langchao/dumpdata/FTMSDIP_DATA.dbf                                                                 3
/oradata/ftms_zx_data.dbf                                                                           0
/oradata/ftms_zx_data2.dbf                                                                          0
/oradata/ftms_zx_data3.dbf                                                                          0
/langchao/dumpdata/QIANZHANG_DATA.dbf                                                               5
/oradata/ftms_zx_data4.dbf                                                                          0
/langchao/dumpdata/FTMS_NEWBL_DATA.dbf                                                             30
/oradata/ftms_credit_data.dbf                                                                       0
/langchao/dumpdata/PJDIP_DATA.dbf                                                                  20
/oradata/ftms_credit_data2.dbf                                                                      0
/oradata/ftms_old.dbf                                                                               0
/langchao/dumpdata/FTMS_JD_DATA.dbf                                                                15
/oradata/ftms_credit_data_6.dbf                                                                     0
/oradata/udpcount_01.dbf                                                                            5
/oradata/udpcount_02.dbf                                                                            5
/oradata/rTB_MBFE                                                                              .03125
/oradata/LCBIPECDS_DAT.DBF                                                                         .5
/langchao/dumpdata/FTMS_CS_DATA.dbf                                                                30
/langchao/dumpdata/FTMS_NEWBL2_DATA.dbf                                                            30
/oradata/dump_data/FTMS_CSBF_DATA.dbf                                                               0
/oradata/ftmshtdata.dbf                                                                    .087890625
/oradata/ftms_credit_data5.dbf                                                                      0
/langchao/dumpdata/FTMS_JD_DATA2.dbf                                                                3
/langchao/dumpdata/FTMS_CS_DATA2.dbf                                                       31.9999847
/langchao/dumpdata/FTMS_CS_DATA3.dbf                                                               10
/oradata/ftms_zx_data5.dbf                                                                          0
/langchao/dumpdata/FTMS_CS_DATA4.dbf                                                        12.109375
/oradata/ftms_zx_test01.dbf                                                                19.0527344
/langchao/dumpdata/FTMS_CS_TDE.dbf                                                                  1
/oradata/TESTWCTDB/ftms_cs3_jiamiceshi                                                     .029296875
/oradata/TESTWCTDB/sd.dbf                                                                           0
/oradata/ftms_zx_test_data.dbf                                                             .009765625
/oradata/ftms_zx_test01_data.dbf                                                                    0
/oradata/ftms_zx_test01_data1.dbf                                                                   0
/oradata/ftms_zx_test01_data2.dbf                                                                   0
/oradata/ftms_zx_test01_data3.dbf                                                                   0
/oradata/ftms_zx_test01_data4.dbf                                                                   0
/oradata/ftms_zx_test01_data5.dbf                                                                   0
/oradata/ftms_zx_test01_data6.dbf                                                          12.5976563
/oradata/ftms_zx_test01_data7.dbf                                                          9.08203125
/oradata/ftms_zx_test01_data8.dbf                                                                6.25

45 rows selected.

把数据文件拷贝回来

cp /proc/20611/fd/302   /langchao/orabak/
cp /proc/20611/fd/301   /langchao/orabak/
cp /proc/20611/fd/300   /langchao/orabak/
cp /proc/20611/fd/299   /langchao/orabak/
cp /proc/20611/fd/298   /langchao/orabak/
cp /proc/20611/fd/297   /langchao/orabak/
cp /proc/20611/fd/295   /langchao/orabak/
cp /proc/20611/fd/290   /langchao/orabak/
cp /proc/20611/fd/285   /langchao/orabak/
cp /proc/20611/fd/284   /langchao/orabak/
cp /proc/20611/fd/283   /langchao/orabak/
cp /proc/20611/fd/281   /langchao/orabak/
cp /proc/20611/fd/274   /langchao/orabak/
cp /proc/20611/fd/272   /langchao/orabak/
cp /proc/20611/fd/271   /langchao/orabak/
cp /proc/20611/fd/269   /langchao/orabak/
cp /proc/20611/fd/267   /langchao/orabak/
cp /proc/20611/fd/265   /langchao/orabak/
cp /proc/20611/fd/264   /langchao/orabak/
cp /proc/20611/fd/263   /langchao/orabak/
cp /proc/20611/fd/260   /langchao/orabak/
cp /proc/20611/fd/259   /langchao/orabak/
cp /proc/20611/fd/258   /langchao/orabak/

由于涉及system表空间数据文件被删除,无法在open情况下直接操作,直接关闭数据库,启动到mount状态,重命名数据文件路径,recover数据文件,open库,恢复完成
参考以前类似恢复:
Solaris rm datafile recovery—利用句柄误删除数据文件恢复
如果数据库已经关闭,需要考虑以下类似恢复方式:
dbca删除库和rm删库恢复
记录一次rm -rf 删除数据文件异常恢复