Monthly Archives: 四月 2021

raid强制上线后数据库无法启动故障处理

Posted on by

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：raid强制上线后数据库无法启动故障处理

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

由于raid掉盘过多,强制raid上线,然后启动数据库报以下错误

Mon Apr 19 23:19:28 2021
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 106750 KB redo, 9080 data blocks need recovery
Mon Apr 19 23:19:45 2021
Slave exiting with ORA-1115 exception
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p000_3277.trc:
ORA-01115: IO error reading block from file 9 (block # 339)
ORA-01110: data file 9: '/u01/app/oracle/oradata/orcl/dev02.dbf'
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I
Mon Apr 19 23:19:45 2021
Aborting crash recovery due to slave death, attempting serial crash recovery
Beginning crash recovery of 1 threads
Started redo scan
Completed redo scan
 read 106750 KB redo, 9080 data blocks need recovery
Aborting crash recovery due to error 1115
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_3275.trc:
ORA-01115: IO error reading block from file 9 (block # 329)
ORA-01110: data file 9: '/u01/app/oracle/oradata/orcl/dev02.dbf'
ORA-1115 signalled during: ALTER DATABASE OPEN...

错误提示比较明显IO error,结合客户强行上线raid的操作,比较明显是由于底层io问题导致该错误,直接对此文件dbv检查

[oracle@database orcl]$ dbv file=dev02.dbf

DBVERIFY: Release 11.2.0.4.0 - Production on Mon Apr 19 23:59:03 2021

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /u01/app/oracle/oradata/orcl/dev02.dbf

DBV-00600: Fatal Error - [28] [27061] [0] [0]

对于此类情况,通过工具进行处理

DUL> copy file from  /u01/app/oracle/oradata/orcl/dev02.dbf to /oradata/dev02.dbf
 
starting copy datafile '/u01/app/oracle/oradata/orcl/dev02.dbf' to '/oradata/dev02.dbf'
read data error from file '/u01/app/oracle/oradata/orcl/dev02.dbf'.error message:Input/output error
read block# error: 303
read data error from file '/u01/app/oracle/oradata/orcl/dev02.dbf'.error message:Input/output error
read block# error: 304
read data error from file '/u01/app/oracle/oradata/orcl/dev02.dbf'.error message:Input/output error
read block# error: 329
datafile copy completed with 2 block error.

dbv校验文件

[oracle@database oradata]$ dbv file=dev02.dbf 

DBVERIFY: Release 11.2.0.4.0 - Production on Tue Apr 20 00:28:31 2021

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /oradata/dev02.dbf
Page 303 is marked corrupt
Corrupt block relative dba: 0x0240012f (file 9, block 303)
Completely zero block found during dbv: 

Page 304 is marked corrupt
Corrupt block relative dba: 0x02400130 (file 9, block 304)
Completely zero block found during dbv: 

Page 329 is marked corrupt
Corrupt block relative dba: 0x02400149 (file 9, block 329)
Completely zero block found during dbv: 



DBVERIFY - Verification complete

Total Pages Examined         : 3932160
Total Pages Processed (Data) : 3213723
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 714294
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 4139
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 1
Total Pages Marked Corrupt   : 3
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 85078875 (6.85078875)

通过对io error的文件进行处理,最终损坏三个block,最大限度抢救数据.使用被恢复出来的文件,尝试open库遭遇以下错误

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [6], [85035771], [6],
[85084136], [12583040], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [2662], [6], [85035770], [6],
[85084136], [12583040], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [6], [85035764], [6],
[85084136], [12583040], [], [], [], [], [], []
Process ID: 6733
Session ID: 570 Serial number: 3

ora-600 2662这个错误比较明显,处理文件头scn,继续open库

SQL> alter database open ;
alter database open 
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 6840
Session ID: 570 Serial number: 3

查看alert日志信息

Tue Apr 20 01:22:27 2021
alter database open upgrade
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 1 KB redo, 3 data blocks need recovery
Started redo application at
 Thread 1: logseq 1, block 3
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/orcl/redo01.log
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 1, block 5, scn 25854859541
 3 data blocks read, 3 data blocks written, 1 redo k-bytes read
Tue Apr 20 01:22:28 2021
Thread 1 advanced to log sequence 2 (thread open)
Thread 1 opened at log sequence 2
  Current log# 2 seq# 2 mem# 0: /u01/app/oracle/oradata/orcl/redo02.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Apr 20 01:22:28 2021
SMON: enabling cache recovery
[6840] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:5902014 end:5905574 diff:3560 (35 seconds)
Dictionary check beginning
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_6824.trc  (incident=63970):
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_63970/orcl_smon_6824_i63970.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ORACLE Instance orcl (pid = 14) - Error 600 encountered while recovering transaction (24, 2) on object 89023.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_6824.trc:
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], []
Tue Apr 20 01:22:38 2021
ORACLE Instance orcl (pid = 14) - Error 600 encountered while recovering transaction (63, 3) on object 89023.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_6824.trc:
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_6824.trc  (incident=63974):
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_63974/orcl_smon_6824_i63974.trc
Tue Apr 20 01:22:55 2021
PMON (ospid: 6798): terminating the instance due to error 474

这个错误是比较常见的错误,参考:ORACLE Instance XFF (pid = 18) – Error 600 encountered while recovering transaction ，通过处理之后,数据库open成功

SQL> startup mount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area 1603411968 bytes
Fixed Size                  2253664 bytes
Variable Size            1023413408 bytes
Database Buffers          570425344 bytes
Redo Buffers                7319552 bytes
Database mounted.
SQL> recover database;
Media recovery complete.
SQL> alter database open;

Database altered.

后续安排逻辑导出,导入新库

KGL-heap-size-exceeded报错

Posted on by

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：KGL-heap-size-exceeded报错

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

alert日志报错

2021-04-19T15:06:08.208159+08:00
Memory Notification: Library Cache Object loaded into SGA
Heap size 512000K exceeds notification threshold (51200K)
Details in trace file /mnt/app/diag/rdbms/xff/xff/trace/xff_ora_12300.trc
2021-04-19T15:06:08.208212+08:00
KGL object name :select DISTINCT(BEGIN_TIME) beginTime,
    MILE mile,
    END_TIME endTime,
    LAST_TIME lastTime,
    BEGIN_TIME beginTime2,
    END_TIME endTime2,
    TEMPERATURE1 temperature1,
    TEMPERATURE2 temperature2,
    TEMPERATURE3 temperature3,
    TEMPERATURE4 temperature4,
    LATITUDE latitude,
    LONGITUDE longitude
    from T_XIFENFEI
    where VEHICLE_ID = :1 
    and END_TIME is not null
    and BEGIN_TIME BETWEEN TO_DATE(:2 , 'yyyy-MM-dd hh24:mi:ss') and TO_DATE(:3 , 'yyyy-MM-dd hh24
Errors in file /mnt/app/diag/rdbms/xff/xff/trace/xff_ora_12300.trc  (incident=162089):
ORA-00600: internal error code, arguments: [KGL-heap-size-exceeded], 
[0x1C4782708], [0], [524288008], [], [], [], [], [], [], [], []
Incident details in: /mnt/app/diag/rdbms/xff/xff/incident/incdir_162089/xff_ora_12300_i162089.trc
Use ADRCI or Support Workbench to package the incident.

数据库版本信息

Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
Build label:    RDBMS_12.2.0.1.0_LINUX.X64_170125
ORACLE_HOME:    /mnt/app/product/12.2.0/dbhome_1
System name:    Linux
Node name:      izbp1399kdj96ra6bjtwyxz
Release:        3.10.0-693.2.2.el7.x86_64
Version:        #1 SMP Tue Sep 12 22:26:13 UTC 2017
Machine:        x86_64

Call Stack信息

----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst()+119         call     kgdsdst()            7FFD6CB9A258 000000002
                                                   7FFD6CB7BCC0 ? 7FFD6CB7BDD8 ?
                                                   000000000 000000082 ?
dbkedDefDump()+1200  call     ksedst()             000000000 000000002 ?
                                                   7FFD6CB7BCC0 ? 7FFD6CB7BDD8 ?
                                                   000000000 ? 000000082 ?
ksedmp()+259         call     dbkedDefDump()       000000003 000000002
                                                   7FFD6CB7BCC0 ? 7FFD6CB7BDD8 ?
                                                   000000000 ? 000000082 ?
dbgexPhaseII()+2130  call     ksedmp()             0000003EB 000000002 ?
                                                   7FFD6CB7BCC0 ? 7FFD6CB7BDD8 ?
                                                   000000000 ? 000000082 ?
dbgexExplicitEndInc  call     dbgexPhaseII()       7F4853DB46C0 7F4853DB77A0
()+602                                             7FFD6CB9E388 7FFD6CB7BDD8 ?
                                                   000000000 ? 000000082 ?
dbgeEndDDEInvocatio  call     dbgexExplicitEndInc  7F4853DB46C0 7F4853DB77A0
nImpl()+658                   ()                   7FFD6CB9E388 ? 7FFD6CB7BDD8 ?
                                                   000000000 ? 000000082 ?
kglLargeHeapWarning  call     dbgeEndDDEInvocatio  7F4853DB46C0 7F4853DB77A0
()+1544                       nImpl()              7F4853DF49A0 7FFD6CB7BDD8 ?
                                                   7F4853DF49A0 000000082 ?
kglHeapAllocCbk()+2  call     kglLargeHeapWarning  7F4853DF49A0 ? 1C4782708 ?
27                            ()                   000000000 01F400060 ?
                                                   0B6DADD88 000000082 ?
kghalp()+1086        call     kglHeapAllocCbk()    7F4853DF49A0 ? 1C4782708 ?
                                                   2EE171B70 01F400060 ?
                                                   000000058 ? 000000082 ?
kkocsRegBindEqvCtxC  call     kghalp()             7F4853DF49A0 ? 0B6DADD88 ?
ommon()+145                                        2EE171B70 ? 000000001 ?
                                                   000000058 ? 0119E7860 ?
kkocsAddBeElem()+81  call     kkocsRegBindEqvCtxC  7F4853DF49A0 ? 7FFD6CB9F460 ?
1                             ommon()              2EE171B70 ? 000000001 ?
                                                   000000002 ? 0119E7860 ?
kkocsProcBSensPredC  call     kkocsAddBeElem()     7F484E2EF998 ? 7F484E2EFAC8 ?
B()+431                                            0BBED4798 ? 209DFEA60 ?
                                                   000000002 ? 0119E7860 ?
qksopVisitPreds()+5  call     kkocsProcBSensPredC  7F484E2EF998 ? 7FFD6CB9F640 ?
39                            B()                  0BBED4798 ? 209DFEA60 ?
                                                   000000002 ? 0119E7860 ?
kkocsBuildBESelCtx(  call     qksopVisitPreds()    7F484E2EF998 ? 7FFD6CB9F640 ?
)+153                                              0BBED4798 ? 209DFEA60 ?
                                                   000000002 ? 0119E7860 ?
kkocsBindSensCheck(  call     kkocsBuildBESelCtx(  7F484E2EF998 ? 7FFD6CB9F640 ?
)+585                         )                    0BBED4798 ? 209DFEA60 ?
                                                   000000002 ? 0119E7860 ?
kkoipt()+901         call     kkocsBindSensCheck(  7F484E2EFA20 ? 7FFD6CB9F640 ?
                              )                    0BBED4798 ? 209DFEA60 ?
                                                   000000002 ? 0119E7860 ?
kkoqbc()+4677        call     kkoipt()             7F484E31F1E0 ? 7FFD6CB9F640 ?
                                                   0BBED4798 ? 209DFEA60 ?
                                                   000000002 ? 000000000
apakkoqb()+183       call     kkoqbc()             7F484E31F1E0 ? 7F4853CC2958 ?
                                                   0BBED4798 ? 209DFEA60 ?
                                                   000000001 000000000 ?
apaqbdDescendents()  call     apakkoqb()           7FFD6CBA1870 ? 000000EB5 ?
+503                                               0BBED4C18 ? 209DFEA60 ?
                                                   000000001 ? 000000000 ?
apaqbd()+136         call     apaqbdDescendents()  7FFD6CBA1870 ? 7F4853CC2958 ?
                                                   0BBED4C18 ? 209DFEA60 ?
                                                   000000001 ? 000000000 ?
apadrv()+1228        call     apaqbd()             7FFD6CBA1870 ? 7F4853CC2958 ?
                                                   0BBED4C18 ? 209DFEA60 ?
                                                   000000001 ? 000000000 ?
opitca()+2139        call     apadrv()             0BBED4C18 ? 7F4853CC2958 ?
                                                   0BBED4C18 ? 209DFEA60 ?
                                                   000000001 ? 000000000 ?
kksFullTypeCheck()+  call     opitca()             7F484E2F2A08 0BBED4C18
86                                                 7FFD6CBA42F0 209DFEA60 ?
                                                   000000001 ? 000000000 ?
rpiswu2()+627        call     kksFullTypeCheck()   7FFD6CBA2FF8 ? 0BBED4C18 ?
                                                   7FFD6CBA42F0 ? 209DFEA60 ?
                                                   000000001 ? 000000000 ?
kksLoadChild()+8003  call     rpiswu2()            7FFD6CBA2FF8 ? 0BBED4C18 ?
                                                   7FFD6CBA42F0 ? 209DFEA60 ?
                                                   7FFD6CBA24F8 ? 00000006A ?
kxsGetRuntimeLock()  call     kksLoadChild()       7F4853DF49A0 0BBED4C18 ?
+2029                                              7FFD6CBA42F0 ? 209DFEA60 ?
                                                   7FFD6CBA24F8 ? 00000006A ?
kksfbc()+15091       call     kxsGetRuntimeLock()  7F4853DF49A0 7F484E2F2A08
                                                   7FFD6CBA4270 209DFEA60 ?
                                                   7FFD6CBA24F8 ? 1C4782708
opiexe()+2932        call     kksfbc()             7F484E2F2A08 7F484E2F2A08 ?
                                                   7FFD6CBA4270 ? 209DFEA60 ?
                                                   7FFD6CBA24F8 ? 1C4782708 ?
kpoal8()+2679        call     opiexe()             000000049 7F484E2F2A08 ?
                                                   7FFD6CBA58F0 209DFEA60 ?
                                                   7FFD6CBA24F8 ? 1C4782708 ?
opiodr()+1229        call     kpoal8()             00000005E 000000026
                                                   7FFD6CBA9300 209DFEA60 ?
                                                   7FFD6CBA24F8 ? 1C4782708 ?
ttcpip()+1257        call     opiodr()             00000005E 000000026
                                                   7FFD6CBA9300 000000000
                                                   7FFD6CBA24F8 ? 1C4782708 ?
opitsk()+1940        call     ttcpip()             7F4853E11750 ? 000000026 ?
                                                   7FFD6CBA9300 00000017D ?
                                                   7FFD6CBA8D50 7FFD6CBA955C
opiino()+941         call     opitsk()             000000000 000000000
                                                   7FFD6CBA9300 ? 00000017D ?
                                                   7FFD6CBA8D50 ? 7FFD6CBA955C ?
opiodr()+1229        call     opiino()             00000003C 000000004
                                                   7FFD6CBAAA18 00000017D ?
                                                   7FFD6CBA8D50 ? 7FFD6CBA955C ?
opidrv()+1021        call     opiodr()             00000003C 000000004
                                                   7FFD6CBAAA18 000000000
                                                   7FFD6CBA8D50 ? 7FFD6CBA955C ?
sou2o()+145          call     opidrv()             00000003C 000000004
                                                   7FFD6CBAAA18 000000000 ?
                                                   7FFD6CBA8D50 ? 7FFD6CBA955C ?
opimai_real()+455    call     sou2o()              7FFD6CBAA9F0 00000003C
                                                   000000004 7FFD6CBAAA18
                                                   7FFD6CBA8D50 ? 7FFD6CBA955C ?
ssthrdmain()+417     call     opimai_real()        000000000 7FFD6CBAAD00
                                                   000000004 ? 7FFD6CBAAA18 ?
                                                   7FFD6CBA8D50 ? 7FFD6CBA955C ?
main()+262           call     ssthrdmain()         000000000 000000002
                                                   7FFD6CBAAD00 000000001
                                                   000000000 7FFD6CBA955C ?
__libc_start_main()  call     main()               000000000 7FFD6CBAAF38
+245                                               7FFD6CBAAD00 ? 000000001 ?
                                                   000000000 ? 7FFD6CBA955C ?
_start()+41          call     __libc_start_main()  000D04D40 000000002
                                                   7FFD6CBAAF38 7F484F241555 ?
                                                   000000000 ? 7FFD6CBA955C ?
--------------------- Binary Stack Dump ---------------------

根据官方描述,该问题是由于Bug 20917487 – CORRUPT KKOCS STRUCTURES AFTER PARTIAL PURGE OF CURSOR HEAP引起
解决方案：
1. 升级数据库到19.1+版本
2. 应用 Patch 20917487
3. Disable adaptive cursor sharing

alter system set "_optimizer_adaptive_cursor_sharing" = false;
alter system set "_optimizer_extended_cursor_sharing_rel" = "none";

4. 通过隐含参数设置一个足够大的值

alter system set "_kgl_large_heap_warning_threshold"=83886080 scope=spfile; --一个足够大的值
--or
alter system set "_kgl_large_heap_warning_threshold"=0 scope=spfile;
alter system set "_kgl_large_heap_assert_threshold"=0 scope=spfile;

shutdown immediate
startup

参考文档：
ORA-600 [KGL-heap-size-exceeded] Errors in the Alert Log (Doc ID 2628072.1)
Memory Notification: Library Cache Object loaded into SGA / ORA-600 [KGL-heap-size-exceeded] (Doc ID 330239.1)

11.2.0.4 打patch遭遇extjobO: Operation not permitted处理

Posted on by

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：11.2.0.4 打patch遭遇extjobO: Operation not permitted处理

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有一段时间没有给11.2.0.4打psu补丁了,今天晚上在给一个客户打的过程中发现类似错误

OPatch found the word "error" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
chmod: changing permissions of `/u01/app/oracle/product/11.2.0/db_1/bin/extjobO': Operation not permitted
make: [iextjob] Error 1 (ignored)


Composite patch 31537677 successfully applied.
OPatch Session completed with warnings.
Log file location: /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-04-16_23-11-48PM_1.log

OPatch completed with warnings.

对比extjob权限信息

[oracle@localhost trace]$ ls -l /u01/app/oracle/product/11.2.0/db_1/bin/extjob*
-rwxr-xr-x 1 oracle oinstall 1254496 Apr 16 23:15 /u01/app/oracle/product/11.2.0/db_1/bin/extjob
-rwx------ 1 oracle oinstall 1254496 Apr 16 23:15 /u01/app/oracle/product/11.2.0/db_1/bin/extjobo
-rwsr-x--- 1 root   oinstall 1254244 Feb 13  2019 /u01/app/oracle/product/11.2.0/db_1/bin/extjobO
-rw------- 1 oracle oinstall 1254244 Feb 13  2019 /u01/app/oracle/product/11.2.0/db_1/bin/extjoboO

证明新的extjob程序确实权限不对,不过根据官方的建议可以忽略这个,参考link:

https://updates.oracle.com/Orion/Services/download?type=readme&aru=20775452

Applying Proactive Bundle / PSU Patch fails with Error: “chmod: changing permissions of `$ORACLE_HOME/bin/extjobO’: Operation not permitted” (Doc ID 2265726.1)
给出来的理由是执行root.sh之后权限会变成正常的,另外这个external jobs基本上很少人会使用到,尝试执行root.sh修复

[root@localhost home]# /u01/app/oracle/product/11.2.0/db_1/root.sh
Performing root user operation for Oracle 11g 

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/11.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[root@localhost home]# ls -l  /u01/app/oracle/product/11.2.0/db_1/bin/extjob*
-rwsr-x--- 1 root   oinstall 1254496 Apr 16 23:15 /u01/app/oracle/product/11.2.0/db_1/bin/extjob
-rwx------ 1 oracle oinstall 1254496 Apr 16 23:15 /u01/app/oracle/product/11.2.0/db_1/bin/extjobo
-rwsr-x--- 1 root   oinstall 1254244 Feb 13  2019 /u01/app/oracle/product/11.2.0/db_1/bin/extjobO
-rw------- 1 oracle oinstall 1254244 Feb 13  2019 /u01/app/oracle/product/11.2.0/db_1/bin/extjoboO

确实通过执行root.sh修复该问题.

ORA-10485故障解决

Posted on by

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：ORA-10485故障解决

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

今天对主库进行打了psu和ojvm成功之后，对standby库打上了psu和ojvm，然后给备库启动mrp进程发现余下报错

Fri Apr 16 23:18:54 2021
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION
Attempt to start background Managed Standby Recovery process (orcl1)
Fri Apr 16 23:18:54 2021
MRP0 started with pid=32, OS id=15961 
MRP0: Background Managed Standby Recovery process started (orcl1)
 started logmerger process
Fri Apr 16 23:18:59 2021
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 32 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63308_j7m6domc_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41599_j7m70kyx_.arc
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63309_j7m6dop8_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63310_j7m70fmp_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63311_j7m70hlj_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63312_j7m75cnk_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41600_j7m75b77_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41601_j7m75bcx_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63313_j7m7vs0h_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41602_j7m7vn3g_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41603_j7m7vq2b_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41604_j7m81g5r_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63314_j7m81cz2_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63315_j7m81d31_.arc
Fri Apr 16 23:19:10 2021
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41605_j7m8vwz5_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41606_j7m8vzcr_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63316_j7m8vxgm_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41607_j7m8vzfk_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63317_j7m8w213_.arc
Errors with log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63317_j7m8w213_.arc
MRP0: Background Media Recovery terminated with error 10485
Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcl1/trace/orcl1_pr00_15963.trc:
ORA-10485: Real-Time Query cannot be enabled while applying migration redo.
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Fri Apr 16 23:19:12 2021
MRP0: Background Media Recovery process shutdown (orcl1)

由于redo中有migration操作,导致备库mrp应用进程失败,解决此类问题最简单的方法,就是把备库重启到mount状态,然后使用rman进行恢复操作,然后再继续open库,启动mrp解决这类问题

oracle@localhost trace]$ ss

SQL*Plus: Release 11.2.0.4.0 Production on Fri Apr 16 23:24:55 2021

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 4.2758E+10 bytes
Fixed Size                  2262656 bytes
Variable Size            5905582464 bytes
Database Buffers         3.6776E+10 bytes
Redo Buffers               74420224 bytes
Database mounted.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@localhost trace]$ rman target /

Recovery Manager: Release 11.2.0.4.0 - Production on Fri Apr 16 23:25:12 2021

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL (DBID=1442395283, not open)

RMAN> recover database;

Starting recover at 16-APR-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=143 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=286 device type=DISK
allocated channel: ORA_DISK_3
channel ORA_DISK_3: SID=428 device type=DISK
allocated channel: ORA_DISK_4
channel ORA_DISK_4: SID=570 device type=DISK

starting media recovery

archived log for thread 1 with sequence 63317 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63317_j7m8w213_.arc
archived log for thread 1 with sequence 63318 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63318_j7m932ms_.arc
archived log for thread 1 with sequence 63319 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63319_j7m93719_.arc
archived log for thread 1 with sequence 63320 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63320_j7m96b3g_.arc
archived log for thread 2 with sequence 41607 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41607_j7m8vzfk_.arc
archived log for thread 2 with sequence 41608 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41608_j7m963jp_.arc
archived log for thread 2 with sequence 41609 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41609_j7m967wm_.arc
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63317_j7m8w213_.arc thread=1 sequence=63317
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41607_j7m8vzfk_.arc thread=2 sequence=41607
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63318_j7m932ms_.arc thread=1 sequence=63318
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63319_j7m93719_.arc thread=1 sequence=63319
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63320_j7m96b3g_.arc thread=1 sequence=63320
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41608_j7m963jp_.arc thread=2 sequence=41608
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41609_j7m967wm_.arc thread=2 sequence=41609
unable to find archived log
archived log thread=2 sequence=41610
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 04/16/2021 23:25:25
RMAN-06054: media recovery requesting unknown archived log for thread 2 with sequence 41610 and starting SCN of 15880064787

SQL> alter database open;

Database altered.

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION;

Database altered.

Fri Apr 16 23:26:45 2021
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION
Fri Apr 16 23:26:47 2021
RFS[2]: Assigned to RFS process 16538
RFS[2]: Opened log for thread 2 sequence 41611 dbid 1442395283 branch 914614547
Archived Log entry 54459 added for thread 2 sequence 41611 rlc 914614547 ID 0x5b2c46ea dest 2:
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41611_j7mc5qcc_.arc
Media Recovery Waiting for thread 2 sequence 41612
Fetching gap sequence in thread 2, gap sequence 41612-41612
Fri Apr 16 23:26:50 2021
RFS[1]: Opened log for thread 2 sequence 41612 dbid 1442395283 branch 914614547
Archived Log entry 54460 added for thread 2 sequence 41612 rlc 914614547 ID 0x5b2c46ea dest 2:
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41612_j7mc5ttr_.arc
Media Recovery Waiting for thread 2 sequence 41613
Fetching gap sequence in thread 2, gap sequence 41613-41613
RFS[1]: Opened log for thread 2 sequence 41613 dbid 1442395283 branch 914614547
Archived Log entry 54461 added for thread 2 sequence 41613 rlc 914614547 ID 0x5b2c46ea dest 2:
Fri Apr 16 23:26:55 2021
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41613_j7mc5y6k_.arc
Media Recovery Waiting for thread 1 sequence 63323
Deleted Oracle managed file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_03_25/o1_mf_2_40942_j5s61m0d_.arc
RFS[1]: Opened log for thread 1 sequence 63323 dbid 1442395283 branch 914614547
Archived Log entry 54462 added for thread 1 sequence 63323 rlc 914614547 ID 0x5b2c46ea dest 2:
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63323_j7mc61gg_.arc
Media Recovery Waiting for thread 1 sequence 63324
Fri Apr 16 23:28:20 2021
RFS[3]: Assigned to RFS process 16545
RFS[3]: Opened log for thread 2 sequence 41615 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:20 2021
RFS[4]: Assigned to RFS process 16543
RFS[4]: Opened log for thread 1 sequence 63326 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:20 2021
RFS[5]: Assigned to RFS process 16551
RFS[5]: Opened log for thread 1 sequence 63324 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:20 2021
RFS[6]: Assigned to RFS process 16549
RFS[6]: Opened log for thread 2 sequence 41614 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:20 2021
RFS[7]: Assigned to RFS process 16547
RFS[7]: Opened log for thread 1 sequence 63325 dbid 1442395283 branch 914614547
Archived Log entry 54463 added for thread 2 sequence 41615 rlc 914614547 ID 0x5b2c46ea dest 2:
Fri Apr 16 23:28:20 2021
RFS[8]: Assigned to RFS process 16553
RFS[8]: Opened log for thread 2 sequence 41616 dbid 1442395283 branch 914614547
Archived Log entry 54464 added for thread 1 sequence 63326 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54465 added for thread 2 sequence 41616 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54466 added for thread 1 sequence 63325 rlc 914614547 ID 0x5b2c46ea dest 2:
RFS[3]: Opened log for thread 2 sequence 41617 dbid 1442395283 branch 914614547
RFS[8]: Opened log for thread 2 sequence 41618 dbid 1442395283 branch 914614547
RFS[4]: Opened log for thread 1 sequence 63327 dbid 1442395283 branch 914614547
Archived Log entry 54467 added for thread 2 sequence 41617 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54468 added for thread 1 sequence 63327 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54469 added for thread 2 sequence 41618 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54470 added for thread 1 sequence 63324 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54471 added for thread 2 sequence 41614 rlc 914614547 ID 0x5b2c46ea dest 2:
Fri Apr 16 23:28:20 2021
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63324_j7mc8n1n_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41614_j7mc8n1r_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41615_j7mc8n03_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41616_j7mc8n2n_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41617_j7mc8n6m_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41618_j7mc8n6p_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63325_j7mc8n21_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63326_j7mc8n11_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63327_j7mc8n6t_.arc
Media Recovery Waiting for thread 1 sequence 63328
RFS[5]: Selected log 20 for thread 1 sequence 63328 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:23 2021
Primary database is in MAXIMUM PERFORMANCE mode
Re-archiving standby log 20 thread 1 sequence 63328
Fri Apr 16 23:28:23 2021
Archived Log entry 54472 added for thread 1 sequence 63328 ID 0x5b2c46ea dest 1:
RFS[9]: Assigned to RFS process 16555
RFS[9]: Selected log 20 for thread 1 sequence 63329 dbid 1442395283 branch 914614547
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63328_j7mc8q4n_.arc
Media Recovery Waiting for thread 2 sequence 41619
Fri Apr 16 23:28:24 2021
Primary database is in MAXIMUM PERFORMANCE mode
RFS[10]: Assigned to RFS process 16557
RFS[10]: Selected log 30 for thread 2 sequence 41620 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:24 2021
RFS[11]: Assigned to RFS process 16559
RFS[11]: Selected log 31 for thread 2 sequence 41619 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:24 2021
Archived Log entry 54473 added for thread 2 sequence 41619 ID 0x5b2c46ea dest 1:
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41619_j7mc8rdz_.arc
Media Recovery Waiting for thread 1 sequence 63329 (in transit)
Recovery of Online Redo Log: Thread 1 Group 20 Seq 63329 Reading mem 0
  Mem# 0: /u01/app/oradata/orcl/std_redo20.log
Media Recovery Waiting for thread 2 sequence 41620 (in transit)
Recovery of Online Redo Log: Thread 2 Group 30 Seq 41620 Reading mem 0
  Mem# 0: /u01/app/oradata/orcl/std_redo30.log

mos上有类似文章供参考:MRP process getting terminated with error ORA-10485 (Doc ID 1618485.1)

记录一次oracle现场故障处理经过

Posted on by

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：记录一次oracle现场故障处理经过

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

近期到现场进行了一个数据库恢复,我在恢复之前该库先由于硬件进行恢复,然后由其他人对其进行了一系列数据库恢复,但是未恢复成功,客户希望我们到现场进行处理(因为网络原因无法远程).接手库之后,处理第一个问题,是客户在进行现场备份的时候(把linux数据拷贝到win的过程中)发现有几个文件拷贝异常,这个错误很可能是由于当初的硬件故障修复之后留下的后遗症(由于io设备错误,无法运行此项请求),通过工具进行拷贝,恢复出来
20210403210131

DUL> copy file from  /oradata2/xifenfeidata.dbf to /oradata2/xifenfeidata.dbf

starting copy datafile '/oradata1/xifenfeidata.dbf' to '/oradata2/xifenfeidata.dbf'
read data error from file '/oradata1/xifenfeidata.dbf'.error message:Input/output error
read block# error: 560171
read data error from file '/oradata1/xifenfeidata.dbf'.error message:Input/output error
read block# error: 560179
datafile copy completed with 2 block error.

[oracle@localhost ~]$ dbv file=/oradata2/xifenfeidata.dbf blocksize=16384

DBVERIFY: Release 11.2.0.3.0 - Production on Mon Mar 29 17:28:17 2021

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /oradata2/xifenfeidata.dbf
Page 560171 is marked corrupt
Corrupt block relative dba: 0x3bc88c2b (file 239, block 560171)
Completely zero block found during dbv: 

Page 560179 is marked corrupt
Corrupt block relative dba: 0x3bc88c33 (file 239, block 560179)
Completely zero block found during dbv: 



DBVERIFY - Verification complete

Total Pages Examined         : 4194302
Total Pages Processed (Data) : 2230726
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 1936953
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 26618
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 3
Total Pages Marked Corrupt   : 2
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 304929867 (106.304929867)

修复完相关无法拷贝文件之后,启动数据库报控制文件异常

Mon Mar 29 15:03:38 2021
alter database mount
USER (ospid: 29044): terminating the instance
Mon Mar 29 15:03:42 2021
System state dump requested by (instance=1, osid=29044), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/xff/xff/trace/xff_diag_28961.trc
Instance terminated by USER, pid = 29044

尝试重建ctl

[oracle@localhost ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Mar 29 17:40:17 2021

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount pfile='/tmp/pfile'
ORACLE instance started.

Total System Global Area 1.7704E+10 bytes
Fixed Size                  2235568 bytes
Variable Size            2348811088 bytes
Database Buffers         1.5301E+10 bytes
Redo Buffers               52580352 bytes
SQL> @/tmp/ctl.sql
CREATE CONTROLFILE REUSE DATABASE xff NORESETLOGS  NOARCHIVELOG
*
ERROR at line 1:
ORA-01503: CREATE CONTROLFILE failed
ORA-01189: file is from a different RESETLOGS than previous files
ORA-01110: data file 249: '/oradata/xff/system03.dbf'

初步判断是由于对方之前恢复导致部分文件resetlogs scn异常,通过bbed进行判断确认

BBED> set file 1
        FILE#           1

BBED> p kcvfhrls
struct kcvfhrls, 8 bytes                    @116     
   ub4 kscnbas                              @116      0x00000001
   ub2 kscnwrp                              @120      0x0000

BBED> set file 249
        FILE#           249

BBED> p kcvfhrls
struct kcvfhrls, 8 bytes                    @116     
   ub4 kscnbas                              @116      0x00000001
   ub2 kscnwrp                              @120      0x0000

通过bbed修改相关值,然后重建控制文件成功,尝试resetlogs库，报ORA-01248错误

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01248: file 234 was created in the future of incomplete recovery
ORA-01110: data file 234: '/oradata1/xifenfeidata5.DBF'

关于ORA-01248的错误解释

01248, 00000, "file %s was created in the future of incomplete recovery"
// *Cause:  Attempting to do a RESETLOGS open with a file entry in the
//          control file that was originally created after the UNTIL time 
//          of the incomplete recovery.
//          Allowing such an entry may hide the version of the file that 
//          is needed at this time.  The file number may be in use for 
//          a different file which would be lost if the RESETLOGS was allowed.
// *Action: If more recovery is desired then apply redo until the creation
//          time of the file is reached. If the file is not wanted and the
//          same file number is not in use at the stop time of the recovery,
//          then the file can be taken offline with the FOR DROP option.
//          Otherwise a different control file is needed to allow the RESETLOGS.
//          Another backup can be restored and recovered, or a control file can
//          be created via CREATE CONTROLFILE.

大概的意思是文件的创建时间大于文件当前的scn,通过查询确实如此

SQL> select file#,CREATION_CHANGE#,CREATION_TIME from v$datafile_header where file#=234;

           FILE# CREATION_CHANGE# CREATION_
---------------- ---------------- ---------
             234     419298664864 02-AUG-19

SQL> SELECT status,  
  2  to_char(checkpoint_change#,'9999999999999999') "SCN",
  3  to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,FUZZY,
  4  count(*) ROW_NUM
  5  FROM v$datafile_header
  6  GROUP BY status, checkpoint_change#, to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss'),fuzzy
  7  ORDER BY status, checkpoint_change#, checkpoint_time;

STATUS  SCN               CHECKPOINT_TIME     FUZ          ROW_NUM
------- ----------------- ------------------- --- ----------------
ONLINE       417750848223 2021-02-23 23:50:46 YES                7
ONLINE       417750848223 2021-03-21 11:44:25 NO               396

通过对部分scn进行修改（比如减小创建时间的scn)，然后尝试resetlogs库

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 5 with name
"_SYSSMU5_2708889888$" too small
Process ID: 3182
Session ID: 1 Serial number: 3

这个错误比较简单,参考以前的部分文章:在数据库open过程中常遇到ORA-01555汇总数据库open过程遭遇ORA-1555对应sql语句补充,处理之后,数据库open成功

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1.7704E+10 bytes
Fixed Size                  2235568 bytes
Variable Size            2348811088 bytes
Database Buffers         1.5301E+10 bytes
Redo Buffers               52580352 bytes
Database mounted.
SQL> alter database open;

Database altered.

本次数据库恢复基本上完成,已经最大限度恢复数据,导出数据到新库,完成恢复任务