11.2.0.4 打patch遭遇extjobO: Operation not permitted处理

联系:手机/微信(+86 13429648788) QQ(107644445)QQ咨询惜分飞

标题:11.2.0.4 打patch遭遇extjobO: Operation not permitted处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有一段时间没有给11.2.0.4打psu补丁了,今天晚上在给一个客户打的过程中发现类似错误

OPatch found the word "error" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
chmod: changing permissions of `/u01/app/oracle/product/11.2.0/db_1/bin/extjobO': Operation not permitted
make: [iextjob] Error 1 (ignored)


Composite patch 31537677 successfully applied.
OPatch Session completed with warnings.
Log file location: /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-04-16_23-11-48PM_1.log

OPatch completed with warnings.

对比extjob权限信息

[oracle@localhost trace]$ ls -l /u01/app/oracle/product/11.2.0/db_1/bin/extjob*
-rwxr-xr-x 1 oracle oinstall 1254496 Apr 16 23:15 /u01/app/oracle/product/11.2.0/db_1/bin/extjob
-rwx------ 1 oracle oinstall 1254496 Apr 16 23:15 /u01/app/oracle/product/11.2.0/db_1/bin/extjobo
-rwsr-x--- 1 root   oinstall 1254244 Feb 13  2019 /u01/app/oracle/product/11.2.0/db_1/bin/extjobO
-rw------- 1 oracle oinstall 1254244 Feb 13  2019 /u01/app/oracle/product/11.2.0/db_1/bin/extjoboO

证明新的extjob程序确实权限不对,不过根据官方的建议可以忽略这个,参考link:

https://updates.oracle.com/Orion/Services/download?type=readme&aru=20775452

Applying Proactive Bundle / PSU Patch fails with Error: “chmod: changing permissions of `$ORACLE_HOME/bin/extjobO’: Operation not permitted” (Doc ID 2265726.1)
给出来的理由是执行root.sh之后权限会变成正常的,另外这个external jobs基本上很少人会使用到,尝试执行root.sh修复

[root@localhost home]# /u01/app/oracle/product/11.2.0/db_1/root.sh
Performing root user operation for Oracle 11g 

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/11.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[root@localhost home]# ls -l  /u01/app/oracle/product/11.2.0/db_1/bin/extjob*
-rwsr-x--- 1 root   oinstall 1254496 Apr 16 23:15 /u01/app/oracle/product/11.2.0/db_1/bin/extjob
-rwx------ 1 oracle oinstall 1254496 Apr 16 23:15 /u01/app/oracle/product/11.2.0/db_1/bin/extjobo
-rwsr-x--- 1 root   oinstall 1254244 Feb 13  2019 /u01/app/oracle/product/11.2.0/db_1/bin/extjobO
-rw------- 1 oracle oinstall 1254244 Feb 13  2019 /u01/app/oracle/product/11.2.0/db_1/bin/extjoboO

确实通过执行root.sh修复该问题.

ORA-10485故障解决

联系:手机/微信(+86 13429648788) QQ(107644445)QQ咨询惜分飞

标题:ORA-10485故障解决

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

今天对主库进行打了psu和ojvm成功之后,对standby库打上了psu和ojvm,然后给备库启动mrp进程发现余下报错

Fri Apr 16 23:18:54 2021
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION
Attempt to start background Managed Standby Recovery process (orcl1)
Fri Apr 16 23:18:54 2021
MRP0 started with pid=32, OS id=15961 
MRP0: Background Managed Standby Recovery process started (orcl1)
 started logmerger process
Fri Apr 16 23:18:59 2021
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 32 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63308_j7m6domc_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41599_j7m70kyx_.arc
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63309_j7m6dop8_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63310_j7m70fmp_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63311_j7m70hlj_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63312_j7m75cnk_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41600_j7m75b77_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41601_j7m75bcx_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63313_j7m7vs0h_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41602_j7m7vn3g_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41603_j7m7vq2b_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41604_j7m81g5r_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63314_j7m81cz2_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63315_j7m81d31_.arc
Fri Apr 16 23:19:10 2021
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41605_j7m8vwz5_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41606_j7m8vzcr_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63316_j7m8vxgm_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41607_j7m8vzfk_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63317_j7m8w213_.arc
Errors with log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63317_j7m8w213_.arc
MRP0: Background Media Recovery terminated with error 10485
Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcl1/trace/orcl1_pr00_15963.trc:
ORA-10485: Real-Time Query cannot be enabled while applying migration redo.
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Fri Apr 16 23:19:12 2021
MRP0: Background Media Recovery process shutdown (orcl1)

由于redo中有migration操作,导致备库mrp应用进程失败,解决此类问题最简单的方法,就是把备库重启到mount状态,然后使用rman进行恢复操作,然后再继续open库,启动mrp解决这类问题

oracle@localhost trace]$ ss

SQL*Plus: Release 11.2.0.4.0 Production on Fri Apr 16 23:24:55 2021

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 4.2758E+10 bytes
Fixed Size                  2262656 bytes
Variable Size            5905582464 bytes
Database Buffers         3.6776E+10 bytes
Redo Buffers               74420224 bytes
Database mounted.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@localhost trace]$ rman target /

Recovery Manager: Release 11.2.0.4.0 - Production on Fri Apr 16 23:25:12 2021

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL (DBID=1442395283, not open)

RMAN> recover database;

Starting recover at 16-APR-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=143 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=286 device type=DISK
allocated channel: ORA_DISK_3
channel ORA_DISK_3: SID=428 device type=DISK
allocated channel: ORA_DISK_4
channel ORA_DISK_4: SID=570 device type=DISK

starting media recovery

archived log for thread 1 with sequence 63317 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63317_j7m8w213_.arc
archived log for thread 1 with sequence 63318 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63318_j7m932ms_.arc
archived log for thread 1 with sequence 63319 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63319_j7m93719_.arc
archived log for thread 1 with sequence 63320 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63320_j7m96b3g_.arc
archived log for thread 2 with sequence 41607 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41607_j7m8vzfk_.arc
archived log for thread 2 with sequence 41608 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41608_j7m963jp_.arc
archived log for thread 2 with sequence 41609 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41609_j7m967wm_.arc
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63317_j7m8w213_.arc thread=1 sequence=63317
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41607_j7m8vzfk_.arc thread=2 sequence=41607
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63318_j7m932ms_.arc thread=1 sequence=63318
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63319_j7m93719_.arc thread=1 sequence=63319
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63320_j7m96b3g_.arc thread=1 sequence=63320
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41608_j7m963jp_.arc thread=2 sequence=41608
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41609_j7m967wm_.arc thread=2 sequence=41609
unable to find archived log
archived log thread=2 sequence=41610
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 04/16/2021 23:25:25
RMAN-06054: media recovery requesting unknown archived log for thread 2 with sequence 41610 and starting SCN of 15880064787
SQL> alter database open;

Database altered.

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION;

Database altered.
Fri Apr 16 23:26:45 2021
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION
Fri Apr 16 23:26:47 2021
RFS[2]: Assigned to RFS process 16538
RFS[2]: Opened log for thread 2 sequence 41611 dbid 1442395283 branch 914614547
Archived Log entry 54459 added for thread 2 sequence 41611 rlc 914614547 ID 0x5b2c46ea dest 2:
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41611_j7mc5qcc_.arc
Media Recovery Waiting for thread 2 sequence 41612
Fetching gap sequence in thread 2, gap sequence 41612-41612
Fri Apr 16 23:26:50 2021
RFS[1]: Opened log for thread 2 sequence 41612 dbid 1442395283 branch 914614547
Archived Log entry 54460 added for thread 2 sequence 41612 rlc 914614547 ID 0x5b2c46ea dest 2:
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41612_j7mc5ttr_.arc
Media Recovery Waiting for thread 2 sequence 41613
Fetching gap sequence in thread 2, gap sequence 41613-41613
RFS[1]: Opened log for thread 2 sequence 41613 dbid 1442395283 branch 914614547
Archived Log entry 54461 added for thread 2 sequence 41613 rlc 914614547 ID 0x5b2c46ea dest 2:
Fri Apr 16 23:26:55 2021
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41613_j7mc5y6k_.arc
Media Recovery Waiting for thread 1 sequence 63323
Deleted Oracle managed file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_03_25/o1_mf_2_40942_j5s61m0d_.arc
RFS[1]: Opened log for thread 1 sequence 63323 dbid 1442395283 branch 914614547
Archived Log entry 54462 added for thread 1 sequence 63323 rlc 914614547 ID 0x5b2c46ea dest 2:
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63323_j7mc61gg_.arc
Media Recovery Waiting for thread 1 sequence 63324
Fri Apr 16 23:28:20 2021
RFS[3]: Assigned to RFS process 16545
RFS[3]: Opened log for thread 2 sequence 41615 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:20 2021
RFS[4]: Assigned to RFS process 16543
RFS[4]: Opened log for thread 1 sequence 63326 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:20 2021
RFS[5]: Assigned to RFS process 16551
RFS[5]: Opened log for thread 1 sequence 63324 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:20 2021
RFS[6]: Assigned to RFS process 16549
RFS[6]: Opened log for thread 2 sequence 41614 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:20 2021
RFS[7]: Assigned to RFS process 16547
RFS[7]: Opened log for thread 1 sequence 63325 dbid 1442395283 branch 914614547
Archived Log entry 54463 added for thread 2 sequence 41615 rlc 914614547 ID 0x5b2c46ea dest 2:
Fri Apr 16 23:28:20 2021
RFS[8]: Assigned to RFS process 16553
RFS[8]: Opened log for thread 2 sequence 41616 dbid 1442395283 branch 914614547
Archived Log entry 54464 added for thread 1 sequence 63326 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54465 added for thread 2 sequence 41616 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54466 added for thread 1 sequence 63325 rlc 914614547 ID 0x5b2c46ea dest 2:
RFS[3]: Opened log for thread 2 sequence 41617 dbid 1442395283 branch 914614547
RFS[8]: Opened log for thread 2 sequence 41618 dbid 1442395283 branch 914614547
RFS[4]: Opened log for thread 1 sequence 63327 dbid 1442395283 branch 914614547
Archived Log entry 54467 added for thread 2 sequence 41617 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54468 added for thread 1 sequence 63327 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54469 added for thread 2 sequence 41618 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54470 added for thread 1 sequence 63324 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54471 added for thread 2 sequence 41614 rlc 914614547 ID 0x5b2c46ea dest 2:
Fri Apr 16 23:28:20 2021
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63324_j7mc8n1n_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41614_j7mc8n1r_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41615_j7mc8n03_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41616_j7mc8n2n_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41617_j7mc8n6m_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41618_j7mc8n6p_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63325_j7mc8n21_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63326_j7mc8n11_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63327_j7mc8n6t_.arc
Media Recovery Waiting for thread 1 sequence 63328
RFS[5]: Selected log 20 for thread 1 sequence 63328 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:23 2021
Primary database is in MAXIMUM PERFORMANCE mode
Re-archiving standby log 20 thread 1 sequence 63328
Fri Apr 16 23:28:23 2021
Archived Log entry 54472 added for thread 1 sequence 63328 ID 0x5b2c46ea dest 1:
RFS[9]: Assigned to RFS process 16555
RFS[9]: Selected log 20 for thread 1 sequence 63329 dbid 1442395283 branch 914614547
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63328_j7mc8q4n_.arc
Media Recovery Waiting for thread 2 sequence 41619
Fri Apr 16 23:28:24 2021
Primary database is in MAXIMUM PERFORMANCE mode
RFS[10]: Assigned to RFS process 16557
RFS[10]: Selected log 30 for thread 2 sequence 41620 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:24 2021
RFS[11]: Assigned to RFS process 16559
RFS[11]: Selected log 31 for thread 2 sequence 41619 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:24 2021
Archived Log entry 54473 added for thread 2 sequence 41619 ID 0x5b2c46ea dest 1:
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41619_j7mc8rdz_.arc
Media Recovery Waiting for thread 1 sequence 63329 (in transit)
Recovery of Online Redo Log: Thread 1 Group 20 Seq 63329 Reading mem 0
  Mem# 0: /u01/app/oradata/orcl/std_redo20.log
Media Recovery Waiting for thread 2 sequence 41620 (in transit)
Recovery of Online Redo Log: Thread 2 Group 30 Seq 41620 Reading mem 0
  Mem# 0: /u01/app/oradata/orcl/std_redo30.log

mos上有类似文章供参考:MRP process getting terminated with error ORA-10485 (Doc ID 1618485.1)

记录一次oracle现场故障处理经过

联系:手机/微信(+86 13429648788) QQ(107644445)QQ咨询惜分飞

标题:记录一次oracle现场故障处理经过

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

近期到现场进行了一个数据库恢复,我在恢复之前该库先由于硬件进行恢复,然后由其他人对其进行了一系列数据库恢复,但是未恢复成功,客户希望我们到现场进行处理(因为网络原因无法远程).接手库之后,处理第一个问题,是客户在进行现场备份的时候(把linux数据拷贝到win的过程中)发现有几个文件拷贝异常,这个错误很可能是由于当初的硬件故障修复之后留下的后遗症(由于io设备错误,无法运行此项请求),通过工具进行拷贝,恢复出来
20210403210131


DUL> copy file from  /oradata2/xifenfeidata.dbf to /oradata2/xifenfeidata.dbf

starting copy datafile '/oradata1/xifenfeidata.dbf' to '/oradata2/xifenfeidata.dbf'
read data error from file '/oradata1/xifenfeidata.dbf'.error message:Input/output error
read block# error: 560171
read data error from file '/oradata1/xifenfeidata.dbf'.error message:Input/output error
read block# error: 560179
datafile copy completed with 2 block error.
[oracle@localhost ~]$ dbv file=/oradata2/xifenfeidata.dbf blocksize=16384

DBVERIFY: Release 11.2.0.3.0 - Production on Mon Mar 29 17:28:17 2021

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /oradata2/xifenfeidata.dbf
Page 560171 is marked corrupt
Corrupt block relative dba: 0x3bc88c2b (file 239, block 560171)
Completely zero block found during dbv: 

Page 560179 is marked corrupt
Corrupt block relative dba: 0x3bc88c33 (file 239, block 560179)
Completely zero block found during dbv: 



DBVERIFY - Verification complete

Total Pages Examined         : 4194302
Total Pages Processed (Data) : 2230726
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 1936953
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 26618
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 3
Total Pages Marked Corrupt   : 2
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 304929867 (106.304929867)

修复完相关无法拷贝文件之后,启动数据库报控制文件异常

Mon Mar 29 15:03:38 2021
alter database mount
USER (ospid: 29044): terminating the instance
Mon Mar 29 15:03:42 2021
System state dump requested by (instance=1, osid=29044), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/xff/xff/trace/xff_diag_28961.trc
Instance terminated by USER, pid = 29044

尝试重建ctl

[oracle@localhost ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Mar 29 17:40:17 2021

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount pfile='/tmp/pfile'
ORACLE instance started.

Total System Global Area 1.7704E+10 bytes
Fixed Size                  2235568 bytes
Variable Size            2348811088 bytes
Database Buffers         1.5301E+10 bytes
Redo Buffers               52580352 bytes
SQL> @/tmp/ctl.sql
CREATE CONTROLFILE REUSE DATABASE xff NORESETLOGS  NOARCHIVELOG
*
ERROR at line 1:
ORA-01503: CREATE CONTROLFILE failed
ORA-01189: file is from a different RESETLOGS than previous files
ORA-01110: data file 249: '/oradata/xff/system03.dbf'

初步判断是由于对方之前恢复导致部分文件resetlogs scn异常,通过bbed进行判断确认

BBED> set file 1
        FILE#           1

BBED> p kcvfhrls
struct kcvfhrls, 8 bytes                    @116     
   ub4 kscnbas                              @116      0x00000001
   ub2 kscnwrp                              @120      0x0000

BBED> set file 249
        FILE#           249

BBED> p kcvfhrls
struct kcvfhrls, 8 bytes                    @116     
   ub4 kscnbas                              @116      0x00000001
   ub2 kscnwrp                              @120      0x0000

通过bbed修改相关值,然后重建控制文件成功,尝试resetlogs库,报ORA-01248错误

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01248: file 234 was created in the future of incomplete recovery
ORA-01110: data file 234: '/oradata1/xifenfeidata5.DBF'

关于ORA-01248的错误解释

01248, 00000, "file %s was created in the future of incomplete recovery"
// *Cause:  Attempting to do a RESETLOGS open with a file entry in the
//          control file that was originally created after the UNTIL time 
//          of the incomplete recovery.
//          Allowing such an entry may hide the version of the file that 
//          is needed at this time.  The file number may be in use for 
//          a different file which would be lost if the RESETLOGS was allowed.
// *Action: If more recovery is desired then apply redo until the creation
//          time of the file is reached. If the file is not wanted and the
//          same file number is not in use at the stop time of the recovery,
//          then the file can be taken offline with the FOR DROP option.
//          Otherwise a different control file is needed to allow the RESETLOGS.
//          Another backup can be restored and recovered, or a control file can
//          be created via CREATE CONTROLFILE.

大概的意思是文件的创建时间大于文件当前的scn,通过查询确实如此

SQL> select file#,CREATION_CHANGE#,CREATION_TIME from v$datafile_header where file#=234;

           FILE# CREATION_CHANGE# CREATION_
---------------- ---------------- ---------
             234     419298664864 02-AUG-19

SQL> SELECT status,  
  2  to_char(checkpoint_change#,'9999999999999999') "SCN",
  3  to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,FUZZY,
  4  count(*) ROW_NUM
  5  FROM v$datafile_header
  6  GROUP BY status, checkpoint_change#, to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss'),fuzzy
  7  ORDER BY status, checkpoint_change#, checkpoint_time;

STATUS  SCN               CHECKPOINT_TIME     FUZ          ROW_NUM
------- ----------------- ------------------- --- ----------------
ONLINE       417750848223 2021-02-23 23:50:46 YES                7
ONLINE       417750848223 2021-03-21 11:44:25 NO               396

通过对部分scn进行修改(比如减小创建时间的scn),然后尝试resetlogs库

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 5 with name
"_SYSSMU5_2708889888$" too small
Process ID: 3182
Session ID: 1 Serial number: 3

这个错误比较简单,参考以前的部分文章:在数据库open过程中常遇到ORA-01555汇总数据库open过程遭遇ORA-1555对应sql语句补充,处理之后,数据库open成功

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1.7704E+10 bytes
Fixed Size                  2235568 bytes
Variable Size            2348811088 bytes
Database Buffers         1.5301E+10 bytes
Redo Buffers               52580352 bytes
Database mounted.
SQL> alter database open;

Database altered.

本次数据库恢复基本上完成,已经最大限度恢复数据,导出数据到新库,完成恢复任务

ORA-600 kcratr_scan_lastbwr 恢复

联系:手机/微信(+86 13429648788) QQ(107644445)QQ咨询惜分飞

标题:ORA-600 kcratr_scan_lastbwr 恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有朋友找到我们,系统断电之后,数据库无法正常启动,报ora-600 kcratr_scan_lastbwr错误

Thu Mar 25 20:33:45 2021
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: alter database mount exclusive
alter database open
Ping without log force is disabled
.
Thu Mar 25 20:33:47 2021
Beginning crash recovery of 1 threads
 parallel recovery started with 32 processes
Thu Mar 25 20:33:47 2021
Started redo scan
Hex dump of (file 10, block 176517) in trace file C:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_4176.trc

Reading datafile 'C:\APP\ADMINISTRATOR\ORADATA\ORCL\XFF.DBF' for corruption at rdba: 0x0282b185 (file 10, block 176517)
Reread (file 10, block 176517) found same corrupt data (logically corrupt)
Write verification failed for File 10 Block 176517 (rdba 0x282b185)
Errors in file C:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_4176.trc  (incident=165355):
ORA-00600: ??????, ??: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Incident details in: C:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_165355\orcl_ora_4176_i165355.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Thu Mar 25 20:33:50 2021
Slave encountered ORA-10388 exception during crash recovery
Thu Mar 25 20:33:50 2021
Slave encountered ORA-10388 exception during crash recovery
Thu Mar 25 20:33:50 2021
Aborting crash recovery due to error 600
Thu Mar 25 20:33:59 2021
Errors in file C:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_4176.trc:
ORA-00600: ??????, ??: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Thu Mar 25 20:33:59 2021
Errors in file C:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_4176.trc:
ORA-00600: ??????, ??: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
ORA-600 signalled during: alter database open...

故障原因,写丢失导致

Crash or instance recovery may fail because of a lost write even
though one of the mirrors has a good copy.  Reading a file header 
can corrupt a good mirror copy with a bad one.
 
Rediscovery Notes:
 ORA-600 [kcratr_scan_lostwrt] or ORA-600 [kcratr_scan_lastbwr] are signaled 
 even though one of the mirrors has a good copy.

解决方案比较简单直接recover顺利open库
20210326113024


ORA-600 16703直接把orachk备份表插入到tab$恢复

联系:手机/微信(+86 13429648788) QQ(107644445)QQ咨询惜分飞

标题:ORA-600 16703直接把orachk备份表插入到tab$恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有一个朋友和我说,他们数据库出现了以下错误ORA-600 16703 错误
20210324195416


他们是在虚拟化环境中,可以恢复到上一个快照点,但是主机启动之后,数据库依旧异常,让我们进行处理

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed Mar 24 17:04:01 2021

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE

SQL> select count(1) from tab$;

  COUNT(1)
----------
         0

很明显tab$已经被清空,数据库无法正常使用.因为库没有crash,尝试把备份的orachk表插入进来

SQL> insert into tab$ select * from ORACHKB514061BDCB10EBA9CF58F3;

6318 rows created.

SQL> commit;

Commit complete.

SQL> select 'DROP TRIGGER '||owner||'."'||TRIGGER_NAME||'";' from dba_triggers w
here TRIGGER_NAME like 'DBMS_%_INTERNAL% '
  2  union all
  3  select 'DROP PROCEDURE '||owner||'."'||a.object_name||'";' from dba_procedu
res a where a.object_name like 'DBMS_%_INTERNAL% '
  4  union all
  5  select 'drop '||object_type||' '||owner||'.'||object_name||';' from dba_obj
ects where object_name in('DBMS_SUPPORT_DBMONITOR','DBMS_SUPPORT_DBMONITORP');

'DROPTRIGGER'||OWNER||'."'||TRIGGER_NAME||'";'
--------------------------------------------------------------------------------

drop PROCEDURE SYS.DBMS_SUPPORT_DBMONITORP;
drop TRIGGER SYS.DBMS_SUPPORT_DBMONITOR;

SQL> drop PROCEDURE SYS.DBMS_SUPPORT_DBMONITORP;

Procedure dropped.

SQL> drop TRIGGER SYS.DBMS_SUPPORT_DBMONITOR;

Trigger dropped.

SQL> commit;

Commit complete.

SQL>

重启数据库,该故障恢复完成,数据完美恢复0丢失.