ORA-00704 ORA-00943故障恢复

Posted on 2021 年 05 月 09 日 by 惜分飞

有朋友找到我们,他们数据库启动报ORA-01092 ORA-00704 ORA-00702错

MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_4868.trc:
ORA-00704: 引导程序进程失败
ORA-00702: 引导程序版本 '' 与版本 '8.0.0.0.0' 不一致
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_4868.trc:
ORA-00704: 引导程序进程失败
ORA-00702: 引导程序版本 '' 与版本 '8.0.0.0.0' 不一致
Error 704 happened during db open, shutting down database
USER (ospid: 4868): terminating the instance due to error 704
Thu May 06 09:20:28 2021
Instance terminated by USER, pid = 4868
ORA-1092 signalled during: ALTER DATABASE OPEN...

以前有过类似恢复经历:
2019恢复第一单—ORA-704-ORA-702恢复
 恶意删除bootstrap$导致数据库无法正常启动
 ORA-00702: bootstrap verison ” inconsistent with version ’8.0.0.0.0′
这个问题本身相对比较简单,客户根据网上介绍,自行尝试恢复,经过客户一系列的处理之后,数据库启动报ORA-00704 ORA-00943错

MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_6600.trc:
ORA-00704: 引导程序进程失败
ORA-00943: 簇不存在
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_6600.trc:
ORA-00704: 引导程序进程失败
ORA-00943: 簇不存在
Error 704 happened during db open, shutting down database
USER (ospid: 6600): terminating the instance due to error 704
Thu May 06 22:17:18 2021
Instance terminated by USER, pid = 6600
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (6600) as a result of ORA-1092

因为根据最初的错误知道是bootstrap$异常,现在报该错误,应该是bootstrap$没有处理正确导致,我们通过对bootstrap$重新处理,数据库直接open成功

win 11.2.0.4打patch后服务无法正常启动处理

Posted on 2021 年 05 月 08 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：win 11.2.0.4打patch后服务无法正常启动处理

最近一直困惑在win 11.2.0.4中打扩展patch的之后,oracle相关无法无法正常启动的问题,今天在朋友的指导下,终于搞定,这里具体描述下这次打patch的相关事宜.
0. win版本信息

1. 关闭oracle相关服务之后,打patch 提示CheckActiveFilesAndExecutables

对于这个问题,可以通过tasklist /M ora*命令找出相关进程,然后杀掉即可

2. 打上psu patch之后,相关服务无法启动

对于这个问题,是由于打上patch之后由于oracle程序调用的需要的MFC动态库不对,需要升级到Microsoft Visual C++ 2005 Service Pack 1 Redistributable Package MFC(https://www.microsoft.com/zh-cn/download/confirmation.aspx?id=26347)

启动相关服务正常

安装相关patch成功

expdp报ORA-39064 ORA-29285错误

Posted on 2021 年 05 月 04 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：expdp报ORA-39064 ORA-29285错误

expdp导出数据,报错ORA-39064 ORA-29285错误,导致datapump的logfile记录的日志不全

. . 导出了 "HCP"."HR_ADJ_SAL_ADV_SETUP_M"                  0 KB       0 行
ORA-39064: 无法写入日志文件
ORA-29285: 文件写入错误
. . 导出了 "HCP"."HR_ADJ_SAL_CONFIRM"                      0 KB       0 行

查询数据库NLS_CHARACTERSET

SQL> select NAME, value$ from props$ where name like 'NLS_CHARACTERSET';

NAME
------------------------------
VALUE$
--------------------------------------------------------------------------------
NLS_CHARACTERSET
UTF8

查看客户端字符集


SQL> select userenv('language') from dual;

USERENV('LANGUAGE')
--------------------------------------------------------------------------------
SIMPLIFIED CHINESE_CHINA.ZHS16GBK

出现这个问题,是由于expdp本身调用UTL_FILE,在Oracle Database PL/SQL Packages and Types Reference中有When data encoded in one character set is read and Globalization Support is told (such as by means of NLS_LANG) that it is encoded in another character set, the result is indeterminate. If NLS_LANG is set, it should be the same as the database character set.
基于这样的情况，通过设置NLS_LANG在客户端字符集和服务端一致,就不会出现该问题

ora-600 2662和ora-600 kclchkblk_4恢复

Posted on 2021 年 04 月 29 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ora-600 2662和ora-600 kclchkblk_4恢复

这两天连续处理两个case,一个是12.1.0.2版本数据库屏蔽一致性,强制open之后,报ORA-600 2662故障

这个错误本身是一个非常常见的错误,直接推scn即可解决,但是问题是12.1.0.2版本,oracle不允许以前常规的操作方法,就连oradebug都报错oradebug poke ORA-32521/ORA-32519故障解决,而且可以是rac环境,bbed修改文件头也相当麻烦,最后我们使用patch方法轻松解决

另外一例是11.2.0.4版本,强制open库报ora-600 kclchkblk_4

Wed Apr 28 21:25:38 2021
SMON: enabling cache recovery
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Errors in file /u01/app/oracle/diag/rdbms/dc/dc1/trace/dc1_ora_27832.trc  (incident=564430):
ORA-00600: internal error code, arguments: [kclchkblk_4], [2959], [904341694], [2959], [904131717], []
Incident details in: /u01/app/oracle/diag/rdbms/dc/dc1/incident/incdir_564430/dc1_ora_27832_i564430.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/dc/dc1/trace/dc1_ora_27832.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kclchkblk_4], [2959], [904341694], [2959], [904131717], []
Errors in file /u01/app/oracle/diag/rdbms/dc/dc1/trace/dc1_ora_27832.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kclchkblk_4], [2959], [904341694], [2959], [904131717], []
Error 704 happened during db open, shutting down database
USER (ospid: 27832): terminating the instance due to error 704
Instance terminated by USER, pid = 27832
ORA-1092 signalled during: alter database open resetlogs...

这个比较简单,参考redo异常 ORA-600 kclchkblk_4 故障恢复.

19c 打RU patch遇到oui-patch.xml (Permission denied)问题

Posted on 2021 年 04 月 28 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：19c 打RU patch遇到oui-patch.xml (Permission denied)问题

有一段时间没有做实施的活,今天安装一套19c rac,并且计划安装Patch 32545008 – GI Release Update 19.11.0.0.210420,在安装过程中遇到oui-patch.xml问题,记录下来供参考:
1. OPATCHAUTO-72088

[root@jbsbdb1 ~]# opatchauto apply /u01/soft/32545008

OPatchauto session is initiated at Wed Apr 28 14:20:24 2021

System initialization log file is /u01/app/19.0/grid/cfgtoollogs/opatchautodb/systemconfig2021-04-28_02-20-29PM.log.

Session log file is /u01/app/19.0/grid/cfgtoollogs/opatchauto/opatchauto2021-04-28_02-20-57PM.log
The id for this session is N2PG

Wrong OPatch software installed in following homes:
Host:jbsbdb2, Home:/u01/app/oracle/product/19.0/db_1

Host:jbsbdb2, Home:/u01/app/19.0/grid

OPATCHAUTO-72088: OPatch version check failed.
OPATCHAUTO-72088: OPatch software version in homes selected for patching are different.
OPATCHAUTO-72088: Please install same OPatch software in all homes.
OPatchAuto failed.

OPatchauto session completed at Wed Apr 28 14:21:15 2021
Time taken to complete the session 0 minute, 51 seconds

 opatchauto failed with error code 42

故障原因,只是升级了节点1的grid和oracle的opatch,把节点2的opatch也升级之后该问题解决

2. oui-patch.xml (Permission denied)文件处理

[root@jbsbdb2 soft]# opatchauto apply /u01/soft/32545008

OPatchauto session is initiated at Wed Apr 28 14:52:29 2021

System initialization log file is /u01/app/19.0/grid/cfgtoollogs/opatchautodb/systemconfig2021-04-28_02-52-32PM.log.

Session log file is /u01/app/19.0/grid/cfgtoollogs/opatchauto/opatchauto2021-04-28_02-52-58PM.log
The id for this session is T6ST

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0/db_1
Patch applicability verified successfully on home /u01/app/19.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.0/db_1


Executing patch validation checks on home /u01/app/19.0/grid
Patch validation checks successfully completed on home /u01/app/19.0/grid


Executing patch validation checks on home /u01/app/oracle/product/19.0/db_1
Patch validation checks successfully completed on home /u01/app/oracle/product/19.0/db_1


Verifying SQL patch applicability on home /u01/app/oracle/product/19.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0/db_1


Preparing to bring down database service on home /u01/app/oracle/product/19.0/db_1
Successfully prepared home /u01/app/oracle/product/19.0/db_1 to bring down database service


Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19.0/grid
Prepatch operation log file location: 
/u01/app/grid/crsdata/jbsbdb2/crsconfig/crs_prepatch_apply_inplace_jbsbdb2_2021-04-28_02-54-34PM.log
CRS service brought down successfully on home /u01/app/19.0/grid


Performing prepatch operation on home /u01/app/oracle/product/19.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.0/db_1


Start applying binary patch on home /u01/app/oracle/product/19.0/db_1
Failed while applying binary patches on home /u01/app/oracle/product/19.0/db_1

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : jbsbdb2->/u01/app/oracle/product/19.0/db_1 Type[rac]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/oracle/product/19.0/db_1, host: jbsbdb2.
Command failed:  /u01/app/oracle/product/19.0/db_1/OPatch/opatchauto 
 apply /u01/soft/32545008 -oh /u01/app/oracle/product/19.0/db_1 -target_type rac_database
 -binary -invPtrLoc /u01/app/19.0/grid/oraInst.loc -jre /u01/app/19.0/grid/OPatch/jre -persistresult
/u01/app/oracle/product/19.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_jbsbdb2_rac_2.ser -analyzedresult
 /u01/app/oracle/product/19.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_jbsbdb2_rac_2.ser
Command failure output: 
==Following patches FAILED in apply:

Patch: /u01/soft/32545008/32545013
Log: /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2021-04-28_15-00-49PM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: 
ApplySession failed in system modification phase... 
'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: 
java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)' 

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Wed Apr 28 15:05:29 2021
Time taken to complete the session 13 minutes, 0 second

 opatchauto failed with error code 42

在节点1打patch成功之后,对节点2进行打patch,遇到类似:ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException:
java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)问题,通过查询mos发现类似文章opatchauto apply Results java.io.FileNotFoundException: /ContentsXML/oui-patch.xml (Permission denied) Error in Non-OUI Nodes (Doc ID 2582139.1),确认是bug 29859410 在20.1版本中修复.检查系统中对应oui-patch.xml信息.

[root@jbsbdb2 soft]# ls -l /u01/app/oraInventory/ContentsXML/oui-patch.xml
-rw-r----- 1 grid oinstall 174 Apr 28 14:04 /u01/app/oraInventory/ContentsXML/oui-patch.xml

确实文件权限不对,对其授权处理

[root@jbsbdb2 soft]# chmod 660 /u01/app/oraInventory/ContentsXML/oui-patch.xml
[root@jbsbdb2 soft]# ls -l /u01/app/oraInventory/ContentsXML/oui-patch.xml
-rw-rw---- 1 grid oinstall 174 Apr 28 14:04 /u01/app/oraInventory/ContentsXML/oui-patch.xml

处理oui-patch.xml文件异常之后,尝试回滚操作

[root@jbsbdb2 soft]# opatchauto rollback /u01/soft/32545008

OPatchauto session is initiated at Wed Apr 28 15:13:58 2021

System initialization log file is /u01/app/19.0/grid/cfgtoollogs/opatchautodb/systemconfig2021-04-28_03-14-00PM.log.

Session log file is /u01/app/19.0/grid/cfgtoollogs/opatchauto/opatchauto2021-04-28_03-14-28PM.log
The id for this session is KFYI

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0/db_1

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0/grid
Patch applicability verified successfully on home /u01/app/19.0/grid

Patch applicability verification failed on home /u01/app/oracle/product/19.0/db_1

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : jbsbdb2->/u01/app/oracle/product/19.0/db_1 Type[rac]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/oracle/product/19.0/db_1, host: jbsbdb2.
Command failed:  /u01/app/oracle/product/19.0/db_1/OPatch/opatchauto  rollback
 /u01/soft/32545008 -oh /u01/app/oracle/product/19.0/db_1 
-target_type rac_database -binary -invPtrLoc /u01/app/19.0/grid/oraInst.loc 
-jre /u01/app/19.0/grid/OPatch/jre -persistresult 
/u01/app/oracle/product/19.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_jbsbdb2_rac_2.ser
 -analyze -online -prepare_home
Command failure output: 
==Following patches FAILED in analysis for rollback:

Patch: /u01/soft/32545008/32579761
Log: 
Reason: Failed during listing in Analysis: java.lang.Exception: oracle.opatch.opatchsdk.OPatchException: 
Unable to create patchObject
Possible causes are:
   ORACLE_HOME/inventory/oneoffs/32545013 is corrupted. PatchObject constructor: 
Input file "/u01/app/oracle/product/19.0/db_1/inventory/oneoffs/32545013/etc/config/actions" 
or "/u01/app/oracle/product/19.0/db_1/inventory/oneoffs/32545013/etc/config/inventory" does not exist.


Patch: /u01/soft/32545008/32545013
Log: 
Reason: Failed during listing in Analysis: java.lang.Exception: oracle.opatch.opatchsdk.OPatchException: 
Unable to create patchObject
Possible causes are:
   ORACLE_HOME/inventory/oneoffs/32545013 is corrupted. PatchObject constructor:
 Input file "/u01/app/oracle/product/19.0/db_1/inventory/oneoffs/32545013/etc/config/actions" 
or "/u01/app/oracle/product/19.0/db_1/inventory/oneoffs/32545013/etc/config/inventory" does not exist. 

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Wed Apr 28 15:14:50 2021
Time taken to complete the session 0 minute, 53 seconds

 opatchauto failed with error code 42

检查发现/u01/app/oracle/product/19.0/db_1/inventory/oneoffs/32545013确实不存在,从节点1把该文件夹tar过来,然后再次回滚

[root@jbsbdb2 ~]# /u01/app/oracle/product/19.0/db_1/OPatch/opatchauto rollback /u01/soft/32545008 -oh 
>/u01/app/oracle/product/19.0/db_1

OPatchauto session is initiated at Wed Apr 28 16:12:33 2021

System initialization log file is 
/u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatchautodb/systemconfig2021-04-28_04-12-36PM.log.

Session log file is /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatchauto/opatchauto2021-04-28_04-12-53PM.log
The id for this session is JRS3

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.0/db_1


Executing patch validation checks on home /u01/app/oracle/product/19.0/db_1
Patch validation checks successfully completed on home /u01/app/oracle/product/19.0/db_1


Verifying SQL patch applicability on home /u01/app/oracle/product/19.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0/db_1


Preparing to bring down database service on home /u01/app/oracle/product/19.0/db_1
Successfully prepared home /u01/app/oracle/product/19.0/db_1 to bring down database service


Bringing down database service on home /u01/app/oracle/product/19.0/db_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: racdb
Database service successfully brought down on home /u01/app/oracle/product/19.0/db_1


Performing prepatch operation on home /u01/app/oracle/product/19.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.0/db_1


Start rolling back binary patch on home /u01/app/oracle/product/19.0/db_1
Binary patch rolled back successfully on home /u01/app/oracle/product/19.0/db_1


Performing postpatch operation on home /u01/app/oracle/product/19.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.0/db_1


Starting database service on home /u01/app/oracle/product/19.0/db_1
Database service successfully started on home /u01/app/oracle/product/19.0/db_1


Preparing home /u01/app/oracle/product/19.0/db_1 after database service restarted
No step execution required.........
 

Trying to roll back SQL patch on home /u01/app/oracle/product/19.0/db_1
SQL patch rolled back successfully on home /u01/app/oracle/product/19.0/db_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:jbsbdb2
RAC Home:/u01/app/oracle/product/19.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/soft/32545008/32576499
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/soft/32545008/32585572
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/soft/32545008/32584670
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/soft/32545008/32579761
Reason: This Patch does not exist in the home, it cannot be rolled back.


==Following patches were SUCCESSFULLY rolled back:

Patch: /u01/soft/32545008/32545013
Log: /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2021-04-28_16-15-05PM_1.log

回滚成功,直接使用opatchauto apply /u01/soft/32545008打节点2成功
3. RU安装成功结果

[grid@jbsbdb2 ~]$ opatch lspatches
32585572;DBWLM RELEASE UPDATE 19.0.0.0.0 (32585572)
32584670;TOMCAT RELEASE UPDATE 19.0.0.0.0 (32584670)
32579761;OCW RELEASE UPDATE 19.11.0.0.0 (32579761)
32576499;ACFS RELEASE UPDATE 19.11.0.0.0 (32576499)
32545013;Database Release Update : 19.11.0.0.210420 (32545013)

OPatch succeeded.

[oracle@jbsbdb2 ~]$ opatch lspatches
32579761;OCW RELEASE UPDATE 19.11.0.0.0 (32579761)
32545013;Database Release Update : 19.11.0.0.210420 (32545013)

OPatch succeeded.
[grid@jbsbdb2 ~]$ crsctl query crs softwarepatch
Oracle Clusterware patch level on node jbsbdb2 is [3331580692].
[grid@jbsbdb2 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. 
The cluster upgrade state is [NORMAL]. The cluster active patch level is [3331580692].

SQL> select PATCH_ID,DESCRIPTION from dba_registry_sqlpatch;

  PATCH_ID
----------
DESCRIPTION
--------------------------------------------------------------------------------
  29517242
Database Release Update : 19.3.0.0.190416 (29517242)

  32545013
Database Release Update : 19.11.0.0.210420 (32545013)

在打patch之前已经知晓该问题,比较粗心的把oui-patch.xml文件从节点1scp拷贝到节点2,没有确认oui-patch.xml权限(拷贝过来是640)从而引起后续很多麻烦