ORA-600 16703故障再现

Posted on 2024 年 09 月 16 日 by 惜分飞

从第一次发现ORA-600 16703(警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703)至今已经7年多时间了,最近依旧有客户中招,提醒各位注意该问题

Sat Sep 14 21:43:29 2024
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sat Sep 14 21:43:29 2024
SMON: enabling cache recovery
Errors in file D:\ORACLE\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_6264.trc  (incident=8561):
ORA-00600: 内部错误代码, 参数: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Incident details in: D:\ORACLE\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_8561\orcl_ora_6264_i8561.trc
Sat Sep 14 21:43:31 2024
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file D:\ORACLE\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_6264.trc:
ORA-00704: 引导程序进程失败
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Errors in file D:\ORACLE\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_6264.trc:
ORA-00704: 引导程序进程失败
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 6264): terminating the instance due to error 704
Instance terminated by USER, pid = 6264
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (6264) as a result of ORA-1092

由于此类故障出现较多,破坏性加大,对其进行了深入的研究,在没有破坏现场的情况下,通过对tab$进行直接重建,实现数据库完美恢复(数据0丢失,数据库无需逻辑迁移[原库直接可用])

以前关于此类报错的文章:
10g数据库遭遇ORA-600 16703
12C数据库遭遇ORA-600 16703
ORA-600 kzrini:!uprofile处理
ORA-600 16703故障解析—tab$表被清空
近期又遇到ORA-600 16703和ORA-702故障
ORA-00600: internal error code, arguments: [16703], [1403], [4] 原因
tab$异常被处理之后报ORA-600 13304故障处理
最近遇到几起ORA-600 16703故障(tab$被清空),请引起重视
ORA-600 16703直接把orachk备份表插入到tab$恢复
警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703
aix平台tab$被删除可能出现ORA-600 [16703], [1403], [28]错误
ORA-00600: internal error code, arguments: [16703], [1403], [4] 故障处理
ORA-00600: internal error code, arguments: [16703], [1403], [32]
ORA-600 16703故障,客户找人恢复数据库,数据库被进一步恶意破坏—ORA-00704 ORA-00922
尽可能不要从互联网下载Oracle安装介质和Patch,避免被注入恶意脚本,并检查已经存在的安装介质的sha256码

数据库启动报ORA-27102 OSD-00026 O/S-Error: (OS 1455)

Posted on 2024 年 09 月 13 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：数据库启动报ORA-27102 OSD-00026 O/S-Error: (OS 1455)

一个win上运行的11.2.0.4库启动的时候报ORA-27102 OSD-00026 O/S-Error: (OS 1455)错误

SQL> STARTUP
ORA-27102: out of memory
OSD-00026: 附加错误信息
O/S-Error: (OS 1455) 页面文件太小，无法完成操作。

数据库alert日志信息

Fri Sep 13 17:17:39 2024
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 24
Number of processor cores in the system is 12
Number of processor sockets in the system is 1
Picked latch-free SCN scheme 3
Error: Failed to allocate SGA granule addr 0000000760000000 size 268435456 
 mode 131073 locality 0
Errors in file D:\app\Administrator\diag\rdbms\xff\xff\trace\xff_ora_77728.trc:
ORA-27102: out of memory
OSD-00026: 附加错误信息
O/S-Error: (OS 1455) 页面文件太小，无法完成操作。
Error: Failed to allocate SGA granule addr 0000000750000000 size 268435456 
 mode 131073 locality 0
Errors in file D:\app\Administrator\diag\rdbms\xff\xff\trace\xff_ora_77728.trc:

看报错信息,第一感觉和内存有关系,可能内存不足无法满足sga分配需求,查看系统空闲内存情况

系统明显有足够内存,出现该问题的原因可能和win操作系统本身有关系,由于要快速恢复业务,直接重启系统数据库启动成功

RAC默认服务配置优先节点

Posted on 2024 年 09 月 07 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：RAC默认服务配置优先节点

在某些rac情况下需要,需要对数据库默认的db_name对应的服务名进行修改,实现优先节点访问的效果.分析了下在默认值情况下,db_name影响到db_unique_name,然后决定了数据库的service_names.现有库的db_name无法修改,那就只能在db_unique_name上做手脚(只是修改service_names,对应的db_unique_name还是会创建默认服务,这样该服务依旧可以连接).但是在rac环境中db_unique_name记录到了crs资源之中,无法直接在数据库级别修改(修改会报ORA-32017 ORA-65500错误)

SQL> alter system set db_unique_name='nxifenfei' sid='*' scope=spfile;
alter system set db_unique_name='nxifenfei' sid='*' scope=spfile
*
ERROR at line 1:
ORA-32017: failure in updating SPFILE
ORA-65500: could not modify DB_UNIQUE_NAME, resource exists

只能先删除crs中关于db的资源,然后再进行修改服务名,再增加db资源

[oracle@xffdb1 ~]$ srvctl remove database -d xifenfei  -f

SQL> alter system set db_unique_name='nxifenfei' sid='*' scope=spfile;

[oracle@xffdb1 ~]$ srvctl add database -d nxifenfei -o /u01/app/oracle/product/19c/db_1 -p \
  +DATADG/XIFENFEI/PARAMETERFILE/spfile.271.1174153165 -pwfile +DATADG/XIFENFEI/PASSWORD/pwdxifenfei.256.1174152463
[oracle@xffdb1 ~]$ srvctl add instance -d nxifenfei -i xifenfei1 -n xffdb1
[oracle@xffdb1 ~]$ srvctl add instance -d nxifenfei -i xifenfei2 -n xffdb2
[oracle@xffdb1 ~]$ srvctl add instance -d nxifenfei -i xifenfei3 -n xffdb3

创建新服务(和db_name同名,和现在的db_unique_name不同名)

[oracle@xffdb1 ~]$ srvctl add service -db nxifenfei -service xifenfei -r xifenfei2 -a xifenfei1,xifenfei3 \
  -failovertype SESSION -failovermethod BASIC -failoverdelay 10 -failoverretry 3 -failback YES
[oracle@xffdb1 ~]$ srvctl start service -db nxifenfei -service xifenfei

[oracle@xffdb1 ~]$ srvctl config service -d nxifenfei -service xifenfei
Service name: xifenfei
Server pool:
Cardinality: 1
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type: SESSION
Failover method: BASIC
Failover retries: 3
Failover delay: 10
Failover restore: NONE
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name:
Hub service:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Failback :  yes
Replay Initiation Time: 300 seconds
Drain timeout:
Stop option:
Session State Consistency: DYNAMIC
GSM Flags: 0
Service is enabled
Preferred instances: xifenfei2
Available instances: xifenfei1,xifenfei3
CSS critical: no
Service uses Java: false
[grid@xffdb1 ~]$

服务的其他操作

--调整服务的优先节点
srvctl modify service -db nxifenfei -service xifenfei -modifyconfig -preferred "xifenfei1" -available "xifenfei2,xifenfei3"
srvctl stop service -db nxifenfei -service xifenfei 
srvctl start service -db nxifenfei -service xifenfei 

--切换服务所在节点
srvctl relocate service -db nxifenfei -service xifenfei -oldinst xifenfei2 -newinst xifenfei1

--删除服务
srvctl stop service -db nxifenfei -service xifenfei
srvctl remove service -db nxifenfei -service xifenfei

Oracle 19c RAC 替换私网操作

Posted on 2024 年 09 月 06 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle 19c RAC 替换私网操作

19c的三节点集群,需要替换一个私网网卡(如果有足够的停机窗口有一个更加简单的方法,直接通过修改网卡名称实现替换,不过需要主机重启一次,参考:Linux 8 修改网卡名称)
1. 先在主机层面确认新配置网络能够相互ping通,在hosts文件加入私网信息,并且确认ssh 可以相互访问

ssh xffdb1-priv3 date;ssh xffdb2-priv3 date;ssh xffdb3-priv3 date;

2. 删除掉需要删除的网络上的asm监听和该network信息

[grid@xffdb1 ~]$ srvctl config listener -asmlistener
Name: ASMNET1LSNR_ASM
Type: ASM Listener
Owner: grid
Subnet: 172.16.16.0
Home: <CRS home>
End points: TCP:1525
Listener is enabled.
Listener is individually enabled on nodes:
Listener is individually disabled on nodes:
Name: ASMNET2LSNR_ASM
Type: ASM Listener
Owner: grid
Subnet: 172.17.17.0
Home: <CRS home>
End points: TCP:1526
Listener is enabled.
Listener is individually enabled on nodes:
Listener is individually disabled on nodes:
[grid@xffdb1 ~]$ srvctl config asmnetwork
ASM network 1 exists
Subnet IPv4: 172.16.16.0//
Subnet IPv6:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
ASM network 2 exists
Subnet IPv4: 172.17.17.0//
Subnet IPv6:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
[grid@xffdb1 ~]$

[grid@xffdb3 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +DATA/orapwASM
Backup of Password file: +DATA/orapwASM_backup
ASM listener: LISTENER
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM,ASMNET2LSNR_ASM
[grid@xffdb3 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       xffdb1                   STABLE
               ONLINE  ONLINE       xffdb2                   STABLE
               ONLINE  ONLINE       xffdb3                   STABLE
ora.chad
               ONLINE  ONLINE       xffdb1                   STABLE
               ONLINE  ONLINE       xffdb2                   STABLE
               ONLINE  ONLINE       xffdb3                   STABLE
ora.net1.network
               ONLINE  ONLINE       xffdb1                   STABLE
               ONLINE  ONLINE       xffdb2                   STABLE
               ONLINE  ONLINE       xffdb3                   STABLE
ora.ons
               ONLINE  ONLINE       xffdb1                   STABLE
               ONLINE  ONLINE       xffdb2                   STABLE
               ONLINE  ONLINE       xffdb3                   STABLE
ora.proxy_advm
               OFFLINE OFFLINE      xffdb1                   STABLE
               OFFLINE OFFLINE      xffdb2                   STABLE
               OFFLINE OFFLINE      xffdb3                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       xffdb1                   STABLE
      2        ONLINE  ONLINE       xffdb2                   STABLE
      3        ONLINE  ONLINE       xffdb3                   STABLE
ora.ASMNET2LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       xffdb1                   STABLE
      2        ONLINE  ONLINE       xffdb2                   STABLE
      3        ONLINE  ONLINE       xffdb3                   STABLE
ora.OCR.dg(ora.asmgroup)
      1        ONLINE  ONLINE       xffdb1                   STABLE
      2        ONLINE  ONLINE       xffdb2                   STABLE
      3        ONLINE  ONLINE       xffdb3                   STABLE
ora.DATADG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       xffdb1                   STABLE
      2        ONLINE  ONLINE       xffdb2                   STABLE
      3        ONLINE  ONLINE       xffdb3                   STABLE
ora.FRADG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       xffdb1                   STABLE
      2        ONLINE  ONLINE       xffdb2                   STABLE
      3        ONLINE  ONLINE       xffdb3                   STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       xffdb2                   STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       xffdb1                   STABLE
      2        ONLINE  ONLINE       xffdb2                   STABLE
      3        ONLINE  ONLINE       xffdb3                   Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       xffdb1                   STABLE
      2        ONLINE  ONLINE       xffdb2                   STABLE
      3        ONLINE  ONLINE       xffdb3                   STABLE
ora.asmnet2.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       xffdb1                   STABLE
      2        ONLINE  ONLINE       xffdb2                   STABLE
      3        ONLINE  ONLINE       xffdb3                   STABLE
ora.cvu
      1        ONLINE  ONLINE       xffdb2                   STABLE
ora.xffdb1.vip
      1        ONLINE  ONLINE       xffdb1                   STABLE
ora.xffdb2.vip
      1        ONLINE  ONLINE       xffdb2                   STABLE
ora.xffdb3.vip
      1        ONLINE  ONLINE       xffdb3                   STABLE
ora.xifenfei.db
      1        ONLINE  ONLINE       xffdb1                   Open,HOME=/u01/app/o
                                                             racle/product/19c/db
                                                             _1,STABLE
      2        ONLINE  ONLINE       xffdb2                   Open,HOME=/u01/app/o
                                                             racle/product/19c/db
                                                             _1,STABLE
      3        ONLINE  ONLINE       xffdb3                   Open,HOME=/u01/app/o
                                                             racle/product/19c/db
                                                             _1,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       xffdb2                   STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       xffdb2                   STABLE
--------------------------------------------------------------------------------

[grid@xffdb1 peer]$ srvctl update listener -listener ASMNET2LSNR_ASM -asm -remove -force
[grid@xffdb1 peer]$ srvctl remove asmnetwork -netnum 2 -force
PRCR-1028 : Failed to remove resource ora.asmnet2.asmnetwork
PRCR-1072 : Failed to unregister resource ora.asmnet2.asmnetwork
CRS-0245:  User doesn't have enough privilege to perform the operation
[root@xffdb1 ~]# source /home/grid/.bash_profile
[root@xffdb1 ~]# srvctl remove asmnetwork -netnum 2 -force
[root@xffdb1 ~]#
[root@xffdb1 ~]#
[root@xffdb1 ~]# crsctl status res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       xffdb1                   STABLE
               ONLINE  ONLINE       xffdb2                   STABLE
               ONLINE  ONLINE       xffdb3                   STABLE
ora.chad
               ONLINE  ONLINE       xffdb1                   STABLE
               ONLINE  ONLINE       xffdb2                   STABLE
               ONLINE  ONLINE       xffdb3                   STABLE
ora.net1.network
               ONLINE  ONLINE       xffdb1                   STABLE
               ONLINE  ONLINE       xffdb2                   STABLE
               ONLINE  ONLINE       xffdb3                   STABLE
ora.ons
               ONLINE  ONLINE       xffdb1                   STABLE
               ONLINE  ONLINE       xffdb2                   STABLE
               ONLINE  ONLINE       xffdb3                   STABLE
ora.proxy_advm
               OFFLINE OFFLINE      xffdb1                   STABLE
               OFFLINE OFFLINE      xffdb2                   STABLE
               OFFLINE OFFLINE      xffdb3                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       xffdb1                   STABLE
      2        ONLINE  ONLINE       xffdb2                   STABLE
      3        ONLINE  ONLINE       xffdb3                   STABLE
ora.OCR.dg(ora.asmgroup)
      1        ONLINE  ONLINE       xffdb1                   STABLE
      2        ONLINE  ONLINE       xffdb2                   STABLE
      3        ONLINE  ONLINE       xffdb3                   STABLE
ora.DATADG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       xffdb1                   STABLE
      2        ONLINE  ONLINE       xffdb2                   STABLE
      3        ONLINE  ONLINE       xffdb3                   STABLE
ora.FRADG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       xffdb1                   STABLE
      2        ONLINE  ONLINE       xffdb2                   STABLE
      3        ONLINE  ONLINE       xffdb3                   STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       xffdb2                   STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       xffdb1                   STABLE
      2        ONLINE  ONLINE       xffdb2                   STABLE
      3        ONLINE  ONLINE       xffdb3                   Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       xffdb1                   STABLE
      2        ONLINE  ONLINE       xffdb2                   STABLE
      3        ONLINE  ONLINE       xffdb3                   STABLE
ora.cvu
      1        ONLINE  ONLINE       xffdb2                   STABLE
ora.xffdb1.vip
      1        ONLINE  ONLINE       xffdb1                   STABLE
ora.xffdb2.vip
      1        ONLINE  ONLINE       xffdb2                   STABLE
ora.xffdb3.vip
      1        ONLINE  ONLINE       xffdb3                   STABLE
ora.xifenfei.db
      1        ONLINE  ONLINE       xffdb1                   Open,HOME=/u01/app/o
                                                             racle/product/19c/db
                                                             _1,STABLE
      2        ONLINE  ONLINE       xffdb2                   Open,HOME=/u01/app/o
                                                             racle/product/19c/db
                                                             _1,STABLE
      3        ONLINE  ONLINE       xffdb3                   Open,HOME=/u01/app/o
                                                             racle/product/19c/db
                                                             _1,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       xffdb2                   STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       xffdb2                   STABLE
--------------------------------------------------------------------------------
[grid@xffdb2 peer]$ srvctl config listener -asmlistener
Name: ASMNET1LSNR_ASM
Type: ASM Listener
Owner: grid
Subnet: 172.16.16.0
Home: <CRS home>
End points: TCP:1525
Listener is enabled.
Listener is individually enabled on nodes:
Listener is individually disabled on nodes:
[grid@xffdb2 peer]$ srvctl config asmnetwork
ASM network 1 exists
Subnet IPv4: 172.16.16.0//
Subnet IPv6:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:

3. 替换集群私网操作

[grid@xffdb1 ~]$ oifcfg getif
bond0  192.168.20.0  global  public
ens9f0  172.16.16.0  global  cluster_interconnect,asm
ens9f1  172.17.17.0  global  cluster_interconnect,asm
[grid@xffdb1 ~]$ oifcfg setif -global ens6f0np0/172.18.18.0:cluster_interconnect,asm
[grid@xffdb1 ~]$ oifcfg getif
bond0  192.168.20.0  global  public
ens9f0  172.16.16.0  global  cluster_interconnect,asm
ens9f1  172.17.17.0  global  cluster_interconnect,asm
ens6f0np0  172.18.18.0  global  cluster_interconnect,asm
[grid@xffdb1 ~]$ oifcfg delif -global ens9f1/172.17.17.0
[grid@xffdb1 ~]$  oifcfg getif
bond0  192.168.20.0  global  public
ens9f0  172.16.16.0  global  cluster_interconnect,asm
ens6f0np0  172.18.18.0  global  cluster_interconnect,asm
[grid@xffdb1 ~]$ oifcfg delif -global ens9f1/172.17.17.0
[grid@xffdb1 ~]$  oifcfg getif
bond0  192.168.20.0  global  public
ens9f0  172.16.16.0  global  cluster_interconnect,asm
ens6f0np0  172.18.18.0  global  cluster_interconnect,asm

4. 依次重启集群三个节点(ASMNET2LSNR_ASM监听需要人工kill),集群网络替换完成(因为asm listener已经有一个,另外一个私网不准备给他们加上asm listener),如果要增加可以进行如下操作

# srvctl add asmnetwork -netnum 2 -subnet 172.18.18.0
% srvctl add listener -asmlistener -l ASMNET1LSNR_ASM -subnet 172.18.18.0

监听报TNS-12541 TNS-12560 TNS-00511错误

Posted on 2024 年 09 月 05 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：监听报TNS-12541 TNS-12560 TNS-00511错误

有客户运行在win平台上的oracle 11.2.0.1,监听无法正常工作,使用status查看,卡很长时间然后出现如下TNS-12541 TNS-12560 TNS-00511 64-bit Windows Error: 2: No such file or directory错误

LSNRCTL> status
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   64-bit Windows Error: 2: No such file or directory
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=LOCALHOST)(PORT=1521)))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   64-bit Windows Error: 61: Unknown error

根据以往经验,监听服务本身启动成功,运行status hang很久,有可能和监听日志有关系,通过查看确认监听日志达到4G,关闭服务,重命名监听日志,再次启动监听,状态正常
listener