ORA-01034 ORA-27101故障分析

联系:手机/微信(+86 13429648788) QQ(107644445)QQ咨询惜分飞

标题:ORA-01034 ORA-27101故障分析

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户的数据库通过监听访问报ORA-01034 ORA-27101错误无法正常使用:

SQL> conn app/xxxxx@192.168.129.145/orcl
ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0

SQL>  conn  app/xxxxx
Connected.

检查数据库状态和tnsping都正常

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE

SQL> !tnsping orcl

TNS Ping Utility for Linux: Version 11.2.0.4.0 - Production on 18-OCT-2020 22:11:49

Copyright (c) 1997, 2013, Oracle.  All rights reserved.

Used parameter files:


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.129.1)
(PORT = 1521))) (CONNECT_DATA = (SERVICE_NAME = orcl)))
OK (0 msec)

检查主机之间ping操作也正常:

[XIFENFEI@DB1 trace]$
[XIFENFEI@DB1 trace]$ ping 192.168.129.1
PING 192.168.129.1 (192.168.129.1) 56(84) bytes of data.
64 bytes from 192.168.129.1: icmp_seq=1 ttl=64 time=0.025 ms
64 bytes from 192.168.129.1: icmp_seq=2 ttl=64 time=0.032 ms
64 bytes from 192.168.129.1: icmp_seq=3 ttl=64 time=0.034 ms
^C
--- 192.168.129.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2318ms
rtt min/avg/max/mdev = 0.025/0.030/0.034/0.006 ms

检查数据库和监听配置:

[XIFENFEI@DB1 trace]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sun Oct 18 22:13:50 2020

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> show parameter name;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
cell_offloadgroup_name               string
db_file_name_convert                 string
db_name                              string      orcl
db_unique_name                       string      orcl
global_names                         boolean     FALSE
instance_name                        string      oracle
lock_name_space                      string
log_file_name_convert                string
processor_group_name                 string
service_names                        string      orcl
SQL>
SQL>
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[XIFENFEI@DB1 trace]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 18-OCT-2020 22:15:16

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                18-OCT-2020 22:05:04
Uptime                    0 days 0 hr. 10 min. 12 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /usr/local/oracle/product/11.2.0/db_1/network/admin/listener.ora
Listener Log File         /usr/local/oracle/diag/tnslsnr/DB1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.129.1)(PORT=1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "orcl" has 1 instance(s).
  Instance "orcl", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
[XIFENFEI@DB1 trace]$ cat /usr/local/oracle/product/11.2.0/db_1/network/admin/listener.ora
# listener.ora Network Configuration File: /u01/app/oracle/product/11.2.0/db_1/network/admin/listener.ora

# Generated by Oracle configuration tools.


SID_LIST_LISTENER =
  (SID_LIST =
   (SID_DESC =
      (SID_NAME = orcl)
      (ORACLE_HOME = /usr/local/oracle/product/11.2.0/db_1)
      (GLOBAL_DBNAME= orcl)
    )
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = /usr/local/oracle/product/11.2.0/db_1)
      (PROGRAM = extproc)
    )
  )

LISTENER =

 (DESCRIPTION_LIST =

   (DESCRIPTION =

    (ADDRESS = (PROTOCOL = IPC)  (KEY = EXTPROC1521))

    (ADDRESS = (PROTOCOL = TCP) (HOST = 192.168.129.1) (PORT = 1521))

   )

 )



ADR_BASE_LISTENER = /usr/local/oracle
[XIFENFEI@DB1 trace]$
[XIFENFEI@DB1 trace]$

这里看到数据库配置和监听配置不匹配,因此需要调整。

[XIFENFEI@DB1 admin]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sun Oct 18 22:21:16 2020

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options


SQL> conn app/xxxxx@192.168.129.145/orcl
Connected.
SQL>
SQL>
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[XIFENFEI@DB1 admin]$

利用VNCR修复CVE-2012-1675漏洞

联系:手机/微信(+86 13429648788) QQ(107644445)QQ咨询惜分飞

标题:利用VNCR修复CVE-2012-1675漏洞

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

随着对安全的重视,TNS Listener远程数据投毒漏洞(CVE-2012-1675)被很多单位要求进行整改,而且级别是高危
1


如果是11.2.0.4之前版本,特别是在rac环境中,如果要整改该问题相对比较麻烦,需要通过配置Oracle wallet来实现,配置比较复杂,而且还要重启实例,影响比较大,具体参考:Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle RAC (Doc ID 1340831.1),对于单机环境直接参考以前文章:Oracle Database Server ‘TNS Listener’远程数据投毒漏洞(CVE-2012-1675)的解决方案.从11.2.0.4开始Oracle引入了Valid Node Checking For Registration (VNCR)新特性,可以通过简单的配置即可完成该漏洞修复
在listener.ora文件中增加(grid/oracle用户)

VALID_NODE_CHECKING_REGISTRATION_LISTENER=ON
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN1=ON
REGISTRATION_INVITED_NODES_LISTENER_SCAN1=(hisdb1,hisdb2)

重启监听

LSNRCTL>set current_listener listener_name
LSNRCTL>stop
LSNRCTL>start

验证是否生效
1.在远程机器尝试远程注册

alter system set remote_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.115.15)(PORT=1521))' scope=memory;

2.观察监听日志
VNCR


类似这样证明已经生效.

补充说明

VALID_NODE_CHECKING_REGISTRATION_listener_name
Values:

OFF/0 - Disable VNCR//禁用VNCR,此功能不会对注册过来的service进行check;

ON/1/LOCAL - The default. Enable VNCR. All local machine IPs can register.
启用VNCR,默认只允许本机的所有IP的服务注册到本监听,可通过REGISTRATION_INVITED_NODES参数添加其他有必要的服务器;

SUBNET/2 - All machines in the subnet are allowed registration.//指定子网内的服务器可以注册过来


REGISTRATION_INVITED_NODES_listener-name 
控制允许链接过来的节点,可以通过IP地址/主机名/网段来指定

Values are valid IPs, valid hosts, a subnet using CIDR notation (for ip4/6), or wildcard (*) for ipv4. 
For example:REGISTRATION_INVITED_NODES_Listener=(net-vm1, 127.98.45.209, 127.42.5.*)

Note 
that when an INVITED list is set, it will automatically include the 
machine's local IP in the list. There is no need to include it.

--11.2.0.4和12c不一样之处
在12.1 RAC数据库上,listener的参数VALID_NODE_CHECKING_REGISTRATION_listener_name默认设置为SUBNET / 2,
即子网中的所有计算机都允许注册.所以12c默认不能解决CVE-2012-1675漏洞

参考文档
Valid Node Checking For Registration (VNCR) (Doc ID 1600630.1)
How to Enable VNCR on RAC Database to Register only Local Instances (Doc ID 1914282.1)

WARNING: Subscription for node down event still pending

监听日志中出现很多”WARNING: Subscription for node down event still pending”警告

--监听日志中内容
……
26-NOV-2012 09:35:48 * ping * 0
WARNING: Subscription for node down event still pending
……
26-NOV-2012 09:45:49 * ping * 0
WARNING: Subscription for node down event still pending
……

--监听日志大小
$ ls -l /oracle/product/10g/network/log/listener.log
-rw-r-----    1 oracle   dba       229526148 Nov 26 14:20 /oracle/product/10g/network/log/listener.log

--总计条数
$ grep "Subscription for node down event still pending" \
> /oracle/product/10g/network/log/listener.log|wc -l
   77306

数据库版本和平台

SQL> select * from v$version;

BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bi
PL/SQL Release 10.2.0.5.0 - Production
CORE    10.2.0.5.0      Production
TNS for IBM/AIX RISC System/6000: Version 10.2.0.5.0 - Productio
NLSRTL Version 10.2.0.5.0 - Production

监听日志配置

$ more /oracle/product/10g/network/admin/listener.ora
SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = /oracle/product/10g)
      (PROGRAM = extproc)
    )
  )

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = xifenfei)(PORT = 1521))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC0))
    )
  )

解决方法

--监听文件中增加
SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name>=OFF

--reload监听
lsnrctl reload

补充说明

SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name>=OFF will prevent the messages 
from being written to the log file and may also prevent the TNS Listener from hanging periodically. 

Setting SUBSCRIBE_FOR_NODE_DOWN_<listener_name> to OFF disables 
a necessary RAC functionality(Oracle Notification Service[ONS],fast application notification[FAN]). 
The above workaround is recommended only for non-RAC environments. 
The issue may be present in all 10g and newer installations.

具体参考:372959.1和340091.1

因IPC导致多个监听不能正常启动

在一台机器上装了9.2.0.4和10.1.0.4数据库,启动监听时候发现有一个启动不了
9i监听配置

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS_LIST =
        (ADDRESS = (PROTOCOL = TCP)(HOST = xifenfei.com)(PORT = 1522))
      )
      (ADDRESS_LIST =
        (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
      )
    )
  )

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = xifenfei)
      (ORACLE_HOME = /u01/oracle/9.2.0/db_1)
      (SID_NAME = xff)
    )
  )

10g监听配置

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = /u02/app/oracle/db10g)
      (PROGRAM = extproc)
    )
  )

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS_LIST =
        (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
      )
      (ADDRESS_LIST =
        (ADDRESS = (PROTOCOL = TCP)(HOST = xifenfei.com)(PORT = 1521))
      )
    )
  )

分别启动监听
启动10g监听

[oraem@xifenfei ~]$ lsnrctl start

LSNRCTL for Linux: Version 10.1.0.4.0 - Production on 12-JUN-2012 15:28:51

Copyright (c) 1991, 2004, Oracle.  All rights reserved.

Starting /u02/app/oracle/db10g/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 10.1.0.4.0 - Production
System parameter file is /u02/app/oracle/db10g/network/admin/listener.ora
Log messages written to /u02/app/oracle/db10g/network/log/listener.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=xifenfei.com)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 10.1.0.4.0 - Production
Start Date                12-JUN-2012 15:28:51
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u02/app/oracle/db10g/network/admin/listener.ora
Listener Log File         /u02/app/oracle/db10g/network/log/listener.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=xifenfei.com)(PORT=1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

启动9i监听

[oracle@xifenfei ~]$ lsnrctl start

LSNRCTL for Linux: Version 9.2.0.4.0 - Production on 12-JUN-2012 15:01:44

Copyright (c) 1991, 2002, Oracle Corporation.  All rights reserved.

Starting /u01/oracle/9.2.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 9.2.0.4.0 - Production
System parameter file is /u01/oracle/9.2.0/db_1/network/admin/listener.ora
Log messages written to /u01/oracle/9.2.0/db_1/network/log/listener.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=xifenfei.com)(PORT=1522)))
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))
TNS-12542: TNS:address already in use
 TNS-12560: TNS:protocol adapter error
  TNS-00512: Address already in use
   Linux Error: 98: Address already in use

Listener failed to start. See the error message(s) above...

这里很奇怪两个监听使用不同的端口,为什么不能提示Address被占用呢,难道9i的已经启动了

查看9i的监听状态

[oracle@xifenfei ~]$ lsnrctl status

LSNRCTL for Linux: Version 9.2.0.4.0 - Production on 12-JUN-2012 15:02:23

Copyright (c) 1991, 2002, Oracle Corporation.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=xifenfei)(PORT=1522)))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 111: Connection refused
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))
TNS-12618: TNS:versions are incompatible

9i监听不能启动原因
这里有重大发现TNS:versions are incompatible.我这台服务器只有一个9i和一个10g的数据库出现版本不兼容,那就是说这个命令可能访问了10g的库中的每个监听信息.继续往上看,发现(DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC))).
When a process is on the same machine as the server, use the IPC protocol for connectivity instead of TCP. Inner Process Communication on the same machine does not have the overhead of packet building and deciphering that TCP has.
通过这段话可以看出IPC(Inner Process Communication)是使用于本机的内部通讯,不用包的封装,可以很大程度上提高程序执行效率.看到上面的IPC提示,我们可以确定该错误是因为10g中已经启用了IPC,然后9i的监听也要来启动这个,从而导致该错误.

解决办法
因为在同一个机器上,只能其中的一个数据库启用IPC.修改9i的监听配置

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS_LIST =
        (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1522))
      )
    #  (ADDRESS_LIST =
    #    (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
    #  )
    )
  )

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = /u01/oracle/9.2.0/db_1)
      (PROGRAM = extproc)
    )
    (SID_DESC =
      (GLOBAL_DBNAME = xifenfei)
      (ORACLE_HOME = /u01/oracle/9.2.0/db_1)
      (SID_NAME = xff)
    )
  )

启动9i监听

[oracle@xifenfei ~]$ lsnrctl

LSNRCTL for Linux: Version 9.2.0.4.0 - Production on 12-JUN-2012 15:12:55

Copyright (c) 1991, 2002, Oracle Corporation.  All rights reserved.

Welcome to LSNRCTL, type "help" for information.

LSNRCTL> start
Starting /u01/oracle/9.2.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 9.2.0.4.0 - Production
System parameter file is /u01/oracle/9.2.0/db_1/network/admin/listener.ora
Log messages written to /u01/oracle/9.2.0/db_1/network/log/listener.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=xifenfei.com)(PORT=1522)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1522)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 9.2.0.4.0 - Production
Start Date                12-JUN-2012 15:12:57
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  OFF
SNMP                      OFF
Listener Parameter File   /u01/oracle/9.2.0/db_1/network/admin/listener.ora
Listener Log File         /u01/oracle/9.2.0/db_1/network/log/listener.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=xifenfei.com)(PORT=1522)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "xifenfei" has 1 instance(s).
  Instance "xff", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

查看两个监听进程

[oracle@xifenfei ~]$ ps -ef|grep tns|grep -v grep
oraem    18099     1  0 13:27 ?        00:00:00 /u02/app/oracle/db10g/bin/tnslsnr LISTENER -inherit
oracle   24312     1  0 15:12 pts/0    00:00:00 /u01/oracle/9.2.0/db_1/bin/tnslsnr LISTENER -inherit

连续两次REMOTE_LISTENER 设置为null导致pmon和listener异常

平台系统版本相关信息

SQL> select * from v$version;

BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bi
PL/SQL Release 10.2.0.4.0 - Production
CORE    10.2.0.4.0      Production
TNS for IBM/AIX RISC System/6000: Version 10.2.0.4.0 - Productio
NLSRTL Version 10.2.0.4.0 - Production

SQL> show parameter cluster;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
cluster_database                     boolean     TRUE
cluster_database_instances           integer     2
cluster_interconnects                string      192.168.16.11

SQL> select to_char(sysdate,'yyyy-mm-dd hh24:mi:ss') "www.orasos.com" from dual;

www.orasos.com
-------------------
2012-05-30 11:07:12

pmon和监听负载
pmon和LISTENER进程负载均比较高

  PID     %CPU ResSize    Char Command    
10617230  72.9  143924 21014  ora_pmon_ahunicom1  
22675560  49.9  142000  1547  oracleahunicom1 (LOCAL=NO)
 5243206  30.6   49728  2579  /oracle10/app/product/db/10.2.0/bin/tnslsnr LISTENER -inherit

监听日志
每秒钟很多类此pmon注册监听信息

27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0
27-FEB-2012 14:12:04 * service_update * ahunicom1 * 0

通过这两点可以确定是因为pmon在不停的动态注册监听导致监听日志,pmon,listener进程异常

查询MOS[ID 982068.1]
问题原因

After altering the value of the parameter REMOTE_LISTENER, 
excessive CPU is seen for the TNS listener process (TNSLSNR) and the listener.log file grows rapidly. 
Alert log confirms the REMOTE_LISTENER parameter in the SPFILE was altered.
Listener.log shows continuous service_update triggered from PMON to the TNS listener, 100's per second. 
REMOTE_LISTENER had been set to null twice

查询alert日志,果真发现:
ALTER SYSTEM SET remote_listener='' SCOPE=BOTH SID='AHUNICOM1';
ALTER SYSTEM SET remote_listener='' SCOPE=BOTH;

解决方案

alter system set remote_listener = 'remote_rac' scope=memory sid = 'AHUNICOM1';
alter system set remote_listener = '' scope=memory sid = 'AHUNICOM1';
--然后重启节点,pmon和监听恢复正常