oracle 11.2进入扩展服务—普通权限mos无法下载最新psu

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:oracle 11.2进入扩展服务—普通权限mos无法下载最新psu

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

oracle 4月份补丁集发布了,11.2已经进入了扩展服务,普通的mos帐号(无11.2扩展服务权限帐号)已经无法下载相关psu补丁
29141056-1


有下载权限帐号显示如下
29141056

11.2版本数据库后续如果要下载最新psu需要购买扩展服务的用户才可以了,有些客户可以考虑升级了

CLSRSC-400: A system reboot is required to continue installing.

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:CLSRSC-400: A system reboot is required to continue installing.

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

在以前的文章中写过关于12.2.0.1在linux 7.3中安装可能遇到的oracle rac 12.2 执行root.sh报CLSRSC-400问题,最近mos上面提供的相应的patch(25078431)以及处理方法,供参考
12.2.0.1 Cluster Installation Failure – ACFS-9459 With RH 7.3 (文档 ID 2273119.1)文章提供方法
25078431-1


ALERT: root.sh Fails With “CLSRSC-400″ While Installing GI 12.2.0.1 on RHEL or OL with RedHat Compatible Kernel (RHCK) 7.3 (文档 ID 2284463.1)文章提供方法
applyOneOffs

12.1人工修改操作系统时间导致数据库异常

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:12.1人工修改操作系统时间导致数据库异常

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户数据库版本为12.1.0.1 版本RAC,突发发生重启,让协助分析原因
数据库alert日志报ORA-15064错误

Mon Apr 15 15:06:26 2019
WARNING: inbound connection timed out (ORA-3136)
Mon Apr 15 15:41:26 2019
NOTE: ASMB terminating
Mon Apr 15 15:41:26 2019
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_asmb_61426.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID: 
Session ID: 1892 Serial number: 29
Mon Apr 15 15:41:26 2019
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_asmb_61426.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID: 
Session ID: 1892 Serial number: 29
Mon Apr 15 15:41:26 2019
System state dump requested by (instance=1, osid=61426 (ASMB)), summary=[abnormal instance termination].
Mon Apr 15 15:41:26 2019
USER (ospid: 61426): terminating the instance due to error 15064
Mon Apr 15 15:41:26 2019
System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_61287.trc
Mon Apr 15 15:41:27 2019
opiodr aborting process unknown ospid (1171) as a result of ORA-1092
Mon Apr 15 15:41:27 2019
ORA-1092 : opitsk aborting process

这里看,明显asmb异常导致数据库无法正常访问asm从而出现数据库crash的问题.

分析asm日志

Mon Apr 15 15:41:26 2019
WARNING: client [+ASM1:+ASM] not responsive for 2069s; state=0x1. pid 23155
NOTE: umbilicus traces dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_gen0_23050.trc
WARNING: client [orcl1:orcl] not responsive for 2069s; state=0x1. killing pid 61436
NOTE: umbilicus traces dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_gen0_23050.trc
WARNING: fencing client [orcl1:orcl] after 2069 seconds (mbr 2)
WARNING: client [-MGMTDB:_mgmtdb] not responsive for 2070s; state=0x1. killing pid 24026
NOTE: umbilicus traces dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_gen0_23050.trc
WARNING: fencing client [-MGMTDB:_mgmtdb] after 2070 seconds (mbr 1)
Mon Apr 15 15:41:26 2019
NOTE: cleaned up ASM client -MGMTDB:_mgmtdb
NOTE: cleaned up ASM client orcl1:orcl
Mon Apr 15 15:41:43 2019
NOTE: Standard client -MGMTDB:_mgmtdb registered, osid 183707, mbr 0x1 (reg:1371965153)
Mon Apr 15 15:42:16 2019
NOTE: Standard client orcl1:orcl registered, osid 184063, mbr 0x2 (reg:2088418628)
Mon Apr 15 15:44:30 2019
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.

asm日志中和mos中的GEN0 terminating the ASM instance due to error 15082 (文档 ID 2096988.1)描述比较匹配.根据客户反馈,他们使用ntp进行修改了时间,基本上可以确定是由于oracle的Bug 19032250(在12.1.0.2中修复)在ntp修改时间跨度过大触发的相关问题(人工直接修改时间也可能出现类似问题)

对于rac修改时间建议
1. 如果时间慢了,关闭数据库和集群直接把时间向前调整,启动集群和数据库
2. 如果时间快了,关闭数据库和集群等实际时间过关闭集群和库的时间之后,再往回调整时间,启动集群和数据库

tab$被恶意删除sys用户之外记录

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:tab$被恶意删除sys用户之外记录

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库open成功,但是alert日志报大量ORA-00600错误

Sun Apr 14 14:30:46 2019
SMCO started with pid=53, OS id=6761 
Completed: ALTER DATABASE OPEN /* db agent *//* {1:65047:2} */
Sun Apr 14 14:30:49 2019
Starting background process CJQ0
Sun Apr 14 14:30:49 2019
CJQ0 started with pid=54, OS id=6776 
Setting Resource Manager plan SCHEDULER[0x32DF]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Sun Apr 14 14:30:52 2019
Starting background process VKRM
Sun Apr 14 14:30:52 2019
VKRM started with pid=37, OS id=6809 
Sun Apr 14 14:30:54 2019
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_6811.trc  (incident=288633):
ORA-00600: internal error code, arguments: [kkpo_rcinfo_defstg:delseg], [84638], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288633/xifenfei1_j000_6811_i288633.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_6811.trc:
ORA-00600: internal error code, arguments: [kkpo_rcinfo_defstg:delseg], [84638], [], [], [], [], [], [], [], [], [], []
ORA-06512: at "APEX_030200.WWV_FLOW_MAIL", line 695
ORA-06512: at line 1
Sun Apr 14 14:30:57 2019
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_7491.trc  (incident=288658):
ORA-00600: 内部错误代码, 参数: [16659], [kqldtu], [INS], [0], [206196], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288658/xifenfei1_j000_7491_i288658.trc
Sun Apr 14 14:34:10 2019
Dumping diagnostic data in directory=[cdmp_20190414143410], requested by (instance=1, osid=7491 (J000)), summary=[incident=288658].
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_7491.trc:
ORA-00600: 内部错误代码, 参数: [16659], [kqldtu], [INS], [0], [206196], [], [], [], [], [], [], []
ORA-06512: 在 "WEBCSMS.P_YGERROR", line 3
Sun Apr 14 14:39:08 2019
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_8515.trc  (incident=288593):
ORA-00600: 内部错误代码, 参数: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288593/xifenfei1_j000_8515_i288593.trc
ORA-06512: 在 line 1
Sun Apr 14 14:52:14 2019
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j001_11062.trc  (incident=288705):
ORA-00600: internal error code, arguments: [16607], [0x3CFB04C90], [257], [9], [0x000000000], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288705/xifenfei1_j001_11062_i288705.trc
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_16712.trc  (incident=288597):
ORA-00600: 内部错误代码, 参数: [16607], [0x3C7CEA678], [1281], [9], [0x000000000], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288597/xifenfei1_ora_16712_i288597.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

报错比较多,客户还反馈登录数据库之后,发现所有的表都丢失。第一反应可能数据字典损坏了,然后让客户查看备库,现在dg的备库也一样表都丢失了,进一步确认字典可能异常,让客户提供system文件进行本地分析.发现DBMS_SUPPORT_DBMONITOR触发器调用DBMS_SUPPORT_DBMONITORP存储过程,和警告:互联网中有oracle介质被注入恶意程序导致—ORA-600 16703中的名称非常类似,但是有点不一样,以前的恶意脚本中都是被注入并且触发之后,数据库无法正常启动,这次数据库能够正常open成功.分析恶意脚本,确认原因
1
2
3
确实这次的恶意脚本是在2016年8月份被创建在库中,在600天之后重启被触发,而且是删除非sys的tab$中记录.知道了恶意脚本的源头,那恢复就比较容易,直接通过批量bbed程序对tab$反删除可以实现比较完美恢复.原则上这样的故障可以实现数据库完美恢复,原库继续使用.

ORACLE Instance XFF (pid = 18) – Error 600 encountered while recovering transaction

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORACLE Instance XFF (pid = 18) – Error 600 encountered while recovering transaction

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

分享一次由于一个表异常导致数据库报类似:ORACLE Instance XFF (pid = 18) – Error 600 encountered while recovering transaction故障的案例
一个10.2.0.4的数据库,正常运行的库突然出现如下错误

Sun Apr 07 11:07:12 2019
Thread 1 advanced to log sequence 602883 (LGWR switch)
  Current log# 3 seq# 602883 mem# 0: L:\ORADATA\XFF\REDO03.LOG
Sun Apr 07 11:10:38 2019
Thread 1 advanced to log sequence 602884 (LGWR switch)
  Current log# 1 seq# 602884 mem# 0: L:\ORADATA\XFF\REDO01.LOG
Sun Apr 07 11:11:56 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\udump\XFF_ora_22956.trc:
ORA-00600: 内部错误代码, 参数: [ktspgfb-1], [], [], [], [], [], [], []

Sun Apr 07 11:12:46 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\udump\XFF_ora_27408.trc:
ORA-00600: 内部错误代码, 参数: [kcbnew_3], [0], [1], [168354056], [], [], [], []

Sun Apr 07 11:13:57 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\udump\XFF_ora_6632.trc:
ORA-00600: 内部错误代码, 参数: [ktspgfb-1], [], [], [], [], [], [], []

过一段时间报,然后实例直接crash

Tue Apr 09 07:47:35 2019
ORACLE Instance XFF (pid = 18) - Error 600 encountered while recovering transaction (1, 1) on object 113718002.
Tue Apr 09 07:47:35 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_smon_12948.trc:
ORA-00600: internal error code, arguments: [kcbgcur_3], [168454497], [8], [4], [0], [], [], []

Tue Apr 09 07:55:23 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_pmon_22652.trc:
ORA-00474: SMON process terminated with error

Tue Apr 09 07:55:24 2019
PMON: terminating instance due to error 474
Tue Apr 09 07:55:24 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_lgwr_28608.trc:
ORA-00474: SMON process terminated with error

Tue Apr 09 07:55:34 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_psp0_12544.trc:
ORA-00474: SMON process terminated with error

Tue Apr 09 07:55:34 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_j000_5216.trc:
ORA-00474: SMON process terminated with error

Tue Apr 09 07:55:35 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_ckpt_28204.trc:
ORA-00474: SMON process terminated with error

Tue Apr 09 07:55:36 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_mman_9320.trc:
ORA-00474: SMON process terminated with error

Tue Apr 09 07:55:44 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_q002_24384.trc:
ORA-00474: SMON process terminated with error

Tue Apr 09 07:55:53 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_reco_24124.trc:
ORA-00474: SMON process terminated with error

根据以上报错,数据库crash的原因是由于undo异常导致,通过对undo进行重建,解决掉异常undo,但是业务运行之后,一样的问题又重现,最后通过分析确认是对象异常导致

SQL> create table XFF.T_XIFENFEI_xff as select * from XFF.T_XIFENFEI;
create table XFF.T_XIFENFEI_xff as select * from XFF.T_XIFENFEI
                                                           *
ERROR at line 1:
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ], [0], [0], [1], [], [], [], []

屏蔽相关block obj的check之后
SQL> create table XFF.T_XIFENFEI_xff as select * from XFF.T_XIFENFEI;
create table XFF.T_XIFENFEI_xff as select * from XFF.T_XIFENFEI
                                                           *
ERROR at line 1:
ORA-00600: internal error code, arguments: [ktspScanInit-l1], [], [], [], [],[], [], []

比较明显该表对象出现逻辑异常,通过基于rowid的方式对该表数据进行抽取

SQL> create table XFF.T_XIFENFEI_new
  2  as
  3  select * from XFF.T_XIFENFEI where 1=0;

Table created.

SQL> set serveroutput on
SQL> set concat off
SQL> DECLARE
  2   nrows number;
  3   rid rowid;
  4   dobj number;
  5   ROWSPERBLOCK number;
  6  BEGIN
  7   ROWSPERBLOCK:=1000;
  8   nrows:=0;
  9   select data_object_id  into dobj
 10   from dba_objects
 11   where owner = 'XFF'
 12   and object_name = 'T_XIFENFEI'
 13   ;
 14   for i in (select relative_fno, block_id, block_id+blocks-1 totblocks
 15             from dba_extents
 16             where owner = 'XFF'
 17               and segment_name = 'T_XIFENFEI'
 18            order by extent_id)
 19   loop
 20     for br in i.block_id..i.totblocks loop
 21      for j in 1..ROWSPERBLOCK loop
 22      begin
 23        rid := dbms_rowid.ROWID_CREATE(1,dobj,i.relative_fno, br , j-1);
 24        insert into XFF.T_XIFENFEI_NEW
 25        select /*+ ROWID(A) */ *
 26        from XFF.T_XIFENFEI A
 27        where rowid = rid;
 28        if sql%rowcount = 1 then nrows:=nrows+1; end if;
 29        if (mod(nrows,10000)=0) then commit; end if;
 30      exception when others then null;
 31      end;
 32      end loop;
 33    end loop;
 34   end loop;
 35   COMMIT;
 36   dbms_output.put_line('Total rows: '||to_char(nrows));
 37  END;
 38  /
Total rows: 227000

PL/SQL procedure successfully completed.

再次观察数据库恢复正常,也不再crash和报错,恢复完成