ORA-04020导致adg异常

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-04020导致adg异常

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

今日早上有客户反馈adg停止同步了,通过检查alert日志发现

Tue Dec 24 18:17:41 2019
Media Recovery Waiting for thread 1 sequence 56655 (in transit)
Recovery of Online Redo Log: Thread 1 Group 11 Seq 56655 Reading mem 0
  Mem# 0: Y:\ORACLE\ORADATA\ORACLE11\STD_REDO11.LOG
Archived Log entry 56248 added for thread 1 sequence 56654 ID 0x5b6bcf9b dest 1:
Tue Dec 24 18:18:11 2019
Errors in file C:\APP\ADMINISTRATOR\diag\rdbms\oracle11dg\oracle11\trace\oracle11_lgwr_3252.trc:
ORA-04020: deadlock detected while trying to lock object SYS.orcl
LGWR (ospid: 3252): terminating the instance due to error 4020
Tue Dec 24 18:18:11 2019
System state dump requested by (instance=1, osid=3252 (LGWR)), summary=[abnormal instance termination].
System State dumped to trace file C:\APP\ADMINISTRATOR\diag\rdbms\oracle11dg\oracle11\trace\oracle11_diag_3236_20191224181811.trc
Dumping diagnostic data in directory=[cdmp_20191224181811], requested by (instance=1, osid=3252 (LGWR)), summary=[abnormal instance termination].
Instance terminated by LGWR, pid = 3252

由于lgwr进程遭遇ORA-04020,从而使得lgwr进程异常,进而整个数据库crash.

分析trace文件

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
Windows NT Version V6.1 Service Pack 1 
CPU                 : 8 - type 8664, 8 Physical Cores
Process Affinity    : 0x0x0000000000000000
Memory (Avail/Total): Ph:8395M/32733M, Ph+PgF:41002M/65464M 
Instance name: oracle11
Redo thread mounted by this instance: 1
Oracle process number: 10
Windows thread id: 3252, image: ORACLE.EXE (LGWR)


*** 2019-12-24 18:18:11.072
*** SESSION ID:(384.1) 2019-12-24 18:18:11.072
*** CLIENT ID:() 2019-12-24 18:18:11.072
*** SERVICE NAME:(SYS$BACKGROUND) 2019-12-24 18:18:11.072
*** MODULE NAME:() 2019-12-24 18:18:11.072
*** ACTION NAME:() 2019-12-24 18:18:11.072
 
A deadlock among DDL and parse locks is detected.
This deadlock is usually due to user errors in
the design of an application or from issuing a set
of concurrent statements which can cause a deadlock.
This should not be reported to Oracle Support.
The following information may aid in finding
the errors which cause the deadlock:
ORA-04020: deadlock detected while trying to lock object SYS.orcl
--------------------------------------------------------
  object   waiting  waiting       blocking blocking
  handle   session     lock mode   session     lock mode
--------  -------- -------- ----  -------- -------- ----
0000000676C20F08  000000066D22BE10 00000006738AB970    X  000000066D22BE10 00000006738A04B0    S
0000000677DF2E80  00000006792E2880 0000000673B13AE8    X  000000066D22BE10 00000006738A19B8    S
--------------------------------------------------------
---------- DUMP OF WAITING AND BLOCKING LOCKS ----------
--------------------------------------------------------
------------- WAITING LOCK -------------
----------------------------------------
SO: 0x00000006738AB970, type: 78, owner: 0x000000065D440498, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0x000000066CDE7AD0, name=LIBRARY OBJECT LOCK, file=kgl.h LINE:8751, pg=0

LibraryObjectLock:  Address=00000006738AB970 Handle=0000000676C20F08 RequestMode=X CanBeBrokenCount=2 Incarnation=3 ExecutionCount=0  
………………
SO: 0x00000006738A19B8, type: 78, owner: 0x000000065A38D6C0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0x000000066CDE7AD0, name=LIBRARY OBJECT LOCK, file=kgl.h LINE:8751, pg=0

LibraryObjectLock:  Address=00000006738A19B8 Handle=0000000677DF2E80 Mode=S CanBeBrokenCount=1 Incarnation=1 ExecutionCount=0   
  
  User=000000066D22BE10 Session=000000066D22BE10 ReferenceCount=1 Flags=CNB/[0001] SavepointNum=1b749 
  LibraryHandle:  Address=0000000677DF2E80 Hash=675351da LockMode=S PinMode=0 LoadLockMode=0 Status=0 
  ObjectName:  Name=SYS.orcl   
    FullHashValue=285b654fe3f440652c403c98675351da Namespace=DBINSTANCE(74) Type=CURSOR(00) Identifier=1 OwnerIdn=0 
  Statistics:  InvalidationCount=0 ExecutionCount=0 LoadCount=0 ActiveLocks=1 TotalLockCount=74719 TotalPinCount=0 
  Counters:  BrokenCount=1 RevocablePointer=1 KeepDependency=0 Version=0 BucketInUse=2 HandleInUse=2 HandleReferenceCount=0 
  Concurrency:  DependencyMutex=0000000677DF2F30(0, 0, 0, 0) Mutex=0000000677DF2FC0(0, 149471, 1, 0) 
  Flags=RON/PIN/KEP/BSO/[00810003] 
  WaitersLists:  
    Lock=0000000677DF2F10[0000000673B13B58,000000067382E2F0] 
    Pin=0000000677DF2EF0[0000000677DF2EF0,0000000677DF2EF0] 
    LoadLock=0000000677DF2F68[0000000677DF2F68,0000000677DF2F68] 
  Timestamp:  
  HandleReference:  Address=0000000677DF3030 Handle=0000000000000000 Flags=[00] ---------------------------------
This lock request was aborted.
error 4020 detected in background process
ORA-04020: deadlock detected while trying to lock object SYS.orcl
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+585<-kjzdssdmp()+329<-kjzduptcctx()+288<-kjzdicrshnfy()+99<-ksuitm()+1525<-ksbrdp()+4578<-opirip()
+853<-opidrv()+906<-sou2o()+98<-opimai_real()+280<-opimai()+191<-BackgroundThreadStart()+646<-0000000076CF59CD<-0000000076E2A561 
----- End of Abridged Call Stack Trace -----

*** 2019-12-24 18:18:11.165
LGWR (ospid: 3252): terminating the instance due to error 4020

*** 2019-12-24 18:18:17.483
ksuitm: waiting up to [5] seconds before killing DIAG(3236)

 

日志显示由于lgwr进程等待LIBRARY OBJECT LOCK超时,从而引起异常,根据经验此类问题一般是由于bug导致,查询mos发现匹配bug信息Bug 18515268 ORA-4020 in ADG Standby Database causing instance crash by LGWR
20191225112622


可以根据需要打上相关Patch 18515268: ACTIVE DATAGUARD STANDBY CRASHES DUE TO AN ORA-4020 ENCOUNTERED BY LGWR

helpservis@horsefucker.org加密恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:helpservis@horsefucker.org加密恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

最近有客户环境中大量服务器被感染类似如下病毒,邮箱为:helpservis@horsefucker.org,每个机器不一样的id,使用该id为文件后缀名
20191222182600
16进制方式打开文件头发现破坏如下
20191222182709


进一步分析发现文件中依旧有大量block没有被破坏
20191222182727

经过分析我们基本上可以较为完美的恢复该病毒加密数据库(sql server,oracle等)
20191222184051

这类的数据库加密,我们可以直接通过解密处理,恢复之后基本立即可用.如果需要可以联系我们从底层进行恢复:Phone:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com

xfs删除数据文件恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:xfs删除数据文件恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

在linux7的环境(xfs文件系统格式)中,由于误操作执行了rm -rf /操作,导致系统大部分文件被删除(oracle数据库相关文件).我们对其现场进行分析,确认相关数据在磁盘底层依旧存在
20191219042943


20191219043052


运气不错通过底层恢复,实现数据库完美open
20191219221348

fuser命令缺失导致19c打补丁失败

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:fuser命令缺失导致19c打补丁失败

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

Oracle 19.3数据库打最新补丁报错

[oracle@www.xifenfei.com 30125133]$ opatch apply ./
Oracle Interim Patch Installer version 12.2.0.1.18
Copyright (c) 2019, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/19.0/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/19.0/db_1/oraInst.loc
OPatch version    : 12.2.0.1.18
OUI version       : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatch/opatch2019-12-16_08-00-53AM_1.log

分析日志

Verifying environment and performing prerequisite checks...
Prerequisite check "CheckSystemCommandAvailable" failed.
The details are:
Missing command :fuser
UtilSession failed: 
Prerequisite check "CheckSystemCommandAvailable" failed.
Log file location: /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatch/opatch2019-12-16_08-00-53AM_1.log

OPatch failed with error code 73


[Dec 16, 2019 8:00:57 AM] [INFO]    Following patches can be applied:  30125133
[Dec 16, 2019 8:00:57 AM] [INFO]    Following patches are not required:
[Dec 16, 2019 8:00:57 AM] [INFO]    Following patches are auto rollbackable:
[Dec 16, 2019 8:00:57 AM] [INFO]    Finished checking prereq checkConflictAgainstOHWithDetail
[Dec 16, 2019 8:00:58 AM] [INFO]    Running prerequisite checks...
[Dec 16, 2019 8:00:58 AM] [INFO]    Space Needed : 3052.647MB
[Dec 16, 2019 8:00:58 AM] [INFO]    Missing command :fuser
[Dec 16, 2019 8:00:58 AM] [INFO]    Prerequisite check "CheckSystemCommandAvailable" failed.
                                    The details are:
                                    Missing command :fuser
[Dec 16, 2019 8:00:58 AM] [SEVERE]  OUI-67073:UtilSession failed:
                                    Prerequisite check "CheckSystemCommandAvailable" failed.

基本上可以确定是由于无fuser命令导致CheckSystemCommandAvailable失败.处理办法为安装上相关程序

[root@www.xifenfei.com tmp]# yum search fuser
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
==================================== Matched: fuser ===========================
psmisc.x86_64 : Utilities for managing processes on your system
[root@www.xifenfei.com tmp]# 


[root@www.xifenfei.com tmp]# mount /dev/cdrom /media/ -o loop
[root@www.xifenfei.com tmp]# yum install -y psmisc
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
rhel-yum                       | 4.3 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package psmisc.x86_64 0:22.20-15.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================
 Package                    Arch              Version                    Repository     Size
=============================================================================================
Installing:
 psmisc                     x86_64            22.20-15.el7               rhel-yum      141 k

Transaction Summary
=============================================================================================
Install  1 Package

Total download size: 141 k
Installed size: 475 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : psmisc-22.20-15.el7.x86_64          1/1 
  Verifying  : psmisc-22.20-15.el7.x86_64          1/1 

Installed:
  psmisc.x86_64 0:22.20-15.el7     

Complete!

重新打补丁

[oracle@www.xifenfei.com 30125133]$ opatch apply ./
Oracle Interim Patch Installer version 12.2.0.1.18
Copyright (c) 2019, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/19.0/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/19.0/db_1/oraInst.loc
OPatch version    : 12.2.0.1.18
OUI version       : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatch/opatch2019-12-16_08-08-44AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   30125133  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/19.0/db_1')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '30125133' to OH '/u01/app/oracle/product/19.0/db_1'
ApplySession: Optional component(s) [ oracle.network.gsm, 19.0.0.0.0 ],[ oracle.rdbms.ic, 19.0.0.0.0 ],
[ oracle.network.cman, 19.0.0.0.0 ],[ oracle.net.cman, 19.0.0.0.0 ],[ oracle.options.olap.awm, 19.0.0.0.0],
[ oracle.oraolap.mgmt, 19.0.0.0.0 ],[ oracle.assistants.usm, 19.0.0.0.0 ],[ oracle.assistants.asm, 19.0.0.0.0],
[ oracle.tfa, 19.0.0.0.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 19.0.0.0.0...

Patching component oracle.rdbms.rsf, 19.0.0.0.0...

Patching component oracle.assistants.acf, 19.0.0.0.0...

Patching component oracle.assistants.deconfig, 19.0.0.0.0...

Patching component oracle.assistants.server, 19.0.0.0.0...

Patching component oracle.buildtools.rsf, 19.0.0.0.0...

Patching component oracle.ctx, 19.0.0.0.0...

Patching component oracle.ldap.rsf, 19.0.0.0.0...

Patching component oracle.network.rsf, 19.0.0.0.0...

Patching component oracle.rdbms.dbscripts, 19.0.0.0.0...

Patching component oracle.sdo, 19.0.0.0.0...

Patching component oracle.sqlplus, 19.0.0.0.0...

Patching component oracle.ldap.rsf.ic, 19.0.0.0.0...

Patching component oracle.rdbms.rman, 19.0.0.0.0...

Patching component oracle.ctx.atg, 19.0.0.0.0...

Patching component oracle.rdbms.oci, 19.0.0.0.0...

Patching component oracle.rdbms.util, 19.0.0.0.0...

Patching component oracle.xdk, 19.0.0.0.0...

Patching component oracle.ovm, 19.0.0.0.0...

Patching component oracle.network.listener, 19.0.0.0.0...

Patching component oracle.rdbms.install.plugins, 19.0.0.0.0...

Patching component oracle.dbjava.jdbc, 19.0.0.0.0...

Patching component oracle.dbdev, 19.0.0.0.0...

Patching component oracle.rdbms.deconfig, 19.0.0.0.0...

Patching component oracle.nlsrtl.rsf, 19.0.0.0.0...

Patching component oracle.oraolap.dbscripts, 19.0.0.0.0...

Patching component oracle.install.deinstalltool, 19.0.0.0.0...

Patching component oracle.dbjava.ic, 19.0.0.0.0...

Patching component oracle.sdo.locator, 19.0.0.0.0...

Patching component oracle.rdbms.scheduler, 19.0.0.0.0...

Patching component oracle.rdbms.dv, 19.0.0.0.0...

Patching component oracle.ons, 19.0.0.0.0...

Patching component oracle.ldap.security.osdt, 19.0.0.0.0...

Patching component oracle.ctx.rsf, 19.0.0.0.0...

Patching component oracle.duma, 19.0.0.0.0...

Patching component oracle.ldap.owm, 19.0.0.0.0...

Patching component oracle.oracore.rsf, 19.0.0.0.0...

Patching component oracle.rdbms.install.seeddb, 19.0.0.0.0...

Patching component oracle.odbc, 19.0.0.0.0...

Patching component oracle.sdo.locator.jrf, 19.0.0.0.0...

Patching component oracle.network.client, 19.0.0.0.0...

Patching component oracle.sqlplus.ic, 19.0.0.0.0...

Patching component oracle.dbjava.ucp, 19.0.0.0.0...

Patching component oracle.xdk.rsf, 19.0.0.0.0...

Patching component oracle.marvel, 19.0.0.0.0...

Patching component oracle.xdk.parser.java, 19.0.0.0.0...

Patching component oracle.rdbms.rsf.ic, 19.0.0.0.0...

Patching component oracle.nlsrtl.rsf.core, 19.0.0.0.0...

Patching component oracle.precomp.common, 19.0.0.0.0...

Patching component oracle.precomp.lang, 19.0.0.0.0...
Patch 30125133 successfully applied.
Sub-set patch [29517242] has become inactive due to the application of a super-set patch [30125133].
Please refer to Doc ID 2161861.1 for any possible further required actions.
Log file location: /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatch/opatch2019-12-16_08-08-44AM_1.log

OPatch succeeded.


[oracle@www.xifenfei.com 30128191]$ opatch apply ./
Oracle Interim Patch Installer version 12.2.0.1.18
Copyright (c) 2019, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/19.0/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/19.0/db_1/oraInst.loc
OPatch version    : 12.2.0.1.18
OUI version       : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatch/opatch2019-12-16_08-12-44AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   30128191  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/19.0/db_1')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '30128191' to OH '/u01/app/oracle/product/19.0/db_1'

Patching component oracle.javavm.server, 19.0.0.0.0...

Patching component oracle.javavm.server.core, 19.0.0.0.0...

Patching component oracle.rdbms.dbscripts, 19.0.0.0.0...

Patching component oracle.rdbms, 19.0.0.0.0...
Patch 30128191 successfully applied.
Log file location: /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatch/opatch2019-12-16_08-12-44AM_1.log

OPatch succeeded.

确认补丁安装成功

[oracle@www.xifenfei.com 30128191]$ opatch lspatches
30128191;OJVM RELEASE UPDATE: 19.5.0.0.191015 (30128191)
30125133;Database Release Update : 19.5.0.0.191015 (30125133)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

OPatch succeeded.

mos上也有类似文章可参考:Prerequisite check “CheckSystemCommandAvailable” failed (Doc ID 1581604.1)

.megac加密恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:.megac加密恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

oracle数据库被加密为.megac.megac后缀名,加密情况如下:
20191211223347


20191211222459

加密说明文件类似:
20191211223423

恢复数据截图:
20191211224008

这类的数据库加密,我们可以直接通过解密处理,open数据库,导出数据,恢复之后立即可用.如果需要可以联系我们从底层进行恢复:Phone:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com