联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
今日早上有客户反馈adg停止同步了,通过检查alert日志发现
Tue Dec 24 18:17:41 2019 Media Recovery Waiting for thread 1 sequence 56655 (in transit) Recovery of Online Redo Log: Thread 1 Group 11 Seq 56655 Reading mem 0 Mem# 0: Y:\ORACLE\ORADATA\ORACLE11\STD_REDO11.LOG Archived Log entry 56248 added for thread 1 sequence 56654 ID 0x5b6bcf9b dest 1: Tue Dec 24 18:18:11 2019 Errors in file C:\APP\ADMINISTRATOR\diag\rdbms\oracle11dg\oracle11\trace\oracle11_lgwr_3252.trc: ORA-04020: deadlock detected while trying to lock object SYS.orcl LGWR (ospid: 3252): terminating the instance due to error 4020 Tue Dec 24 18:18:11 2019 System state dump requested by (instance=1, osid=3252 (LGWR)), summary=[abnormal instance termination]. System State dumped to trace file C:\APP\ADMINISTRATOR\diag\rdbms\oracle11dg\oracle11\trace\oracle11_diag_3236_20191224181811.trc Dumping diagnostic data in directory=[cdmp_20191224181811], requested by (instance=1, osid=3252 (LGWR)), summary=[abnormal instance termination]. Instance terminated by LGWR, pid = 3252
由于lgwr进程遭遇ORA-04020,从而使得lgwr进程异常,进而整个数据库crash.
分析trace文件
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
Windows NT Version V6.1 Service Pack 1
CPU : 8 - type 8664, 8 Physical Cores
Process Affinity : 0x0x0000000000000000
Memory (Avail/Total): Ph:8395M/32733M, Ph+PgF:41002M/65464M
Instance name: oracle11
Redo thread mounted by this instance: 1
Oracle process number: 10
Windows thread id: 3252, image: ORACLE.EXE (LGWR)
*** 2019-12-24 18:18:11.072
*** SESSION ID:(384.1) 2019-12-24 18:18:11.072
*** CLIENT ID:() 2019-12-24 18:18:11.072
*** SERVICE NAME:(SYS$BACKGROUND) 2019-12-24 18:18:11.072
*** MODULE NAME:() 2019-12-24 18:18:11.072
*** ACTION NAME:() 2019-12-24 18:18:11.072
A deadlock among DDL and parse locks is detected.
This deadlock is usually due to user errors in
the design of an application or from issuing a set
of concurrent statements which can cause a deadlock.
This should not be reported to Oracle Support.
The following information may aid in finding
the errors which cause the deadlock:
ORA-04020: deadlock detected while trying to lock object SYS.orcl
--------------------------------------------------------
object waiting waiting blocking blocking
handle session lock mode session lock mode
-------- -------- -------- ---- -------- -------- ----
0000000676C20F08 000000066D22BE10 00000006738AB970 X 000000066D22BE10 00000006738A04B0 S
0000000677DF2E80 00000006792E2880 0000000673B13AE8 X 000000066D22BE10 00000006738A19B8 S
--------------------------------------------------------
---------- DUMP OF WAITING AND BLOCKING LOCKS ----------
--------------------------------------------------------
------------- WAITING LOCK -------------
----------------------------------------
SO: 0x00000006738AB970, type: 78, owner: 0x000000065D440498, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x000000066CDE7AD0, name=LIBRARY OBJECT LOCK, file=kgl.h LINE:8751, pg=0
LibraryObjectLock: Address=00000006738AB970 Handle=0000000676C20F08 RequestMode=X CanBeBrokenCount=2 Incarnation=3 ExecutionCount=0
………………
SO: 0x00000006738A19B8, type: 78, owner: 0x000000065A38D6C0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x000000066CDE7AD0, name=LIBRARY OBJECT LOCK, file=kgl.h LINE:8751, pg=0
LibraryObjectLock: Address=00000006738A19B8 Handle=0000000677DF2E80 Mode=S CanBeBrokenCount=1 Incarnation=1 ExecutionCount=0
User=000000066D22BE10 Session=000000066D22BE10 ReferenceCount=1 Flags=CNB/[0001] SavepointNum=1b749
LibraryHandle: Address=0000000677DF2E80 Hash=675351da LockMode=S PinMode=0 LoadLockMode=0 Status=0
ObjectName: Name=SYS.orcl
FullHashValue=285b654fe3f440652c403c98675351da Namespace=DBINSTANCE(74) Type=CURSOR(00) Identifier=1 OwnerIdn=0
Statistics: InvalidationCount=0 ExecutionCount=0 LoadCount=0 ActiveLocks=1 TotalLockCount=74719 TotalPinCount=0
Counters: BrokenCount=1 RevocablePointer=1 KeepDependency=0 Version=0 BucketInUse=2 HandleInUse=2 HandleReferenceCount=0
Concurrency: DependencyMutex=0000000677DF2F30(0, 0, 0, 0) Mutex=0000000677DF2FC0(0, 149471, 1, 0)
Flags=RON/PIN/KEP/BSO/[00810003]
WaitersLists:
Lock=0000000677DF2F10[0000000673B13B58,000000067382E2F0]
Pin=0000000677DF2EF0[0000000677DF2EF0,0000000677DF2EF0]
LoadLock=0000000677DF2F68[0000000677DF2F68,0000000677DF2F68]
Timestamp:
HandleReference: Address=0000000677DF3030 Handle=0000000000000000 Flags=[00] ---------------------------------
This lock request was aborted.
error 4020 detected in background process
ORA-04020: deadlock detected while trying to lock object SYS.orcl
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+585<-kjzdssdmp()+329<-kjzduptcctx()+288<-kjzdicrshnfy()+99<-ksuitm()+1525<-ksbrdp()+4578<-opirip()
+853<-opidrv()+906<-sou2o()+98<-opimai_real()+280<-opimai()+191<-BackgroundThreadStart()+646<-0000000076CF59CD<-0000000076E2A561
----- End of Abridged Call Stack Trace -----
*** 2019-12-24 18:18:11.165
LGWR (ospid: 3252): terminating the instance due to error 4020
*** 2019-12-24 18:18:17.483
ksuitm: waiting up to [5] seconds before killing DIAG(3236)
日志显示由于lgwr进程等待LIBRARY OBJECT LOCK超时,从而引起异常,根据经验此类问题一般是由于bug导致,查询mos发现匹配bug信息Bug 18515268 ORA-4020 in ADG Standby Database causing instance crash by LGWR
可以根据需要打上相关Patch 18515268: ACTIVE DATAGUARD STANDBY CRASHES DUE TO AN ORA-4020 ENCOUNTERED BY LGWR
