kcratr_nab_less_than_odr和system坏块故障处理

Posted on 2026 年 08 月 25 日 by 惜分飞

标题：kcratr_nab_less_than_odr和system坏块故障处理

学校客户由于机房断电,导致数据库异常.现场工程师进行了现场备份,然后尝试强制拉库,结果没有成功,大概报错和操作过程如下:
断电之后启动数据库,数据库报ORA-600 kcratr_nab_less_than_odr故障解决错误,这个是一种非常常见的操作,一般是由于写丢失导致,以前有过很多类似恢复经历:
ORA-600 kcratr_nab_less_than_odr故障解决
 差点被误操作的ORA-600 kcratr_nab_less_than_odr故障
 ORA-600 kcratr_nab_less_than_odr和ORA-600 2662故障处理
 ORA-600 kcratr_nab_less_than_odr和ORA-600 4193故障处理

Tue Aug 18 23:59:19 2026
ALTER DATABASE OPEN
This instance was first to open
Beginning crash recovery of 2 threads
 parallel recovery started with 19 processes
Started redo scan
Completed redo scan
 read 21634 KB redo, 800 data blocks need recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_275950.trc  (incident=703037):
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6916], [3], [4], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_703037/orcl2_ora_275950_i703037.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Abort recovery for domain 0
Aborting crash recovery due to error 600
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_275950.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6916], [3], [4], [], [], [], []
Abort recovery for domain 0
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_275950.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6916], [3], [4], [], [], [], []
ORA-600 signalled during: ALTER DATABASE OPEN...

现场恢复人员上来之后,直接尝试做强制resetlogs操作,数据库报ORA-600 ORA-600 krsi_al_hdr_update.15错误,主要是由于redo写丢失导致无法resetlogs成功,具体参考：Alter Database Open Resetlogs returns error ORA-00600: [krsi_al_hdr_update.15], (Doc ID 2026541.1) Oracle断电故障处理

Wed Aug 19 00:28:47 2026
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 29721342127
Archived Log entry 38935 added for thread 1 sequence 6916 ID 0xceea62af dest 1:
ARCH: All Archive destinations made inactive due to error 742
ARCH: Closing local archive destination LOG_ARCHIVE_DEST_1: '+ARCHDG/2_19951_1224781365.arc' (error 742)(orcl2)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_287355.trc  (incident=727038):
ORA-00600: internal error code, arguments: [krsi_al_hdr_update.15], [4294967295], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_727038/orcl2_ora_287355_i727038.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Master archival failure: 600
Archive all online redo logfiles failed:600
ORA-600 signalled during: alter database open resetlogs...

通过 ALTER DATABASE RECOVER database using backup controlfile until cancel之后,继续尝试强制打开库，报ORA-600 2662错误.

Wed Aug 19 00:40:25 2026
Checker run found 32 new persistent data failures
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 29721342127
Archived Log entry 38936 added for thread 1 sequence 6915 ID 0xceea62af dest 1:
Archived Log entry 38937 added for thread 1 sequence 6916 ID 0xceea62af dest 1:
Archived Log entry 38938 added for thread 2 sequence 19951 ID 0xceea62af dest 1:
Archived Log entry 38939 added for thread 2 sequence 19950 ID 0xceea62af dest 1:
Clearing online redo logfile 1 +DATADG/orcl/onlinelog/group_1.319.1224781365
Clearing online log 1 of thread 1 sequence number 6915
Wed Aug 19 00:40:34 2026
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 +DATADG/orcl/onlinelog/group_2.320.1224781367
Clearing online log 2 of thread 1 sequence number 6916
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 +DATADG/orcl/onlinelog/group_3.323.1224781463
Clearing online log 3 of thread 2 sequence number 19951
Clearing online redo logfile 3 complete
Clearing online redo logfile 4 +DATADG/orcl/onlinelog/group_4.324.1224781465
Clearing online log 4 of thread 2 sequence number 19950
Clearing online redo logfile 4 complete
Resetting resetlogs activation ID 3471467183 (0xceea62af)
Online log +DATADG/orcl/onlinelog/group_1.319.1224781365: Thread 1 Group 1 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_1.4574.1224781367: Thread 1 Group 1 was previously cleared
Online log +DATADG/orcl/onlinelog/group_2.320.1224781367: Thread 1 Group 2 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_2.10729.1224781369: Thread 1 Group 2 was previously cleared
Online log +DATADG/orcl/onlinelog/group_3.323.1224781463: Thread 2 Group 3 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_3.12866.1224781463: Thread 2 Group 3 was previously cleared
Online log +DATADG/orcl/onlinelog/group_4.324.1224781465: Thread 2 Group 4 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_4.7527.1224781465: Thread 2 Group 4 was previously cleared
Wed Aug 19 00:40:43 2026
Setting recovery target incarnation to 3
Wed Aug 19 00:40:43 2026
Assigning activation ID 3488382552 (0xcfec7e58)
Thread 2 opened at log sequence 1
  Current log# 3 seq# 1 mem# 0: +DATADG/orcl/onlinelog/group_3.323.1224781463
  Current log# 3 seq# 1 mem# 1: +ARCHDG/orcl/onlinelog/group_3.12866.1224781463
Successful open of redo thread 2
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Aug 19 00:40:43 2026
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_291405.trc  (incident=739007):
ORA-00600: internal error code, arguments: [2662], [6], [3951555809], [6], [3951556555], [12583040], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_739007/orcl2_ora_291405_i739007.trc
Wed Aug 19 00:40:45 2026
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_291405.trc:
ORA-00600: internal error code, arguments: [2662], [6], [3951555809], [6], [3951556555], [12583040], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_291405.trc:
ORA-00600: internal error code, arguments: [2662], [6], [3951555809], [6], [3951556555], [12583040], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 291405): terminating the instance due to error 600
Instance terminated by USER, pid = 291405
ORA-1092 signalled during: alter database open resetlogs...

到这一步,现场停止了继续尝试,我接手故障处理.先dbv检测坏块,发现system有两个坏块

[oracle@db3 ~]$ dbv userid=sys/oracle file=/datapool/cold_backup_20260818/SYSTEM.314.1224781265

DBVERIFY: Release 11.2.0.4.0 - Production on Sat Aug 22 10:39:26 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /datapool/cold_backup_20260818/SYSTEM.314.1224781265
Page 94587 is marked corrupt
Corrupt block relative dba: 0x0041717b (file 1, block 94587)
Bad header found during dbv: 
Data in bad block:
 type: 11 format: 2 rdba: 0x00400001
 last change scn: 0x0000.00000000 seq: 0x1 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x00000b01
 check value in block header: 0xd49
 computed block checksum: 0x0

Page 95021 is marked corrupt
Corrupt block relative dba: 0x0041732d (file 1, block 95021)
Bad header found during dbv: 
Data in bad block:
 type: 11 format: 2 rdba: 0x00400001
 last change scn: 0x0000.00000000 seq: 0x1 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x00000b01
 check value in block header: 0xd49
 computed block checksum: 0x0



DBVERIFY - Verification complete

Total Pages Examined         : 157440
Total Pages Processed (Data) : 72079
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 21854
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 49484
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 14021
Total Pages Marked Corrupt   : 2
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 0 (0.0)

OBET> dbv file 1

===============================================
DBV (Data Block Verification)
Block Size: 8192 bytes
Endian:     little-endian (x86)
Target File: #1(only)
===============================================

Verifying file #1: /datapool/cold_backup_20260818/SYSTEM.314.1224781265 (157441 blocks) - Started: 2026-08-22 11:14:48
File #32: rfile=1 (0x00000001)  header_block_num=157440 (0x00026700)  filesize_status:OK
  Progress: 100000 / 157441 blocks checked...
  File #1completed: 0 all zero, 0 soft corrupted, 2 tailchk error, 0 checksum error, 0 rdba error

DBV completed at: 2026-08-22 11:14:59

===============================================
DBV Summary:
Total blocks checked: 157439
Total all zero blocks found: 0
Total all rdba error blocks found: 0
Total all tailchk error blocks found: 2
Total all soft corrupted blocks found: 0
Total all checksum error blocks found: 0
Total bad blocks found: 2
Execution time: 11.00 seconds
Throughput: 111.82 MB/s
===============================================

Detailed report saved to: dbv_file_32_20260822111448.log
Files processed: 1

OBET> list corrupt
file 1(/datapool/cold_backup_20260818/SYSTEM.314.1224781265) total bad blocks: 2
block#   bad block type
94587   tailchk
95021   tailchk

使用obet修复坏块
Oracle Block Editor Tool使用手册

OBET> set file 32
filename set to: /datapool/cold_backup_20260818/SYSTEM.314.1224781265 (file#1)

OBET> set block 94587
block set to: 94587

OBET> set mode edit
mode set to: edit

OBET> repair block
Warning: Missing value for 'block', using global setting: 94587

Repairing block 94587 in file /datapool/cold_backup_20260818/SYSTEM.314.1224781265...

Repair analysis for block 94587:
1. seq_kcbh check: 0x01 -> OK
2. Tailchk check: 0x0106C0E4 -> needs repair (0xE4C00601)
3. Checksum check: 0x5EC5 -> needs repair (0x7DE6)

Confirm repair operations:
File: /datapool/cold_backup_20260818/SYSTEM.314.1224781265
Block: 94587
Operations needed: fix tailchk, fix checksum
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x0106C0E4 OK
3. Checksum: 0x7DE6 OK

Block 94587 repair completed successfully.

OBET> set block 95021
block set to: 95021

OBET> repair block 
Warning: Missing value for 'block', using global setting: 95021

Repairing block 95021 in file /datapool/cold_backup_20260818/SYSTEM.314.1224781265...

Repair analysis for block 95021:
1. seq_kcbh check: 0x01 -> OK
2. Tailchk check: 0xF8FA0601 -> needs repair (0xE06C0601)
3. Checksum check: 0x81BA -> OK

Confirm repair operations:
File: /datapool/cold_backup_20260818/SYSTEM.314.1224781265
Block: 95021
Operations needed: fix tailchk
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x01066CE0 OK
3. Checksum: 0x81BA OK

Block 95021 repair completed successfully.

dbv检查确认坏块修复成功

[oracle@db3 ~]$ dbv file=/datapool/cold_backup_20260818/SYSTEM.314.1224781265

DBVERIFY: Release 11.2.0.4.0 - Production on Sat Aug 22 11:17:30 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /datapool/cold_backup_20260818/SYSTEM.314.1224781265


DBVERIFY - Verification complete

Total Pages Examined         : 157440
Total Pages Processed (Data) : 72080
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 21855
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 49484
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 14021
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3951556890 (6.3951556890)

后面的恢复比较简单,使用客户恢复之前的备份,直接重建ctl,然后打开库成功,并且做rman校验没有异常,直接把恢复之后的库备份还原到asm里面,完成本次恢复任务

通过obet 恢复system坏块,打开数据库

Posted on 2026 年 08 月 22 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：通过obet 恢复system坏块,打开数据库

客户由于误操作直接在虚拟化平台点击电源键,强制关闭了正在运行的数据库服务器虚机,导致数据库无法正常启动,检查发现ntfs文件系统损坏

通过obet工具检测数据文件坏块(obet dbv功能完整说明),发现核心的system文件上面有一些坏块

File #1: D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF (225281 blocks) - Started: 2026-08-21 16:40:23
File #1: rfile=1 (0x00000001)  header_block_num=225280 (0x00037000)  filesize_status:OK
file 1, block 212238: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 213507: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 213515: tailchk error (expected 0x01065E92, got 0x0106276F), bad block
file 1, block 213547: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 213555: tailchk error (expected 0x01065E92, got 0x0106276F), bad block
file 1, block 221387: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 221403: tailchk error (expected 0x01065E92, got 0x0106B19C), bad block
file 1, block 221419: tailchk error (expected 0x01065E92, got 0x0106C7E3), bad block
file 1, block 222117: tailchk error (expected 0x01065E92, got 0x0106276F), bad block
file 1, block 222133: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 222141: tailchk error (expected 0x01065E92, got 0x0106276F), bad block
file 1, block 222173: tailchk error (expected 0x0106C7E3, got 0x01065E92), bad block
  File #1 completed: 0 all zero, 0 soft corrupted, 12 tailchk error, 0 checksum error, 0 rdba error

这个12个坏块的数量和dbv检测结果一致


DBVERIFY: Release 19.0.0.0.0 - Production on 星期五 8月 21 17:37:02 2026

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.


DBVERIFY - 开始验证: FILE = D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF
页 212238 流入 - 很可能是介质损坏
Corrupt block relative dba: 0x00433d0e (file 1, block 212238)
Fractured block found during dbv: 
Data in bad block:
 type: 6 format: 2 rdba: 0x00433d0e
 last change scn: 0x0000.0002.d71a6f27 seq: 0x1 flg: 0x06
 spare3: 0x0
 consistency value in tail: 0x925e0601
 check value in block header: 0x93bb
 computed block checksum: 0xfd79

…………

页 222173 流入 - 很可能是介质损坏
Corrupt block relative dba: 0x004363dd (file 1, block 222173)
Fractured block found during dbv: 
Data in bad block:
 type: 6 format: 2 rdba: 0x004363dd
 last change scn: 0x0000.0002.d708e3c7 seq: 0x1 flg: 0x06
 spare3: 0x0
 consistency value in tail: 0x925e0601
 check value in block header: 0xe089
 computed block checksum: 0x7199


DBVERIFY - 验证完成

检查的页总数: 225280
处理的页总数 (数据): 125451
失败的页总数 (数据): 0
处理的页总数 (索引): 29984
失败的页总数 (索引): 0
处理的页总数 (其他): 48214
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 21619
标记为损坏的总页数: 12
流入的页总数: 12
加密的总页数        : 0
最高块 SCN            : 12199365950 (2.3609431358)

使用obet的reair block功能进行修复(obet repair block使用说明)

OBET> set file 1
filename set to: D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF (file#1)

OBET> set block 212238
block set to: 212238

OBET> d

File: D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF
Block: 212238                Offsets:     0 to    31
--------------------------------------------------------------------------------
67A1C000 06A20000 0E3D4300 276F1AD7 02000106 BB930000 02000000 D0020000 E46E1AD7

<32 bytes read>

OBET> tailchk
Check tailchk for File D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF, Block 212238:
current = 0x925E0601, required = 0x6F270601

OBET> backup
Backing up file #1, block 212238
Successfully backed up block 212238 from D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF 
   to backup_blk\SYSTEM01.DBF.212238_20260821173825.blk

OBET> set mode edit
mode set to: edit

OBET> repair
Usage: repair [subcommand]
  repair block [file N] [X]  - Repair tailchk/checksum (optional: file N, block X)
  repair blkscn [file N] [X] - Repair block SCN (optional: file N, block X)

OBET> repair block
Warning: Missing value for 'block', using global setting: 212238

Repairing block 212238 in file D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF...

Repair analysis for block 212238:
1. seq_kcbh check: 0x01 -> OK
2. Tailchk check: 0x925E0601 -> needs repair (0x6F270601)
3. Checksum check: 0xBB93 -> OK

Confirm repair operations:
File: D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF
Block: 212238
Operations needed: fix tailchk
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x6F270601 OK
3. Checksum: 0xBB93 OK

Block 212238 repair completed successfully.

所有坏块依次进行修复,然后再次使用dbv检测


DBVERIFY: Release 19.0.0.0.0 - Production on 星期五 8月 21 17:42:02 2026

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.


DBVERIFY - 开始验证: FILE = D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF
Block Checking: DBA = 4407811, Block Type = KTB-managed data block
**** actual rows locked by itl 2  = 0 != # in trans. header = 1
**** actual rows locked by itl 3  = 0 != # in trans. header = 1
---- end index block validation
页 213507 失败, 校验代码为 6401
Block Checking: DBA = 4407819, Block Type = KTB-managed data block
**** row 120: key out of order
**** actual rows locked by itl 2  = 0 != # in trans. header = 1
**** actual rows marked deleted = 1 != kdxlende = 0
---- end index block validation
页 213515 失败, 校验代码为 6401

DBVERIFY - 验证完成

检查的页总数: 225280
处理的页总数 (数据): 125451
失败的页总数 (数据): 0
处理的页总数 (索引): 29996
失败的页总数 (索引): 2
处理的页总数 (其他): 48214
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 21619
标记为损坏的总页数: 0
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 12199365950 (2.3609431358)

有两个block有少量逻辑错误,可以通过数据库级别设置进行跳过,基本上实现了这个12个坏块的自动修复.后续就是数据库的打开过程

SQL> startup mount pfile='d:/pfile.txt'
ORACLE 例程已经启动。

Total System Global Area 1.2885E+10 bytes
Fixed Size                 16120656 bytes
Variable Size            7751073792 bytes
Database Buffers         5100273664 bytes
Redo Buffers               17432576 bytes
数据库装载完毕。
SQL> recover database;
ORA-00283: 恢复会话因错误而取消
ORA-00742: 日志读取在线程 1 序列 19828 块 11511 中检测到写入丢失情况
ORA-00312: 联机日志 2 线程 1: 'D:\APP\ADMINISTRATOR\ORADATA\NHIS\REDO02.LOG'

SQL> RECOVER DATABASE UNTIL TIME '2026-08-21:13:28:56' USING BACKUP CONTROLFILE;
ORA-00279: change 12198905983 generated at 08/21/2026 12:43:18 needed for
thread 1
ORA-00289: suggestion :
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000019827_1199133733.0001
ORA-00280: change 12198905983 for thread 1 is in sequence #19827


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ADMINISTRATOR\ORADATA\NHIS\REDO01.LOG
ORA-00279: change 12199301614 generated at 08/21/2026 13:27:06 needed for
thread 1
ORA-00289: suggestion :
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000019828_1199133733.0001
ORA-00280: change 12199301614 for thread 1 is in sequence #19828
ORA-00278: log file 'D:\APP\ADMINISTRATOR\ORADATA\NHIS\REDO01.LOG' no longer
needed for this recovery


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ADMINISTRATOR\ORADATA\NHIS\REDO02.LOG
ORA-00756: 鎭㈠鎿嶄綔妫€娴嬪埌鏁版嵁鍧楀啓鍏ヤ涪澶?ORA-10567:
Redo is inconsistent with data block (file# 4, block# 31663, file offset is 259383296 bytes)
ORA-10564: tablespace UNDOTBS1
ORA-01110: 鏁版嵁鏂囦欢 4: 'D:\APP\ADMINISTRATOR\ORADATA\NHIS\UNDOTBS01.DBF'
ORA-10560: block type 'KTU UNDO BLOCK'

ORA-01112: media recovery not started

SQL> alter database open resetlogs;

Database altered.

使用expdp导出数据,完成本次恢复任务

分享一例运行在aix上的sap系统数据库恢复过程

Posted on 2026 年 08 月 14 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：分享一例运行在aix上的sap系统数据库恢复过程

情况描述
客户sap系统运行在aix系统,ibm v7000存储上,数据存放在三个1T的lun组成的vg的多个lv中.异常断电之后,但是给数据库使用的三个lun丢失了2个,从而使得所有vg/lv异常,通过硬件恢复出来异常的2个lun和剩余的1个lun一起,把所有的数据文件恢复出来.但是由于某种原因,出现部分block被覆盖(其中还包括两个文件头损坏).

坏块检测
对于恢复的所有文件,为了快速做一遍坏块检查,直接在恢复的win机器上使用过obet做了一次dbv检查obet实现对数据文件坏块检测功能(使用obet的dbv检查有几个好处:1>可以在win上面检测aix的数据文件;2>可以检测数据文件头损坏的数据文件其他block;3>检测速度比原生dbv快[每个文件内部加了并行检测]),检测结果如下

--其中两个文件头损坏
File #62: E:\sr3_54\sr3.data54 (1280000 blocks) - Started: 2026-07-28 23:57:53
File #62: rfile=0 (0x00000000)  header_block_num=0 (0x00000000)  filesize_status:NO
file 62, block 0: block all zero
file 62, block 1: block all zero

File #84: E:\sr3_76\sr3.data76 (4185601 blocks) - Started: 2026-07-29 00:55:32
File #84: rfile=638937491 (0x26156993)  header_block_num=77152556 (0x0499412C)  filesize_status:NO
file 84, block 1: rdba error (expected 1, got 69760), bad block
file 84, block 2: rdba error (expected 2, got 589836), bad block

--坏块汇总
DBV completed at: 2026-07-29 00:58:10
===============================================
DBV Summary:
Total blocks checked: 196287237
Total all zero blocks found: 337417
Total all rdba error blocks found: 1062468
Total all tailchk error blocks found: 1
Total all soft corrupted blocks found: 0
Total all checksum error blocks found: 0
Total bad blocks found: 1399886
Execution time: 10507.00 seconds
===============================================

这个统计下来好的block在99.3%左右,证明硬件层面的会效果已经非常好.

碎片工具进一步恢复
恢复公司文件系统层面恢复有数据块遗漏的可能,通过碎片工具(OraScan(Oracle 碎片扫描工具) 使用说明)进一步扫描

通过确认62号文件还有少量block可以进一步恢复（也就是说碎片扫描到的62号文件的block多于硬件公司恢复出来的62号文件里面好的block数量）,通过obet的merge功能进行填补

基于上述操作,对于lun里面的数据文件实现了最大效果恢复.

数据库恢复操作
1. 上次恢复文件到aix,offline掉异常文件头数据文件打开数据库

sapprd2:oraprd 8> sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Aug 3 18:41:28 2026

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select status from v$instance;

STATUS
------------------------------------
STARTED

SQL> alter database mount;
alter database mount
*
ERROR at line 1:
ORA-00214: control file '/oracle/PRD/origlogA/cntrl/cntrlPRD.dbf' version
9742943 inconsistent with file '/oracle/PRD/sapdata1/cntrl/cntrlPRD.dbf'
version 9742931

解决ctl不一致问题之后继续mount库恢复

sapprd2:oraprd 11> sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Aug 3 18:42:53 2026

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> alter database mount;

Database altered.

SQL> alter database datafile 62,84 offline;

Database altered.

SQL> recover database ;
Media recovery complete.
SQL> alter database open;

Database altered.

2.对于两个损坏的数据文件头进行修复
由于现在aix环境的客户比较少,obet没有在aix环境下进行编译,因此直接使用bbed工具进行文件头修复（列举了主要操作过程）

BBED> copy file 83 block 1 to file 84 block 1
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:    0 to   31           Dba:0x15000001
------------------------------------------------------------------------
 0ba20000 14c00001 00000000 00000104 22250000 00000000 0b200000 72bc991d

 <32 bytes per line>

BBED> set offset 368
        OFFSET          368

BBED> d
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:  368 to  399           Dba:0x15000001
------------------------------------------------------------------------
 00000053 00000000 00000000 495dc097 00000000 00000000 00000000 00000000

 <32 bytes per line>

BBED> m /x 00000054
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:  368 to  399           Dba:0x15000001
------------------------------------------------------------------------
 00000054 00000000 00000000 495dc097 00000000 00000000 00000000 00000000

 <32 bytes per line>

BBED> set offset 52
        OFFSET          52

BBED> d
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:   52 to   83           Dba:0x15000001
------------------------------------------------------------------------
 00530003 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 <32 bytes per line>

BBED> m /x 0054
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:   52 to   83           Dba:0x15000001
------------------------------------------------------------------------
 00540003 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 <32 bytes per line>

BBED> set offset 4
        OFFSET          4

BBED> d
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:    4 to   35           Dba:0x15000001
------------------------------------------------------------------------
 14c00001 00000000 00000104 22250000 00000000 0b200000 72bc991d 50524400

 <32 bytes per line>

BBED> m /x 15000001
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:    4 to   35           Dba:0x15000001
------------------------------------------------------------------------
 15000001 00000000 00000104 22250000 00000000 0b200000 72bc991d 50524400

 <32 bytes per line>


BBED> m /x E1B0
Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:  100 to  611           Dba:0x15000001
------------------------------------------------------------------------
 e1b0853f 00040000 4083fcfe 33606b63 02502452 0000ca28 49680fb3 93b9f6b9

 <32 bytes per line>

BBED> set offset +2
        OFFSET          102

BBED> m /x 8542
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:  102 to  613           Dba:0x15000001
------------------------------------------------------------------------
 85420004 00004083 fcfe3360 6b630250 24520000 ca284968 0fb393b9 f6b90005

 <32 bytes per line>


BBED> sum
Check value for File 84, Block 1:
current = 0x2225, required = 0x23e5

BBED> sum apply
Check value for File 84, Block 1:
current = 0x23e5, required = 0x23e5

BBED> verify
DBVERIFY - Verification starting
FILE = /oracle/PRD/sapdata3/sr3_76/sr3.data76
BLOCK = 1


DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 0
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED

修改完成之后,还出现过几个错误

SQL> alter database open ;
alter database open
*
ERROR at line 1:
ORA-01122: database file 62 failed verification check
ORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54'
ORA-01200: actual file size of 1279999 is smaller than correct size of 1280000

ORA-01200是由于数据文件比文件头记录信息小一个block,通过补上这个block解决

Read of datafile '/oracle/PRD/sapdata2/sr3_54/sr3.data54' (fno 62) header failed with ORA-01202
Rereading datafile 62 header failed with ORA-01202
Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_11272278.trc:
ORA-01122: database file 62 failed verification check
ORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54'
ORA-01202: wrong incarnation of this file - wrong creation time

ORA-01202是由于create time没有修改正确导致,重新修改解决

Rereading datafile 84 header failed with ORA-01203
Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_17105252.trc:
ORA-01122: database file 84 failed verification check
ORA-01110: data file 84: '/oracle/PRD/sapdata3/sr3_76/sr3.data76'
ORA-01203: wrong incarnation of this file - wrong creation SCN

ORA-01203是由于create scn没有修改正确导致,重新修改解决

Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_17105278.trc:
ORA-01122: database file 62 failed verification check
ORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54'
ORA-01207: file is more recent than control file - old control file
ORA-1122 signalled during: alter database open .

ORA-01207是由于数据文件的ckp信息比控制文件的新,重建ctl解决,解决这些问题之后,顺利打开数据库

SQL> startup mount;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 3.7548E+10 bytes
Fixed Size                  2254136 bytes
Variable Size            1.9193E+10 bytes
Database Buffers         1.8254E+10 bytes
Redo Buffers               98996224 bytes
Database mounted.
SQL> alter database open;

Database altered.

SQL> select status,count(1) from v$datafile_header group by status;

STATUS    COUNT(1)
------- ----------
ONLINE          84

然后跳过坏块,导出数据,对于无法导出的异常表进行特殊处理,完成本次恢复工作,最终恢复结果总结

几乎动用了所有手段的Oracle故障恢复

Posted on 2026 年 08 月 01 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：几乎动用了所有手段的Oracle故障恢复

最近处理了一个比较复杂的恢复,使用了十八般武艺基本上完成了恢复,最大限度恢复客户的数据
故障背景
1. 客户两个1.8T盘做raid 1,操作系统是linux（分区swap,/ [lv方式]）运行oracle数据库(跑的是一家医院的his系统),前些年一直这样运行
2. 今年4月份发现空间不足,维护人员发现主机上有一个sdb盘(裸盘,而且没有被使用),直接加入到/分区的lv中
3. 前几天系统突然故障,数据库无法连接,他们排查原因的时候发现备份一体机的备份相关的配置和以前设置不一样了,以为故障和这个相关,然后他们就让备份一体机厂商把备份相关配置还原恢复,结果这个操作导致两个问题:1)备份一体机中所有的关于这个主机上的备份文件全部被删除;2)以前在这个主机上看到的sdb的lun也不见了(后来确认是备份一体机映射出来的,现在被回收回去了)
4. 由于调整一体机相关设置之后,依旧无法解决问题,他们开始怀疑是raid磁盘问题(可能这个时候也发现了raid上面有告警),然后多次尝试对两个盘进行插拔操作,最后导致raid也异常
5. 客户的备份机制:先备份在本地的/分区(也就是说备份文件很可能部分写入到了sdb盘),然后再传输到备份一体机中.

恢复思路
1. 镜像损坏的raid磁盘
2. 通过损坏的磁盘中恢复出来可以恢复的数据文件
3. 通过碎片扫描磁盘,把由于元数据丢失导致部分在镜像磁盘中的数据块恢复出来
4. 通过提取镜像磁盘中的备份,并尽可能的恢复出来需要的业务数据块(基于客户这边的情况,备份是最近写的,大概率是在sdb盘上面占比大,所以直接可以还原的概率很小)
5. 通过工具把3和4中提取出来的block,整合到2中的数据文件中
6. 然后打开数据导出数据(设置跳过坏块)

具体恢复操作
1. 恢复损坏磁盘中的数据文件
接手这个故障之后,让客户想对raid 1中的其中一块磁盘进行镜像,通过工具分析确认是vg少了一块盘

通过镜像文件恢复数据文件(由于lv包含了已经丢失的sdb盘),所以数据肯定不完整,但是理论上今年4月份之前的数据应该相对完整,先通过工具对镜像进行解析,恢复镜像中的数据文件

在拷贝过程中发现部分文件有报错,通过文件系统元数据查看报错数据文件分布元数据信息,确认部分数据段存放在了sdb盘上面

通过对恢复出来的所有数据文件使用obet的dbv功能检测(obet实现对数据文件坏块检测功能),并确认有6个数据文件异常

PS I:\2026年7月31日> Get-Content .\dbv_20260731085210.log | Select-String "filesize_status:NO" -Context 2,0

  File #1: I:\20260731-jn\xxxx\system01.dbf (4167680 blocks) - Started: 2026-07-31 08:52:10
> File #1: rfile=1 (0x00000001)  header_block_num=4175360 (0x003FB600)  filesize_status:NO

  File #2: I:\20260731-jn\xxxx\sysaux01.dbf (2470912 blocks) - Started: 2026-07-31 08:52:44
> File #2: rfile=2 (0x00000002)  header_block_num=2536960 (0x0026B600)  filesize_status:NO

  File #24: I:\20260731-jn\xxxx\XXX406.DBF (2331648 blocks) - Started: 2026-07-31 08:55:58
> File #24: rfile=24 (0x00000018)  header_block_num=2522880 (0x00267F00)  filesize_status:NO

  File #25: I:\20260731-jn\xxxx\XXX407.DBF (2292736 blocks) - Started: 2026-07-31 08:56:17
> File #25: rfile=25 (0x00000019)  header_block_num=2484480 (0x0025E900)  filesize_status:NO

  File #26: I:\20260731-jn\xxxx\XXXX408.DBF (2291712 blocks) - Started: 2026-07-31 08:56:36
> File #26: rfile=26 (0x0000001A)  header_block_num=2483200 (0x0025E400)  filesize_status:NO

  File #27: I:\20260731-jn\xxxx\sysaux02.dbf (832512 blocks) - Started: 2026-07-31 08:56:55
> File #27: rfile=27 (0x0000001B)  header_block_num=920832 (0x000E0D00)  filesize_status:NO

这里主要是file 24,25,26涉及业务数据,对于system 大概率是aud$记录,sysaux可以直接忽略.

2. 对镜像盘进行碎片扫描,尽可能多的找数据块
对于异常文件缺少的数据块,先尝试对进行磁盘进行碎片扫描最大限度恢复可以由于文件系统元数据写入到sdb磁盘导致的丢失数据,使用OraScan(Oracle 碎片扫描工具)进行碎片扫描,恢复出来数据文件

3. 先从操作系统中恢复出来备份文件,然后使用工具在备份中强制提取file为24,25,26数据文件

4. 使用obet最近开发的merge功能对文件进行整合,类似操作

5. 最终恢复之后效果,所有业务数据块总的只有15149个损坏block(无法找出来)

C:\Users\XFF>grep "File #"  C:\Users\XFF\dbv_20260801233813.log
File #24: I:\20260731-jn\xxxx\XXX406.DBF (2522881 blocks) - Started: 2026-08-01 23:38:13
File #24: rfile=24 (0x00000018)  header_block_num=2522880 (0x00267F00)  filesize_status:OK
  File #24 completed: 0 all zero, 0 soft corrupted, 0 tailchk error, 0 checksum error, 4288 rdba error
File #25: I:\20260731-jn\xxxx\XXX407.DBF (2484481 blocks) - Started: 2026-08-01 23:39:14
File #25: rfile=25 (0x00000019)  header_block_num=2484480 (0x0025E900)  filesize_status:OK
  File #25 completed: 0 all zero, 1 soft corrupted, 0 tailchk error, 0 checksum error, 10604 rdba error
File #26: I:\20260731-jn\xxxx\XXX408.DBF (2483201 blocks) - Started: 2026-08-01 23:40:13
File #26: rfile=26 (0x0000001A)  header_block_num=2483200 (0x0025E400)  filesize_status:OK
  File #26 completed: 0 all zero, 0 soft corrupted, 0 tailchk error, 0 checksum error, 256 rdba error

6. 尝试打开数据库过程中遇到还有文件大小不对的问题

SQL> /
CREATE CONTROLFILE REUSE DATABASE "XXX" NORESETLOGS FORCE LOGGING NOARCHIVELOG
*
第 1 行出现错误:
ORA-01503: CREATE CONTROLFILE ??
ORA-01200: 853727 ????????? 920832 ??????
ORA-01110: ???? 27: 'I:\20260731-jn\xxxx\sysaux02.dbf'

使用obet的extend功能进行处理

OBET> extend file 1
File #1: I:\20260731-jn\xxx\sysaux02.dbf
  BlockSize:       8192 bytes
  Header Blocks:   920832 (offset 44)
  Expected Size:   7543463936 bytes ((920832+1)*8192)
  Actual Size:     6993739776 bytes
  Status: File is smaller than header record.
  Extending file by 549724160 bytes with zero-filled padding...
  Confirm? (Y/yes to proceed): Y
  Done. File extended to 7543463936 bytes.

然后打开数据库过程报各种错误处理

SQL> recover database;
ORA-10562: Error occurred while applying redo to data block (file# 24, block# 2086190)
ORA-10564: tablespace TS_HIS4
ORA-01110: ???????? 24: 'I:\20260731-JN\XXXX\XXX406.DBF'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 93878
ORA-00600: ????????????, ????: [6122], [0], [81963], [0], [], [], [], [], [], [], [], []

SQL> recover database until cancel;
ORA-00279: 更改 3573202216 (在  生成) 对于线程 1 是必需的


指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: 警告: RECOVER 成功但 OPEN RESETLOGS 将出现如下错误
ORA-01194: 文件 1 需要更多的恢复来保持一致性
ORA-01110: 数据文件 1: 'I:\20260731-JN\XXXX\SYSTEM01.DBF'


ORA-01112: 未启动介质恢复


SQL> alter database open resetlogs ;
alter database open resetlogs
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [0], [3573202226], [0],
[3573222676], [12583040], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [2662], [0], [3573202225], [0],
[3573222676], [12583040], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [0], [3573202223], [0],
[3573222676], [12583040], [], [], [], [], [], []
进程 ID: 16884
会话 ID: 1 序列号: 3

通过patch_scn(Patch SCN一键解决ORA-600 2662故障)解决这个问题之后,数据库顺利打开

SQL> alter database open ;
alter database open
*
第 1 行出现错误:
ORA-01113: ?? 1 ??????
ORA-01110: ???? 1: 'I:\20260731-JN\XXXX\SYSTEM01.DBF'

SQL> recover database;
完成介质恢复。
SQL> alter database open;

数据库已更改。

然后按照客户需求导出数据,完成本次恢复任务.

Oracle 19c 202607补丁（RUs+OJVM）-19.32

Posted on 2026 年 07 月 27 日 by 惜分飞

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle 19c 202607补丁（RUs+OJVM）-19.32

Release	Database Update	GI Update	Windows Bundle Patch
JUL2026 (19.32.0.0.0)	39472050	39467003	39418910
APR2026 (19.31.0.0.0)	39034528	39036936	38818049
JAN2026 (19.30.0.0.0)	38632161	38629535	38597735
OCT2025 (19.29.0.0.0)	38291812	38298204	38111211
JUL2025 (19.28.0.0.0)	37960098	37957391	37962957
APR2025 (19.27.0.0.0)	37642901	37641958	37532350
JAN2025 (19.26.0.0.0)	37260974	37257886	37486199
OCT2024 (19.25.0.0.0)	36912597	36916690	36878821
JUL2024 (19.24.0.0.0)	36582781	36582629	36521936
APR2024 (19.23.0.0.0)	36233263	36233126	36219938
JAN2024 (19.22.0.0.0)	35943157	35940989	35962832
OCT2023 (19.21.0.0.0)	35643107	35642822	35681552
JUL2023 (19.20.0.0.0)	35320081	35319490	35348034
APR2023 (19.19.0.0.0)	35042068	35037840	35046439
JAN2023 (19.18.0.0.0)	34765931	34762026	34750795
Oct2022 (19.17.0.0.0)	34419443	34416665	34468114
JUL2022 (19.16.0.0.0)	34133642	34130714	34110685
APR2022 (19.15.0.0.0)	33806152	33803476	33829175
JAN2022 (19.14.0.0.0)	33515361	33509923	33575656
OCT2021(19.13.0.0.0)	33192793	33182768	33155330
JUL2021 (19.12.0.0.0)	32904851	32895426	32832237
APR2021 (19.11.0.0.0)	32545013	32545008	32409154
JAN2021 (19.10.0.0.0)	32218454	32226239	32062765
OCT2020 (19.9.0.0.0)	31771877	31750108	31719903
JUL2020 (19.8.0.0.0)	31281355	31305339	31247621
APR2020 (19.7.0.0.0)	30869156	30899722	30901317
JAN2020 (19.6.0.0.0)	30557433	30501910	30445947
OCT2019 (19.5.0.0.0)	30125133	30116789	30151705
JUL2019 (19.4.0.0.0)	29834717	29708769	-
APR2019 (19.3.0.0.0)	29517242	29517302	-

Release	OJVM Update	OJVM + DB Update	OJVM + GI Update
JUL2026 (19.32.0.0.260721)	39222882	39618649	39618711
APR2026 (19.31.0.0.260421)	38906621	39062931	39062956
JAN2026 (19.30.0.0.260120)	38523609	38658587	38658588
OCT2025 (19.29.0.0.251021)	38194382	38273545	38273558
JUL2025 (19.28.0.0.250715)	37847857	37952354	37952382
APR2025 (19.27.0.0.250415)	37499406	37591483	37591516
JAN2025 (19.26.0.0.250121)	37102264	37262172	37262208
OCT2024 (19.25.0.0.241015)	36878697	36866623	36866740
JUL2024 (19.24.0.0.240716)	36414915	36522340	36522439
APR2024 (19.23.0.0.240416)	36199232	36209492	36209493
JAN2024 (19.22.0.0.240116)	35926646	36031426	36031453
OCT2023 (19.21.0.0.231017)	35648110	35742413	35742441
JUL2023 (19.20.0.0.230718)	35354406	35370174	35370167
APR2023 (19.19.0.0.230418)	35050341	35058163	35058172
JAN2023 (19.18.0.0.230117)	34786990	34773489	34773504
OCT2022 (19.17.0.0.221018)	34411846	34449114	34449117
JUL2022 (19.16.0.0.220719)	34086870	34160831	34160854
APR2022 (19.15.0.0.220419)	33808367	33859194	33859214
JAN2022 (19.14.0.0.220118)	33561310	33567270	33567274
OCT2021 (19.13.0.0.211019)	33192694	33248420	33248471
JUL2021 (19.12.0.0.210720)	32876380	32900021	32900083
APR2021 (19.11.0.0.210420)	32399816	32578972	32578973
JAN2021 (19.10.0.0.210119)	32067171	32126828	32126842
OCT2020 (19.9.0.0.201020)	31668882	31720396	31720429
JUL2020 (19.8.0.0.200714)	31219897	31326362	31326369
APR2020 (19.7.0.0.200414)	30805684	30783543	30783556
JAN2020 (19.6.0.0.200114)	30484981	30463595	30463609
OCT2019 (19.5.0.0.191015)	30128191	30133124	30133178
JUL2019 (19.4.0.0.190716)	29774421	29699079	29699097
APR2019 (19.3.0.0.190416)	29548437	29621253	29621299