断电引起的oracle数据库异常恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:断电引起的oracle数据库异常恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

服务器断电,数据库mount失败

SQL> startup mount pfile='d:/pfile.txt'
ORACLE 例程已经启动。

Total System Global Area 1185853440 bytes
Fixed Size                  2175168 bytes
Variable Size             335548224 bytes
Database Buffers          838860800 bytes
Redo Buffers                9269248 bytes
ORA-00205: ?????????, ??????, ???????

alert日志报错信息

Sun Mar 19 20:18:29 2023
ALTER DATABASE   MOUNT
Errors in file d:\app\xifenfei\diag\rdbms\orcl\orcl\trace\orcl_ckpt_15064.trc  (incident=3697):
ORA-00227: ????????????: (? 1, # ? 1)
ORA-00202: ????: ''D:\BAIDUNETDISKDOWNLOAD\ORCL\CONTROL01.CTL''
Incident details in: d:\app\xifenfei\diag\rdbms\orcl\orcl\incident\incdir_3697\orcl_ckpt_15064_i3697.trc
Sun Mar 19 20:18:30 2023
Errors in file d:\app\xifenfei\diag\rdbms\orcl\orcl\trace\orcl_m000_18084.trc  (incident=3761):
ORA-00227: ????????????: (? 1, # ? 1)
ORA-00202: ????: ''D:\BAIDUNETDISKDOWNLOAD\ORCL\CONTROL01.CTL''
Incident details in: d:\app\xifenfei\diag\rdbms\orcl\orcl\incident\incdir_3761\orcl_m000_18084_i3761.trc
Sun Mar 19 20:18:29 2023
MMNL started with pid=16, OS id=9404 
ORA-00227: ????????????: (? 1, # ? 1)
ORA-00202: ????: ''D:\BAIDUNETDISKDOWNLOAD\ORCL\CONTROL01.CTL''
Checker run found 1 new persistent data failures
Trace dumping is performing id=[cdmp_20230319201831]
ORA-205 signalled during: ALTER DATABASE   MOUNT...

错误比较明显由于控制文件的block损坏导致数据库在mount的时候提示ORA-00205,重试重建ctl

SQL> CREATE CONTROLFILE REUSE DATABASE "orcl" NORESETLOGS  NOARCHIVELOG
  2      MAXLOGFILES 50
  3      MAXLOGMEMBERS 5
  4      MAXDATAFILES 100
  5      MAXINSTANCES 8
  6      MAXLOGHISTORY 226
  7  LOGFILE
  8    GROUP 1 'D:\BaiduNetdiskDownload\orcl/redo01.log'  SIZE 50M,
  9    GROUP 2 'D:\BaiduNetdiskDownload\orcl/redo02.log'  SIZE 50M,
 10    GROUP 3 'D:\BaiduNetdiskDownload\orcl/redo03.log'  SIZE 50M
 11  DATAFILE
 12  'D:\BaiduNetdiskDownload\orcl\EXAMPLE01.DBF',
 13  'D:\BaiduNetdiskDownload\orcl\GHZN.DBF',
 14  'D:\BaiduNetdiskDownload\orcl\GHZN2.DBF',
 15  'D:\BaiduNetdiskDownload\orcl\SYSAUX01.DBF',
 16  'D:\BaiduNetdiskDownload\orcl\SYSTEM01.DBF',
 17  'D:\BaiduNetdiskDownload\orcl\UNDOTBS01.DBF',
 18  'D:\BaiduNetdiskDownload\orcl\USERS01.DBF'
 19  CHARACTER SET ZHS16GBK
 20  ;
CREATE CONTROLFILE REUSE DATABASE "orcl" NORESETLOGS  NOARCHIVELOG
*
ERROR at line 1:
ORA-01503: CREATE CONTROLFILE failed
ORA-01565: error in identifying file
'D:\BaiduNetdiskDownload\orcl\UNDOTBS01.DBF'
ORA-27041: unable to open file
OSD-04001: 逻辑块大小无效 (OS 2613931212)

由于undo文件异常(大小不是block size的整数倍),因此报OSD-04001: 逻辑块大小无效错误.对undo文件及其其他文件进行检查发现数据库文件有不少坏块,而且undo文件的文件头损坏
20230319202417


通过抛弃undo文件并进行一些处理,重建ctl成功,并且recover 数据库成功,顺利open数据库

SQL> CREATE CONTROLFILE REUSE DATABASE "orcl" NORESETLOGS  NOARCHIVELOG
  2      MAXLOGFILES 50
  3      MAXLOGMEMBERS 5
  4      MAXDATAFILES 100
  5      MAXINSTANCES 8
  6      MAXLOGHISTORY 226
  7  LOGFILE
  8    GROUP 1 'D:\BaiduNetdiskDownload\orcl/redo01.log'  SIZE 50M,
  9    GROUP 2 'D:\BaiduNetdiskDownload\orcl/redo02.log'  SIZE 50M,
 10    GROUP 3 'D:\BaiduNetdiskDownload\orcl/redo03.log'  SIZE 50M
 11  DATAFILE
 12  'D:\BaiduNetdiskDownload\orcl\EXAMPLE01.DBF',
 13  'D:\BaiduNetdiskDownload\orcl\GHZN.DBF',
 14  'D:\BaiduNetdiskDownload\orcl\GHZN2.DBF',
 15  'D:\BaiduNetdiskDownload\orcl\SYSAUX01.DBF',
 16  'D:\BaiduNetdiskDownload\orcl\SYSTEM01.DBF',
 17  'D:\BaiduNetdiskDownload\orcl\USERS01.DBF'
 18  CHARACTER SET ZHS16GBK
 19  ;

Control file created.

SQL> recover database;
Media recovery complete.
SQL> shutdown immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> startup mount pfile='d:/pfile.txt'
ORACLE instance started.

Total System Global Area 1185853440 bytes
Fixed Size                  2175168 bytes
Variable Size             335548224 bytes
Database Buffers          838860800 bytes
Redo Buffers                9269248 bytes
Database mounted.
SQL> alter database open;

Database altered.

然后使用逻辑方式导出数据,运气不错业务文件没有任何坏块,system坏块在aud$上,无任何业务数据丢失.

等保修改oracle SYS用户名要求的请注意—ORA-00600 kokasgi1

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:等保修改oracle SYS用户名要求的请注意—ORA-00600 kokasgi1

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

在2019年处理过第一起ORA-00600 kokasgi1 故障到现在,已经过去了近4年,今天又有客户依旧因为修改sys重启库遇到该问题
ora-600-kokasgi1


通过分析确认客户那边在等保的时候要求修改oracle的SYS用户
20230319001510

然后重启数据库,数据库就开始报ORA-600 kokasgi1错误.
再次呼吁:
1. oracle的sys用户名不能修改,这个东西是写在oracle代码里面的,启动的时候会去读取
2.如果已经修改了sys用户名的,请在数据库重启之前一定修改回来

Buffer I/O error on dev故障数据库恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:Buffer I/O error on dev故障数据库恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户对虚拟化环境进行物理机加内存,结果发现数据库服务器无法正常启动,报Buffer I/O error on dev错误
buffer io error


通过对底层虚拟化磁盘进行分析,确定vg为centos,数据所在的磁盘组为root
20230313213019

进一步恢复其中数据
20230313233918

然后顺利打开数据库,数据0丢失

[oracle@xifenfei u01]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Mar 13 23:27:42 2023
Version 19.13.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.

Connected to an idle instance.
 
SQL> startup
ORACLE instance started.

Total System Global Area 1.0100E+10 bytes
Fixed Size                 13625520 bytes
Variable Size            6241124352 bytes
Database Buffers         3825205248 bytes
Redo Buffers               19927040 bytes
Database mounted.
Database opened.
SQL> select status,count(1) from v$datafile group by status;

STATUS    COUNT(1)
------- ----------
SYSTEM           1
ONLINE          21

win强制修改盘符导致oracle异常恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:win强制修改盘符导致oracle异常恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户反馈,他们在系统没有关闭数据库的情况下,强制修改了win系统盘符,然后导致数据库异常,启动报错

Sat Feb 25 12:50:40 2023
Recovery of Online Redo Log: Thread 1 Group 2 Seq 10440 Reading mem 0
  Mem# 0 errs 0: G:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO02.LOG
Sat Feb 25 12:50:40 2023
Completed redo application
Sat Feb 25 12:50:40 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_p001_5604.trc:
ORA-00600: 内部错误代码, 参数: [2037], [25801018], [2973409798], [6], [255], [25], [1198764346], [100796692]

Sat Feb 25 12:50:40 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\udump\orcl_ora_7648.trc:
ORA-07445: 出现异常错误: 核心转储 [ACCESS_VIOLATION][_kslwlmod+166][PC:0x469742][ADDR:0x54F8][UNABLE_TO_WRITE][]
ORA-04096: 触发器 '' 的 WHEN 子句过大, 限量为 2K

Sat Feb 25 12:50:40 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\udump\orcl_ora_7252.trc:
ORA-00600: internal error code, arguments: [ksuapc2], [258], [0], [2], [1], [2], [], []

Sat Feb 25 12:50:43 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_p001_5604.trc:
ORA-00081: 地址范围 [0x77240440, 0x77240444) 不可读
ORA-07445: 出现异常错误: 核心转储 [ACCESS_VIOLATION][_ksl_cleanup+723][PC:0x46E373][ADDR:0x1C][UNABLE_TO_READ][]
ORA-00081: 地址范围 [0x77240440, 0x77240444) 不可读
ORA-00600: 内部错误代码, 参数: [2037], [25801018], [2973409798], [6], [255], [25], [1198764346], [100796692]

Sat Feb 25 12:50:45 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_dbw0_6332.trc:
ORA-07445: ??????: ???? [ACCESS_VIOLATION][_kews_idle_wait+378][PC:0x604AE6][ADDR:0xED30C470][UNABLE_TO_WRITE][]

Sat Feb 25 12:50:48 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\udump\orcl_ora_7648.trc:
ORA-00600: 内部错误代码, 参数: [kslwlflux:1], [0xAB805400], [0x549C], [2], [], [], [], []
ORA-00081: 地址范围 [0x74480443, 0x74480447) 不可读
ORA-00600: 内部错误代码, 参数: [kslwlflux:1], [0xAB805400], [0x549C], [2], [], [], [], []
ORA-00081: 地址范围 [0x74480443, 0x74480447) 不可读
ORA-00600: 内部错误代码, 参数: [kslwlflux:1], [0xAB805400], [0x549C], [2], [], [], [], []
ORA-00081: 地址范围 [0x74480443, 0x74480447) 不可读
ORA-00600: 内部错误代码, 参数: [kslwlflux:1], [0xAB805400], [0x549C], [2], [], [], [], []
ORA-00081: 地址范围 [0x74480443, 0x74480447) 不可读
ORA-00600: 内部错误代码, 参数: [kslwlflux:1], [0xAB805400], [0x549C], [2], [], [], [], []
ORA-00081: 地址范围 [0x74480443, 0x74480447) 不可读
ORA-00600: 内部错误代码, 参数: [kslwlflux:1], [0xAB805400], [0x549C], [2], [], [], [], []
ORA-00081: 地址范围 [0x74480443, 0x74480447) 不可读
ORA-00600: 内部错误代码, 参数: [kslwlflux:1], [0xAB805400], [0x549C], [2], [], [], [], []
ORA-00081: 地址范围 [0x74480443, 0x74480447) 不可读
ORA-00600: 内部错误代码, 参

Sat Feb 25 12:51:34 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_d000_8116.trc:
ORA-07445: ??????: ???? [ACCESS_VIOLATION][_kmcgms+121][PC:0x5D6C71][ADDR:0x50][UNABLE_TO_WRITE][]

Sat Feb 25 12:52:04 2023
USER: terminating instance due to error 472
Sat Feb 25 12:52:48 2023
USER: terminating instance due to error 472
Sat Feb 25 12:52:48 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\udump\orcl_ora_6252.trc:
ORA-07445: exception encountered: core dump [ACCESS_VIOLATION][_ksuitm+631][PC:0x410C07][ADDR:0x1][UNABLE_TO_READ][]

Sat Feb 25 12:55:35 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_7656.trc:
ORA-07445: ??????: ???? [ACCESS_VIOLATION][_kews_idle_wait+378][PC:0x604AE6][ADDR:0xE530C470][UNABLE_TO_WRITE][]

通过恢复一些恢复之后,数据库open之后又挂掉

Sat Feb 25 15:05:49 2023
SMON: enabling tx recovery
Sat Feb 25 15:05:49 2023
Database Characterset is ZHS16GBK
Sat Feb 25 15:05:50 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\udump\orcl_ora_5308.trc:
ORA-00600: 内部错误代码, 参数: [4194], [34], [31], [], [], [], [], []

Sat Feb 25 15:05:50 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_7568.trc:
ORA-00600: 内部错误代码, 参数: [kcbgtcr_13], [], [], [], [], [], [], []

Sat Feb 25 15:05:51 2023
Non-fatal internal error happenned while SMON was doing logging scn->time mapping.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
Sat Feb 25 15:05:51 2023
Doing block recovery for file 2 block 2951
Block recovery from logseq 10441, block 78 to scn 109906860017
Sat Feb 25 15:05:51 2023
Recovery of Online Redo Log: Thread 1 Group 3 Seq 10441 Reading mem 0
  Mem# 0 errs 0: G:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG
Block recovery stopped at EOT rba 10441.81.16
Block recovery completed at rba 10441.81.16, scn 25.2532677517
Doing block recovery for file 2 block 113
Block recovery from logseq 10441, block 78 to scn 109906859718
Sat Feb 25 15:05:52 2023
Recovery of Online Redo Log: Thread 1 Group 3 Seq 10441 Reading mem 0
  Mem# 0 errs 0: G:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG
Block recovery completed at rba 10441.80.16, scn 25.2532677516
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=30, OS id=6904
Sat Feb 25 15:05:53 2023
db_recovery_file_dest_size of 2048 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Sat Feb 25 15:05:53 2023
Completed: alter database open
Sat Feb 25 15:05:53 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j000_5400.trc:
ORA-00600: 内部错误代码, 参数: [4194], [6], [4], [], [], [], [], []

Sat Feb 25 15:05:54 2023
DEBUG: Replaying xcb 0xac458698, pmd 0x8d7e9b9c for failed op 8
Doing block recovery for file 2 block 1515
No block recovery was needed
Sat Feb 25 15:05:55 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j000_5400.trc:
ORA-00600: 内部错误代码, 参数: [4194], [6], [4], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [4194], [6], [4], [], [], [], [], []

Sat Feb 25 15:05:56 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j000_5400.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4194], [6], [4], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [6], [4], [], [], [], [], []

Sat Feb 25 15:05:57 2023
Doing block recovery for file 2 block 2951
Block recovery from logseq 10441, block 78 to scn 109906860017
Sat Feb 25 15:05:57 2023
Recovery of Online Redo Log: Thread 1 Group 3 Seq 10441 Reading mem 0
  Mem# 0 errs 0: G:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG
Block recovery completed at rba 10441.81.16, scn 25.2532677620
Doing block recovery for file 2 block 113
Block recovery from logseq 10441, block 78 to scn 109906860083
Sat Feb 25 15:05:57 2023
Recovery of Online Redo Log: Thread 1 Group 3 Seq 10441 Reading mem 0
  Mem# 0 errs 0: G:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG
Block recovery completed at rba 10441.138.16, scn 25.2532677684
Sat Feb 25 15:05:57 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j003_1716.trc:
ORA-12012: 自动执行作业 42568 出错
ORA-00607: 当更改数据块时出现内部错误
ORA-00607: 当更改数据块时出现内部错误

Sat Feb 25 15:05:59 2023
Flush retried for xcb 0xac4c5a80, pmd 0x8c0cec74
Doing block recovery for file 2 block 2951
Block recovery from logseq 10441, block 78 to scn 109906860017
Sat Feb 25 15:05:59 2023
Recovery of Online Redo Log: Thread 1 Group 3 Seq 10441 Reading mem 0
  Mem# 0 errs 0: G:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG
Sat Feb 25 15:05:59 2023
DEBUG: Replaying xcb 0xac458698, pmd 0x8d7e9b9c for failed op 8
Doing block recovery for file 2 block 1515
No block recovery was needed
Sat Feb 25 15:05:59 2023
Block recovery completed at rba 10441.81.16, scn 25.2532677620
Sat Feb 25 15:06:00 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j002_6400.trc:
ORA-00600: internal error code, arguments: [4194], [6], [4], [], [], [], [], []

Sat Feb 25 15:06:02 2023
DEBUG: Replaying xcb 0xac458698, pmd 0x8d7e9b9c for failed op 8
Doing block recovery for file 2 block 1515
No block recovery was needed
Sat Feb 25 15:06:02 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_pmon_1076.trc:
ORA-00600: 内部错误代码, 参数: [4194], [6], [4], [], [], [], [], []

Sat Feb 25 15:06:03 2023
PMON: terminating instance due to error 472
Sat Feb 25 15:06:03 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j007_7188.trc:
ORA-00472: PMON 进程因错误而终止

Sat Feb 25 15:06:03 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j006_7624.trc:
ORA-00472: PMON 进程因错误而终止

Sat Feb 25 15:06:03 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j005_5688.trc:
ORA-00472: PMON  process terminated with error

Sat Feb 25 15:06:10 2023
Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_7568.trc:
ORA-00472: PMON 进程因错误而终止

Instance terminated by PMON, pid = 1076

这个ORA-600 4194错误主要是由于undo异常,从而引起pmon异常,报ORA-00472错误.对undo进行处理,数据库稳定open,逻辑导出数据,完成恢复工作,完美帮忙客户恢复数据.

再一例asm disk被误加入vg并且扩容lv恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:再一例asm disk被误加入vg并且扩容lv恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

又一客户把三块asm disk磁盘加入到vg里面(两个节点的root vg中),并且还进行了扩容
sdj-asmdisk


加入之后,还对该lv里面进行了expdp数据导出(导出一半失败了,数据库挂了),进而引起了大量的asm磁盘中数据块被文件系统中的expdp导出的dmp复写
20230217200348

通过对损坏磁盘进行kfed分析大概判断损坏到90GB位置
20230217200625

对于这种asm磁盘损坏比较多的情况,常规kfed无法进行修复,只能采用底层基于block层面的扫描恢复,参考:
asm disk header 彻底损坏恢复
通过底层处理恢复出来数据文件:
datafile

然后通过可以有的2天之前的备份,结合dul工具,恢复出来最近数据,业务层面进行整合,完成本次数据恢复
有过类似的恢复案例:
asm disk被加入vg恢复
又一例asm disk 加入vg故障