It Takes a Village – Collaborative Steps to Breaking Botnets: How Level 3 and Cisco Worked Together to Improve the Internet’s Security and Stop SSHPsychos

The information security community’s ability to respond to threats and vulnerability discovery improves with each passing month. The collective reaction from the security community to a new file hash, new technique, or communication method has never been stronger. However, attackers are also keeping up, or even exceeding the security world’s defenses.

One way to balance this problem is to not only focus on identifying the threat, but also to find an effective method of removing it from the internet. Too often problem identification is confused with problem removal, leaving attackers observed, yet still able to pursue their goals.

This is why Level 3 Threat Research Labs and Cisco’s Talos Group worked together to investigate and mitigate the risk posed by an attacker’s internet-wide scanning and DDoS botnet, SSHPsychos.

Investigating Malicious Actors

In late September 2014, the Malware Must Die! blog detailed a new Linux malware and rootkit used for DDoS attacks.  Despite their well-written description, the threat persisted.  More than four months later, FireEye identified an unusually large SSH brute force attack attempting to load the same malware, which was still an extremely effective rootkit and DDoS tool when coupled with SSH brute force login attempts.

Fortunately for the internet community, the Talos Group at Cisco did not stop tracking the campaign. In Q1-2015, Talos’ honeypots saw more authentication attempts from this one attacker (103.41.125.0/23 and 43.255.190.0/23) than all other hosts combined.

(Source: Cisco Talos Group.  RED/PINK/ORANGE – Attacker.  GREEN – Rest of Internet)

In late March, Level 3 Threat Research Labs began discussions with the Cisco Talos Group to work together on mitigating this threat to the internet. Level 3’s network data confirmed the massive scale that this one attacker attained when compared with overall internet traffic for SSH. At times, this single attacker accounted for more than 35% of total internet SSH traffic.

(Source: Level 3 Threat Research Labs)

Clearly the security community’s awareness of the overall event was not enough to discourage the attacker and something more substantive had to be done for any improvement to occur.

Our goal, when confirming an internet risk, is to remove it as broadly as possible; however, before removing anything from the internet, it is important to fully understand the impact that may have to more benign hosts. To do this, we must understand more details of the attacker’s tools and infrastructure.

The Talos Group’s honeypot data allowed them to identify what actions were taken after a successful brute force root SSH login occurred. In this example, the chain of events led to the download of a file from a web server running on 23.234.60.140 (resolved from ftp.rxxiaoao.com) and 23.234.19.202. The first host serves files within /install named 8000.rar through 8008.rar, and the second host a06 through a11 within the /i directory.

Upon downloading the files, Level 3 Threat Research Labs confirmed the information supplied in the Malware Must Die! blog from September, the filename is structured to be aligned with the port on which the eventual botnet communication will occur. However, the attacker had moved on from port 3502 through 3505 that were used back in September and is now making use of TCP ports 8000 through 8008 and 3306.

The first host’s file is a more detailed shell script, which provides an insight to some of the logic used by the attacker. Although named with a .rar extension, each file is actually a Unix shell script, using the same obfuscation as from September:

dec(){ echo $@|tr “[a-zA-Z0-9\;-=+*\/]” “[.0-9a-zA-Z\/\/\:]”; }

The translation string did not change at all.

This simple bash function dec() takes obfuscated text and translates characters back to the intended text. With this function we can extract the interesting strings from the 8000.rar file retrieved:

(Source: Level 3 Threat Research Labs)

The most important lines here are the __download_url__ variable where the actual executable is retrieved and the __remote__ variable, which is used for command and control communication.

The only difference between each script (8000.rar vs 8008.rar for example) is the __download_url__ filename and port numbers for __remote__  change, which match the number from the filename of the script.

After confirming that nothing structural had changed in the malware, Level 3 Threat Research Labs loaded it inside a CentOS 7 VM as root and watched it work.

Monitoring Botnet Communication in Action

After retrieving and executing the binary executable from the __download_url__ at 23.234.60.140 and 23.234.19.202, the bot immediately begins searching for its command and control host. The executable has hardcoded 8.8.8.8 and 8.8.4.4 as its DNS resolvers and proceeds to attempt DNS resolution for the hostnames configured. Next, it attempts connections to the IPs and resolved hostnames that were contained within the __remote__ line of the shell script.

In the case of our VM the various versions of the malware were able to establish connectivity to C2s at:

• 240.140.152 (resolved from ndns.dsaj2a.org and ns2.hostasa.org)
• 218.112.7 (resolved from ndns.dsaj2a1.org)
• 143.5.25 (resolved from ndns.dsaj2a1.org and ns1.hostasa.org)
• 240.141.54 (resolved from ndns.dsaj2a.com, ndns.hcxiaoao.com, and ns3.hostasa.org)

Other communication behaviors were as expected: As part of a DDoS botnet, our bot was instructed to launch SYN floods each time it connected to the C2. The SYN packets were null padded to maximize bandwidth usage and did not bother to spoof their origin.  The attack communication from the C2 telling our bot to attack was confirmed to be XORed with the same key (BB2FA36AAA9541F0) that has been in use since the original malware research.

The other noteworthy communication was the every 30 minute retrieval of /config.rar from the original malware hosted site (23.234.60.140), this time by resolving the hostname info.3000uc.com. The file was retrieved with the user-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; TencentTraveler ; .NET CLR 1.1.4322)

This file, just like the others, was not a RAR file, but this time was XORed with the same 128bit key as the C2 communication. After decoding it, the team found a configuration file containing the following keys: md5, denyip, filename, and rmfile. These contain values referencing a number of other known trojans and malware. Researchers at avast! confirmed that the config is used to remove and kill competing malware so the bot can have full control of the machine.

Now that we have confirmed what the malware does, how it communicates, and with whom it is speaking, we began to assess its impact to victims on the wider internet.

Victim Impact

(Source: Level 3 Threat Research Labs)

Over a two-week period in late March and early April, we monitored a large number of IPs scanned by the attackers. We also identified which hosts in the Internet were active participants in the botnet. Of the botnet hosts, we are able to see that C2 communication was occurring over TCP ports 8000-8008 and 3306, as seen in the current version of the malware. However, a number of hosts were still communicating over TCP ports 3502-3508 as seen in previous versions.

After assessing the massive scale, impact, and duration, we decided it was time to work on taking this malicious infrastructure off the Internet.

Takedown

On Tuesday, April 7, 2015, Level 3 took action by blackholing all attacker traffic inside of our global networks. This ensured that no traffic to the attacker would be successfully sent through Level 3. We began outreach to other network operators, briefing them on the threat, and asking them to follow suit in removing it from the global Internet permanently.

We are also monitoring the attacker’s actions to maintain awareness of any changes they make to the attack infrastructure. Throughout the analysis period the attacker shifted their SSH scanning operation from 103.41.124.0/23 to 43.255.190.0/23, along with shifting multiple C2 and malware IPs. Our team expects to continue this monitoring as the attacker attempts to resurrect their DDoS capability.

What You Should Do

For those of you who have Linux machines running sshd on the open internet, be sure to follow the best practice of disabling root login in your sshd config file. That step alone would stop this particular attacker from being successful in your environment.

Additionally, you should consider running your SSH daemon in a way that avoids these types of attacks. Running a firewall locally on the Linux machine to protect against unknown access attempts is a strong step when possible. However, when unsophisticated scans occur, even the extremely simple step of running SSH on a non-standard port can be used as an avoidance method. Most commodity scanners and malware clients will not search for services on non-standard ports.

It is also important to ensure that passwords have a minimum complexity and that common dictionary attacks are not effective against any user’s password. To help protect against this problem we have attached the passwords attempted by this attacker as collected by Cisco’s honeypots. The list can be encrypted and compared to user passwords to validate they cannot be easily attacked.

We also recommend that everyone monitor DNS traffic traversing their networks for abnormalities. This particular malware hardcoded open resolver IPs that would have been an indicator to victims infected that something was abnormal within their environment.

Of course, you should take a look for traffic to any of the IPs or hostnames mentioned here to ensure that you do not have any devices speaking with the attacker or participating in the botnet:

Defending the Internet

After an attacker is found, it is critical that the security community clean up the attacker’s infrastructure and ability to execute, rather than just observe the problem. Such collaboration leads to an improvement in the security posture of the internet as a whole.  In the case of SSHPsychos, the work done by the combined teams of Level 3 and Cisco have resulted in an improvement in security for everyone using the internet. Level 3 and Cisco hope to continue that trend together and see others join us in that effort. It takes a village.