What Happened?
Cisco Incident Response (CSIRS) recently responded to an incident involving the Watchbog cryptomining botnet. The attackers were able to exploit CVE-2018-1000861 to gain a foothold and install the Watchbog malware on the affected systems.

This Linux-based malware relied heavily on Pastebin for command and control (C2) and operated openly. CSIRS gained an accurate understanding of the attacker's intentions and abilities on a customer's network by analyzing the various Pastebins. As the investigation progressed, CSIRS identified and de-obfuscated multiple pastes using artifacts left on compromised hosts.

There were some attempts at obfuscation, such as base64 encoding URLs and Pastebins, but the attack was still relatively simple to uncover - this attacker did not practice particularly strong operational security.

The attackers behind Watchbog claimed to be providing a service by identifying security vulnerabilities and aiding the organization by exploiting said weaknesses before any "real" hackers could do so. During the investigation, Cisco IR found signs of hosts becoming a part of a separate botnet around the time of the Watchbog activity. This raises serious doubts about the "positive" intentions of this adversary. Below is a message left on a compromised system by the adversary:

What does Watchbog do?  
The Watchbog botnet mines Monero cryptocurrency for its owners. While researching our variant we came across a post by Alibaba Cloud Security that provides some insights into Watchbog. This post coincided with our findings as we found an installation script that performs the following activities.

First the installation script checks for running processes matching other cryptocurrency miners. If the system was previously configured to mine cryptocurrency, the installation script would terminate their execution using the kill command:

The script then uses the touch command to determine its capability to write to various directories on the filesystem.

It also checks the architecture of the system to determine if it is executing on a 32-bit or 64-bit operating system and then makes three attempts to download and install a 'kerberods' dropper using wget or curl.

Depending on permissions, the kerberods dropper is saved to one of the following directories:

  • The current working directory
  • /usr/bin
  • /usr/libexec
  • /usr/local/bin
  • /tmp
  • /usr/sbin

The script also retrieves the contents of a Pastebin URL containing a Monero wallet ID and mining information. CSIRS verified this as the same wallet ID as the one used by the attacker referenced in the Alibaba cloud post referenced earlier.

Though the Pastebin URL in the previous screenshot is no longer accessible, the next step in the infection process is to download the cryptocurrency miner. We identified a script that 'kerberods' likely runs to reach out to GitHub to install the XMR-Stak Monero miner.

The main part of the script checks to see if a process called 'watchbog' is running.

If the 'watchbog' process is not detected, the 'testa' or 'download' functions are called to install the version of the miner that's compatible with the host operating system and architecture and execute it to begin the mining process.

'Testa' function
As previously mentioned, the 'testa' function may be called to facilitate the infection process. Below is the code associated with this function. This code is responsible for writing the various configuration data used by the mining software. The function declares three variables and assigns base64 encoded data to each of them.

The base64 encoded data is then decoded and written to various files.

The base64 encoded values correspond to the following:

  • St_64: This variable contains the URL of the Github repository that hosts the XMR-Stak mining client.
  • hXXps://github[.]com/fireice-uk/xmr-stak/releases/download/2.10.3/xmr-stak-linux-2.10.3-cpu.tar.xz
  • con_url: This variable contains the Pastebin URL that is used to host the configuration file for the mining client.
  • hXXps://pastebin[.]com/raw/YJH8sWr
  • Cpu_url: This variable contains an additional Pastebin URL. During our investigation the Pastebin URL was no longer accessible, but likely contains an additional configuration file to be used by the mining client.
  • hXXps://pastebin[.]com/raw/irzk5mSh
  • poo_url:This variable contains an additional Pastebin URL. During our investigation the Pastebin URL was no longer accessible, but likely contains an additional configuration file to be used by the mining client.
  • hXXps://pastebin[.]com/raw/aJkbTx6Y

The script then starts the Watchbog process and deletes the text file after downloading the encoded Pastebins as a text file and giving it execution permissions. The following screenshot shows the configuration file that is referenced by the con_url variable in the 'testa' function.

'download' function
The following code is associated with the 'download' function referenced by the installation script previously described. Similar to what was described in the 'testa' function, it contains three declared variables with base64 encoded assignments.

These base64 encoded strings correspond to the following:

  • mi_64: This variable contains the Github URL that hosts the XMrig monero mining client.
  • hXXps://github[.]com/xmrig/xmrig/releases/download/v2.14.1/xmrig-2.14.1-xenial-x64.tar.gz
  • mi_32: This variable contains a Pixeldrain URL. During our investigation the URL was no longer accessible.
  • hXXps://pixeldrain[.]com/api/file/ZuVWceWG
  • der_ke: This variable contains a Pastebin URL. The URL was used to host a file containing the attacker(s) Monero Wallet ID for the miner to use. This Wallet ID is used to facilitate payment to the attacker. All Monero successfully mined by clients under the attacker's control will transfer the Monero to the Wallet ID specified in this file. The same wallet is included in the Alibaba Cloud post mentioned earlier.
  • hXXps://pastebin[.]com/raw/hURdMBLd

The download function then writes the contents retrieved from the specified URLs to various file locations. It then determines the architecture of the system and installs the appropriate mining client and executes it to initiate the mining process.

The following screenshot contains the contents of the Monero wallet configuration associated with the der_ke variable in the 'download' function described earlier. It specifies the configuration parameters that will be used by the mining client, including the Wallet ID, mining pool URL, and other parameters that can be used to control CPU usage, logging, etc.

Lateral movement via SSH
CSIRS identified that the adversary was using SSH to spread laterally. Although local logs were unavailable, we were able to use network logs to gain an understanding of how the malware was spreading. As we viewed the logs, it was easy to determine Watchbog's lateral movement mechanism because they were generating a large amount of SSH traffic. This could have been easily detected using internal traffic flow monitoring, such as with StealthWatch Cloud or other netflow-monitoring capability.

The following Bash script was used to facilitate the lateral movement process. It retrieves the contents of the known_hosts file on the infected system and then attempts to SSH into those systems. It also checks for the existence of SSH keys and leverages them to authenticate to the systems in the known_hosts file. If successful, it will retrieve the contents of the Pastebin URL previously described and initiate the infection process.

Lateral movement via Jenkins and Redis servers
In addition to leveraging SSH for lateral movement, the Watchbog adversary also attempted to leverage a Python script that scans for open Jenkins and Redis ports on the host's subnet. If the script finds any vulnerable servers, it attempts to use the curl or wget commands to retrieve a payload from Pastebin and execute it on the target.

Based on the following string on line 71, the script targets CVE-2018-1000861, a vulnerability in the Staple web framework for versions up to Jenkins 2.138.1 or 2.145 which handles HTTP requests. It can provide attackers with RCE through particularly crafted URLs. A post by Orange Tsai shows how to exploit this vulnerability by using cross reference objects to bypass ACL policy.

Though the pastes accessed in the script were no longer available, we believe the payload was the installation script for the XMR-Stak miner previously described. The following Python script is also downloaded and executed from the XMR-Stak miner script described above in a function called 'party.'

As can be seen above, the payload variable contains a base64 encoded blob which is then decoded and written to the /tmp directory and executes it. This base64 encoded blob contains a Pastebin URL (hXXps://pastebin[.]com/raw/DzgYb9mu) which was used to host the following Python script. The Python script is used to facilitate the exploitation of the aforementioned vulnerability and initiate the infection process. The following screenshots are associated with this Python script.

Persistence
Watchbog's main persistence mechanism appears to have been using cron jobs. Below is the 'system' function from the 'kerberods' installation script which ensures the dropper will call out to Pastebins every hour for new information. The below screenshot shows the way that Watchbog configures the cron jobs responsible for achieving persistence on infected systems.

In a post by Renato Marinho from Morphus Labs, he mentions a very interesting way 'kerberods' achieves persistence as well. If it has root privileges, it will download and load a library into the operating system which hooks parts of Glibc to modify Glibc's behavior. The post also specifies that the hooks allow the miner to run as anyone (including root) and also obfuscates the network connection to the mining pool as well as the Redis/Jenkins server scans.

Covering their tracks
Evidence deletion has been identified in previous Watchbog variants. The Watchbog variant in our incident continued this trend. Evidence deletion was performed in a clear manner with files and logs being deleted or overwritten. The evidence deletion was typically added to the end of a handful of the Pastebin scripts, with the Xmr-stak download and the SSH Lateral Movement scripts being prime examples. The loss of those key pieces of evidence made analysis difficult, but not impossible. We were able to rely upon our clients centralized logging to fill in those holes, and the hosts themselves still had evidence. The most obvious being the malware variants themselves.

Conclusion  
Unpatched web applications vulnerable to known CVEs are a major target for attackers. Adversaries can leverage the vulnerability to gain a foothold into the web server and network environment in which the web server is deployed. Once that foothold has been established, the attacker can then connect to their C2, achieve persistent long-term access to the environment and spread laterally — which is exactly what happened in this case. The best way to prevent such activity would be to ensure that all enterprise web applications are up to date. Patching can cause some operational gaps and delays, so it’s also important to have a maintenance window and a test environment to ensure that the new patches do not cause any issues. Identifying cryptomining activity can be done effectively by following security fundamentals. Establish a baseline for internal network traffic and if any significant deviations occur, identify and investigate them. Even if there is an existing theory for the activity. In this case, Watchbog generated a noticeable spike in the organization’s SSH traffic.

Coverage Intrusion prevention systems such as SNORT® provide an effective tool to detect China Chopper activity due to specific signatures present at the end of each command. In addition to intrusion prevention systems, it is advisable to employ endpoint detection and response tools (EDR) such as Cisco AMP for Endpoints, which gives users the ability to track process invocation and inspect processes. Try AMP for free here.

Additional ways our customers can detect and block these threats are listed below.

Cisco Cloud Web Security (CWS) orWeb Security Appliance (WSA) web scanning prevents access to malicious websites and detects malware used in these attacks.

Email Security can block malicious emails sent by threat actors as part of their campaign.

Network Security appliances such asNext-Generation Firewall (NGFW), Next-Generation Intrusion Prevention System (NGIPS), and Meraki MX can detect malicious activity associated with this threat.

AMP Threat Grid helps identify malicious binaries and build protection into all Cisco Security products.

Umbrella, our secure internet gateway (SIG), blocks users from connecting to malicious domains, IPs, and URLs, whether users are on or off the corporate network.

Open Source SNORTⓇ Subscriber Rule Set customers can stay up to date by downloading the latest rule pack available for purchase on Snort.org.

Indicators of Compromise (IOCs)
The following IOCs have been observed associated with Watchbog.

Hashes (SHA256):
b383d0fdfa5036ccfa5d9c2b43cbfd814bce8778978873057b86678e5295fc61 0b0567c9b45ea0a3ea4267001f0760ccdf2b8224fceaf8979d32fcceb2d6fb7a

3A6271A90D0F6CC8A2D31D45D931E8401F13F7377932BA07D871DC42F252B9CA

Domains:
aziplcr72qjhzvin[.]onion[.]to

Misc:
Monero Wallet (Same wallet as the Alibaba Cloud Post)

47k2wdnyyBoMT6N9ho5Y7uQg1J6gPsTboKP6JXfB5msf3jUUvTfEceK5U7KLnWir5VZPKgUVxpkXnJLmijau3VZ8D2zsyL7