Detection of malware is a constant battle between the technologies designed to detect and prevent malware and the authors creating them. One common technique adversaries leverage is packing binaries. Packing an executable is similar to applying compression or encryption and can inhibit the ability of some technologies to detect the packed malware. High entropy is traditionally a tell-tale sign of the presence of a packer, but many malware analysts may have probably encountered low-entropy packers more than once. Numerous popular tools (e.g., PEiD, Manalyze, Detect It Easy), malware-related courses, and even reference books on the topic, affirm that packed malware often shows a high entropy. As a consequence, many researchers use this heuristic in their analysis routines. It is also well known that the tools typically used to detect packers are based on signature matching and may sometimes combine other heuristics, but again, the results are not completely faithful, as many of the signatures that circulate are prone to false positives.

This signature can be found in many packer signature databases available online, and is responsible for many false positives (e.g. 7z.exe, a commonly used tool, is flagged as being packed by Armadillo).

This imprecision has many consequences on malware related systems and studies:

• Sample ingestion pipelines often rely on static data, which is not reliable if a sample is packed.
• Machine learning based classifiers need to be trained with a solid source of ground truth. Polluted datasets negatively affect the reliability and performance of these approaches.
• A wrong classification of packed vs. not packed malware can affect studies that analyze trends in malware.
  Researchers have sporadically mentioned the existence of low-entropy packers, but it was still unclear if this phenomenon is relevant or negligible. We conducted a systematic study over a curated dataset of 50,000 low-entropy malicious PE files belonging to multiple families, obtained from the VirusTotal public feed, observed between 2013 and 2019. Then, we leveraged a dynamic analysis system based on the PANDA binary instrumentation framework in order to provide ground truth about the presence of a packer and the scheme used to keep entropy low: byte padding, encoding, transposition, monoalphabetic and polyalphabetic substitution.

For further details about these schemes, our dynamic analysis component, and how we designed and conducted these experiments, you can refer to the full paper. Our results show that over 30 percent of these low-entropy files adopt some type of run-time packing. Similarly, we obtained a dataset belonging to APT samples obtained from numerous reports and whitepapers, conducted a similar experiment and observed that up to 15 percent of these low-entropy files were packed. These numbers confirm that the phenomenon is not negligible and must not be ignored in future studies.

In a second test, we evaluated the detection rates of commonly used tools such as PEiD, DIE, and Manalyze. All the tools had difficulties to recognize low-entropy packers as packed, and some of them showed false positive detections of common off-the-shelf packers due to weak heuristics or signatures.

Finally, we conducted an experiment to evaluate several machine-learning based classification approaches to distinguish between packed and not packed files. For this, we collected all the features used to date in different academic publications. In all cases, these classifiers showed an important performance degradation when trained and evaluated on a dataset with low-entropy packers: even the best classifier was able to detect only 70 percent of the packed samples in our dataset.

For more on our research, the authors of this research paper will be presenting their work at NDSS on February 26th, 2020. You can also download the full research paper here...