Machine learning based fileless malware traffic classification using image visualization

Abstract

In today’s interconnected world, network traffic is replete with adversarial attacks. As technology evolves, these attacks are also becoming increasingly sophisticated, making them even harder to detect. Fortunately, artificial intelligence (AI) and, specifically machine learning (ML), have shown great success in fast and accurate detection, classification, and even analysis of such threats. Accordingly, there is a growing body of literature addressing how subfields of AI/ML (e.g., natural language processing (NLP)) are getting leveraged to accurately detect evasive malicious patterns in network traffic. In this paper, we delve into the current advancements in ML-based network traffic classification using image visualization. Through a rigorous experimental methodology, we first explore the process of network traffic to image conversion. Subsequently, we investigate how machine learning techniques can effectively leverage image visualization to accurately classify evasive malicious traces within network traffic. Through the utilization of production-level tools and utilities in realistic experiments, our proposed solution achieves an impressive accuracy rate of 99.48% in detecting fileless malware, which is widely regarded as one of the most elusive classes of malicious software.

Introduction

Network traffic flow classification is an essential network function that paves the way for dynamic and agile network management. It empowers network operators to handle different service requirements and constraints to allocate resources more efficiently while maximizing their utilization. It also enables security-based network traffic engineers to identify malicious patterns affecting network services’ availability and performance. As most devices maintain their connectivity through networks, most cyber threats are triggered through network traffic which ranges from malware infection to distributed denial of service (DDoS) attacks. Hence, it is essential to identify malicious network traffic that targets the underlying devices.

Fileless malware (Kumar 2020) is a type of evasive malware that is notable for its capability to reside solely in the system’s main memory, without leaving traces on the disk or file system. These malware attacks employ manipulation techniques on legitimate libraries and utilities of benign platforms to achieve their objectives (Saad et al. 2019). Conventional methods of file-based detection operate by scanning for any malicious programs or software. However, benign software that is included in a whitelist is never subjected to testing because the malware detection system does not classify them as malicious (Saad et al. 2019). This allows fileless malware to exploit vulnerabilities in trusted and widely used applications (Smelcer 2017), such as web browsers, text-processing applications, and video players. Commonly abused tools by fileless malware include Bitsadmin, certutil, MS Office macros, mshta, msiexec, PowerShell, psexec, registry, regsvr32, task scheduler, WMI, and VBscript (Borana et al. 2021).

Popular malware detection systems like Signature-based and Static analysis are ineffective at detecting fileless malware because this type of malware does not leave any residual traces in the file system after the attack has concluded. As a result, fileless malware has an advantage in evading traditional detection solutions (Barnes 2021). The ability to leave no trace after an attack also makes it challenging for forensic investigators to use reverse engineering to analyze fileless malware attacks (Saad et al. 2019).

In general, three unique properties of fileless malware make them different from file-based malware (Babar 2020). First, fileless malware does not require data to be written on the target’s file system at the time of infection and propagation. Second, it does not download any file on disk. Third, fileless malware depends on pre-installed benign software libraries and utilities to execute the malicious payload.

Recent advances in combining network traffic visualization with machine learning techniques have shown promising results in the detection of malicious patterns in network traffic (Joo et al. 2021; Shapira and Shavitt 2019; Wang et al. 2017). The conversion of network traffic to image facilitates the visualization of possible changes in network traffic patterns before and during different attacks. Research has shown that network traffic exhibits strong patterns of behavior across multiple timescales (Taheri et al. 2018; Wang et al. 2017; Xu et al. 2021). By leveraging visualization techniques, the complex patterns and characteristics within network data can be effectively analyzed, enabling the machine learning algorithms to make more informed and precise classifications. This combination of visualization and machine learning plays a vital role in enhancing the effectiveness and reliability of detecting malicious activities in network traffic. Image-based techniques can offer advanced visual aids that enable the quick detection of suspicious, unknown malware and the timely alerting of anomalous behavior patterns. Representing malware as an image enables the detection of even small changes while preserving the overall structure of samples from the same malware family (Gibert et al. 2019). Visual representations of malware patterns can also provide a concise overview of potential attacks. Additionally, converting malicious executables into images makes it easier to differentiate between different binary sections.

Fig. 1

Related malware families

Distinct sections of the executable binary are depicted as distinctive image textures, which facilitates faster analysis and can increase the productivity of malware analysts. Figure 1 demonstrates that related malware classes exhibit stronger visual similarities with one another than with unrelated classes, as shown in Fig. 2.

Fig. 2

Unrelated malware families

Summary of contributions

This paper introduces a solution for detecting fileless malware traffic using image visualization based on Convolutional Neural Networks (CNN). The key contributions of this paper are as follows:

• We propose a novel approach for malware detection using image visualization, specifically targeting fileless malware and low-rate attacks. Existing studies in image visualization-based malware detection have primarily focused on file-based malware detection and classification. However, the application of such visualization techniques for detecting evasive traces generated by stealthy fileless malware or low-rate attacks (Khorsandroo and Tosun 2018, 2019) has not been extensively explored. Notably, at the time of writing this paper, no published work has addressed the classification of Cobalt Strike beacons using machine learning and image visualization.

• We explored the viability of employing a Convolutional Neural Network (CNN) for an image-based network traffic classification to effectively identify elusive traces of fileless malware. To obtain a realistic network traffic payload, we utilized Cobalt Strike (Rahman 2021), a commercial adversary simulation software. It is worth noting that while penetration testing tools like Mimikatz, Cobalt Strike, and Metasploit are designed for assessing the security of IT networks from a red teaming perspective, Cobalt Strike, in particular, has gained notoriety for its utilization by threat actors.

• The objective of our approach is twofold: firstly, to address the relatively under-explored challenge of classifying Cobalt Strike beacons, and secondly, to efficiently identify this form of fileless malware within a vast dataset. Unlike previous works conducted in laboratory settings, our experiment was conducted in a production-level environment, ensuring the representativeness of the dataset in relation to real-world malware (Yadav and Tokekar 2021). In this experiment, we assess the performance of our model and compare it to prior techniques, taking into account the varying behaviors exhibited by Cobalt Strike beacons. We convert the collected Cobalt Strike beacon payloads into grayscale images and employ image transformation techniques to enhance their effectiveness. These generated images are then utilized to train a CNN model, enabling the classification of network traffic as either benign or malicious. Our results demonstrate that our solution surpasses existing methods in accurately detecting stealthy malicious traffic associated with fileless malware.

The remainder of the paper is organized as follows. “Related work” Section presents a detailed literature review on image-based malware classification and fileless malware detection mechanisms. This provides a foundation for understanding the existing research in the field. Building upon the literature review, “Types of use cases” Section explores the various use cases of image-based network traffic classification. By examining different scenarios and applications, we demonstrate the practical relevance of this approach. In “Image representation methods” Section, we delve into the investigation of image representation methods. This analysis allows us to identify the most effective techniques for representing images in the context of malware detection. “Methodology: from network traffic to image classifier” Section elaborates on the proposed methodology we used to visualize Cobalt Strike beacon payloads and automatically classify them using images as input. We explain the steps and techniques involved in this process, highlighting the unique aspects of our approach. To support the visualization and classification process, “Proposed CNN architecture” Section provides details on the designed Convolutional Neural Network (CNN). We describe the architecture and components of the CNN model, highlighting its suitability for our purposes. Moving forward, in “Experimental setup and evaluation” Section, we discuss our experimental setup and the evaluation of the proposed approach. We present the data used, the metrics employed, and the results obtained, providing a comprehensive assessment of our methodology. Finally, in “Conclusion” Section, we conclude the paper and discuss future directions. We summarize the key findings and contributions of our work and highlight potential avenues for further research in the field of image-based network traffic classification and fileless malware detection.

Related work

Numerous visualization techniques relying on images have been suggested in academic studies for the purpose of facilitating the analysis of network traffic classification. This section presents a summary of image-based approaches to network traffic classification, employing various machine learning algorithms. Additionally, it explores mechanisms designed for the detection of fileless malware.

Overview of image-based network traffic classification

Nataraj et al. (2011) were among the early adopters of the malware visualization technique for the purpose of classifying diverse malware families. The method proposed by the authors involves converting malware binary data into 2D grayscale images. They noted that malware variants belonging to the same family exhibit similarities in image texture. Initially, the authors extracted GIST features to classify malware, followed by the application of K-Nearest Neighbors (KNN) algorithm using Euclidean distance. KNN is a supervised learning technique commonly used for classification and regression problems. In KNN, unclassified samples are assigned to a specific class based on a majority vote from their neighboring samples (Shabtai et al. 2009). If a significant proportion of the k nearest neighbors belong to a particular class, the sample will be assigned to that class (Xie et al. 2018). In their extensive analysis, the authors utilized a dataset of 9458 malware samples, encompassing 25 malware families, and achieved a classification accuracy of 97.18%.

Dai et al. (2018) proposed a technique involving extracting grayscale images from malware memory dump files. They employed K-Nearest Neighbors (KNN), Random Forest (RF), and Multi-Layer Perceptron (MLP) for malware classification. RF, being an ensemble model, generates a response variable based on results from different decision tree models and has proven effective in solving classification and regression problems (Xie et al. 2018). When running a random forest, it is important to specify parameters such as training data, response variables, the number of trees, predictor variables, error calculation parameters, and other relevant factors. Furthermore, MLP (Taud and Mas 2018), belonging to the class of powerful Artificial Neural Networks (ANNs), is widely used. It employs supervised training to generate a non-linear model capable of predicting output based on input data. To extract image features, the authors employed histograms of gradients and followed three key steps for classifying memory dumps. Initially, they extracted runtime malware memory dump files using a sandbox. Next, they converted the memory dump into grayscale images and extracted feature vectors using bi-cubic interpolation. To validate their approach, the authors conducted experiments using actual malware samples, demonstrating the effectiveness of their methodology.

Gibert et al. (2019) introduced a Convolutional Neural Network(CNN) to the Nataraj et al. (2011) approach to classify assembly language within portable executable (PE) files. A Convolutional Neural Network is part of deep learning whose connectivity pattern is inspired by mammals’ visual cortex structure (Hubel and Wiesel 1968). Without human supervision, CNN can process an image’s high dimensionality. CNN accepts the raw pixels of an image as input and performs feature extraction. By transforming the input image, CNN can learn important objects constituted within. In their work, the authors represented malware binary content as a grayscale image to find a pattern and classify malware into families. By deriving features such as local and invariant out of the image, they inferred that the presence of malware patterns could be detected.

Ni et al. (2018) proposed a malware family classification technique that combines image-based malware visualization and Convolutional Neural Network. In this work, the authors extracted similar hash values of similar malware codes using the locality-sensitive hashing technique. In this regard, similar Hash values produce a similar grayscale image. The performance of the proposed algorithm was improved by considering additional methods such as multi-hash, major block selection, and bilinear interpolation. Finally, Convolutional Neural Networks were used to identify which family the malware belongs to.

Zhang et al. (2019) Converted network flow data to a grayscale image to classify encrypted network traffic. The authors combined Long Short-Term Memory (LSTM) and Convolutional Neural Network(CNN) to extract statistical features. As a class of recurrent neural networks (RNN), long-short-term memory networks (LSTM) are capable of dealing with long-term dependency (Liu et al. 2018). Along with the hidden state, LSTM has an additional information flow path compared to other forms of RNN. Input, output, and forget gates are the three components of Cell which is the additional processor introduced in LSTM. The Cell controls the flow of information and decides which information needs to be dropped by going through the network (Li et al. 2021). This property of Cell helps LSTM to store information that is acquired from previous steps. In their work, the authors apply the LSTM model to extract the representation vectors of the images. On the other hand, to extract representation features and classify the network CNN algorithm is used.

Fileless malware detection mechanisms

van der Eijk and Schuijt (2020) developed an algorithm to detect the presence of a Cobalt Strike Command and Control (C2) beacon in a network communication based on NetFlow data. The authors established a network topology that consists of a Cobalt Strike C2 server, domain redirection, and target infrastructure. And configure a NetFlow that helps them distinguish between malicious and benign data. They analyzed the characteristics of the regular HTTP and HTTPS network connection of Cobalt Strike. For their experiment, they used a static algorithm.

Hendler et al. (2018) proposed a Deep Neural Networks-based fileless malware detection method. In this study, the authors implemented several detectors for identifying malicious PowerShell commands. The evaluation of their detection model demonstrated high recall values and an exceptionally low false positive rate. The authors proposed that combining Natural Language Processing (NLP) techniques with Convolutional Neural Networks (CNNs) resulted in the optimal performance for this task.

Authors in Bucevschi et al. (2019) proposed a machine learning based fileless attack prevention. They tested the presence of an anomaly in command lines such as PowerShell scripts, Windows Management Instrumentation (WMI) scripts, Windows tasks, shortcut files (LNK), and Batch scripts. In this paper, the authors use One Side Class perception(OSC) which is a modified version of the Perceptron algorithm. By providing an extra stage that will maintain a small false positive rate, OSC ensures the correctness of classification in the training phase. To validate their proposed model, the authors utilized a dataset consisting of a total of 37,546 samples that were labeled as both anomaly and anomaly-free. In their experimental evaluation, the authors achieved a detection rate of 83.32%.

Rigaki and Garcia (2018) used Generative Adversarial Networks(GANs) to mimic network traffic. GAN generates network traffic, and the malware source codes were modified to accept parameters from GAN. GAN allows them to mimic legitimate application traffic and bypass malware detection. Real-life experiments were conducted on Facebook chat traffic to train it with GAN. The authors successfully mimic Facebook’s chat by modifying the behavioral patterns of real-world malware samples. In this work, the authors applied deep learning to create malware samples. Additionally, they evaluated the effectiveness of these enhanced malware samples in evading detection by a machine learning-based malicious traffic detection system known as Stratosphere Linux IPS (slips). Their experiment revealed that approximately 63.42% of the malicious samples successfully bypassed the detection capabilities of the Stratosphere Linux IPS system.

The presence of malicious PowerShell was tested by FireEye (Fang 2018). The author used the Natural Language Processing (NLP) pipeline. The key to the proposed NLP module are Decoder, Named Entity Recognition (NER), Tokenizer, stemmer, Vocabulary Vectorizer, Supervised classifier(Kernel SVM, Gradient Boosted Trees, Deep Neural Networks), and Reasoning. Any encoded text within the PowerShell script will be detected and decoded. Then the PowerShell command will be tokenized to create a list of tokens. All semantically identical tokens will go through the stemming algorithm to reduce them to the original word form. Prior to inputting the token list into the machine learning algorithm, a vectorization process is applied to ensure a machine learning-friendly format. This process transforms the tokens into a suitable representation. Subsequently, the supervised model executes the prediction.

Borana et al. (2021) proposed an assistive tool to detect fileless malware. This tool will perform a forensic examination to identify abnormal processes and abnormal activities on the system and network. The authors also discussed fileless malware life cycles along with their infection strategies. In their study, Handaya et al. (2020) suggested three machine learning algorithms(KNN, SVM, Random Forest) to detect fileless cryptocurrency mining malware accurately. The authors recommended that researchers use the EMBER dataset that contains more than 1 million SHA-256 hashes collected from portable executable (PE) files. This dataset contains 900 K samples for training and 200 K samples for testing.

Types of use cases

Image-based traffic classification holds potential for various applications in network traffic analytics. The existing body of literature primarily emphasizes two key use cases: malware detection and traffic engineering.

Internet traffic classification

Internet traffic classification encompasses the classification of network traffic into different traffic classes, utilizing a range of features (Dhote et al. 2015). Two main approaches have been employed for network traffic classification: rule-based and machine learning-based methods. Rule-based methods involve techniques such as port-based and payload-based approaches. Conversely, machine learning-based methods encompass statistical-based and behavioral-based approaches, as depicted in Fig. 3.

Fig. 3

Internet traffic classification approaches

Rule-based approaches

In this approach, packets that enter the network will be classified according to their predefined hard-coded rules (Lim et al. 2019; Wang et al. 2017). This method is one of the traditional classification methods that suffer from dynamic ports, and encrypted applications (Zhang et al. 2012). Thus, the following two approaches are frequently employed.

• (i) Port-Based Approaches: As one of the traditional classification methods, port-based methods rely mainly on port numbers, and it uses only information from the packet’s header (Dhote et al. 2015; Tahaei et al. 2020). This method was successful because many application hosts use a well-known port to communicate with other hosts. In early times, it was easier to find a packet’s target port number as most applications will have their port numbers registered at the Internet Assigned Numbers Authority (IANA). Then using the classifier, the port numbers will be associated with the application. For example, port number 53 is associated with DNS, and port 80 is for HTTP traffic. Even though this method is fast and has a simple implementation, it fails to detect correctly if a fake port number is in the traffic (Tahaei et al. 2020). In addition, the existence of dynamic port allocation in recent applications which are private ports and are not available in the IANA’s list, and encryption of packet header of the IP layer which creates obfuscation in TCP or UDP port number, makes port-based approaches obsolete (Barut et al. 2020).

• (ii) Payload-Based Approaches: To address the limitations of port-based approaches, payload or deep packet inspection (DPI) has been utilized. With this method, network traffic is classified by examining both the packet header and the payload information from the application layer (Lim et al. 2019; Tahaei et al. 2020), and matching them to stored signatures (Lim et al. 2019). In this approach, the payloads are scrutinized bit by bit to identify a match for a predetermined byte sequence. Then stored signatures are compared with the matched bit stream and classification will be performed accordingly (Dhote et al. 2015). This method solves the problem of port number dependency (Lim et al. 2019) and performs the network application classification accurately. Despite its advantage, this method fails to classify correctly if signatures are not up to date or if the payload is encrypted (Barut et al. 2020). Additionally, DPI can have high computational requirements, which can cause delays in network traffic (Dhote et al. 2015; Tahaei et al. 2020). Stochastic packet inspection (SPI) was proposed as a solution to the limitations of payload-based classification methods. SPI uses statistical methods to analyze the traffic flow of packets rather than inspecting the payload itself. SPI captures statistical features of the traffic flow such as packet arrival times, packet size, inter-packet time, and direction of the flow (Zhao et al. 2021). These statistical features are then used to create models for different traffic classes. SPI is more resilient to encrypted traffic and can provide a more accurate classification than payload-based methods (Tahaei et al. 2020; Zhao et al. 2021). Moreover, this method exposes user data privacy as the packet’s content is inspected thoroughly (Tahaei et al. 2020).

Machine learning-based approaches

The limitations of previous techniques directed researchers to apply machine learning approaches that do not depend only on the port number or payload (Barut et al. 2020). Accordingly, the following two approaches are commonly used.

• (i) Statistical-Based Approaches: This approach uses flow-level properties such as flow duration, flow idle time, packet inter-arrival time, and packet length with the assumption that traffic at the network or transport layers will be unique for certain classes of applications (Dhote et al. 2015; Nguyen and Armitage 2008; Tahaei et al. 2020). These methods solve the problem of payload-based approaches as they avoid content inspection (Tahaei et al. 2020).

• (ii) Behavioral-Based Approaches: In this approach, flow level, packet level, and connection level data were used in order to check a host’s behavior. Packet header fields such as IP address, port number, and protocol type play the main role in behavior-based classification to identify an application behavior in a host (Zhao et al. 2021).

Image representation methods

Data preprocessing plays a vital role in preparing raw traffic data for classification and detection algorithms. Its objective is to convert the raw data into a suitable format that machine learning algorithms can effectively utilize. One common approach is to convert the data into an image format, which can be fed into machine learning models for classification. Representing traffic data as an image enables the extraction of significant features and patterns that assist in the identification of different traffic classes and the detection of anomalies (Nataraj et al. 2011; Naeem et al. 2019).

Network flow to image conversion

In line with existing literature, raw data will be captured as packet capture files (PCAP) or binary files (BIN). The collected network traces will go through three stages: traffic splitting, traffic sanitizing, and outlier removal. Traffic splitting categorizes the captured traffic into different representations, such as flow based on header fields, layer seven information, connection sessions with header fields, or connection sessions with only layer seven information. Flows based on session information are typically stored as PCAP files, while flows based on layer seven information are saved in BIN format.

During sanitization, Media Access Control(MAC) and Internet Protocol(IP) addresses in the data link and network layers are randomized, removing identical, duplicated, and empty files without altering the data format. This process helps address biases during machine learning model training.

Outliers, files abnormally larger or smaller than the rest, are removed before generating image files from the captured traffic. To ensure uniformity, the input data size (images from network traffic) is adjusted by trimming or padding. Trimming reduces the file size to the desired length, while padding adds 0 × 00 to smaller files. The preprocessed data, now uniform in size, is transformed into grayscale images by representing each byte as a pixel in the image. A cutoff size is chosen to ensure equal length and width, e.g., a 28 × 28 pixel grayscale image corresponds to a cutoff size of 784 bytes. Figure 4 illustrates the general concept of mapping a byte array to a grayscale image.

Fig. 4

Byte array to grayscale image conversion

Binary code to image conversion

There are research works (e.g., Nataraj et al. 2011; Ni et al. 2018; Naeem et al. 2019; Su et al. 2018) that discuss techniques for converting raw binary code into an 8-bit vector and generating a grayscale image as shown in Fig. 5. The grayscale image represents the binary code as a 2-dimensional array, where each pixel has an 8-bit unsigned integer value ranging from 0 to 255. Black is represented by 0, and white is represented by 255. Each pixel in the grayscale image represents intensity information, ranging from 0 to 255 (Kumar et al. 2016). In the conversion process, the binary bit string is divided into 8-bit substrings, which are then converted to decimal numbers. The conversion is done by summing the binary digits multiplied by powers of 2. For example, a binary number $B=(b_{n-1}...b_{4} b_{3} b_{2}b_{1}b_{0})$ can be converted to a decimal number $D =(b_{0}*2^{0}+b_{1}*2^{1}+b_{2}*2^{2}+b_{3}*2^{3}+b_{4}*2^{4}...)$). As Fig. 6 illustrates, a bit string such as $B = 0110000010101100$ can be split into two substrings of $B1 = 01100000$ and $B2 = 10101100$. Then, B1 and B2 can be converted further to decimal numbers (that is, $B1 = 01100000 \rightarrow 96$ and $B2 = 10101100 \rightarrow 172$). The resulting decimal numbers form a 1D vector representing the intensity of each pixel (Kancherla and Mukkamala 2013). The 1-dimensional array can be transformed into a 2-dimensional matrix based on the preferred width. Since there are varying input samples, the grayscale output image will have distinct widths, and heights (Gibert et al. 2019).

Fig. 5

Binary to grayscale image

Fig. 6

Binary to decimal conversion

Binary to color code mapping

A recent work (Chukka and Devi 2021) has demonstrated the ability to better capture patterns within malware instruction using opcode as a color-coded pixel. The binary-to-color code mapping process involves five steps: collecting opcodes from all code sections, identifying unique opcodes, mapping each unique opcode to a distinct color, arranging opcodes in a two-dimensional image grid, and replacing the opcodes on the grid with the corresponding color codes from the mapping. According to Chukka and Devi (2021), using grayscale images has limited effectiveness in identifying behavioral patterns in binaries. Grayscale representation of raw binary files becomes noisy due to the varying sections in Portable Executable (PE) binaries (e.g.,.text,.data).

Binary to RGB image

RGB color format uses three colors (Red, Green, Blue) represented by 8 bits each, totaling 24 bits to define a color out of millions of possibilities. A recent technique for malware detection involves converting raw binary files into RGB images. In RGB images, each pixel represents three sequential bytes, allowing for capturing more bytes in each row and ensuring consistent byte-level alignment. This approach facilitates placing more information in each row, making visual similarities between similar samples more apparent and easily identifiable. Unlike grayscale-based approaches, RGB-based encoding enables more compact images by reducing pixel space with a 1/3 ratio, resulting in less distortion during post-image resizing. However, this approach may have drawbacks when byte-level variations become prominent in raw binary data, compared to 8-bit grayscale encoding (Bozkir et al. 2019, 2021).

Non-image data into an image conversion

The authors in Sharma et al. (2019) introduced the concept of DeepInsight, which transforms non-image data into an image through three main steps. Firstly, non-linear dimensionality reduction techniques like kernel PCA or t-SNE are applied to map features into a lower-dimensional space, typically two-dimensional. Secondly, the smallest rectangle box encompassing all points is determined using the convex hull algorithm, and the image is rotated to fit horizontally or vertically. Lastly, the pixels are framed and mapped to obtain the final pixel coordinates. In DeepInsight, pixels that do not contribute to representing features are left blank, particularly in the presence of outliers, which can sometimes account for significant portions of the image. Additionally, compared to similar techniques such as IGTD (Zhu et al. 2021), DeepInsight produces larger images, necessitating more memory and longer training times.

Symbolic data conversion

In datasets containing both numeric and symbolic data types, symbolic features like protocol types (e.g., TCP, UDP, ICMP) need to be converted into numeric form to be used in machine learning models. One-hot encoding is a common technique for this purpose. It represents categorical variables as binary vectors. In the context of network traffic analysis, one-hot encoding can be applied to represent the byte values of the traffic. Each distinct category is mapped to an integer value, and then each integer value is represented as a binary vector. Entries corresponding to categories not represented by the vector are set to zero, while the entry representing the category is set to one. Thus, the one-hot encoder takes an input value of m integers and outputs an m-sized binary vector. For example, to represent the byte value ‘0’, a 256-sized vector is constructed with the first entry set to one and the others set to zero. This transformation replaces single values with vectors and converts the original 1D vector of integers into a 2D binary array (Krupski et al. 2021).

Methodology: from network traffic to image classifier

This section elaborates on how the captured network traffic will be converted into an image. It then discusses the architecture of the Convolutional Neural Network (CNN), which takes the produced images as input.

Image generation: implementation

We perform image conversion based on the payload collected from one of the prominent attack toolkits used by malicious actors, namely Cobalt Strike (Seazzu 2016). Cobalt Strike is a threat-emulating proprietary software suite that emulates embedded actor beacons in a network. A Cobalt Strike beacon is a fileless malware attack. Fileless malware (Kumar 2020) is an evasive malware that does not rely on files. Instead, it exploits vulnerabilities on legitimate, trusted, and widely used applications to infect a system (Smelcer 2017). When fileless malware is in action, it performs malicious activity using native tools built into a system to steal data, interfere with operations, or use compute resources. Moreover, traditional methods of detecting malware will no longer work to detect these threats since they cannot leave any traces behind. It is why fileless malware attacks have 10 times higher success rate than traditional file-based attacks (Sanjay et al. 2018). The use of beacons within network traffic is a common method for communicating with external servers and emulating malicious commands. Cobalt Strike beacons, in particular, are known for their ability to blend in with legitimate traffic due to their communication flexibility, making them difficult to detect (Seazzu 2016). Apart from spear-phishing, Cobalt Strike is capable of mimicking malware and other sophisticated threat techniques to obtain unauthorized access to systems (Rahman 2021).

The Cobalt Strike beacon payload employs asynchronous communication patterns characterized as “low and slow.” For beacon communication, Cobalt Strike uses three alternative transport mechanisms, such as HTTP, HTTP(s), and DNS. In this experiment, we used an HTTP request header. Through an HTTP GET or POST request, Cobalt Strike’s beacon payload retrieves tasks from its team server. The size of the traffic flow for HTTP requests generated by the Cobalt Strike beacon depends and influenced by a variety of factors. These factors include the type and size of the requested resource, the HTTP headers that are sent, and the particular configuration of the beacon payload being used. By default, to complete most task packages in a single request, the payload limits its data usage to 1 MB of encrypted data per request. Beacon does not make HTTP requests in parallel; instead, it sends one request and waits for its response. If any data that is intended for the team server is beyond the limit, the beacon chunker will divide it into 100-byte chunks (Mudge 2019). Every component sends a separate HTTP request back to the team server. We used a Cobalt Strike payload with a base64 encoded HTTP request header to implement image conversion. The process starts with decoding the payloads into byte format. Since the payload length is very small, the image created will also have a small size. To have an image that can be readily visible, we choose an image dimension of 64 × 64 pixels, an image in a perfect square shape. The input payload will be resized into the next perfect square by applying the zero-padding technique. Then, bytes are converted to their corresponding ASCII values and create a one-dimensional (1D) array. The generated 1D array will then be fed into the Python NumPy (Bressert 2012) Reshape module to be converted to a two-dimensional (2D) array. Finally, OpenCv (Culjak et al. 2012) interpolation transformation is used to resize the images to the model’s desired size. If the image has a smaller size than expected, it will be enlarged. It may also be shrunk if it exceeds the preferred image dimension of 64 × 64 pixels. Figure 7 depicts the process mentioned above. In contrast, Algorithm 1 presents the pseudo-code used to convert incoming network traffic payload to a grayscale image which is fed into the CNN. The network traffic payload is enhanced to fit the length and width requirement of the CNN algorithm using standard interpolation techniques.

Fig. 7

Turning payload into grayscale image

Proposed CNN architecture

In this section, we will elaborate on the CNN architecture used for our model.

A convolutional neural network(CNN) is part of deep learning whose connectivity pattern is inspired by mammals’ visual cortex structure (Hubel and Wiesel 1968). Without any human supervision, CNN is capable of processing an image’s high dimensionality. CNN was originally created for image classification and can accept the raw pixels of an image as input and perform feature extraction. By transforming the input image, CNN can learn about important objects contained within the image. CNN is selected for this malware image classification experiment because of its ability to learn spatial hierarchies of features from images and handle the high dimensional input space.

Traditional neural networks like Support Vector Machines (SVM) and Random Forests are inefficient at processing images as they require feature engineering and assume that each input feature is independent. However, in an image, each pixel is correlated with its neighboring pixels, which results in a high dimensional input space. CNNs solve this problem by using convolutional layers that can extract relevant features from the input image by sliding a filter over it and learning a set of weights. The outputs of these convolutional layers are then passed through pooling layers, which reduce the dimensionality of the output while retaining the important features. Moreover, CNNs can also learn local invariances in translations and rotations. Robustness to variation is essential for image classification tasks where the object of interest can appear in different parts of the image. CNN achieves this by using shared weights for the filters in convolutional layers, which allows the network to recognize the same feature in different parts of the image.

In our experiments, once the payload is transformed into an image, it is processed further using the CNN architecture. The convolutional layer produces feature maps proportional to the input image. As Fig. 8 shows, the proposed CNN model has three convolutional layers. A two-dimensional (2D) convolution layer is followed by a Rectified Linear Unit (ReLU) activation layer, followed by batch normalization and a max pooling layer. CNN-based models are proven to be effective at analyzing images. We opt for Convolutional Neural Networks to classify sessions because they possess the ability to learn directly from raw bytes in network traffic, eliminating the requirement for feature extraction. This is important for detecting fileless malware, as it does not leave traditional artifacts on disk that can be used for detection. By representing the data in an image format, it becomes easier to identify patterns and anomalies that may indicate the presence of malware. We chose a dense neural network to classify malicious and benign sessions based on the features learned by the convolutional layer from images.

Fig. 8

Proposed CNN architecture

During the training phase, to prevent overfitting, we incorporated batch normalization. We further utilized a two-dimensional (2D) Maxpooling layer to downsample the images. Additionally, we applied the Adam algorithm (Kingma and Ba 2014) and tuned various parameters to achieve a true positive rate of over 95% and a false positive rate of less than 0.01%. These included the distribution of benign and malicious sessions in the training dataset, the number of CNN and dense layers, types of optimizers, learning rate, dropout rate, number of filters, size of filters, and the number of training epochs. We closely monitored the loss and accuracy of our model on the validation dataset during the training process to determine the optimal parameters. By utilizing the Adam algorithm, we were able to identify the best hyperparameters that produced the most optimal validation results. We selected a batch size of 32 that proved to work best in our case. We flatten the outputs of the last convolutional module and pass them through a couple of fully connected layers. At the same time, we applied regularization through dropout modules between these layers. We set the dropout probability equal to 0.4.

Table 1 Summary of the developed model

The final layer of our model employed softmax to transform the logits generated by the dense layer into probability distributions. During the training process, the cross-entropy loss function was utilized to minimize errors. The loss value obtained was assessed on a scale ranging from 0 to 1, where a value of 0 indicated a perfect model with no errors. Table 1 shows a summary of the developed model. This CNN model managed a total of 51,993,026 parameters.

Experimental setup and evaluation

Experimental setup

To implement the proposed CNN architecture, we used TensorFlow (Abadi et al. 2016), and Ketkar (2017), which will help the model extract high-level features from the images. Our experiments are executed on Palo Alto private Cloud. The virtual machines (VMs) used for this experiment run Ubuntu 18.04 LTS with 32 GB of RAM with dedicated GPU support.

Dataset

Our experimental analysis is based on the Cobalt Strike beacon payload collected from Palo Alto Networks’ research infrastructure. We acquired a benign dataset of network traffic from an enterprise network, comprising over 3.6 million sessions collected over the span of a week. We labeled the sessions as benign by running them through VirusTotal, and Palo Alto Networks products. Additionally, we obtained a dataset of malicious traffic by simulating command and control communications of Cobalt Strike exploit kits. The Cobalt Strike payloads, referred to as beacons, were created to communicate with its team server using malleable profiles from a public GitHub repository (Mudge 2018), and additional malleable profiles collected by the Palo Alto Networks. In total, we created 6,271 malicious sessions. All the benign and malicious sessions were stored as PCAP files.

Due to skewed class proportions in the dataset, it was challenging for the model to work and generalize well. To mitigate this, we employed resampling techniques by downsampling the benign dataset and upsampling the malicious dataset. Specifically, we used undersampling to reduce the size of the benign dataset by applying deduplication on the hostname and URI (Uniform Resource Identifier) path. We clustered the sessions with the same hostname and URI path in the request header of the first packet in the session and randomly selected one session from each group. After deduplication, we were left with 567,620 sessions in the benign dataset.

To upsample the malicious dataset, we shuffled the arrangement of the header fields in an HTTP packet. This was possible because shuffling the structure of the header fields in HTTP payloads does not change the functionality of the HTTP request. However, from our preliminary experiments on model selection, we observed that the model learned to classify benign and malicious sessions based on the structure of the header fields, resulting in false positives based on the similarity in the header structure. Since each header had more than five header fields, we could generate 120 sessions with different permutations of the header fields. After running a few experiments, we found that upsampling the malicious sessions by a factor of 30 achieved the best results and a benign-to-malicious dataset ratio of 75:25. Following upsampling, the size of our malicious dataset was expanded to 188,130 samples. After model optimization, we performed our testing using new unknown samples to our model to find out testing accuracy. Running our model on a dataset it has never seen before will help our model to avoid overfitting and obtain an unbiased assessment of the performance of our model. We used a total of 865,206 (859,885 benign and 5321 malicious) samples for testing. Table 2 summarizes the total dataset used for training and testing.

Table 2 Summary of dataset

Fig. 9

Training loss versus validation loss

Evaluation

We evaluated the model generated from the training pipeline on the non-overlapping sets of the malicious dataset obtained from the wild to measure the true positive rate of the model. The false positive rate was measured by testing the model in the sessions we collected by crawling the top Alexa websites. It was computationally expensive for us to run multiple iterations of cross-validation, as our model had 51 million parameters and more than 800,000 samples. However, during training and validation, we split our dataset at different random seeds, and we observed consistent results. We eventually fixed the random seed for final model to make our results reproducible. Figure 9 presents how loss variation was optimized. From the graph, training loss and validation loss follow each other, which shows that the model is not overfitted. Furthermore, Fig. 10 illustrates the disparity between the achieved training accuracy and validation accuracy. The model’s overall classification accuracy is 99.48%. Table 3 presents the training and validation accuracies attained by the model across 10 epochs.

Fig. 10

Training accuracy versus validation accuracy

Table 3 Training and validation accuracy for 10 epochs

Performance metrics in our experimentation include: i) True Positive (TP) to present the malicious data which is correctly classified as malicious), ii) True Negative (TN) to manifest the benign data which is correctly classified as benign), iii) False Negative (FN) to show the malicious data which is classified as benign, and iv) False Positive (FP) to express benign data which has been classified as malicious.

A confusion matrix is known to work well with binary classification. Therefore, the true positive rate (TPR), which describes how accurately the model classifies, can also be described in terms of a confusion matrix. The true positive rate (TPR) can be calculated using the confusion matrix as shown in Eq. (1):

$$\begin{aligned} TPR = \frac{TP}{TP + FN} \end{aligned}$$

(1)

Furthermore, the False Negative Rate (FNR) shows an incorrect negative classification and is defined in Eq. (2):

$$\begin{aligned} FNR = \frac{FN}{TP + FN} \end{aligned}$$

(2)

On the other hand, the True Negative Rate (TNR) deduces the correct negative classification and is defined in Eq. (3):

$$\begin{aligned} TNR = \frac{TN}{TN + FP} \end{aligned}$$

(3)

Finally, False Positive Rate (FPR) represents the ratio of incorrect classifications and is defined in Eq. (4):

$$\begin{aligned} FPR = \frac{FP}{TN + FP} \end{aligned}$$

(4)

It is also noteworthy that we define our accuracy, precision, recall, and F1 score metric in terms of a confusion matrix. The four metrics are defined in Eqs. (5), (6), (7), and (8) respectively.

$$\begin{aligned} Accuracy= & {} \frac{TP +TN}{TP +TN + FP + FN} \end{aligned}$$

(5)

$$\begin{aligned} Precision= & {} \frac{TP}{TP + FP } \end{aligned}$$

(6)

$$\begin{aligned} Recall= & {} \frac{TP}{TP + FN} \end{aligned}$$

(7)

$$\begin{aligned} F1-score= & {} \frac{2*(Precision*Recall)}{Precision + Recall} \end{aligned}$$

(8)

Accuracy serves as a measure of the overall correctness of a model’s predictions, whereas precision rate gauges the model’s capability to accurately identify positive samples. Recall, on the other hand, signifies the model’s proficiency in correctly identifying all positive samples. F1-score represents a harmonious balance between precision and recall, indicating an effective trade-off between the two metrics.

Table 4 Evaluation metrics results

Discussion

The integration of machine learning and image visualization for fileless malware classification is a relatively recent field. Table 4 presents the detection rates achieved by our model on a test dataset. We collected a total of 5321 malicious samples from four different sources (Wildfire, APTs in the wild, Palo Alto Customers, and Public profiles of Cobalt Strike). As for the benign samples, we utilized the Top Alexa website request headers dataset, consisting of 859,885 benign samples. The model successfully identified and classified 4916 malicious samples as malicious, resulting in an overall average detection rate of 92.38%. Table 3 demonstrates that our approach exhibits improving training and validation accuracy with each epoch, ultimately achieving an average accuracy of 99.4% and 99.5% for the training and validation phases, respectively.

In addition, we conducted a comparison of our classification accuracy with that of existing approaches (i.e., van der Eijk and Schuijt 2020; Khalid et al. 2023), as shown in Table 5. Our experiments involved 193,451 fileless malware samples and resulted in an accuracy of 99.48%, precision of 80.2%, recall of 92.38%, and F1-score of 85.86%. As presented in Table 5, the research conducted by van der Eijk and Schuijt (2020) employed a dataset comprising only 17 Cobalt Strike beacons and achieved an accuracy of 99.99%. The precision, recall, and F1-score were reported as 75%, 88.2%, and 81% respectively. It is worth noting that van der Eijk and Schuijt (2020) utilized a deep packet inspection based static algorithm to detect Cobalt Strike beacons in network traffic. Furthermore, a small number of non-real-world malware samples were used in their experiment to identify features. In contrast, Khalid et al. (2023) used 26 fileless malware samples and achieved an accuracy of 93.3%, with a true positive rate (TPR) of 87.5% and a false positive rate (FPR) of 0%. Precision, recall, and F1-score were not reported in Khalid et al. (2023). What sets our study apart is the utilization of a substantial real-world dataset comprising fileless malware samples. This approach enhances the reliability and generalizability of our findings.

Table 5 Comparison of evaluation metrics with related works

Table 6 Comparison of this work with prior image visualization based malware classification techniques

Table 6 provides a comparison between our approach and previous literature that incorporates both machine learning and image-based visualization. It is important to note that all the works presented in Table 6 focus on file-based malware. The table highlights that while prior techniques may achieve high accuracy, they often come with significant computational costs due to computationally intensive image pre-processing steps.

Limitations and challenges

One of the main challenges in this study stems from the limited availability of samples. There is a notable imbalance between the number of benign and malicious instances collected, which presents obstacles to achieving optimal performance and generalization of the model. Furthermore, obtaining labeled fileless malware samples for training purposes is challenging due to their scarce availability. To tackle this issue, we utilized resampling techniques, specifically downsampling the benign dataset and upsampling the malicious dataset, to mitigate the imbalanced class proportions.

While convolutional neural networks (CNNs) have shown effectiveness in various computer vision tasks, their application in classifying network traffic and detecting malware may face certain limitations. Unlike images, network traffic data is sequential and requires capturing temporal dependencies and contextual information. CNNs, on their own, may not be able to adequately model and utilize such dependencies, potentially impacting their effectiveness in these tasks. Another limitation is CNNs need fixed-size inputs, which is difficult to achieve with network traffic data. Network packets vary in length, and different protocols have distinct structures. Preprocessing data to fit fixed-size inputs may cause information loss or distortion.

Conclusion

Previous studies in the field of malware detection and classification primarily focused on file-based malware and utilized image visualization techniques. However, this study represents one of the early works specifically concentrating on fileless malware detection using image-based visualization and Convolutional Neural Networks (CNNs). Fileless malware exploits vulnerabilities in legitimate applications, making it difficult for conventional file-based methods to detect them. In this research, we conducted experimental research using evasive and realistic offensive traffic generated by proprietary Cobalt Strike beacons at the production level. By converting Cobalt Strike beacon payloads into grayscale images and training our proposed CNN model on them, we achieved a high level of accuracy. The outcome demonstrates that our approach effectively detects traces of fileless malware in network traffic.

Moving forward, our future studies will focus on innovative image enhancements aimed at enhancing the local contrast within regions of images that represent fileless malware traffic. This approach will enable us to identify similar pixel values, improve the interoperability of fileless malware images, and provide better input for the machine-learning classifier. Additionally, we plan to explore different image conversion algorithms, such as converting payloads into bitmaps and generating RGB-colored 3D images, in order to investigate their impact on the accuracy of machine learning-based traffic classification.