Security-first architecture: deploying physically isolated active security processors for safeguarding the future of computing

Software vulnerabilities are inevitable

Software security vulnerabilities exist because software systems are too complicated. During the past two decades, software has been increasingly woven into the human society, and the scale and complexity of software systems have been increasing significantly. The complexity of most applications goes far beyond the understanding of a single person. For example, the Firefox browser contains 36 million lines of code committed by 6313 contributors over ten years (Mozilla Firefox 2018). Complex code would no wonder have a higher chance of having security vulnerabilities exploitable by attacks (Shin et al. 2011). As mentioned in the June 2017 update, Mozilla fixed a total of 32 bugs of the Firefox browser, among which the most dangerous is a use-after-free vulnerability, CVE-2017-5472, using a destroyed node when regenerating trees. This vulnerability is obviously a severe security vulnerability resulted at least partially from code complexity. Another representative instance is the Linux operating system. The size of Linux kernel is increasing and the function is more and more complicated. As shown in Fig. 1 (Common Vulnerabilities and Exposures 2018), in 2017 the number of newly exposed Linux kernel vulnerabilities hit a historical high of 453, more than twice as many as in 2016. Overflow and memory corruption are classic security vulnerabilities and should have been avoided as much as possible. However, in 2017 the yearly numbers of CVE exposed Linux kernel vulnerabilities of the two types are still the highest, reaching 51 and 26, respectively. Due to great complexity, many critical software systems are extremely difficult to test. Also, the complexity in software has reached beyond the capability of existing verification methodologies. Without whole-system verification, current software systems have no proven security. As the complexity of modern software systems increases continuously, software vulnerabilities are inevitable.

Hardware components are insecure

Similar to the inevitable software vulnerabilities, hardware vulnerabilities broadly exist as well. Unsafe hardware architecture and complex production processes are two main factors of hardware vulnerabilities.

(1) Unsafe hardware architecture

Security risks widely exist in performance-oriented architecture design. In such design, inherent security risks are associated with branch prediction, cache, instruction prefetch and so on. These security risks are well reflected in the following two examples. First, cache is often the target of attacks. Exploiting the time difference between cache miss and hit, side channel attacks (Liu et al. 2015) can successfully speculate code paths. Private keys can be extracted from a code path because the code paths of an encrypted program are deterministic. Second, out-of-order and speculative execution are two fundamental optimization techniques for improving chip-level performance, but from the security viewpoint they can also be viewed as two security vulnerabilities. The recently disclosed Meltdown (Lipp et al. 2018) and Spectre (Kocher et al. 2018) attacks respectively exploited these two vulnerabilities to compromize computers.

(2) Complex production processes

Modern processor chips are produced by complex manufacturing processes. Due to the complexity, product security cannot be guaranteed. To make sure that the cost is not too high, modern processor chips generally adopt the OEM (Original Equipment Manufacturer) production mode. In this mode, design, layout, manufacturing, packaging and testing are in general separated; each process has different vendors. The design process frequently uses third party IP, standard cell, and a variety of EDA (Electronics Design Automation) tools. Due to globalization, a product often needs to “travel” from one country to another. Figure 2 illustrates the production process of CPU chips and highlights the potential hazards in relevant stages. These uncontrollable factors are likely to introduce additional security risks. On one hand, it is possible for disgruntled or malicious chip designers to implant malicious logic or circuits without being noticed. On the other hand, foundries also have the opportunities to deploy malicious circuits on the chip layout without even modifying the design logic, as shown in the A2 attack (Yang et al. 2016). Finally, real-world manufacturing processes have very limited ability to detect malicious circuits; and very limited ability to verify that circuits are free of malice. This also aggravates the aforementioned hardware security problem.

Attacks are becoming more sophisticated

The complexity of attacks is mainly shown in two aspects:

(1) Width of attack range. Attacks can be launched from any places, including all layers from application software to physical hardware. Many popular attacks are mainly at the software level, such as SQL injection (Halfond et al. 2006) and CSRF (Cross-site request forgery) (Lin et al. 2009). In addition, ARP (Address Resolution Protocol) Spoofing (Whalen 2001), session hijacking and others indicate that the network protocols can be attacked. Furthermore, various vulnerabilities also show that hackers are able to launch hardware-level attacks, such as Rowhammer (Seaborn and Dullien 2015; Kim et al. 2014), Meltdown (Lipp et al. 2018) and Spectre (Kocher et al. 2018).

(2) Variety of attack means. The simple means of attack rely on unintentional yet harmful operations, such as downloading a program bundled with a Trojan. Next, worms and botnets are designed to self-replicate and spread. These attack means automatically exploit software vulnerabilities. The exploitation of software vulnerabilities has also evolved from code injection attacks to code reuse attacks such as ROP (Return-oriented programming) (Checkoway et al. 2010) and JOP (Jump-oriented programming) (Bletsch et al. 2011). Shifting the target from software to hardware, many side channel attacks exploit hardware vulnerability.

Fundamental issues

Building a secure system atop the current computer architecture is fundamentally difficult. The fast-growing complexity in software inevitably leads to exploitable security vulnerabilities. IP (Intellectural Property) reuse and the separated ASIC manufacture procedure with multiple vendors make it impossible to thoroughly inspect the whole design and the manufacture procedures. Some architecture-level vulnerabilities are difficult to remove due to the conflicting interests between performance and security. In the meanwhile, attacks are constantly evolving with new means to outwit existing defenses. To meet this challenge, this paper proposes security-priority architecture, a concept which separates the security tasks from the normal tasks by physical isolation. The architecture can remedy the deficiencies of existing security mechanisms and provide a new direction worth exploring.

Existing architecture-level defenses are usually resulted from passively responding to specific attacks

Several micro-architecture level defense mechanisms have been integrated into commercial chips. Here, we discuss three representative types of architecture-level defenses.

Memory overflow defense: Memory corruption bugs enable attackers to maliciously change a program’s behavior (van der Veen et al. 2012; Szekeres et al. 2013). Applications written in low-level languages like C/C++ are prone to these kinds of bugs due to the lack of memory safety. Since it is difficult to figure out all potential memory overflow bugs, an efficient way to enforce memory safety is to add extra hardware protection mechanisms. For example, Intel recently released a new ISA extension, named as Memory Protection Extensions (MPX) (Oleksenko et al. 2017). To facilitate memory safety checking, application developers are allowed to insert boundary record instructions to store the boundaries of protected memory regions, like arrays, at the places where these data structures are defined, and insert boundary check instructions to ensure that the access to these data structures are inside their valid ranges. Unfortunately, MPX suffers from high performance overhead which limits its adoption.

Pointer integrity defense: The Pointer Authentication (PA) mechanism is added to ARMv8.3-A to prevent memory corruption attacks (Qualcomm Technologies 2017). PA guarantees the integrity of pointers by binding each pointer with a Pointer Authentication Code (PAC). As the actual address space in 64-bit architectures is less than 64 bits, PA places the PAC to the unused bits in the pointer value to minimize the size and performance impact. PACs are computed by a lightweight cryptography algorithm (Avanzi 2017) and added to the pointer values by extended PAC instructions. The integrity of pointers is verified and restored by the AUT instructions. To restrict the accesses to pointers in special context, PA allocates keys for instruction pointers, data pointers and general-purpose instructions. The keys are still managed by software.

Control-flow integrity defense: Control-flow integrity (CFI) (Abadi et al. 2005; Davi 2015; Burow et al. 2017) is considered as one general and promising defense against code-reuse attacks (Shacham 2007; Bletsch et al. 2011; Carlini and Wagner 2014; Schuster et al. 2015). CFI restricts the control-flow of an application program to valid execution traces; and the program’s predefined Control-Flow Graph (CFG) tells what is valid. Forward-edge control-flow represents transfers caused by indirect jumps and function calls. Backward-edge control-flow represents transfers caused by function return instructions. Intel has an ISA-level CFI extension, named as Control-flow Enforcement Technology (CET) (Intel Corporation 2016), which protects forward-edge CFI by indirect branch tracking and ensures backward-edge CFI by hardware shadow stacks. However, CET suffers from two problems. One is the difficulty in defining a complete and precise legal CFG (Evans et al. 2015), because some informaion is only available at runtime, such as the target of an indirect jump. The other is the limited size of the hardware shadow stack. The stack has to rely on the operating system (OS) for handling context switches and deeply nested function calls (Frantzen and Shuey 2001).

In summary, a common characteristic of three representative types of architecture-level defenses is that they seek to protect the chip or system from a specific vulnerability. In other words, existing architecture-level defenses are usually resulted from passively responding to specific attacks. As a consequence, such patch-like approaches are usually not generic and cannot handle attacks which exploit zero-day vulnerabilities.

Trusted computing alone is not secure

Trusted computing ensures that executable binaries are not tampered (David et al. 2008; Trusted Computing 2008). The foundation for trusted computing relies on a dedicated chip, which is defined as a Trusted Platform Module (TPM), to measure the integrity of executable binaries by verifying their hash values. To implement this measuring idea, the TPM is designed as a coprocessor that helps the platform software to verify itself using a predefined sequence as well as a cryptographic engine that accelerates encryption, digital signatures, and hashing.

The TPM was initially used as the static root of trust for measurement (SRTM) (Trusted Computing Group 2003). SRTM utilizes the TPM to verify the integrity of booting processes. If a chain of trust is established during a booting process, the boundary of trust can be extended to include more than one level of software within the system. To set the system into a clean state without rebooting, researchers propose dynamic root of trust for measurement (DRTM). Two DRTM implementations are as follows. Intel develops the Trusted Execution Technology (TXT) (Intel Corporation 2006) to securely launch software (such as the hypervisor and security kernel) at arbitrary time. AMD offers similar capabilities with its Secure Virtual Machine (SVM) extensions.

However, the security requirements are far beyond guaranteeing the integrity of executable binaries. Even if the executable binaries are unchanged, there are still security risks as the control flow integrity (CFI) and data flow integrity (DFI) can be exploited by malware. Dedicated mechanisms implemented in processor chips could be very helpful in defending such kind of advanced attacks.

Logical isolation suffers from information leak through physical side-channels

ARM TrustZone (Wojtczuk and Rutkowska 2017) and Intel SGX (McKeen et al. 2013) are similar technologies which adopt logical isolation to provide a trusted execution environment for security-sensitive data and code. For performance reasons, secure and non-secure programs (or worlds) still share substantial physical hardware resources, including the on-chip cache hierarchy, TLB and others. This sharing leads to the risk of side-channel information leakage.

ARM TrustZone adds a NS bit in the memory system to divide the processor into two worlds: a normal world and an isolated secure world. The non-secure world cannot directly access the resources used by the secure world (ARM Limited 2009). The two worlds communicate with each other through a security monitor. To improve the system performance, caches are not flushed during world switches. However, this allows cache lines from the secure world to be evicted by the cache lines from the normal world and vice versa. Such evictions can be exploited as a cache side-channel to leak security-sensitive information. Exploiting the cache side-channel by evicting the secure cache lines cached in the shared cache belonging to the normal world, a prime+probe attack (Zhang et al. 2016) was able to infer the full AES128 secret key in 2.5 seconds from the normal world kernel or 14 minutes from a user space Android application. Allowing non-secure and secure cache lines to co-exist in caches may also result in cache incoherence behavior. This incoherence can be exploited to install rootkit, which evades the memory introspection mechanisms (Zhang et al. 2016).

Intel SGX is designed to increase the security of software through an “inverse sandbox” mechanism. In this approach, rather than attempting to identify and isolate all the malware on the platform, legitimate software can be sealed inside an enclave and protected from attacks by the malware, irrespective of the privilege level of the latter. In other words, this allows an application to create a secure enclave at the CPU level which is protected from the OS upon which it is running. Data inside an enclave is allowed to be accessed only by the codes located in the same enclave. Moreover, the content of an enclave is encrypted when stored in the memory. Even if an attacker has the ability to snoop the memory bus, she cannot get any useful information. Although the OS cannot directly access the memory region used by an enclave, to simplify the deployment and improve the performance, SGX still leaves the OS in charge of setting up the page tables used by enclaves. By observing an application’s page faults and page table attributes, a malicious OS can infer part of the memory access pattern of the application. By hacking the page fault handler of the OS, attackers can track the access patterns of the enclave at page size granularity. Xu et al. (2015) implemented a controlled-channel attack on SGX-enabled platforms to reveal the input-dependent control transfers and data accesses of the target program. A similar but stealthier attack can be launched utilizing the access bit inside the page table entry (Wang et al. 2017). Leveraging the contention in memory resources including caches and TLBs etc., Wang et al. implemented a traditional prime+probe attack which is more fine-grained and powerful on SGX-enabled platforms.

Current architecture-level security subsystems are not really secure

Most of current server chips include a subsystem dedicated for system management. The subsystem is basically a tiny computer-within-a-computer normally embedded directly in the chip along with the host processor cores. Typical examples of such subsystems include Intel Management Engine (ME) (Datenschutz and Pataky 2017; Bogowitz and Swinford 2004), AMD Platform Security Processor (PSP) (Advanced Micro Devices 2018b; Wikimedia Foundation 2018) and Power On Chip Controller (OCC) (Sinharoy et al. 2015). The functions of these subsystems include managing the boot process, keeping the system under thermal limits, and running based on user input modes and parameters. The subsystems can also be used to monitor the system for identifying suspicious activities or events, and make appropriate responses. As security subsystems usually contain a rich set of hardware-based cryptographic primitives, they process security related functions, such as secure boot, secure updates, secure debug and secure communication, significantly faster than software-only solutions.

However, current designs (of these subsystems) need improvement in terms of performance and security. Firstly, complicated logic is expected to run on these subsystems to detect and handle potential malicious behaviors, even unknown attacks. These security inspection workloads can have strong hardware performance requirements, but popular designs use tiny embeded processor cores, such as ARM M series processors. Hence, there is a performance gap between the required workloads and the actual computation power. Secondly, the security of such a subsystem itself is a concern. As the subsystem has accessves to everything, it might provide an attacker with a powerful backdoor if compromised. The subsystem consists of layers of quite complex software. For example, the Intel ME runs a hidden MINIX OS (Datenschutz and Pataky 2017; Bogowitz and Swinford 2004). Various software components also run on the ME, such as the Intel Active Management Technology (AMT) (Bogowitz and Swinford 2004). On other hand, the complexity provides an attacker with a good chance to compromise the subsystem. On the other hand, the subsystem is implemented with embedded processors (Sinharoy et al. 2015; Datenschutz and Pataky 2017; Advanced Micro Devices 2018b) which usually lack the common security hardening such as stack cookies, No-eXecute (NX) flags, or address space layout randomization (ASLR). Hence, there is also a security gap between the security requirements and the current security hardening. It has been reported that the vulnerabilities in Intel ME and AMD PSP allow attackers to gain administrative capabilities (Intel Security 2017; Advanced Micro Devices 2018a).

Architecture overview

According to the analysis in “Weaknesses of current countermeasures” section, the design of security mechanism has the trend which is evolving from passive to active. For example, existing architecture-level defenses are usually resulted from passively responding to specific attacks. And the techniques like trust computing and SGX/trustzone are of much more active defenses, which provide generic secure execution check or environment instead of targeting at specific attacks. However, these active defence mechanisms still suffer from some problems. Firstly, since security is usually not the first priority design goal, logical isolation is preferred than physical isolation, which causes side-channel information leakages. And secondly, even for independent secure environment like AMD PSP or Intel ME, there is a performance gap between the required security workloads and the actual computation power. More seriously, existing active mechanisms usually need to be invoked by other compute units such as CPU. Therefore, once the computer has been compromised, these active security mechanisms are possibly evaded.

Figure 3 illustrates our proposed security-first architecture. In existing commercial systems, tasks in different security modes or with different privilege levels are being performed consuming one set of physical resources, including processor cores and shared cache. This in principle would result in side-channel information leakages. As a comparison, security-first architecture explicitly differentiates the role of a processor in the same node into two types: conventional processors named as Computation Processors (CP), and newly designed “dedicated to security” processors named as Active Security Processors (ASP). CPs are monitored and protected by these ASPs. The two types of processors execute on their own resources, including memory, disks, network and I/O devices. The ASPs are provided with dedicated channels to access all the resources of the CPs but not vice versa. Such uni-directional information flows lead to physical isolation for computation and security systems. The functionalities of computation and security are physically decoupled. Most of the security tasks or functions shall be assigned to ASP, and all other tasks are still kept on CP as what commercial systems are currently doing. Due to physical isolation, the likelihood to suffer from side-channel attacks will be significantly reduced for the ASP system.

Active Security Processor (ASP)

A physically isolated ASP is a key component for security-first architecture. On one hand, the security of security tasks and mechanisms is guaranteed due to physical isolation. On the other hand, the performance impacts on the execution of computational workloads are minimum due to the transfer the security tasks from CPs to ASPs. This section introduces the detailed ASP design.

Another architecture-level feature is the asymmetric memory space, which enables the physical isolation between the Computation domain and the Security domain. CP and ASP have their own memory, chip sets and I/O devices. And the inter-domain communications are unidirectional. That is to say, ASP can access all resources including the internal status of CP, the memory and I/O devices in the Computation domain, while CP cannot access any information within the Security domain.

At micro-architecture level, security-first architecture adopts the classic general-purpose processor hardware optimizations including Out-of-Order execution and speculative technique to meet the performance and flexibility requirements of diverse security tasks. Besides these, unique hardware optimizations like heterogeneous computing are also considered in the micro-architecture design of ASP. For instance, typical operations like encryption/de-encryption, and pattern matching are selected as acceleration targets via domain-specific hardware accelerators.