Security Functions for Virtual Machines via Introspection

Abstract

The recent renaissance of virtualization brought with it the resurgence of ideas for hypervisor based security services. As such, virtual machine introspection (VMI) has been proposed for both passive and active monitoring. While passive monitoring is the method for detecting intrusions, active monitoring allows intervention of a Virtual Machine (VM) behavior, which is proper for intrusion prevention. Several VMI techniques for security purposes have been deployed in different virtualization solutions. XenProbes, XenAccess, and Ether are examples of deployed VMI for Xen. The goal of this thesis is the design and the implementation of a security function that actively monitors the integrity aspect of guest virtual machines. OS debugging is the method used for active VMI. In this method, Xen built-in capability for OS debugging is used, to control, and to intervene in the behavior of guest virtual machines. A well-known drawback of VMI in "high-rate" applications is the cost of context switches between the trusted monitor and the virtual machine being monitored. As a result, low-rate security functions are probably more suitable candidates for VMI applications. The proposed security functions are low-rate solutions for systems integrity property. In the attempt to define proper low-rate security functions different available filesystem integrity solutions like DigSig and IMA are surveyed. As DigSig is limited to ELF files and IMA is not developed completely and is not immune against rootkits, a new security function is developed in this thesis. In this process, IMA is used as the basis of the designed security function. The security function validates the RSA signature of accessed files in guest virtual machines. It prevents file access in case of violation. This security function starts early in the boot process of a guest VM to properly ensure its integrity property. Having implemented the security function, its security strength, performance, and limitations are analyzed. Finally it is concluded, while this security function imposes negligible performance penalty, it improves the security attributes of a virtual machine.