Detecting Malicious Java Code Using Virtual Machine ckrintz/papers/ Malicious Java Code Using Virtual Machine Auditing Sunil Soman Chandra Krintz Giovanni Vigna Computer Science Department

Embed Size (px)

Text of Detecting Malicious Java Code Using Virtual Machine ckrintz/papers/ Malicious Java Code Using...

  • Detecting Malicious Java Code Using Virtual Machine Auditing

    Sunil Soman Chandra Krintz Giovanni Vigna

    Computer Science DepartmentUniversity of California, Santa Barbara{sunils,ckrintz,vigna}


    The Java Virtual Machine (JVM) is evolving as aninfrastructure for the efficient execution of large-scale,network-based applications. To enable secure execu-tion in this environment, industrial and academic effortshave implemented extensive support for verification oftype-safety, authentication, and access control. However,JVMs continue to lack intrinsic support for intrusion de-tection.

    Existing operating system auditing facilities and host-based intrusion detection systems operate at the processlevel, with the assumption that one application is mappedonto one process. However, in many cases, multiple Javaapplications are executed concurrently as threads withina single JVM process. As such, it is difficult to analyzethe behavior of Java applications using the correspond-ing OS-level audit trail. In addition, the malicious ac-tions of a single Java application may trigger a responsethat disables an entire execution environment. To over-come these limitations, we have developed a thread-levelauditing facility for the Java Virtual Machine and an in-trusion detection tool that uses audit data generated bythis facility to detect attacks by malicious Java code. Thispaper describes the JVM auditing mechanisms, the in-trusion detection tool, and the quantitative evaluation oftheir performance.

    1 Introduction

    Java technology [18] was initially used by web page de-signers to embed active content. As a result of the wide-spread success and popularity of Java, developers now

    use the language for implementation of a wide rangeof large-scale systems, e.g., robust, mobile code sys-tems [36, 19, 33] and complex server systems [43, 32].In these systems, multiple applications and code compo-nents are uploaded by multiple (possibly untrusted) usersfor concurrent execution within a single Java Virtual Ma-chine (JVM) [38]. The portability, flexibility, and secu-rity features of Java make it an ideal technology for theimplementation of systems that support the execution ofmobile code.

    The use of Java enables portability because programsare encoded using an architecture-independent transferformat that can be executed without modification on anyplatform for which a JVM has been developed. Javasupports flexibility by allowing applications to be up-graded and extended at run time through the use of dy-namic code loading. In addition, the Java type systemand verification mechanisms provide protection againstsome programming errors and malicious attacks.

    The key areas that must be improved to enable con-tinued wide-spread use of Java for Internet-scale serverapplications are performance and security. Recent ad-vances in Just-In-Time (JIT) compilation and optimiza-tion techniques [35, 6, 8] offer significant improvementsin Java program performance. Currently, to support se-curity in server applications, Java provides mechanismsfor authentication and access control [17, 2]. However,additional mechanisms are required to detect attacks thatcircumvent (or attempt to circumvent) the existing pro-tections or abuse legitimate access.

    Host-based Intrusion Detection Systems (HIDSs) pro-vide a suitable solution to this problem [3, 12]. Ex-isting HIDSs use process-level execution events, col-lected by an auditing facility in the operating system,

  • to identify and respond to security threats and viola-tions [13, 26, 52].

    Unfortunately, since a JVM executes as a single,multi-threaded user process, we must presume that allsuspicious activity reported by the auditing facility forthe JVM process ID, is caused by the JVM itself. How-ever, the culprit may be one of the many applications run-ning within the JVM process. As users increasingly de-mand higher availability and reliability from the serversexecuting their applications, an intrusion response mech-anism that simply terminates the JVM process that is ex-ecuting malicious code becomes unacceptable. There-fore, novel auditing and intrusion detection techniquesare needed, to provide finer-grained threat identificationand response.

    To this end, we have developed a JVM auditing facil-ity and an intrusion detection system that detects attacksthat are initiated by malicious Java code. To implementour system, We leveraged two existing technologies: ahigh-performance, open-source, Java Virtual Machine,called JikesRVM [1]; and an intrusion detection frame-work, called STAT, that we developed in prior work [50].In this paper, we describe the JVM auditing mechanismsand the intrusion detection tool that we implemented, andprovide a quantitative evaluation of the performance ofour system.

    In the next section, we place our work in the context ofexisting approaches to intrusion detection. In Section 3,we present an overview of our approach. In Section 4, wedescribe the JVM auditing system. Then, in Section 5,we present the intrusion detection system. Section 6 dis-cusses the experimental evaluation of our system. Sec-tion 7 presents related work on Java security. Finally,Section 8 draws conclusions and outlines future work.

    2 Extant Approachesto Intrusion Detection

    Intrusion detection is performed by identifying the man-ifestation of a security violation in an input event stream.In host-based intrusion detection systems, the input eventstream is usually represented by the audit records pro-duced by the auditing facility of an operating system,such as the Solaris Basic Security Module [47]. Otherpossible input streams are system call traces and UNIXsyslog messages.

    The detection process can be performed according todifferent techniques. For example, it is possible to usestatistical measures to characterize the normal behaviorof users and applications [13, 23, 29]. Deviations fromthe established profiles are assumed to be evidence of anattack. The problem with these approaches is the diffi-culty to create a reliable model of the application behav-ior. An imprecise model may lead to both missed de-tections (called false negatives) and false alarms (calledfalse positives).

    Other approaches rely on the formal specification ofthe acceptable behavior of an application [34, 54, 52].An execution history that does not conform to this be-havior is considered malicious. The advantage of this ap-proach is that the generation of false positives is greatlyreduced. On the other hand, the generation of the specifi-cation requires access to the application source code andconsiderable effort, even when supported by tools. Forthis reason, these techniques are not in wide-spread use.

    The most commonly used approach relies on mod-els of attacks to analyze the input event stream. Sys-tems based on this approach [20, 26, 39] are equippedwith a number of attack signatures. These signatures arematched against the stream of audit data to identify themanifestation of an attack. The advantage of this ap-proach is that it supports very effective intrusion detec-tion and produces few false positives. In addition, signa-tures are not limited to modeling attacks against an ap-plication. For example, it is possible to describe attacksthat represent abuses of legitimate access to the system.The disadvantage is that the signature set must be up-dated continuously as new ways of attacking a systemare found.

    In the work we present herein, we describe the de-sign and implementation of a signature-based intrusiondetection system. The system is an extension of previouswork [49] in which we developed an auditing facility andan intrusion detection system for a mobile agent system,called Aglets [36]. This prior system detects maliciousagent activity through the monitoring of the Aglets exe-cution environment. In that case, the Java server applica-tion that was responsible for transferring and executingthe mobile agents was instrumented to produce agent-related information.

    The development of this agent monitoring system sug-gested a far more general approach in which the JVMitself is extended to produce the necessary informa-

  • !

    Figure 1: Architecture of the Java Virtual Machine auditing system and the STAT-based intrusion detection system

    tion. Therefore, we developed a mechanism that collectsevents that give information about the activity of threadswithin a JVM. The resulting auditing system can moni-tor the activity of any Java application, including varioustechnologies supporting mobile code. We also developedan intrusion detection system that takes advantage of thefiner-grained information produced by the JVM audit-ing system to detect attacks coming from malicious Javacode.

    To the best of our knowledge, no existing system per-forms auditing and threat detection at the Java threadlevel.

    3 System Overview

    Figure 1 shows a high-level overview of our intrusiondetection architecture. The auditing system monitorsexecuting Java threads (possibly associated with multi-ple, independent applications) and produces an audit logcomposed of records related to thread activity. The log isconverted to an event stream by an event provider. Theevent stream is subsequently analyzed by an intrusiondetection system for possible security threats or attacks.More precisely, the intrusion system compares patternsin the event stream against known attack scenarios. A

    match indicates that an intrusion or threat is in progressand that a response should be initiated. To do so, theintrusion detection system contacts the JVM which im-mediately terminates all offending threads. The JVM au-diting system maps authenticated user IDs to threads sothat malicious users can be identified and their threadsselectively terminated.

    Our system implementation couples and extends twoex