Automated Experimentation System - Malware Analysis System · Automated Experimentation System – Malware Analysis System . User and Administrator Guide. Christopher McKenzie . AEPOS

Automated Experimentation System – Malware Analysis System User and Administrator Guide

Christopher McKenzie AEPOS Technologies Corporation Prepared By: Christopher McKenzie AEPOS Technologies Corporation 116 Albert St., Suite 600 Ottawa, Ontario K1P 5G3 Contract Project Manager: Jonathan Risto, (613) 990-6015 PWGSC Contract Number: W7714-115223 CSA: Marc Grégoire, 613-998-2113 The scientific or technical validity of this Contractor Report is entirely the responsibility of the contractor and the contents do not necessarily have the approval or endorsement of Defence R&D Canada. .

Defence R&D Canada – Ottawa Contract Report

DRDC Ottawa CR 2013-101 November 2013

Automated Experimentation System - Malware Analysis System User and Administrator Guide

Christopher McKenzie AEPOS Technologies Corporation Prepared By: Christopher McKenzie AEPOS Technologies Corporation 116 Albert St., Suite 600 Ottawa, Ontario K1P 5G3 Contract Project Manager: Jonathan Risto, (613) 990-6015 PWGSC Contract Number: W7714-115223 CSA: Marc Gregoire, 613-998-2113 The scientific or technical validity of this Contractor Report is entirely the responsibility of the contractor and the contents do not necessarily have the approval or endorsement of Defence R&D Canada.

Defence R&D Canada – Ottawa Contract Report DRDC Ottawa CR 2013-101 November 2013

Contract Scientific Authority

Original signed by Marc Grégoire

Marc Grégoire

Group Leader

Approved by

Original signed by Peter Mason

Peter Mason

Acting Head / COSW Section

Approved for release by

Original signed by Chris McMillan

Chris McMillan

Head / Document Review Panel

© Her Majesty the Queen in Right of Canada, as represented by the Minister of National Defence, 2013

© Sa Majesté la Reine (en droit du Canada), telle que représentée par le ministre de la Défense nationale, 2013

DRDC Ottawa CR 2013-101 i

Abstract ……..

This is the user and administrator guide for the Automated Experimentation System (AES) - Malware Analysis System. It is intended to cover end-user operation of AES for malware analysis, and administrator configuration, system design, and troubleshooting.

Résumé ….....

Ce document est le manuel de l’utilisateur et du gestionnaire pour le Système d’Expérimentation Automatisé (SEA) – Système d’Analyse de Logiciel Malveillant. Les sujets couverts sont le fonctionnement pour l’utilisateur, la configuration pour le gestionnaire du système, la conception du système et le dépannage.

ii DRDC Ottawa CR 2013-101

This page intentionally left blank.

DRDC Ottawa CR 2013-101 iii

Executive summary

Automated Experimentation System - Malware Analysis System: User and Administrator Guide

Christopher McKenzie; DRDC Ottawa CR 2013-101; Defence R&D Canada – Ottawa; November 2013.

Introduction or background: The Automated Experimentation System (AES) is a software library and virtual testing environment designed by Communications Research Centre Canada (CRCC) in order to conduct network security experimentations. AES for malware analysis is used to conduct experiments on malware samples within an isolated virtual network. The virtual network provides a simulated environment where malware can be executed and observed for analysis. The isolated network is comprised of multiple VMware ESX hosted virtual machines.

This document is intended as a user guide for malware experiment users and as an administrator guide for administrators responsible for the configuration and maintenance of the AES infrastructure.

Significance: The AES malware analysis system provides both a research and an operational capability.

iv DRDC Ottawa CR 2013-101

Sommaire .....


Christopher McKenzie; DRDC Ottawa CR 2013-101; R & D pour la défence Canada –Ottawa; November 2013.

Introduction ou contexte : Le Système d’Expérimentation Automatisé (SEA) est une bibliothèque de logiciels et un environnement d’essai virtuel conçu par le Centre de recherches sur les communications Canada dans le but de procéder à des expériences dans le domaine de la sécurité informatique. Cette version du SEA est dédiée à l’analyse de logiciels malveillants. Le système est utilisé pour procéder à des expériences avec des échantillons de logiciels malveillants dans un environnement virtuel isolé. Un réseau virtuel est créé, offrant ainsi un environnement simulé où le logiciel malveillant peut-être exécuté de façon sécuritaire et dont le comportement est observé à des fins d’analyse. Le réseau isolé créé est constitué d’une multitude de machines virtuelles avec un hôte VMware ESX.

Importance : Le Système d’Expérimentation Automatisé (SEA) – Système d’Analyse de Logiciel Malveillant offre à la fois un outil de recherche et une capacité opérationnelle.

DRDC Ottawa CR 2013-101 v

Table of contents

Abstract …….. ................................................................................................................................. i Résumé …..... ................................................................................................................................... i Executive summary ........................................................................................................................ iii Sommaire ..... .................................................................................................................................. iv

Table of contents ............................................................................................................................. v

List of figures ................................................................................................................................ vii 1 Overview ................................................................................................................................... 1

1.1 Automated Experimentation System ............................................................................. 1

1.2 Solution Stack ................................................................................................................ 2

1.3 Using This Guide ........................................................................................................... 4

2 User Guide ................................................................................................................................ 5

2.1 Preparation ..................................................................................................................... 5

2.1.1 User Interface....................................................................................................... 5

2.1.2 Supported Malware .............................................................................................. 7

2.1.3 Transferring Malware .......................................................................................... 8

2.1.4 Malware Encapsulation ....................................................................................... 8

2.1.5 Malware Samples ................................................................................................. 9

2.2 Experimentation .......................................................................................................... 10

2.3 Analysis ....................................................................................................................... 14

2.3.1 Summary report ................................................................................................. 14

2.3.1.1 Static Report............................................................................................. 15

2.3.1.2 Dynamic Report ....................................................................................... 17

OpenDPI 19

Behaviour 20

HTTP 20

SMTP 21

IRC 21

Bittorrent 21

2.3.2 Detailed report package ..................................................................................... 22

2.3.3 Experiment Logs ................................................................................................ 23

2.3.4 Experiment Screenshot ...................................................................................... 24

3 Administrator Guide ............................................................................................................... 25

3.1 Deployment ................................................................................................................. 25

3.2 Initial Configuration .................................................................................................... 26

3.3 Licensing and activation .............................................................................................. 29

3.4 Virtual Infrastructure ................................................................................................... 30

3.4.1 Hardware Requirements .................................................................................... 30

vi DRDC Ottawa CR 2013-101

3.4.2 Experiment Manager ......................................................................................... 31

3.4.2.1 Network Design ....................................................................................... 33

3.4.2.2 Virtual Machines ...................................................................................... 34

3.4.3 Static Experiment ............................................................................................... 35

3.4.3.1 Network Design ....................................................................................... 36


3.4.4 Dynamic Experiment ......................................................................................... 38

3.4.4.1 Network design ........................................................................................ 39


3.5 Update Experiment Systems ........................................................................................ 43

3.5.1 Updating Virtual Machine Systems ................................................................... 43

3.5.2 Updating Snort IDS ........................................................................................... 46

3.5.3 Adding a New Malware System ........................................................................ 47

3.5.3.1 Creating a New Virtual Machine ............................................................. 47

3.5.3.2 Preparing a Malware Virtual Machine ..................................................... 49

3.5.3.3 Installing AES Supporting Software ........................................................ 52

3.5.3.4 Staging a Malware Virtual Machine ........................................................ 53

3.5.3.5 Adding New Malware Virtual Machines to AES .................................... 54

4 FAQ and Troubleshooting ...................................................................................................... 58

4.1 Known Issues and Limitations .................................................................................... 58

4.1.1 OpenDPI Deep Packet Inspection...................................................................... 58

4.1.2 Document Scanning ........................................................................................... 58

4.1.3 Known experiment errors .................................................................................. 58

4.2 Powering Down AES Server ....................................................................................... 62

4.3 Machine Accounts and Console Access ...................................................................... 62

4.4 Backup and Restore ..................................................................................................... 64

4.4.1 Experiment Reports ........................................................................................... 64

4.4.2 Experiment Virtual Machines ............................................................................ 65

4.4.2.1 Connecting to the AES Infrastructure ...................................................... 66

4.4.2.2 Backing up to an External Drive .............................................................. 66

4.4.2.3 Restoring from an External Drive ............................................................ 67

4.4.2.4 Backing up to Secondary Storage ............................................................ 68

4.4.2.5 Restoring from Secondary Storage .......................................................... 69

4.5 Recovering from an AES Service failure .................................................................... 70

4.5.1 Manually restarting the AES Service ................................................................. 71

4.5.2 Manually removing pending experiments ......................................................... 72

4.6 Malware Submission Failure ....................................................................................... 73

4.7 Experiment Virtual Machines Fail To Start ................................................................ 74

DRDC Ottawa CR 2013-101 vii

List of figures

Figure 1 – AES Solution stack ........................................................................................................ 2

Figure 2 – AES management diagram ............................................................................................. 3

Figure 3 - Experiment Manager Web Interface ............................................................................... 6

Figure 4 - Experiment Manager Malware Analysis Interface ....................................................... 10

Figure 5 - Experiment Manager Console Log ............................................................................... 12

Figure 6 - Experiment Manager Debug Log ................................................................................. 12

Figure 7 - Experiment Manager Pending Experiments ................................................................. 12

Figure 8 - Experiment Manager Reports ....................................................................................... 13

Figure 9 - Detailed Report Package Contents ............................................................................... 22

Figure 10 - Detailed Report Package Experiment Directory Contents .......................................... 22

Figure 11 - Experiment Screenshot Selection ............................................................................... 24

Figure 12 – AES Experiment Manager Architecture .................................................................... 31

Figure 13 – AES Management Network Diagram ........................................................................ 33

Figure 14 – Static Experiment Virtual Topology .......................................................................... 35

Figure 15 – Static Experiment Network Diagram ......................................................................... 36

Figure 16 – Dynamic Experiment Virtual Topology ..................................................................... 38

Figure 17 – Dynamic Experiment Network Diagram.................................................................... 39

Figure 19 - AVG Antivirus Update Download Web Page ............................................................ 44

Figure 20 - Experiment Manager Web Interface Agent ISO ......................................................... 52

Figure 21 - Agent zip and iso File Contents .................................................................................. 52

viii DRDC Ottawa CR 2013-101


DRDC Ottawa CR 2013-101 1

1 Overview

1.1 Automated Experimentation System

The Automated Experimentation System (AES) is a software library and virtual testing environment designed by Communications Research Centre Canada (CRCC) in order to conduct network security experimentations. AES for malware analysis is used to conduct experiments on malware samples within an isolated virtual network. The virtual network provides a simulated environment where malware can be executed and observed for analysis. The isolated network is comprised of multiple VMware ESX hosted virtual machines.

AES leverages Deep Packet Inspection (DPI) techniques to identify malware network behaviour at the application protocol layer. For instance, malware samples which generate HTTP or IRC network traffic on non-standard ports, to the Internet or within the affected network, will be identified. A selection of Anti-Virus solutions is provided for testing malware file detection capabilities.

A Web-based user interface is used for initiating malware sample experimentation within the isolated network. It provides progress on running experiments and the resulting experiment reports.

The submitted malware is executed by AES on the Malware VM. There are two types of malware experiments that can be conducted:

Static: A Static experiment environment is intended for analyzing malware samples without executing them. The Static experiment virtual machines available include various Anti-Virus security software and file identification tools for scanning malware in a controlled environment.

Dynamic: A Dynamic experiment environment is intended for observing submitted and executed malware behaviour on an isolated network, which simulates a limited private network with access to the Internet. Simulating access to the Internet is accomplished using a specially designed firewall which implements traffic shaping to direct Internet based IP address requests to a virtual machine within the environment.

2 DRDC Ottawa CR 2013-101

1.2 Solution Stack

The AES solution stack is built on a VMware vSphere ESX virtualization infrastructure installed on a dedicated physical server. It is a combination of specially designed virtual machines, custom AES software and ESX configuration.

Figure 1 – AES Solution stack


The virtual environment is managed from a separate AES Management client machine using the vCenter client. Intended use of AES is from a Web interface hosted by the Experiment Manager virtual machine which is accessed from the separate AES Management client machine over HTTP. The Experiment Manager provides a Web interface for facilitating the following functional usage:

Submitting malware sample files for experimentation

Selecting the type of environment and target virtual machine

Monitoring experiment progress

Retrieving experimentation results

Troubleshooting experiment operation issues

Figure 2 – AES management diagram

The experiment environments have separate virtualized machine and network configurations for conducting their respective experiments. See section 3.4 for more details on the virtual infrastructure.


1.3 Using This Guide

This guide is divided into two general sections; a User Guide and an Administrator Guide.

The Administrator Guide section must first be reviewed by an individual tasked with administrating the AES Infrastructure. Once the AES Infrastructure has been installed and configured, it is ready for AES user experimentation.

For AES users, start with section 2 “User Guide”, which contains the instruction for investigating malware. The user operating AES is expected to be highly familiar with Microsoft Windows computer systems, networking communication protocols and ports, and methods of safely handling malware. Experience or understanding of VMware and virtual environments is recommended.

For AES administrators, section 3 “Administrator Guide” contains the administrative portion of this document intended for technical staff tasked with deploying and managing the AES environment. This would likely include both physical and virtual aspects to the environment. The individual administrating the AES system should be highly familiar with server and network hardware installation, Microsoft Windows computers systems, network configuration, and VMware vSphere ESX. Each AES supporting virtual machine is carefully configured for automation purposes, and so attention to detail is important.


2 User Guide

This section covers general usage of the AES Experiment Manager for conducting malware experiments, including submission of malware for experimentation, and analysis of the resulting reports.

See section 3 “Administrator Guide” for further details on the AES Infrastructure, virtual machine or experiment environment configuration.

2.1 Preparation

Malware experimentation is initiated, monitored and reviewed from a separate physical computer termed the AES Management system. Ensure the AES administrator has prepared a suitable AES Management system, and that it is accessible.

2.1.1 User Interface

The Experiment Manager Web interface is the primary interface for conducting and reviewing malware experiments. It is accessed using a Web browser from the AES Management system. The interface is divided into four functional sections:

Add Malware Analysis: Used to upload malware sample files and select which experiments to conduct with the sample.

Running: Displays running malware experiments with an accompanying progress log.

Pending: Lists pending malware experiments.

Reports: Lists all completed malware experiments. Each experiment includes links to a summary report, log file, screen shot, and the detailed report package containing all resulting experiment files.


The following image illustrates an Experiment Manager Web interface with running, pending and completed experiments.

Figure 3 - Experiment Manager Web Interface


2.1.2 Supported Malware

AES supports six malware file types for execution:

Executable: Malware executable with the .exe file extension that can be executed on the Windows operating system as normal.

Windows Batch: Windows batch file with the .bat file extension that can be executed on the Windows operating system as normal.

Adobe PDF: Document file with the .pdf file extension that contains code which is executed when opened with the Adobe Acrobat Reader.

Microsoft Word: Document file with the .doc or .docx file extension that contains code which is executed when opened with Microsoft Office Word.

Microsoft Excel: Spreadsheet document with the .xls or .xlsx file extension that contains code which is executed when opened with Microsoft Office Excel.

Microsoft PowerPoint: Presentation document with the .ppt or .pptx file extension that contains code which is executed when opened with Microsoft Office PowerPoint.

AES will only process malware sample files of these types as described. The experiment types available for use will depend on the malware same type. Submitting unsupported malware will result in the experiment failing.


2.1.3 Transferring Malware

All experimentation malware files must be transported to the AES Management client machine for submission to AES through the Experiment Manager Web interface. The method of transportation used is at the discretion of the user, such as CD/DVD, or read-only USB drive. It is assumed that any malware files used during experimentation are handled with care outside of the experimentation network.

2.1.4 Malware Encapsulation

To aid in avoiding accidental execution of malware files on the AES Management client machine, AES supports submission of the encapsulated malware files in the following optional container types:

None: The malware file is un-encapsulated and unmodified.

Zip: The malware file has been compressed using the password “infected” into the .zip format using WinZip or a similar compatible application. AES will un-compress the file for use during testing.

Gzip: The malware file has been compressed using the password “infected” into the .zip format, then further encapsulated into the .gzip format. WinZip, or a similar compatible application, may be used to compress and password protect the malware sample into the .zip format. Gzip, or a similar compatible application, may be used to compress the .zip file into the .gzip format. AES will un-compress the .gzip file, then un-compress the .zip file for use during testing.

Regardless of the container type, AES must be able to use (open or execute) the malware file type once un-encapsulated.

Transport the malware file to the AES Management client machine and proceed to the next section.

The time estimate for a Dynamic experiment is 6.5 minutes. The malware sample will be executed for 1 minute during this time. For a Static experiment, the time estimate is 1.5 minute.


2.1.5 Malware Samples A set of sample malware files have been included for testing purposes. They are not destructive live malware files, but samples developed specifically for AES testing purposes. These samples may be used to determine if AES is operating correctly, for user training, or for producing sample experiment reports. The malware samples are available from the “fake malware” link at the bottom of the Experiment Manager Web interface. The zip file contains directories for the six supported malware types. See 2.1.2. “Supported Malware” for more information on supported malware types.


2.2 Experimentation

The following steps are a guide on how to submit malware files for experimentation.

1) Login to the AES Management client machine.

2) Open an Internet browser and access the Experiment Manager Web interface: http://192.168.66.7

The malware submission web page should load.

Figure 4 - Experiment Manager Malware Analysis Interface

3) In the “Add Malware Analysis” section, click the “Choose File” button and navigate to the malware file located on the AES Management client machine.

A set of sample malware files have been included for testing purposes. They are contained in a compressed zip file downloadable from the “fake malware (zip)” link at the bottom of the Experiment Manager Web interface. The contained sample malware files may be submitted from the “Add Malware Analysis” section.

4) Select the malware file type. This will determine how AES treats the malware in the experiment environment. For example, a .doc will be launched with Microsoft Word.

AES supports six malware file types for execution: .exe, .pdf, .doc, .xls, .ppt, and .bat

5) If the malware file is contained within a zip or gzip container format, select the appropriate Container type.

http://192.168.66.176/


6) Select the experiment type and targets for the experiment environment. These selections correspond with the available AES virtual machines which implement the experiment type.

For example, a Static experiment using the AVG 2012 option will provision the AES-MAS-EM01.AVAVG2012 virtual machine as the Static Scanner in the experiment. See section 3.4.3 for more information on Static experiment architecture and section 3.4.3.2 for details on the virtual machines included. For example, a Dynamic experiment using the Windows XP Pro SP3 option will provision the AES-MAS-EM01.MalwareWinXPProSP3 virtual machine as the Malware system in the experiment. See section 3.4.4 for more information on Dynamic experiment architecture and section 3.4.4.2 for details on the virtual machines included. The “File/TrID/Hashes/ssdeep” Static experiment includes static scanning of a submitted malware sample using a set of file identification and hash signature generation tools. See section 2.3.1.1 “Static Report” for sample results. Multiple experiment type options may be selected. Each experiment selected will be executed in sequential order. As each experiment completes, an average time duration label will be displayed next each experiment selection item For a Dynamic experiment, the estimated duration is 6:30 minutes. The malware sample will be executed for 1 minute during this time. For a Static experiment, the time duration estimate is 1:30 minutes.

7) Click the Submit button to begin the experiment.

AES will begin the experiment. The AES Service will provision and configure the appropriate virtual machines, and the malware will be transported to the intended virtual machine for experimentation.


The experiment in progress will be displayed in the “Running” section, including an updating progress log of the experiment steps being executed within the environment.

The Console and Debug icons maybe clicked to switch between a user friendly experiment progress log, and a verbose debug progress log.

Figure 5 - Experiment Manager Console Log

Figure 6 - Experiment Manager Debug Log

Pending experiments will be listed and executed in sequential order. A maximum of 50 pending experiments at any time is supported.

Figure 7 - Experiment Manager Pending Experiments


As each experiment completes, the report and accompanying files will be made available in the Reports section. Each report includes buttons that may be used to download a summary report, the experiment progress log file, a zip format compressed version that includes all collected experiment files, a desktop screenshot of the Malware system, and an option to delete the report files. A maximum of 800 reports will be retained, with older reports being deleted.

Figure 8 - Experiment Manager Reports

See section 2.3 “Analysis” for more information on experiment reports.

See section 4.5 “Recovering from an AES Service failure” for more information on failed experiment reports.


2.3 Analysis

Experiment reports will be made available in the Reports section of the Experiment Management Web interface. A summary report and a detailed zip package are provided. The detailed package includes pcap packet captures and log files. In most cases the summary report will be enough to review the malware behaviour.

2.3.1 Summary report

An experiment summary report is an automated interpretation of the AES Service collected logs and files from experiment systems. An experiment summary report may be accessed from the AES Experiment Manager Web interface by clicking the summary report icon in the “Reports” section.

The summary report has four primary sections: Information, Experiments, Static and Dynamic. The report is intended to capture activity of interest without the need to parse through all the collected logs and files.

The “Information” section includes basic file and timestamp details.

[Information]

File: Malicious_Document.pdf

MD5: 6db4c24b54ed2887c6e5eab05c3a545d

Generated on: Wed Jun 27 2012 at 13:52:57 EDT

The “Experiments” section indicates, with an X, the experiment types executed.

[Experiments]

STATIC

[_] File / TrID / Hashes / ssdeep

[X] AVG 2012

[_] Avira CLS

DYNAMIC

[_] Windows XP Pro SP2

[X] Windows XP Pro SP3

[_] Windows 7 Enterprise x86 SP1

[_] Windows 7 Enterprise x64 SP1

See 2.3.1.1 “Static Report” and 2.3.1.2 “Dynamic Report” for information on how to interpret the summary report “Static” and “Dynamic” sections.


2.3.1.1 Static Report

The Static section of the summary report includes all summary details collected from the target experiment virtual machine. For example, if the AVG 2012 experiment was selected, then the “AVG 2012” Static sub-section will be populated with the AVG output from scanning the malware sample. If an available experiment type was not selected and executed, results for it will be displayed as “Not available”. The following example includes the “File/TrID/Hashes/ssdeep” and “AVG 2012” experiment results.

====================================================================

STATIC

====================================================================

[File / TrID / Hashes / ssdeep]

MD5: 1e25fbaab306f57f0bce5de1c1be17ed

SHA1: 17b4b10cf9bd569db13ac49b57c1ce68c94d97bf

SHA256: b272ee56881883221e8c57dbbf9268cc9e940e7256efdf02ef139deb56074b5c

SHA512:

e5573847c73fa1c1fa9fee3bd4613e310d999e60c1d4ce79153930083e6a4fd04d1ae81ce7257c4276f3001221

4846f352d25277776d1dfba9d2bbfca50fe00d

Size: 373295 bytes

SSDeep Size: 6144

SSDeep Hash: EuIlWqB+ihabs7Ch9KwyF5LeLodp2D1Mmakda0qLqIb2YeJXKzeQStBf4X

File v4.21:

PE32 executable (GUI) Intel 80386, for MS Windows, UPX compressed

TRiD v2.10:

45.1% (.EXE) UPX compressed Win32 Executable (30569/9/7)

39.2% (.EXE) Win32 EXE Yoda's Crypter (26569/9/4)

9.6% (.EXE) Win32 Executable Generic (6514/8/2)

2.9% (.EXE) Generic Win/DOS Executable (2002/3)

2.9% (.EXE) DOS Executable Generic (2000/1)

====================================================================

[AVG 2012]

AVG 2012 Anti-Virus command line scanner

Copyright (c) 1992 - 2012 AVG Technologies

Program version 2012.0.2180, engine 2012.0.2433

Virus Database: Version 2433/5062 2012-06-11

------------------------------------------------------------

Test started: 12.6.2012 11:45:38

Duration of test: 2 second(s)

------------------------------------------------------------

Objects scanned : 3

Found infections : 0

Found PUPs : 0

Healed infections : 0

Healed PUPs : 0

Warnings : 0

------------------------------------------------------------

====================================================================


The “File/TrID/Hashes/ssdeep” experiment includes static scanning of the submitted malware sample using the following tools:

File: A UNIX tool used to classify file types.

TrID: A file identification tool designed to identify file types by their binary signature.

Hashes: Used to generate common hash signatures including MD5, SHA1, SHA256 and SHA512.

Ssdeep: A tool used to compute Context Triggered Piecewise Hashes (CTPH), also known as fuzzy hashes. CTPH can be used to identify modified versions of a file even if data has been inserted, modified or deleted in the resulting new file.

This information may be used to determine if the malware sample is detectable by the anti-virus solutions tested, and to safely get a clear finger print of the file for possible detection using other security tools within an organization.


2.3.1.2 Dynamic Report

The Dynamic section of the summary report includes all summary details collected from Dynamic experiment systems and tools. When analyzing a dynamic report, consider the following experiment specific systems and IP addresses. This list should aid in determining if the observed network communication is to an internal virtual machine, or to an external destination which is not part of the AES network infrastructure. Additionally, consider that some observed network communication may be to legitimate Internet services, such as the Adobe update service that launches when a .pdf malware sample is opened in Adobe Acrobat. Inventory name or service Network details AES-MAS-EM01.FirewallChannel VMware vSwitch: AES-MAS-EM01.VMNet3

IP Address: 10.92.65.1 AES-MAS-EM01.AVAviraCLS VMware vSwitch: AES-MAS-EM01.VMNet3

IP Address: 10.92.94.1 AES-MAS-EM01.AVFile421 VMware vSwitch: AES-MAS-EM01.VMNet3

IP Address: 10.92.94.1 AES-MAS-EM01.AVAVG2012 VMware vSwitch: AES-MAS-EM01.VMNet3

IP Address: 10.92.94.1 AES-MAS-EM01.DNAT Interface: 1

VMware vSwitch: AES-MAS-EM01.VMNet3 IP Address: 10.92.64.1 Interface: 2 VMware vSwitch: AES-MAS-EM01.VMNet2 IP Address: 10.92.32.1

AES-MAS-EM01.MalwareWinXPProSP3 AES-MAS-EM01.MalwareWinXPProSP2 AES-MAS-EM01.MalwareWin7Ent32SP1 AES-MAS-EM01.MalwareWin7Ent64SP1

VMware vSwitch: AES-MAS-EM01.VMNet3 IP Address: 10.92.95.2

AES-MAS-EM01.Honeypot VMware vSwitch: AES-MAS-EM01.VMNet3 IP Address: 10.92.36.1

Depending on the behavioral collection tool being used and the malware sample type submitted, the following sub-sections should be present. If a collection tool is not available for a malware type, results for it will be displayed as “Not available”.


IDS - Snort The Intrusion Detection System Snort is used to inspect the collected packet capture .pcap file resulting from the experiment for malware related network activity. Network probing or attacks detected by Snort will be displayed in the “IDS – Snort” section in a readable summary format. This information can be used to better understand how the malware attempts to spread, probe for network weaknesses, and possibly communicate with an outside command system.

[IDS - Snort]

1:2000345:9 - ET ATTACK_RESPONSE IRC - Nick change on non-std port (1 time)

1:2000347:9 - ET ATTACK_RESPONSE IRC - Private message on non-std port (3 times)

1:2000348:9 - ET ATTACK_RESPONSE IRC - Channel JOIN on non-std port (1 time)

1:2008350:5 - ET POLICY Autoit Windows Automation tool User-Agent in HTTP Request -

Possibly Hostile (1 time)

1:2100483:6 - GPL SCAN PING CyberKit 2.2 Windows (2 times)

DNS Query All Malware system DNS queries are extracted from the collected packet capture .pcap file resulting from the experiment. If the submitted malware is capable of network access, all hostname queries will be recorded and displayed in the “DNS Query” section. This information can be used to determine malware Internet destinations by hostname.

[DNS Query]

Query A for gnsnoitze.dynserv.com, Response A with 10.92.37.216

Query A for hvdbepsyo.afraid.org, Response A with 10.92.41.141

Query PTR for 10.92.36.109, Response PTR with csptoim.1dumb.com

TCP Packet capturing is used to record all network communication activity. A summary of the TCP connections attempted from the infected Malware system are displayed in the “TCP” section. This information can be used to view all malware network TCP communication ports and protocols.

[TCP]

10.92.52.222 on port 80, syn (2 times)

10.92.40.180 on port 80, syn (2 times)

10.92.50.88 on port 80, synack (2 times)

10.92.32.168 on port 80, syn (2 times)

10.92.35.3 on port 80, synack (2 times)


UDP Packet capturing is used to record all network communication activity. A summary of the UDP connections attempted from the infected Malware system are displayed in the “UDP” section. This information can be used to view all malware network UDP communication ports and protocols.

[UDP]

10.92.36.1 on port 53, to-client (351 times)

10.92.95.255 on port 138, to-server (2 times)

10.92.36.1 on port 53, to-server (351 times)

OpenDPI

The packet categorization tool OpenDPI is used to identify and categorize all network communication activity. A summary of the connections and their identified communication type are displayed in the “OpenDPI” section. This information can be used to determine malware protocols used on non-standard ports. Note that custom or modified protocols implemented by a malware sample may result in an unknown protocol type.

[Connections (OpenDPI)]

10.0.0.0 (0) -----------> IGMP -----------> 224.0.0.1 (0)

10.92.95.2 (1033) ------> DNS ------------> 10.92.36.1 (53)

10.92.95.2 (1043) ------> unknown --------> 10.92.48.169 (65520)

10.92.95.2 (1046) ------> Mail_SMTP ------> 10.92.59.126 (25)


Behaviour

Procmon, also known as “Process Monitor”, is a Microsoft SysInternals tool used by AES on the Malware system to monitor local process activities for submitted exe type malware samples. Procmon will monitor all read, write and update activity for local files and the Windows system registry. A summary of all process file activity is displayed in the “Behaviour” section. This may include AES specific process activity as well. The information can be used to determine if the malware affects the local system in any way. The malware sample file submitted is renamed “suspicious_file.exe” in the summary report. In the following example, the malware was observed reading files, loading mapped system .dlls in memory, creating a new executable file, and invoking the cmd.exe shell.

[BEHAVIOUR]

suspicious_file.exe Read itself C:\suspicious_file.exe

suspicious_file.exe Read file C:\WINDOWS\system32\ntdll.dll

suspicious_file.exe Create mapping of file C:\WINDOWS\system32\kernel32.dll

suspicious_file.exe Create mapping of file C:\WINDOWS\system32\imm32.dll

suspicious_file.exe Read file C:\WINDOWS\system32\sortkey.nls

suspicious_file.exe Create file C:\a.bat

suspicious_file.exe Create registry key

HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders

suspicious_file.exe Modify registry key

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders\Common AppData

suspicious_file.exe Create process C:\WINDOWS\system32\cmd.exe

cmd.exe Read file C:\WINDOWS\Prefetch\CMD.EXE-

087B4001.pf

cmd.exe Create mapping of file C:\WINDOWS\system32\shimeng.dll

cmd.exe Create mapping of file C:\WINDOWS\AppPatch\sysmain.sdb

cmd.exe Create mapping of file C:\WINDOWS\AppPatch\AcGenral.dll

HTTP

All HTTP requests are captured by AES and displayed in the “HTTP Request” section. This information may be used to determine details of HTTP requests to an external Internet destination. These HTTP requests may include valuable information on what a malware sample may have collected or how it reports back to a command system.

[HTTP Request]

GET /reg?u=33617B11&v=187&s=1123&su=2039&p=1&e=0&o=0&a=1&wr=75 (HTTP/1.1)

Host: lmehzb.1dumb.com (10.92.57.247)

User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)

GET /reg?u=33617B11&v=187&s=1123&su=2039&p=1&e=0&o=0&a=1&wr=75 (HTTP/1.1)

Host: cqtaejgtgp.3-a.net (10.92.35.26)

User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)


SMTP

All SMTP requests are captured by AES and displayed in the “SMTP” section. This information may be used to determine SMTP request details to an external Internet destination. A malware sample which attempts to send e-mails to internal systems or external destinations can be a spambot or is attempting to communicate with an external command system via SMTP.

[SMTP]

From: <[email protected]>

To: [email protected]

Subject: IuE7WsWu47to6z

Attachment (1): file.zip

Body: Hello,

:

: my name is Williams

: Regards

:

: .

IRC

All IRC requests are captured by AES and displayed in the “IRC” section. This information may be used to determine IRC request details to an external Internet destination. Monitoring a malware sample attempting to communicate with an external IRC server can be used to determine if and where an IRC command system may exist.

[IRC]

Server: 138.17.213.65 (1095)

Channel(s): #moti

Nick: pejo

User: mogi 8 * : nepo

Session: NICK pejo

USER mogi 8 * : nepo

JOIN #moti

PRIVMSG #moti :mogi writing many msgs

PRIVMSG #moti :to confuse the walruses

PRIVMSG #moti :and gnomes

QUIT :mogi complete, signing off

Bittorrent

If the malware sample implements a Bittorrent client, any attempts to communicate with a public Bittorrent seed server will be captured. This information may be used to determine if the malware sample is capable of downloading illegal or potentially malicious content over distributed Bittorrent servers and clients.

[Bittorrent Announce]

/announce.php

Host: 166.95.109.5:9400 (166.95.109.5)

Info-Hash: SUPERHASH1234567890

User-Agent: AutoIt

mailto:[email protected]



2.3.2 Detailed report package

The detailed report package includes all collected logs and files from the experiment systems. An experiment report package may be accessed from the AES Experiment Manager Web interface by clicking the experiment package icon in the Reports section.

The compressed zip file contains the summary report, experiment log and debug files. Sub-directories for each experiment type will be present. Depending on the experiment type, the collected files can vary.

Figure 9 - Detailed Report Package Contents

A Static experiment sub-directory will contain the target anti-virus solution or tool output. The AES experiment instruction xml file will also be present.

A Dynamic experiment sub-directory will contain the following:

Packet capture .pcap files from both network interfaces from the firewall in the experiment network.

Snort IDS and OpenDPI processed .csv files from the packet capture files.

A screenshot of the Malware virtual machine desktop after the malware has executed.

Procmon related .csv files capturing malware sample behaviour on the Malware virtual machine.

Figure 10 - Detailed Report Package Experiment Directory Contents


2.3.3 Experiment Logs

The experiment debug log file may be viewed from the AES Experiment Manager Web interface by clicking the log file icon in the Reports section.

The debug log file will be displayed and can be saved. See section 4.1.3 “Known experiment” errors for more information on interpreting failed experiment errors.

[ USER ] Initializing experiment

[ USER ] Step: AES1 "Setting up VM configurations" (Fri Jun 29 08:49:23 EDT 2012) ...

Done (2s)

[ USER ] Step: AES2 "Validating NICs" (Fri Jun 29 08:49:25 EDT 2012) ... Done (0s)

[ USER ] Step: AES3 "Validating snapshots" (Fri Jun 29 08:49:26 EDT 2012) ... Done (1s)

[ USER ] Step: AES4 "Setting snapshots" (Fri Jun 29 08:49:27 EDT 2012) ... Done (13s)

[ USER ] Step: AES5.1 "Powering on VMs" (Fri Jun 29 08:49:41 EDT 2012) ... Done (1m 20s)

[ USER ] Step: AES5.2 "Setting up CD-ROMs" (Fri Jun 29 08:51:01 EDT 2012) ... Done (0s)

[ USER ] Step: AES5.3 "Setting up NICs" (Fri Jun 29 08:51:01 EDT 2012) ... Done (16s)

[ USER ] Step: AES6 "Opening channels" (Fri Jun 29 08:51:18 EDT 2012) ... Done (0s)

[ USER ] Step: AES7 "Creating output directories" (Fri Jun 29 08:51:18 EDT 2012) ... Done

(0s)

[ USER ] Running experiment

[ USER ] Step: S1 "Configuring /etc/hosts for DNAT" (Fri Jun 29 08:51:18 EDT 2012) ...

true (1s)

[ USER ] Step: S2 "Send DNAT Script" (Fri Jun 29 08:51:19 EDT 2012) ... true (1s)

[ USER ] Step: S3 "Send IP rule file" (Fri Jun 29 08:51:20 EDT 2012) ... true (2s)

[ USER ] Step: S4 "Run DNAT Script" (Fri Jun 29 08:51:22 EDT 2012) ... true (2s)

[ USER ] Step: S5 "Start DNAT Process" (Fri Jun 29 08:51:24 EDT 2012) ... true (22s)

[ USER ] Step: S6 "Configuring /etc/hosts for Honeypot" (Fri Jun 29 08:51:46 EDT 2012)

... true (12s)

[ USER ] Step: S7 "Start Honeypot scripts" (Fri Jun 29 08:51:58 EDT 2012) ...


2.3.4 Experiment Screenshot

A screenshot of the Malware virtual machine desktop is captured after the malware sample has been executed. This screenshot may be viewed from the AES Experiment Manager Web interface by clicking the desktop icon in the Reports section.

Clicking on the experiment report screenshot icon will display thumb nail images of all included experiment screenshots. Each thumbnail image can be clicked to view a large image of the Malware virtual machine desktop. This screenshot can be used to visually inspect the state of the Malware virtual machine desktop. If the malware opened various applications or communicated a message to the user, a visual representation would show this.

Figure 11 - Experiment Screenshot Selection


3 Administrator Guide

When first deploying the AES server hardware, initial steps must be completed before end user experiment usage is possible. This section covers AES Infrastructure deployment, networking requirements, and maintenance procedures.

See section 2 “User Guide” for more information on how to submit malware samples and conduct experimentations.

3.1 Deployment

Decide on a suitable location for the AES server where the following requirements are met:

Server power requirements. Enough electrical power outlets to meet all server power supplies needed, including any redundant power supplies. To avoid data loss or corruption in the event of a power failure, it is recommended that a UPS power backup solution be included.

A separate Windows machine representing the AES Management client.

A network switch should be available for facilitating network communication between the AES Management client machine and the physical AES server.

The AES server and AES Management client machine should be isolated from existing networks due to the potentially dangerous nature of testing malware files.

Deploy the AES server into the target lab location, power on all physical hardware and proceed with the section 3.2 “Initial Configuration”.


3.2 Initial Configuration

Similar to commercial network equipment, the AES Server and Experiment Manager virtual machine are pre-configured with default network configurations:

AES Server default network configuration Network: IP Address: 192.168.66.176

Network Mask: 255.255.255.0

Experiment Manager default network configuration Network: IP Address: 192.168.66.7

Network Mask: 255.255.255.0

If the physical environment is not already prepared, complete the following steps:

1) Ensure any physical network configuration is complete. The AES Management client machine requires network access to the AES Server.

2) Plugin and power on the AES server.

3) Power on the separate AES Management client machine.

Once the AES Management and AES Server systems are powered on complete the following steps:


2) Re-configure the network configuration settings to allow the client machine to reach the default AES network 192.168.66.0/24. For example, configure the client machine IP address to 192.168.66.100 with a network mask of 255.255.255.0.

3) Ensure the AES Management system can reach the AES Server by opening an Internet browser and connecting to: http://192.168.66.176

4) From an external system, download the VMware vCenter client 4.1 software from the vmware.com web site:

http://vsphereclient.vmware.com/vsphereclient/4/9/1/5/5/7/VMware-viclient-all-4.1.0-491557.exe

VMware vCenter client versions are generally backward compatible with older versions of ESX. AES is built upon VMware ESX 4.1 update 2, and so the above client is the minimum version required.

5) Transfer the VMware vCenter client software to the AES Management client machine and install.

http://192.168.66.176/




6) Start the VMware vCenter client software and connect to the AES Server using the following settings:

IP address/Name: 192.168.66.176 User name: root Password: aesmas303

7) From the VMware vCenter client select the Experiment Manager (AES-MAS-EM01) virtual machine and click the Power on icon.

8) From an Internet browser, ensure the Experiment Manager Web interface can be accessed at: http://192.168.66.7

At this point the AES Management client can access the Experiment Manager Web interface, and manage the virtual infrastructure through the VMware vCenter client. Malware samples may now be submitted for experimentation.

See section 4.2 “Powering Down AES Server” on how to power manage the AES Server virtual machine environment. Note that the virtual machines may include a machine state snapshot. Do not modify these snapshots or create new snapshots.

See section 4.3 “Machine Accounts and Console Access” for more information on default user accounts.

http://192.168.66.7/


Before AES can be turned over to experiment users, a test experiment should be conducted to ensure that the environment is operating correctly.

Proceed to section 2 “User Guide” for guidance on submitting malware using a supplied fake malware sample. Ensure the malware experiment completes successfully.


3.3 Licensing and activation

The AES environment is comprised of integrated vendor software, some of which require purchased commercial licensing. It is the responsibility of the AES administrator to obtain and replace all vendor software licensing with their own

The following list details all vendor purchased licensing that should be replaced:

VMware vSphere Essentials Kit for ESX 4.x The AES infrastructure leverages VMware ESX software for virtualizing the experiment environment. A license key is required to enable the VMsafe API used by the AES Experiment Manager for provisioning and configuration virtual machines. The license may be obtained directly from the VMware store. The VMware Knowledge Base article 1010839 contains guidance on updating VMware ESX and vCenter licensing. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010839

Microsoft Windows Most of the AES virtual machines use the Windows operating system. Microsoft offers the MSDN (Microsoft Subscription Developer Network) service as a method of obtaining Microsoft software licensing for development and testing purposes. The following virtual machines contain the Microsoft Windows operating system with testing only licensing: Inventory name Licensed software AES-MAS-EM01 Windows XP AES-MAS-EM01.AVAviraCLS Windows XP AES-MAS-EM01.AVFile421 Windows XP AES-MAS-EM01.AVAVG2012 Windows XP AES-MAS-EM01.MalwareWinXPProSP3 Windows XP AES-MAS-EM01.TargetWinXPProSP2 Windows XP AES-MAS-EM01.MalwareWinXPProSP3 Windows XP AES-MAS-EM01.MalwareWin7Ent32SP1 Windows 7 AES-MAS-EM01.MalwareWin7Ent64SP1 Windows 7

When making changes to an AES virtual machine, a new virtual machine snapshot will need to be taken of the updated configuration. See section 3.5.1 “Updating Virtual Machine Systems” for more information on how to update a virtual machine for AES use. Once a virtual machine has been updated, it should be backed up for future use in the event of a failure or disaster. See section 4.4 “Backup and Restore” for more information on backing up and restoring virtual machines.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010839

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010839


3.4 Virtual Infrastructure

AES is an entirely virtualized environment, designed to be hosted on a single server VMware vSphere infrastructure. A library of customized virtual machines is provided. Depending on the experiment type, the virtual machines and network architecture used will change.

3.4.1 Hardware Requirements

The recommended AES Virtual Infrastructure server hardware requirements are as follows:

4 Core CPU

12 GB of ram

2 1 TB 7200 RPM hard disks or faster


3.4.2 Experiment Manager

The Experiment Manager is the AES Management system representing the initialing actor preparing the experiment environment and issuing the sequence of experiment instructions required. It is a virtual machine which is hosting an Apache Web Server with the AES PHP Web application and a Java based AES Service.

Figure 12 – AES Experiment Manager Architecture

The intended usage consists of an experiment user accessing the AES Web interface using a Web browser from a separate physical AES Management machine. Malware can be submitted through the Web interface, which is then placed in a local storage staging area.

The AES Service will observe the experiment files and begin preparing the virtual experimentation VMs, automatically configuring the virtual lab networks and virtual machines before execution and analysis of the submitted malware. The experiment instructions and malware are packaged and transferred to the target experiment type virtual machine. The AES Agent software on the target virtual machine will proceed according to the experiment instructions. For example, in a Dynamic experiment the AES Agent will execute the malware appropriately, and in a Static experiment the included malware file tools will be executed on the submitted malware.


Then the AES Service will collect logs and files from the experiment virtual machines and transfer them to the Experiment Manager local storage staging area for processing. Reports are then generated from the collected experiment logs and files, which the experiment user can access from the Web interface.

After a successful experiment, the AES will restore each virtual machine involved to a clean state; virtual network configurations will be unassigned, and the virtual machine will be restored to an initial clean snapshot. If an experiment failed and a machine is found to be in an unclean state, the AES will restore the machine to an initial clean state and proceed with any remaining queued experiments.

The Experiment Manager itself requires network access to the VMware Infrastructure physical network interface to automate the provisioning of experiment virtual machines. It includes two local hard disks for separation of operating and staging areas. All operating system, AES application code and services are located on the C: drive. While the AES staging and reporting directories are located on the D: drive.

The AES application code is located under the C:\aes directory. This directory contains the Apache and PHP Experiment Manager Web application code, the AES Service Java jar file, and any supporting configuration files.

See section 4.4 “Backup and Restore” for more information on how to backup and maintain the Experiment Manager local disk space.


3.4.2.1 Network Design

The Experiment Manager virtual machine requires an external network to facilitate experiment user AES Management client access to the Web interface, and access to the VMware virtual infrastructure for provisioning virtual machines. The network configuration for the separate AES Management machine, Experiment Manager virtual machine, and VMware vSphere ESX infrastructure should permit network connectivity between them.

The following diagram illustrates each system configured with an example network address space. Each system can communicate with each other through a physical switch or hub.

Figure 13 – AES Management Network Diagram


3.4.2.2 Virtual Machines

The following details the base AES virtual machine inventory and their essential configuration.

Experiment Manager The Experiment Manager virtual machine is used to manage the experiment environment. Inventory name: AES-MAS-EM01 Network: Interface: 1 (Management)

VMware vSwitch: Management IP Address: 10.92.66.7 Network Mask: 255.255.224.0 Interface: 2 (Experiment) VMware vSwitch: AES-MAS-EM01.VMNet4 IP Address: 10.92.68.7 Network Mask: 255.255.224.0

Applications and Services: Windows XP SP3 Apache Web Server with PHP AES Web application AES Service

Experiment Firewall The Experiment Firewall virtual machine is used to safely route network communication requests initiated from the Experiment Manager to the deployed Static Scanner virtual machine. Inventory name: AES-MAS-EM01.FirewallChannel Network: Interface: 1 (Management)

VMware vSwitch: AES-MAS-EM01.VMNet4 IP Address: 10.92.68.1 Network Mask: 255.255.224.0 Interface: 2 (Experiment) VMware vSwitch: AES-MAS-EM01.VMNet3 IP Address: 10.92.65.1 Network Mask: 255.255.224.0

Applications and Services: pfSense Firewall operating system


3.4.3 Static Experiment

When a Static experiment environment is deployed, the AES will power on the virtual machines selected in the Experiment Manager Web interface and configure any virtual network settings required.

The AES Experiment Manager will communicate with the Static Scanner virtual machine using SSH through the Experiment Firewall. The Experiment Firewall VM is designed to only allow inbound network communication from the Experiment Manager to the experimentation VMs. No experimentation VM network traffic is permitted to leave the isolated experiment network unless it is solicited by the Experiment Manager.

The submitted malware file will be transferred to the Static Scanner virtual machine local hard disk, and any file scanning tools included with the Static Scanner virtual machine selected will be initiated. The resulting scan report will be collected by the AES Experiment Manager for review.

The following diagram illustrates both the physical and virtual organization of an AES Static experiment. The VMware vSwitch names were used when identifying the environment networks.

Figure 14 – Static Experiment Virtual Topology


3.4.3.1 Network Design

The Static Experiment network is designed to allow for scanning of submitted malware files on an isolated and controlled computer system.

Figure 15 – Static Experiment Network Diagram



The AES VM inventory includes a number of specially designed machines for Static experimentation purposes. Effort has been made to name them appropriately in the VMware infrastructure inventory. The following Static Experiment specific virtual machines work in conjunction with the Experiment Manager virtual machines. See section 3.4.2.2 for more information on the Experiment Manager virtual machines.

Static Scanner The Static Scanner system virtual machine is used as a target host for scanning submitted malware. Depending on the Static virtual machine selected when initiating the experiment through the Experiment Manager Web interface, the tools used to scan the malware can vary. The following tables detail the Static Scanner virtual machines provided and the tools used during the malware file scanning experiment. Inventory name: AES-MAS-EM01.AVAviraCLS Network: VMware vSwitch: AES-MAS-EM01.VMNet3

IP Address: 10.92.94.1 Network Mask: 255.255.224.0

Applications and Services: Microsoft Windows XP SP2 Avira Free for Windows 1.9.156

Inventory name: AES-MAS-EM01.AVFile421 Network: VMware vSwitch: AES-MAS-EM01.VMNet3


Applications and Services: Microsoft Windows XP SP2 File file identifier TrID File Identifier ssdeep

Inventory name: AES-MAS-EM01.AVAVG2012 Network: VMware vSwitch: AES-MAS-EM01.VMNet3


Applications and Services: Microsoft Windows XP SP2 AVG Anti-Virus 2012


3.4.4 Dynamic Experiment

A Dynamic experiment environment simulates a small network with access to the Internet and local networks. The AES will power on the Dynamic experiment virtual machines required and configure any virtual network settings required.

The following diagram illustrates both the physical and virtual organization of an AES Dynamic Experiment. The VMware vSwitch names were used when identifying the environment networks.

Figure 16 – Dynamic Experiment Virtual Topology

The AES Experiment Manager will communicate with the experiment virtual machines using SSH through the Experiment Firewall, which is designed to only allow inbound network communication from the Experiment Manager to the experimentation VMs. The DNAT firewall is configured to permit communication from the Experiment Manager to the Honeypot and Target virtual machine. The submitted malware file will be transferred to the Malware virtual machine, and then executed. Monitoring tools on the Malware virtual machine will observe local file behaviour. The DNAT virtual machine will route all localized network communication to the


Target virtual machine, and all Internet bound network communication to the Honeypot virtual machine. The Target is intended to represent a local network machine. The Honeypot is configured with packet capture and Deep Packet Inspection network monitoring tools.

Once the experiment time limit is reached, the AES Experiment Manager will collect log files from all virtual machines, process the data and generate a summary report for review.

3.4.4.1 Network design

The Dynamic Experiment network is designed to simulate a small organization network with access to the Internet, which is directed to the experiment Honeypot.

Figure 17 – Dynamic Experiment Network Diagram



The AES VM inventory includes a number of specially designed machines for Dynamic experimentation purposes. Effort has been made to name them appropriately in the VMware infrastructure inventory. The following Dynamic Experiment specific virtual machines work in conjunction with the Experiment Manager virtual machines. See section 3.4.2.2 for more information on the Experiment Manager virtual machines.

DNAT The DNAT system virtual machine is a Linux based virtual machine using the ipchains firewall with a custom packet reshaper intended to simulate local and Internet networks by routing network traffic from the Malware virtual machine to the Honeypot or Target virtual machines. It additionally supports the routing of Experiment Manager communications through the Experiment Firewall to the Honeypot and Target VMs for initiating experiment instructions and collecting logs and reports. Inventory name: AES-MAS-EM01.DNAT Network: Interface: 1

VMware vSwitch: AES-MAS-EM01.VMNet3 IP Address: 10.92.64.1 Network Mask: 255.255.224.0 Interface: 2 VMware vSwitch: AES-MAS-EM01.VMNet2 IP Address: 10.92.32.1 Network Mask: 255.255.224.0

Applications and Services: ipchains AES packet reshaper

Malware The Malware system virtual machine represents a vulnerable target system into which the experiment submitted malware will be executed. Several virtual machines with different software configurations are provided. Sensors, such as Procmon, are used to capture file activity related to the malware. Inventory name: AES-MAS-EM01.MalwareWinXPProSP3 Network: VMware vSwitch: AES-MAS-EM01.VMNet3


Applications and Services: Microsoft Windows XP SP3 Adobe Reader 10.1.3 Office 2003


Inventory name: AES-MAS-EM01.MalwareWinXPProSP2 Network: VMware vSwitch: AES-MAS-EM01.VMNet3


Applications and Services: Microsoft Windows XP SP2 Adobe Reader 8.0.0 Internet Explorer 7.0.5730.11 Office 2003 Java 6u2

Inventory name: AES-MAS-EM01.MalwareWin7Ent32SP1 Network: VMware vSwitch: AES-MAS-EM01.VMNet3


Applications and Services: Microsoft Windows 7 Enterprise 32-bit SP1 Adobe Reader X 10.1.3 Internet Explorer 8.0.7601.1751 Office 2010 Java 7u4

Inventory name: AES-MAS-EM01.MalwareWin7Ent64SP1 Network: VMware vSwitch: AES-MAS-EM01.VMNet3


Applications and Services: Microsoft Windows 7 Enterprise 64-bit SP1 Adobe Reader X 10.1.3 Internet Explorer 8.0.7601.1751 Office 2010 Java 7u4

Honeypot The Honeypot system virtual machine is a series of sensors, tools and scripts used to emulate the following Internet services:

HTTP DNS SMTP IRC

The DNAT system will route network communication from the Malware system to the Honeypot system that is Internet bound.


Inventory name: AES-MAS-EM01.Honeypot Network: VMware vSwitch: AES-MAS-EM01.VMNet3


Applications and Services: DNS HTTP SMTP IRC

Target The Target system virtual machine is a standard network system with typical services to give the malware a local system to communicate with. It will react to the malware traffic, representing a target on the local network. The DNAT system will route network communication from the Malware system to the Target system that is not Internet bound. Inventory name: AES-MAS-EM01.TargetWinXPProSP2 Network: VMware vSwitch: AES-MAS-EM01.VMNet3


Applications and Services: Microsoft Windows XP SP2


3.5 Update Experiment Systems

Many of the experimentation virtual machines include anti-virus solution software and other tools which will become out-of-date over time. Additionally, newer or different Malware virtual machine target software maybe required for experimentation on a different malware samples. The isolated test environment by default prohibits access to the Internet for vendor supplied software and virus definition updates. The following sections contain guidance on methods for updating these tools and solutions.

3.5.1 Updating Virtual Machine Systems

The Static Scanner virtual machine, used in Static experiments, includes anti-virus tools during experimentation.

The Malware virtual machine, used in Dynamic experiments, includes a baseline of operating system and software tools used during experimentation. Submitted malware will be executed on the selected Malware system, and it is possible a change in any existing combination of operating system patch level or a software component version may be needed. Depending on the malware experiment, an updated Malware virtual machine may be required.

The recommended method for updating software on these existing virtual machines is to download the vendor software, burn it to a CD/DVD disc, then mount the disc in the target virtual machine from the VMware vCenter client. The virtual machine can then be powered on and the update applied from the CD/DVD. Once the update is complete, the target virtual machine must be configured in order to be ready for AES experiment automation.

It is possible to copy the target virtual machine and export it to an Internet accessible location, then perform any Internet updating. However the sensitive nature of the experimentation subject matter may prohibit this. It is at the discretion of the AES administrator to proceed with care during this update process.

For the purposes of this guide, the AES-MAS-EM01.AVAVG2012 Static Scanner virtual machine for Static type experiments will be updated to the latest AVG anti-virus definition file. If the software to update is to the Windows operating system, all required patch files should be downloaded at one time to avoid repeating the procedure.


1) From an external machine, download the appropriate software updates from the vendor web site. In the case of a Static Scanner, each anti-virus vendor has their own method of offline updating. Free AVG 2012 anti-virus updates are available from http://free.avg.com/us-en/download-update.

Figure 19 - AVG Antivirus Update Download Web Page

2) Once downloaded, create and burn a CD/DVD containing the downloaded software.

3) Insert the CD/DVD disc or copy the ISO file to the AES Management client machine. (Not the AES Virtual Infrastructure machine)

4) Power on the paused target virtual machine. The default snapshot should be “Before Attack Scenario”.

5) While the target virtual machine is selected in the left menu, click the disc icon in the toolbar. Under “CD/DVD Drive 1”, select “Connect to D:” or the drive letter of the AES Management client machine CD/DVD drive.

In the target virtual machine console, the CD/DVD disc should be mounted and the contained files accessible.

6) Access the target virtual machine console by clicking the “Console” tab in the right pane.

Proceed with installing the software update. For AVG 2012, the AVG User Interface should be launched and under Tools, “Update from directory…” the CD/DVD update file should be selected.

7) Once update is complete, click the CD/DVD icon in the vCenter client menu toolbar and select “CD/DVD Drive 1”, and click “Disconnect from D:” to unmount the disc or ISO file. Only one virtual machine may access a mounted CD/DVD device on the AES Management machine at the same time.

http://free.avg.com/us-en/download-update


For all Static Scanner virtual machines which have been modified, ensure the following steps are complete before the target virtual machine can be used in AES experiments again.

1) Shutdown any update related software that may have been opened during the previous update steps.

2) If the software update requires a reboot, then the AES runner Agent software will need to be re-started manually:

a) Click the Start menu button. Select Run and enter “cmd”.

b) Type the following: cd \

java -jar runner.jar

The AES runner Agent software should start and be ready for the next step.

3) With the target virtual machine selected, click the Pause button to suspend the virtual machine.

Then click the “Take a snapshot of this virtual machine” icon in the VMware vCenter client toolbar.

4) Name the snapshot “Before Attack Scenario”.

Rename the previous snapshot to an appropriate alternative name. For example, “Before Attack Scenario - 06-18-2012”. AES will only accept 1 snapshot named “Before Attack Scenario”.

Do not delete the original “Before Attack Scenario” snapshot. If the virtual machine encounters any issues during experimentation with the new snapshot, the original snapshot state should be reverted to in order to re-attempt the update process.

The updated target virtual machine is now ready for normal experiment usage.


3.5.2 Updating Snort IDS

Snort IDS is used by AES on the Experiment Manager to analyze captured malware network packets. As newer Snort rule sets are released, the Experiment Manager virtual machine should be periodically updated with them. The Snort IDS 2.9.0.5 engine is used by AES, and as a consequence the Snort rule sets must be compatible with it.

AES uses the Emerging Threats free Snort rule set from: http://www.emergingthreats.net/

The latest Emerging Threats Snort IDS 2.9 rule set can be found here: http://rules.emergingthreats.net/open/snort-2.9.0/emerging-all.rules

When updating this rule set file with a newer version, complete the following:

1) Download and create a CD/DVD with the rule set from an external machine.

2) Insert the CD/DVD disc into the AES Management client machine. (Not the AES Virtual Infrastructure machine)

3) Login to the VMware vCenter client and select the AES-MAS-EM01 virtual machine.

4) Click the disc icon in the toolbar. Under “CD/DVD Drive 1”, select “Connect to D:” or the drive letter of the AES Management client machine CD/DVD drive.

In the AES-MAS-EM01 virtual machine console, the CD/DVD disc should be mounted and the contained files accessible.

5) Access the AES-MAS-EM01 virtual machine console by clicking the “Console” tab in the right pane.

6) While logged into the AES-MAS-EM01 virtual machine console, copy the emerging-all.rules file from the CD/DVD D: drive to the C:\aes\tools\snort\ET.snort-2.9.0\rules directory. Overwrite the existing rule file, or rename it.

7) Once complete, click the CD/DVD icon in the vCenter client menu toolbar and select “CD/DVD Drive 1”, and click “Disconnect from D:” to unmount the disc or ISO file. Only one virtual machine may access a mounted CD/DVD device on the AES Management machine at the same time.

The new Snort IDS rule set will now be used when analyzing collected malware packets.

AES does not currently support multiple Snort IDS rule sets. It is possible to replace the Emerging Threats rule file with a different rule set, such as from the official Sourcefire Snort rule subscription. In all cases, the rule file must be named emerging-all.rules and located under the C:\aes\tools\snort\ET.snort-2.9.0\rules\ directory.

* See section 4.3 “Machine Accounts and Console Access” for more information on default user accounts.

http://www.emergingthreats.net/

http://rules.emergingthreats.net/open/snort-2.9.0/emerging-all.rules


3.5.3 Adding a New Malware System

The Dynamic type experiment Malware system virtual machines include a baseline of operating system and software tools used during experimentation. Alternative target configurations may be needed for experimentation purposes. This section attempts to provide baseline guidance on creation of a new virtual machine which can be used with AES. It is possible that a required software version or application results in a Malware virtual machine which is not compatible with AES.

The process has been divided into logical steps which may be optional depending on how the virtual machine is created. Review all of them before attempting to import a new Malware virtual machine.

3.5.3.1 Creating a New Virtual Machine

When creating a new Malware virtual machine with the AES Infrastructure, the virtual machine may be created outside AES or directly to the AES Infrastructure local storage from the AES Management vCenter client.

The following steps detail how to create a virtual machine directly to the AES Infrastructure local storage:

1) From the AES Management client, login to the VMware vCenter client.

2) Right click the AES infrastructure menu item and select “New Virtual Machine”.

3) Complete the new virtual machine creation wizard with the following guiding settings:

Name: Determine a descriptive name for the new Malware system and prefix it with AES-MAS-EM01. For example, AES-MAS-EM01.MalwareWindows7.

Note this Malware system name when completing section 3.5.3.5 “Adding New Malware Virtual Machines to AES”. E.g. MalwareWindows7

Datastore: Select the data store named [vms01]. This is the AES infrastructure inventory location.

Guest Operating System: AES only supports the Microsoft Windows operating system for a Malware virtual machine.

Create a Disk: Limit the virtual disk size to 14GB. Optionally the “Thin Provisioning” setting may be selected to minimize datastore disk usage.

4) Once the new virtual machine has been created, select it in the menu list, right click and edit settings.


5) Ensure the following guiding configuration is met in the new virtual machine settings:

Memory: Allocate no more than 1024GB.

Network adapter 1: Select the “nothing” Network label.

CD/DVD Drive 1: Select the Client Device option.

Depending on the Windows operating system, it may be necessary to adjust other virtual machine configuration settings. For example, Windows 2003 will require the LSI Logic Parallel SCSI controller to be selected.

6) While the new virtual machine is selected in the left menu, click the Power on icon in the toolbar.

7) Click the disc icon in the toolbar and under “CD/DVD Drive 1”, select “Connect to D:” or the drive letter of the AES Management client machine CD/DVD drive.

8) Insert the operating system installation media into the AES Management client machine.

9) Select the virtual machine Console tab and click into the console display. Press the Ctrl + Alt + Insert keys to reboot within the virtual machine. Note that clicking the virtual machine restart button will disconnect the disc device.

Complete the operating system installation process from the virtual machine console.

If prompted for an initial user during installation of the operating system, consider the following default:

Name: John Doe Username: johndoe Password: password1


After the new virtual machine operating system has been installed, apply any software updates needed via the same method. Download any required software or updates from an external machine, burn it to a CD/DVD, and repeat steps 6 through 8.

Once the new Malware virtual machine operating system has been installed, proceed to the next section.


3.5.3.2 Preparing a Malware Virtual Machine

A new virtual machine intended for use as an AES Malware system must be prepared for automation use. The following guidelines should be considered when configuring a Malware virtual machine operating system:

Install any vendor software licensing offline. For Microsoft Windows this will require dialing the 1-800 number for Windows offline activation.

Change the local Administrator password to “password1”. If the default Administrator user account is disabled, enable it first then reset the password.

By default Windows 7 disables the Administrator user account. At the Start menu, type lusrmgr.msc, locate the Administrator user account, enable it and reset the password.

Enable Windows auto-login for the Administrator user account:

o Windows 7: http://windows.microsoft.com/en-US/windows-vista/Turn-on-automatic-logon

o Windows XP: http://support.microsoft.com/kb/315231

o Windows Vista: http://windows.microsoft.com/en-US/windows-vista/Turn-on-automatic-logon

Disable Windows User Account Control (UAC):

o Windows 7: http://windows.microsoft.com/en-us/windows-vista/Turn-User-Account-Control-on-or-off

o Windows Vista: http://windows.microsoft.com/en-us/windows-vista/Turn-User-Account-Control-on-or-off

Disable all operating system software update mechanisms.

For Windows Vista and Windows 7 disable all IPv6 support.

This can be accomplished by opening the Network and Sharing Center, selecting Change adapter settings, and double-clicking the Local Area Connection network adapter. Uncheck any TCP/IPv6 list items and click OK.

Avoid installing VMware Tools.

Disable any screen savers and time based user account lock out settings.

Disable any operating system firewall software. Ensure the Windows firewall is disabled for both Public and Private networks.

http://windows.microsoft.com/en-US/windows-vista/Turn-on-automatic-logon


http://support.microsoft.com/kb/315231



http://windows.microsoft.com/en-us/windows-vista/Turn-User-Account-Control-on-or-off





The following network configuration must be applied:

IP address: 10.92.95.2 Subnet mask: 255.255.224.0 Default gateway: 10.92.64.1 DNS Server: 8.8.8.8 Hostname: Select a descriptive hostname using the WORKGROUP workgroup.

A Malware virtual machine requires a basic set of applications installed to support proper execution of the various malware file types. For example, Microsoft Office Word is required for execution of malicious .doc files.

Install Microsoft Office 2003, 2007 or 2010:

o Install all Office components. Specifically Word, Excel, and PowerPoint are required.

o Install any vendor software licensing offline. For Microsoft Office this will require dialing the 1-800 number for offline activation.

o Launch Word, Excel and PowerPoint by opening .doc, .xls and .ppt files directly. E.g. double-click the .doc file. Ensure that they are able to cleanly open the files and that any first time prompts are accepted.

o During automation, AES must be able to open the files without any interruptions.

Install the Adobe Acrobat Reader version desired for experimentation purposes. Adobe provides various archived versions of the Acrobat Reader at ftp.adobe.com. Note that Adobe does remove archived versions which are known to be vulnerable to high risk exploits. Specific versions of Acrobat Reader may have to be sourced from alternative locations.

o Once installed, launch Acrobat Reader and accept the license prompts.

o Open Adobe Reader, go to Preferences, and set the update to never check for newer versions.

o Open a .pdf file directly by double-clicking it. Ensure that it is able to open cleanly without any prompts.

o Go to the Windows Services panel. Stop, and then disable the Adobe Acrobat Update Service.

Launch the Windows Media Player software and unselect the “Acquire licenses automatically for protected content” setting.

ftp://ftp.adobe.com/


A default user must be created for automation use, but also as a possible target user account for executed malware. This user may have already been created during installation of the operating system. For example:

Name: John Doe Username: johndoe Password: password1

If Microsoft Outlook or Outlook Express is included use the following configuration:

Name: John Doe E-mail address: [email protected] Incoming mail: POP3, pop3.bait1.com Outgoing mail: SMTP, smtp.bail1.com Account name: johndoe Password: janetjoe Login using SPA: No Address book entries:

o Ben Doe ([email protected]) o Joe Doe ([email protected]) o Kathy Smith ([email protected]) o Other fake e-mail addresses which malware might discover and make use of

Do not attempt to test the POP and SMTP settings. The virtual machine will not be able to reach the experiment environment services at this time.

Add as many fake e-mail addresses to the “John Doe” user Outlook address book as desired. A malware sample may attempt to discover e-mail addresses in order to further propagate or spam.

Once the new Malware virtual machine has all required applications and configuration applied, proceed to the next section.





3.5.3.3 Installing AES Supporting Software

A new Malware system will require AES specific software installed and configured for automation purposes. Complete the following steps to prepare the new Malware virtual machine for AES automation.

1) From the AES Management client, open an Internet browser and access the Experiment Manager Web interface: http://192.168.66.7

2) Scroll to the bottom of the page and download the “agent” iso file to the AES Management machine.

Figure 20 - Experiment Manager Web Interface Agent ISO

The separate “agent” zip and “iso” files contain all the required AES agent software.

Figure 21 - Agent zip and iso File Contents

3) From the AES Management client, login to the VMware vCenter client and select the new Malware virtual machine.

4) Click the disc icon in the toolbar. Under “CD/DVD Drive 1”, select “Connect to ISO image on local disk” and select the downloaded AES iso file.

In the new Malware virtual machine console, the iso file should be mounted as a CD/DVD disc and the contained files accessible.

http://192.168.66.176/


5) Access the new Malware virtual machine console by clicking the “Console” tab in the right pane.

6) Open Windows Explorer and navigate to the D: drive. Extract the agent.zip file directly to the local C: drive.

7) The C:\install\ directory contains all the installable software components required. Install every included application. E.g. Sun JDK, 7zip, etc…

8) Execute C:\Procmon.exe, accept the license agreement and close the application.

9) Copy the files in C:\My Documents\ to the John Doe and Administrator user’s My Documents directory. For example, C:\Users\johndoe\Documents and C:\Users\administrator\Documents

10) Login as the Administrator account and add the C:\runner.bat file to the Windows Startup for the Administrator user account.

Depending on the version of Windows, this may be accomplished in different ways. Generally, from Windows Explorer, click and drag the runner.bat file to the Start menu icon, navigate to the Programs, and release it in the Startup menu directory.

The runner.bat file can also be copied to the Startup directory directly. For example, on Windows 7:

C:\Users\johndoe\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\


Once the AES supporting software has been installed and configured, and the new Malware virtual machine has any required applications and configuration applied, proceed to the next section.

3.5.3.4 Staging a Malware Virtual Machine

A Malware virtual machine must have a special snapshot available before AES can automate provisioning it for experiments.

1) From the AES Management client, login to the VMware vCenter client.

2) Select the new Malware virtual machine and click the Power on button.

3) Click the “Console” tab in the right pane.


4) Ensure that runner.jar is running in a Console window.

If not, manually start it by executing the C:\runner.bat file.

This batch file should have been added to the Windows Startup directory. It should be possible to perform a system reboot, and have runner.jar execute automatically.

5) With the the new Malware virtual machine selected, click the Pause button to suspend the virtual machine.

Then click the “Take a snapshot of this virtual machine” icon in the VMware vCenter client toolbar.

Name the snapshot “Before Attack Scenario”. Note that AES will only accept 1 snapshot named “Before Attack Scenario”.

The Malware virtual machine is now ready for use with AES. Proceed to the next section to add the new Malware system to the Experiment Manager for experimentation use.

3.5.3.5 Adding New Malware Virtual Machines to AES

Once a new Malware virtual machine has been created, prepared and staged, the AES Experiment Manager must be updated to be aware of it for experiment and reporting purposes.

1) Determine the new Malware system name that was used in section 3.5.3.1 “Creating a New Virtual Machine”, step 3. E.g. MalwareWindows7

The Malware system name will be used repeatedly in the proceeding steps.

2) From the AES Management client, login to the VMware vCenter client and select the Experiment Manager virtual machine.

3) Access the Experiment Manager virtual machine console by clicking the “Console” tab in the right pane.

4) Open Windows Explorer and navigate to C:\aes\etc\Templates\experiments. Make a copy of the VMWinXPProSP3.xml template and name it based on the new Malware system name. e.g. MalwareWindows7.xml


5) Edit the new template and save the following changes:

Change the displayname attribute to reflect the name of the Dynamic experiment virtual machine as it will appear in the Experiment Manager Web interface. e.g. Windows 7

<experiment displayname="Windows 7" …>

The existing filetypesupport attribute can be used as is. However, if the new Malware virtual machine includes application support for file types other than those listed, it may be modified for those file extensions. The AES agent will execute the submitted Malware as if it were double-clicked by a user.

Additionally, the Experiment Manager Web interface will automatically display any new file types, and control the Malware virtual machines which support a selected file type.

To add or modify the new Malware system support for experiment malware file types, locate the filetypesupport attribute and modify its value according to the desired malware file extension.

For example, the following file types could be for a Malware virtual machine which includes Microsoft Office 2010 and Microsoft Visio:

<experiment displayname="Windows 7" filetypesupport="exe,pdf,doc,xls,ppt,bat,

docx,xlsx,pptx,vsd" ...>

Search and replace all instances of MalwareWinXPProSP3 with the new Malware name. e.g. replace MalwareWinXPProSP3 with MalwareWindows7.

<actor id="MalwareWindows7" >

...

<step id="S8 ... actor="MalwareWindows7" ...>

6) Navigate to C:\aes\etc\Templates\reports. Make a copy of the VMWinXPProSP3.txt template and name it based on the new Malware name. e.g. MalwareWindows7.txt


7) Edit the summary.txt file and save the following changes:

In the [Experiments] section under DYNAMIC, add a new line using the following format where <name> is the new Malware name from previous sections, and <display name> is the new Malware system name for reporting purposes:

[{EXPERIMENT_<name>}] <display name>

Note that it is important the Malware name be uppercase.

For example:

[Experiments]

…

DYNAMIC

[{EXPERIMENT_VMWINXPPROSP3}] Windows XP Pro SP3

[{EXPERIMENT_MALWAREWINDOWS7}] Windows 7

Under the DYNAMIC section at the bottom, add a new report using the following format where <name> is the new Malware name from previous sections, and <display name> is the new Malware system name for reporting purposes:

[<display name>]

{REPORT_<name>}

====================================================================

Note that it is important the Malware name be uppercase.

For example: ====================================================================

DYNAMIC

====================================================================

[Windows XP Pro SP3]

{REPORT_VMWINXPPROSP3}

====================================================================

[Windows 7]

{REPORT_ MALWAREWINDOWS7}

====================================================================

8) Navigate to C:\aes\etc\ExperimentConfig. Make a copy of the VMWinXPProSP3 directory and name it based on the new Malware name. e.g. MalwareWindows7


9) Edit c:\aes\etc\VLab\ftp-channel.conf and duplicate the last line in the file and modify it to reflect the Malware name.

MalwareWinXPProSP3,192.168.68.921,Administrator,password1,/vlab/,/

MalwareWindows7,192.168.68.921,Administrator,password1,/vlab/,/

10) (Optional) Navigate to C:\aes\etc\Templates\profiles. Make a copy of any .yml file and name it based on the new Malware name. e.g. MalwareWindows7.yml

Edit the copied .yml file as you see fit. Its contents will be displayed in the ? icon next to the new Malware experiment in the Dynamic Experiment section of the Experiment Manager Web interface.

11) Reboot the Experiment Manager virtual machine, or manually restart the AES Service by closing the runner.jar Console window and clicking the AES-MAS shortcut on the Desktop.

12) The Experiment Manager virtual machine has now been modified. Save the changes made by taking a snapshot.

Proceed to section 2 “User Guide” for guidance on conducting malware experiments. Confirm that the new Malware system is available for experimentation and test the new Malware system with all included malware sample files to confirm it is compatible with AES.

See section 2.1.5 “Malware Samples” for more information on test malware samples included with AES.


4 FAQ and Troubleshooting

4.1 Known Issues and Limitations

4.1.1 OpenDPI Deep Packet Inspection

The OpenDPI Deep Packet Inspection engine used by AES is limited to identifying known application layer protocols used in TCP and UDP traffic flows. In the event that the submitted malware employs an unknown protocol for communication, the traffic will not be identified by OpenDPI.

4.1.2 Document Scanning

AES cannot monitor internal Malware system activity perpetrated by submitted malware samples which are embedded in document files, such as .doc or .pdf. This type of monitored process activity is only available when submitting executable malware samples. The executable malware process activity such as process identifiers, spawned processes and utilization are normally included in the experiment report under the “Behaviour” section.

This limitation is due to how the AES post-processes the Procmon log files.

4.1.3 Known experiment errors

In the event that the AES system experiences a runtime, communication or file error, it will likely be captured in the experiment log file. The experiment report will be displayed in a red font to indicate an error has occurred. Effort has been made to correctly identify and highlight errors in experiment log files. However it is recommended that experiment log files be reviewed for possible errors or incorrectly identified error cases.

From the Experiment Manager Web interface, click the log icon to view the log. Locate the error by searching for any Java exceptions.

When diagnosing an AES experiment log exception, consider that it is possible for the submitted malware sample to interrupt the experiment process in a way that can cause the AES Service to fail unexpectedly. The AES Service will revert all virtual machine states when an experiment begins. In most cases, retrying the experiment can remedy the issue. In rarer cases, restarting the AES Management virtual machine or AES Infrastructure and retrying the experiment may work. In the worst case, it may not be possible to successfully complete the experiment with the malware sample. This can occur if the malware prevents AES services from operating as normal.


The following sample log exceptions can be resolved by retrying the experiment. They are related to the provisioning and configuration of VMware machines.

[ ERROR ] RemoteException in findByPath. Reconnect.

[ ERROR ] AxisFault

faultCode: ServerFaultCode

faultSubcode:

faultString: The session is not authenticated.

faultActor:

faultNode:

faultDetail:

{urn:vim25}NotAuthenticatedFault:<object type="SearchIndex">ha-

searchindex</object><privilegeId>System.View</privilegeId>

[ ERROR ] The session is not authenticated.

[ ERROR ] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

...

VMware API stale session error

[ ERROR ] Unexpected behaviour:

[ ERROR ] Reconnecting...

[ ERROR ] RemoteException in reconnect.

[ ERROR ] AxisFault

faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException

faultSubcode:

faultString: java.net.SocketException: Connection reset

faultActor:

faultNode:

faultDetail:

{http://xml.apache.org/axis/}stackTrace:java.net.SocketException: Connection reset

at java.net.SocketInputStream.read(Unknown Source)

...

VMware API connection broken error


If the AES Service or deployed agents are unable to collect required experiment data or there is some dependant process which failed that AES does not have absolute control over but is required, it will produce a fatal error case. The following is an example log file where a required file was not available to an experiment step possibly due to a timing issue, communication failure, or dependant process being interrupted. This type of error case may be resolved be retrying the experiment.

[ USER ] Initializing experiment

[ USER ] Step: AES1 "Setting up VM configurations" (Fri Jun 29 08:49:23 EDT 2012) ...

Done (2s)

[ USER ] Step: AES2 "Validating NICs" (Fri Jun 29 08:49:25 EDT 2012) ... Done (0s)

[ USER ] Step: AES3 "Validating snapshots" (Fri Jun 29 08:49:26 EDT 2012) ... Done (1s)

[ USER ] Step: AES4 "Setting snapshots" (Fri Jun 29 08:49:27 EDT 2012) ... Done (13s)

[ USER ] Step: AES5.1 "Powering on VMs" (Fri Jun 29 08:49:41 EDT 2012) ... Done (1m 20s)

[ USER ] Step: AES5.2 "Setting up CD-ROMs" (Fri Jun 29 08:51:01 EDT 2012) ... Done (0s)

[ USER ] Step: AES5.3 "Setting up NICs" (Fri Jun 29 08:51:01 EDT 2012) ... Done (16s)

[ USER ] Step: AES6 "Opening channels" (Fri Jun 29 08:51:18 EDT 2012) ... Done (0s)

[ USER ] Step: AES7 "Creating output directories" (Fri Jun 29 08:51:18 EDT 2012) ... Done

(0s)

[ USER ] Running experiment

[ USER ] Step: S1 "Configuring /etc/hosts for DNAT" (Fri Jun 29 08:51:18 EDT 2012) ...

true (1s)

[ USER ] Step: S2 "Send DNAT Script" (Fri Jun 29 08:51:19 EDT 2012) ... true (1s)

[ USER ] Step: S3 "Send IP rule file" (Fri Jun 29 08:51:20 EDT 2012) ... true (2s)

[ USER ] Step: S4 "Run DNAT Script" (Fri Jun 29 08:51:22 EDT 2012) ... true (2s)

[ USER ] Step: S5 "Start DNAT Process" (Fri Jun 29 08:51:24 EDT 2012) ... true (22s)

[ USER ] Step: S6 "Configuring /etc/hosts for Honeypot" (Fri Jun 29 08:51:46 EDT 2012)

... true (12s)

[ USER ] Step: S7 "Start Honeypot scripts" (Fri Jun 29 08:51:58 EDT 2012) ...

[ ERROR ] java.io.FileNotFoundException: vlab\ExperimentOutputTemp\AES-MAS-EM01-

0\c942c5709a68a55a6c9b2df877489712(dataset=drop;)[honeypot;].log (The system cannot find

the path specified)

[ ERROR ] at java.io.FileOutputStream.openAppend(Native Method)

...

[ ERROR ] Task did not complete.

[ ERROR ] Stopping experiment.

...

AES Service missing file error


If the AES Service is not able to communicate with the agents on the experiment virtual machines, the log file will likely indicate a connection failure error message:

Connecting to SSH2 server 192.168.68.1:952.

ERROR: Connection failed. Connect() failed: Windows error 10060: A connection attempt

failed because the connected party did not properly respond after a period of time, or

established connection failed because connected host has failed to respond.

[ ERROR ] at

crc.utils.io.net.ssh.TunnelierSSHConnection.execOnce(TunnelierSSHConnection.java:182)

[ ERROR ] at

crc.utils.io.net.ssh.TunnelierSSHConnection.exec(TunnelierSSHConnection.java:116)

[ ERROR ] at

crc.utils.io.net.ssh.TunnelierSSHConnection.exec(TunnelierSSHConnection.java:43)

[ ERROR ] at

crc.utils.vlab.manager.VirtualRuntime.runCommand(VirtualRuntime.java:140)

[ ERROR ] at crc.utils.vlab.manager.VirtualRuntime.exec(VirtualRuntime.java:49)

[ ERROR ] at

crc.utils.vlab.manager.AbstractExperimentManager$1.run(AbstractExperimentManager.java:352)

[ ERROR ] doVirtualMachineStep throwing ExperimentException after Thread

[ ERROR ] Command : /cygdrive/c/Progra~1/ICW/bin/sh -c 'echo 10.92.65.1 aes-manager >>

/cygdrive/c/WINDOWS/system32/drivers/etc/hosts'

[ ERROR ] Machine : AES-MAS-EM01.MalwareWinXPProSP3

[ ERROR ] Timeout : 0

[ ERROR ] Output : vlab\ExperimentOutputTemp\AES-MAS-EM01-

0\3f14d3a6c4060d72722e1731631e6ff1(dataset=drop;)[malware;].log

...

AES Service communication error

AES currently performs internal host monitoring on the Malware VM using SysInternals Process Monitor (Procmon). When conducting an experiment with an .exe type malware sample, Procmon is used to collect the process related activities of the malware. In the event that Procmon crashes as a consequence of some malware behaviour or the computer reboots unexpectedly, its log file is unreadable because the symbol table is only written to disk when the process terminates gracefully. If a Procmon error is encountered and its log file is unreadable, the following error will be displayed in the experiment log file:

[ ERROR ] Procmon file corrupted

...

Procmon error

This is not considered a fatal error. Procmon is an optional component of .exe type malware experiments and the “Behaviour” section of the summary report will indicate that this information couldn’t be collected.

http://192.168.68.1:952/


4.2 Powering Down AES Server

When shutting down the AES server, the Experiment Manager virtual machine should be paused or gracefully shutdown first and there should not be any running or pending experiments. If it is an emergency shutdown and there are experiments running or pending, the Experiment Manager can be shutdown manually and the AES Service will terminate. If an experiment is in mid-process at the time of the AES Service shutdown, any experiment virtual machines will still be running. They should be paused or shutdown from the VMware vCenter client before shutting down the virtual infrastructure. The primary concern is loss of data as the experiment virtual machines are automated.

Each experiment virtual machine will include a machine state snapshot. These snapshots are required for AES to improve experiment time performance. Do not modify these snapshots or create new snapshots.

4.3 Machine Accounts and Console Access

If required, the following default accounts may be used to access AES physical and virtual systems.

AES Server (VMware vSphere ESX) Username: root Password: aesmas303 Experiment Manager (virtual machine) Username: administrator Password: aesmanager202 Malware (virtual machine) Username: administrator Password: password1 Static Scanner (virtual machine) Username: administrator Password: password1 Target (virtual machine) Username: administrator Password: password1 DNAT (virtual machine) Username: root Password: VMLinuxCentOS52DNAT


Honeypot (virtual machine) Username: root Password: VMLinuxFedora9Honeypot Experiment Firewall (virtual machine) Username: admin Password: pfsense101 Virtual machine console access can be reached from the VMware vCenter client:


2) Open the VMware vCenter client software and login to the AES Server VMware vSphere ESX server.

3) Right click on any powered on virtual machine, such as the Experiment Manager (AES-MAS-EM01) and select “Open Console”. A console window will open and remote access directly to the virtual machine may be accessed through it.


4.4 Backup and Restore

AES is designed to provide continuous operation for malware experimentation, but in the event of data loss, key aspects of the file system should be backed up. Procedures on backing up experiment reports and virtual machine systems have been included in the following sections.

Although backup of the AES virtual machine inventory is discussed, full VMware ESX operating system backup is not. The level of integrated automation with VMware ESX implemented by AES makes full ESX backup challenging. In the event of a hardware failure, the restored ESX would require an identical hardware configuration that is beyond the scope of this guide.

4.4.1 Experiment Reports

The Experiment Manager hard disk is 300GB and will allow for up to 800 experiment reports. Any experiments conducted past this limit will result in the oldest reports being deleted. It is the responsibility of the AES administrator to backup reports of interest to another location.

1) Log on to the Experiment Manager virtual machine console from the VMware vCenter client.

2) Open the File Explorer and navigate to D:\reports. All the report files will be located here in chronological order. Each experiment is stored in a sub-directory named after the time of the experiment which has been hashed.

3) Backup or delete any experiment files desired.

Under normal circumstances, AES should clean all temporary or operational directories. If recovering from low disk space, they should be inspected and cleaned manually if needed. When doing so, the AES Service should be stopped first and no experiments should be pending.


2) Stop the AES Service. See section 4.5.1 “Manually restarting the AES Service” for more information on how to do so.

3) Open the File Explorer and delete the following directories: D:\results\

D:\state\

D:\sources\

D:\channel\

D:\drop\

D:\config\

4) Start the AES Service. See section 4.5.1 “Manually restarting the AES Service” for more information on how to do so.

SSH/SFTP

AES Virtual Infrastructure

vms01

AES Management vms02

/vmfs/volumes/

SSH/SFTP

AES Virtual Infrastructure

vms01

vms02

USB Hard Drive

AES Management

http://www.bitvise.com/tunnelier


Ensure that no experiments are in process or pending before attempting to backup virtual machine files. The backup will require approximately 750GB of disk space and can take as long as 15 hours to complete using a USB 2.0 external drive.

1) Navigate to /vmfs/volumes/ directory.

2) Attach a 1TB or larger external storage device to the AES Management client machine to be used for external backup.

3) Transfer only the vms01 and manager directories over to the USB hard drive.

The vms02 directory contains a local copy of vms01 and backing it up is unnecessary. The esx directory cannot be easily restored and so backing it up is unnecessary.

4.4.2.3 Restoring from an External Drive

Ensure that no experiments are in process or pending before attempting to restore virtual machine files. Depending on the number of virtual machine files being restored, the file transfer can take as long as 15 hours to complete.

From the AES Management client machine:

1) Login to VMware vCenter client and suspend to power down all virtual machines.

2) Attach the external storage device containing the AES virtual machine file backup to the AES Management client machine.

3) Connect to the AES Infrastructure over SFTP. See 4.4.2.1 “Connecting to the AES Infrastructure” for more information on doing so.

4) Transfer target backup directories from the USB hard drive to the original VMware ESX local store.

It is possible to only copy specific virtual machine directories located under the vms01 directory. In the event that a damaged virtual machine needs to be restored, only copy the vms01 virtual machine sub-directory from the USB hard drive to the /vmfs/volumes/vms01 directory.

For example, to restore a damaged Experiment Manager virtual machine from the USB hard drive, copy the Hermione-EM7 virtual machine sub-directory from the manager directory to the AES Infrastructure manager local data store.

To restore a damaged AVG 2012 Static Experiment (AES-MAS-EM01-AVAVG2012) from the USB hard drive, copy the Hermione-EM7.VMWinXPProSP3AVG2012AV virtual machine sub-directory from the vms01 directory to the AES Infrastructure vms01 local data store.


5) From the VMware vCenter client, select the Experiment Manager virtual machine (AES-MAS-EM01), and click the Power on icon. Test all restored virtual machines by conducting an experiment which utilizes them.

4.4.2.4 Backing up to Secondary Storage

When backing up AES experiment virtual machines to the second local storage hard drive, first connect to the AES Infrastructure over SFTP. See 4.4.2.1 “Connecting to the AES Infrastructure” for more information on doing so.

Ensure that no experiments are in process or pending before attempting to backup virtual machine files.

1) Navigate to /vmfs/volumes/ directory.

2) Transfer all sub-directories from the vms01 directory to the vms02 directory.

It is possible to only backup specific virtual machine directories located under the vms01 directory. In the case where a specific virtual machine has been updated and a local backup is desired; only the virtual machine in question need be copied.

For example, when updating Static experiment or Dynamic Malware experiment virtual machine software, the change should be backed up to the secondary vms02 location. Copy the updated virtual machine sub-directory from the vms01 directory to the vms02 directory and replace the old virtual machine backup sub-directory.


4.4.2.5 Restoring from Secondary Storage

Ensure that no experiments are in process or pending before attempting to restore virtual machine files. Depending on the number of virtual machine files being restored, the file transfer can take as long as 1 hour to complete.

From the AES Management client machine:

1) Login to VMware vCenter client and suspend to power down all virtual machines.

2) Connect to the AES Infrastructure over SFTP. See 4.4.2.1 “Connecting to the AES Infrastructure” for more information on doing so.

3) Copy target virtual machine directories from the secondary hard drive store to the original VMware ESX local store.

It is possible to only copy specific virtual machine directories located under the vms01 directory. In the event that a damaged virtual machine needs to be restored, only copy the vms01 virtual machine sub-directory from the USB hard drive to the /vmfs/volumes/vms01 directory.

For example, to restore a damaged AVG 2012 Static Experiment (AES-MAS-EM01-AVAVG2012) from the secondary local storage backup, copy the Hermione-EM7.VMWinXPProSP3AVG2012AV virtual machine sub-directory from the vms02 directory to the vms01 local data store.

4) From the VMware vCenter client, select the Experiment Manager virtual machine, AES-MAS-EM01, and click the Power on icon. Test all restored virtual machines by conducting an experiment which utilizes them.


4.5 Recovering from an AES Service failure

The Experiment Manager AES Service is designed to be resilient to experiment failures and produce experiment log files with the reason. In the event that the AES Service itself fails, it will be unable to produce the experiment report or continue with any pending experiments.

AES is capable of experimenting with various malware that can affect the experiment systems and service in negative ways. AES will then continue with any pending experiments. Experiment virtual machines should not require any manual intervention. At the beginning of each experiment, all relevant virtual machines will be reverted to a clean state before continuing.

If the AES Service itself fails due to an unexpected error during an experiment, the service status indicator will show that AES is not running while at the same time displaying the experiment progress.

The user or debug log may not include an exception error as the AES Service may not have been able to log the error before failing.

To recover from this case, the AES Service will need to be restarted. See 4.5.1 “Manually restarting the AES Service” for information on this procedure.

After being restarted, AES will automatically attempt to restart the experiment. If the same experiment repeatedly causes the AES Service to fail, it can be manually removed from the experiment queue. See 4.5.2 “Manually removing pending experiments” for information on this procedure.


4.5.1 Manually restarting the AES Service

To manually stop the AES Service:

1) Log on to the Experiment Manager* virtual machine console from the VMware vCenter client.

2) Locate the AES-MAS Console window and type Control-C or close the window.

To manually start the AES Service:


2) Reboot the Experiment Manager virtual machine, or manually start the AES Service by clicking the AES-MAS shortcut on the Desktop.



4.5.2 Manually removing pending experiments

Pending experiments waiting to be executed can be removed from the Experiment Manager Web Interface by clicking the X icon next to the experiment in the Pending section.

In the event that a failed or pending experiment needs to be removed from the AES experiment queue manually, complete the following steps:

1) From the Experiment Manager Web interface, note the MD5 field of the running experiment. For example, bdabb4aed559f54fefaffb435692844d.


3) Open the File Explorer and navigate to D:\sources. Search and delete all directories that match the MD5 value observed in step 1.

Once the experiment folders are deleted, the AES Service will not be able to run the experiment again.


4) Reboot the Experiment Manager virtual machine, or manually start the AES Service by clicking the AES-MAS shortcut on the Desktop.


4.6 Malware Submission Failure When submitting malware samples to the AES Experiment Manager Web interface, the Apache service may need to be restarted if the Internet browser does not complete the file upload within two minutes. This will not affect AES Service running or queued experiments.


2) Locate the XAMPP Control Panel Application icon in the system tray and open it.

3) Stop and Start the Apache service module.

4) From the AES Management system, enter the AES Experiment Manager Web interface. It is recommended that the web form data not be resubmitted, but instead the user reloads the page and fills out the submission form again.



4.7 Experiment Virtual Machines Fail To Start

When VMware virtual machines are moved or modified, the VMware ESX software will acknowledge this and offer to create a new and unique network interface MAC address. This prompt will interrupt the AES Service’s automation process and put the powered on experiment virtual machine in a 95% start up state.

If five minutes has passed after starting an experiment, without the AES Experiment Manager Web interface displaying any continued Running progress, it is possible an unexpected or accidental machine change has occurred.


2) At the bottom of the vCenter client in the Recent Tasks section, inspect if a Recent Task named “Power On virtual machine” is stuck at 95%.

In the machine inventory list, inspect the Target virtual machine for an information mark bubble icon. The information mark bubble icon indicates if user input is required to complete the current Power on operation.


Select the Target machine from the inventory list and click the “Summary” tab. If a user input action is required, the prompt will be available here. Select “I moved it” and click OK. The virtual machine will complete the operation and the experiment should proceed without further input.

If “I copied it” was accidentally selected, it is possible a second set of network interfaces on the virtual machine will be created for the new generated MAC address. An operating system typically assigns a network configuration to a network interface by MAC address, and changing it can interfere with the AES Service automated configuration.

In order to recover from this mistaken case, select the virtual machine Snapshot Manager icon and revert the virtual machine to the snapshot named “Before Attack Scenario”. If the power on issue reoccurred, select “I moved it” this time.




DOCUMENT CONTROL DATA (Security classification of title, body of abstract and indexing annotation must be entered when the overall document is classified)

1. ORIGINATOR (The name and address of the organization preparing the document.Organizations for whom the document was prepared, e.g. Centre sponsoring a contractor's report, or tasking agency, are entered in section 8.)

Christopher McKenzieAEPOS Technologies Corporation116 Albert St., Suite 600Ottawa, Ontario K1P 5G3

2. SECURITY CLASSIFICATION (Overall security classification of the document including special warning terms if applicable.)

UNCLASSIFIED(NON-CONTROLLED GOODS)DMC AREVIEW: GCEC JUNE 2010

3. TITLE (The complete document title as indicated on the title page. Its classification should be indicated by the appropriate abbreviation (S, C or U) in parentheses after the title.)


4. AUTHORS (last name, followed by initials – ranks, titles, etc. not to be used)

McKenzie, C.

5. DATE OF PUBLICATION (Month and year of publication of document.)

November 2013

6a. NO. OF PAGES (Total containing information, including Annexes, Appendices, etc.)

90

6b. NO. OF REFS (Total cited in document.)

0 7. DESCRIPTIVE NOTES (The category of the document, e.g. technical report, technical note or memorandum. If appropriate, enter the type of report, e.g.

interim, progress, summary, annual or final. Give the inclusive dates when a specific reporting period is covered.)

Contract Report

8. SPONSORING ACTIVITY (The name of the department project office or laboratory sponsoring the research and development – include address.)Defence R&D Canada – Ottawa3701 Carling AvenueOttawa, Ontario K1A 0Z4

9a. PROJECT OR GRANT NO. (If appropriate, the applicable research and development project or grant number under which the document was written. Please specify whether project or grant.)

15bb02

9b. CONTRACT NO. (If appropriate, the applicable number under which the document was written.)

W774-115223

10a. ORIGINATOR'S DOCUMENT NUMBER (The official document number by which the document is identified by the originating activity. This number must be unique to this document.)

10b. OTHER DOCUMENT NO(s). (Any other numbers which may be assigned this document either by the originator or by the sponsor.)

DRDC Ottawa CR 2013-101

11. DOCUMENT AVAILABILITY (Any limitations on further dissemination of the document, other than those imposed by security classification.)

Unlimited

12. DOCUMENT ANNOUNCEMENT (Any limitation to the bibliographic announcement of this document. This will normally correspond to theDocument Availability (11). However, where further distribution (beyond the audience specified in (11) is possible, a wider announcement audience may be selected.)) Unlimited

13. ABSTRACT (A brief and factual summary of the document. It may also appear elsewhere in the body of the document itself. It is highly desirable that the abstract of classified documents be unclassified. Each paragraph of the abstract shall begin with an indication of the security classification of the information in the paragraph (unless the document itself is unclassified) represented as (S), (C), (R), or (U). It is not necessary to include here abstracts in both official languages unless the text is bilingual.)

This is the user and administrator guide for the Automated Experimentation System (AES) - Malware Analysis System. It is intended to cover end-user operation of AES for malware analysis, and administrator configuration, system design, and troubleshooting.

Ce document est le manuel de l’utilisateur et du gestionnaire pour le Système d’Expérimentation Automatisé (SEA) – Système d’Analyse de Logiciel Malveillant. Les sujets couverts sont le fonctionnement pour l’utilisateur, la configuration pour le gestionnaire du système, la conception du système et le dépannage.

14. KEYWORDS, DESCRIPTORS or IDENTIFIERS (Technically meaningful terms or short phrases that characterize a document and could be helpful in cataloguing the document. They should be selected so that no security classification is required. Identifiers, such as equipment model designation, trade name, military project code name, geographic location may also be included. If possible keywords should be selected from a published thesaurus, e.g. Thesaurus of Engineering and Scientific Terms (TEST) and that thesaurus identified. If it is not possible to select indexing terms which are Unclassified, the classification of each should be indicated as with the title.)

malware analysis, automated experimentation, user guide, administrator guide