Memory Forensics for IR – Leveraging Volatility to Hunt Advanced Actors

Jared GreenhillNovember 4 2014

#Personal Bio Jared Greenhill (@jared703)

Volatility user since 2012.

MS. In Computer Forensics from George Mason University.

cfrs.gmu.edu

#Intro

In July 2013 a global Non-Profit reached out to us about a FBI notification they received.

We asked for Firewall Logs and/or a memory image for a related host if available.

We quickly went from knowing little to understanding much more because of Volatility.

#Hunting for APT using VolatilityVolatility for IR – Specifically:

– Timelining (MFT/Registry)– ShimCache Review– Dumping Scheduled Job artifacts– Dumping Malware

Why is this case so critical to IR?– Speed of initial triage– One memory image broke things open– Find indicators to use across the

environment

#Overview FBI notification in Early July 2013 that potential data exfiltration was likely coming from an IP from their network range.

#Overview & HistoryA Firewall log revealed ~1 GB of data leaving the customer’s network to suspect IP through two ~.5GB transactions on June 13th 2013, originating from a suspect internal host… 

We reviewed the ASA log structure and began some greping for data going outbound/potential exfil.

Customer indicated that they did not have any Familiarity with this IP address.

#Firewall Log Reviewgrep 'bytes [0-9]\{9\}’ SyslogFW.log

#Overview & History Shortly after, the client provided

memory in the form of a .VMSN file for the Host after our FW log analysis confirmed the FBI’s notification.

Luckily for us, Nir Izraeli’s VMSNparser had recently been adopted for the Beta version of Volatility 2.3. in May 2013

#Initial Analysis Steps Determine the Server type for correct profile use.

– Volatility is profile based as data structures vary through OS versions.

vol.py –f host.raw imageinfo– Suggested Profile(s) : Win2003SP0x86, Win2003SP1x86, Win2003SP2x86 (Instantiated with WinXPSP2x86)

– Image date and time : 2013-07-17 14:51:22 UTC+0000– Image local date and time : 2013-07-17 10:51:22 -0400

▪ Client indicated the host was a Win2k3 SP2 server.▪ Confirmed w/Imageinfo

#Initial Analysis Continued Everyone has investigative techniques/bias.

I started by looking for Interesting Processes and Network communications related to the 6/13/13 exfil date.

▪ vol.py -f host.raw --profile Win2003SP2x86 yarascan --wide --yara-rules=“2xx.xx5.x2.9”

▪ vol.py -f host.raw --profile Win2003SP2x86 pslist— Most processes were started on 6/25/13, indicated it could have been rebooted then.— The Exfil date was 6/13/13, so we could have lost things in memory during the reboot

which likely occurred on 6/25/13 based on process start times.

▪ Vol.py -f host.raw --profile Win2003SP2x86 sockets▪ vol.py -f host.raw --profile Win2003SP2x86 connections

#Connections Connections were internal/RFC1918, except one - 198.55.120.205:80 connection which was benign.

#Triaging ShimCache• ShimCache (AppCompactCache/AppCompatability Reg keys) – data resides in the

SYSTEM hive

• Records file path, size, last modified time, last execution time

• Parses the HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\AppCompatability Registry Key for XP

• Parses the HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\AppCompatCache key for Server 2003/Vista/Win7/Server 2008

• Command:• vol.py -f host.raw --profile Win2003SP2x86 shimcache

#Triaging ShimCache: APT Problems…Redacted_2012_9—2013-3-27.log file – a quick google of the customer and the filename revealed that this person worked in the Chinese arena; explainingAPT interest in their email. APT tool activity appeared to go back to July 12, 2012…

#Looks like APT• At this point we gave the client our initial triage findings:

FW logs confirming the FBI’s exfil notification:• ~1 GB out the door.

ShimCache results:• Likely APT tool use.• Likely exfil of an Employee in Chinese relations found in the filename.• Likely tool activity since (at least) July 2012

• The client engaged us for IR… 3 months later (Oct 2013).• I kept focusing on this memory sample…

#Likely APT – Scheduled Jobs?• Schedlgu.txt – Windows task scheduler output log from at.exe

• lists scheduled jobs executed on a system. (C:\Windows\Tasks)

• Volatility can be leveraged to extract this file from memory • Use the “Handles” plugin to find the virtual offset of “schedlgu.txt”:

• vol.py -f host.raw --profile Win2003SP2x86 handles | grep -i schedlgu.txt

• 0x8a744028 1156 0x334 0x12019f File \Device\HarddiskVolume1\WINDOWS\Tasks\SchedLgU.Txt

• Use the virtual offset with the “filescan” plugin to find the physical offset.• vol.py -f host.raw --profile Win2003SP2x86 filescan | grep -i

schedlgu.txt• 0x0a744028 1 1 RW-r-- \Device\HarddiskVolume1\WINDOWS\

Tasks\SchedLgU.Txt

#Scheduled Jobs Cont’d• The 0x0A744028 address is the physical offset of SchedLgU.Txt in

memory. We can now use the “dumpfile” plugin to dump “Schedlgu.txt” from memory.• vol.py -f host.raw --profile=Win2003SP2x86 dumpfiles -Q

0x0a744028 –D.• This command dumps the Schedlgu.txt (file.None.0x8a7385e0.dat)

file from the physical location 0x0a744028 to the current working directory (denoted by the “.”). Output in a text editor:

• sx86.exe executed successfully on 6/25/13 via AT1.job

#Timelining with Volatility

#Timelining Cont’d• At this point I had a timeline in CSV format (timeline.csv) that we can

GREP or turn into an Excel spreadsheet. • It’s critical to have a place to start. Knowing our suspected exfil event

occurred on 6/13/13, the 6/25 AT1.job that executed successfully was a solid starting point.

• My first step was to grep/search for “sx86.exe” from “Schedlgu.txt”. • Unfortunately there was no entry for “sx86.exe”.

• Next, I looked for other items of interest on or around 6/25/2013 at 12:15:00 AM in the timeline…

#Timelining Cont’dRoughly 23 minutes earlier (6/24/2013 11:53 PM) I saw a $MFT entry for “WINDOWS\Temp\trbsgmxq_kl.dll”. Suspect location for a legitimate DLL to reside?

#Timelining Cont’d – trbsgmxq_kl.dll• Next, I grep’d for “trbsgmxq_kl” across my CSV timeline.• Looks like it is running is two processes – PID’s 376 & 1156

#Using “Dlllist” to see if the DLL is running• Checked if the DLL existed in any processes:• vol.py –f host.raw --profile=Win2003SP2x86 dlllist

Using “memdump” to dump processes• The “memdump” command extracts all memory from a running process:

• vol.py -f host.raw --profile=Win2003SP2x86 memdump -p 376 -D dump/

• vol.py -f host.raw --profile=Win2003SP2x86 memdump -p 1156 -D dump/

Using “memdump” to dump processesNext, use the linux strings command to pull out strings from the two running processes:

• strings -a 376.dmp > strings376.txt (creates)• strings -a -e l 376.dmp >> strings376.txt (appends)

• strings -a 1156.dmp > strings1156.txt• strings -a –e l 1156.dmp >> strings1156.txt

PID 376 did not have the same interesting strings that PID 1156 did.

Strings Analysis related to trbsgmxq_kl.dllInteresting strings from the “svchost.exe” process (PID 1156) :

• C:\WINDOWS\Temp\trbsgmxq_kl.dll• C:\WINDOWS\TEMP\RarSFX0\AROTutorial.exeo• C:\WINDOWS\Temp\iismgr.dat

Interesting API Strings which indicated trbsgmxq_kl.dll had Keylogging activity:• GetWindowTextA (Copies the text of the specified window's title bar)• EnableWindow (Enables or disables mouse and keyboard input to the specified window or control. When input is enabled, the window receives all input.)  • GetKeyState (Retrieves the status of the specified virtual key. The status specifies whether the key is up, down, or toggled (on, off—alternating each time the key is pressed).• GetSystemMenu (returns a handle to the copy of the window menu currently in use)

Dumping trbsgmxq_kl.dll from memoryUse DLLLIST to obtain the physical offset:

vol.py -f host.raw --profile=Win2003SP2x86 dlllist | grep trbsgmxq_kl.dll

Now that we have the physical offset, 0x06370000

I dumped the DLL: vol.py -f host.raw --profile=Win2003SP2x86 dlldump --base=0x06370000 -D dump/

#Dumping trbsgmxq_kl.dll from memorySanity check to make sure we have a DLL in our dump directory.

Analysis of this DLL confirmed that it was a keylogger which captured keystrokes to the file “iismgr.dat” at the location “C:\WINDOWS\Temp” and was single byte XOR’d w/ 0xC2.

#Next Steps• Memory analysis yielded multiple high fidelity indicators! We used these to hunt during our IR efforts…

• Keylogger - trbsgmxq_kl.dll which logs keystrokes to iismgr.dat

• Scheduled Task “AT1.job” which executed sx86.exe

• Various .log files (x.log,error.log, victim_2012_9—2013-3-27.log) found in C:\WINDOWS\addins\

• C:\F5 - mm.exe, set.exe, dllhosts.exe, MyWce32.exe, wce32.exe

• C:\RECYCLER\gs.exe

#Pivot• IR finally began 3 months later. Goal is to answer everything possible.• 2xx.xx5.8x.9 Exfil…? Yeah, it happened.

• Searching through Disk (memory.dmp crash file) revealed that “cssrs.exe” was executed at the suspect Exfil time from the FW logs – “cssrs.exe 2xx.xx5.8x.9 443” on Jun 13 2013 @ 13:26

#2xx.xx5.8x.9 / CSSRS.exe• Analysis of cssrs.exe revealed that it was Trojan.Hikit, a full featured Remote Access Tool

(RAT).

• That same crash file showed the attacker packaging up the previously mentioned employees PST file with a tool “csss” on 6/13/13 at 9:11am ~4 hours before it likely was exfiled.

#Wrap Up• June 25 2013 AT1.job -- C:\

sx86.exe dropper for Trojan FF-RAT:• frtest.dat & Windows Config.wav

which we refer to as “Trojan-FF-RAT”.

• Signed malware “Xuzhou Chenju Technology Co.,Ltd

• We are still seeing malware signed by this signed cert serial # via Yara/VT.

#Wrap Up• Memory was taken on 7/17/13 @ 10.51am EST.

• After getting the disk image a few months later, the keylogger output file “iismgr.dat” was created on the system on 7/17/13 @ 11:28 am EST.

• APT actors were active on the system and had domain admin creds 37 minutes later through the trbsgmxq_kl.dll keylogger…

#Total Damage

• Total of 8 Unique Trojan families, several of which were signed.

• We found evidence of multiple actor groups, no conclusive attribution.

• 18% of endpoints had APT malware, or were accessed laterally by APT.

• Keystroke loggers and GUI access (RDP & RAT’s).

• Total compromise ~3 years.

#Malware SummaryDroppers

• SX86.exe - dropper for Trojan FF-RAT (frtest.dat & Windows Config.wav)

• mm.exe – unrecovered• set.exe - unrecovered

RATS• cssrs.exe – Trojan.Hikit• FF-RAT (frtest.dat &

Windows Config.wav)

Keylogger• trbsgmxq_kl.dll• iismgr.dat (output file)

Scheduled Jobs• AT1.job SX86.exe on

June 25 13