Memory Forensics for Incident Response

Jared Greenhill – RSA IR

@jared703

Memory Forensics for IR – Leveraging Volatility to Hunt Advanced Actors

2© Copyright 2015 EMC Corporation. All rights reserved.

#Bio

• Member of RSA’s IR Team for 2+ years

• Former malware analyst at US-CERT

• M.S. Computer Forensics from George Mason University

• Teaching Grad Level Memory Forensics Course Spring ‘16

• CFRS.GMU.EDU


#Agenda

• Memory Forensics & IR Criticality

• Case Intro, Overview and Discussion

• Memory Analysis – Volatility Usage and APT Artifacts

• Pivot from Memory to Disk Based Artifacts

• Case Summary / Overall Findings / Additional Work


#Memory Forensics & IR Criticality

– Memory triage can provide timely answers in IR & Intrusions.

– Memory acquisition usually fast due to size

– The goal is to quickly determine if badness occurred on a host.



– If yes, what type of badness?

– Can we map it to our initial detection?• Network Hunting, IDS/IPS event, AV hit,

3rd party notification

– Is there Malware?

– If so, Commodity or Targeted?



• Why is this case so critical to IR?

– Speed of initial triage

– One memory image broke things open

– Discovery of critical indicators to sweep across the environment



• Volatility for IR – Specifically:

– ShimCache parsing

– Dumping scheduled jobs & artifacts

– Timelining ($MFT/Registry/Process creation)

– Dumping Malware from memory


#Case Intro• In July 2013 a global

Non-Profit reached out to us about a FBI notification they received.

• We asked for Firewall Logs and/or a memory image for a related host if available.

• We quickly went from knowing little to understanding much more through Memory Forensics and Volatility.

Discover New compromises

Notification July 2013

Firewall Log Review

Develop Network & Host based Signatures

Memory Analysis

Malware Analysis

Host Based Forensics



– Memory can provide a wealth of critical data:

– Network Connections • Active connections

– Critical to collect memory when system is on the network

• Residual connections sometimes exist

– Malware • Injected into a process• Can exist in both memory and on disk• $MFT can provide context


#Case Overview• FBI notification came

in Early July 2013 (not their 1st).

• Data exfiltration was likely coming from an IP in their IP space.

• June 2013 was the timeframe provided.


#Overview & History

• A co-worker and myself reviewed the ASA log structure and greping for outbound data.

• We discovered ~1 GB of data leaving outbound to 206.205.82.9 through two ~.5GB transactions on June 13th 2013, originating from a suspect internal host…


#Firewall Log Review

grep 'bytes [0-9]\{9\}’ SyslogFW.log


#Overview & History

Shortly after, the client provided memory in the form of a .VMSN file for the Host after our FW log analysis confirmed the FBI’s notification.

Luckily Nir Izraeli’s VMSNparser had recently been adopted for the Beta version of Volatility 2.3. in May 2013


#Initial Analysis Steps

Determine the Server type for correct profile use.– Volatility is profile based as data structures vary through OS

versions.

vol.py –f host.raw imageinfo– Suggested Profile(s) : Win2003SP0x86, Win2003SP1x86,

Win2003SP2x86 (Instantiated with WinXPSP2x86)– Image date and time : 2013-07-17 14:51:22 UTC+0000– Image local date and time : 2013-07-17 10:51:22 -0400

• Client indicated the host was a Win2k3 SP2 server.• Confirmed w/Imageinfo


#Initial Analysis Continued

Everyone has investigative techniques/bias.

• I started by looking for Interesting Processes and Network communications related to the 6/13/13 exfil date.

– vol.py -f host.raw --profile Win2003SP2x86 yarascan --wide --yara-rules=“206.205.82.9”

– vol.py -f host.raw --profile Win2003SP2x86 pslist• Most processes were started on 6/25/13, indicated it could have been

rebooted then.• The Exfil date was 6/13/13, so we could have lost things in memory

during the reboot which likely occurred on 6/25/13 based on process start times.

– Vol.py -f host.raw --profile Win2003SP2x86 sockets– vol.py -f host.raw --profile Win2003SP2x86 connections


#Network Connections?Connections were internal/RFC1918, except one - 198.55.120.205:80 – connection was legitamate.


#Triaging ShimCache• ShimCache (AppCompactCache/AppCompatability Reg keys) – data

resides in the SYSTEM hive

• Records last execution time (depends on OS) file path, size, last modified time

– Win XP Registry Key: • HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\

AppCompatibility\AppCompatCache

– Win 2003/Vista/Win7/Server 2008 Registry Key:• HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\

AppCompatCache\AppCompatCache

vol.py -f host.raw --profile Win2003SP2x86 shimcache

http://www.mandiant.com/library/Whitepaper_ShimCacheParser.pdf


#Triaging ShimCache: APT Problems?

It appeared that APT tool activity went back to at least July 2012with the files “set.exe”, Gs.exe & MyWce32.exe.

Last Modified Path------------------------------ ----2008-11-02 05:02:34 UTC+0000 \??\C:\WINDOWS\addins\1.exe2010-12-08 10:45:00 UTC+0000 \??\C:\WINDOWS\addins\gsec1.exe2010-04-27 14:04:06 UTC+0000 \??\C:\WINDOWS\addins\psloglist.exe2010-04-27 15:04:04 UTC+0000 \??\C:\WINDOWS\addins\PsLoggedon.exe2010-04-27 15:04:04 UTC+0000 \??\C:\WINDOWS\addins\PsList.exe2010-04-27 15:04:06 UTC+0000 \??\C:\WINDOWS\addins\p.exe2011-12-08 19:42:16 UTC+0000 \??\C:\WINDOWS\addins\csss.exe2011-06-24 20:42:18 UTC+0000 \??\C:\WINDOWS\addins\AIO_.exe2011-08-25 21:07:10 UTC+0000 \??\C:\WINDOWS\addins\lssas.exe2011-08-25 23:29:34 UTC+0000 \??\C:\WINDOWS\addins\sumlist.exe2012-12-28 15:12:00 UTC+0000 \??\C:\WINDOWS\addins\cssrs.exe


#Triaging ShimCache: Cont’dRedacted_2012_9—2013-3-27.log file – a quick Internet search of the ORG/filename revealed that this person was their China Director.

Last Modified Path------------------------------ ----2008-11-02 05:02:34 UTC+0000 \??\C:\WINDOWS\addins\1.exe2012-05-17 18:31:32 UTC+0000 \??\C:\WINDOWS\addins\FileTime.exe2012-07-12 16:49:32 UTC+0000 \??\C:\RECYCLER\gs.exe2012-08-11 05:08:14 UTC+0000 \??\C:\WINDOWS\addins\wce32.exe2012-08-28 19:17:52 UTC+0000 \??\C:\WINDOWS\addins\x.txt2012-10-17 05:24:36 UTC+0000 \??\C:\WINDOWS\addins\MyWce32.exe2013-01-09 12:47:41 UTC+0000 \??\c:\set.exe2013-03-11 14:10:10 UTC+0000 \??\C:\F5\mm.exe2013-03-26 18:09:37 UTC+0000 \??\C:\WINDOWS\addins\redact_2012_9--2013-3-27.log2013-03-26 18:10:42 UTC+0000 \??\C:\WINDOWS\addins\err.log2013-03-28 21:30:25 UTC+0000 \??\C:\WINDOWS\addins\x.log2013-03-28 19:34:51 UTC+0000 \??\C:\WINDOWS\addins\log.log


#Looks like APT…

At this point we gave the client our initial triage findings:

FW logs confirming the FBI’s exfil notification:– ~1 GB data went out the door on June 13, 2013.

ShimCache results:– Likely APT tool use based on suspect filenames/locations.– Likely exfil w/China Director’s name in the filename.– Likely active since at least July 2012, possibly since 2008.

• The client engaged us for IR… 3 months later (Oct 2013).

• I kept focusing on this memory sample…


#Scheduled Jobs in Memory?

Schedlgu.txt – Windows Task Scheduler output log from at.exe

– The output file lists the scheduled jobs executed on a system. – Located at C:\Windows\Tasks\Schedlgu.txt

Reviewing this file is a quick way to see if jobs executed on a host.


#Scheduled Jobs & APT Actors

Use the “Handles” plugin to find the virtual offset of “Schedlgu.txt”:• vol.py -f host.raw --profile Win2003SP2x86 handles |

grep -i schedlgu.txt • Offset 0x8a744028 1156 0x334 0x12019f File \Device\

HarddiskVolume1\WINDOWS\Tasks\Schedlgu.txt• Use the virtual offset with the “filescan” plugin to find

the physical offset.• vol.py -f host.raw --profile Win2003SP2x86 filescan |

grep -i schedlgu.txt• 0x0a744028 1 1 RW-r-- \Device\HarddiskVolume1\

WINDOWS\Tasks\SchedLgU.Txt


#Scheduled Jobs Cont’dThe physical memory offset of SchedLgU.Txt in memory. We can now use the “dumpfile” plugin to dump “Schedlgu.txt” from memory.• vol.py -f host.raw --profile=Win2003SP2x86 dumpfiles -Q

0x0a744028 –D.• This command dumps the Schedlgu.txt (file.None.0x8a7385e0.dat) file

from the physical location 0x0a744028 to the working directory. • The file sx86.exe was executed successfully on 6/25/13 via AT1.job.


#Scheduled Jobs Cont’d• Additional review of Schedlgu.txt

yielded more successful file executions and some failures:

• Set.exe on 1/9/13 @ 9:01AM

• Successfully executed• Log.bat on 9/12/12 @

2:27AM Failed.


#Scheduled Jobs & APT Actors…Scheduled Jobs are commonly used by APT actors to:

– Run batch scripts, execute malware/droppers & Move Laterally.

APT actors create Jobs to run at a certain time:– Hour/Minute are provided during job creation

• EX. 12:00AM – and they are run exactly at that time.

• These jobs leave execution time “00” on the seconds, which also stand out.

– Per Microsoft: • exit code (0) means that the scheduled task completed successfully.• exit code (1) means that the scheduled task failed. (Incorrect Function)


#Scheduled Jobs & ShimCache Comparison

These run at the highest privilege level “SYSTEM” by default.• The first manually created job is named “AT1.job”, the next is

“AT2.job”…etc• The job file is stored as a file, AT1.job, at C:\Windows\Tasks\AT1.job• These are deleted by design but can be found in slackspace• Job files reside in a proprietary binary format – many tools can parse

them…


#Timelining Memory with Volatility

• Timelining reins in the temporal data that exists in memory

• Volatility’s “timeliner” plugin

• Provides a listing & creation times of Processes, Registry entries, prefetch & other file system artifacts.

• Can be combined with Mftparser & Shellbags• Multiple output file types: Body, XLSM, XML, pipe-delimited

• Allow analyst to walk memory and quickly find badness


#Timelining Memory with Volatility


#Timelining Cont’d

• At this point I had a timeline in CSV format (timeline.csv) that we can GREP or turn into an Excel spreadsheet.

• It’s critical to have a place to start. Knowing our suspected exfil event occurred on 6/13/13, the 6/25 AT1.job that executed successfully was a solid starting point.

• My first step was to grep/search for “sx86.exe” from “Schedlgu.txt”. – Unfortunately there was no entry for “sx86.exe”.

• Next, I looked for other items of interest on or around 6/25/2013 at 12:15:00 AM in the timeline…


#Timelining Cont’d

Roughly 23 minutes earlier (6/24/2013 11:53 PM) I saw a $MFT entry for “WINDOWS\Temp\trbsgmxq_kl.dll” on 6/24/13 @ 23:59 EST.

Suspect location for a legitimate DLL to reside…?


#Timelining Cont’d – trbsgmxq_kl.dll

• Next, I grep’d for “trbsgmxq_kl” across my CSV timeline.• It’s was running in two processes – PID’s 376 & 1156


#Using “Dlllist” to list running DLL’s

• Confirmation of the suspect DLL exists in any processes…• vol.py –f host.raw --profile=Win2003SP2x86 dlllist | grep trbsgmxq


#Dumping Processes with Memdump

The “memdump” command dumps memory from a running process:

vol.py -f host.raw --profile=Win2003SP2x86 memdump -p 376 -D dump/

vol.py -f host.raw --profile=Win2003SP2x86 memdump -p 1156 -D dump/


#Strings & Dumped Processes

Next, I used strings to pull out strings from the two running processes:

strings -a 376.dmp > strings376.txtstrings -a -e l 376.dmp >> strings376.txt

strings -a 1156.dmp > strings1156.txtstrings -a –e l 1156.dmp >> strings1156.txt


#trbsgmxq_kl.dll Strings AnalysisInteresting strings from the “svchost.exe” process (PID 1156) :

• C:\WINDOWS\Temp\trbsgmxq_kl.dll• C:\WINDOWS\TEMP\RarSFX0\AROTutorial.exeo• C:\WINDOWS\Temp\iismgr.dat

Additional Interesting API Strings which indicated trbsgmxq_kl.dll had Keylogging activity:

• GetWindowTextA (Copies the text of the specified window's title bar)

• EnableWindow (Enables or disables mouse and keyboard input to the specified window or control. When input is enabled, the window receives all input.)

• GetKeyState (Retrieves the status of the specified virtual key. The status specifies whether the key is up, down, or toggled (on, off—alternating each time the key is pressed).

• GetSystemMenu (returns a handle to menu window currently in use)


#Dumping trbsgmxq_kl.dll from memoryFirst - Use DLLLIST to obtain the physical offset:

vol.py -f host.raw --profile=Win2003SP2x86 dlllist | grep trbsgmxq_kl.dllNow that we have the physical offset, 0x06370000 I dumped the DLL: vol.py -f host.raw --profile=Win2003SP2x86 dlldump --base=0x06370000 -D dump/


#Dumping trbsgmxq_kl.dll from memoryPerformed a quick Sanity Check to make sure I had a DLL:

• Analysis of this DLL revealed it was a keylogger that captured keystrokes to the file “iismgr.dat” at the location “C:\WINDOWS\Temp”

• Single byte XOR’d w/ 0xC2


#Next Steps• Memory analysis yielded multiple high fidelity indicators! We used

these to hunt during our IR efforts across hundreds of hosts…

• Keylogger - trbsgmxq_kl.dll that logged keystrokes to iismgr.dat– iismgr.dat was XOR’d with 0xC2

• Scheduled Task “AT1.job” which executed sx86.exe

• Various .log files (x.log,error.log, victim_2012_9—2013-3-27.log) found in C:\WINDOWS\addins\

• C:\F5 - mm.exe, set.exe, dllhosts.exe, MyWce32.exe, wce32.exe

• C:\RECYCLER\gs.exe


#Pivot – cssrs.exe/Trojan.Hikit & 206.205.82.9• IR finally began 3 months later.

• 206.205.82.9 Exfil from June 13/FBI notification…? It happened.

• Searching Disk revealed a crash file several GB’s in size located in the “C:\Windows\” folder.– “cssrs.exe” was executed at the suspect Exfil time from

the FW logs – via CMD “cssrs.exe 206.205.82.9 443” on Jun 13 2013 @ 13:26

– This time is inline with the Firewall logs we reviewed.

– csrss.exe = Win. Client/Server runtime subsystem (LEGIT)

– cssrs.exe = Trojan.Hitkit - A full featured RAT


#Pivot – cssrs.exe/Trojan.Hikit & 206.205.82.9

• MEMORY.DMP (Kernel dump) picked up attacker commands…• Capture a lot of CLI activity from the attacker spanning several

months.


#Email /.PST Exfil & csss.exe

• Analysis of C:\Windows\Addins\cssrs.exe revealed that it was Trojan.Hikit, a full featured RAT.

• The crash file showed the attacker packaging up the China Director’s PST file with a tool “csss” on 6/13/13 at 9:11am ~4 hours before it likely was exfiled.

• Both csss.exe (likely winRAR) and cssrs.exe (HiKit) were seen in ShimCache…


#Crash Dump (Memory.dmp) Analysis• The crash dump had a WEALTH of attacker CLI actions• Including confirmation of sx86.exe on disk right before AT1.job


#Further AnalysisJune 25 2013 AT1.job -- C:\sx86.exe dropper for Trojan FF-RAT:

– frtest.dat & Windows Config.wav which we refer to as “Trojan-FF-RAT”.

– Signed malware “Xuzhou Chenju Technology Co.,Ltd

VT sigs still hitting on this.


#Additional Analysis• Memory was taken on 7/17/13 @ 10.51am EST. • After getting the disk image a few months later, the keylogger

output file “iismgr.dat” was created on the system on 7/17/13 @ 11:28 am EST.

• APT actors were active on the system and had domain admin creds 37 minutes later through the trbsgmxq_kl.dll keylogger…


#Total Damage – Overall Intrusion• Total of 8 Unique Trojan families, several of which

were signed executables.

• We found evidence of multiple (unique) actor groups, no conclusive attribution.

• 18% of endpoints were either infected with APT malware, or were accessed laterally by APT.

• Keystroke loggers and GUI access (RDP & RAT’s).

• Total organizational compromise existed for > 3 years.


#Malware Summary

RATS• cssrs.exe

Trojan.Hikit• FF-RAT (frtest.dat

& Windows Config.wav)

Droppers• SX86.exe - dropper for

Trojan FF-RAT (frtest.dat & Windows Config.wav)

• mm.exe – unrecovered• set.exe – unrecovered

(Dropped FF-RAT)

Keylogger trbsgmxq_kl.dll iismgr.dat (output file)


#Questions

EMC, RSA, the EMC logo and the RSA logo are trademarks of EMC Corporation in the U.S. and other countries.