Embed Size (px)
Citation preview
Copyright © 2014 Splunk Inc.
Chris Kurtz System Architect Arizona State University
Using Splunk to Protect Students, Faculty and the University
Disclaimer
2
During the course of this presentaGon, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cauGon you that such statements reflect our current expectaGons and
esGmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements,
please review our filings with the SEC. The forward-‐looking statements made in the this presentaGon are being made as of the Gme and date of its live presentaGon. If reviewed aRer its live presentaGon, this presentaGon may not contain current or accurate informaGon. We do not assume any obligaGon to update any forward-‐looking statements we may make. In addiGon, any informaGon about our roadmap outlines our general product direcGon and is subject to change at any Gme without noGce. It is for informaGonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligaGon either to develop the features or funcGonality described or to
include any such feature or funcGonality in a future release.
AddiGonal Speaker Disclaimer: While I am speaking as an employee of Arizona State University, I do not speak for the University nor dictate policy, procedures, or purchases. Any and all statements made in this presentaGon are mine
alone, and do not in any way represent an official statement from ASU. The opinions and comments contained herein are enGrely my own. ASU does not endorse or represent any product menGoned, up to and including Splunk.
Agenda
! IntroducGon to me and Arizona State University – About ASU – About me – Our Environment and our challenges
! Use Cases and Examples – ProtecGng Direct Deposit, two versions – Phishing as a teaching tool – Leveraging your insGtuGonal data with lookups and apps
! Conclusion: Where we’ve been, where we’re going!
3
IntroducGon
4
5
! Largest single University in the US ! More than 80,000 acGve students… ! …and another 20,000 accounts (faculty/staff, alumni, affiliates) ! Located in Tempe, Arizona, suburb of Phoenix, 6th Largest City in US ! Not located on the surface of the sun…but you can see it from here!
Obligatory About Me: Professionally
6
! Unix/Linux System Administrator by trade, 23 years experience ! Supported NASA/JPL Mars projects at ASU for more than 10 years:
– TES & THEMIS Instrument onboard Mars Global Surveyor & Mars Odyssey – MTES Instrument on the Mars ExploraGon Rovers Spirit and Opportunity
! ASU’s “Splunk Guy” (System Architect) since early 2013 ! Splunk Video Interview “Value of Higher EducaGon and Splunk” ! Author of the ISO 3166 Splunk App – more on this later!
Obligatory About Me: Personally
7
• Self-‐proclaimed Geek, what’s it to ya? • Steampunk Enthusiast (I made my hat,
goggles, and the gun!) • Beginning Maker (Steampunk and
Arduino/Electronics) • xoff on #splunk on efnet
Liqle known fact about me: Clyde Tombaugh, the discoverer of Pluto,
was a personal friend growing up
hqp://about.me/chk
8
! First Google Apps for EducaGon customer ! MulGple campuses with a diverse IT infrastructure ! Many organic, home-‐grown, custom, and proprietary systems ! Large number of governing requirements: FERPA, HIPPA, DARPA, DoJ, NASA, JPL, etc
! Clear separaGon of responsibiliGes inside the University Technology Office: the InformaGon Security Office (ISO) does not have access to the systems (and more importantly the logs) run by OperaGons
The Power of Splunk Splunk as ASU’s universal aggregator of all machine generated logs
9
Logs reside in mulGple locaGons, depending on when and where the system was installed: web logs in one locaGon, system logs mulGple others (depending on OS); some are on single log concentrator and some in an “old, slow, and unsupported” proprietary search database. ISO requests logs for incident. Ops has to use the proprietary tools (or just as oRen, just grep through mulGple logfiles) based on ISO descripGon and email/share logs. ISO likely has to revise request at least once.
ISO directly accesses logs in Splunk, oRen using pre-‐built dashboards, alerts, and saved searches. Ops can concentrate on OperaGons.
Typical response Gme to incident: mul$ple business days
Typical response Gme to incident: minutes!
Without Splunk
With Splunk
Splunk and Arizona State University
10
Licensing • 750gb/day
– Started at 50gb in November 2012 – …to 150gb in February 2013 – …to 500gb in June 2013 – …to 750gb in July of 2014 – On track to reach 1TB this FY
Infrastructure • Physical indexers in cluster
– ~14TB in Hardware RAID10 – NFS for Cold (being phased out) – Architected for 1TB (10 indexers)
• Search head pooling – 3 virtual servers (12 CPUs, 32gb) – NFS SSD storage for shared data
• Virtual support servers: – Deployment server – License manager – Cluster master
The value of Splunk to the InformaGon Security Office has driven the rapid growth…
but other groups are starGng to see the value!
We Didn’t Know…
11
“To ASU, Splunk was like the invenGon of the microscope: we didn’t know what we couldn’t see.”
– Mar&n Idaszak Security Architect, Arizona State University
ProtecGng Direct Deposit
12
Use Case: ProtecGng Direct Deposit
13
! Being able to change your employee informaGon online is a great convenience, but a target for hackers
! Because of ASU’s internaGonal students, faculty, and staff, just blocking other countries isn’t acceptable…
Splunk is the soluGon!
How We Did It…Before Splunk
14
1. Payroll gets a call that an employee didn’t get their direct deposit 2. …invesGgates, sees a foreign bank deposit…and contacts the
InformaGon Security Office 3. ISO changes the user’s password 4. ISO requests webserver single sign-‐on and HR HR system logs from
OperaGons and our HR Vendor (could take days!) 5. Eventually details are discovered (compromised account) and user
is informed. Funds are long gone, and ASU has to re-‐issue the employee’s check, eaGng the loss
How We Did It With Splunk, Version 1
15
1. Logs from webserver single sign-‐on and PeoplesoR now go to Splunk. No more waiGng on OperaGons to retrieve logs! This makes both ISO and Ops very happy!
2. Splunk monitors for Direct Deposit changes via a schedule search, building a transacGons to link the change back to the user’s webserver authenGcaGon. Ok, now we have an originaGng IP and a username…so we run geolocaGon on the originaGng IP so it’s easier to create reports based on locaGon of the change
Web auth DB records
IP username
Geo tag country
user address
All unusual changes
IP username state/country
Version 1 Stop Here:
16
! ISO creates a scheduled report of unusual originaGng IPs (Malaysia, etc) and sends it to Payroll before the close of each payroll run
! Payroll contacts users with unusual changes for verificaGon before payroll is run and if it was a fraudulent change, the change is reverted, so no funds are lost
! Even at this point, Payroll is ecstaGc and saves over 30 hours per payroll run reviewing direct deposit, and ASU saves tens of thousands of dollars per payroll run!
Now…How Do we Improve This?
17
We asked the quesGon: Where do you change your direct deposit from?
1. Home
So, let’s think about it: If your direct deposit changes from Malaysia, it’s probably fraud…
but what about Ohio, if you live in Arizona?
2. Work
That’s likely fraud, too!
So let’s leverage Splunk’s geolocaGon features!
…Version 2 (Now in Progress)
18
1. StarGng with the originaGng IP and username from Version 1…we use a custom lookup tables (more later!) to leverage HR system data, so we can lookup a username’s informaGon: Name, address, etc
2. GeolocaGon informaGon about the user’s home zip code (via the zip code) is generated
3. Using a free Splunk app called haversine, we calculate the distance between the user’s home (technically, the lat/lon of the center of their zipcode) and the lat/lon of the IP the change was made from. We realize both of these are a bit vague, but we’re really only looking for scale
4. If the distance is unusual (~50 miles) the result will be flagged for Payroll review automaGcally
Lessons Learned…and You Can Do This Too!
19
1. GET YOUR DATA INTO SPLUNK! 2. One of the beauGful things about Splunk is that you can modify how the data appears
(field extracGons, etc.) once it’s already in Splunk, and that applies to already indexed data. The focus should be gewng it into Splunk first, and figure out fields later. Think of it as schema on demand!
3. When you find people who “get it” use them to evangelize Splunk to others in the organizaGon
4. When you find people who resist, show them how much Gme and effort they can save, especially interacGng with other departments (if appropriate) by using Splunk. We won several people over when they discovered that the number of requests from groups like ISO dropped from 3-‐5 per week (each taking hours to do) to zero once the data was in Splunk
5. Don’t get caught up on “use cases”: Once you have the data in Splunk, use cases present themselves again and again! Think of it as use case on demand!
Flexibility
20
“It’s not only it’s schema-‐on-‐the-‐fly, it’s use-‐case-‐on-‐the-‐fly.”
– Barak Reeves Splunk Sales Engineer, Team TK-‐421
Phishing as a Teaching Tool
21
Use Case: Phishing as a Teaching Tool
22
! As a public University, a large amount of our informaGon is mandated to be publically available, including a directory of email addresses…and we have over 100,000 users, and each can have as many email addresses as they want…
! This means ASU receives a lot of email: In fact, we used Splunk to determine exactly how much. In the last 12 months, ASU received more than ONE BILLION email messages, and more than 750 million of them were spam and phishing!
As usual…Splunk is the soluGon!
Mandatory Pie Chart
Phishing and ASU
Inbound Phishing Email
Mail Filter
Email Stored
User clicks on phishing link
Firewall blocks some
Some gets through
Firewall
23
ASU is Hard to Protect
24
ASU, as an enGty, is very hard to protect. We have students from all across the world, and by their nature, they are very transient: they move apartments, dorms, travel the US and abroad, and access ASU systems from almost everywhere. Unlike most corporaGons, we can’t assume that access to ASU from Nigeria, China, or Malaysia are hacking aqempts…in fact, it’s probably legiGmate!
One of the very first things we saw with Splunk were logins on campus and from India for the same user on the same day. What was this? Hacking? VPN? MulGple people using the same login? Turns out Indian students oRen gave their passwords (gasp!) to their parents, who insisted on it, so the parents could regularly check their grades! This led another project to provide limited access to secondary accounts
(just for this purpose) know that their efforts were valid and necessary!
Use the Data You Have…
25
To protect ASU from spam, we use Barracuda Spam & Virus Firewalls, but there is no Splunk app (yet) so we make custom field extracGons from the Barracuda logs. …but ASU does not store user emails in Splunk, only the headers of the messages that transit our system
Do managers ever ask you if a product is worthwhile? We regularly use Splunk to show that other products
are doing their jobs!
Seems legit?
Phishing and ASU Correlate firewall informaGon with our mail logs to get a list of every user who clicked on a phishing link.
Firewall log Email log
IP Bad URL user Email with link
Table of user clicking bad link
CMDB for contact
26
…and Let Your Data Combine!
27
! BUT…ASU also uses Palo Alto Firewalls to protect our users. These firewalls very oRen catch phishing URLs that users click on, either via mistake or lack of understanding… and we correlate that Palo Alto informaGon with our mail logs to get a list of every user who clicked on a phishing link.
! The ISO can then directly contact the users who clicked on a phishing link, explain to them why they need to change their password (and probably run a virus/malware scan), and use the opportunity to explain to the user why what they did was bad. The users are thankful that the University is watching out for them, and some of the potenGal vicGms have become our best reporGng sources for received phishing and spear phishing emails!
This too is being automated! We plan to use workflows to allow ISO to easily flag a potenGally compromised account in Splunk, which (via a REST API call to our authenGcaGon system) is automaGcally disabled and (via another REST API) a Gcket is created for the helpdesk, so they can explain the situaGon to the user when they call in because their password no longer works.
…Version 2 (Now in Progress)
28
1. ISO acGvely follows phishing links (from a secure and isolated Virtual Machine) and enters bogus credenGals. We are now using Splunk to alert on aqempted logins using those honeypot credenGals. These acGve hackers are then blocked on the Palo Alto Firewalls in a quick but manual process…this protects users who might click on the phishing. Eventually, we plan to semi-‐automate this using Splunk workflows that let ISO directly block several different types of aqackers from Splunk, using the Palo Alto’s APIs.
2. ASU is invesGgaGng using honeypot full email accounts that will be scraped from the public directory and then sent spam/phishing aqempts just like real users. The plan is to use Splunk to index the enGre email, so we will have the full body of phishing and spam emails as well as headers. Phishing URLs idenGfied would be blocked using a workflow to the Palo Alto APIs, as above, and the from addresses would be blocked on the Barracudas with their APIs.
Lessons Learned…and You Can Do This Too!
29
1. LEVERAGE YOUR DATA! 2. Combining data from mulGple sources is amazing! We use data from the
Barracuda Spam Firewalls as well as the Palo Alto Firewalls to provide mulGple points of visibility into phishing.
3. Standardize your data! Follow Splunk’s Common InformaGon Model so that field names are consistent across data types. Once you realize that src_ip, for example, exists in mulGple datasets, the possibiliGes just jump out at you!
4. Fill in the gaps. When you find gaps in your data models, work on how to fill them in. For us, it’s the honeypot registraGons and full-‐email indexing. Once we realized full-‐email indexing was possible (and easy!) all sorts of new use cases appeared!
Value of Splunk
30
“This is the best tool we’ve seen in 10 years.” – Jay Steed
AVP for UTO Opera&ons, Arizona State University
Leveraging Your Own Custom Data
31
The Power of Splunk!
32
! No schemas! This means if you need to alter your data structure (field extracGons, calculated fields, etc.) you can easily do it on the fly, and it’s retroacGve!
! No types! Splunk really doesn’t care if “42” is a string or a number, so you can divide 42 by 7 and get 6, or add a string to make it “42 is the answer” just as easily to modify a field or make a new one on the fly.
! Eval is your friend! ! Remember…It doesn’t maqer if data is from a logfile, database, texyile, script
output, or anything else…combine it in any way you want, on the fly!
Why menGon this? Because as a Splunk Admin, always remember: the data structure is mutable! If it doesn’t work for your needs, change it on the fly!
To Correlate Data, You Have to Have Data to Correlate…
33
! Having data from machine logs such as mailservers and firewalls is great, it’s the first (and easiest) data to get into Splunk.
! Without a common key, there is no way to know that two pieces of data refer to the same individual.
! For ASU, the master datasource is the Data Warehouse. These databases contain the records for every student and employee.
Does the email [email protected] belong to John Bunbury?
Lookups from Databases
34
! Isolated Splunk server running Database Connect (DBX) runs SQL Queries on several databases, and writes a series of lookup tables (with the affiliate ID) every 4 hours
! Linux ionoGfy monitors the lookup tables, and on write-‐close copies data to producGon systems (sanity checking applies)
Data Warehouse Isolated Splunk running DBX
ProducGon Splunk
100000001, jbunbury7, John Bunbury, [email protected], student 100000002, jbunbury, Jane Bunbury, [email protected], employee
35
! Splunk (and most other applicaGons) use the ISO3166 standard “alpha-‐2” country codes (US for United States, for example). This is standard for geolocaGon services in Splunk.
! But…our Oracle Databases for Student data get the data from the students, oRen their passports. And machine-‐readable passports use the ISO3166 “alpha-‐3” country codes…and there isn’t a simple conversion!
! If the Country Code is not in the standard geolocaGon format, I can’t do any geolocaGon, which means the data is far less useful.
! I looked on the Splunk Apps site (hqp://apps.splunk.com) but didn’t find a soluGon…
Problem is…
Country alpha-‐3 alpha-‐2
United States USA US
China CHN CN
Nigeria NGA NG
So, I Wrote the App Myself!
36
Very simple structure, but so useful! I took the online ISO 3166 country codes (3 kinds: alpha-‐3, alpha-‐2, and numeric) and built a lookup table, which I call in the dbquery search before outpuwng the lookup table
Lookup Sample: alpha-‐2,alpha-‐3,numeric US,USA,840 CN,CHN,156 NG,NGA,566
| dbquery "PS PRD" "SELECT EMPLID,CITY,STATE,POSTAL,COUNTRY_CODE FROM EDS_ADDRESS" | dedup EMPLID CITY STATE POSTAL COUNTRY_CODE | lookup iso3166 iso3166_alpha-‐3 as COUNTRY_CODE | eval city=upper(substr(CITY,1,1)).lower(substr(CITY,2)) | rename STATE as region_name EMPLID as affiliate_id POSTAL as postal_code iso3166_alpha-‐2 as country_code | eval postal_code=if(country_code="US",substr(postal_code,1,5),postal_code) | table affiliate_id,city,region_name,postal_code,country_code | outputlookup affiliate_to_address.csv
Why bother publishing as an app? Because it might be useful to someone else, and at least 2 people have now said to me:“Wow, thanks, that solves my problem!”
Building an App is Simple!
37
1. In etc/apps, create a directory for your app, with appropriate subdirs (default is mandatory)
2. All config files go in default – nothing in local! 3. Write an appropriate default/app.conf (look at other apps) 4. Create a README file and other appropriate documentaGon. 5. Package and test on a generic Splunk install for sanity (hint .spl files are
just tgz files!) 6. Upload to apps.splunk.com – if something isn’t right, it’ll let you know. 7. Make sure to put the docs online!
hqp://wiki.splunk.com/Community:CreaGng_your_first_applicaGon
My app took me about a day to do, including an obsessive amount of research on how to do it.
#splunk
38
“It is days like today when I am stuck with a piece of crappy soRware with horrible documentaGon and support that I am very thankful that I spend the rest of my Gme
dealing with Splunk.” – David Shpritz (automine) Splunk IRC channel
Conclusion
39
The Past and the Future
40
! ASU has heavily invested in Splunk because it solves many of our outstanding issues, and a culture of “how can we use Splunk to solve this?” is developing
! First round (FY14) of data onboarding concentrated on the needs of the InformaGon Security Office. Second round (FY15) is focusing on OperaGons needs, with some interesGng use cases thrown in as they appear
! Splunk is expensive, but the savings in man hours, extreme flexibility, use to validate other systems, and goals to replace anGquated systems has very much paid off
Get Some Help!
41
! Splunk Docs (hqp://docs.splunk.com) – I use Splunk docs so much I have a Chrome shortcut to just search it. And if you do occasionally find something that is unclear, use the links at the boqom to provide feedback…the team is great at responding!
! Splunk Answers (hqp://answers.splunk.com) – I always look (and oRen post in Answers) here before I contact support. Just looking at what others are posGng is oRen just what you need to rephrase the quesGon to find the answers you need. The users who are on answers are the true heroes of Splunk. In fact there is only one group beqer…
! The Splunk Wiki – specifically hqp://wiki.splunk.com/Things_I_wish_I_knew_then ! The #splunk IRC channel on efnet (hqp://wiki.splunk.com/Community:IRC) – Ok, I admit
it, I’m a Splunk IRC junkie. This group is just the best…a great mix of Splunkers (aka Splunk employees), customers, and professional services and hysterical to boot. Props to the crew: Piebob, cgales, ^Brian^, DaGryph, Coccyx, amrit, Duckfez, Yorokobi, Madscient, automine, starcher, jtrucks, and even Trex (a fellow ASUer)
! Also check out @splunk, @splunkdev, and @splunkanswers on Twiqer!
42
"I look to the future because that’s where I’m going to spend the rest of my life."
– George Burns
QuesGons and MenGoned Links
43
! My Splunk App to do ISO 3166 translaGons: hqp://apps.splunk.com/app/1775/
! Free Splunk App to calculate distances on a globe (a “Great Circle” or haversine calculaGon): hqp://apps.splunk.com/app/936/
! My Splunk Video: hqp://www.splunk.com/view/SP-‐CAAAJPW
44
Security office hours: 11:00 AM – 2:00 PM @Room 103 Everyday Geek out, share ideas with Enterprise Security developers
Red Team / Blue Team -‐ Challenge your skills and learn new tricks Mon-‐Wed: 3:00 PM – 6:00 PM @Splunk Community Lounge Thurs: 11:00 AM – 2:00 PM
Learn, share and hack
Birds of a feather-‐ Collaborate and brainstorm with security ninjas Thurs: 12:00 PM – 1:00 PM @Meal Room
THANK YOU
