Upload
orsen
View
45
Download
3
Embed Size (px)
DESCRIPTION
Processing Intelligence Feeds with Open Source Software. Chris Horsley, SC Leung, Tomas Lima, L. Aaron Kaplan, Raphael Vinot. Overview. Current topics in automatic incident handling for CERTs IFAS HKCERT , IFAS and use-cases IHAP project ContactDB project Current R&D. IFAS. - PowerPoint PPT Presentation
Citation preview
Processing Intelligence Processing Intelligence Feeds with Open Source Feeds with Open Source SoftwareSoftware
Chris Horsley, SC Leung, Tomas Lima, L. Aaron Chris Horsley, SC Leung, Tomas Lima, L. Aaron Kaplan, Raphael VinotKaplan, Raphael Vinot
OverviewOverview
• Current topics in automatic incident handling for CERTsCurrent topics in automatic incident handling for CERTs
• IFASIFAS
• HKCERT , IFAS and use-casesHKCERT , IFAS and use-cases
• IHAP projectIHAP project
• ContactDB projectContactDB project
• Current R&DCurrent R&D
IFASIFAS
• Information Feed Analysis SystemInformation Feed Analysis System
Knowing whatKnowing what’’s going ons going on
How do national CSIRTs know whatHow do national CSIRTs know what ’’s s happening?happening?
National CSIRTs need visibility on network in their economyNational CSIRTs need visibility on network in their economy
However, many national CSIRTs donHowever, many national CSIRTs don’’t operate networks t operate networks themselves, and normally donthemselves, and normally don’’t have global (or any) direct t have global (or any) direct visibilityvisibility
How does the CSIRT know whatHow does the CSIRT know what’’s going on in their country?s going on in their country?
The kindness of strangersThe kindness of strangers
Luckily, there are a lot of network operators, research teams, Luckily, there are a lot of network operators, research teams, vendors, and other CSIRTs out there that collect information, vendors, and other CSIRTs out there that collect information, and will share it with national CSIRTs.and will share it with national CSIRTs.
And here comes the And here comes the ““butbut””......
So much data, so many So much data, so many formatsformats
There are many feeds, all with their own data formats and There are many feeds, all with their own data formats and mediums:mediums:
Formats: CSV, JSON, XML, STIX, IODEFFormats: CSV, JSON, XML, STIX, IODEF
Mediums: HTML, RSS, email, HTTP APIsMediums: HTML, RSS, email, HTTP APIs
While there are efforts to standardise data formats, this will While there are efforts to standardise data formats, this will take a long time, and will likely never cover 100% of feedstake a long time, and will likely never cover 100% of feeds
We canWe can’’t change the format of remote feeds - we can only t change the format of remote feeds - we can only change what we do with the data.change what we do with the data.
The need for standardsThe need for standards
Different feeds use many terms to mean the same thing:Different feeds use many terms to mean the same thing:
ip, source_ip, src_ip, endpoint, attacker_ip, cnc_ip...ip, source_ip, src_ip, endpoint, attacker_ip, cnc_ip...
If we receive events from many feeds, we need to normalise If we receive events from many feeds, we need to normalise so we can compare them together.so we can compare them together.
The need for storageThe need for storage
As a national CSIRT, weAs a national CSIRT, we’’re concerned with the health of re concerned with the health of national networks: which means measurement.national networks: which means measurement.
We can only measure longterm if we store events, enabling We can only measure longterm if we store events, enabling us to analyse them.us to analyse them.
We also want to search through events, like:We also want to search through events, like:
C&C servers in domestic networks in last weekC&C servers in domestic networks in last week
Bots infected with Trojan.abc on BigISPBots infected with Trojan.abc on BigISP
Defaced web sites targeting gov.zzDefaced web sites targeting gov.zz
Need for automationNeed for automation
ThereThere’’s way too much network event data out there to s way too much network event data out there to manually processmanually process
Options:Options:
a) use lots of analyst time doing tedious log processinga) use lots of analyst time doing tedious log processing
b) write lots of small, independent scriptsb) write lots of small, independent scripts
c) ignore inbound logs completelyc) ignore inbound logs completely
d) use an automated processing systemd) use an automated processing system
So what do we need?So what do we need?
We need something which automatically:We need something which automatically:
Gathers many different types of feedsGathers many different types of feeds
Normalises the data in those feedsNormalises the data in those feeds
Stores that data somewhereStores that data somewhere
Allows search and performs statistical analysisAllows search and performs statistical analysis
IFASIFAS
IFAS = Information Feed Analysis SystemIFAS = Information Feed Analysis System
Project sponsored by HKCERT and developed by HKCERT and Project sponsored by HKCERT and developed by HKCERT and CSIRT FoundryCSIRT Foundry
An integration of open source tools, released as open source An integration of open source tools, released as open source for CSIRTsfor CSIRTs
ArchitectureArchitecture
ArchitectureArchitecture
Abusehelper: gather, process, and enrich feeds, generate Abusehelper: gather, process, and enrich feeds, generate eventsevents
Logstash: process and normalise feedsLogstash: process and normalise feeds
Elasticsearch: store events in schema-free index serverElasticsearch: store events in schema-free index server
Kibana: search through eventsKibana: search through events
IFAS Reporter: get overall statistics, build realtime IFAS Reporter: get overall statistics, build realtime dashboardsdashboards
Kibana event searchesKibana event searches
Freeform statistical Freeform statistical reportingreporting
Nesting, filtering, Nesting, filtering, deduplicationdeduplication
IFAS – DashboardIFAS – Dashboard
Visualize informationVisualize information
*Drill down right at the chart
What you need to startWhat you need to start
SoftwareSoftware
Open source under Apache 2.0 LicenseOpen source under Apache 2.0 License
Only possible with the hard work released under open source Only possible with the hard work released under open source licenses from Abusehelper and Elasticsearch teamslicenses from Abusehelper and Elasticsearch teams
Contributions, bug reports, feature requests most welcome!Contributions, bug reports, feature requests most welcome!
HardwareHardware
Production: 8-16GB memory machineProduction: 8-16GB memory machine
Dev: 4GB possibleDev: 4GB possible
Multi-core machine (4+ ideal)Multi-core machine (4+ ideal)
Runs in a VM no problemRuns in a VM no problem
Out of the box feedsOut of the box feeds
Other developed Plugins Malc0de Malicious Domain List Arbor SRF Shadowserver Zone-H
Future … more, and your own
Out of Box Feed Plugins(4 publicly available)Abuse.chCleanMXMillersmilesPhishtank
Where to get itWhere to get it
Currently under closed pilot to trusted CSIRTsCurrently under closed pilot to trusted CSIRTs
Eventually public releaseEventually public release
Please contact Please contact [email protected] for details for details
DemosDemos
IFAS and Use CasesIFAS and Use Cases
SC Leung, HKCERTSC Leung, HKCERT
Give a sense of Today’s Give a sense of Today’s EventsEvents
IFAS - Log SearchIFAS - Log Search
Powerful search on all the information collectedPowerful search on all the information collected
Keywords here
Add columns of interests
Feed Details
IFAS - ReporterIFAS - Reporter Statistical analysis-Trends & DistributionsStatistical analysis-Trends & Distributions
Free form statistical reportsFree form statistical reports
1.1.
5.5. 2.2. 4.4.
6.6.
3.3.
Nesting, filtering, Nesting, filtering, deduplicationdeduplicationNumber of phishings in “.AU” in each ASN by brand
IFAS - AlertIFAS - Alert Set tracking criteria – get notify ASAPSet tracking criteria – get notify ASAP
domain: domain: *.gov.hk*.gov.hk
Alert lists : educational institutions (Alert lists : educational institutions (hkeduhkedu), NGOs (), NGOs (hkorghkorg))
!!
DashboardDashboard Real-time situational awareness Real-time situational awareness for CERT management for CERT management
Public Situational Public Situational AwarenessAwareness
on on Compromised Servers / Compromised Servers /
PCsPCs
Hong Kong Security Watch ReportHong Kong Security Watch Report
• Correlate Cryptolocker 2013-Oct with ZeusCorrelate Cryptolocker 2013-Oct with Zeus
Analysis of Trend with Analysis of Trend with EventsEvents
Engage ISPs for large scale incident Engage ISPs for large scale incident handlinghandling
• Data do help Data do help HKCERT HKCERT engaging ISPs engaging ISPs (their sales (their sales team)team)
• Data do help a Data do help a server hosting server hosting SP understand SP understand their customers’ their customers’ security security problemsproblems
ISP
Converting security events into Converting security events into incident reportsincident reports
• DefacementDefacement
• PhishingPhishing
Export to CSV for batch processing, with Export to CSV for batch processing, with some other scriptssome other scripts
• Malware hosting – a bit difficultMalware hosting – a bit difficult
• Large volume of incidents – need prioritisationLarge volume of incidents – need prioritisation
Future of IFAS - a Future of IFAS - a collaboration platformcollaboration platform
• All you can useAll you can use
• All you can contributeAll you can contribute
• Add input filters for new feedsAdd input filters for new feeds
• Add new plug-in modulesAdd new plug-in modules
• Add new chart and visualizationAdd new chart and visualization
• Integrate with other systems, e.g. RTIRIntegrate with other systems, e.g. RTIR
• ……
• Standard languageStandard language: STIX, taxonomy of ENISA: STIX, taxonomy of ENISA
• An ongoing project that turn security events into Actionable An ongoing project that turn security events into Actionable DataData
• Set Priority, Choose Monitors, Consolidate ResultsSet Priority, Choose Monitors, Consolidate Results
DSMS DSMS (Decision Support & Monitoring (Decision Support & Monitoring System)System)
Decision Decision SupportSupport
Sub-systemSub-system
Decision Decision SupportSupport
Sub-systemSub-system
IFASIFAS
Interfaces to Interfaces to MonitorsMonitors
PrivatePrivate analysis sysanalysis sys
PrivatePrivate analysis sysanalysis sys
Public Public analysis sys analysis sys (VirusTotal, (VirusTotal,
ThreatExpert)ThreatExpert)
Public Public analysis sys analysis sys (VirusTotal, (VirusTotal,
ThreatExpert)ThreatExpert)
Web Web reputation reputation (D-Shield)(D-Shield)
Web Web reputation reputation (D-Shield)(D-Shield)
Interface Interface ModulesModulesInterface Interface ModulesModules
Interface Interface ModuleModule
Interface Interface ModuleModule
Interface Interface Modules Modules Interface Interface Modules Modules
Request Request to to
monitormonitor
OutputOutput
StoryStoryStoryStory
ProfileProfile
Input Input URLURL
TasksTasks
IncideIncidentnt
MgmtMgmt
Status CheckStatus Check(HTTP, DNS) (HTTP, DNS)
via proxyvia proxy
Status CheckStatus Check(HTTP, DNS) (HTTP, DNS)
via proxyvia proxy
Status Status ??Interface Interface ModuleModule
Interface Interface ModuleModule (online /offline)(online /offline)
MonitoringMonitoringServicesServices
ConsolidateConsolidated Resultsd Results
IHAPIHAPIncident handling automation projectIncident handling automation project
IHAPIHAP
• Very similar to IFAS, developed in parallel by CERT.pt, Very similar to IFAS, developed in parallel by CERT.pt, CERT.atCERT.at
• Also uses Logstash, Elastic Search and AbusehelperAlso uses Logstash, Elastic Search and Abusehelper
• Less work on the Webinterface, more work on Ontology, Less work on the Webinterface, more work on Ontology, „Data harmonisation document“„Data harmonisation document“
IHAP - HistoryIHAP - History
• Discussions about CERT.AT developments/documentsDiscussions about CERT.AT developments/documents
• Discussions about cooperation between CERTsDiscussions about cooperation between CERTs
• ENISA supportENISA support
IHAP - GoalsIHAP - Goals• Open SourceOpen Source
• MaintainableMaintainable
• Flexible and Modular Flexible and Modular - must be possible to integrate existing - must be possible to integrate existing software and modules (Pastemon, AbuseHelper, etc..)software and modules (Pastemon, AbuseHelper, etc..)
• ReusableReusable
• Easily ExtendableEasily Extendable - should require little knowledge and basic - should require little knowledge and basic programming skillsprogramming skills
• Easily DeployableEasily Deployable
• Easily Updatable Easily Updatable – easy to share new developments with other – easy to share new developments with other CERTs and update the system with that new codeCERTs and update the system with that new code
• Easily Configurable Easily Configurable - config files that can be easily modified to fit - config files that can be easily modified to fit CERT‘s needsCERT‘s needs
• DocumentedDocumented - must be well documented - must be well documented
Links & CodeLinks & Code
http://www.enisa.europa.eu/activities/cert/support/incident-http://www.enisa.europa.eu/activities/cert/support/incident-handling-automationhandling-automation
Common field names for Common field names for AHAH• https://bitbucket.org/clarifiednetworks/abusehelper/wiki/Data
%20Harmonization%20Ontology
• A standard set of well defined field names within A standard set of well defined field names within Abusehelper (AH)Abusehelper (AH)
• Allows CERTs to:Allows CERTs to:
• Write bots which are interoperable within AHWrite bots which are interoperable within AH
• Measure in identical waysMeasure in identical ways
• Easier to parse different feeds („generic santizer bot“) : Easier to parse different feeds („generic santizer bot“) : you just have to define the mappingsyou just have to define the mappings
contactDBcontactDB
Background/ problemBackground/ problem
• abuse@ lookups suck (IRT object not in use, no standard; abuse@ lookups suck (IRT object not in use, no standard; Just now RIPE DB is changing with abuse-c:)Just now RIPE DB is changing with abuse-c:)
• Getting the right lookup is non-trivial, complex Getting the right lookup is non-trivial, complex
• Many (national) CERTs create their own abuse contact Many (national) CERTs create their own abuse contact lookup DBs.lookup DBs.
• National CERT DB, TI directory, FIRST data can not be looked National CERT DB, TI directory, FIRST data can not be looked up automatically via scripts.up automatically via scripts.
IdeaIdea
• A caching contact database with more specific internal dataA caching contact database with more specific internal data
• Some of this data (tel nos, etc) will never be in the public Some of this data (tel nos, etc) will never be in the public whoiswhois
• Unify with TI, FIRST etc dataUnify with TI, FIRST etc data
• Make it query-able by scriptsMake it query-able by scripts
Abuse contact lookup - flowAbuse contact lookup - flowWhat databases exist? What can we query?What databases exist? What can we query?
Number based Number based resource:resource:
IP addr, netblock, IP addr, netblock, ASNASN
Number based Number based resource:resource:
IP addr, netblock, IP addr, netblock, ASNASN
Name based Name based resource:resource:
domain name, domain name, hostnamehostname
Name based Name based resource:resource:
domain name, domain name, hostnamehostname
MaxmindMaxmindRIPE DBRIPE DBCymru, ..Cymru, ..
..
MaxmindMaxmindRIPE DBRIPE DBCymru, ..Cymru, ..
..
Get Get country()country()
National National CERT DBCERT DBCERT.orgCERT.org
National National CERT DBCERT DBCERT.orgCERT.org
Email AddressEmail AddressEmail AddressEmail Address
Whois Whois DB DB
(RIPE, (RIPE, ARIN, ..)ARIN, ..)
Whois Whois DB DB
(RIPE, (RIPE, ARIN, ..)ARIN, ..)
IRT object, abuse-c, IRT object, abuse-c, ......
Country codeCountry code
TI, FIRST, TI, FIRST, CERT.org CERT.org
DBsDBs
TI, FIRST, TI, FIRST, CERT.org CERT.org
DBsDBs
Whois DB Whois DB (registrant(registrant, registrar), registrar)
Whois DB Whois DB (registrant(registrant, registrar), registrar)
IANA IANA ccTLD listccTLD list
IANA IANA ccTLD listccTLD list
Extract ccTLDExtract ccTLD
National CERT for countryNational CERT for country
Country codeCountry code
Gethostbyname()Gethostbyname()
What exists now?What exists now?
• Public code repo ;-)Public code repo ;-)
• Whois server (thx Mauro)Whois server (thx Mauro)
• RESTful API (Mauro, Rafiot)RESTful API (Mauro, Rafiot)
• Some scripts to import TI data (Aaron, David)Some scripts to import TI data (Aaron, David)
• Still some bugs ;-)Still some bugs ;-)
Code & document with Code & document with RIPERIPE• Document (WIP):Document (WIP):
• https://github.com/certtools/contactdb/blob/master/doc/contact-databases-for-abuse-handling.mkd
• Codebase:Codebase:https://github.com/certtools/contactdb
• (thx Rafiot, David, Mauro!)(thx Rafiot, David, Mauro!)
SummarySummary
SummarySummary
• The CERT community has limited ressources for The CERT community has limited ressources for developmentdevelopment
• We re-implement the same thing all the timeWe re-implement the same thing all the time
• Let‘s share code or at least exchange ideas on how to Let‘s share code or at least exchange ideas on how to automate incident handling!automate incident handling!
• Let‘s share on how to measure successLet‘s share on how to measure success
• Thanks HKCERT, ENISA, CERT.at, CERT.pt, CIRCL, etc..Thanks HKCERT, ENISA, CERT.at, CERT.pt, CIRCL, etc..
• Mailinglist: Mailinglist: https://tiss.trusted-introducer.org/mailman/listinfo/ihap
Thanks!Thanks!