Download pptx - Core intel

ING PP Example Reference 16x9 - June 2015

Core IntelKrzysztof Adamski, Krzysztof mijOn the bank secret service

Max. widthMax. height

ING OrangeRGB= 255, 98, 0

ING Light GreyRGB= 168, 168, 168

ING IndigoRGB= 82, 81, 153

ING SkyRGB= 96, 166, 218Colour Guidelines

ING FuchsiaRGB= 171, 0, 102

ING LimeRGB= 208, 217, 60

ING LeafRGB= 52, 150, 81

ING Mid GreyRGB= 118, 118, 118

Text ColourRGB= 51, 51, 51









Text ColourRGB= 51, 51, 51No content below the grey lineAre security breaches common?

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/432412/bis-15-302-information_security_breaches_survey_2015-full-report.pdf









Text ColourRGB= 51, 51, 51No content below the grey lineCarbanak3

https://securelist.com/blog/research/68732/the-great-bank-robbery-the-carbanak-apt/









Text ColourRGB= 51, 51, 51No content below the grey line

Core Intel is a part of ING Cyber Crime Resilience Programme to structurally improve the capabilities for the cybercrime

preventiondetection and the response

CoreIntel 4










Measures against e-banking fraud, DDoS and Advanced Persistent Threats (APTs). Threat intelligence allow to respond to, or even prevent, a cybercrime attack (This kind of intelligence is available via internal and external parties and includes both open and closed communities)Monitoring, detection and response to spear phishingDetection/mitigation of infected ING systems Baselining network traffic/anomaly detectionResponse to incidents (knowledge, tools, IT environment)Automated feeds, automated analysis and historical data analysis

The reasoning5










The world is not enough



















6

So the challenge is



















7

Market leaders Benelux

Growth markets

Commercial Banking

Challengers

Most of our data is within Europe8









Text ColourRGB= 51, 51, 51No content below the grey lineMarket leaders Benelux

Growth markets

Commercial Banking

Challengersbut we operate globally9









Text ColourRGB= 51, 51, 51No content below the grey lineExpect the unexpected to collect all the data10









Text ColourRGB= 51, 51, 51No content below the grey lineWhat kind of data do we need?Where is our data located?How we can potentially capture it?What are the legal implications?

So there is a challenge to capture all the data11










Core Intel architecture



















12

So what you would like to see isPhoto credit:edgarpierceviaFoter.com/CC BY









Text ColourRGB= 51, 51, 51No content below the grey lineIn fact it is slightly more complicated


















Text ColourRGB= 51, 51, 51No content below the grey lineAll has its own purpose. Lets see in details.15Photo credit:https://www.pexels.com/photo/dslr-camera-equipments-147462/









Text ColourRGB= 51, 51, 51No content below the grey lineLocal data collector16









Text ColourRGB= 51, 51, 51No content below the grey lineBut tell how to capture that data17

https://observer.viavisolutions.com/includes/popups/taps/tap-vs-span.php










17

Broker settings:Replication factor >= 3min.insync.replicas = 2unclean.leader.election.enable = falsereplica.lag.time.max.ms

Producer settings:acks = allretries = Integer.MAX_VALUEmax.block.ms = Long.MAX_VALUEblock.on.buffer.full = true

To have data in ordermax.in.flight.requests.per.connection = 1

Kafka producer configuration (as we dont like losing data)18









Text ColourRGB= 51, 51, 51No content below the grey lineCentral data collector19









Text ColourRGB= 51, 51, 51No content below the grey lineTime is crucial here20Photo credit:Cargo CultviaFoter.com/CC BY









Text ColourRGB= 51, 51, 51No content below the grey lineBut your business data more, so proceed with caution21Photo credit:https://www.pexels.com/photo/white-caution-cone-on-keyboard-211151/










Network bandwidth controlquota.consumer.defaultquota.producer.default

Kafka mirror maker configuration22










Secure data:listeners=SSL://host.name:portssl.client.auth=requiredssl.keystore.locationssl.keystore.passwordssl.key.passwordssl.truststore.locationssl.truststore.password

Kafka mirror maker configuration23

Secure data in transit









Text ColourRGB= 51, 51, 51No content below the grey lineStreaming data24










spark.yarn.maxAppAttemptsspark.yarn.am.attemptFailuresValidityIntervalspark.yarn.max.executor.failuresspark.yarn.executor.failuresValidityIntervalspark.task.maxFailuresspark.hadoop.fs.hdfs.impl.disable.cachespark.streaming.backpressure.enabled=truespark.streaming.kafka.maxRatePerPartition

Spark on yarn streaming configuration25









Text ColourRGB= 51, 51, 51No content below the grey lineIn memory data grid26

valrddFromMap=sc.fromHazelcastMap("map-name-to-be-loaded")









Text ColourRGB= 51, 51, 51No content below the grey lineLets find something in these logs27Photo credit: https://www.flickr.com/photos/[email protected]/12726065645/in/[email protected]/









Text ColourRGB= 51, 51, 51No content below the grey lineMatching28

Tornado - a Python web framework and asynchronous networking library - http://www.tornadoweb.org/MessagePack binary transport formathttp://msgpack.org/










Automatically & continually match network logs threat intel When new threat intel arrives, against full history network logsWhen new network logs arrive, against full history threat intelAlerts are shown in a hit dashboardDashboard is a web-based interfaces that provide flexible charts, querying, aggregation and browsing Quality/relevance of an alert is subject to the quality of IoC feeds and completeness of internal log data. Hit, alerts and dashboards29










29

Be smart with your tooling30Photo credit https://www.flickr.com/photos/[email protected]/









Text ColourRGB= 51, 51, 51No content below the grey lineand leverage e.g. elasticsearch templates31










31

Data mapping: - doc_value - fielddata - fields

Cluster settings to check:gateway.recover_after_nodesgateway.recover_after_master_nodesgateway.recover_after_data_nodesindices.recovery.max_bytes_per_secindices.breaker.total.limitindices.breaker.fielddata.limit

Elasticsearch configuration32









Text ColourRGB= 51, 51, 51No content below the grey lineFor those who know how to use heavy equipment33Photo credit:News Collection & Public Distribution @techpearce2viaFoter.com/CC BY









Text ColourRGB= 51, 51, 51No content below the grey lineLong data storage - HDFS34









Text ColourRGB= 51, 51, 51No content below the grey lineKafka offset management 35










35

Core Intel allows users to perform advanced analytics on network logs using a set of powerful tools

Spark API to write code to process large data sets on a clusterperform complex aggregations to collect interesting statisticsrun large scale clustering algorithms with Sparks MLLibrun graph analyses on network logs using Sparks GraphXtransform and extract data for use in another system (which are better for specific analytics or visualization purposes)Kafka, co you can write own Consumers and Producers to work with your datato perform streaming analysis on your datato implement your own alerting logicToolsetProgramming languages: Scala, Java, PythonIDEs: Eclipse / Scala IDE, IPython Notebook and R StudioAdvanced analytics36










How do we schedule the jobs37










37

How to keep everything under control38Photo credit:https://www.flickr.com/photos/martijn141









Text ColourRGB= 51, 51, 51No content below the grey lineMonitoring crucial points in your data pipeline39









Text ColourRGB= 51, 51, 51No content below the grey lineSomething for smart guys40Photo credit: https://www.flickr.com/photos/jdhancock/5173498203/









Text ColourRGB= 51, 51, 51No content below the grey linePlenty of data to analyze 41









Text ColourRGB= 51, 51, 51No content below the grey lineUpcoming challenges on the operations side42










Shaken,not stirred?43










44









Text ColourRGB= 51, 51, 51No content below the grey lineFollow us to stay a step aheadING.com

YouTube.com/ING

SlideShare.net/ING

@ING_News

LinkedIn.com/company/ING

Flickr.com/INGGroup

Facebook.com/ING

















