Upload
flink-forward
View
54
Download
3
Embed Size (px)
Citation preview
ING PP Example Reference 16x9 - June 2015
StreamING modelsRealtime model deployment of ML capabilities
Erik de Nooij, IT Chapter Lead Fraud&Cybersec.
Max. widthMax. height
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
1
IT Chapter Lead within the Fraud & Cybersecuritydepartment, based in Amsterdam
Before ING implemented Enterprise Software,mainly knowledge management and CRM related
Background in: Scala, Java, C# (MCSD), Tomcat, Websphere, Oracle, Cassandra and now.Flink
https://www.linkedin.com/in/erik-de-nooij-93ab1a/
[email protected] Am I?2
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
2
About ING
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
3
Worldwide35 Million customers51.000 EmployeesPresence in over 40 countries
Netherlands9 Million CustomersBillion logins yearly on https://www.ing.nl1 million transactions per day
About ING4
Market leaders Benelux
Growth markets
Commercial Banking
ChallengersThe Netherlands
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
4
ThreatsRule based detection
Model based detectionCriminal organizationResponseScanomaly detection
Fake ID Skimming Phishing APT ?2008 2010 2012 2014 2017Threats related to fraud & cybersecurity 5
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
5
Carbanak APT (Advanced Persistent Threat)6This started via a phishing email
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
6
Support various types of (ML) modelsTools to create models versus scoring modelsOne codebase, SaaS deployment modelMake changes instantly (no downtime)Multiple domainsGoals7
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
7
Support various types of (ML) modelsOne codebase, SaaS deployment modelPre-processor, Decoupled architectureMake changes instantly (no downtime)Multiple domainsGoals8
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
8
Support various types of (ML) modelsOne codebase, SaaS deployment modelMake changes instantly (no downtime)Use caseFeature extractionEnriching streamsEnd user toolingDemoMultiple domainsGoals9
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
9
Support various types of (ML) modelsOne codebase, SaaS deployment modelMake changes instantly (no downtime)Multiple domainsexamplesGoals10
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
10
Support various types of models
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
11
Model creation
HDFS
offline
Model execution
Streaming platformonlineCreating models offline, scoring online12
{PFA}
Portable model
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
12
The Predictive Model Markup Language (PMML)is an XML-based predictive model interchange format Predictive Model Markup Language (PMML)13
if field1 > 500 ANDfield2 == 1 ANDfield3 > 1
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
13
The Predictive Model Markup Language (PMML)is an XML-based predictive model interchange format Predictive Model Markup Language (PMML)14
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
14
Machine learning tools supporting pmml15
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
15
Parse the pmml file(s)Pass on the Feature Set to the model(s)Run the predict function which returns the output of the model(s)16Model scoring using OpenScoring.io libraryControl streamData streamScore
Feature setsmodelscoring
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
16
Supported models17Supported models(*)Association rulesRegressionCluster modelRule setGeneral regressionScorecardNaive BayesSupport Vector Machinek-Nearest neighboursTree modelNeural networkEnsemble model
(*) supported models by http://openscoring.io/
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
17
Goals18Use of various types of modelsOne codebase, SaaS Deployment model Pre-processor, Decoupled architectureMake changes instantly (no downtime)Multiple domains
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
18
Market leaders Benelux
Growth markets
Commercial Banking
ChallengersOne Bank Strategy19
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
19
How flexible is this architecture?20
Feature extraction &Model scoring
Amount = 42,00Amountincents = 4200Amount = 42.00
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
20
Decoupled architecture21
Feature extraction &Model scoring
Pre-Processor
Business events
Amount = 42.00Amountincents = 4200Amount = 42.00Amountincents = 4200
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
21
Goals22Use of various types of modelsOne codebase, SaaS Deployment modelMake changes instantly (no downtime)Use caseFeature extractionEnriching streamsEnd user toolingDemoMultiple domains
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
22
Your phone with the banking app installed is stolenLimit on the banking app is 1.000,-Funds are transferred from your account (A) to a mule account (B)Use case23
ING OrangeRGB= 255, 98, 0
ING Light GreyRGB= 168, 168, 168
ING IndigoRGB= 82, 81, 153
ING SkyRGB= 96, 166, 218Colour Guidelines
ING FuchsiaRGB= 171, 0, 102
ING LimeRGB= 208, 217, 60
ING LeafRGB= 52, 150, 81
ING Mid GreyRGB= 118, 118, 118
Text ColourRGB= 51, 51, 51No content below the grey line
23
Model features and model output24Amount > 500NrOf Trxs Last 1hFirst Trx