Upload
somkiat-puisungnoen
View
694
Download
1
Embed Size (px)
Citation preview
Introduction elasticsearch
@somkiat
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
realtime, search and analytics
engine
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
LuceneDistributed
Scalability
open-sourcedocumentation
JSONAPI
RESTFul
document store
High-Availability
JAVA
Plug-ins
History
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
First version 0.4 in Feb 2010Rewrite from Compass projectAdd scalability
Created by Shay Banon
Current version 1.7.1
Who use ?
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
https://www.elastic.co/use-cases
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
• Search repositories, users, issues, pull request• Search sourcecode 130 พันล้านบรรทัด• Track alerts, events และ logs
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
• ใ้ Full text seach + geolocation• ใ้ feature More-like-this ในการหาคําถาม และ คําตอบ
Clients
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
• Java• PHP• Ruby• Python• JavaScript
• Go• Scala• .Net• Clojure• Erlang
• NodeJS • R
http://www.elasticsearch.org/guide/en/elasticsearch/client/community/current/clients.html
https://github.com/github-sprint3r/elasticsearch_workshop
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
Let’s start
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
Installation …
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
elasticsearch
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
• real-time• distributed• search• analytics
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
mapping
analysis query dsl
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
query dsl• flexible• powerful• query language
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
queries• relevance
filters• boolean yes/no
• full text • exact values• not cached • cached• slower • faster
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
Filter firstquery remaining documents
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { . . . }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “match” : { “title” : “search” } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “match_all” : { } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “filtered” : { “query” : { … }, “filter” : { … } } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “filtered” : { “query” : { “match” : { “title” : “search” } }, “filter” : { … } } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “filtered” : { “query” : { “match” : { “title” : “search” } }, “filter” : { “term” : { “status” : “active” } } } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “filtered” : { “query” : { “match_all” : { } }, “filter” : { “term” : { “status” : “active” } } } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “filtered” : { “query” : { “match_all” : { } }, “filter” : { “term” : { “status” : “active” } } } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
How data is indexed ?
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
{ "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen" }
{ "title": "Keeping pets healthy", "content": "My quick brown fox eats rabbits on a regular basis" }
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
where content like
“%brown%fox%”
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
Analysis
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
{ "title": "Quick brown rabbits", "content": "Brown rabbits are commonly seen" }
{ "title": "Keeping pets healthy", "content": "My quick brown fox eats rabbits on a regular basis" }
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
{ "title": [quick,brown,rabbits], "content": [brown,rabbits,are,commonly,seen] }
{ "title": [keeping,pets,healthy], "content": [my,quick,brown,fox,eats,rabbits,on ,a,regular,basis] }
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
field: content Term Document 1 Document 2
aare
basisbrown
commonlyeatsfoxmyon
quick rabbitsregularseen
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
field: content Term Document 1 Document 2
aare
basisbrown
commonlyeatsfoxmyon
quick rabbitsregularseen
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
Inverted indexed
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
where field = “value”
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
where field contains “value”
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
where field contains “value”term filter
“term” : { “title” : “brown” }
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “filtered” : { “query” : { “match_all” : { } }, “filter” : { “term” : { “title” : “brown” } } } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
field: title Term Document 1 Document 2
brown
healthy
keeping
pets
quick
rabbits
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
field: title Term Document 1 Document 2
brown
healthy
keeping
pets
quick
rabbits
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
Result
bitset[ 1, 0 ]cache as “title:brown”
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
where field IN [“value1”, … ]
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
terms filter
“terms” : { “title” : [“quick”, “pets”] }
where field IN [“value1”, … ]
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
field: title Term Document 1 Document 2
brown
healthy
keeping
pets
quick
rabbits
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
Result
bitset[ 1, 1 ]cache as “title:quick title:pets”
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
where field >= “value 1” and field < “value 2”
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
where field >= “value 1” and field < “value 2”
range filter
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
“range” : { “content” : { “gte” : “a”, “lt” : “m”
} }
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
field: content Term Document 1 Document 2
aare
basisbrown
commonlyeatsfoxmyon
quick rabbitsregularseen
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
Result
bitset[ 1, 1 ]cache as “content:[a TO m]”
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
AND … OR … NOT
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
AND … OR … NOTbool filter
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
“bool” : { “must” : [ <filters> ], “should” : [ <filters> ], “must_not” : [ <filters> ], }
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
how relevant is this term ?
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
how relevant is this term ?term query
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “term” : { “title” : “brown” } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
how relevant is this document ?
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
Relevant score
Term in this documentmore is better
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
Relevant score
Term in all documentsless is better
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
Relevant score
How long of this document ?shorter is better
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
AND … OR … NOTbool query
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
“bool” : { “must” : [ <filters> ], #AND “should” : [ <filters> ], ??? “must_not” : [ <filters> ], #NOT }
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
“bool” : { “must” : [ <filters> ], #AND “should” : [ <filters> ], ??? “must_not” : [ <filters> ], #NOT “minimum_should_match” : ? }
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
bool filter -> True | False
bool query -> _score
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
match query
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
match queryhigh level query
understand mapping and analysis
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
match query1. analyze query string2. rewrite query
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
{ “match” : { “title” : “QUICK!” } }
title:quick
{ “term” : { “title” : “quick” } }
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
{ “match” : { “title” : “QUICK FOX!” } }
title:quick OR title:fox
{ “bool” : { “should” : [ { “term” : { “title” : “quick” } }, { “term” : { “title” : “fox” } } ] }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
Need all words must match ?
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “term” : { “title” : “QUICK FOX!” } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “term” : { “title” : { “query” : “QUICK FOX!”, “operator” : “and” } } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ "query": {
"bool": { "must": [
{ "term": { "title" : "quick" } }, { "term": { "title" : "fox" } }
] }
} }
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “match_pharse” : { “title” : { “query” : “QUICK BROWN” } } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “match_pharse” : { “title” : { “query” : “QUICK RABBITS” } } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “match_pharse” : { “title” : { “query” : “BROWN QUICK” } } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
Limit long tail ?
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ “query” : { “term” : { “title” : { “query” : “QUICK BROWN FOX!”, “minimum_should_match” : “75”% } } }}
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
GET /_search{ "query": {
"bool": { "should": [
{ "term": { "title" : "quick" } }, { "term": { "title" : "brown" } }, { "term": { "title" : "fox" } }
], “minimum_should_match” : 2 }
} }
SPRINT3R Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance
Aggregation APIanalytic your dataexplore your data