Battle of the Giants Round 2 - Apache Solr vs. Elasticsearch

Round 2

Battle of the Giants

Rafał Kuć – Sematext Group, Inc.@kucrafal @sematext sematext.com

Ich bin ein…

Sematext consultant & engineerSolr Cookbook series author„ElasticSearch Server” author„Mastering ElasticSearch” authorSolr.pl co-founderFather and husband

Under the Hood

Lucene 4.3Lucene 4.3

ExpectationsScalabilityFault tolerananceHigh availablityFeaturesManageabilityEase of installationTools Support

Expectations vs Reality

Only ElasticSearch nodesSingle leader

Solr + ZooKeeperLeader per shard

DistributedFault tolerant

Automatic leader election

All Time Top Committers

Active Contributors

The Code

The Mailing Lists

Trends

Collection vs Index

Collections and Indices can be spread among different nodes in the cluster

Collection – main logical index

Index – main logical structure

Apache Solr Index Structure

Field and types defined in schemaAutomatic value copyingDynamic fieldsCustom similarityCustom postings formatMultiple document types require shared schemaCan be read using API

ElasticSearch Index Structure

Schema - lessFields and types defined with HTTP APIMulti – field supportNested and parent – child documentsCustom similarity Custom postings format Multiple document with different structureCan be read and written using API

Shards and Replicas

Many shards0 or more replicasReplica can become leader Replicas can be created on live cluster

Configuration

Static in solrconfig.xmlCan be reloaded with

core reload

Static in elasticsearch.yml

Changable at runtime

Discovery

Zen DiscoveryApache Zookeeper

Solr & ZooKeeper

Requires additional softwarePrevents split – brain situationsHolds collections configurationsZooKeeper ensemble needed

ElasticSearch Zen Discovery

Automatic node discoveryMulticast and unicast discovery methodsAutomatic master detectionTwo - way failure detection

HTTP FTW

HTTP REST API in ElasticSearch or Query String for simple queriesHTTP with Query String in Apache SolrBoth provide specialized Java API

Results Grouping

Group on: field value query result function query

Prospective Search

Called PercolatorMatches documents to stored queries

Full Text Search Capabilities

Variety of queriesControl score calculationDifferent query parsers Advanced Lucene queries

Score Calculation

Leverage Lucene scoring Control importance of: documents queries terms phrasesSimiliarity configuration

Apache Solr and Score Influence

Index - time boostingQuery - time

Term boostsField boostsPhrases boostFunction queriesSub-queries used for boosting

ElasticSearch and Score Influence

Index - timeQuery - time

Different queries provide different boost controlsCan calculate distributed term frequenciesNegative and Positive boosting queriesCustom score filters

Scripts

ElasticSearch Query Rescore

Reorders top N hits by using other queryExecuted on shards before results are returned to the node handling itNot executed with scan and count

ElasticSearch Nested Objects

Indexed as separate documentsStored in the same part of index as root docHidden from standard queries and filtersNeed appropriate queries and filters (nested)Top level documents can be sorted on the basis of nested ones

Solr Parent – Child Relationship

Used at query timeMulti core joins possible

select?q={!join from=parent to=id}color:Yellow

ElasticSearch Parent – Child

Proper indexing requiredIndexed as separate documentsStandard queries don’t return child documentsRetrieve parent docs using queries and filters (has_child, has_parent, top_children)

FiltersUsed to narrown down query results

Good candidates for caching and reuse

AddictiveCan use different query parsersCan use local paramsNarrows down faceting results

Defined using Query DSLCan be used for score calculation Doesn’t narrow down faceting results

Faceting

TermsRange & queryTerms statisticsSpatial distance

Pivot Histograms

Real Time Or Not ?

Get not yet indexed docs from transaction logDon’t need searcher reopening

Separate Get and Multi Get API

Separate Realtime Get Handler

Data Handling

Single and batch indexing supported

JSON in / JSON out(and YAML)

Different formats allowed (XML, JSON, CSV, binary)

Partial Document Updates

Not based on LUCENE-3837Server-side doc reindexingBoth servers use versioning Decreases network traffic

Apache Solr Partial Doc Update

Sent to the standard update handlerRequires _version_ field

curl 'localhost:8983/solr/update?commit=true' -H 'Content-type:application/json' -d '[ { "id" : "12345", "enabled" : { "set" : true } } ]'

ElasticSearch Partial Doc Update

Special end – point exposed - _updateSupports parameters like routing, parent, replication, percolate, etc (similar to Index API)Uses scripts to perform document updates

curl -XPOST 'localhost:9200/sematext/test/12345/_update' -d '{ "script" : "ctx._source.enabled = enabled", "params" : { "enabled" : true }}'

Solr Collections API

Collection creation reload deletion shards splitting

ElasticSearch Indices REST API

Index creation deletion closing and opening refreshing existence checking

Apache Solr Shard Splitting

admin/collections?action=SPLITSHARD&collection=collection1&shard=shard1

Cluster State Monitoring

Multiple MBeans exposed by JMX

Multiple REST end – points exposed to get different statistics

ElasticSearch Statistics API

Health and state checkNodes informationCache statisticsSegments informationIndex informationMappings information

SPM – „One to rule them all”

ElasticSearch Cluster Settings Update

Control rebalancing recovery allocationChange cluster configuration properties

ElasticSearch Custom Shard Allocation

Cluster level:

Index level:

curl -XPUT localhost:9200/_cluster/settings -d '{ "persistent" : { "cluster.routing.allocation.exclude._ip" : "192.168.2.1" }}'

curl -XPUT localhost:9200/sematext/_settings/ -d '{ "index.routing.allocation.include.tag" : "nodeOne,nodeTwo"}'

Moving Shards and Replicas

Move shards between nodes on demand

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{ "commands" : [ {"move" : {"index" : "sematext", "shard" : 0, "from_node" : "node1", "to_node" : "node2"}}, {"allocate" : {"index" : "sematext", "shard" : 1, "node" : "node3"}} ] }'

The Verdict

And The Winner Is ?

The Users

We Are Hiring !

Dig Search ?Dig Analytics ?Dig Big Data ?Dig Performance ?Dig working with and in open – source ?We’re hiring world – wide !

http://sematext.com/about/jobs.html

Rafał Kuć @kucrafal rafal.kuc@sematext.com

Sematext @sematext http://sematext.com http://blog.sematext.com

ElasticSearch Server 25% off:MREESS25

Thank You !

Battle of the Giants Round 2 - Apache Solr vs. Elasticsearch

Technology

Combining Solr and Elasticsearch to Improve Autosuggestion ...files.meetup.com/7646592/2016-10-12 Elastic Meetup Local Autosuggest.pdfautosuggestion feature – Number of typed characters

Big-Data-Technologien - Überblick - ihk-nuernberg.de · Suche mit Solr/ElasticSearch . Prof. Dr. Jens Albrecht Big Data Technologien

Side by Side with Elasticsearch and Solr

Suchgetriebene Anwendungen mit Elasticsearch und Solr · Autocomplete Did you mean Synonyme . Für Suchmaschinen gibt es viele Einsatzszenarien Volltextsuche Speicher Cache Geo-Suche

Amazon CloudSearch Amazon Elasticsearch Service...Amazon Elasticsearch Service AWS Black Belt Tech Webinar 2016 ... •CloudSearchとAmazon ESの比較. Agenda ... –Lucene Solr RevolutionでのEvernoteやLinkedInの発表

Scaling Solr with Solr Cloud

in Lucene, Solr and ElasticSearch and the eco-system...Language support and linguistics in Lucene, Solr and ElasticSearch and the eco-system Christian Moen cm@atilika.com June 3rd,

iFinder Enterprise Search für Mitarbeiterportal „MyRhenus“ · Lucene, Solr und Elasticsearch. • Das Enterprise Search-Produkt iFinder umfasst eine benutzerfreundliche und leistungsfähige

Elasticsearch and Solr for Logs

Search Evolution - Von Lucene zu Solr und ElasticSearch (Majug 20.06.2013)

Search Evolution - Von Lucene zu Solr und ElasticSearch

Solr and ElasticSearch demo and speaker feb 2014

Improving Drupal search experience with Apache Solr and ... · Improving Drupal search experience with Apache Solrand Elasticsearch When to use Elasticsearch 33 • The Elastic Stack

Solr vs ElasticSearch

Solr Fusion a Solr Proxy

Elasticsearch vs Apache Solr - eValais.chevalais.ch/wp-content/uploads/2017/10/Elasticsearch-vs-Solr.pdf · Elasticsearch vs Apache Solr Battle of the Open-Source Search Giants !

Combining Solr and Elasticsearch to Improve Autosuggestion ... · Apache: Big Data 2015 Combining Solr and Elasticsearch to Improve Autosuggestion on Mobile Local Search Toan Vinh

Lucene Tutorial - Klinton Bicknell · Introduction to Information Retrieval Open source IR systems Widely used academic systems ... Things built on it: Solr, ElasticSearch A few others

Battle of the giants: Apache Solr vs ElasticSearch

Optimizing SOLR to Improve Searchinfo2.magento.com/rs/magentosoftware/images/SOLR... · Agenda ! Overview of SOLR ! Basic Solr Troubleshooting – Common SOLR Troubleshooting and