Gregg Donovan @Lucene Revolution 2011

8/6/2019 Gregg Donovan @Lucene Revolution 2011

1/84


2/84

1.5 years Solr & Lucene at Etsy.com

3 years Solr & Lucene at TheLadders.com


3/84


4/84

8+ million members


5/84

9.3 million items


6/84


7/84

1+ billion pageviews / month


8/84


9/84


10/84


11/84


12/84


13/84


14/84

Maximize Solr out-of-the-box


15/84


16/84

Know when to do each


17/84


18/84

Or


19/84


20/84

Dont fear trunk


21/84

builds.apache.org/job/Solr-trunk/changes


22/84


23/84


24/84


25/84

{!lucene}

{!field}

{!func}

{!dismax}

{!edismax}

{!boost}

{!term}


26/84

Cheap ranking awesomeness


27/84


28/84

ExternalFileField ftw!


29/84

schema.xml:

/search/data/treasury/external_hotness.1306390802088:

1=2.32=1.7

3=1.1

Solr query:

sort={!func}hotness+desc


30/84

ExternalFileField caveats


31/84

More relevance: boost query


32/84

http://localhost:8983/solr/listings/select?q={!boost b=$rel v=$qq}

&rel=category:furniture^10+OR+((-material:acrylic)

^5)

&qq=desk
http://livepage.apple.com/http://livepage.apple.com/http://livepage.apple.com/http://livepage.apple.com/


33/84

Impression tracking


34/84

etsy.com/search?q=desk&explain=1


35/84

Side-by-Side testing


36/84


37/84

Cheap performance wins


38/84

Put off sharding till you must


39/84

cat ${indexDir}/* > /dev/null


40/84

Return IDs, minimize stored fields


41/84

RAM: $10-20 / GB


42/84

SSD: 0.1ms vs 10ms seek


43/84

Custom?


44/84

solr-user


45/84

Tools for low-level hacking


46/84

Continuous deployment


47/84


48/84

One button.So easy a dog could do it.


49/84


50/84


51/84


52/84


53/84


54/84

github.com/etsy/logster


55/84

Tracking GC


56/84

export GC_DEBUG="-verbose:gc -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:

+PrintGCApplicationStoppedTime -XX:+PrintAdaptiveSizePolicy -

XX:AdaptiveSizePolicyOutputInterval=1 -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -Xloggc:/var/log/search/gc.log"


57/84


58/84


59/84

Alerting


60/84

Testing


61/84


62/84

SaveAsFixture


63/84

Profiling


64/84

Java Primitive Libraryfastutil

trove4j


65/84

Know the hooks

QParserPlugin

SolrEventListener

SolrRequestHandler

SearchComponent

SolrCache

ValueSourceParser


66/84

SolrIndexSearcher gotchasreference countingusing it as a cache key:

WeakHashMap myCache...


67/84

Example:personalized collections


68/84


69/84

fq={!term f=id}123 OR {!term f=id}456


70/84

Need a map of PK to docId


71/84

Use custom SolrCache plus SolrEventListener

to fill it


72/84


73/84

i18n currency sorting and filtering


74/84


75/84

currency.xml:

! ! !

! !

! !

! !

...

!

!

! !

! !

! !

! !

...


76/84

price:[10.00USD to 50.00USD]

price:20.00EUR

price:[$10.00 to $50.00]


77/84


78/84

Replication gotcha


79/84

SOLR-2202


80/84

Related Searches


81/84

Autosuggest!

bjewlery dewelry ejewelry ejwelry ewelery ewerly ewlery fewelryfewlery fjewelery fjewelry gewerly gewlery hewelery hewelry hewerly

h l hj l i l ij l j l j l j l j l


82/84

hewlery hjewelry iewelry ijewelry jawelery jawlery jeawlery jeelery

jeelryjeewelery jeewelry jeewlery jeewlry jefwelry jejelry jelelry

jelery jellery jelwelery jelwelry jelwlery jemelry jemerly jemwelry

jeqwelry jerelery jerelry jerely jererly jerlery jerwelery jerwelry

jerwely jerwerlyjeselery jeselry jevelry jeverly jewalery jewdelryjewedlry jeweelrry jeweelry jeweely jeweer jeweery jeweilry jeweiry

jewejery jewejlry jewejrly jewejry jewekey jewekry jewelary jeweldy

jewele jewelee jewelelryjewelera jewelerey jewelerly jewelert

jewelerty jeweleru jeweleruy jeweleryl jewelerys jeweleryy jewelet

jewelety jeweleya jewelfry jewelfy jeweliy jewellryp jewelltry

jewelly jewelory jewelra jewelray jewelrejewelree jewelreyyjewelrfy jewelrh jewelri jewelrky jewelrly jewelrr jewelrs jewelrsy

jewelrt jewelrty jewelru jewelruy jewelrye jewelryh jewelryl

jewelrym jewelryr jewelrys jewelryt jewelryu jewelryuk

jewelryyjewelrz jewelsry jewelsy jeweltry jewelty jewelw jewelwery

jewelwey jewelwy jewelya jewelyj jewelyr jewelyry jewelyu jewelyy

jewelzry jeweory jewerey jeweriy jewerky jewerlary jewerley jewerlijewerlly jewerlsjewerlt jewerlu jewerlyh jewerlyr jewerlys jewerlyu

jewerry jeweryl jewetry jewewlry jewewly jewewrly jewewry jeweylry

jewiery jewilary jewkery jewlary jewledy jewleery jewlelery jewlely


83/84


84/84

Documents

Gregg Donovan @Lucene Revolution 2011