View
2.473
Download
3
Category
Tags:
Preview:
Citation preview
Creare il proprio motore di ricerca con Apache Solr
alfonso.focareta@pronetics.it (@afocareta) Pro-netics S.p.A.angelo.quercioli@pronetics.it Pro-netics S.p.A
Alfonso FocaretaAngelo Quercioli
Solr & LuceneAlfonso FocaretaAngelo Quercioli
alfonso.focareta@pronetics.it Angelo.quercioli@pronetics.it
Lucene: featuresAlfonso FocaretaAngelo Quercioli
• High performance, full-text & scalable search library
• 100% pure Java
• Focus: Indexing + Searching Documents (“Document” is just a list of name+value pairs)
• No crawlers or document parsing Flexible Text Analysis (tokenizers + token filters)
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: featuresAlfonso FocaretaAngelo Quercioli
• A full text search server based on Lucene• XML/HTTP, JSON Interfaces• Faceted Search (category counting)• Flexible data schema to define types and fields• Hit Highlighting• Configurable Advanced Caching• Index Replication• Extensible Open Architecture, Plugins• Web Administration Interface• Written in Java5, deployable as a WAR
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: licenseAlfonso FocaretaAngelo Quercioli
OPEN SOURCE!!Apache License
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: ArchitectureAlfonso FocaretaAngelo Quercioli
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Installing and StartingAlfonso FocaretaAngelo Quercioli
• JDK5 or above intsalled
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
http://localhost:8983/solr/admin/ in your web browser for admin it
Solr: Define a schema.xmlAlfonso FocaretaAngelo Quercioli
Define a Schema (schema.xml)
The file schema.xml describes the structures of the data indexed.
• Type definitions• Field definitions• CopyField section• Additional definitions
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Define a schema.xml (type definition)Alfonso FocaretaAngelo Quercioli
Type Definition
List of type and component (simple and complex)• Primitive type• WhiteSpaceTokenizerFactory• StopFilterFactory• WordDelimiterFilterFactory• LowerCaseFilterFactory• SnowBallFilterFactory (stemming)
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Define a schema.xml (type definition- example)
Alfonso FocaretaAngelo Quercioli
Type Definition - Example
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Define a schema.xml (type definition- example)
Alfonso FocaretaAngelo Quercioli
Field Definitions
• Field Attributes: name, type, indexed, stored, multiValued, omitNorms, termVectors
<field name="id“ type="string" indexed="true" stored="true"/><field name="sku“ type="textTight” indexed="true" stored="true"/><field name="name“ type="text“ indexed="true" stored="true"/><field name=“inStock“ type=“boolean“ indexed="true“ stored=“false"/><field name=“price“ type=“sfloat“ indexed="true“ stored=“false"/><field name=“category“ type=”text_ws“ indexed=”true” stored=“true”
multiValued="true"/>
• Dynamic Fields
<dynamicField name="*_i" type="sint“ indexed="true" stored="true"/><dynamicField name="*_s" type="string“ indexed="true" stored="true"/><dynamicField name="*_t" type="text“ indexed="true" stored="true"/>
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Define a schema.xml (Copy Field- example)
Alfonso FocaretaAngelo Quercioli
Copy Field
Copies one field to another at index time.Case#1: Analyze same field different ways
– copy into a field with a different analyzer– boost exact-case, exact-punctuation matches– language translations, thesaurus, soundex
<field name=“title” type=“text”/><field name=“title_exact” type=“text_exact” stored=“false”/><copyField source=“title” dest=“title_exact”/>
Case #2: Index multiple fields into single searchable field
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Indexing MethodAlfonso FocaretaAngelo Quercioli
Indexing Method
You put documents in it (called "indexing") via :
• XML• JSON• CSV• Binary over http (multipart request)
alfonso.focareta@pronetics.it Angelo.quercioli@pronetics.it
Solr: Indexing (Java Api)Alfonso FocaretaAngelo Quercioli
Indexing by Solrj
Send an xml like this
alfonso.focareta@pronetics.it Angelo.quercioli@pronetics.it
<add><doc <field name=“id”>043564</field> <field name=“name”>Alfonso</field> <field name=“surname”>Focareta</field> <field name=“category”>developer</field> <field name=“language”>Italian</field> <field name=“language”>English</field></doc></add>
Solr: Indexing (Solrj)Alfonso FocaretaAngelo Quercioli
Solrj
Solrj is a java client to access solr, It offers a java interface to add, update, and query the solr index
Example ->
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Indexing (Solrj) ExampleAlfonso FocaretaAngelo Quercioli
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Delete DocumentAlfonso FocaretaAngelo Quercioli
Delete document(s)
• Delete by Id(most efficient)<delete>
<id>05591</id> <id>32552</id>
</delete>
• Delete by Query<delete>
<query>language:english</query>
</delete>
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Commit and OptimizeAlfonso FocaretaAngelo Quercioli
Commit and Optimize
Commit : when you are indexing documents to Solr none of the changes you are making will appear until you run the commit command!
Optimize: the command that reorganize the index into segments (increasing search speed) and remove any deleted (replaced) documents.
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: SearchingAlfonso FocaretaAngelo Quercioli
Searching You can search document in Solr by http or by solrj
library.http:/
/localhost:8983/solr/select?q=language:italian&start=0&rows=2&fl=name,surname
<response> <result numFound=“15" start="0"> <doc> <str name=“name">Angelo</str> <str name=“surname”>quercioli</str> </doc> <doc> <str name=“name">Alfonso</str> <str name=“surname”>Focareta</str> </doc> </result></response>
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Searching (Response Format)Alfonso FocaretaAngelo Quercioli
Response FormatYou can add &wt=json for JSON formatted response
{“result": {"numFound":15, "start":0, "docs": [ {“name”:”Angelo”, “surname”:”Quercioli”}, {“name”:” Alfonso”, “surname”:” Focareta”} ]}
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Searching – Query SyntaxAlfonso FocaretaAngelo Quercioli
Lucene Query Syntax
• Italian englishEquiv: italian OR englishQueryParser default operator is “OR”/optional
• Wildcard searches: ang?o, alf*o, rom*
• +italian+english –name:angelo Equiv: italian AND english NOT name:angelo
• “justice league” –name:aquaman• releaseDate:[2012-01-01T00-00-00Z TO 2013-12-
31T23:59:59Z]• description:“legge roma”~100•
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Searching – Query Syntax 2Alfonso FocaretaAngelo Quercioli
Lucene Query Syntax 2
• *:*• (angelo AND “pier francesco”) OR
(+federico +paolo)
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Function QueryAlfonso FocaretaAngelo Quercioli
Function Query• Allows adding function of field value to score– Boost recently added or popular documents
• Current parser only supports function notation• Example: log(sum(popularity,1))• sum, min, max, log, sqrt, currency, ms … etc• scale(x, target_min, target_max)– calculates min & max of x across all docs
• map(x, min, max, target)– useful for dealing with defaults
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Boosted QueryAlfonso FocaretaAngelo Quercioli
Boosted Query
• Score is multiplied instead of added– New local params {!...} syntax added
&q={!boost b=sqrt(popularity)}”super man”
• Parameter dereferencing in local params&q={!boost b=$boost v=$userq}&boost=sqrt(popularity)&userq=“super man”
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Solr: Facet QueryAlfonso FocaretaAngelo Quercioli
Facet QueryFaceted search breaks up search result into multiple
categories
http://solr/select?q=foo&wt=json&indent=on &facet=true&facet.field=cat &facet.query=price:[0 TO 100] &facet.query=manu:IBM
{"response":{"numFound":26,"start":0,"docs":[…]}, “facet_counts":{ "facet_queries":{ "price:[0 TO 100]":6, “manu:IBM":2}, "facet_fields":{ "cat":[ "electronics",14, "memory",3, "card",2, "connector",2] }}}
alfonso.focareta@pronetics.it angelo.quercioli@pronetic.it
Solr: Filter QueryAlfonso FocaretaAngelo Quercioli
Filter Query
• Filters are restrictions in addition to the query• Use in faceting to narrow the results• Filters are cached separately for speed
User queries for memory, query sent to solr is &q=memory&fq=inStock:true&facet=true&…2. User selects 1GB memory size &q=memory&fq=inStock:true&fq=size:1GB&…3. User selects DDR2 memory type &q=memory&fq=inStock:true&fq=size:1GB &fq=type:DDR2&…
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Demo!Alfonso FocaretaAngelo Quercioli
Demo!
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Demo!Alfonso FocaretaAngelo Quercioli
Questions ?
alfonso.focareta@pronetics.it angelo.quercioli@pronetics.it
Recommended