If you can't read please download the document
Upload
charlie-hull
View
1.439
Download
0
Embed Size (px)
Citation preview
Charlie Hull - Managing Director20th October 2015Search Solutions
www.flax.co.uk/blog+44 (0) 8700 118334Twitter: @FlaxSearch
Towards a new model of test-based relevance tuning
Building open source search applications since 2001
Independent, honest advice and analysis
Expert design & development, Apache Solr committers
Test-driven relevancy and performance tuning
Custom training & mentoring for your staff
Flexible support up to 24/7/365 with SLAs
Building open source search applications since 2001
Independent, honest advice and analysis
Expert design & development, Apache Solr committers
Test-driven relevancy and performance tuning
Custom training & mentoring for your staff
Flexible support up to 24/7/365 with SLAs
Come and join the open source search community (tonight?)
Why bother testing?
Throwing it over the fence
Some (slightly) better methods
A collaborative model
Quepid & other tools
A better way to test
@FlaxSearch
Search is Magic
Search doesn't affect the bottom line
The new search engine is better than the old one
We can just fix this one problem here...
Why bother testing?
@FlaxSearch
Business users / content creators know search is broken
Managers tell search developers to 'fix it'
Search developers don't understand why it needs fixing
Business users / content creators don't understand side effects of a fix
Bad communication, internal politics, search gets worse!
Throwing it over the fence
@FlaxSearch
Avoiding the HiPPO
@FlaxSearch
Copyright William J Bagshaw and licensed for reuse under this Creative Commons Licence
Identify what to testQuery logs
'Most valuable' queries
Languages/markets
Segmented query types
Keep proper recordsManual query testing
Record relevance judgements Per page or per result?
Say why
Have an overall score
Use the same test system
Some (slightly) better methods
@FlaxSearch
Some (slightly) better methods
@FlaxSearch
Problems:Slow iterations
Lots of error-prone copy-and-paste
Unwieldy
Not really collaborative
Improvements:Build test UIs
Use a better scoring algorithme.g. average discounted distance
Bring in other data e.g. web analyticsBut remember clicks only 45% predict relevance (75% is achievable) (Susan Dumais, Microsoft)
Some (slightly) better methods
@FlaxSearch
In software we use tests to collaborate1. What should happen in this case?2. Write test code to check
TDD can lead to improved software quality
Why not with search relevancy?
Test-based relevancy
@FlaxSearch
A client approached Flax:What's the current state of the art /academically proven way to test relevance?
Er.....
I remembered something called Quepid...
So this happened....
@FlaxSearch
Built by Doug Turnbull of OpenSource Connections
A browser-based tool for tuning relevance
Needed some development for enterprise use - We did this working with our client
Let's take a look...
Quepid
@FlaxSearch
You should test your searches in a methodical way
Collaboration between 'the business' and developers is vital
Some tools now exist to help
Hopefully this is the first step to better relevance tuning
Conclusions
@FlaxSearch
Thankyou!
Any questions?
[email protected]/blog+44 (0) 8700 118334Twitter: @FlaxSearch