25
NLGraphs: A Hybrid Approach toward Interac9vely Querying Seman9c Data Khadija Elbedweihy Suvodeep Mazumdar Stuart N. Wrigley Fabio Ciravegna OAK Group University of Sheffield Sheffield, UK k.Elbedweihy | s.mazumdar | s.wrigley | f.ciravegna | @ shef.ac.uk ovus00

NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

  • Upload
    others

  • View
    5

  • Download
    0

Embed Size (px)

Citation preview

Page 1: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

NL-­‐Graphs:  A  Hybrid  Approach  toward  Interac9vely  Querying  Seman9c  Data  

Khadija  Elbedweihy  Suvodeep  Mazumdar  Stuart  N.  Wrigley  Fabio  Ciravegna  

OAK  Group    University  of  Sheffield  

Sheffield,  UK  

k.Elbedweihy  |  s.mazumdar  |  s.wrigley  |  f.ciravegna  |  @  shef.ac.uk          ovus00  

Page 2: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Users  and  User  Experience  

•  Seman9c  Search  System    •  Assist  users  sa9sfy  their  informa9on  needs  

•  User  Sa9sfac9on  –  Query  approach  –  Performance  of  Search  system  –  Presenta9on  of  results  –  ….  

•  Usability  and  User  Experience  largely  ignored  –  Need  to  focus  on  users  

Page 3: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Query  Approaches  

•  Types  •  Interac9on  mechanisms  •  Formalisms  (Natural  language  –  Forms  –  Formal  Query)  

•  O[en  Task  or  applica9on  Specific  –  Keyword-­‐based  approach  to  quickly  iden9fy  possible  relevant  

instances  –  Formal  query  for  more  complex  tasks  (nega9on,  group-­‐by  etc.)  

•  Most  SW  applica9ons  focus  on  one  approach  

Page 4: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

 

   Esther  Kaufmann  and  Abraham  Bernstein.  Evalua9ng  the  usability  of  natural  language  query  languages  and  interfaces  to  seman9c  web  knowledge  bases.  Journal  of  Web  Seman/cs,  8(4):377–393,  2010.    

 

Spectrum  of  Query  Approaches  

4  

Page 5: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Issues  +  Mo9va9ons  •  Comparison  of  Query  Approaches  (Expert  and  Casual)  [2]  

§  NL  Approach  + Prior  experience  with  Google-­‐like  approaches  + Easy  and  quick  to  write  ques9ons/keywords  -  Less  support  for  Complex  queries    

§  View  Based  Approach  + Good  support  in  complex  queries  + Visualising  the  search  space  -  Time  and  effort  required  formula9ng  queries  

•  Learnability  study  with  Graph  based  approach  (Experts)  [3]  +  Improvement  in  user  sa9sfac9on  and  performance  over  9me  -  S9ll  needed  9me  and  effort  to  formulate  queries  -  Extensive  use  of  keyword-­‐search  for  concepts    

[2]  K.  Elbedweihy,  S.  N.  Wrigley,  and  F.  Ciravegna.  Evalua9ng  Seman9c  Search  Query  Approaches  with  Expert  and  Casual  Users.  In  Proceedings  of  the  Evalua9ons  and  Experiments  Track,  ISWC,  2012.  [3]  S.Mazumdar,  D.Petrelli,  K.  Elbedweihy,  V.  Lanfranchi,  and  F.  Ciravegna.  Affec9ve  Graphs:  The  visual  appeal  of  linked  data.  Seman9c  web  :  interoperability,  usability,  applicability,  2013.  In  Press.  

Page 6: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Affec9ve  Graphs  

Page 7: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Affec9ve  Graphs  

•  Demo  

Page 8: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

HybridGraphs  

•  NL  [4]  +  Graph  [3]  =  Hybrid  Query  Approach  •  Builds  on  Advantages  of  either  approach  

•  Quick  access  to  relevant  data    •  Easy  and  visual  means  to  formulate  complex  queries  •  Visually  inspect  search  space  

•  Alleviates  the  disadvantages  •  Lesser  9me  taken  to  build  complex  queries  

 

 [3]  S.Mazumdar,  D.Petrelli,  K.  Elbedweihy,  V.  Lanfranchi,  and  F.  Ciravegna.  Affec9ve  Graphs:  The  visual  appeal  of  linked  data.  Seman9c  web  :  interoperability,  usability,  applicability,  2013.  In  Press.  [4]  K.  Elbedweihy,  S.N.  Wrigley,  F.  Ciravegna,  and  Z.  Zhang  (2013).  Using  BabelNet  in  bridging  the  gap  between  natural  language  queries  and  linked  data  concepts.  NLP  &  DBpedia  Workshop,  12th  Interna9onal  Seman9c  Web  Conference  (ISWC2013),  21-­‐25  October,  Sydney,  Australia.    

Page 9: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Hybrid  Approach  

Page 10: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Evalua9on  •  Previous  evalua9ons  on  NL  component  and  Affec9ve  Graphs  [3,4]  

(performance,  usability,  effec9veness  etc.)  •  Focus  on  hybrid  approach  as  a  new  query  mechanism  and  the  

support  provided    •  Methodology  

–  Expert  (SW  experts  in  Computer  Science  +  So[ware  development  firm)  and  Casual  users  (staff  and  students  at  the  University  of  Sheffield)  

–  24  subjects  (12  +  12)  –  Aged  between  18  and  53,  mean  age  31  years  –  Short  demonstra9on  of  a  scenario  based  example  –  Familiarisa9on  with  the  system  

[3]  S.Mazumdar,  D.Petrelli,  K.  Elbedweihy,  V.  Lanfranchi,  and  F.  Ciravegna.  Affec9ve  Graphs:  The  visual  appeal  of  linked  data.  Seman9c  web  :  interoperability,  usability,  applicability,  2013.  In  Press.  [4]  K.  Elbedweihy,  S.N.  Wrigley,  F.  Ciravegna,  and  Z.  Zhang  (2013).  Using  BabelNet  in  bridging  the  gap  between  natural  language  queries  and  linked  data  concepts.  NLP  &  DBpedia  Workshop,  12th  Interna9onal  Seman9c  Web  Conference  (ISWC2013),  21-­‐25  October,  Sydney,  Australia.    

Page 11: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Evalua9on  Tasks  

•  5  ques9ons  derived  from  QALD  dataset  [5]  •  Different  levels  of  difficulty  

–  1.  When  was  Capcom  founded?  –  2.  What  did  Bruce  Carver  die  from?  –  3.  Who  was  the  wife  of  U.S.  president  Lincoln?  –  4.  Give  me  all  ci9es  in  Alaska  with  more  than  10000  inhabitants.  –  5.  Show  me  all  songs  from  Bruce  Springsteen  released  between  1980  

and  1990.  

[5]  V.  Lopez,  C.  Unger,  P.  Cimiano,  and  E.  Moqa.  Evalua9ng  ques9on  answering  over  linked  data.  Web  Seman9cs:  Science,  Services  and  Agents  on  the  World  Wide  Web,  2013.  

Page 12: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Evalua9on  Analysis  

•  Objec9ve  and  Subjec9ve  data  collected  •  Input  Time,  number  of  aqempts,  Success  rate    •  Post  evalua9on  ques9onnaires  –  System  Usability  Scale  (SUS),  

Extended  ques9onnaire  –  The  query  construc9on  process  was  X.  (5-­‐point  Likert  scale,  ranging  

from  Laborious  to  Effortless)  –  What  did  you  like  about  the  hybrid  approach  as  a  mechanism  for  

expressing  your  query?  and  why?  –  What  things  you  didn't  like  about  the  hybrid  approach  as  a  mechanism  

for  expressing  your  query?  and  why?  •  Aesthe9cs  ques9onnaire  

Page 13: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Results  

•  Overall,  experts  (73.75)  liked  NL  Graphs  more  than  casuals  (61.25)  –  Graph  based  approach  more  complicated  –  Experts  more  familiar  with  Seman9c  Web  and  graph  data  

•  Success  rate  100%  (all  users  were  able  to  successfully  complete  the  tasks)  mostly  on  one  aqempt  

•  Query  Construc9on  process  scored  effortless  •  Most  users  commented  posi9vely  on  the  usability  and  support  

provided  by  the  hybrid  approach  •  Four  users  (1e  +  3c)    

–  Complicated  and  non-­‐intui9ve  –  Why  are  there  more  steps  required  to  build  queries  than  with  Google?  –  “It  seemed  an  extra  step  to  get  to  your  answer  rather  than  just  typing  

in  a  search  and  it  appearing  in  results.”  

Page 14: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Results  Contd.  

•  More  assistance  needed/desired  •  Need  for  more  Training    

•  Experts  took  longer  (94.48s)  to  formulate  queries  than  casual  (76.88s)  users  –  Follow  logic  and  prior  understanding  of  SW  concepts  –  Increased  interest  in  the  interface,  trying  out  new  features,  exploring  

technical  aspects  (e.g.  SPARQL  query  generated)    

Page 15: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Results  Contd.  

Page 16: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Results  Contd.  

•  Q4  –  Give  me  all  ci9es  in  Alaska  with    more  than  10000  inhabitants  

–  Use  of  terms  like  “Alaskan”  –  DbPedia  proper9es  such  as  isPartOf  –  Numerical  constraints  >10000    

•  Q5  -­‐  Show  me  all  songs  from  Bruce  Springsteen  released  between  1980  and  1990  –  Sezng  dates  as  a  constraint  is  a  manual  process  

Page 17: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Summary  

•  Hybrid  approach  to  querying  Linked  Data  •  It  works  !  J  •  High  SUS  scores  indicate  usability  and  usefulness  •  Future  work    

–  Several  re-­‐designs  iden9fied/suggested  in  the  evalua9on  –  A  longer  term  study  directly  comparing  Hybrid  approach  to  other  

approaches  in  iden9cal  condi9ons  –  Study  with  larger  set  of  user  groups    –  Improving  NL  system  

[3]  S.Mazumdar,  D.Petrelli,  K.  Elbedweihy,  V.  Lanfranchi,  and  F.  Ciravegna.  Affec9ve  Graphs:  The  visual  appeal  of  linked  data.  Seman9c  web  :  interoperability,  usability,  applicability,  2013  

Aesthe9c  design  in  Seman9c  Web  Applica9ons  [3]  

Page 18: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Thank  You  

• Ques9ons  ?  

Page 19: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Hybrid  Approach  

•  Usability  +  Interac9vity  •  NL  [4]  +  Graph  [3]  =  Hybrid  Query  Approach  •  Builds  on  Advantages  of  either  approach  

–  Quick  access  to  relevant  data    –  Easy  and  visual  means  to  formulate  complex  queries  –  Visualise  nearby  concepts    –  Visually  inspect  search  space  

•  Alleviates  the  disadvantages  –  Lesser  9me  taken  to  build  complex  queries  

[3]  S.Mazumdar,  D.Petrelli,  K.  Elbedweihy,  V.  Lanfranchi,  and  F.  Ciravegna.  Affec9ve  Graphs:  The  visual  appeal  of  linked  data.  Seman9c  web  :  interoperability,  usability,  applicability,  2013.  In  Press.  [4]  K.  Elbedweihy,  S.N.  Wrigley,  F.  Ciravegna,  and  Z.  Zhang  (2013).  Using  BabelNet  in  bridging  the  gap  between  natural  language  queries  and  linked  data  concepts.  NLP  &  DBpedia  Workshop,  12th  Interna9onal  Seman9c  Web  Conference  (ISWC2013),  21-­‐25  October,  Sydney,  Australia.    

Page 20: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Quan9ta9ve  Evalua9on  

•  Query  Execu9on  phase  (98ms  to  19.9s)          

20  

Page 21: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Evalua9ng.views. –  Performance  Test  Results  

     

21  

Remote  database  Test  Results  –  Variability  between.  0.00026  ms  to  6.48ms  

Page 22: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

10  Principles  of  Aesthe9cs  for  SW  

•  Review  of  literature  10  general  guidelines  &  

22  

Mazumdar,  S.,  Petrelli,  D.,  Elbedweihy,  K.,  Lanfranchi,  V.,  &  Ciravegna,  F.  Affec9ve  Graphs:  The  Visual  Appeal  of  Linked  Data.  Seman9c  Web:  Interoperability,  Usability,  Applicability.  

Page 23: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

9  node-­‐linked  visualisa9on  guidelines    

Page 24: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Measuring  Aesthe9cs  

•  Screenshots  of  interfaces  fed-­‐in  to  a  Java  program,  where  each  element  was  “marked-­‐up”  using  mouse  gestures  

24  

 D.C.L.  Ngo,  A.  Samsudin,  and  R.  Abdullah.  Aesthe9c  measures  for  assessing  graphic  screens.  J.  INF.  SCI.  ENG,  16(1):97–116,  2000.  D.C.L.  Ngo,  L.S.  Teo,  and  J.G.  Byrne.  Modelling  interface  aesthe9cs.  Informa9on  Sciences,  152:25–46,  2003.      

Page 25: NL#Graphs:+A+Hybrid+Approach+toward+ …NL#Graphs:+A+Hybrid+Approach+toward+ Interac9vely+Querying+Seman9c+Data Khadija Elbedweihy+ Suvodeep+Mazumdar+ StuartN.+Wrigley+ Fabio+Ciravegna

Results  

25  

0.652  

0.561