1KJE.n
Summary Statistics
Run ID 1KJE.n
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 915
Summary Statistics
Run ID 1KJE.n
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1193
Recall Level Precision Averages
Recall Precision
0.00 0.6592
0.10 0.5252
0.20 0.3953
0.30 0.3098
0.40 0.2667
0.50 0.2090
0.60 0.1456
0.70 0.1037
0.80 0.0581
0.90 0.0333
1.00 0.0199
Average precision over all
relevant documents
non-interpolated 0.2255
Document Level Averages
Precision
At 5 docs 0.3774
At 10 docs 0.3019
At 15 docs 0.2654
At 20 docs 0.2406
At 30 docs 0.1981
At 100 docs 0.0957
At 200 docs 0.0590
At 500 docs 0.0298
At 1000 docs 0.0173
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2478
Recall Level Precision Averages
Recall Precision
0.00 0.7127
0.10 0.5159
0.20 0.4017
0.30 0.3323
0.40 0.2806
0.50 0.2185
0.60 0.1778
0.70 0.1237
0.80 0.0554
0.90 0.0369
1.00 0.0098
Average precision over all
relevant documents
non-interpolated 0.2360
Document Level Averages
Precision
At 5 docs 0.4302
At 10 docs 0.3434
At 15 docs 0.3057
At 20 docs 0.2811
At 30 docs 0.2396
At 100 docs 0.1221
At 200 docs 0.0773
At 500 docs 0.0388
At 1000 docs 0.0225
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2603
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
1KJE
Summary Statistics
Run ID 1KJE
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 930
Summary Statistics
Run ID 1KJE
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1209
Recall Level Precision Averages
Recall Precision
0.00 0.6356
0.10 0.5108
0.20 0.4256
0.30 0.3269
0.40 0.2755
0.50 0.2318
0.60 0.1771
0.70 0.1147
0.80 0.0702
0.90 0.0408
1.00 0.0270
Average precision over all
relevant documents
non-interpolated 0.2389
Document Level Averages
Precision
At 5 docs 0.3925
At 10 docs 0.3151
At 15 docs 0.2730
At 20 docs 0.2538
At 30 docs 0.2107
At 100 docs 0.0979
At 200 docs 0.0597
At 500 docs 0.0289
At 1000 docs 0.0175
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2588
Recall Level Precision Averages
Recall Precision
0.00 0.6968
0.10 0.5251
0.20 0.4535
0.30 0.3680
0.40 0.3090
0.50 0.2503
0.60 0.1985
0.70 0.1150
0.80 0.0587
0.90 0.0338
1.00 0.0120
Average precision over all
relevant documents
non-interpolated 0.2549
Document Level Averages
Precision
At 5 docs 0.4415
At 10 docs 0.3792
At 15 docs 0.3358
At 20 docs 0.3123
At 30 docs 0.2566
At 100 docs 0.1258
At 200 docs 0.0768
At 500 docs 0.0378
At 1000 docs 0.0228
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2811
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
1KJE3
Summary Statistics
Run ID 1KJE3
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 961
Summary Statistics
Run ID 1KJE3
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1240
Recall Level Precision Averages
Recall Precision
0.00 0.6592
0.10 0.5349
0.20 0.4400
0.30 0.3504
0.40 0.2936
0.50 0.2358
0.60 0.1849
0.70 0.1050
0.80 0.0652
0.90 0.0406
1.00 0.0270
Average precision over all
relevant documents
non-interpolated 0.2466
Document Level Averages
Precision
At 5 docs 0.4075
At 10 docs 0.3151
At 15 docs 0.2767
At 20 docs 0.2632
At 30 docs 0.2088
At 100 docs 0.1013
At 200 docs 0.0616
At 500 docs 0.0305
At 1000 docs 0.0181
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2638
Recall Level Precision Averages
Recall Precision
0.00 0.7192
0.10 0.5499
0.20 0.4713
0.30 0.3862
0.40 0.3030
0.50 0.2471
0.60 0.2005
0.70 0.1003
0.80 0.0577
0.90 0.0343
1.00 0.0126
Average precision over all
relevant documents
non-interpolated 0.2585
Document Level Averages
Precision
At 5 docs 0.4566
At 10 docs 0.3906
At 15 docs 0.3384
At 20 docs 0.3198
At 30 docs 0.2541
At 100 docs 0.1279
At 200 docs 0.0783
At 500 docs 0.0394
At 1000 docs 0.0234
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2838
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
1KJE ij
Summary Statistics
Run ID 1KJE ij
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 52
Total number of documents over all topics
Retrieved: 52000
Relevant: 1878
Rel-ret: 926
Summary Statistics
Run ID 1KJE ij
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 52
Total number of documents over all topics
Retrieved: 52000
Relevant: 2309
Rel-ret: 1204
Recall Level Precision Averages
Recall Precision
0.00 0.6524
0.10 0.5415
0.20 0.4306
0.30 0.3285
0.40 0.2893
0.50 0.2306
0.60 0.1659
0.70 0.0983
0.80 0.0612
0.90 0.0353
1.00 0.0231
Average precision over all
relevant documents
non-interpolated 0.2407
Document Level Averages
Precision
At 5 docs 0.4038
At 10 docs 0.3154
At 15 docs 0.2923
At 20 docs 0.2538
At 30 docs 0.2032
At 100 docs 0.0990
At 200 docs 0.0612
At 500 docs 0.0311
At 1000 docs 0.0178
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2668
Recall Level Precision Averages
Recall Precision
0.00 0.7088
0.10 0.5469
0.20 0.4485
0.30 0.3547
0.40 0.3055
0.50 0.2351
0.60 0.1687
0.70 0.1123
0.80 0.0596
0.90 0.0369
1.00 0.0107
Average precision over all
relevant documents
non-interpolated 0.2507
Document Level Averages
Precision
At 5 docs 0.4615
At 10 docs 0.3731
At 15 docs 0.3487
At 20 docs 0.3038
At 30 docs 0.2468
At 100 docs 0.1254
At 200 docs 0.0792
At 500 docs 0.0404
At 1000 docs 0.0232
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2744
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
ANNO
Summary Statistics
Run ID ANNO
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 38396
Relevant: 1910
Rel-ret: 868
Summary Statistics
Run ID ANNO
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 38396
Relevant: 2345
Rel-ret: 1099
Recall Level Precision Averages
Recall Precision
0.00 0.4867
0.10 0.3901
0.20 0.2536
0.30 0.1939
0.40 0.1654
0.50 0.1442
0.60 0.1019
0.70 0.0652
0.80 0.0376
0.90 0.0194
1.00 0.0145
Average precision over all
relevant documents
non-interpolated 0.1473
Document Level Averages
Precision
At 5 docs 0.2491
At 10 docs 0.1868
At 15 docs 0.1648
At 20 docs 0.1415
At 30 docs 0.1245
At 100 docs 0.0798
At 200 docs 0.0526
At 500 docs 0.0284
At 1000 docs 0.0164
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.1607
Recall Level Precision Averages
Recall Precision
0.00 0.5242
0.10 0.4070
0.20 0.2585
0.30 0.2021
0.40 0.1779
0.50 0.1511
0.60 0.1055
0.70 0.0723
0.80 0.0508
0.90 0.0255
1.00 0.0127
Average precision over all
relevant documents
non-interpolated 0.1497
Document Level Averages
Precision
At 5 docs 0.2717
At 10 docs 0.2113
At 15 docs 0.1962
At 20 docs 0.1745
At 30 docs 0.1572
At 100 docs 0.1068
At 200 docs 0.0684
At 500 docs 0.0366
At 1000 docs 0.0207
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.1673
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
BKJJBIDS
Summary Statistics
Run ID BKJJBIDS
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1050
Summary Statistics
Run ID BKJJBIDS
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1342
Recall Level Precision Averages
Recall Precision
0.00 0.7247
0.10 0.5529
0.20 0.4261
0.30 0.3575
0.40 0.2936
0.50 0.2570
0.60 0.1922
0.70 0.1625
0.80 0.1204
0.90 0.0678
1.00 0.0556
Average precision over all
relevant documents
non-interpolated 0.2715
Document Level Averages
Precision
At 5 docs 0.4642
At 10 docs 0.3623
At 15 docs 0.3170
At 20 docs 0.2689
At 30 docs 0.2239
At 100 docs 0.1077
At 200 docs 0.0683
At 500 docs 0.0341
At 1000 docs 0.0198
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2817
Recall Level Precision Averages
Recall Precision
0.00 0.7751
0.10 0.5800
0.20 0.4623
0.30 0.3896
0.40 0.3207
0.50 0.2722
0.60 0.2150
0.70 0.1809
0.80 0.1372
0.90 0.0777
1.00 0.0541
Average precision over all
relevant documents
non-interpolated 0.2927
Document Level Averages
Precision
At 5 docs 0.5245
At 10 docs 0.4264
At 15 docs 0.3811
At 20 docs 0.3283
At 30 docs 0.2799
At 100 docs 0.1396
At 200 docs 0.0880
At 500 docs 0.0438
At 1000 docs 0.0253
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3147
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
BKJJBIFU
Summary Statistics
Run ID BKJJBIFU
Task adhoc
Relevant Level Relevant
Run Description automatic,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1232
Summary Statistics
Run ID BKJJBIFU
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1628
Recall Level Precision Averages
Recall Precision
0.00 0.7907
0.10 0.7341
0.20 0.6317
0.30 0.5328
0.40 0.4680
0.50 0.4189
0.60 0.3484
0.70 0.2732
0.80 0.1779
0.90 0.0933
1.00 0.0616
Average precision over all
relevant documents
non-interpolated 0.3966
Document Level Averages
Precision
At 5 docs 0.5736
At 10 docs 0.4755
At 15 docs 0.4214
At 20 docs 0.3830
At 30 docs 0.3233
At 100 docs 0.1477
At 200 docs 0.0881
At 500 docs 0.0411
At 1000 docs 0.0232
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.4160
Recall Level Precision Averages
Recall Precision
0.00 0.8848
0.10 0.8020
0.20 0.7020
0.30 0.5882
0.40 0.5323
0.50 0.4557
0.60 0.3653
0.70 0.2625
0.80 0.1990
0.90 0.1219
1.00 0.0552
Average precision over all
relevant documents
non-interpolated 0.4350
Document Level Averages
Precision
At 5 docs 0.6642
At 10 docs 0.5830
At 15 docs 0.5170
At 20 docs 0.4689
At 30 docs 0.4000
At 100 docs 0.1943
At 200 docs 0.1185
At 500 docs 0.0554
At 1000 docs 0.0307
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.4456
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long with concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long with concepts)
BKJJDCFU
Summary Statistics
Run ID BKJJDCFU
Task adhoc
Relevant Level Relevant
Run Description automatic,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1093
Summary Statistics
Run ID BKJJDCFU
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1462
Recall Level Precision Averages
Recall Precision
0.00 0.7478
0.10 0.6352
0.20 0.5373
0.30 0.4575
0.40 0.3924
0.50 0.3220
0.60 0.2656
0.70 0.2016
0.80 0.1274
0.90 0.0571
1.00 0.0441
Average precision over all
relevant documents
non-interpolated 0.3252
Document Level Averages
Precision
At 5 docs 0.5208
At 10 docs 0.4264
At 15 docs 0.3686
At 20 docs 0.3198
At 30 docs 0.2629
At 100 docs 0.1277
At 200 docs 0.0777
At 500 docs 0.0372
At 1000 docs 0.0206
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3656
Recall Level Precision Averages
Recall Precision
0.00 0.8325
0.10 0.7228
0.20 0.5817
0.30 0.4992
0.40 0.4119
0.50 0.3405
0.60 0.2843
0.70 0.2093
0.80 0.1401
0.90 0.0630
1.00 0.0414
Average precision over all
relevant documents
non-interpolated 0.3536
Document Level Averages
Precision
At 5 docs 0.6000
At 10 docs 0.5075
At 15 docs 0.4478
At 20 docs 0.4000
At 30 docs 0.3346
At 100 docs 0.1713
At 200 docs 0.1054
At 500 docs 0.0501
At 1000 docs 0.0276
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3936
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long with concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long with concepts)
CRL10
Summary Statistics
Run ID CRL10
Task adhoc
Relevant Level Relevant
Run Description interactive,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 52051
Relevant: 1910
Rel-ret: 1526
Summary Statistics
Run ID CRL10
Task adhoc
Relevant Level Partial Relevant
Run Description interactive,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 52051
Relevant: 2345
Rel-ret: 1936
Recall Level Precision Averages
Recall Precision
0.00 0.8178
0.10 0.6953
0.20 0.6060
0.30 0.5351
0.40 0.4579
0.50 0.4132
0.60 0.3415
0.70 0.2683
0.80 0.2000
0.90 0.1166
1.00 0.0614
Average precision over all
relevant documents
non-interpolated 0.3889
Document Level Averages
Precision
At 5 docs 0.5623
At 10 docs 0.4774
At 15 docs 0.4189
At 20 docs 0.3651
At 30 docs 0.3038
At 100 docs 0.1621
At 200 docs 0.1018
At 500 docs 0.0508
At 1000 docs 0.0288
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3956
Recall Level Precision Averages
Recall Precision
0.00 0.9048
0.10 0.7661
0.20 0.6796
0.30 0.6155
0.40 0.5063
0.50 0.4620
0.60 0.3814
0.70 0.3011
0.80 0.2254
0.90 0.1429
1.00 0.0434
Average precision over all
relevant documents
non-interpolated 0.4404
Document Level Averages
Precision
At 5 docs 0.6717
At 10 docs 0.5679
At 15 docs 0.5132
At 20 docs 0.4538
At 30 docs 0.3824
At 100 docs 0.2096
At 200 docs 0.1322
At 500 docs 0.0655
At 1000 docs 0.0365
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.4404
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (long with concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (long with concepts)
CRL12
Summary Statistics
Run ID CRL12
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1135
Summary Statistics
Run ID CRL12
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1473
Recall Level Precision Averages
Recall Precision
0.00 0.6129
0.10 0.5332
0.20 0.4030
0.30 0.3386
0.40 0.2806
0.50 0.2442
0.60 0.2037
0.70 0.1657
0.80 0.1296
0.90 0.0736
1.00 0.0495
Average precision over all
relevant documents
non-interpolated 0.2575
Document Level Averages
Precision
At 5 docs 0.4075
At 10 docs 0.3245
At 15 docs 0.2906
At 20 docs 0.2585
At 30 docs 0.2057
At 100 docs 0.1087
At 200 docs 0.0710
At 500 docs 0.0364
At 1000 docs 0.0214
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2699
Recall Level Precision Averages
Recall Precision
0.00 0.6678
0.10 0.5410
0.20 0.3994
0.30 0.3413
0.40 0.2910
0.50 0.2581
0.60 0.2189
0.70 0.1708
0.80 0.1339
0.90 0.0794
1.00 0.0279
Average precision over all
relevant documents
non-interpolated 0.2654
Document Level Averages
Precision
At 5 docs 0.4340
At 10 docs 0.3547
At 15 docs 0.3270
At 20 docs 0.2934
At 30 docs 0.2409
At 100 docs 0.1377
At 200 docs 0.0912
At 500 docs 0.0474
At 1000 docs 0.0278
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2753
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
CRL13
Summary Statistics
Run ID CRL13
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1146
Summary Statistics
Run ID CRL13
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1484
Recall Level Precision Averages
Recall Precision
0.00 0.6182
0.10 0.5355
0.20 0.4243
0.30 0.3452
0.40 0.2785
0.50 0.2471
0.60 0.1946
0.70 0.1600
0.80 0.1308
0.90 0.0694
1.00 0.0427
Average precision over all
relevant documents
non-interpolated 0.2584
Document Level Averages
Precision
At 5 docs 0.4113
At 10 docs 0.3340
At 15 docs 0.2843
At 20 docs 0.2557
At 30 docs 0.2107
At 100 docs 0.1102
At 200 docs 0.0708
At 500 docs 0.0368
At 1000 docs 0.0216
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2763
Recall Level Precision Averages
Recall Precision
0.00 0.6630
0.10 0.5360
0.20 0.4275
0.30 0.3508
0.40 0.2922
0.50 0.2618
0.60 0.2096
0.70 0.1667
0.80 0.1352
0.90 0.0819
1.00 0.0270
Average precision over all
relevant documents
non-interpolated 0.2675
Document Level Averages
Precision
At 5 docs 0.4453
At 10 docs 0.3774
At 15 docs 0.3245
At 20 docs 0.2925
At 30 docs 0.2484
At 100 docs 0.1392
At 200 docs 0.0912
At 500 docs 0.0479
At 1000 docs 0.0280
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2910
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
CRL14
Summary Statistics
Run ID CRL14
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1148
Summary Statistics
Run ID CRL14
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1503
Recall Level Precision Averages
Recall Precision
0.00 0.6213
0.10 0.5402
0.20 0.4409
0.30 0.3624
0.40 0.2874
0.50 0.2467
0.60 0.1965
0.70 0.1635
0.80 0.1348
0.90 0.0689
1.00 0.0397
Average precision over all
relevant documents
non-interpolated 0.2627
Document Level Averages
Precision
At 5 docs 0.4075
At 10 docs 0.3434
At 15 docs 0.2943
At 20 docs 0.2585
At 30 docs 0.2176
At 100 docs 0.1119
At 200 docs 0.0708
At 500 docs 0.0371
At 1000 docs 0.0217
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2872
Recall Level Precision Averages
Recall Precision
0.00 0.6664
0.10 0.5415
0.20 0.4439
0.30 0.3679
0.40 0.2991
0.50 0.2613
0.60 0.2127
0.70 0.1737
0.80 0.1413
0.90 0.0793
1.00 0.0288
Average precision over all
relevant documents
non-interpolated 0.2732
Document Level Averages
Precision
At 5 docs 0.4491
At 10 docs 0.3849
At 15 docs 0.3346
At 20 docs 0.2943
At 30 docs 0.2560
At 100 docs 0.1430
At 200 docs 0.0921
At 500 docs 0.0486
At 1000 docs 0.0284
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3040
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
CRL3
Summary Statistics
Run ID CRL3
Task adhoc
Relevant Level Relevant
Run Description automatic,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1295
Summary Statistics
Run ID CRL3
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1677
Recall Level Precision Averages
Recall Precision
0.00 0.7192
0.10 0.6432
0.20 0.5475
0.30 0.4679
0.40 0.3802
0.50 0.3299
0.60 0.2706
0.70 0.2116
0.80 0.1631
0.90 0.0743
1.00 0.0463
Average precision over all
relevant documents
non-interpolated 0.3337
Document Level Averages
Precision
At 5 docs 0.4981
At 10 docs 0.4170
At 15 docs 0.3597
At 20 docs 0.3189
At 30 docs 0.2755
At 100 docs 0.1372
At 200 docs 0.0851
At 500 docs 0.0428
At 1000 docs 0.0244
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3310
Recall Level Precision Averages
Recall Precision
0.00 0.8151
0.10 0.6890
0.20 0.5878
0.30 0.4936
0.40 0.4191
0.50 0.3445
0.60 0.2875
0.70 0.2100
0.80 0.1623
0.90 0.0896
1.00 0.0431
Average precision over all
relevant documents
non-interpolated 0.3575
Document Level Averages
Precision
At 5 docs 0.5623
At 10 docs 0.4849
At 15 docs 0.4239
At 20 docs 0.3802
At 30 docs 0.3321
At 100 docs 0.1726
At 200 docs 0.1095
At 500 docs 0.0557
At 1000 docs 0.0316
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3665
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long with concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long with concepts)
CRL4
Summary Statistics
Run ID CRL4
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 51779
Relevant: 1910
Rel-ret: 1186
Summary Statistics
Run ID CRL4
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 51779
Relevant: 2345
Rel-ret: 1498
Recall Level Precision Averages
Recall Precision
0.00 0.5768
0.10 0.4856
0.20 0.3876
0.30 0.3275
0.40 0.2842
0.50 0.2354
0.60 0.1883
0.70 0.1471
0.80 0.1158
0.90 0.0724
1.00 0.0579
Average precision over all
relevant documents
non-interpolated 0.2404
Document Level Averages
Precision
At 5 docs 0.3660
At 10 docs 0.3019
At 15 docs 0.2616
At 20 docs 0.2406
At 30 docs 0.2025
At 100 docs 0.1062
At 200 docs 0.0680
At 500 docs 0.0368
At 1000 docs 0.0224
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2490
Recall Level Precision Averages
Recall Precision
0.00 0.6314
0.10 0.4983
0.20 0.4144
0.30 0.3460
0.40 0.2800
0.50 0.2468
0.60 0.2034
0.70 0.1576
0.80 0.1176
0.90 0.0743
1.00 0.0388
Average precision over all
relevant documents
non-interpolated 0.2492
Document Level Averages
Precision
At 5 docs 0.4075
At 10 docs 0.3396
At 15 docs 0.3044
At 20 docs 0.2792
At 30 docs 0.2371
At 100 docs 0.1315
At 200 docs 0.0849
At 500 docs 0.0468
At 1000 docs 0.0283
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2602
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
CRL5
Summary Statistics
Run ID CRL5
Task adhoc
Relevant Level Relevant
Run Description interactive,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 50665
Relevant: 1910
Rel-ret: 1582
Summary Statistics
Run ID CRL5
Task adhoc
Relevant Level Partial Relevant
Run Description interactive,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 50665
Relevant: 2345
Rel-ret: 1984
Recall Level Precision Averages
Recall Precision
0.00 0.7723
0.10 0.6562
0.20 0.5612
0.30 0.4955
0.40 0.4046
0.50 0.3573
0.60 0.2949
0.70 0.2483
0.80 0.1561
0.90 0.0957
1.00 0.0670
Average precision over all
relevant documents
non-interpolated 0.3479
Document Level Averages
Precision
At 5 docs 0.5019
At 10 docs 0.4283
At 15 docs 0.3723
At 20 docs 0.3302
At 30 docs 0.2792
At 100 docs 0.1500
At 200 docs 0.0992
At 500 docs 0.0491
At 1000 docs 0.0298
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3775
Recall Level Precision Averages
Recall Precision
0.00 0.8503
0.10 0.7413
0.20 0.6359
0.30 0.5581
0.40 0.4497
0.50 0.4078
0.60 0.3429
0.70 0.2741
0.80 0.1864
0.90 0.1145
1.00 0.0555
Average precision over all
relevant documents
non-interpolated 0.3922
Document Level Averages
Precision
At 5 docs 0.6151
At 10 docs 0.5170
At 15 docs 0.4579
At 20 docs 0.4132
At 30 docs 0.3547
At 100 docs 0.1923
At 200 docs 0.1280
At 500 docs 0.0633
At 1000 docs 0.0374
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.4160
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (long with concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (long with concepts)
CRL6
Summary Statistics
Run ID CRL6
Task adhoc
Relevant Level Relevant
Run Description automatic,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1334
Summary Statistics
Run ID CRL6
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1734
Recall Level Precision Averages
Recall Precision
0.00 0.7491
0.10 0.6729
0.20 0.5889
0.30 0.5054
0.40 0.4367
0.50 0.3911
0.60 0.3238
0.70 0.2512
0.80 0.1967
0.90 0.0921
1.00 0.0602
Average precision over all
relevant documents
non-interpolated 0.3716
Document Level Averages
Precision
At 5 docs 0.5358
At 10 docs 0.4736
At 15 docs 0.3962
At 20 docs 0.3575
At 30 docs 0.2981
At 100 docs 0.1509
At 200 docs 0.0938
At 500 docs 0.0451
At 1000 docs 0.0252
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3815
Recall Level Precision Averages
Recall Precision
0.00 0.8310
0.10 0.7323
0.20 0.6512
0.30 0.5505
0.40 0.4894
0.50 0.4159
0.60 0.3538
0.70 0.2729
0.80 0.2150
0.90 0.1080
1.00 0.0576
Average precision over all
relevant documents
non-interpolated 0.4075
Document Level Averages
Precision
At 5 docs 0.6264
At 10 docs 0.5623
At 15 docs 0.4855
At 20 docs 0.4358
At 30 docs 0.3616
At 100 docs 0.1957
At 200 docs 0.1240
At 500 docs 0.0596
At 1000 docs 0.0327
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.4122
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long with concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long with concepts)
CRL7
Summary Statistics
Run ID CRL7
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 50601
Relevant: 1910
Rel-ret: 1113
Summary Statistics
Run ID CRL7
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 50601
Relevant: 2345
Rel-ret: 1425
Recall Level Precision Averages
Recall Precision
0.00 0.5854
0.10 0.4734
0.20 0.3729
0.30 0.3235
0.40 0.2725
0.50 0.2211
0.60 0.1750
0.70 0.1380
0.80 0.1051
0.90 0.0696
1.00 0.0579
Average precision over all
relevant documents
non-interpolated 0.2290
Document Level Averages
Precision
At 5 docs 0.3774
At 10 docs 0.2925
At 15 docs 0.2667
At 20 docs 0.2340
At 30 docs 0.2013
At 100 docs 0.1002
At 200 docs 0.0632
At 500 docs 0.0344
At 1000 docs 0.0210
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2419
Recall Level Precision Averages
Recall Precision
0.00 0.6432
0.10 0.4947
0.20 0.3926
0.30 0.3333
0.40 0.2800
0.50 0.2286
0.60 0.1956
0.70 0.1440
0.80 0.1128
0.90 0.0711
1.00 0.0381
Average precision over all
relevant documents
non-interpolated 0.2402
Document Level Averages
Precision
At 5 docs 0.4151
At 10 docs 0.3340
At 15 docs 0.3031
At 20 docs 0.2689
At 30 docs 0.2346
At 100 docs 0.1251
At 200 docs 0.0799
At 500 docs 0.0445
At 1000 docs 0.0269
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2481
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
CRL8
Summary Statistics
Run ID CRL8
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 40899
Relevant: 1910
Rel-ret: 1115
Summary Statistics
Run ID CRL8
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 40899
Relevant: 2345
Rel-ret: 1438
Recall Level Precision Averages
Recall Precision
0.00 0.6129
0.10 0.5332
0.20 0.4033
0.30 0.3390
0.40 0.2797
0.50 0.2435
0.60 0.2035
0.70 0.1654
0.80 0.1293
0.90 0.0723
1.00 0.0472
Average precision over all
relevant documents
non-interpolated 0.2571
Document Level Averages
Precision
At 5 docs 0.4075
At 10 docs 0.3245
At 15 docs 0.2906
At 20 docs 0.2585
At 30 docs 0.2063
At 100 docs 0.1094
At 200 docs 0.0708
At 500 docs 0.0361
At 1000 docs 0.0210
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2707
Recall Level Precision Averages
Recall Precision
0.00 0.6677
0.10 0.5409
0.20 0.3997
0.30 0.3416
0.40 0.2904
0.50 0.2570
0.60 0.2165
0.70 0.1688
0.80 0.1313
0.90 0.0774
1.00 0.0281
Average precision over all
relevant documents
non-interpolated 0.2646
Document Level Averages
Precision
At 5 docs 0.4340
At 10 docs 0.3547
At 15 docs 0.3270
At 20 docs 0.2934
At 30 docs 0.2415
At 100 docs 0.1387
At 200 docs 0.0910
At 500 docs 0.0469
At 1000 docs 0.0271
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2761
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
CRL9
Summary Statistics
Run ID CRL9
Task adhoc
Relevant Level Relevant
Run Description automatic,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1340
Summary Statistics
Run ID CRL9
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1747
Recall Level Precision Averages
Recall Precision
0.00 0.7928
0.10 0.7013
0.20 0.5936
0.30 0.5324
0.40 0.4619
0.50 0.4003
0.60 0.3389
0.70 0.2714
0.80 0.2163
0.90 0.1317
1.00 0.0753
Average precision over all
relevant documents
non-interpolated 0.3917
Document Level Averages
Precision
At 5 docs 0.5660
At 10 docs 0.4717
At 15 docs 0.4138
At 20 docs 0.3679
At 30 docs 0.3107
At 100 docs 0.1604
At 200 docs 0.0980
At 500 docs 0.0468
At 1000 docs 0.0253
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3905
Recall Level Precision Averages
Recall Precision
0.00 0.8750
0.10 0.7487
0.20 0.6681
0.30 0.5825
0.40 0.5278
0.50 0.4432
0.60 0.3753
0.70 0.2907
0.80 0.2370
0.90 0.1422
1.00 0.0671
Average precision over all
relevant documents
non-interpolated 0.4323
Document Level Averages
Precision
At 5 docs 0.6415
At 10 docs 0.5623
At 15 docs 0.4956
At 20 docs 0.4434
At 30 docs 0.3862
At 100 docs 0.2089
At 200 docs 0.1284
At 500 docs 0.0611
At 1000 docs 0.0330
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.4354
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long with concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long with concepts)
DOVE1
Summary Statistics
Run ID DOVE1
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 977
Summary Statistics
Run ID DOVE1
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1300
Recall Level Precision Averages
Recall Precision
0.00 0.3658
0.10 0.2613
0.20 0.1934
0.30 0.1698
0.40 0.1316
0.50 0.1207
0.60 0.0761
0.70 0.0677
0.80 0.0485
0.90 0.0365
1.00 0.0117
Average precision over all
relevant documents
non-interpolated 0.1170
Document Level Averages
Precision
At 5 docs 0.1736
At 10 docs 0.1340
At 15 docs 0.1182
At 20 docs 0.1179
At 30 docs 0.1013
At 100 docs 0.0625
At 200 docs 0.0491
At 500 docs 0.0295
At 1000 docs 0.0184
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.1334
Recall Level Precision Averages
Recall Precision
0.00 0.4079
0.10 0.2786
0.20 0.2210
0.30 0.1790
0.40 0.1482
0.50 0.1299
0.60 0.0907
0.70 0.0726
0.80 0.0564
0.90 0.0422
1.00 0.0090
Average precision over all
relevant documents
non-interpolated 0.1272
Document Level Averages
Precision
At 5 docs 0.2038
At 10 docs 0.1642
At 15 docs 0.1447
At 20 docs 0.1434
At 30 docs 0.1302
At 100 docs 0.0868
At 200 docs 0.0685
At 500 docs 0.0400
At 1000 docs 0.0245
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.1438
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
DOVE2
Summary Statistics
Run ID DOVE2
Task adhoc
Relevant Level Relevant
Run Description interactive,long without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1194
Summary Statistics
Run ID DOVE2
Task adhoc
Relevant Level Partial Relevant
Run Description interactive,long without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1539
Recall Level Precision Averages
Recall Precision
0.00 0.7189
0.10 0.5388
0.20 0.3793
0.30 0.3202
0.40 0.2713
0.50 0.2192
0.60 0.1717
0.70 0.1183
0.80 0.0910
0.90 0.0420
1.00 0.0167
Average precision over all
relevant documents
non-interpolated 0.2396
Document Level Averages
Precision
At 5 docs 0.4075
At 10 docs 0.2811
At 15 docs 0.2327
At 20 docs 0.2075
At 30 docs 0.1723
At 100 docs 0.1030
At 200 docs 0.0670
At 500 docs 0.0374
At 1000 docs 0.0225
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2491
Recall Level Precision Averages
Recall Precision
0.00 0.7828
0.10 0.5362
0.20 0.4019
0.30 0.3371
0.40 0.2773
0.50 0.2382
0.60 0.1761
0.70 0.1296
0.80 0.0911
0.90 0.0508
1.00 0.0161
Average precision over all
relevant documents
non-interpolated 0.2484
Document Level Averages
Precision
At 5 docs 0.4642
At 10 docs 0.3453
At 15 docs 0.2830
At 20 docs 0.2538
At 30 docs 0.2119
At 100 docs 0.1315
At 200 docs 0.0880
At 500 docs 0.0483
At 1000 docs 0.0290
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2611
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (long without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (long without concepts)
DOVE3
Summary Statistics
Run ID DOVE3
Task adhoc
Relevant Level Relevant
Run Description interactive,long without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1173
Summary Statistics
Run ID DOVE3
Task adhoc
Relevant Level Partial Relevant
Run Description interactive,long without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1495
Recall Level Precision Averages
Recall Precision
0.00 0.6033
0.10 0.5180
0.20 0.4450
0.30 0.3508
0.40 0.2808
0.50 0.2171
0.60 0.1703
0.70 0.1054
0.80 0.0725
0.90 0.0281
1.00 0.0108
Average precision over all
relevant documents
non-interpolated 0.2317
Document Level Averages
Precision
At 5 docs 0.3736
At 10 docs 0.3132
At 15 docs 0.2591
At 20 docs 0.2368
At 30 docs 0.1956
At 100 docs 0.1017
At 200 docs 0.0704
At 500 docs 0.0375
At 1000 docs 0.0221
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2727
Recall Level Precision Averages
Recall Precision
0.00 0.6777
0.10 0.5723
0.20 0.4909
0.30 0.3684
0.40 0.2999
0.50 0.2407
0.60 0.1838
0.70 0.1224
0.80 0.0879
0.90 0.0383
1.00 0.0144
Average precision over all
relevant documents
non-interpolated 0.2556
Document Level Averages
Precision
At 5 docs 0.4528
At 10 docs 0.3830
At 15 docs 0.3195
At 20 docs 0.2962
At 30 docs 0.2447
At 100 docs 0.1298
At 200 docs 0.0924
At 500 docs 0.0489
At 1000 docs 0.0282
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2951
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (long without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (long without concepts)
JALAB
Summary Statistics
Run ID JALAB
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 52051
Relevant: 1910
Rel-ret: 84
Summary Statistics
Run ID JALAB
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 52051
Relevant: 2345
Rel-ret: 164
Recall Level Precision Averages
Recall Precision
0.00 0.0075
0.10 0.0064
0.20 0.0049
0.30 0.0025
0.40 0.0014
0.50 0.0006
0.60 0.0002
0.70 0.0000
0.80 0.0000
0.90 0.0000
1.00 0.0000
Average precision over all
relevant documents
non-interpolated 0.0017
Document Level Averages
Precision
At 5 docs 0.0000
At 10 docs 0.0000
At 15 docs 0.0000
At 20 docs 0.0009
At 30 docs 0.0025
At 100 docs 0.0034
At 200 docs 0.0029
At 500 docs 0.0022
At 1000 docs 0.0016
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.0003
Recall Level Precision Averages
Recall Precision
0.00 0.0194
0.10 0.0108
0.20 0.0053
0.30 0.0034
0.40 0.0025
0.50 0.0006
0.60 0.0002
0.70 0.0000
0.80 0.0000
0.90 0.0000
1.00 0.0000
Average precision over all
relevant documents
non-interpolated 0.0028
Document Level Averages
Precision
At 5 docs 0.0075
At 10 docs 0.0075
At 15 docs 0.0050
At 20 docs 0.0057
At 30 docs 0.0069
At 100 docs 0.0072
At 200 docs 0.0062
At 500 docs 0.0042
At 1000 docs 0.0031
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.0066
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
K32001
Summary Statistics
Run ID K32001
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1198
Summary Statistics
Run ID K32001
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1559
Recall Level Precision Averages
Recall Precision
0.00 0.7148
0.10 0.5550
0.20 0.4739
0.30 0.4313
0.40 0.3920
0.50 0.3051
0.60 0.2518
0.70 0.1995
0.80 0.1584
0.90 0.0958
1.00 0.0748
Average precision over all
relevant documents
non-interpolated 0.3159
Document Level Averages
Precision
At 5 docs 0.4566
At 10 docs 0.3811
At 15 docs 0.3296
At 20 docs 0.2981
At 30 docs 0.2434
At 100 docs 0.1262
At 200 docs 0.0798
At 500 docs 0.0392
At 1000 docs 0.0226
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3317
Recall Level Precision Averages
Recall Precision
0.00 0.7837
0.10 0.6284
0.20 0.4996
0.30 0.4651
0.40 0.3968
0.50 0.3383
0.60 0.2890
0.70 0.2267
0.80 0.1764
0.90 0.1111
1.00 0.0721
Average precision over all
relevant documents
non-interpolated 0.3454
Document Level Averages
Precision
At 5 docs 0.5434
At 10 docs 0.4623
At 15 docs 0.4038
At 20 docs 0.3717
At 30 docs 0.3126
At 100 docs 0.1666
At 200 docs 0.1050
At 500 docs 0.0512
At 1000 docs 0.0294
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3573
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
K32002
Summary Statistics
Run ID K32002
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1219
Summary Statistics
Run ID K32002
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1588
Recall Level Precision Averages
Recall Precision
0.00 0.7031
0.10 0.5983
0.20 0.4872
0.30 0.4451
0.40 0.3747
0.50 0.3158
0.60 0.2537
0.70 0.2122
0.80 0.1598
0.90 0.0948
1.00 0.0630
Average precision over all
relevant documents
non-interpolated 0.3181
Document Level Averages
Precision
At 5 docs 0.4717
At 10 docs 0.3981
At 15 docs 0.3484
At 20 docs 0.3142
At 30 docs 0.2535
At 100 docs 0.1298
At 200 docs 0.0808
At 500 docs 0.0399
At 1000 docs 0.0230
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3273
Recall Level Precision Averages
Recall Precision
0.00 0.7650
0.10 0.6441
0.20 0.5140
0.30 0.4763
0.40 0.4077
0.50 0.3501
0.60 0.2875
0.70 0.2360
0.80 0.1825
0.90 0.1147
1.00 0.0628
Average precision over all
relevant documents
non-interpolated 0.3495
Document Level Averages
Precision
At 5 docs 0.5585
At 10 docs 0.4792
At 15 docs 0.4189
At 20 docs 0.3877
At 30 docs 0.3252
At 100 docs 0.1711
At 200 docs 0.1059
At 500 docs 0.0522
At 1000 docs 0.0300
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3657
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
KCTRG1
Summary Statistics
Run ID KCTRG1
Task adhoc
Relevant Level Relevant
Run Description automatic,long without cocepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 989
Summary Statistics
Run ID KCTRG1
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,long without cocepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1270
Recall Level Precision Averages
Recall Precision
0.00 0.5638
0.10 0.3765
0.20 0.2761
0.30 0.2169
0.40 0.1593
0.50 0.1316
0.60 0.0848
0.70 0.0531
0.80 0.0328
0.90 0.0115
1.00 0.0089
Average precision over all
relevant documents
non-interpolated 0.1544
Document Level Averages
Precision
At 5 docs 0.2604
At 10 docs 0.2208
At 15 docs 0.2063
At 20 docs 0.1906
At 30 docs 0.1560
At 100 docs 0.0826
At 200 docs 0.0549
At 500 docs 0.0304
At 1000 docs 0.0187
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.1851
Recall Level Precision Averages
Recall Precision
0.00 0.6397
0.10 0.3988
0.20 0.3070
0.30 0.2517
0.40 0.1938
0.50 0.1638
0.60 0.1096
0.70 0.0713
0.80 0.0513
0.90 0.0203
1.00 0.0066
Average precision over all
relevant documents
non-interpolated 0.1768
Document Level Averages
Precision
At 5 docs 0.3321
At 10 docs 0.2925
At 15 docs 0.2704
At 20 docs 0.2443
At 30 docs 0.2069
At 100 docs 0.1104
At 200 docs 0.0725
At 500 docs 0.0395
At 1000 docs 0.0240
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2142
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long without concepts)
KCTRG2
Summary Statistics
Run ID KCTRG2
Task adhoc
Relevant Level Relevant
Run Description automatic,long without cocepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1088
Summary Statistics
Run ID KCTRG2
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,long without cocepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1443
Recall Level Precision Averages
Recall Precision
0.00 0.6904
0.10 0.5378
0.20 0.4025
0.30 0.3073
0.40 0.2289
0.50 0.1874
0.60 0.1438
0.70 0.1010
0.80 0.0666
0.90 0.0366
1.00 0.0208
Average precision over all
relevant documents
non-interpolated 0.2280
Document Level Averages
Precision
At 5 docs 0.3887
At 10 docs 0.3226
At 15 docs 0.2805
At 20 docs 0.2500
At 30 docs 0.2031
At 100 docs 0.1074
At 200 docs 0.0698
At 500 docs 0.0355
At 1000 docs 0.0205
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2452
Recall Level Precision Averages
Recall Precision
0.00 0.7796
0.10 0.5776
0.20 0.4510
0.30 0.3532
0.40 0.2771
0.50 0.2262
0.60 0.1770
0.70 0.1193
0.80 0.0754
0.90 0.0401
1.00 0.0172
Average precision over all
relevant documents
non-interpolated 0.2584
Document Level Averages
Precision
At 5 docs 0.4679
At 10 docs 0.3981
At 15 docs 0.3484
At 20 docs 0.3160
At 30 docs 0.2654
At 100 docs 0.1460
At 200 docs 0.0939
At 500 docs 0.0475
At 1000 docs 0.0272
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2824
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long without concepts)
KLAB1.dat
Summary Statistics
Run ID KLAB1.dat
Task adhoc
Relevant Level Relevant
Run Description automatic,short with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 33830
Relevant: 1910
Rel-ret: 747
Summary Statistics
Run ID KLAB1.dat
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 33830
Relevant: 2345
Rel-ret: 982
Recall Level Precision Averages
Recall Precision
0.00 0.1619
0.10 0.1267
0.20 0.1120
0.30 0.1078
0.40 0.0848
0.50 0.0536
0.60 0.0472
0.70 0.0400
0.80 0.0300
0.90 0.0205
1.00 0.0151
Average precision over all
relevant documents
non-interpolated 0.0717
Document Level Averages
Precision
At 5 docs 0.0566
At 10 docs 0.0547
At 15 docs 0.0528
At 20 docs 0.0509
At 30 docs 0.0497
At 100 docs 0.0402
At 200 docs 0.0293
At 500 docs 0.0203
At 1000 docs 0.0141
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.0612
Recall Level Precision Averages
Recall Precision
0.00 0.1917
0.10 0.1500
0.20 0.1350
0.30 0.1252
0.40 0.0923
0.50 0.0580
0.60 0.0504
0.70 0.0456
0.80 0.0287
0.90 0.0264
1.00 0.0258
Average precision over all
relevant documents
non-interpolated 0.0808
Document Level Averages
Precision
At 5 docs 0.0642
At 10 docs 0.0736
At 15 docs 0.0755
At 20 docs 0.0698
At 30 docs 0.0660
At 100 docs 0.0528
At 200 docs 0.0395
At 500 docs 0.0274
At 1000 docs 0.0185
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.0738
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short with concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short with concepts)
KLAB2.dat
Summary Statistics
Run ID KLAB2.dat
Task adhoc
Relevant Level Relevant
Run Description automatic,short with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 41979
Relevant: 1910
Rel-ret: 475
Summary Statistics
Run ID KLAB2.dat
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 41979
Relevant: 2345
Rel-ret: 603
Recall Level Precision Averages
Recall Precision
0.00 0.0525
0.10 0.0507
0.20 0.0482
0.30 0.0455
0.40 0.0314
0.50 0.0287
0.60 0.0245
0.70 0.0182
0.80 0.0097
0.90 0.0087
1.00 0.0087
Average precision over all
relevant documents
non-interpolated 0.0327
Document Level Averages
Precision
At 5 docs 0.0038
At 10 docs 0.0132
At 15 docs 0.0201
At 20 docs 0.0255
At 30 docs 0.0226
At 100 docs 0.0194
At 200 docs 0.0165
At 500 docs 0.0129
At 1000 docs 0.0090
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.0220
Recall Level Precision Averages
Recall Precision
0.00 0.0721
0.10 0.0655
0.20 0.0614
0.30 0.0535
0.40 0.0449
0.50 0.0331
0.60 0.0289
0.70 0.0253
0.80 0.0112
0.90 0.0112
1.00 0.0112
Average precision over all
relevant documents
non-interpolated 0.0381
Document Level Averages
Precision
At 5 docs 0.0038
At 10 docs 0.0226
At 15 docs 0.0314
At 20 docs 0.0377
At 30 docs 0.0333
At 100 docs 0.0251
At 200 docs 0.0209
At 500 docs 0.0168
At 1000 docs 0.0114
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.0318
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short with concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short with concepts)
NTE151
Summary Statistics
Run ID NTE151
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 51221
Relevant: 1910
Rel-ret: 966
Summary Statistics
Run ID NTE151
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 51221
Relevant: 2345
Rel-ret: 1203
Recall Level Precision Averages
Recall Precision
0.00 0.6472
0.10 0.5460
0.20 0.4249
0.30 0.3654
0.40 0.2958
0.50 0.2443
0.60 0.1856
0.70 0.1426
0.80 0.1066
0.90 0.0523
1.00 0.0391
Average precision over all
relevant documents
non-interpolated 0.2560
Document Level Averages
Precision
At 5 docs 0.4189
At 10 docs 0.3358
At 15 docs 0.2818
At 20 docs 0.2481
At 30 docs 0.1981
At 100 docs 0.1008
At 200 docs 0.0636
At 500 docs 0.0318
At 1000 docs 0.0182
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2725
Recall Level Precision Averages
Recall Precision
0.00 0.6910
0.10 0.5538
0.20 0.4208
0.30 0.3497
0.40 0.2920
0.50 0.2465
0.60 0.1875
0.70 0.1416
0.80 0.0957
0.90 0.0489
1.00 0.0289
Average precision over all
relevant documents
non-interpolated 0.2586
Document Level Averages
Precision
At 5 docs 0.4717
At 10 docs 0.3811
At 15 docs 0.3270
At 20 docs 0.2906
At 30 docs 0.2403
At 100 docs 0.1253
At 200 docs 0.0804
At 500 docs 0.0402
At 1000 docs 0.0227
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2779
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
NTE152
Summary Statistics
Run ID NTE152
Task adhoc
Relevant Level Relevant
Run Description automatic,short with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1135
Summary Statistics
Run ID NTE152
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1443
Recall Level Precision Averages
Recall Precision
0.00 0.7496
0.10 0.6695
0.20 0.5649
0.30 0.4729
0.40 0.3826
0.50 0.3243
0.60 0.2533
0.70 0.1829
0.80 0.1393
0.90 0.0846
1.00 0.0684
Average precision over all
relevant documents
non-interpolated 0.3316
Document Level Averages
Precision
At 5 docs 0.5132
At 10 docs 0.4151
At 15 docs 0.3522
At 20 docs 0.3038
At 30 docs 0.2497
At 100 docs 0.1196
At 200 docs 0.0743
At 500 docs 0.0371
At 1000 docs 0.0214
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3358
Recall Level Precision Averages
Recall Precision
0.00 0.8080
0.10 0.7049
0.20 0.5862
0.30 0.4810
0.40 0.3962
0.50 0.3321
0.60 0.2672
0.70 0.1955
0.80 0.1384
0.90 0.0788
1.00 0.0534
Average precision over all
relevant documents
non-interpolated 0.3490
Document Level Averages
Precision
At 5 docs 0.5925
At 10 docs 0.4830
At 15 docs 0.4176
At 20 docs 0.3698
At 30 docs 0.3050
At 100 docs 0.1500
At 200 docs 0.0947
At 500 docs 0.0474
At 1000 docs 0.0272
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3485
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short with concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short with concepts)
R2D21
Summary Statistics
Run ID R2D21
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1127
Summary Statistics
Run ID R2D21
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1427
Recall Level Precision Averages
Recall Precision
0.00 0.7033
0.10 0.6064
0.20 0.4943
0.30 0.4114
0.40 0.3415
0.50 0.2914
0.60 0.2329
0.70 0.1688
0.80 0.1342
0.90 0.0455
1.00 0.0352
Average precision over all
relevant documents
non-interpolated 0.2933
Document Level Averages
Precision
At 5 docs 0.4830
At 10 docs 0.3925
At 15 docs 0.3346
At 20 docs 0.2953
At 30 docs 0.2421
At 100 docs 0.1215
At 200 docs 0.0743
At 500 docs 0.0368
At 1000 docs 0.0213
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3017
Recall Level Precision Averages
Recall Precision
0.00 0.7772
0.10 0.6205
0.20 0.5137
0.30 0.4104
0.40 0.3494
0.50 0.2938
0.60 0.2457
0.70 0.1922
0.80 0.1509
0.90 0.0600
1.00 0.0358
Average precision over all
relevant documents
non-interpolated 0.3097
Document Level Averages
Precision
At 5 docs 0.5358
At 10 docs 0.4528
At 15 docs 0.3925
At 20 docs 0.3528
At 30 docs 0.2975
At 100 docs 0.1511
At 200 docs 0.0928
At 500 docs 0.0463
At 1000 docs 0.0269
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3199
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
R2D22
Summary Statistics
Run ID R2D22
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1278
Summary Statistics
Run ID R2D22
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1600
Recall Level Precision Averages
Recall Precision
0.00 0.7181
0.10 0.6057
0.20 0.5370
0.30 0.4631
0.40 0.3865
0.50 0.3295
0.60 0.2524
0.70 0.2063
0.80 0.1692
0.90 0.0791
1.00 0.0550
Average precision over all
relevant documents
non-interpolated 0.3289
Document Level Averages
Precision
At 5 docs 0.4906
At 10 docs 0.4094
At 15 docs 0.3547
At 20 docs 0.3113
At 30 docs 0.2629
At 100 docs 0.1370
At 200 docs 0.0852
At 500 docs 0.0422
At 1000 docs 0.0241
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3323
Recall Level Precision Averages
Recall Precision
0.00 0.7802
0.10 0.6252
0.20 0.5554
0.30 0.4797
0.40 0.3906
0.50 0.3444
0.60 0.2715
0.70 0.2353
0.80 0.1820
0.90 0.0933
1.00 0.0503
Average precision over all
relevant documents
non-interpolated 0.3473
Document Level Averages
Precision
At 5 docs 0.5509
At 10 docs 0.4698
At 15 docs 0.4201
At 20 docs 0.3802
At 30 docs 0.3270
At 100 docs 0.1732
At 200 docs 0.1072
At 500 docs 0.0532
At 1000 docs 0.0302
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3555
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
R2D23
Summary Statistics
Run ID R2D23
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1039
Summary Statistics
Run ID R2D23
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1318
Recall Level Precision Averages
Recall Precision
0.00 0.6958
0.10 0.5858
0.20 0.4735
0.30 0.3957
0.40 0.3234
0.50 0.2776
0.60 0.2177
0.70 0.1493
0.80 0.1171
0.90 0.0443
1.00 0.0336
Average precision over all
relevant documents
non-interpolated 0.2784
Document Level Averages
Precision
At 5 docs 0.4755
At 10 docs 0.3868
At 15 docs 0.3220
At 20 docs 0.2840
At 30 docs 0.2302
At 100 docs 0.1085
At 200 docs 0.0701
At 500 docs 0.0331
At 1000 docs 0.0196
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3010
Recall Level Precision Averages
Recall Precision
0.00 0.7521
0.10 0.5835
0.20 0.4974
0.30 0.3958
0.40 0.3176
0.50 0.2644
0.60 0.2053
0.70 0.1638
0.80 0.1277
0.90 0.0577
1.00 0.0281
Average precision over all
relevant documents
non-interpolated 0.2862
Document Level Averages
Precision
At 5 docs 0.5245
At 10 docs 0.4340
At 15 docs 0.3736
At 20 docs 0.3387
At 30 docs 0.2836
At 100 docs 0.1358
At 200 docs 0.0872
At 500 docs 0.0419
At 1000 docs 0.0249
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3139
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
R2D24
Summary Statistics
Run ID R2D24
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1184
Summary Statistics
Run ID R2D24
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1492
Recall Level Precision Averages
Recall Precision
0.00 0.7022
0.10 0.5985
0.20 0.5294
0.30 0.4294
0.40 0.3542
0.50 0.3014
0.60 0.2375
0.70 0.1899
0.80 0.1540
0.90 0.0598
1.00 0.0452
Average precision over all
relevant documents
non-interpolated 0.3103
Document Level Averages
Precision
At 5 docs 0.4868
At 10 docs 0.4094
At 15 docs 0.3447
At 20 docs 0.2925
At 30 docs 0.2434
At 100 docs 0.1292
At 200 docs 0.0785
At 500 docs 0.0389
At 1000 docs 0.0223
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3225
Recall Level Precision Averages
Recall Precision
0.00 0.7468
0.10 0.6147
0.20 0.5416
0.30 0.4304
0.40 0.3540
0.50 0.3018
0.60 0.2541
0.70 0.2145
0.80 0.1605
0.90 0.0740
1.00 0.0400
Average precision over all
relevant documents
non-interpolated 0.3219
Document Level Averages
Precision
At 5 docs 0.5245
At 10 docs 0.4566
At 15 docs 0.4038
At 20 docs 0.3566
At 30 docs 0.3025
At 100 docs 0.1626
At 200 docs 0.0990
At 500 docs 0.0494
At 1000 docs 0.0282
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3328
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
STIX1
Summary Statistics
Run ID STIX1
Task adhoc
Relevant Level Relevant
Run Description interactive,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 52404
Relevant: 1910
Rel-ret: 350
Summary Statistics
Run ID STIX1
Task adhoc
Relevant Level Partial Relevant
Run Description interactive,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 52404
Relevant: 2345
Rel-ret: 445
Recall Level Precision Averages
Recall Precision
0.00 0.3218
0.10 0.1968
0.20 0.0967
0.30 0.0467
0.40 0.0087
0.50 0.0020
0.60 0.0006
0.70 0.0002
0.80 0.0002
0.90 0.0000
1.00 0.0000
Average precision over all
relevant documents
non-interpolated 0.0460
Document Level Averages
Precision
At 5 docs 0.1434
At 10 docs 0.0981
At 15 docs 0.0742
At 20 docs 0.0623
At 30 docs 0.0465
At 100 docs 0.0249
At 200 docs 0.0155
At 500 docs 0.0096
At 1000 docs 0.0066
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.0723
Recall Level Precision Averages
Recall Precision
0.00 0.3713
0.10 0.2069
0.20 0.0842
0.30 0.0383
0.40 0.0082
0.50 0.0020
0.60 0.0003
0.70 0.0003
0.80 0.0003
0.90 0.0002
1.00 0.0000
Average precision over all
relevant documents
non-interpolated 0.0438
Document Level Averages
Precision
At 5 docs 0.1774
At 10 docs 0.1170
At 15 docs 0.0868
At 20 docs 0.0726
At 30 docs 0.0541
At 100 docs 0.0309
At 200 docs 0.0200
At 500 docs 0.0117
At 1000 docs 0.0084
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.0723
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (short without concepts)
STIX2
Summary Statistics
Run ID STIX2
Task adhoc
Relevant Level Relevant
Run Description interactive,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 52404
Relevant: 1910
Rel-ret: 350
Summary Statistics
Run ID STIX2
Task adhoc
Relevant Level Partial Relevant
Run Description interactive,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 52404
Relevant: 2345
Rel-ret: 445
Recall Level Precision Averages
Recall Precision
0.00 0.3110
0.10 0.1877
0.20 0.0934
0.30 0.0470
0.40 0.0090
0.50 0.0020
0.60 0.0006
0.70 0.0002
0.80 0.0002
0.90 0.0000
1.00 0.0000
Average precision over all
relevant documents
non-interpolated 0.0446
Document Level Averages
Precision
At 5 docs 0.1472
At 10 docs 0.0943
At 15 docs 0.0679
At 20 docs 0.0557
At 30 docs 0.0434
At 100 docs 0.0249
At 200 docs 0.0156
At 500 docs 0.0095
At 1000 docs 0.0066
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.0710
Recall Level Precision Averages
Recall Precision
0.00 0.3603
0.10 0.1942
0.20 0.0820
0.30 0.0386
0.40 0.0084
0.50 0.0020
0.60 0.0003
0.70 0.0003
0.80 0.0003
0.90 0.0002
1.00 0.0000
Average precision over all
relevant documents
non-interpolated 0.0429
Document Level Averages
Precision
At 5 docs 0.1774
At 10 docs 0.1113
At 15 docs 0.0805
At 20 docs 0.0660
At 30 docs 0.0509
At 100 docs 0.0315
At 200 docs 0.0201
At 500 docs 0.0117
At 1000 docs 0.0084
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.0683
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (short without concepts)
UTS1
Summary Statistics
Run ID UTS1
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 930
Summary Statistics
Run ID UTS1
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1210
Recall Level Precision Averages
Recall Precision
0.00 0.4052
0.10 0.3262
0.20 0.2365
0.30 0.1857
0.40 0.1575
0.50 0.1295
0.60 0.1009
0.70 0.0694
0.80 0.0482
0.90 0.0309
1.00 0.0245
Average precision over all
relevant documents
non-interpolated 0.1412
Document Level Averages
Precision
At 5 docs 0.2151
At 10 docs 0.2019
At 15 docs 0.1899
At 20 docs 0.1698
At 30 docs 0.1390
At 100 docs 0.0791
At 200 docs 0.0540
At 500 docs 0.0294
At 1000 docs 0.0175
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.1645
Recall Level Precision Averages
Recall Precision
0.00 0.4742
0.10 0.3549
0.20 0.2826
0.30 0.2278
0.40 0.1836
0.50 0.1555
0.60 0.1116
0.70 0.0854
0.80 0.0581
0.90 0.0396
1.00 0.0282
Average precision over all
relevant documents
non-interpolated 0.1623
Document Level Averages
Precision
At 5 docs 0.2679
At 10 docs 0.2491
At 15 docs 0.2365
At 20 docs 0.2142
At 30 docs 0.1830
At 100 docs 0.1030
At 200 docs 0.0691
At 500 docs 0.0376
At 1000 docs 0.0228
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.1996
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
UTS2
Summary Statistics
Run ID UTS2
Task adhoc
Relevant Level Relevant
Run Description automatic,long without cocepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1136
Summary Statistics
Run ID UTS2
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,long without cocepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1521
Recall Level Precision Averages
Recall Precision
0.00 0.7009
0.10 0.6219
0.20 0.4960
0.30 0.4380
0.40 0.3737
0.50 0.2984
0.60 0.2593
0.70 0.1913
0.80 0.1307
0.90 0.0664
1.00 0.0429
Average precision over all
relevant documents
non-interpolated 0.3043
Document Level Averages
Precision
At 5 docs 0.4377
At 10 docs 0.3830
At 15 docs 0.3421
At 20 docs 0.3019
At 30 docs 0.2579
At 100 docs 0.1334
At 200 docs 0.0809
At 500 docs 0.0385
At 1000 docs 0.0214
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3243
Recall Level Precision Averages
Recall Precision
0.00 0.7649
0.10 0.6627
0.20 0.5635
0.30 0.4963
0.40 0.4261
0.50 0.3436
0.60 0.2833
0.70 0.2120
0.80 0.1352
0.90 0.0817
1.00 0.0377
Average precision over all
relevant documents
non-interpolated 0.3420
Document Level Averages
Precision
At 5 docs 0.5321
At 10 docs 0.4736
At 15 docs 0.4252
At 20 docs 0.3792
At 30 docs 0.3283
At 100 docs 0.1751
At 200 docs 0.1084
At 500 docs 0.0517
At 1000 docs 0.0287
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3688
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long without concepts)
WSLab1
Summary Statistics
Run ID WSLab1
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 48622
Relevant: 1910
Rel-ret: 595
Summary Statistics
Run ID WSLab1
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 48622
Relevant: 2345
Rel-ret: 825
Recall Level Precision Averages
Recall Precision
0.00 0.1875
0.10 0.1490
0.20 0.1151
0.30 0.0896
0.40 0.0609
0.50 0.0512
0.60 0.0397
0.70 0.0214
0.80 0.0127
0.90 0.0044
1.00 0.0012
Average precision over all
relevant documents
non-interpolated 0.0585
Document Level Averages
Precision
At 5 docs 0.0943
At 10 docs 0.0868
At 15 docs 0.0931
At 20 docs 0.0830
At 30 docs 0.0711
At 100 docs 0.0453
At 200 docs 0.0322
At 500 docs 0.0184
At 1000 docs 0.0112
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.0747
Recall Level Precision Averages
Recall Precision
0.00 0.2272
0.10 0.1680
0.20 0.1333
0.30 0.0899
0.40 0.0697
0.50 0.0605
0.60 0.0439
0.70 0.0200
0.80 0.0125
0.90 0.0042
1.00 0.0012
Average precision over all
relevant documents
non-interpolated 0.0651
Document Level Averages
Precision
At 5 docs 0.1170
At 10 docs 0.1132
At 15 docs 0.1170
At 20 docs 0.1066
At 30 docs 0.0969
At 100 docs 0.0626
At 200 docs 0.0441
At 500 docs 0.0254
At 1000 docs 0.0156
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.0922
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
WSLab2
Summary Statistics
Run ID WSLab2
Task adhoc
Relevant Level Relevant
Run Description automatic,long without cocepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 736
Summary Statistics
Run ID WSLab2
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,long without cocepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1002
Recall Level Precision Averages
Recall Precision
0.00 0.2866
0.10 0.2411
0.20 0.1709
0.30 0.1279
0.40 0.0881
0.50 0.0736
0.60 0.0503
0.70 0.0316
0.80 0.0185
0.90 0.0065
1.00 0.0023
Average precision over all
relevant documents
non-interpolated 0.0882
Document Level Averages
Precision
At 5 docs 0.1623
At 10 docs 0.1283
At 15 docs 0.1233
At 20 docs 0.1123
At 30 docs 0.0975
At 100 docs 0.0589
At 200 docs 0.0417
At 500 docs 0.0227
At 1000 docs 0.0139
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.1084
Recall Level Precision Averages
Recall Precision
0.00 0.3644
0.10 0.2506
0.20 0.1908
0.30 0.1384
0.40 0.1068
0.50 0.0886
0.60 0.0641
0.70 0.0355
0.80 0.0172
0.90 0.0069
1.00 0.0023
Average precision over all
relevant documents
non-interpolated 0.1001
Document Level Averages
Precision
At 5 docs 0.1811
At 10 docs 0.1566
At 15 docs 0.1585
At 20 docs 0.1500
At 30 docs 0.1283
At 100 docs 0.0770
At 200 docs 0.0541
At 500 docs 0.0305
At 1000 docs 0.0189
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.1326
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long without concepts)
jscb1
Summary Statistics
Run ID jscb1
Task adhoc
Relevant Level Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1374
Summary Statistics
Run ID jscb1
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,short without concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1764
Recall Level Precision Averages
Recall Precision
0.00 0.6834
0.10 0.6172
0.20 0.5209
0.30 0.4820
0.40 0.4131
0.50 0.3731
0.60 0.3148
0.70 0.2565
0.80 0.2203
0.90 0.1437
1.00 0.0923
Average precision over all
relevant documents
non-interpolated 0.3596
Document Level Averages
Precision
At 5 docs 0.4528
At 10 docs 0.3962
At 15 docs 0.3572
At 20 docs 0.3245
At 30 docs 0.2843
At 100 docs 0.1455
At 200 docs 0.0919
At 500 docs 0.0462
At 1000 docs 0.0259
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3505
Recall Level Precision Averages
Recall Precision
0.00 0.7163
0.10 0.6351
0.20 0.5678
0.30 0.5209
0.40 0.4367
0.50 0.3987
0.60 0.3393
0.70 0.2801
0.80 0.2275
0.90 0.1311
1.00 0.0616
Average precision over all
relevant documents
non-interpolated 0.3790
Document Level Averages
Precision
At 5 docs 0.5321
At 10 docs 0.4755
At 15 docs 0.4239
At 20 docs 0.3896
At 30 docs 0.3491
At 100 docs 0.1887
At 200 docs 0.1208
At 500 docs 0.0600
At 1000 docs 0.0333
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.3804
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (short without concepts)
jscb2
Summary Statistics
Run ID jscb2
Task adhoc
Relevant Level Relevant
Run Description automatic,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1480
Summary Statistics
Run ID jscb2
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1895
Recall Level Precision Averages
Recall Precision
0.00 0.8398
0.10 0.7408
0.20 0.6687
0.30 0.5896
0.40 0.5231
0.50 0.4660
0.60 0.4004
0.70 0.3350
0.80 0.2609
0.90 0.1603
1.00 0.0965
Average precision over all
relevant documents
non-interpolated 0.4436
Document Level Averages
Precision
At 5 docs 0.5962
At 10 docs 0.5170
At 15 docs 0.4377
At 20 docs 0.3887
At 30 docs 0.3327
At 100 docs 0.1711
At 200 docs 0.1058
At 500 docs 0.0512
At 1000 docs 0.0279
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.4301
Recall Level Precision Averages
Recall Precision
0.00 0.9224
0.10 0.8041
0.20 0.7403
0.30 0.6551
0.40 0.5814
0.50 0.5097
0.60 0.4323
0.70 0.3381
0.80 0.2773
0.90 0.1744
1.00 0.0794
Average precision over all
relevant documents
non-interpolated 0.4852
Document Level Averages
Precision
At 5 docs 0.6981
At 10 docs 0.6170
At 15 docs 0.5358
At 20 docs 0.4783
At 30 docs 0.4101
At 100 docs 0.2221
At 200 docs 0.1388
At 500 docs 0.0660
At 1000 docs 0.0358
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.4673
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long with concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long with concepts)
jscb3
Summary Statistics
Run ID jscb3
Task adhoc
Relevant Level Relevant
Run Description interactive,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 1571
Summary Statistics
Run ID jscb3
Task adhoc
Relevant Level Partial Relevant
Run Description interactive,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1989
Recall Level Precision Averages
Recall Precision
0.00 0.8626
0.10 0.7769
0.20 0.6925
0.30 0.6363
0.40 0.5606
0.50 0.5044
0.60 0.4481
0.70 0.3945
0.80 0.3329
0.90 0.2073
1.00 0.1128
Average precision over all
relevant documents
non-interpolated 0.4855
Document Level Averages
Precision
At 5 docs 0.6679
At 10 docs 0.5340
At 15 docs 0.4579
At 20 docs 0.4066
At 30 docs 0.3522
At 100 docs 0.1823
At 200 docs 0.1133
At 500 docs 0.0544
At 1000 docs 0.0296
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.4648
Recall Level Precision Averages
Recall Precision
0.00 0.9431
0.10 0.8597
0.20 0.7727
0.30 0.7122
0.40 0.6340
0.50 0.5724
0.60 0.4966
0.70 0.4143
0.80 0.3196
0.90 0.2214
1.00 0.0951
Average precision over all
relevant documents
non-interpolated 0.5358
Document Level Averages
Precision
At 5 docs 0.7811
At 10 docs 0.6434
At 15 docs 0.5610
At 20 docs 0.5000
At 30 docs 0.4358
At 100 docs 0.2364
At 200 docs 0.1471
At 500 docs 0.0695
At 1000 docs 0.0375
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.5007
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (long with concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (long with concepts)
rmit02
Summary Statistics
Run ID rmit02
Task adhoc
Relevant Level Relevant
Run Description automatic,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 963
Summary Statistics
Run ID rmit02
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1289
Recall Level Precision Averages
Recall Precision
0.00 0.6029
0.10 0.4583
0.20 0.3761
0.30 0.3211
0.40 0.2566
0.50 0.2025
0.60 0.1408
0.70 0.0940
0.80 0.0708
0.90 0.0478
1.00 0.0275
Average precision over all
relevant documents
non-interpolated 0.2166
Document Level Averages
Precision
At 5 docs 0.3811
At 10 docs 0.3094
At 15 docs 0.2604
At 20 docs 0.2330
At 30 docs 0.1943
At 100 docs 0.0970
At 200 docs 0.0592
At 500 docs 0.0310
At 1000 docs 0.0182
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2447
Recall Level Precision Averages
Recall Precision
0.00 0.6990
0.10 0.5242
0.20 0.4414
0.30 0.3657
0.40 0.2922
0.50 0.2351
0.60 0.1569
0.70 0.1085
0.80 0.0706
0.90 0.0552
1.00 0.0337
Average precision over all
relevant documents
non-interpolated 0.2490
Document Level Averages
Precision
At 5 docs 0.4830
At 10 docs 0.3906
At 15 docs 0.3396
At 20 docs 0.3047
At 30 docs 0.2560
At 100 docs 0.1289
At 200 docs 0.0793
At 500 docs 0.0418
At 1000 docs 0.0243
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2800
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long with concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long with concepts)
sstut3
Summary Statistics
Run ID sstut3
Task adhoc
Relevant Level Relevant
Run Description automatic,long without cocepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 52999
Relevant: 1910
Rel-ret: 1028
Summary Statistics
Run ID sstut3
Task adhoc
Relevant Level Partial Relevant
Run Description automatic,long without cocepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 52999
Relevant: 2345
Rel-ret: 1318
Recall Level Precision Averages
Recall Precision
0.00 0.6456
0.10 0.5406
0.20 0.4147
0.30 0.3356
0.40 0.2642
0.50 0.2004
0.60 0.1427
0.70 0.1028
0.80 0.0693
0.90 0.0256
1.00 0.0215
Average precision over all
relevant documents
non-interpolated 0.2303
Document Level Averages
Precision
At 5 docs 0.4075
At 10 docs 0.3340
At 15 docs 0.2943
At 20 docs 0.2651
At 30 docs 0.2151
At 100 docs 0.1091
At 200 docs 0.0673
At 500 docs 0.0334
At 1000 docs 0.0194
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2544
Recall Level Precision Averages
Recall Precision
0.00 0.7586
0.10 0.5823
0.20 0.4822
0.30 0.3624
0.40 0.3029
0.50 0.2172
0.60 0.1615
0.70 0.1155
0.80 0.0783
0.90 0.0356
1.00 0.0186
Average precision over all
relevant documents
non-interpolated 0.2583
Document Level Averages
Precision
At 5 docs 0.4868
At 10 docs 0.4189
At 15 docs 0.3648
At 20 docs 0.3264
At 30 docs 0.2667
At 100 docs 0.1379
At 200 docs 0.0858
At 500 docs 0.0431
At 1000 docs 0.0249
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.2948
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long without concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(automatic) (long without concepts)
tgl1
Summary Statistics
Run ID tgl1
Task adhoc
Relevant Level Relevant
Run Description interactive,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 1910
Rel-ret: 857
Summary Statistics
Run ID tgl1
Task adhoc
Relevant Level Partial Relevant
Run Description interactive,long with concepts
Number of Topics 53
Total number of documents over all topics
Retrieved: 53000
Relevant: 2345
Rel-ret: 1139
Recall Level Precision Averages
Recall Precision
0.00 0.4267
0.10 0.2937
0.20 0.1866
0.30 0.1462
0.40 0.1181
0.50 0.1018
0.60 0.0763
0.70 0.0532
0.80 0.0232
0.90 0.0088
1.00 0.0052
Average precision over all
relevant documents
non-interpolated 0.1168
Document Level Averages
Precision
At 5 docs 0.2000
At 10 docs 0.1698
At 15 docs 0.1459
At 20 docs 0.1396
At 30 docs 0.1195
At 100 docs 0.0657
At 200 docs 0.0461
At 500 docs 0.0257
At 1000 docs 0.0162
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.1379
Recall Level Precision Averages
Recall Precision
0.00 0.4755
0.10 0.3113
0.20 0.2059
0.30 0.1708
0.40 0.1365
0.50 0.1173
0.60 0.0794
0.70 0.0351
0.80 0.0184
0.90 0.0087
1.00 0.0034
Average precision over all
relevant documents
non-interpolated 0.1262
Document Level Averages
Precision
At 5 docs 0.2491
At 10 docs 0.2094
At 15 docs 0.1836
At 20 docs 0.1745
At 30 docs 0.1491
At 100 docs 0.0864
At 200 docs 0.0612
At 500 docs 0.0342
At 1000 docs 0.0215
R-Presicion (presicion after
R docs retreived (where R is
the number of relevant
documnets))
Exact 0.1581
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Pre
cisi
on
Recall
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Recall-Precision Curve Di�erence from Median in
Average Precision per Topic
(all)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (long with concepts)
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
-1-0.8-0.6-0.4-0.2
00.20.40.60.8
1
40 50 60 70 80
Diff
eren
ce
Topic
Di�erence from Median in Di�erence from Median in
Average Precision per Topic Average Precision per Topic
(interactive) (long with concepts)