Optimal aggregation algorithms for middleware Ronald Fagin, Amnon Lotem , and Moni Naor

1

Optimal aggregation algorithms for

middlewareRonald Fagin, Amnon Lotem,

and Moni NaorPresented by Suresh Barukula

2011csz8090

2

Top-k query processing means finding k-objects, that have highest overall grades.

A query in multimedia database *combines different graded attributes

through an aggregation function *Overall grade for each object will be

calculated using an aggregation function, and we can return top-k objects.

What is top-k query processing

3

In general multimedia databases contains fuzzy data.

For example: We want to retrieve all red objects

What we can say about the below object? Is it red or not? We can’t say whether it is red or not, but

we can grade it by the amount of redness. attribute values are typically graded [0,1]

Why is it important?

4

FA-Fagin’s Algorithm TA-Threshold Algorithm TAZ Algorithm NRA- No Random Access CA- Combined Algorithm

What are the ways?

5

N-Number of Objects m-No of attributes xi Є [0,1] Database is consisting of m sorted lists L1…Lm; each of length N . We may refer to Li as list i. Each entry of Li is of the form (R, xi), where xi is the ith field of object R, Each list Li is sorted in descending order by the xi value.

The database model

6

Sorted access Random access The cost of the middleware is sCS+ rCR

Where s is the no of sorted accesses, r is no of random accesses , CS is sorted access cost and CR is random access cost. 

Modes of accessing the database

7

Example – Simple Database model

(a, 0.9)

(b, 0.8)

(c, 0.72)

(d, 0.6)

.

.

.

.

Sorted L1

(d, 0.9)

(a, 0.85)

(b, 0.7)

(c, 0.2)

.

.

.

.N

a

b

c

d ....

ObjectID

0.9

0.8

0.72

0.6

.

.

.

.

Attribute 1

0.85

0.2

0.9....

Attribute 2

0.7

MSorted L2

8

Find the top 2 (k = 2) objects on the following ‘query’ executed on the middleware:

A1 & A2 (eg: color=red & shape=round)

Example – Simple Query

A1 & A2 as a ‘query’ to the middlewareresults in combining of the grades of A1 andA2 by min(A1,A2)

9

c

ID A1 A2 Min(A1,A2)

STEP 1: Read attributes from every sorted list• Stop when k objects have been seen in common from all lists

(a, 0.9)

(b, 0.8)

(c, 0.72)

(d, 0.6)

.

.

.

.

L1 L2

(d, 0.9)

(a, 0.85)

(b, 0.7)

(c, 0.2)

.

.

.

.

a

d

0.9

0.9

0.85

b 0.8

0.72

0.7

Example – Fagin’s Algorithm

10

c

ID A1 A2 Min(A1,A2)

STEP 2: Random access to find missing grades

(a, 0.9)

(b, 0.8)

(c, 0.72)

(d, 0.6)

.

.

.

.

L1 L2

(d, 0.9)

(a, 0.85)

(b, 0.7)

(c, 0.2)

.

.

.

.

a

d

0.9

0.9

0.85

b 0.8

0.72

0.7

0.6

0.2

Example – Fagin’s Algortihm

11

c

ID A1 A2 Min(A1,A2)

STEP 3• Compute the grades of the seen objects.• Return the k highest graded objects.

(a, 0.9)

(b, 0.8)

(c, 0.72)

(d, 0.6)

.

.

.

.

L1 L2

(d, 0.9)

(a, 0.85)

(b, 0.7)

(c, 0.2)

.

.

.

.

a

d

0.9

0.9

0.85

b 0.8

0.72

0.7

0.6

0.2

0.85

0.6

0.7

0.2

Example – Fagin’s Algortihm

12

Read all grades of an object once seen from a sorted access• No need to wait until the lists give k common objects

Do sorted access (and corresponding random accesses) until you have seen the top k answers.

• How do we know that grades of seen objects are higher than the grades of unseen objects ?• Predict maximum possible grade unseen objects:

a: 0.9b: 0.8c: 0.72

.

.

.

.

L1 L2

d: 0.9a: 0.85b: 0.7

c: 0.2

.

.

.

.f: 0.65d: 0.6

f: 0.6

Seen

Possibly unseen Threshold value

Threshold Algorithm (TA)

T = min(0.72, 0.7) = 0.7

13

ID A1 A2 Min(A1,A2)

Step 1: - parallel sorted access to each list

(a, 0.9)

(b, 0.8)

(c, 0.72)

(d, 0.6)

.

.

.

.

L1 L2

(d, 0.9)

(a, 0.85)

(b, 0.7)

(c, 0.2)

.

.

.

.

a

d

0.9

0.9

0.85 0.85

0.6 0.6

For each object seen: - get all grades by random access - determine Min(A1,A2) - amongst 2 highest seen ? keep in buffer

Example – Threshold Algorithm

14

ID A1 A2 Min(A1,A2)a: 0.9

b: 0.8

c: 0.72

d: 0.6

.

.

.

.

L1 L2

d: 0.9

a: 0.85

b: 0.7

c: 0.2

.

.

.

.

Step 2: - Determine threshold value based on objects currently seen under sorted access. T = min(L1, L2)

a

d

0.9

0.9

0.85 0.85

0.6 0.6

T = min(0.9, 0.9) = 0.9

- 2 objects with overall grade ≥ threshold value ? stop else go to next entry position in sorted list and repeat step 1

Example – Threshold Algorithm

15

ID A1 A2 Min(A1,A2)

Step 1 (Again): - parallel sorted access to each list

(a, 0.9)

(b, 0.8)

(c, 0.72)

(d, 0.6)

.

.

.

.

L1 L2

(d, 0.9)

(a, 0.85)

(b, 0.7)

(c, 0.2)

.

.

.

.

a

d

0.9

0.9

0.85 0.85

0.6 0.6

For each object seen: - get all grades by random access - determine Min(A1,A2) - amongst 2 highest seen ? keep in buffer

b 0.8 0.7 0.7

Example – Threshold Algorithm

16

ID A1 A2 Min(A1,A2)a: 0.9

b: 0.8

c: 0.72

d: 0.6

.

.

.

.

L1 L2

d: 0.9

a: 0.85

b: 0.7

c: 0.2

.

.

.

.

Step 2 (Again): - Determine threshold value based on objects currently seen. T = min(L1, L2)

a

b

0.9

0.7

0.85 0.85

0.8 0.7

T = min(0.8, 0.85) = 0.8

- 2 objects with overall grade ≥ threshold value ? stop else go to next entry position in sorted list and repeat step 1

Example – Threshold Algorithm

17

ID A1 A2 Min(A1,A2)a: 0.9

b: 0.8

c: 0.72

d: 0.6

.

.

.

.

L1 L2

d: 0.9

a: 0.85

b: 0.7

c: 0.2

.

.

.

.

Situation at stopping condition

a

b

0.9

0.7

0.85 0.85

0.8 0.7

T = min(0.72, 0.7) = 0.7

Example – Threshold Algorithm

18

The middleware cost of the FA is same no matter what the aggregation function is . 

TA stops at least as early as FA TA may perform more random accesses

than FA TA requires only bounded buffers TA can be stopped early(θ-approximation)

TA vs FA

19

A = class of algorithms, A Є A represents an algorithmD = legal inputs to algorithms (databases), D Є D represents a database

Cost(A,D ) = middleware cost when running algorithm A over database D

Concept of instance optimality

Algorithm B is instance optimal over A and D if :B Є A and Cost(B,D ) = O(Cost(A,D )) A Є A, D Є D

Which means that:Cost(B,D ) ≤ c . Cost(A,D ) + c’, A Є A, D Є D

optimality ratio

,

20

Theorem: If the aggregation function t is monotone, TA correctly finds the top K answers.

  Theorem: TA is instance optimal for every

monotone aggregation function, over every database

(Note: if we exclude wild guesses). 

Some facts about TA

21

TAZ Algorithm(No Sorted Access)

(a, 0.9)

(b, 0.8)

(c, 0.72)

(d, 0.6)

.

.

.

.

L1 L2

(d, 0.9)

(a, 0.85)

(b, 0.7)

(c, 0.2)

.

.

.

.

(b, 0.6)

(a, 0.83)

(d, 0.61)

(c, 0.9)

.

.

.

.

L3

1

T=min(0.72,0.7,1)=0.7

22

Can we determine rank of an object without seeing all of it’s grades? The main essence of this algorithm is estimating the rank using best and

worst possible values

NRA(No Random Access)

1

1/3

1/3

1/3

.

.

.

.

1/3

1/3

1/3

1/3

.

.

.

.

23

CA is merge between TA and NRA.

The idea of CA is to run NRA but after every h steps to perform random access step.

Both NRA and CA are instance optimal over all databases, when the aggregation function is monotone

CA(Combined Algorithm)

24

In this paper the authors have studied a simple and elegant algorithm called TA.

They have also studied the variants of TA, when there are no sorted access, no random access etc..,

They have emphasized on instance optimality and they have proved that their algorithms are instance optimal over all algorithms for all databases under normal assumptions.

But they have not considered the computational costs and the data structures that are required to implement the algorithms.

Conclusion

25

Questions?