Querying Sensor Networks

1

Querying Sensor Networks

Sam MaddenUC Berkeley

2

Sensor Networks• Small computers with:

– Radios– Sensing hardware– Batteries

• Remote deployments– Long lived– 10s, 100s, or 1000s Battery Pack

Smart Sensor, aka “Mote”

3

MotesMica Mote4Mhz, 8 bit Atmel RISC uProc40 kbit Radio4 K RAM, 128 K Program Flash, 512 K Data FlashAA battery packBased on TinyOS**Hill, Szewczyk, Woo, Culler, & Pister.

“Systems Architecture Directions for Networked Sensors.” ASPLOS 2000. http://webs.cs.berkeley.edu/tos

4

Programming Sensor Nets Is Hard

– Months of lifetime required from small batteries» 3-5 days naively; can’t recharge often» Interleave sleep with processing

–Lossy, low-bandwidth communication»Nodes coming and going»~20% loss @ 5m»Multi-hop

–Remote, zero administration deployments

–Highly distributed environment–Limited Development Tools»Embedded, LEDs for Debugging!

Need high level abstractions!

Current (mA) by Processing Phase

0

5

10

15

20

Processing Processing &Listening

Processing &Transmitting

I dle

Curr

ent

(mA)

200-800 instructions per bit transmitted!

High-Level Abstraction Is Needed!

5

A Solution: Declarative Queries

• Users specify the data they want– Simple, SQL-like queries– Using predicates, not specific addresses– System: TinyDB

• System processes queries– Challenges

»Language»Data location»Power efficient data collection»Multihop result delivery

“Transparent optimization” System does better* than most programmers* Is more power efficient

6

Sensor Net Sample Apps

Traditional monitoring apparatus.

Earthquake monitoring in shake-test sites.

Vehicle detection: sensors along a road, collect data about passing vehicles.

Habitat Monitoring: Storm petrels on Great Duck Island, microclimates on James Reserve.

7

Overview• TinyDB: Queries for Sensor Nets• Processing Aggregate Queries

(TAG)• Experiments & Optimizations• Acquisitional Query Processing• Other Research • Future Directions

8

Overview• TinyDB: Queries for Sensor Nets• Processing Aggregate Queries

(TAG)• Experiments & Optimizations• Acquisitional Query Processing• Other Research • Future Directions

9

TinyDB Demo

10

TinyOSSchema

Query Processor

Multihop Network

TinyDB Architecture

Schema:•“Catalog” of commands & attributes

Filterlight >

400get (‘temp’)

Aggavg(tem

p)

QueriesSELECT AVG(temp) WHERE light > 400

ResultsT:1, AVG: 225T:2, AVG: 250

Tables Samples got(‘temp’)Name: tempTime to sample: 50 uSCost to sample: 90 uJCalibration Table: 3Units: Deg. FError: ± 5 Deg FGet f : getTempFunc()…

getTempFunc(…)getTempFunc(…)

TinyDBTinyDB

~10,000 Lines Embedded C Code~5,000 Lines (PC-Side) Java~3200 Bytes RAM (w/ 768 byte heap)~58 kB compiled code(3x larger than 2nd largest TinyOS Program)

11

Declarative Queries for Sensor Networks

• Examples:SELECT nodeid, nestNo, lightFROM sensorsWHERE light > 400EPOCH DURATION 1s

1EpocEpoc

hhNodeiNodei

ddnestNnestN

ooLightLight

0 1 17 4550 2 25 3891 1 17 4221 2 25 405

Sensors

“Find the bright nests.”“Count the number occupied nests in each loud region of the island.”

12

Aggregation Queries

Epoch region CNT(…) AVG(…)0 North 3 3600 South 3 5201 North 3 3701 South 3 520

“Find the bright nests.”“Count the number occupied nests in each loud region of the island.”

SELECT region, CNT(occupied) AVG(sound)

FROM sensorsGROUP BY regionHAVING AVG(sound) > 200EPOCH DURATION 10s

3

Regions w/ AVG(sound) > 200

SELECT AVG(sound)FROM sensorsEPOCH DURATION 10s

2

13

Overview• TinyDB: Queries for Sensor Nets• Processing Aggregate Queries

(TAG)• Experiments & Optimizations• Acquisitional Query Processing• Other Research • Future Directions

14

Tiny Aggregation (TAG)• In-network processing of aggregates

– Common data analysis operation» Aka gather operation or reduction in || programming

– Communication reducing» Operator dependent benefit

– Across nodes during same epoch

• Exploit query semantics to improve efficiency!

Madden, Franklin, Hellerstein, Hong. Tiny AGgregation (TAG), OSDI 2002.

15

Query Propagation Via Tree-Based Routing

• Tree-based routing– Used in:

» Query delivery » Data collection

– Topology selection is important; e.g.

» Krishnamachari, DEBS 2002, Intanagonwiwat, ICDCS 2002, Heidemann, SOSP 2001

» SIGMOD 2003– Continuous process

» Mitigates failures

A

B C

D

FE

Q:SELECT …

Q Q

Q

QQ

Q

Q

QQ

Q QQ

R:{…}R:{…}

R:{…}

R:{…} R:{…}

16

Basic Aggregation• In each epoch:

– Each node samples local sensors once– Generates partial state record (PSR)

» local readings » readings from children

– Outputs PSR during assigned comm. interval

• At end of epoch, PSR for whole network output at root

• New result on each successive epoch

• Extras:– Predicate-based partitioning via GROUP BY

1

2 3

4

5

17

Illustration: Aggregation

1 2 3 4 5

4 13

2

1

4

1

2 3

4

51

Sensor #

Inte

rval

#

Interval 4SELECT COUNT(*) FROM sensors

Epoch

18

Illustration: Aggregation

1 2 3 4 5

4 13 22

1

4

1

2 3

4

5

2

Sensor #Interval 3

SELECT COUNT(*) FROM sensors

Inte

rval

#

19

Illustration: Aggregation

1 2 3 4 5

4 13 22 1 31

4

1

2 3

4

5

31

Sensor #Interval 2

SELECT COUNT(*) FROM sensors

Inte

rval

#

20

Illustration: Aggregation

1 2 3 4 5

4 13 22 1 31 54

1

2 3

4

5

5Sensor #

SELECT COUNT(*) FROM sensors Interval 1

Inte

rval

#

21

Illustration: Aggregation

1 2 3 4 5

4 13 22 1 31 54 1

1

2 3

4

51

Sensor #

SELECT COUNT(*) FROM sensors Interval 4

Inte

rval

#

22

Interval Assignment: An Approach

1

2 3

4

5

SELECT SELECT COUNT(*)…COUNT(*)…

•CSMA for collision avoidance

•Time intervals for power conservation

•Time Sync (e.g. Elson & Estrin OSDI 2002)

4 intervals / epochInterval # = Level

4

3

Level = 1

2

Epoch

Comm Interval4 3 2 1 555

ZZ Z Z

Z ZZZ Z

ZZ Z Z

Z Z ZZ Z

ZZ Z Z

Z Z ZZ Z

ZZ Z Z

Z Z ZZ Z

ZZ Z Z

Z Z ZZ Z

ZZ Z

ZZ Z

ZZ Z

ZZ Z

L T

L T

L T

T

L T

L LPipelining: Increase throughput by delaying result arrival until a later epochMadden, Szewczyk, Franklin, Culler. Supporting Aggregate Queries Over Ad-Hoc Wireless Sensor Networks. WMCSA 2002.

23

Aggregation Framework• As in extensible databases, we support any

aggregation function conforming to:Aggn={finit, fmerge, fevaluate}Finit {a0} <a0>Fmerge {<a1>,<a2>} <a12>Fevaluate {<a1>} aggregate value

Example: AverageAVGinit {v} <v,1>AVGmerge {<S1, C1>, <S2, C2>} < S1 + S2 , C1 + C2>AVGevaluate{<S, C>} S/C

Partial State Record (PSR)

Restriction: Merge associative, commutative

24

Types of Aggregates• SQL supports MIN, MAX, SUM, COUNT,

AVERAGE

• Any function over a set can be computed via TAG

• In network benefit for many operations– E.g. Standard deviation, top/bottom N, spatial

union/intersection, histograms, etc. – Compactness of PSR

25

Overview• TinyDB: Queries for Sensor Nets• Processing Aggregate Queries

(TAG)• Experiments & Optimizations• Acquisitional Query Processing• Other Research• Future Directions

26

Taxonomy of Aggregates• TAG insight: classify aggregates according to

various functional properties– Yields a general set of optimizations that can

automatically be appliedProperty Examples Affects

Partial State MEDIAN : unbounded, MAX : 1 record

Effectiveness of TAG

Duplicate Sensitivity

MIN : dup. insensitive,AVG : dup. sensitive

Routing Redundancy

Exemplary vs. Summary

MAX : exemplaryCOUNT: summary

Applicability of Sampling, Effect of Loss

Monotonic COUNT : monotonicAVG : non-monotonic

Hypothesis Testing, Snooping

Drives an API!

27

Taxonomy Related Insights

• Communication Reducing– In-network Aggregation – Hypothesis Testing– Snooping– Sampling

• Quality Increasing– Multiple Parents– Child Cache

28

Simulation Environment• Evaluated TAG via simulation

• Coarse grained event based simulator– Sensors arranged on a grid– Two communication models

»Lossless: All neighbors hear all messages»Lossy: Messages lost with probability that

increases with distance

29

Benefit of In-Network Processing

Simulation Results2500 Nodes50x50 GridDepth = ~10Neighbors = ~20Uniform Dist.

Total Bytes Xmitted vs. Aggregation Function

0100002000030000400005000060000700008000090000

100000

EXTERNAL MAX AVERAGE DI STI NCT MEDI ANAggregation Function

Tota

l Byt

es X

mitt

ed

•Aggregate & depth dependent benefit!

HolisticHolisticUniqueUnique

DistributiveDistributiveAlgebraicAlgebraic

30

Channel Sharing (“Snooping”)

• Insight: Shared channel can reduce communication

• Suppress messages that won’t affect aggregate– E.g., MAX– Applies to all exemplary, monotonic

aggregates

31

Hypothesis Testing• Insight: Guess from root can be used for

suppression– E.g. ‘MIN < 50’– Works for monotonic & exemplary aggregates

»Also summary, if imprecision allowed

• How is hypothesis computed?– Blind or statistically informed guess– Observation over network subset

32

Experiment: Snooping vs. Hypothesis Testing

•Uniform Value Distribution•Dense Packing •Ideal Communication

Messages/ Epoch vs. Network Diameter(SELECT MAX(attr), R(attr) = [0,100])

0

500

1000

1500

2000

2500

3000

10 20 30 40 50Network Diameter

Mes

sage

s /

Epoc

h

No GuessGuess = 50Guess = 90Snooping

Pruning in Network

Pruning at Leaves

33

Use Multiple Parents• Use graph structure

– Increase delivery probability with no communication overhead

• For duplicate insensitive aggregates• Or aggregates that can be expressed as a sum of parts

– Send (part of) aggregate to all parents» In just one message, via multicast

– Assuming independence, decreases variance

SELECT COUNT(*)

A

B C

R

A

B C

c

R

P(link level loss) = pP(loss) = p2

E(cnt) = c * p2

Var(cnt) = c2 * p2 * (1 – p2) V

# of parents = n

E(cnt) = n * (c/n * p2)Var(cnt) = n * (c/n)2 *

p2 * (1 – p2) = V/n

A

B C

c/n c/n

R

n = 2

34

Multiple Parents Results• Better than

previous analysis expected!

• Losses aren’t independent!

• Insight: spreads data over many links

Benefit of Result Splitting (COUNT query)

0

200

400

600

800

1000

1200

1400

(2500 nodes, lossy radio model, 6 parents per node)

Avg.

CO

UNT Splitting

No Splitting

Critical Link!

No Splitting With Splitting

35

TAG Advantages• Simple but powerful data collection language

– Logical partitioning via grouping

• In network processing – Reduces communication

» Power and contention benefits

• Power Aware Routing– Integration of sleeping, computation

• Continuous stream of results

• Taxonomy driven techniques to– Improve quality– Reduce communication

36

Overview• TinyDB: Queries for Sensor Nets• Processing Aggregate Queries

(TAG)• Experiments & Optimizations• Acquisitional Query Processing• Other Research • Future Directions

37

Acquisitional Query Processing (ACQP)

• Closed world assumption does not hold– Could generate an infinite number of samples

• An acqusitional query processor controls – when, – where, – and with what frequency data is collected!

• Versus traditional systems where data is provided a priori

Madden, Franklin, Hellerstein, and Hong. The Design of An Acqusitional Query Processor. SIGMOD, 2003 (to appear).

38

ACQP: What’s Different?• How does the user control acquisition?

– Rates or lifetimes– Event-based triggers

• How should the query be processed?– Sampling as a first class operation– Event – join duality

• Which nodes have relevant data?• Which samples should be transmitted?

39

Lifetime Queries• Lifetime vs. sample rate

SELECT …EPOCH DURATION 10 s

SELECT …LIFETIME 30 days

• Extra: Allow a MAX SAMPLE PERIOD– Discard some samples– Sampling cheaper than transmitting

40

(Single Node) Lifetime PredictionVoltage vs. Time, Measured Vs. ExpectedLif etime Goal = 24 Weeks (4032 Hours. 15 s / sample)

R2 = 0.8455

300400500600700800900

1000

0 1000 2000 3000 4000Time (Hours)

Volta

ge (

Raw

Units

)

Voltage (Expected)Voltage (Measured)Linear Fit

95097099010101030

0 100 200 300

ExpectedMeasured

I nsuffi cient Voltage to Operate (V = 350)

41

• E(sampling mag) >> E(sampling light)

1500 uJ vs. 90 uJ

Operator Ordering: Interleave Sampling + Selection

SELECT light, magFROM sensorsWHERE pred1(mag)AND pred2(light)EPOCH DURATION 1s

(pred1)

(pred2)

mag

light

(pred1)

(pred2)

mag

light

(pred1)

(pred2)

mag light

Traditional DBMS

ACQP

At 1 sample / sec, total power savings could be as much as 3.5mW Comparable to processor!

Correct orderingCorrect ordering(unless pred1 is (unless pred1 is very very selective selective

and pred2 is not):and pred2 is not):

Cheap

Costly

42

Exemplary Aggregate Pushdown

SELECT WINMAX(light,8s,8s)FROM sensorsWHERE mag > xEPOCH DURATION 1s

• Novel, general pushdown technique

• Mag sampling is the most expensive operation!

WINMAX

(mag>x)

mag light

Traditional DBMS

light

mag

(mag>x)

WINMAX

(light > MAX)

ACQP

43

Overview• TinyDB: Queries for Sensor Nets• Processing Aggregate Queries

(TAG)• Experiments & Optimizations• Acquisitional Query Processing• Other Research• Future Directions

44

Fun Stuff• Temporal aggregates

• Sophisticated, sensor network specific aggregates

• Lossy compression– Wavelets

Hellerstein, Hong, Madden, and Stanek. Beyond Average. IPSN 2003 (to appear)

“Isobar Finding”

45

Additional Research• Sensors, TinyDB, TinyOS

– This Talk: » TAG (OSDI 2002)» ACQP (SIGMOD 2003)» WMCSA 2002» IPSN 2003

– TOSSIM. Levis, Lee, Woo, Madden, & Culler. (In submission)

– TinyOS contributions: memory allocator, catalog, network reprogramming, OS support, releases, TinyDB

46

Other Research (Cont)• Stream Query Processing

– CACQ (SIGMOD 2002)» Madden, Shah, Hellerstein, & Raman

– Fjords (ICDE 2002)» Madden & Franklin

– Java Experiences Paper (SIGMOD Record, December 2001)

» Shah, Madden, Franklin, and Hellerstein

– Telegraph Project, FFF & ACM1 Demos» Telegraph Team

47

TinyDB Deployments• Initial efforts:

– Network monitoring– Vehicle tracking

• Ongoing deployments:– Environmental monitoring – Generic Sensor Kit– Building Monitoring– Golden Gate Bridge

48

Overview• TinyDB: Queries for Sensor Nets• Processing Aggregate Queries

(TAG)• Experiments & Optimizations• Acquisitional Query Processing• Other Research• Future Directions

49

TinyDB Future Directions• Expressing lossiness

– No longer a closed world!• Additional Operations

– Joins– Signal Processing

• Integration with Streaming DBMS– In-network vs. external operations

• Heterogeneous Environments• Real Deployments

50

Summary• Higher-level programming abstractions for sensor networks

are necessary• For many apps, queries are the right abstraction!

– Ease of programming– Transparent optimization

• Aggregation is a fundamental operation – Semantically aware optimizations– Close integration with network

• Acquisitional Query Processing – Framework for addressing the new issues that arise in sensor

networks, e.g.» Order of sampling and selection» Languages, indices, approximations that give user control over which

data enters the system• Consideration of database, network, and device issues

http://telegraph.cs.berkeley.edu/tinydb

51

Questions?