Supporting Multi-Fidelity Computations in Mobile Interactive Applications

Dushyanth Narayanan

2

Mobile Interactive Applications

Wearable computing = mobility + interactivity

A mobile system has limited battery energy unpredictable, turbulent network environment other resources are scarce

An interactive system needs low latency (good interactive response) minimal user distraction

Goal: build a system to support mobile interactive apps

3

Some examples

Augmented reality an architect wants to remodel a warehouse. an aircraft technician wants to see hidden wiring. a soldier needs to be aware of enemy positions.

Search find restaurants that I can get to within five minutes. find images that look like this building’s façade. plan an alternate driving route before I miss my exit.

Speech recognition

Image processing surveillance: filter out uninteresting images, report

interesting ones.

4

Why are interactive apps special?

The challenge:

provide low latency, adequate battery lifetimes

in a turbulent, resource-poor environment.

The opportunity:

degrading the result is acceptable

since users are tolerant sinks of output.

render 3-D objects at lower resolutions for augmented reality. simplify search query, or search a subset of the entire

space. use a smaller language model for speech recognition.

5

Multi-fidelity computation

Traditional algorithms have a fixed output specification.

Resource consumption is the dependent variable.

Multi-fidelity computations can have a range of outputs.

Along with the output, the fidelity is indicated.

The computation provides knobs to set the fidelity.

Lets us bound resource usage by varying fidelity.

Dependence of variables is reversed.

6

Thesis statement

Multi-fidelity computation is a viable approach to supporting mobile interactive applications.

There are interesting multi-fidelity, mobile,interactive apps.

Multi-fidelity computation improves their usability.

I plan to build multi-fidelity support in Odyssey.

Design a programming model for multi-fidelity apps.

Support the programming model in Odyssey.

Demonstrate a small number of proof-of-concept apps.

Evaluate the system.

7

Outline

Thesis statement

Fidelity and resources

Programming model

Research plan

Related work

Contributions

8

What is fidelity?

Fidelity of a computational result:

extent to which it matches a reference result.

Multi-dimensional fidelity:

A result might have multiple fidelity metrics.

A multi-fidelity computation provides knobs to set the fidelity metrics.

At each execution, the system has to set these knobs.

9

Example of fidelity metrics

Radiosity colors a 3-D model prior to rendering(might use in an augmented reality app).

Two fidelity metrics: resolution (fraction of polygons retained)

continuous, from 0 to 1

choice of algorithm (hierarchical/progressive)discrete, with two values

At each execution, system picks the fidelity

(e.g., resolution=0.4, method=progessive).

10

Radiosity application (unmodified)

11

Multi-fidelity radiosity app

12

Multi-fidelity radiosity app

13

Multi-fidelity radiosity app

14

Multi-fidelity radiosity app

15

Fidelity and resources

Setting the knobs affects the fidelity of the result;

it also affects the resource consumption.

The resources of interest to the user are:

Latency (user wait time) Battery energy User attention Money

16

What is a resource?

“Resource” is an abstraction for anything required by a computation from its environment .

Fidelity LatencyUser attentionBattery energyMoney CPUMemoryNetworkDisk bwCache state

Resources

Qualitymetrics

Uservisible

Userinvisible

17

Cache state as a resource

Caching remote objects key to avoiding network access.

Cache (or cache state) is a critical resource: the computation makes demands on it. the system provides it at varying levels.

but not a resource in the traditional sense: value is not necessarily a number (set of objects). not shared or allocated in the usual way. not the same resource as cache space.

Example: user wants to search a set of documents.

“Better” cache state means better latency (or fidelity).

18

User attention as a resource

A wearable computer mediates user’s reality.

Interactive app make demands on user’s attention.

A multi-fidelity system has an impact on user attention: Different fidelities demand different levels of attention. System solicits user feedback to make good tradeoffs.

19

Resource usage dependencies

Cache state

Memory

Networktransmitreceive

CPUlocalremote

Disk bwFidelityData size

Money

Latency

Battery

User attention

CPU usage is a function of fidelity, input data size

Battery energy and latency are functions of CPU, network, disk usage.

20

Outline

Thesis statement

Fidelity and resources

Programming model

Research plan

Related work

Contributions

21

Interactive computations

Interactive computation:smallest unit of computation visible to the user.

We care about multi-fidelity interactive computations.

Each execution has a set of quality values, and a utility.

For each execution of such a computation, we need to set the fidelity to maximize the utility.

22

Programming model

The application programmer identifies the multi-fidelity interactive computations. inserts hooks for system to set the fidelity. inserts hooks for system to monitor the execution.

The run-time system: selects the fidelity. monitors each execution and its resource consumption. uses the information to improve fidelity selection.

23

Research plan

Build a system to support this scenario:

User asks radiosity app to color some scene. Radiosity app queries system for fidelity. System makes a fidelity selection.

monitors/predicts resources available to app. predict’s app’s resource consumption

(using dependency graph + resource availability). maps user-visible resources + fidelity to user happiness

(using some utility function). searches over the fidelity space to maximize utility.

24

Mapping fidelity to utilityComputation

Fidelity 1

Fidelity 2

…

Input data

System state

CPU availability

Memory availability

Network bandwidth

Disk bandwidth

Cache state

$ cost of network

Energy cost of CPU

Energy cost of network

Energy cost of disk

User’s distraction level

Quality

Fidelity 1

Fidelity 2

….

Latency

User attention

Money

Battery drain

Utility

ResourceconsumptionCPUMemoryNetworkDisk bwCache stateUser attention

Resourceconsumptionfunctions

Resource consumption functions

Utilityfunction

25

Research plan

Develop an API Extend API to support cache state, user attention

Implement support in Odyssey Predicting resource availability Predicting resource consumption Determining the utility function Maximizing the utility function

Demonstrate proof-of-concept applications Radiosity application Augmented or virtual reality Search Engineering computation and/or 3-D visualization

Evaluate the system

26

Multi-fidelity API

On initialization

register_fidelity({f1:continuous[0..1],

f2:discrete[hier,prog]});

hint_resource(CPU, (f2 == hier) ?

1.4 nf1log(nf1) : 2 n2f12);

hint_constraint(LATENCY, 0.05 seconds);

When the user requests a coloring operation

begin_fidelity_op(1000 input data size in polygons);returns {f1=0.4, f2=hier} values of fidelity metrics

… run the radiosity computation …

end_fidelity_op(); signals end of interactive computation

27

Cache state and user attention

To use cache state in making fidelity decisions: monitor cache state, staleness of objects. extend API to allow disclosure of accesses. predict #objects accessed, hit rate from past history.

To measure user attention: assign cost to each aspect of user interface

(windows, menu bars, audio messages). extend interface to allow app hints on user attention. do cost/benefit when soliciting user feedback on utility.

28

Predicting resource consumption (1)

Resource consumption is a function of input data fidelity

How can we know the resource consumption function?

Solution: a combination of application programmer hints learning from a history database

We represent input data by a single feature(typically size).

29

Predicting resource consumption (2)

Cache state

Memory

Networktransmitreceive

CPUlocalremote

Disk bwFidelityData size

Money

Latency

Battery

User attention

Split complex function into multiple simpler functions.

E.g. battery is an application-independent linear function of CPU, network, and disk usage.

30

Learning the utility function

Two ways of learning utility or user happiness: Implicit (outside the scope of this thesis) Explicit

application programmer hints feedback from user

Hints will be in the form of constraints.programmer-specified bounds on important quality metric(s)

Constraints are simplified utility functions.assume utility is a step-function on constrained quality metric

31

Using user feedback

User feedback: slider bars/buttons/preference files to set user constraints query the user when we have a difficult tradeoff decision allow user to interrupt incremental-fidelity operations

More user feedback leads to better tradeoffs.

This is not free: we are consuming user attention.

There is a meta-tradeoff: between goodness of tradeoff and user attention.

32

Finding sweet spots

Constraints are only an approximation:

assumption that utility is a step-function.

For non-real time apps, we can fudge the constraint.

Example: a search app has a 1s latency constraint.

In 1.2s we can search all documents in the cache.

After that, we need to go over the network.

There is a knee, or sweet spot at 1.2s.

I plan to automatically identify sweet spots.

33

Evaluation

Evaluating the idea: How much can we impact resource consumption by adapting

the fidelity? Does this adaptation lead to better usability? How easy/difficult is it to program to the multi-fidelity API?

Evaluating the system: How accurately do we predict resource consumption? How well do fidelity selections maximize utility? What is the overhead of computing fidelity selections? How accurately do we find sweet spots?

34

Timeline

Dec-99 Mar-00 Jun-00 Sep-00 Nov-00 Feb-01 May-01

writing

evaluation

other apps

radiosity app

user attention

cache state

implementation

design

35

Outline

Thesis statement

Fidelity and resources

Programming model

Research plan

Related work

Contributions

36

Related work

Data fidelity Odyssey [Noble97], Dynamic Distillation [Fox96]

Real-time QoS Multi-resource QoS [Lee99] Rialto [Jones95], Darwin [Chandra98].

Multi-fidelity computations in the real world Approximation algorithms, probabilistic algorithms Online aggregation [Hellerstein97] Odyssey speech recognition

User interfaces Coda user patience model [Mummert95] Mixed-initiative user interfaces [Horvitz99]

37

Contributions

Improving the usability of a mobile interactive system

Broad abstraction of multi-fidelity computation

Multi-fidelity API and system support

Support for new resources (user attention, cache state)

Applying learning to fidelity/resource tradeoff decisions

Demonstration of mobile interactive multi-fidelity applications