Timelapse: interactive record/replay for the web

Timelapse Interactive record/replay for the web

Brian Burg Andrew J. Ko Michael D. Ernst

Computer Science and EngineeringUniversity of Washington

Reproducing program behavior is essential in software development

Demonstration:

interacting with a program to achieve a behavior

Investigation:

interacting with developer tools to understand behavior

The phases of demonstrating and investigating are inseparable.

Task coupling can be hazardous

Example: debugging a color picker widget

Barriers to reproducing behavior

Demonstration

• Tedious, time-consuming, and manual

• Difficult to explain repro. steps (e.g., in a bug report)

Investigation

• Repeated demonstration to investigate same behavior

• The behavior being investigated is not held constant

• Interleaving the two phases incurs task switching penalties

Simplifying by decoupling

The means of investigation could change independently without re-demonstration.

Decoupling behavior demonstration from investigation simplifies reproduction and enables new practices.

Behavior could be demonstrated, and later, investigated, by different people.

Behavior could be reproduced precisely, automatically, quickly, and easily.

http://dryicons.com, http://www.getfirebug.com/

Solution: deterministic record/replay

User-space librariesJockey (C), Mugshot (JavaScript)

Application-specificVideo game engines

Operating systemsdOS, Determinator

Virtual machinesReVirt, LEAP, VMWare Workstation

HardwareDMP, RCDC, Calvin

Old idea: achieve identical execution by controlling nondeterminism.

Recent idea: modern browsers are virtual machines.

New idea: adapt virtual machine record/replay techniques to web browsers

Timelapse: record/replay for the web

Contributions

• Adaptation of VM record/replay techniques to the web

• Infrastructure and tool for interactive record/replay

• Design concepts for visualizing, interacting with recordings

Timelapse is an infrastructure and developer tool to record, replay, and investigate web application behavior.

Demonstrating and Investigating(with Timelapse)

Example: color picker, revisited

1. Begin recording2. Demonstrate behavior3. End recording4. Investigate as needed

Recreating a specific execution

To eliminate nondeterminism,

• Identify what is/isn’t deterministic

• Decide what sources to capture, and how

• Figure out how to replay the recorded nondeterminism

Timelapse adapts definitions and techniques from VM research.

Executions differ because of sources of nondeterminism.

Programs and Inputs

main.js

An execution is determined by a program and its inputs.

main.js

function foo(a,b) { return a + b; }

function bar(c) { return c * Math.random(); }

foo(argv[1], argv[2]);

bar(argv[1]); bar(argv[3]);

main.js

Programs and Inputs

main.js

An execution is determined by a program and its inputs.

main.js

function foo(a,b) { return a + b; }

function bar(c) { return c * Math.random(); }

foo(argv[1], argv[2]);

bar(argv[1]); bar(argv[3]);

main.js

Programs and Inputs

Sources of nondeterminism that affect the course of execution are implicit inputs to the program.

function foo(a,b) { return a + b; }

function bar(c) { return c * Math.random(); }

foo(argv[1], argv[2]);

bar(argv[1]); bar(argv[3]);

An execution is determined by a program and its inputs.

main.js

Interpreters and Inputs

>> foo(a,b) { return a + b; }

>> bar(c) { return c *

Math.random(); }

defs.js

>> load(“defs.js”);

>> foo(1,2);

>> foo(3,4);

>> bar(5);

>> bar(5);

REPL session

1. load defs.js

2. CALL foo 1, 2

3. CALL foo 3, 4

4. CALL bar, 5

5. RET Math.random, 0.453863

6. CALL bar, 5

7. RET Math.random, 0.986364

interpreter inputs

The session could be reproduced exactly by reusing inputs.

In an interpreter session, the program is the interpreter itself.

Web browsers are interpreters

Web browsers interpret JavaScript, CSS, HTML, and user input.

Web Interpreter OutputInput

Web browsers are interpreters

Executions can be reproduced by capturing and reusing inputs.

Web Interpreter OutputInput

Inputs Log

Record/replay goals

Precise deterministic record/replay• Must capture and reuse explicit and implicit inputs• Able to detect divergence of replay from recording

Self-contained, sharable recordings• Must not depend on any machine-specific context• Replay does not require external communication

Integration with existing architecture and tools• Record/replay must have low performance overhead• Controllable and compatible with existing developer tools

Record/replay technique: proxying

During normal execution, Date.now() returns the current time.

>> foo(a,b) { return a + b; }

>bar(c) { return “now:”+

Date.now(); }

/* file: Source/wtf/DateMath.h */

inline double jsCurrentTime(){return floor(WTF::currentTimeMS());}

Record/replay technique: proxying

During recording, the return value of Date.now() is saved.

>> foo(a,b) { return a + b; }

>bar(c) { return “now:”+

Date.now(); }

/* file: Source/wtf/DateMath.h */

inline double jsCurrentTime(){return floor(WTF::currentTimeMS());}

Inputs log

Record/replay technique: proxying

On replay, the logged return value of Date.now() is used.

>> foo(a,b) { return a + b; }

>bar(c) { return “now:”+

Date.now(); }

/* file: Source/wtf/DateMath.h */

inline double jsCurrentTime(){return floor(WTF::currentTimeMS());}

Inputs log

Shim: the thing in the middle

Shims are used to implement deterministic record/replay.

The hard part of implementing record/replay is designing and placing shims.

Background: browser architectures

Web interpreters are embeddable and cross-platform.

User input

Callbacks, visual output

Date.now, setTimeou

t

timer fire

Embedders Web Interpreter(WebKit, Gecko)

Platforms

Shim placement in the browser stack

Shims are placed at the API boundaries of the web interpreter.

Embedders Web Interpreter(WebKit, Gecko)

Platforms

Comparison: shim placement

For record/replay, VM only needs shim between OS/hardware

Why not place shims elsewhere?

At the OS or VM level (VMWare Replay)

Low overhead and exact replay

Can’t integrate with developer tools

At the JS library level (Mugshot: Mickens et al., 2010)

Cross-platform

Slower, lower fidelity, can’t integrate with tools

At the level of DOM events only (WaRR: Adrica et al., 2011)

Most DOM events are deterministic and/or have no effect

Skipping pre-dispatch browser code causes strange behavior

Interpreter inputs by source

User: mouse, keyboard, scroll, resizeNetwork: images, scripts, HTML, AJAXCommands: navigate, open, close page

Internal nondeterminism:Animations, transitions, multimedia,async. actions (load, error, popstate)

Functions: Date.now, Math.randomCaching: resources, cookiesTimers: timer interrupt schedule

“Pull” vs. “Push” inputsPUSH

TOmouse, keyboard, scroll, navigate, network events

resource data,client API hooks, input handling

PULL FROM

Animation, transition, load, error, media events

timer callbacks timestamp, random, cookies, persistent state

interruptsRTDSC, etc

Comparison: “Pull” vs. “Push”

Mechanics of execution replay

Injecting “push” inputs causes dispatch of DOM event(s).

1. mousedown

2. mouseup

3. click

4. focus

5. submit

DOM Event DIspatch

function onClicked(..) {

... = Date.now();

}

JavaScript execution

Comparison: replay timing

Problem: when to inject “push” inputs to cause same execution

VMs and Timelapse share notions of counting and injecting.

• Count branches, instructions

• Inject hardware interrupts

VM record/replay

• Count DOM event dispatches

• Inject “push” inputs

Browser record/replay

Challenge: finding and taming nondeterminism

Taming nondeterminism is easier with principled design.

Networked games like Starcraft 2 rely on record/replay facilities.

Video games explicitly design to a record/replay-like architecture.

Taming nondeterminism with shims relies on well-defined APIs.

Taming nondeterminism post-hoc requires refactoring and hacks.

Challenge: dealing with real web platforms

http://hawanja.deviantart.com/

• >1 MLOC (mainly C++/ JavaScript)

• Hundreds of committers, and 500-1000 commits per week

• 6 independent build systems

• Supports many OS, CPU, GUI toolkits, graphics engines, toggleable features, embedders

Web interpreters are complex, many-limbed, and fast-changing.

Lesson: consider many, implement one

http://commons.wikimedia.org

It’s fine to choose one reasonable configuration. But, don’t ignore essential complexity.

Platform-specific examples:• input method editor handling

• rendering and animation

• resource caching, cookies

Embedder-specific examples:• JavaScript engine

• Embedding API usage

For research, only address essential architectural complexity.

Interacting with a recording

The Omniscient Debugger for JavaControlling video on YouTube

Interacting with Whyline [Ko & Myers]

Whyline for Java: program output Whyline for Java: program output

Interacting with Timelapse

Timelapse Panel inside Web Inspector Application being replayed

Timeline visualization

InfoVis mantra: overview, zoom and filter, details-on-demand.

Filter inputs by

type

Program inputs over

time, by type

Time ruler

Timeline visualization

InfoVis mantra: overview, zoom and filter, details-on-demand.

Filter inputs by time span

Table of program inputs

Input types by

color

Inputs by id and time

Table is adjustable

Global record

controls

Panel-specific controls

InfoVis mantra: overview, zoom and filter, details-on-demand.

Recording, replaying, and modality

• Recording can be started/stopped from any Inspector panel

• Replay is controlled from the Timelapse panel only

• Modality displayed next to record/replay controls

Slider control and modality• Timeline and table are linked by sliders and filters.

• Dragging slider seeks replay to the dropped position.

Sliders show same position Dragging induces drop previews

How do developers use Timelapse?

Actual use cases? Good affordances? Workflow integration?

Formative user study

• Subjects: 9+3 professional web developers

• Tasks: create a test case, fix a bug

• Time: 45 min/task, + tutorial & exit interview

• How: within-subjects design, Camtasia, Timelapse

Important use cases

Play, watch and pause

Visual binary search

Stepping through inputs

Execution anchoring

Refining repro. steps

Important use cases

Play, watch and pause

Visual binary search

Stepping through inputs

Execution anchoring

Refining repro. steps

Important use cases

Play, watch and pause

Visual binary search

Stepping through inputs

Execution anchoring

Refining repro. steps

Important use cases

Play, watch and pause

Visual binary search

Stepping through inputs

Execution anchoring

Refining repro. steps

Important use cases

Play, watch and pause

Visual binary search

Stepping through inputs

Execution anchoring

Refining repro. steps

Important use cases

Play, watch and pause

Visual binary search

Stepping through inputs

Execution anchoring

Refining repro. steps

Study Results

No significant effect on task completion time. Why?

• Timelapse didn’t affect types of information sought

• “Program inputs” not related to existing concepts

• New strategies only possible with tight integration

Interactive record/replay mainly* accelerates existing strategies.

Future Work

“Canned” executions can bring research ideas into practice.

Dynamic analysison-demand race detection, code coverage, optimizations

Program synthesiscreate automated tests from recorded interactions

Corpus analysistrain models, test type systems, or evaluate analyses

Future Work

“Canned” executions can bring research ideas into practice.

Dynamic analysison-demand race detection, code coverage, optimizations

Program synthesiscreate automated tests from recorded interactions

Corpus analysistrain models, test type systems, or evaluate analyses

On-demand dynamic analysis

Dynamic analyses generate data at runtime with instrumentation.

Instrumented execution

Generated data

Dynamic analysis

Analysis Output

On-demand dynamic analysis

Record/replay enables on-demand dynamic analysis tools.

Timelapserecording

Normal execution

RECORD

Instrumented execution

REPLAY

Developer tools

Data and analysis

Conclusion

Timelapse decouples demonstration and investigation using ideas from virtual machine record/replay work.

The Timelapse tool supports interactive inspection.

Timelapse’s infrastructure is useful for other purposes.

The Timelapse research prototype is an open-source fork of WebKit, and is freely available from the project website.

https://bitbucket.org/burg/timelapse/

CALL FOR BACKUP

Network: a real proxy, not a shim

This is a hack.

In the short-term, easier than attempting to recreate platform-dependent network streams/handles

Long-term solution:

A resource loader implementation with built-in record/replay primitives, or a higher-level network API

An HTTP proxy records and replays network traffic.

Replay fidelity and completeness

Web interpreters expose a large and ever-changing API.

Timelapse doesn’t tame all sources of nondeterminism.

Excepting untamed sources, the DOM tree and JavaScript heap are identical for all recorded and replayed executions.

Divergence is automatically detected by comparing DOM events.

Divergence detection supports piecewise implementation.

Performance characteristics

Record and replay slowdowns are negligible because shims are placed along existing architectural boundaries.

Timelapse is ideal for CPU- and user-bound programs.

Network-bound applications are slower, due to prototype shortcuts (disabling caching and pipelining).

Replay/seek speed could be improved dramatically by checkpointing and clipping unnecessary layout/renders.

Performance hasn’t been a priority, but is adequate anyway.

Embedding and platform APIs

Embedders

EMBE

DD

ING

API

PLAT

FORM

API

Web Interpreter(WebKit, Gecko)

Abstraction layers separate web interpreters from platforms/embedders.

Platforms

Embedding and platform APIs

Embedders

EMBE

DD

ING

API

PLAT

FORM

API

Web Interpreter(WebKit, Gecko)

Abstraction layers separate web interpreters from platforms/embedders.

Platforms

Embedding and platform APIs

Embedders Web Interpreter(WebKit, Gecko)

Platforms

EMBE

DD

ING

API

PLAT

FORM

API

Shims sit between the web interpreter and abstraction layers.

Embedding and platform APIs

Embedders Web Interpreter(WebKit, Gecko)

Platforms

EMBE

DD

ING

API

PLAT

FORM

API

Shims sit between the web interpreter and abstraction layers.

Shim behavior when recording

[user mouse press]-> (IPC communication)-> WebKit::WebPage::mouseEvent()-> WebCore::InputProxy::handleMousePress()---> WebCore::DeterminismController::capture()-> WebCore::EventHandler::handleMousePress()-> WebCore::Node::dispatchMouseEvent()

EMBE

DD

ING

API

PLAT

FORM

API

Start recording!

Shim behavior when recording

[user mouse press]-> (IPC communication)-> WebKit::WebPage::mouseEvent()-> WebCore::InputProxy::handleMousePress()---> WebCore::DeterminismController::capture()-> WebCore::EventHandler::handleMousePress()-> WebCore::Node::dispatchMouseEvent()

EMBE

DD

ING

API

PLAT

FORM

API

EMBE

DD

ING

API

PLAT

FORM

API

Shim behavior when replaying

DeterminismController::dispatchNextAction()-> MousePressAction::dispatch()-> InputProxy::handleMousePress(true)-> EventHandler::handleMousePress()-> Node::dispatchMouseEvent()-> WebKit::WebPage::mouseEvent()-> InputProxy::handleMousePress(false)

Start replaying!

EMBE

DD

ING

API

PLAT

FORM

API

Shim behavior when replaying

DeterminismController::dispatchNextAction()-> MousePressAction::dispatch()-> InputProxy::handleMousePress(true)-> EventHandler::handleMousePress()-> Node::dispatchMouseEvent()-> WebKit::WebPage::mouseEvent()-> InputProxy::handleMousePress(false)