SmartSynth: Synthesizing Smartphone Automation

Scripts from Natural Language

Vu Le (UC Davis)Sumit Gulwani (MSR Redmond)

Zhendong Su (UC Davis)

motivation

programming

App Inventor Tasker

natural languageWhen I receive a new SMS, if the phone is connected to my car’s bluetooth, it reads the message content and replies the sender “I’m driving.”

scriptwhen (number, content) := MessageReceived()

if (IsConnectedToBTDevice(Car_BT) then

Speak(content);

SendMessage(number, "I'm driving");

SmartScriptScript P ::= I E T C M

Parameter I ::= input (i1, …, in) | ε

Event E ::= (r1, …, rn) := when Event() | ε

Conversions T ::= F1; …; Fn;

Condition C ::= if (Π1 ∧ … ∧ Πn) then | ε

Body M ::= Stmt1; … ; Stmtn;

Conversion F ::= x := Convert(a)

Predicate Π ::= Predicate (a1, …, an)

Statement Stmt ::= S | foreach x in a do S1; … ; Sn; od

Atomic Stmt. S ::= A | F

Action A ::= (r1, …, rn) := Action (a1, …, an)

Argument a ::= x | i | r | l

approach

Script = Components + Relations

approach

Script = Components + Relations

approach

Script = Components + Relations

approach

Script = Components + Relations

definitiono Component

• API: MessageReceived

• Entity• Literal: “I’m driving”

• API return value: MessageReceived.TextO

o Relation = <Entity, API parameter>• E.g., < “I’m driving”, SendMessage.TextI

>

component discovery

Description Component

When I receive a new SMS

MessageReceived

if the phone is connected to

IsConnectedToBTDevice

my car’s bluetooth Car_BT

reads Speak

the message content MessageReceived.TextO

replies SendMessage

the sender MessageReceived.NumberO

“I’m driving” "I'm driving"

mapping featureso Regular expressionso Bag of wordso Phrase lengtho Punctuationo Parse tree (Stanford NLP parser)

relation discoveryo Rule-based relation detection

• Relative locations of APIs and entities

Return Value or Literal API Parameter

Car_BT IsConnectedToBTDevice.TextI

MessageReceived.TextO Speak.TextI

MessageReceived.NumberO

SendMessage.NumberI

“I’m driving” SendMessage.TextI

script generationwhen (number, content) := MessageReceived()

if (IsConnectedToBTDevice(Car_BT) then

Speak(content);

SendMessage(number, "I'm driving");

ambiguityDescription Possible Component

Mappings

When I receive a new SMS

MessageReceived

if the phone is connected to

IsConnectedToBTDevice

IsConnectedToWifiNetwork

IsConnectedToDataService

my car’s bluetooth Car_BT

reads Speak

the message content MessageReceived.TextO

repliesSendMessage

SendEmail

“I’m driving” "I'm driving"

incompletenesswhen (number, content) := MessageReceived()

if (IsConnectedToBTDevice(Car_BT) then

Speak(content);

SendMessage(number, "I'm driving");

insight

NLP + Program Synthesis

ambiguity{ MessagedReceived, IsConnectedToBTDevice, Car_BT,

Speak, MessageReceived.TextO, (SendMessage OR

SendEmail), "I'm driving“ }

Ranking

Script 1 (SendMessage)

SendMessage

SendEmails

Script 1

Script 2

Synthesizer

incompletenesso Search for most likely missing

relations

q & ao Distinguishing multiple choice

questions• Question: API parameter

• Answers: Equally-likely related entitiesWhat do you want the phone to speak?A. The received message contentB. “I’m driving”

architecture

Component Discovery

ScriptDiscovery

Natural LanguageDescription

MappingComponents

IntendedScript

Natural LanguageQ&A (Optional)

1

2

3

5

User

Feedback onDescription (Optional)

6

Feedback onComp. Mapping

4

evaluationo 50 scripts collected from user forumso User study: give NL descriptions from

scripts• 11 students participated

• Each student was given 25 problems

• 640 correct descriptions (725 total)

component mapping

[1] [1] [2] [1] [2] [3] [1][2][3][4] SmartSynth0

10

20

30

40

50

60

70

80

90

100

Series1[1] Regex + Bags-of-Words [2] Phrase length[3] Punctuation [4] Parse tree

relation detection

1 2 3 4 7 80

1

2

3

4

5

# Relations

# D

etec

ted

Rela

tions

completing ratio

0 1 2 3 4 7 80

50

100

150

200

250

Detected by NLP Engine Completed by Synthesis Engine

# Relations

# D

escr

iptio

ns

synthesis time

2 3 4 5 6 7 8 10 11 120

1

2

3

4

5

6

7

Parsing time Total time

# Components

Tim

e (s

)

related worko General purpose programming using

natural languageo Natural language interfaceso Specification extraction from natural

language

thank you

Script = Components + Relations

NLP

Program Synthesis

Identify

Detect

Resolve

Complete

v.s. Siri

Conversation

Single phone API

Composition of APIs

Siri

SmartSynth

v.s. Tasker

TaskerSmartSynth

Time

Prod

uctiv

ity