A Graphical User Interface for the Hybrid Control of Multi ...1249463/FULLTEXT01.pdf · Medan en agent utfor sin r¨ orelseplan, kan¨ andringar¨ i det viktat ﬁnita overg¨ angssystemet

IN DEGREE PROJECT ELECTRICAL ENGINEERING,SECOND CYCLE, 30 CREDITS

, STOCKHOLM SWEDEN 2018

A Graphical User Interface for the Hybrid Control of Multi-Robot Systems under Linear Temporal Logic

LUKAS SCHÖNBÄCHLER

KTH ROYAL INSTITUTE OF TECHNOLOGYSCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE

A Graphical User Interface forthe Hybrid Control ofMulti-Robot Systems underLinear Temporal Logic

LUKAS SCHONBACHLER

Master in Electrical EngineeringDate: 24th July 2018Supervisor: Lars Lindemann, Meng Guo, Margarita ChliExaminer: Dimos DimarogonasSwedish title: Ett grafiskt anvandargranssnitt for hybridstyrning avflerrobotsystem under linjar temporal logikSchool of Electrical Engineering and Computer Science

Abstract

In recent years the model checking theory was applied successfullyfor robot motion and task planning by specifying the workspace asa weighted finite transition system and tasks as linear temporal logicformulas. The tasks contain hard and soft-specifications which are as-signed to an agent in a multi-robot system. The tasks are assigned aslocal tasks to an agent and are independent from each other.

In this work a Graphical User Interface is presented to define a finitetransition system and specify tasks for each agent as linear temporallogic formulas. Furthermore, the presented GUI allows the user tosynthesize the robot motion and task plan. While an agent executesits motion plan, changes in the weighted finite transition system canbe applied and the agent performs a replanning. A new frameworkfor online task adaption is provided, which allows the user to sendtemporary tasks to an agent while executing its motion plan. In theend of the thesis some case studies are shown to demonstrate the newapproaches in simulation and also the functionality of the GUI in anexperiment. In the experiment the agents are able to avoid other agentsand even avoid moving human beings in the workspace.

iii

Sammanfattning

Under de senaste aren tillampades modellkontrollteori framgangsriktfor robotrorelse och uppdragsplanering genom att ange arbetsrum somett viktat finita overgangssystem och uppdragen som linjara tidsbestamdlogikformler. Uppdraget innehaller harda och mjuka specifikationersom tilldelas en agent i ett flerrobotsystem. Uppdragen ar tilldeladesom lokala uppdragen till en agent och ar oberoende av varandra.

I detta arbete presenteras ett grafiskt anvandargranssnitt for ett de-finierat finita overgangssystem och specificerar uppdrag for varje agensom linjara tidsbestamd logikformler. Dessutom tillater den presente-rade anvandargranssnitt anvandaren att syntetisera robotrorelsen ochuppdragplanering. Medan en agent utfor sin rorelseplan, kan andringari det viktat finita overgangssystemet appliceras och agent planerar omsin rorelse. Ett nytt ramverk for online uppdrag anpassning ar given,vilket gor det mojligt for anvandare att skicka tillfalliga uppdraget tillagent under tillfallen da agenten utfor sin rorelseplan. I slutet av rap-porten, visas nagra fallstudier som visar de nya metoderna i simule-ringen och aven funktionaliteten hos anvandaravsnitt i ett experiment.Det visar sig att agenterna undvika andra medel och till och med und-vika manniskor under rorelse i arbetsytan.

v

Acknowledgements

First of all, I would like to thank my supervisor Lars Lindemann. Yousupervised me in a competent way and I could profit from your exper-tise. Your guidance and the discussions with you helped me to solveseveral problems during my thesis work.

Furthermore, I would like to thank my examiner Dimos Dimarog-onas and Meng Guo to give me the opportunity to come to KTH andwrite my thesis here.

I also want to express my gratitude to my colleagues in the SmartMobility Lab. Thanks to Stefano Mariuzzi, Umar Chughtai, Kuba Nowak,Yu Wang, Xiao Chang and Frank Jiang for your companion and havea good time over the lunch breaks. Special thanks to Pedro Roque forhelping with software and hardware problems during the thesis workand the inspiring discussions.

Great thanks to my friends from the exchange program. AndreuSalcedo Bosch, Torbjørn Slinde and Ludovic Coelho, I appreciated thetime with you discovering Stockholm. Thanks to Alex Lamprecht,Suzanne Le Bihan and Olivier Salvas for planning amazing trips aroundStockholm.

Thanks to my friends back home in Switzerland to come to Stock-holm in their holidays and spending a few days with me.

Last but not least, I want to thank my parents for supporting meduring my studies and helping me making important decisions in mylife.

vii

Contents

Abstract iii

Sammanfattning v

Acknowledgements vii

Symbols x

1 Introduction 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.2 Problem formulation . . . . . . . . . . . . . . . . . . . . . 51.3 Literature review . . . . . . . . . . . . . . . . . . . . . . . 5

2 Notation and Preliminaries 92.1 Coordinate frame transformation . . . . . . . . . . . . . . 92.2 Map coordinate frame . . . . . . . . . . . . . . . . . . . . 12

3 Robot Motion Planning 153.1 Specification level . . . . . . . . . . . . . . . . . . . . . . . 17

3.1.1 Discretized workspace model and Motion abstrac-tion . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.1.2 Task specification as LTL-formulas . . . . . . . . . 193.1.3 Weighted product Buchi automaton . . . . . . . . 21

3.2 Execution level . . . . . . . . . . . . . . . . . . . . . . . . 23

4 Online task adaption 25

5 Workspace adaption 355.1 Knowledge update . . . . . . . . . . . . . . . . . . . . . . 355.2 Revision and Validation . . . . . . . . . . . . . . . . . . . 37

viii

CONTENTS ix

6 Graphical User Interface for Robot Motion Planning 416.1 World selection . . . . . . . . . . . . . . . . . . . . . . . . 446.2 Select Finite transition system . . . . . . . . . . . . . . . . 45

6.2.1 Choose pose of interest . . . . . . . . . . . . . . . . 456.2.2 Select transitions . . . . . . . . . . . . . . . . . . . 476.2.3 Define general atomic propositions . . . . . . . . . 486.2.4 Additional buttons . . . . . . . . . . . . . . . . . . 50

6.3 Define agent type and task specifications . . . . . . . . . 516.3.1 Agent types . . . . . . . . . . . . . . . . . . . . . . 516.3.2 Robot name . . . . . . . . . . . . . . . . . . . . . . 526.3.3 Initial pose . . . . . . . . . . . . . . . . . . . . . . . 526.3.4 Localization and Navigation . . . . . . . . . . . . 526.3.5 Task specification . . . . . . . . . . . . . . . . . . . 55

6.4 Synthesize plan . . . . . . . . . . . . . . . . . . . . . . . . 556.5 Setup simulation or experiment . . . . . . . . . . . . . . . 56

6.5.1 Simulation . . . . . . . . . . . . . . . . . . . . . . . 566.5.2 Experiment . . . . . . . . . . . . . . . . . . . . . . 57

6.6 Execution and runtime modifications . . . . . . . . . . . . 586.6.1 Temporary task . . . . . . . . . . . . . . . . . . . . 616.6.2 Workspace update . . . . . . . . . . . . . . . . . . 61

7 Case studies 637.1 Hotel scenario: Temporary tasks . . . . . . . . . . . . . . 637.2 Hotel scenario: Temporary task with workspace adaption 697.3 Experiment: Multiple robots with collision avoidance . . 71

8 Conclusion 75

9 Future work 77

Bibliography 82

Symbols

Symbols

a atomic propositionAφ non-deterministic Buchi automatonAφ relaxed non-deterministic Buchi automatonAp weighted product Buchi automatonC set of all temporary task sequencesE edge among two poses of interestF accepting states in AφF ′ accepting states in Apg generalG target setl input alphabetL labeling functionL accepted languageK number of propositions in a temporary LTL-formulaK set of agentsM number of temporary tasksM matrixN number of poses of interestp pixel coordinate vector, pure imaginary quaternionP poseq state in Aφq′ state in ApQ set of states in Aφ

x

CONTENTS xi

Q′ set of states in Apq unit quaternionr coordinate vectorR run in Apres resolution [meters/pixel]

S set of true atomic propositionst time [s]

T coordinate frame transformation, time period, time constantu, v pixel coordinatesT weighted finite transition systemW weight functionW workspacex, y, z coordinatesα design parameterγ relative weighting among prefix and suffixΓ set of final proposition of temporary tasksδ transition relation in Aφδ′ transition relation in Apφ LTL-formulaφ temporary taskΨ set of temporary LTL-formulasσ infinite word over the alphabet 2AP as infinite sequenceπ pose of interestΠ set of poses of interestτ path as infinite sequenceω ω-operatorχ set of input alphabetsΞ set of invalid transitionsℵ set of unsafe transitionsΥ sum of changed statesζ set of poses of interest2AP alphabet→ set of transition relations

xii CONTENTS

∧ and¬ not◦ next∪ until� always� eventually∨ disjunction⇒ implication

Indices

A,B, C coordinate framesc weighted finite transition system, combinationd desiredF footprint coordinate frameinit initialI image coordinate framem map coordinate frame in image planeM map coordinate frame in three dimensional spacek k-th agent, k-th propositionO odometry coordinate framep pose, product automatonpre, suf prefix, suffixprop propositionQ Qualisys coordinate framerem remainingtemp temporarytrans transitionu u axisv v axisx x axisy y axis

CONTENTS xiii

z z axis

Acronyms and Abbreviations

AP Atomic PropositionGUI Graphical User InterfaceKTH Royal Institute of TechnologyLTL Linear Temporal LogicNBA Non-deterministic Buchi AutomatonROS Robotic Operation SystemSML Smart Mobility LabUAVs Unmanned Aerial VehiclesUGVs Unmanned Ground VehicleswFTS Weighted Finite Transition System

Chapter 1

Introduction

The research in autonomous robots, such as unmanned ground ve-hicles (UGVs), unmanned aerial vehicles (UAVs) and service robotshas been in high interest recently. Essentially the autonomous robotsshould comprehend daily tasks specified by non-expert users, figureout a plan and finally execute the plan to accomplish the given taskswithout or at least with minimal human intervention. A task couldbe a simple delivery tasks: ”Always if requested, pick up a package froma dispatch station and deliver it to goal destination; otherwise stay at dock-ing station”. An example is the Amazon’s prime air delivery service1 shown in Figure 1.1. Another simple task could be a cleaning task:”Always clean all rooms, if anyone is at home”. This task can be done byexample by autonomous vacuum cleaners as the Roomba 2, shown inFigure 1.2. More developed robots could do a service tasks as: ”Robotgo to the kitchen, make me a coffee and bring it back to me”. Furthermore,it is preferable if an agent is able to react to changes in its environmentor change the task specifications on-the-fly.

In addition, wireless communication technologies enable autonomousrobots to communicate with each other and exchange information abouttheir operation space. Given such a communication, the autonomousrobots are more accurate up-to-date about changes in their environ-ment or can send requests between the autonomous systems to collab-orate witch each other. A group of coordinated autonomous robots can

1Amazon webpage: https://www.amazon.com/Amazon-Prime-Air/b?ie=UTF8&node=8037720011

2Project webpage: http://www.irobot.com/For-the-Home/Vacuuming/Roomba.aspx

1

https://www.amazon.com/Amazon-Prime-Air/b?ie=UTF8&node=8037720011

https://www.amazon.com/Amazon-Prime-Air/b?ie=UTF8&node=8037720011

http://www.irobot.com/For-the-Home/Vacuuming/Roomba.aspx

http://www.irobot.com/For-the-Home/Vacuuming/Roomba.aspx

2 CHAPTER 1. INTRODUCTION

Figure 1.1: The autonomous delivery drone amazon Prime-air.

Figure 1.2: The autonomous roomba vacuum cleaner.

CHAPTER 1. INTRODUCTION 3

Figure 1.3: Waymo the current autonomous car project from Google.

Figure 1.4: An autonomous drone from ULC recording data of thefield.

achieve more complex tasks and can be more efficient than a group ofindependent single robots. For instance, multiple tasks must be exe-cuted at the same time in different regions or a group of robots is ableto carry an heavy object together which can not be carried by one of thesingle robots alone. Designing a suitable coordination scheme that canimprove the overall performance of multi-robot systems while keep-ing the cost and coordination low is one of the core problems. In theremainder of this chapter some motivating application are providedand related work is discussed.


1.1 Motivation

Car manufacturers around the world show interest in developing au-tonomous cars or upgrade existing models with assistance technol-ogy (automatic parking, collision avoiding, adaptive cruise control,etc). The first prototypes of self-driven cars already exist and has beentested, e.g. the actual project from Google named Waymo 3 picturedin Figure 1.3. May the majority of autonomous vehicles could lead toan extremely efficient transportation system that everybody could useby giving driving tasks in high-level language, even if the person doesnot have the ability to drive. A given task could be: ”Park the car asclose as possible to the next supermarket of this brand, but on the way passby the hospital and avoid highways”. This tasks are more complex thansimply ”going from A to B” and have temporal characteristics for lo-cations and properties of interest attached (”next supermarket of thisbrand”). Therefore, the planner needs an extra level of logic reasoningto find the high-level plan as locations to visit.

UAVs were used to perform some simple delivery tasks or for in-spection in agriculture, e.g. ULC 4 uses autonomous drones to obtaindata from crop and vineyard fields to inform the agriculture managerabout the health of their plants. In Figure 1.4 the autonomous drone isshown recording data of the field. Nowadays the drones can fly with-out human interactions to control the motors, but still the input forgoal destinations and routes are needed. An example for an inspec-tion task could be: ”Surveil locations A, B, C and D. In case of detecting atarget notify the control station”.

The last kind of autonomous robots to mention here are servicerobots. The goal of a service robot is to do daily tasks of a human be-ing. This could be making coffee, vacuum cleaning or cooking in thehousehold. Now imagine to use the different kinds of autonomousrobots at the same time in collaboration. Even more complex taskscould be achieved, since with a variety of autonomous robots in col-laboration the advantages of each kind could be exploited. The UAVscould perform an inspection task and if an UAV finds a target, the in-formation could be send to a autonomous ground vehicle to reach thetarget to perform another task.

3Project webpage: https://waymo.com/4Project webpage: http://ulcrobotics.com/uav-agricultural-

inspection-services/

https://waymo.com/

http://ulcrobotics.com/uav-agricultural-inspection-services/

http://ulcrobotics.com/uav-agricultural-inspection-services/


1.2 Problem formulation

Assume we have a given known workspace. The workspace is givenas map, but the weighted finite transition system is not defined withinthe workspace. In this workspace a multi-agent system should executea task given as LTL-formulas. We assume the user knows about robotmotion planning under LTL-specifications, but does only have sparseknowledge about the actual planner software.

Problem 1.2.1. Include planner software in a Graphical User Interface,such that different scenarios can be generated easily and fast, withoutdeep knowledge of the actual software.

Another burning issue is the plan adaption while an agent is exe-cuting its plan. Therefore, an agent should be able to adapt its planduring runtime when it receives a new temporary task.

Problem 1.2.2. Given an agent executing its current run Rt and receiv-ing a new temporary task, adapt the current planRt such that all speci-fications are fulfilled and the agent can accomplish the new temporarytask.

1.3 Literature review

In [1], [2] a given workspace is decomposed into triangular cells togenerate a continuous trajectory from the initial cell to the goal. Forgenerating such a trajectory linear temporal logic (LTL) formulas areused to specify the tasks of an agent. Linear temporal logic is a lan-guage that provides a useful set of operators for defining tasks. Theintroduced tasks are then sending an agent to certain cells or avoid-ing cells (obstacles). In comparison to our work, no decomposition ofthe whole workspace is used, we predefine poses of interests in ourworkspace. The model checking approach with a product automa-ton between Buchi automaton and finite transition system (FTS) is al-ready used in [2], where the Buchi automaton is the conversion of LTL-formulas. A FTS defines which transitions an agent in the workspaceis able to perform and with the LTL-formulas the task specificationsare given. The first approach for LTL planning for groups of robots isgiven in [3]. However, the agents are controlled centralized and thetask is global, whereas our approach aims for decentralized robots.


The communication between robots for periodic synchronization anda fault recovery modeled in LTL for multiple agents are introduced in[4]. In [5] multiple agents solve a global task, i.e. in their providedsimulation three agents reach simultaneously three different goal re-gions. Although the task is given centralized, each robot executesits own part of the global task decentralized. However, synchronousmovements to the next region is needed, given by communication be-tween the agents, whereas we avoid synchronous movements. Forpartially known workspace [6], [7] and [8] used two-player GeneralReactivity GR(1) game between robot and the environment to updatethe workspace and synthesize a plan. In addition, in [8] introducesreceding horizon based framework to reduce the computational com-plexity. A weighted FTS (wFTS) is used in [9] and [10] to find theoptimal plan obtained by using an adapted Djikstra’s algorithm andintroducing the prefix and suffix plan structure, whereas our workwill also build up on the idea of a weighted FTS. The advantage ofa wFTS is that the cost of transitions are defined, which allows to ap-ply graph search algorithms to find an optimal path. A robust optimalpath planning for a multiple robot systems with asynchronous motionis presented in [11]. For partially known workspace [12] algorithms areprovided to update the workspace and revise the current plan online.In [13] a method is described to synthesize a motion plan that fulfillsan infeasible task the most and how the infeasible specifications arerelaxed by using a relaxed product automaton. A robot model com-bined of robot actions map and the robot mobility is introduced in[14] to synthesize a motion plan including navigation and performingactions. Nevertheless performing actions is an essential part of eachagent, in this thesis the focus is set on the movements among differentregions and adding actions to the framework is left for future work.In [15] a knowledge transfer scheme is introduced between multipleagents for distributed plan reconfiguration while using soft- and hardLTL-specifications. A scheme for updating the product automaton andreconfigure the plan in an partially known workspace by gathering in-formation from environment or other agents is shown in [16] on whichthis thesis is mainly built-on. A decomposition of finite LTL missionspecifications into independent tasks is shown in [17]. This allows todecompose a global task for multiple agents in independent local tasksfor each agent. Moreover, resource constraints are considered in [18]as constraint for finding a robot motion plan. In [19] a Human-in-the-


Loop mixed initiative control is presented and new short-term tasksassigned to the agent during run time is introduced. The second partof the thesis is build on the idea of this new introduced short-termtasks. The short-term task in [19] is a simple sequence of two regions ofinterest, whereas in this thesis we extend this approach to a sequencewith more than only two atomic propositions and with multiple tem-porary tasks at the same time.

Chapter 2

Notation and Preliminaries

In this chapter the used notation through this thesis is introduced.Furthermore, the coordinate frame transformation and the most im-portant characteristics of unit quaternions used in this thesis are intro-duced. While we assume that the reader is familiar with unit quater-nions and its basic characteristics, readers interested in a full introduc-tion to quaternions are referred to [20].

2.1 Coordinate frame transformation

Describing positions in R3 we use coordinate vectors r. A vector inthis thesis is denoted with bold letters (e.g. r).

Definition 2.1.1. A coordination vector Ar ∈ R3 is defined as:

Ar =

xyz

, (2.1)

where A is the coordinate frame and x, y and z are the coordinates.

The orientation is denoted by unit quaternions. A unit quaternion is anon-minimal representation of rotations.

Definition 2.1.2. A unit quaternion is defined as:

Aq =

qωqxqyqz

, (2.2)

9

10 CHAPTER 2. NOTATION AND PRELIMINARIES

whereA is the coordinate frame, qω is the real part and qv = qxi+qyj+

qzk = (qx, qy, qz) is the imaginary part.

Combining the position and orientation we get a pose P or a trans-formation T among two coordinate frames.

Definition 2.1.3. A pose P is then given as a tuple: AP = (Ar, Aq),where A is the coordinate frame, Ar the position and Aq the orienta-tion.

Definition 2.1.4. A transformation T among two coordinate frames Aand B is denoted as tuple: ATB = (ArB, AqB), where ArB is the transla-tion from frame A to frame B and AqB is the rotation from frame A toframe B.

Rotate an orientation Aq given in frame A to frame B is done byspecial algebra for quaternion multiplication denoted as ⊗. The orien-tation Aq in frame B is then given by eq. (2.3)[20].

Bq = BqA ⊗ Aq (2.3)

Using special algebra quaternions can be directly used to rotate vec-tors. Therefore, a coordinate vector Ar is expressed as pure imaginaryquaternion p(Ar).

Definition 2.1.5. The pure imaginary quaternion p(Ar) is defined as:

p(Ar) =

[0

Ar

], (2.4)

where Ar is the coordinate vector in frame A.

Given a rotation between two coordinate frames as unit quaternionCqB, it is possible to rotate p(Br) from frame B to frame C, which resultsin p(Cr)[20].

p(Cr) = BqC ⊗ p(Br) ⊗ Bq>C , (2.5)

where the coordinate frame B and C have the same origin but a differ-ent orientation.

Finally a position Br given in coordinate frame B is transformed inframe Awith the given transformation ATB derived from eq. (2.3) andeq. (2.5) given through eq. (2.6).

p(Ar) = p(ArB) + Aq>B ⊗ p(Br) ⊗ AqB (2.6)

CHAPTER 2. NOTATION AND PRELIMINARIES 11

Figure 2.1: Blue: Coordinate frame A; Red: Coordinate frame B;Green: Coordinate frame C; P the position of a point. A and C havethe same orientation and B and C have the same position.

Example 2.1.1. Let us consider a simple example as shown in Fig-ure 2.1. Known are the transformation ATB and the coordinate vectorp(Br) which is the position of point P , expressed in coordinate frameB. We want to know the position of P in frame A. First the vectorp(Br) is transformed by eq. (2.5) in coordinate frame C which has thesame orientation as frameA (CqB = AqB) and the same origin as frameB (ArB = ArC). Then the sum of p(ArB) and p(Cr) represents p(Ar).

Using eq. (2.3) and eq. (2.6) a Pose BP in frame B can be expressedin frameA as AP through transformation ATB. Although there are alsoother methods to transform poses in different coordinate frames (e.g.homogeneous transformation matrix), we transform them directly us-ing unit quaternion, since the pose measurements of our robots aregiven as tuple with coordinate vector and unit quaternion for orienta-tion. In this way computing the rotations matrix from the quaternionis not necessary and we can represent the pose always in the sametype.

12 CHAPTER 2. NOTATION AND PRELIMINARIES

2.2 Map coordinate frame

Maps are given as images and a map has a resolution res inmeters/pixel.Through the resolution we can compute, what a corresponding distanceof a pixel in the real world is. Since maps are given as images, we de-fine the pixel coordinates in a image.

Definition 2.2.1. Pixel coordinates Ip ∈ R2 are defined as:

Ip =

[Iu

Iv

], (2.7)

where I indicates the image coordinate frame and u are the horizon-tal coordinates and v respectively the vertical coordinates. The originof the image coordinate frame I is in the top left corner of the image.However, the origin of the map frame m in the image might not belocated in the top left corner in the image. Therefore, a transforma-tion among image and map frame is needed. The transformation fromimage frame to map frame is then done by eq. (2.8).

mp =

[mu

mv

]=

[Iu − Ium

Ivm − Iv

], (2.8)

where m is the map frame in the image and Ium, Ivm are the coor-dinates of the origin of the map coordinate frame in the image. Weassume that coordinate axis of I and m are parallel and v-axis are re-versed. A simple example of mapped rectangular room is shown inFigure 2.2.

Since we want to know the corresponding coordinates of a pixel inthree dimensional space, we do the transformation from map frame min image plane to the map coordinate frame M in three dimensionalspace by

Mr =

mu·mv·0

· res, (2.9)

whereM is the map frame in three dimensional space, mu and mv thepixel coordinates of the map frame m in two-dimensional space andres the resolution from pixels to meters. The x-axis ofM and u-axis ofm are collinear and also the y-axis and v-axis are collinear.

CHAPTER 2. NOTATION AND PRELIMINARIES 13

Figure 2.2: A simple map of a rectangular room. In blue the imagecoordinate frame I in the top left corner of the image and in green themap coordinate frame m.

Chapter 3

Robot Motion Planning

The robot motion planning can be split up in three process-levels [21].On the first level, the workspace is partitioned and task specificationsare defined, which results in a directed graph. This first level is alsocalled the specification level. Applying graph search algorithms, pos-sible discrete motion plans can be found which satisfy all specifica-tions. A found discrete plan is then carried out on the execution level,which is the second level. Through generating control laws from theexecution-level, the implementation level is reached which outputs theactual controller implementation for agents. While the agent is execut-ing its plan, all three levels are updated through user inputs or sensing.In this thesis the focus is on specification and execution level.

In this chapter the motion and task planning framework for mul-tiple mobile robot systems is introduced. First the abstraction of theworkspace and its weighted finite transition system (wFTS) is constructed.Secondly it is derived how the tasks are specified as temporal logic for-mulas over propositions that are related to the transition system. Thetemporal logic formulas are transformed in a non-deterministic Buchiautomaton (NBA) and the product automaton between wFTS and NBAis formed. This is the specification level. Then a discrete motion plan issynthesized through a graph search algorithm, where the plan satisfiesgiven tasks and minimizes a total cost. This is called the execution level.For completeness and better understanding of the thesis, some slightlyadapted definitions and algorithms are introduced which mostly arebased on the work of [16]. Moreover, a simple example will lead thereader through following sections.

15

16 CHAPTER 3. ROBOT MOTION PLANNING

Figure 3.1: (a) Hierarchical abstraction and computation architecture.Temporal logic formulas over the given workspace in the specificationlevel. Execution of a discrete plan in execution level and in implemen-tation level the control strategy for each agent is given. (b) Team ofrobots moving a in an environment with obstacles and targets whilecommunicating with each other. Figure from [21].

CHAPTER 3. ROBOT MOTION PLANNING 17

3.1 Specification level

3.1.1 Discretized workspace model and Motion abstrac-tion

Each robot’s motion in the given workspace is abstracted as a finitetransition system [22]. The given workspace is a 3-dimensional space,denoted byW0 ⊂ R3. Within this workspace there existN t poses of in-terest πi ⊂ W0,∀i = 1, 2, · · · , N t. The superscript t denotes the currenttime, since in chapter 4 and chapter 5 adaption during runtime are per-formed. The whole set of poses is then denoted by Π t = {π1, · · · , πNt}.Poses of interest in the workspace are goal poses for an agent, whereit might can perform a special task, e.g. by reaching a pose of interestthe agent is able to pick up an object or deliver an object to a cos-tumer. On the other hand poses of interest can be used to lead anagent through a room with multiple poses as way points for the navi-gation path planner. To express the robot’s state we use atomic propo-sitions (AP ), which are boolean variables that can be either True orFalse denoted by >, respectively ⊥. With the set of atomic proposi-tions APp = {ap,i}, i = 1, 2, · · · , N t, where

ap,i =

{> if the robot is at pose πi,⊥ otherwise,

(3.1)

is stated > if an agent reached a pose, where the evaluation is doneby the robots positioning system or external through a motion capturesystem. The subscript p in ap,i stands for pose. More general proposi-tions are given in the setAPg = {ag,1, ag,2, · · · , ag,M}, where g stands forgeneral and all atomic propositions are then in the set AP = APp∪APg.A more general proposition could for example be ”corridor” and is as-signed to all poses of interest that are in a corridor room. Another ex-ample as general proposition could be ”ball” and is assigned to posesof interest where a ball can be picked up.

Definition 3.1.1. The labeling function Lt : Π t → 2AP maps posesπi ∈ Π t to the set of atomic propositions satisfied by πi and ap,i ∈ Lt(πi)by default, ∀i = 1, . . . , N t.

The robot motion is then abstracted as transition among the theposes Π t = {π1, . . . , πNt}. A transition from πi to πj exists if a navi-gation controller exists, that is able to drive the robot from pose πi to


pose πj in finite time. Given the abstraction of the workspace and thetransition relation we can define a weighted finite state transition system(wFTS) similar to [16] as:

Definition 3.1.2. The weighted FTS is a tuple T tc = (Π t,→tc, Π0, AP, L

tc,W

tc ),

where Π t = {π1, . . . , πNt} is the set of states;→tc⊆ Π t ×Π t is the tran-

sition relation; Π0 ⊆ Π t is the set of initial states to indicate where therobot is starting from; AP is the set of atomic propositions; Ltc : Π t →2AP is the labeling function; W t

c : Π t×Π t → R+ is the weight functionas cost of transition in→t

c.

The weight of a transition is approximated as the Euclidean distancebetween two poses of interest and hence gives a cost that we try tominimize.

In addition we assume for all robots the same transition system andthat T tc does not have a terminal state. A terminal state is a state in thewFTS without outgoing transitions. After reaching a terminal state it isnot possible to leave the terminal state in the future. Since we search insection 3.2 for infinite runs in the wFTS, a terminal state would neverbe in the run and is therefore not useful. We define an infinite path ofT tc as an infinite state sequence τ t = π1π2 . . . such that πi ∈ Π t,∀i =

1, 2, · · ·N t and a transition→tc exists from πi−1 to πi for all i > 0. The

sequence of sets of atomic propositions that are true in the states alongthe path is called trace. The segment of an infinite path that has to berepeated infinitely many times is represented with the ω-operator [22].

Example 3.1.1. In Figure 3.2 a simple wFTS is shown. The workspaceconsists of two rooms, where the two rooms are connected with a corri-dor. The wFTS contains four poses of interest (r1, r2, r3, r4) which areindicated with the arrows. Possible transitions among the poses areillustrated with dashed lines and the initial pose is indicated by thegreen dot. An agent has the possibility to reach r4 from r1 by visitingpose r3 or r2 and vice versa.

Note that the abstraction of workspace is closely related to the re-quirements of the used navigation controllers. In many related works,a full partition of workspace (triangulation ([1],[2], [12], [13]), poly-topes ([3], [23], [6], [24], [7], [5])) is needed to apply the controller suc-cessfully. This might lead to an over-approximation or under-approximationof the workspace. Inspired of ([14], [16]) the navigation controller inour case takes goal poses as input to generate the navigation controller


r1

r3

r4

r2

Figure 3.2: A finite transition system in a workspace with two roomsconnected with a corridor as explained in Example 3.1.1.

among the poses of interest which leads to a smaller transition system.However, by using poses of interest the user needs to define the posesmanually, whereas for the full partition of a workspace already devel-oped algorithms can be applied. For instance for triangulation twoalgorithms are introduced in [25].

3.1.2 Task specification as LTL-formulas

Linear Temporal Logic (LTL) provides a formal way to specify a com-plex task in a way that is expressive enough to specify various types oftasks. An LTL-formula can be constructed by using following syntax:

φ ::= > | a | φ1 ∧ φ2 | ¬φ | ◦ φ | φ1 ∪ φ2, (3.2)

where a ∈ AP and ∧ (and), ¬ (not), ◦ (next), ∪ (until). Other usefuloperators like ∨ (disjunction), � (always), � (eventually),⇒ (implication)can be derived from the basic operators [22].

An infinite word over the alphabet 2AP is an infinite sequence σ ∈(2AP )ω that σ = S0S1S2 . . . , where Sk ∈ 2AP for all k = 1, 2, . . . , whereSk is the set of atomic propositions that are true at time step k. Wecan then define the semantics of LTL for an infinite word σ as relation(σ, k) � φ.


Definition 3.1.3. The semantics of LTL are defined as [22]:

(σ, k) � a ↔ a ∈ Sk(σ, k) � ¬ φ ↔ (σ, k) 2 φ(σ, k) � ◦ φ ↔ (σ, k + 1) � φ

(σ, k) � φ1 ∨ φ2 ↔ (σ, k) � φ1 or (σ, k) � φ2

(σ, k) � φ1 ∪ φ2 ↔ ∃k′ ∈ [k,+∞], (σ, k′) � φ2 and∀k′′ ∈ (k, k′), (σ, k′′) � φ1

The set of words that satisfy φ is then given by Words(φ) = {σ ∈(2AP )ω|σ � φ}.

Some simple LTL formulas to specify robot tasks could be: Safety(� ¬ φ1, globally avoiding φ1), ordering ( � (φ1 ∧ � (φ2 ∧ � φ3)), φ1,φ2, φ3 hold in sequence), response (φ1 ⇒ φ2, if φ1 holds, φ2 will hold infuture), repetitive surveillance (� � φ, φ holds infinitely often).

The local task specifications for agent k is of the following form:

φk = φsoftk ∧ φhardk , (3.3)

where φsoftk are ”soft” and φhardk the ”hard” sub-formulas, k ∈ K. Thesubscript k for different agents is neglected further in this thesis, sincethe agents tasks are only assigned locally and are not connected toeach other. The task specifications in φhard must always be guaran-teed and are not allowed to be relaxed, such as safety constraints likecollision-avoidance or energy-supply guarantee. On the other hand,soft specifications are satisfied as much as possible and can be relaxedif the initial task is infeasible.

Furthermore, there exists a non-deterministic Buchi automaton (NBA)Aφ over 2AP corresponding to φ [22].

Definition 3.1.4. The NBAAφ is defined by the tuple: Aφ = (Q, 2AP , δ, Q0,F),where Q is a finite set of states; Q0 ⊆ Q is a set of initial states; 2AP isthe alphabet; δ : Q × 2AP → 2Q is a transition relation; F ⊆ Q is a setof accepting states.

All input alphabets l ∈ 2AP that enable the transition from qm to qnin δ are denoted by

χ(qm, qn) = {l ∈ 2AP |qn ∈ δ(qm, l)}. (3.4)

In this thesis we use the fast translation algorithms of [26], whichgenerates NBA with fewer states and transitions than [22]. The processof generating a NBA can then be done in time and space 2O(|φ|).


An infinite sequence of states that starts from an initial state that fol-lows the transition relation is an infinite run in the NBA. In addition,the run is called accepting if an accepting state appears infinitely often.We denote the accepted language of Aφ by Lω(Aφ), which is the set ofinfinite words that result in an accepting run ofAφ. Due the character-istic of Aφ as non-deterministic, there might exist multiple runs of thesame word.

Given the task specifications in ”soft” and ”hard” sub-formulas ineq. (3.3) a NBAAhard = (Q1, 2

AP , δ1, Q0,1,F1) for the ”hard” sub-formulasis generated and a second NBA for the ”soft” sub-formulas denoted byAsoft = (Q2, 2

AP , δ2, Q0,2,F2). Introduced in [16] a relaxed automata in-tersection between Ahard and Asoft is defined by the following:

Definition 3.1.5. The relaxed intersection of Ahard and Asoft is definedby: Aφ = (Q, 2AP , δ, Q0,F), where Q = Q1 × Q2 × {1, 2}; Q0 = Q0,1 ×Q0,2 × {1}; F = F1 × Q2 × {1}; δ : Q × 2AP → 2Q, with 〈q1, q2, c〉 ∈δ(〈q1, q2, c〉, l) when the following three conditions hold: (i) l ∈ χ1(q1, q1);(ii) χ2(q2, q2) 6= ∅; (iii) q1 /∈ F1 and c = c = 1; or q2 /∈ F2 and c = c = 2;or q1 ∈ F1, c = 1 and c = 2; or q2 ∈ F2, c = 2 and c = 1.

The proof that there is an accepting run R in Aφ while always sat-isfying the hard specifications and the soft specifications as much aspossible can be found in [16].

Example 3.1.2. Consider the same wFTS as in Example 3.1.1. We spec-ify now a surveillance task between pose r1 and r4 and in addition forsafety reasons the agent is not allowed to visit pose r2. We define thisspecifications as hard tasks and do not allow soft tasks. The resultingLTL formula for the given specifications is then: φ = (� � r1) ∧ (� �r4) ∧ (� ¬ r2). The constructed NBA from the LTL formula is shownin Figure 3.3.

3.1.3 Weighted product Buchi automaton

Given the weighted finite transition system by Definition 3.1.2 and therelaxed NBA by Definition 3.1.5 the weighted product Buchi automaton isconstructed, where we rely on the work of [12], [15] and [16].

Definition 3.1.6. The weighted product Buchi automaton is defined asAtp = T tc ×Aφ = (Q′, δ′, Q′0,F ′,Wp), whereQ′ = Π t×Q, q′ = 〈π, q〉 ∈ Q′,


q2

q1

init¬r2

r1 & ¬r2

r4 & ¬r2r4 & ¬r2

r1 & r4 & ¬r2

¬r2

r1 & ¬r2

r1 & r4 & ¬r2

¬r2

Figure 3.3: The constructed NBA of the LTL formula φ = (� � r1) ∧(� � r4) ∧ (� ¬ r2), where state q1 with the double circles is indicatedas accepting state. The NBA was generated by using the algorithm of[26].

for all π ∈ Π t and for all q ∈ Q; δ′ : Q′ → 2Q′ , 〈πj, qn〉 ∈ δ′(〈πi, qm〉) if

and only if (πi, πj) ∈→c and qn ∈ δ(qm, Lt(πi)); Q′0 = Π0 ×Q0 is the setof initial states; F ′ = Π t×F is the set of accepting states; Wp : δ′ → R+

is the weight function to be defined.

In our approach we will calculate Wp of the weighted product Buchiautomaton Ap as given in eq. (3.5).

W tp(〈πi, qm〉, 〈πj, qn〉) = W t

c (πi, πj) + α · Dist(Lt(πi), χ2(q2, q2)), (3.5)

where W tc (πi, πj) is the cost among two poses in the transition system

T tc , α is a design parameter and the function Dist(Lt(πi), χ2(q2, q2))

measures how much the transition violates the constraints given byAsoft. In our case is the cost of the transition system the Euclideandistance among poses and for the derivation of the Dist() functionwe refer the reader to [12] and [16]. In the product automaton, taskand wFTS specifications are included and we are prepared to find anoptimal run through the product automaton.


3.2 Execution level

In this section we show how to find an accepting run Rt through theproduct automaton Atp and convert it to a path. The path of an agentis an infinite sequence of states in T tc that satisfies the specifications ofAφ. The path should be in the prefix-suffix structure:

τ t = 〈τ tpre, τ tsuf〉 = τ tpre[τtsuf ]

ω = π0π1π2 · · · [πfπf+1 · · · πn]ω, (3.6)

where the prefix τ tpre is only performed once and the suffix τ tsuf is re-peated infinitely often. τ t contains the poses πi ∈ Π t to be visited andwe define the trace trace(τ t) as the sequence of atomic propositionsthat are true at the states along the path τ t:

trace(τ t) = Lt(π0)Lt(π1)L

t(π2) · · · [Lt(πf )Lt(πf+1) · · ·Lt(πn)]ω (3.7)

We call an infinite path τ t of T tc valid if (πi, πi+1) ∈→tc, for i = 0, 1, 2, · · ·

which is fulfilled if the path satisfies the specifications in the FTS.Moreover, the path is safe if it satisfies all ”hard” specifications trace(τ t) �φhard and satisfying if trace(τ t) � φ.

The path τ t is found by project an accepting run Rt in Atp on Π t

which is τ t = Rt|Πt . Since τ t is in prefix-suffix structure, the run Rt

needs also to be in the prefix-suffix structure:

Rt = q′0q′1 · · · [q′fq′f+1 · · · q′n]ω, (3.8)

where q′0 ∈ Q′0 and q′f ∈ F ′.As shown in [16], an accepting run Rt in Atp can be found such that

its projection on Π t, τ t = Rt|Πt is both valid and safe for T tc and φ.Moreover, there can be more than one accepting run Rt, but we are

interested in the run that minimizes the cost given in eq. (3.9).

Cost(Rt, Atp) =

f−1∑i=0

Wp(q′i, q′i+1) + γ

n−1∑i=f

Wp(q′i, q′i+1), (3.9)

where γ ≥ 0 as design parameter for the relative weighting amongprefix and suffix [27]. In order to find the optimal run Algorithm 3 in[16] is used, which applies and adapted Dijkstra’s algorithm on Atp. Inparticular, the algorithm computes first all possible prefix. For everyinitial state q′0 ∈ Q′ the shortest path in Atp to all accepting states q′f ∈F ′ are computed. In a second step possible suffix are computed bysearching cycles containing q′f . In the end, the optimal run Rt is therun that minimizes the cost of a combination of prefix and suffix.


r2, initr1, init r4, initr3, init

r2, q1r1, q1 r3, q1

r2, q2r1, q2 r4, q2r3, q2

r4, q1

Figure 3.4: The fully constructed product automaton Atp. The initialstate is shown in green and the accepting states that have to be vis-ited infinitely often are in red. In blue an accepting run through theautomaton is shown.

Example 3.2.1. In order to find an accepting plan for the FTS from Ex-ample 3.1.1 that satisfies the specifications of Example 3.1.2 we buildthe product Buchi automaton, visualized in Figure 3.4. Note that thestates with r2 as atomic proposition can not be reached and the syn-thesized plan visits infinitely often the accepting state 〈r4, q1〉. Finallythe run in prefix-suffix structure is:

Rt = 〈r1, init〉〈r1, q2〉〈r3, q2〉〈r4, q1〉[〈r3, init〉〈r1, q2〉〈r3, q2〉〈r4, q1〉]ω

Chapter 4

Online task adaption

In this chapter a method for online plan adaption for temporary tasks ina given framework is introduced. While an agent is executing its syn-thesized motion plan, the agent might need to adapt the initial planto respond to incoming task requests from other agents or users, thatare not known beforehand. On one hand, all temporary tasks need tobe fulfilled and on the other hand, the tasks should be fulfilled in cer-tain order, depending on the priority of each task. A trivial approachwould be to fulfill the temporary tasks in the same order as the agent re-ceived them. However, this might lead to highly suboptimal solutionswith respect to cost function eq. (3.9). Another approach could be toorder the temporary tasks with respect to minimize the cost functiongiven in eq. (3.9). Nevertheless, the cost would be minimized for thesecond approach, this might result in a case, in which a temporary taskwith high cost never is accomplished. For instance, while the agenthas a temporary task which results in a high cost, it receives frequentlytemporary tasks with low cost. The agent accomplishes the tasks withlow cost first and through receiving them frequently, the task whichleads to high cost will never be accomplished.

First we introduce how to calculate different runs Rtemp which ac-complish all temporary tasks and then how to chose a run according togiven priority and cost as defined in eq. (3.9).

At time t = Tinit all agents are without temporary tasks, where Tinitis the time t when an agent receives its initial task φ. Furthermore, wedenote withM t the number of temporary tasks to fulfill after receivinga new temporary task at time t = T0,new, where T0,new > Tinit is thetime when an agent receives a temporary task. It follows that at time

25

26 CHAPTER 4. ONLINE TASK ADAPTION

t = Tinit the number of temporary tasks is M t = 0. The incomingLTL-formula φtemp,new is given as sequence of eventually. The subsetAPφ,new ⊆ AP includes all atomic propositions used in φtemp,new.

Definition 4.0.1. The framework of an LTL-formula in a temporarytask is defined as: φtemp,i = � (ai,1 ∧ � ( ai,2 ∧ � · · · ∧ � ( ai,Ki−1 ∧� ai,Ki

))), where a ∈ AP ; Ki = |APφ,i| is the size of subset APφ,i ⊆AP ; i = 1, 2, · · ·M t is the subscript for different temporary tasks.

The set of all temporary tasks φtemp,i at time t ≥ T0,Mt is denotedby Ψt = {φtemp,1, φtemp,2, · · · , φtemp,Mt} and the set containing all atomicpropositions a in Ψt is denoted by AP t

Ψ ⊆ AP = APφ,1 ∪ APφ,2 ∪ · · · ∪APφ,Mt = {a1,1, a1,2, · · · , a1,K1 , a2,1, a2,2, · · · , aMt,KMt}.

All temporary tasks φtemp,i ∈ Ψt are accomplished if all atomic propo-sitions ai,ki ∈ AP t

Ψ were True at least once after their receiving time T0,i.Therefore, all possible sequences of propositions over AP t

Ψ are com-puted. A possible sequence containing all propositions ai,ki ∈ AP t

Ψ isdenoted as

rc = a0i,kia1i,kia2i,ki · · · a

p−1i,ki

, (4.1)

where asi,ki ∈ AP tΨ, ∀s = 0, 1, 2, · · · p − 1, where i = 1, 2, · · ·M t, ki =

1, 2 · · ·Ki, p = |AP tΨ| and all propositions ai,ki occur only once in rc.

Moreover, we denote with rc(a) as the index s where proposition a

occurs in rc. Then rc is only a valid sequence if the index rc(ai,ki) issmaller than the index rc(ai,ki+1

),∀i = 1, 2, · · ·M t and ∀k = 1, 2, · · ·Ki

In other words, rc is only a valid sequence over AP tΨ if all ai,ki ∈ AP t

Ψ

are contained in rc and the order of ai,ki ∈ AP tΨ holds with respect to the

sequence of eventually in the temporary task specifications φtemp,i ∈ Ψt.We denote then the set of all valid sequences rc over AP t

Ψ as Ct, wherethe size of Ct is |Ct| = |AP t

Ψ|!∏Mi=1(Ki!)

.

Example 4.0.1. At time t an agent has two temporary tasks and theirLTL-formulas are:

φtemp,1 = � (a ∧ � b)

φtemp,2 = � (c ∧ � (d ∧ � e))

The set containing all atomic propositions a is then AP tΨ = {a, b, c, d, e}

and the set of all valid sequences rc is:

Ct = {abcde, acbde, acdbe, acdeb, cabde,cadbe, cadeb, cdabe, cdaeb, cdeab}

(4.2)

CHAPTER 4. ONLINE TASK ADAPTION 27

For every atomic propositions in a sequence rc we define the set ofstates q′ ∈ Q′ that satisfy atomic proposition as in rc as target set Gs(as).

Definition 4.0.2. The target set for the s-th proposition in rc is definedas Gs(as) = Lt(as)−1 ×Q = {q′s = 〈π, q〉 ∈ Q′ | ∀π ∈ Lt(as)−1, ∀q ∈ Q},where Lt(as)−1 is the inverse of the labeling function, which maps anatomic propositions a ∈ AP to the set of poses where a is satisfied; s isthe s-th proposition in rc.

Since all a in rc need to be True at least once we search for a runRtemp

in the product automaton Atp from the current state q′c = 〈πc, qc〉 at timet through states in Atp, so that in each target set Gs one state q′s ∈ Gswas visited. In addition, with the superscript s we want to ensure thatthe target sets Gs are visited in the right order, which means a stateq′s ∈ Gs−1 needs to be visited before a state q′s ∈ Gs is visited. Whilesearching for runs in Atp, we also ensure that safety specifications aresatisfied. We split Rtemp up in smaller runs between target sets Gs−1and Gs

Rq′s−1,q′s = q′s−10 q′1q′2 · · · q′sn , (4.3)

where q′s−1 ∈ Gs−1 is the start state, q′s ∈ Gs is the end state of the runand if s = 0, q′s−10 = q′c. The optimal run Rq′s−1,q′s is found by applyingDijksTargets(Adj(Atp), q′s−1, Gs) of algorithm 3 line 2 of [16] and itscost is

Wp,temp(Rq′s−1,q′s , Atp) =n−1∑i=0

Wp(q′i, q′i+1), (4.4)

where Rq′s−1,q′s is the run from state q′s−1 to q′s in the product automa-ton Atp.

In addition, the approximate Euclidean distance among two targetsets is then

W tc,temp(Rq′s−1,q′s|Π , T tc ) =

n−1∑i=0

W tc (πi, πi+1), (4.5)

where Rq′s−1,q′s|Π is the projection from state q′s−1 to q′s on Π t. Thedistance among target sets is used later in eq. (4.10) to estimate thetime to fulfill a temporary task.

The run Rtemp of rc accomplishing all temporary tasks is then com-posed to

Rtemp = R0q′c,q′0R1

q′0,q′1 · · ·Rp−1q′p−2,q′p−1 = R0R1R2 · · ·Rp−1. (4.6)


Let us denote ζt as the set which contains all last poses of interest fromthe projection of the runs Rq′s−1,q′s ,∀s = 1, 2, · · · , p− 1.

ζt = {q′0|Πt , q′1|Πt , q′2|Πt , · · · , q′p−1|Πt} (4.7)

Always when the agent visits a pose of interest from ζt a part of atemporary task is accomplished, since Rq′s−1,q′s are runs among targetsets. This is used later during the plan execution to remove the alreadyachieved task fragments.

The run Rtemp would now be a sufficient run to accomplish all tem-porary tasks and for each rc ∈ Ct a corresponding Rtemp can be com-puted. However, the ordering of the tasks might be highly suboptimalor not in desired order.

Since an agent can have multiple temporary tasks at the same time,priority among different temporary tasks needs to be defined. Forevaluating the priority of temporary tasks, the two time constants Td,iand T0,i are used. The first time constant Td,i gives the desired timespan to accomplish the temporary task φtemp,i and the second timeconstant T0,i is the time t when φtemp,i was received. We define thenthe priority for a temporary task φtemp,i as the gradient of theweightFactor(Td,i, T0,i, Tend,i, t) function

weightFactor(Td,i, T0,i, Tend,i, t) =Tend,i + t− T0,i

Td,i, (4.8)

where Tend,i is an estimation, how long it takes until φtemp,i will be ac-complished. In fact, the function weightFactor() is simple a linearfunction, where the factor is equal to 1 if φtemp,i is fulfilled at time Td,i.The weight functions for two different temporary tasks are illustratedin Figure 4.1 at the time t = T0,2.

Through a temporary task φtemp,i and the two time constants Td,iand T0,i a temporary task with priority is defined.

Definition 4.0.3. A temporary task with priority is defined as tuple:φi = (φtemp,i, Td,i, T0,i), where φtemp,i is the temporary task LTL-formula;Td,i desired time span to accomplish the task; T0,i the time t when thetask was received.

The priority of a task φtemp,i increases with decreasing time constantTd,i. Note the time constant Td,i is not a deadline for the temporarytask, it is a desired time span to accomplish the task.


1

Figure 4.1: An agent receives at time t a second temporary task φ2

(red). The gradient of task φ1 (blue) is higher and therefore the weightfactor increases faster with time.

The weightFactor() is then use to weight the cost eq. (3.9) fromthe product automaton of different temporary tasks.

Since we are also interested to weight the cost of a temporary taskφtemp,i by the time it needs to be accomplished, we need to know theindex li of Rli in Rtemp = R0R1R2 · · ·Rli · · ·Rli+1 · · ·Rp−1 that accom-plishes φtemp,i. A task φtemp,i is finished successfully if ai,Ki

∈ APφ,i isTrue in a state q′ ∈ Rtemp. The run

Ritemp = R0R1R2 · · ·Rli (4.9)

is then the run from state q′c to q′li which accomplishes temporary taskφtemp,i. The time span Tend,i to accomplish task φtemp,i is then

Tend,i =W tc (R

itemp|Πt , T tc )

vavg, (4.10)

where W tc,temp(R

itemp|Πt , T tc ) is the distance in T tc as defined in eq. (4.5)

for Ritemp and vavg is the estimated average velocity of an agent.

Finally the new cost function with a cost-weight-factor for ordering


the temporary tasks is given as

Costtemp(Rtemp, Atp) =Mt∑i=1

Wp,temp(Ritemp, Atp)︸︷︷︸

cost from Atp

· Tend,i + (t− T0,i)Td,i︸︷︷︸

weight-factor

,

(4.11)where Wp,temp(R

itemp, Atp) is the cost of run Ri

temp from the product au-tomaton Atp given in eq. (4.4), Tend,i the approximated time span fromeq. (4.10) to finish task φtemp,i, t is the time, T0,i is the time when taskφtemp,i was received and Td,i is the desired time to finish φtemp,i. Therun Rtemp is then the run Rtemp which is build from rc ∈ Ct that has theminimal cost calculated by eq. (4.11).

After all temporary tasks are accomplished the agent has to fulfillits initial task specifications φ. The remaining run Rrem is then againcomputed by algorithm 3 of [16], where the initial state is the end stateq′pn of Rtemp and the accepting set is still the initial set F ′. Finally therun Rt is updated to

Rt = RtempRrem (4.12)

whenever a temporary task φi at time t is received.The procedure to update the run Rt is given in the following three

algorithms(Algorithm 1, Algorithm 2, Algorithm 3). First in Algo-rithm 1 all valid sequence of propositions are computed and the vari-able Γ t stores the last atomic propositions ai,Ki

in the sequence of eachtemporary task. The variable Γ t is used as input in Algorithm 2 toidentify the end of a task in a sequence of propositions rc. The func-tion updateTaskVars() adds the new task φnew to Ψ t, the task endproposition to Γ t and all new propositions of φnew to AP t

Ψ . Withpermutations() all possible permutations are computed withouttaking care of ordering. Not valid permutations are found withcheckSequence() and are removed. In the algorithms we denote aupdated variable with +, e.g. Γ t+ is the updated set of Γ t.


Algorithm 1: Update temporary task UpdTempTask()Input: φnew, Γ t, AP t

Ψ , Ψ t

Output: Γ t, AP tΨ , Ct, Ψ t

1 [Γ t+ , AP t+

Ψ , Ψ t+

] = updateTaskVars(φnew, Γt, AP t

Ψ , Ψt)

2 C = permutations(AP t+

Ψ )

3 foreach rc ∈ C do4 foreach φtemp ∈ Ψ t do5 if checkSequence(rc, φtemp) = False then6 remove rc from C7 end8 end9 end

10 Γ t = Γ t+ , AP tΨ = AP t+

Ψ , Ct = C, Ψ t = Ψ t+

In Algorithm 2 the actual run Rt is updated to fulfill the temporarytasks. The variable ∆temp is used as temporary storage of the cost. Thefunction getCurrentState() returns the current state of an agentin Atp, while executing its plan. Target sets are build with the func-tion buildTargetSet() and last() returns the last state in a se-quence. With getTaskIndex() the index i for a temporary task φi isreturned, if the task φtemp,i is accomplished.


Algorithm 2: Find temporary Run RunTemp()

Input: Atp, Rt, Γ t, AP tΨ , Ct

Output: Rt, ζt

1 Cost =∞2 foreach rc ∈ Ct do3 Rtemp = getCurrentState(Atp)4 ∆temp = 0

5 ζ = ∅6 foreach a ∈ rc do7 G = buildTargetSet(a, Atp)8 q′end = last(Rtemp)

9 Rl = min(DijksTargets(Adj(Atp), q′end,G))10 Rtemp = RtempR

l

11 add last(Rtemp) to ζ12 if a ∈ Γ t then13 i = getTaskIndex(a, Γ t+)14 Tend,i = W t

c,temp(Rtemp|Πt)/vavg15 ∆temp = ∆temp +Wp,temp(Rtemp, Atp)·

weightFactor(Td,i, T0,i, Tend, t)

16 end17 end18 if ∆temp < Cost then19 Cost = ∆temp

20 Rtemp = Rtemp

21 ζt = ζ

22 end23 end24 q′end = last(Rtemp)

25 Rrem = OptRun(Atp, q′end)26 Rt = RtempRrem

In the end Algorithm 3 combines Algorithm 1 and Algorithm 2 andis called whenever an agent receives a new temporary task φnew.


Algorithm 3: Replan with new temporary task ReplanTemp()

Input: φnew, Atp, Γ t, AP tΨ , Ψ t

Output: Γ t, AP tΨ , Ψ t, Rt, ζt

1 [Γ t+ , AP t+

Ψ , Ct, Ψ t+ ] = UpdTempTask(φnew, Atp, Γ t, AP tΨ , Ψ

t)

2 [Rt, ζt] = RunTemp(Atp, Rt, Γ t+ , AP t+

Ψ , Ct)3 Γ t = Γ t+ , AP t

Ψ = AP t+

Ψ , Ψ t = Ψ t+

Chapter 5

Workspace adaption

In the previous chapters we have shown how the initial path for anagent is computed and the adaption for incoming temporary tasks. Inthis chapter, we handle the case if the wFTS has changed during theexecution of the agents path. A change in the wFTS might results inan invalid current path and a replanning needs to be done. Therefore,we introduce first how the wFTS T tc and the product automaton Atpare updated to apply in a second step the replanning. Again the mostof the work is adapted from [16], where especially the changes to [16]should be highlighted, since in [16] does not include temporary tasks.Therefore, it is shown how the temporary task are included in the re-planning after the wFTS has changed.

5.1 Knowledge update

While an agent is executing its current path, it can sense changes inthe current workspace. In our case the sense knowledge is send to allagents at the same time, so that all agent have the same workspacemodel. In comparison to [16], where different agents exchange infor-mation trough a communication protocol. We denote a set of sensinginformation received at time t ≥ 0 as

Senset = {(MP , S, S¬), E, E¬}, (5.1)

whereMP is the pose of a received pose of interest; S ⊆ AP is the set ofpropositions satisfied by this pose; S¬ ⊆ AP is the set of propositionsnot satisfied by this pose; (πi, πj) ∈ E if (πi, πj) is an edge to be addedto→t

c; (πi, πj) ∈ E¬ if (πi, πj) is an edge to be removed from→tc.

35

36 CHAPTER 5. WORKSPACE ADAPTION

Let us denote T t−c as transition system before the update at time tand T t+c after the update time. Denote by πnew = B(MP) as the new re-ceived pose. If a new pose πnew is a not existing pose πnew /∈ Π t− , thenπnew is added to Π t− and Lt

+(πnew) = S. Otherwise if πnew already

exist, then propositions of the pose are adapted so that Lt+(πnew) =

Lt−

(πnew) ∩ S \ S¬. Regarding E,E¬ ∈ Senset, the transitions in E areadded to T t−c and transitions E¬ are removed from T t−c . Algorithm 4describes how to build T t+c from T t−c based on Senset. This algorithmis an adapted version of Algorithm 5 of [16]. The variable Π t ⊆ Π inthe algorithm is used to store the poses of Π t− of which the labelingfunctions has changed during the update. Π t will be an input argu-ment of Algorithm 6 of [16] which is used later.

Algorithm 4: Update knowledge, UpdKnow()

Input: T t− , Senset

Output: T t+ , Π t

1 foreach (MP , S, S¬) ∈ Senset do2 if πnew /∈ Π t− then3 add πnew to Π t−

4 πnew = B(MP) and Lt+

(πnew) = S

5 end6 if πnew ∈ Π t− then7 Lt

+(πnew) = Lt

−(πnew) ∩ S \ S¬

8 end9 end

10 remove (πi, πj) from→t−c , for all (πi, πj) ∈ Et

¬11 add (πi, πj) to→t−

c , for all (πi, πj) ∈ Et

After the transition system was updated also the product automa-ton Atp needs to be updated. This is achieved by applying Algorithm6 UpdProd() of [16]. All changed states πi ∈ Π t whose labeling func-tion has changed, and all added or removed edges (πi, πj) ∈ (Et ∪Et

¬)

have a corresponding state q′s = 〈πi, qm〉 ∈ Q′. This correspondingstates q′s are marked as ”unvisited” in the product automaton Atp. Alltransition of a state marked as ”unvisited” have to be reconstructed.

CHAPTER 5. WORKSPACE ADAPTION 37

5.2 Revision and Validation

If the product automaton Atp has changed, it is necessary to check ifτ t is still valid and safe. The current run Rt is validated by the Algo-rithm 7 ValidRun() of [16]. The algorithm checks if the current runRt is containing states q′s which have been marked as ”unvisited” andif it does so, it checks if the current run Rt is still valid. The outputsof ValidRun() are then Ξt and ℵt, where Ξt is the set invalid tran-sitions and ℵt the set of unsafe transitions in Rt. The run Rt needs tobe revised if Ξt and ℵt are not empty. The revision is performed byan adapted algorithm 8 Revise() of [16]. In fact, line 11 was addedto the Algorithm 8 of [16]. Through this adaption the temporary tasksare also included in revising the current plan. The adapted algorithmis then Algorithm 5, where tail(q′s, Rt−

pre) is the segment of Rt−pre after

q′s not including q′s, head(q′s−1, Rt−pre) is the segment of Rt−

pre before q′s−1not including q′s−1 and first(Rt−

pre) is the first state of Rt−pre.


Algorithm 5: Revise the current plan Revise()

Input: Ξt, ℵt, Atp, Rt− , Γ t

Output: Rt+

1 foreach (q′s, q′s+1) ∈ (Ξt ∪ ℵt) do

2 if (q′s, q′s+1) ∈ Rt−

pre then3 if q′s is first(Rt−

pre) then4 q′s−1 = q′s5 end6 bridge = DijksTarget(Adj(Atp)), q′s−1,

tail(q′s, Rt−pre)

7 if bridge 6= ∅ then8 Rt+

pre = [ head(q′s−1, Rt−pre) bridge

tail(last(bridge), Rt−pre)]

9 else10 Rt+ = OptPlan(Atp)11 Rt+ = RunTemp(Atp, Rt+ , Γ t, AP t

Ψ , Ct)12 break13 end14 end15 if (q′s, q

′s+1) ∈ Rt−

suf then16 if q′sis last(Rt−

pre) then17 tail(q′s, R

t−

suf) = first(Rt−

suf)

18 end19 repeat lines 3-8 but replace Rt−

pre by Rt−

suf

20 end21 end22 Rt+ = [Rt+

pre, Rt+

suf ]

Instead of calling algorithm 3 of [16] and computing a optimal plan,the provided approach is to try to revise the plan locally. Locallymeans that a ”bridge” is searched for unsafe and invalid edges in thecurrent run Rt. A ”bridge” replaces the invalid and unsafe edges bysearching for a run from the start start state of the edge to its targetstate, a simple example is given in Figure 5.1. If no bridge is foundAlgorithm 3 of [16] is called.

The Algorithm 6 is the whole motion planning scheme. Wheneveran agent receives a Senset information the wFTS T tc and the product

CHAPTER 5. WORKSPACE ADAPTION 39

Figure 5.1: The initial run is in blue. The green edge gets removedthrough a workspace update. A bridge is found over the red state.

automaton Atp is update. The current run is validated and if it is in-valid, the run is revised. Furthermore, with Υ t is the sum denotedof changed poses of interest and changed edges in the wFTS. If morechanges than N call were applied in the wFTS a reoptimizing of thewhole run Rt is performed.


Algorithm 6: Motion planning scheme with knowledge-update

Input: T 0, Aφ, Γ t, AP tΨ , Ct

Output: Rt, τ t, Υ t

1 if t = 0 then2 R0

opt = OptPlan(A0p) // Algorithm 3 of [16]

3 end4 read Senset

5 [T t+c , Π t] = UpdKnow(T tc , Senset) // Algorithm 4

6 At+p = UpdProd(At−p , Π t, E, E¬) // Algorithm 6 of [16]

7 [Ξt,ℵt] = ValidRun(At+p , Rt, Π t, E¬) // Algorithm 7 of[16]

8 Υ t+

= Υ t + |Π t|+ |E|+ |E¬|9 if Υ t+ ≥ N call then

10 Rt+ = OptPlan(Atp) // Algorithm 3 of [16]11

12 Rt+ = RunTemp(Atp, Rt+ , Γ t, AP tΨ , Ct) // Algorithm 3

13

14 else15 Rt+ = Revise(At+p , Rt, Ξt,ℵt) // Algorithm 516 end17 Rt = Rt+ , τ t = Rt|Πt , Υ t = Υ t

+ , T tc = T t+c

Chapter 6

Graphical User Interface for RobotMotion Planning

In this chapter the implementation of the Graphical User Interface(GUI) for the robot motion planning introduced in chapter 3 is ex-plained. Therefore, the user can choose among the simulation environ-ment Gazebo1 or setting up an experiment in reality. First an overviewof the whole package is given, second the selection of the FTS, thirdthe definition of task specifications for the agents and at the end of thechapter simulation or experimental setups are clarified.

The GUI connects user inputs with the hybrid-controller softwareby using widgets e.g. buttons, check-boxes etc.. For the implementa-tion of the GUI Qt2 libraries are used. The Qt developing environ-ment has already well developed widgets and it can be connectedwith the Robotics Operation System3 (ROS). The basic functionalityof a GUI is that if a user applies changes to a widget, a signal is emit-ted. The emitted signal contains predefined information and is con-nected to a function as illustrated in Figure 6.1. Then everytime a sig-nal is emitted the function is called. For instance, a button widgetcontains the signal pushed which has a boolean information to indicatethat the button is pushed. The signal pushed is then obviously emit-ted when the button is pushed. The signal pushed is connected withthe function onButtonPushed(). By pressing the button the pushedsignal with boolean information True is emitted and calls the function

1Webpage: http://gazebosim.org/2Webpage: https://www.qt.io/3Webpage: http://www.ros.org/

41

http://gazebosim.org/

https://www.qt.io/

http://www.ros.org/

42 CHAPTER 6. GRAPHICAL USER INTERFACE FOR ROBOTMOTION PLANNING

Button1

User Interface

Button2

Function library:

onButtonPressed1()

onButtonPressed2()

pushed

pushed

Figure 6.1: User interface with two buttons. In blue the signal pushedemitted by the buttons. The red arrow illustrate the connection amongbutton and functions through a signal.

onButtonPushed().The GUI for robot motion planning is connected with the Robotic

Operation System (ROS) environment. ROS is flexible framework forrobot softwares, which has already a collection of tools, libraries andconventions that simplify the task of creating a complex software forrobot motion planning. Moreover, robots such as TurtleBot4 or TIAGo5

use the ROS framework for navigation and planning in simulation andreal world robots. Another advantage of ROS is the real-time mes-sage handling and passing services allow to run different ROS pack-ages at the same time which is beneficial for multiple robot setups andreal-time applications. By using the provided GUI the user can easilychoose different worlds and select a finite transition system withoutchanging parameter files. In addition it is possible to launch multi-ple agents with different local task specifications at the same time andthe current state with its sensor data is for convenience visualized inRVIZ. Within the GUI the following needs to be specified to set up anexperiment or simulation successfully:

1. World selection4Webpage: https://www.turtlebot.com/5Webpage: http://tiago.pal-robotics.com/

https://www.turtlebot.com/

http://tiago.pal-robotics.com/

CHAPTER 6. GRAPHICAL USER INTERFACE FOR ROBOT MOTIONPLANNING 43

Figure 6.2: The main window of the GUI. On the top left the mapwith the wFTS is shown. At the right side the chosen agents with taskspecifications and synthesized plan are displayed.


2. Generate FTS and save it

3. Define agents

• Agent type• Agent name• Initial pose• Used localization• Task specification

– Hard task– Soft task

4. Synthesize plan

5. Setup experiment or simulation

6. Start experiment or simulation

7. Record data

8. Apply changes during runtime

6.1 World selection

A predefined world for simulation in gazebo and its map can be cho-sen by selecting its name on the top left combo box below the mapscreen. The combo box is connected with the function load world()through the signal currentIndexChanged. The signal currentIndexChangedis emitted when another world in the combo box is chosen. After theworld has been chosen load world() loads an image as map anda corresponding parameter file which contains information about thetransformation from image coordinate frame I to the maps coordinateframe m in the image, resolution res [meters/pixel] for converting pix-els in meters and if known the transformation QTQM, where Q is thecoordinate frame of a motion capture system and M the maps coor-dinate system in three dimensional space. A map can be generatedby using gmapping6 to map real environment or a map can be drawn.Detailed information about the parameters can be found on the wikipage for the map server7 ros-package. In Figure 6.3 a map done by

6http://wiki.ros.org/gmapping7http://wiki.ros.org/map_server

http://wiki.ros.org/gmapping

http://wiki.ros.org/map_server


Figure 6.3: A map generated with gmapping.

gmapping is shown.

Add new world

For adding new worlds to the GUI add the world’s name <world name>to the list in parameter file PKG DIR/config/gui config.yaml. Alsoadd the map image and the corresponding map.yaml in the directoryPKG DIR/scenario/<world name>. If the world is used for simula-tions save the world file to PKG DIR/worlds/gazebo/<world name>.world.

6.2 Select Finite transition system

As introduced in section 3.1 the workspace is abstracted in poses ofinterest πi ∈ Π t. For defining a wFTS the exact position and orienta-tion of each pose of interest must be known and transitions→c amongthese poses must be defined. Setting up such a wFTS by adapting pa-rameter files or even hard coding is exhausting work and presumesthe user is familiar with the coding environment. With our providedGUI the user can simply select poses of interest by clicking with themouse on a map and define the edges between poses in a matrix. Forconvenience the chosen wFTS is saved in a parameter file and can bereloaded afterwards. By pressing the button ”Choose region of interest”in the the main window of the GUI a second interface window willpop up in which the map for the current selected world is loaded.

6.2.1 Choose pose of interest

A pose of interest π is chosen with the cursor in the map graphic. Thegraphic emits three different signals:

mouseButtonPressed: The signal mouseButtonPressed contains pixel co-ordinates Ipcurrent and is emitted whenever the left mouse but-


Figure 6.4: The dialog window: Left the loaded map in which with thecursor poses can be generated and right the transition Matrix Mtrans,where transitions are enabled by checking the corresponding boxes.

ton is pressed in the graphic. The connection is to the functionselect position(Ippress).

moveMouse: The signal moveMouse contains pixel coordinates Ipcurrentand is emitted whenever the mouse moves in the graphic. Theconnection is to the function move mouse(Ipcurrent).

mouseButtonReleased: The signal mouseButtonReleased contains pixelcoordinates Iprelease and is emitted when the left mouse buttonis released in the graphic. The connection is to the functionsselect orientation(Ippress, Iprelease) and matrix add().

Whenever the left mouse button is pressed in the map the currentpixels Ippress at cursor position is selected. The pixels Ippress are usedto calculate the position Mr of π through select position(Ippress)which uses eq. (2.8) and eq. (2.9). A red point is added to the graphic toindicate the chosen position. While keeping the mouse button pressedan arrow with its origin at Ippress is following the cursor. The ar-row orientation is updated by move mouse(Ipcurrent). After releasingthe mouse button the arrow is indicating the chosen pose of interestand function select orientation(Ippress, Iprelease) is called. In


select orientation(Ippress, Iprelease) the arrow’s angle θ com-puted by

θ = atan2

(− mvrelease + mvpress

murelease + mupress

)(6.1)

is used to determine quaternion Mq as the orientation of the chosenpose of interest. We assume the orientation is only rotated around thez-axis.

Finally the pose of interest π is then composed by position Mr andthe quaternion Mq. The whole procedure is summed up in Algo-rithm 7. Note that poses can be chosen anywhere in the map althoughit would be in a occupied area. The user should be aware of that andonly select reachable poses. Otherwise the agent will fail the plan exe-cution.

Algorithm 7: Selection Pose of interest MPInput: Pixel coordinates: Ippress, Ipcurrent, IpreleaseOutput: Pose of interest MP , update GraphicsScene

1 Mr = select position(Ippress)2 while mouse button pressed do3 if cursor moved then4 move mouse(Ipcurrent)5 end6 end7 Mq = select orientation(Ippress, Iprelease)8 MP = composePose(Mr, Mq)9 matrix add()

10 return MP

6.2.2 Select transitions

For defining transitions among poses of interest we use a matrix ofcheck boxes. Let us call this matrix transition matrix Mtrans(r, c), wherec,∀c = 1, 2, · · · , N are the column indices and r,∀r = 1, 2, · · · , N therow indices. All check boxes in Mtrans(r, c) emit the following signal:

checkStateChanged: The signal checkStateChanged contains informationabout the check box state and the indices r and c of the check boxposition in Mtrans(r, c). The signal is emitted by checking and


unchecking the box and is connected to the functionedges both ways(state, c, r).

A single check box in Mtrans(r, c) stands for a transition (πc, πr) be-tween two poses πc and πr. If a check box is checked, the transition(πc, πr) is added to→c and as opposite if the box is unchecked the tran-sition (πc, πr) is removed from→c. The function edges both ways(state,c, r) sets the check box at (r, c) in Mtrans to the same state as theone at (c, r) in Mtrans. Through this modification it is ensured thatif the transition (πc, πr) is removed or added also the reverse transi-tion (πr, πc) is removed or added. By adding edges always in bothdirections, terminal states are avoided, since they are not allowed inour wFTS as introduced in Definition 3.1.2. The transitions are visu-alized in the graphic as straight lines. A row and a column is addedto Mtrans if a new pose of interest was selected in the map and if apose is removed from the graphic also the corresponding row and col-umn in Mtrans are removed. Adding a new row and column for a newpose is done by function matrix add() and the removing is done bymatrix remove().

Note the drawn lines among poses in the graphic only representif there exist a transition and do not represent the actual path of anagent. The path of an agent to reach a pose is computed online whileexecuting the plan. In addition, the user has to be aware, that the costof a transition in the wFTS is computed as Euclidean distance. TheEuclidean distance should approximately estimate the length of theexecuted path between poses. If this assumption does not hold for theselected wFTS, the synthesized plan may not be optimal. An inappro-priate example of a wFTS is given in Figure 6.5. A transition amongpose r01 and r04 is added. Since the robot is not able to move throughwalls it will plan a path trough the corridor to reach pose r04 from r01

as shown in blue. The Euclidean distance and the path length differfrom each other significantly which can lead to a not optimal solutionduring the graph search algorithm.

6.2.3 Define general atomic propositions

As introduced in section 3.1.1 general propositions ag are used to addmore propositions to a pose of interest besides the ap proposition whichonly states of the agent reached a pose or not. The button ”General


r1

r3

r4

r2

Figure 6.5: To FTS from Example 3.1.1 a transition from r01 to r04 isadded (red dashed line). In blue the path of an agent while movingfrom r01 to r04.

AP ’s” opens a new window which enables the user to define and as-sign general propositions to poses of interest. On the right side of thenew window a matrix with check boxes Mprop(r, c) is displayed. Simi-lar to Mtrans(r, c) from section 6.2.2 the check boxes in Mprop(r, c) allowthe user to add or remove general AP ’s ag from a pose of interest πc.Each column c = 1, 2, · · · , N t indicates a pose of interest πc ∈ Π t andthe rows indicate possible propositions ag ∈ AP . All check boxes inMprop(r, c) emit the signal:

checkStateChanged: The signal checkStateChanged contains informationabout the check box state and the indices r and c of the checkbox position in Mprop(r, c). The signal is emitted by checking andunchecking the box and is connected to the functionadd remove general prop(state, c, r).

The function add remove general prop(state, c, r) adds thegeneral AP from row r to the region of interest πc if the check box stateis checked or removes the proposition if the state is unchecked.

The definition of a new general AP is done by pressing the but-ton ”Add atomic proposition”. A dialog will pop up where the label forthe new proposition can be defined and saved afterwards. The new


Figure 6.6: Left: The interface to select general AP ’s. Pose r01 has theproposition bucket assigned and r02 the proposition ball ; right: A newproposition can be defined and added to Mprop(r, c).

proposition is then added to Mprop(r, c) by adding a new row to thematrix.

6.2.4 Additional buttons

Delete last ROI

The button ”Remove last ROI” removes the last added pose of interestπN ∈ Π and all it dependent edges (πN , πj) ∈→c. This allows the userto remove wrong chosen poses in a convenient way.

Set all Edges and Delete all Edges

Sometimes almost all transitions are allowed. For this purpose the but-ton ”Set all Edges” can be pressed which will check all boxes in Mtrans

and therefore add all possible transitions (πi, πj) to the wFTS. The op-posite does the ”Delete all Edges” button. It removes all previous se-lected transitions (πi, πj) ∈→c.

Save Finite Transition System

The user can save the chosen wFTS Tc by pressing the button ”SaveFinite Transition System”. The wFTS will be saved by default in the filePKG DIR/FTS/env GUI.yaml and the dialog window is closed.


Reset and Cancel

The ”Reset” button removes all poses of interest and all transitions and”Cancel” closes the dialog window without saving the wFTS.

Load FTS from yaml file

Instead of selecting the wFTS for every simulation or experiment it canalso be loaded from a yaml file by pushing the button ”Load wFTS fromyaml file”.

6.3 Define agent type and task specifications

On the right side in the main window of the GUI, the user has to spec-ify for every agent its type and the task specifications. If the user issatisfied with all selections the plan synthesize can be started after-wards.

6.3.1 Agent types

The supported agent types are at the moment the TurtleBot and TIAGo.The GUI allows to use different agent types at the same time. Theagent type is chosen by the combo box. The combo box is connectedwith the function set agent type() through the signal currentIn-dexChanged. The signal currentIndexChanged is emitted when anotheragent type in the combo box is chosen.

TurtleBot and TIAGo

TurtleBot is a low-cost robot kit with open-source software. The mainhardware consists of a Kobuki Base which drives the robot and a depthcamera is used for localization and navigation purpose in the workspace.Kinematics constraints of the base are given by its differential wheels.TIAGo is a robot from PAL Robotics with a manipulator. Kinematicconstraints are also given by the differential wheels of its base andfor navigation and localization TIAGo has additionally a laser range-finder. Both robots use the move base ros-packages to receive naviga-tion goals, plan the path to the next goal and perform collision avoid-ance during execution.


Add new agent type

Agent types can be added by adapting the file PKG DIR/config/gui config.yaml,where the new can be added to the dictionary of models. However,also the launch files are of course needed.

6.3.2 Robot name

Robot names are used to set the namespace for ROS nodes and topicsthat belong to an agent. The given names must differ among robots.Note certain namespaces are needed while using the motion capturesystem as localization.

6.3.3 Initial pose

Initial poses πi ∈ Π0 can be selected among the preselected poses inthe wFTS. Be aware that in multiple robot systems not more than onerobot can initialized at each pose. The initial pose is selected throughthe combo box, where the combo box is connected with the functionset init pose() trough the signal currentIndexChanged. The sig-nal currentIndexChanged is emitted when another initial pose in thecombo box is chosen.

6.3.4 Localization and Navigation

The localization is by default done by the amcl-package8 for TurtleBotsand TIAGos. Nevertheless the localization with amcl-package works,the use of a motion capture system is more accurate. However, themotion capture system and the map do not have the same coordinateframe. Therefore, a node is transforming all measured poses QP bythe motion capture system in the map coordinate frame M throughthe transformation QTM. Note that the robot name must be exactly thesame as the model name in the motion capture system software so thatthe topics and nodes are given in the correct namespace.

8Wiki page: http://wiki.ros.org/amcl

http://wiki.ros.org/amcl


Integrate Motion Capture System in Navigation

The ros-package move base9 for navigation uses the amcl-package for lo-calization. A motion capture system can replace this localization pack-age. For a successful integration the framework from move base needsto be understand. Therefore, it is shortly shown how the amcl-packageis integrated in move-base and then replaced by the motion capture sys-tem.

The amcl-package estimates the current robot pose, here given asfootprint, with respect to the map frame M and updates then theodometry frame O with dead reckoning. The mobile base estimatesthe transformation OTF among odomtery O and footprint frame F ,where drift and measurement uncertainties effect the accuracy. Theamcl-package adjusts the odometry frame O in order to compensatedrift.

However, while using a motion capture system we want to avoidmeasurements uncertainties by laserscans and drift. Therefore, trans-formation MTO among map M and odometry O frame and transfor-mation OTF from odometry O to footprint F frame are replaced bymeasurements of a motion capture system. A measurement of the mo-tion capture system QP is then the transformation QTF between cap-ture system Q and footprint frame F . As odometry frame O is simplythe initial pose of the robot MPinit taken and from this the transforma-tion to the footprint frame OTF is calculated, as shown in Figure 6.7and Figure 6.8.

Since motion capture system Q and mapM are not in the same co-ordinate frame some transformation need to be applied. The transfor-mation among map and odometry frame is the initial robot poseMPinitgiven in map frameM and is computed by eq. (6.2) and eq. (6.3):

p(MrO) = Qq>M ⊗ p(Qr) ⊗ QqM − Qq>M ⊗ p(QrM) ⊗ QqM (6.2)

MqO = Qq>M ⊗ Qq (6.3)

The transformation OTF from odometry O to footprint F frame issimilarly computed and updated through new measurements of themotion capture system. Given transformationsMTO and OTF move base

9Wiki page: http://wiki.ros.org/move_base

http://wiki.ros.org/move_base


map_frame odom_frame footprint_frame

Dead Reckoning

Odometry Drift

Translation,Orientation

Translation,Orientation

Estimated by AMCL

map_frame odom_frame footprint_frame

Initial pose Motion Capture system

Motion Capture System measurement

AMCL Localization

Motion Capture System

MoCap_frame

Transformation MoCap to Map

Figure 6.7: Upper diagram shows coordinate transformation by usingthe amcl-package. AMCL estimates the transformation from map to thecurrent robot pose, here in footprint frame, but the broadcasted trans-formation is from map frame to the odometry frame; Second diagramshows the coordinate transformations replaced by a motion capturesystem.

can be used as navigation controller. The procedure is given in Algo-rithm 8, where transform() transforms a pose in a new coordinateframe and broadcast() broadcasts a new transformation.

Algorithm 8: Update TF-TreeInput: Measurement QP , TF QTMOutput: Updated TF-Tree

1 MP = transform(QP ,Q TM)2 if first pose then3 MTO = MP4 broadcast(MTO)

5 else6 OP = transform(MP ,M TO)7 OTF = OP8 broadcast(OTF)

9 end


Figure 6.8: Blue: Motion capture system frame Q; Purple: Map frameM; Green: Odometry frame O (Initial pose); Red: Footprint frameF (current Pose); Orange: Agent trajectory; Black: Transformationsbetween coordinate frames.

6.3.5 Task specification

The task specifications are only local which means they apply only forone agent. The LTL-formulas can be written with the syntax given intable 6.1.

As defined in eq. (3.3) the task specification φ is done in sub-formulasfor hard- φhard and φsoft soft-tasks. The user can type them in the cor-responding line in the GUI.

6.4 Synthesize plan

The initial plan can be synthesized after wFTS, agent type and thetask specifications are defined. The GUI synthesizes a separate mo-tion plan that satisfies all specifications for each robot. The plannernode is started from the GUI and it subscribes the task specifications,robot pose and a switch topic which enables the planner. The frame-work is shown in Figure 6.9. The wFTS is loaded from the previ-ous saved file. The non-deterministic Buchi automaton Aφ is con-structed through LTL-formulas φ and the weighted product automa-


Table 6.1: Syntax for LTL operators.Description Operator Syntaxand ∧ &&or ∨ ||not ¬ !next ◦ Xuntil ∪ Ualways � []eventually � <>

implication ⇒ − >

ton A0p is build. On the product automaton A0

p the graph search al-gorithm is applied to find the optimal run R0, as explained in sec-tion 3.1. The found run R0 is send back to the GUI and is displayed inthe corresponding text box. Meanwhile the planner node is in ready-state prepared to send the first motion goal. The activation is doneby a switch topic. The robot specifications are also saved in the filePKG DIR/FTS/env GUI.yamlwhich can be loaded by the button ”Loadscenario” in the main window of the GUI.

6.5 Setup simulation or experiment

While the planner is ready to send goals to the robots in simulationor experiment, the simulation environment needs first to be set up orthe robots have to be prepared before. We assume that the map of theworld has been generated before starting a simulation or experiment,such that we can perform in a known workspace. While setting up thesimulation is done without human intervention, set up an experimenttakes more time and it is crucial that the workspace is exact as possibleas mapped. A data logger writes gathered messages in a logging file ifdesired for later data analyses.

6.5.1 Simulation

The simulation is performed in Gazebo. The GUI takes all informa-tion previous selected and provided. With this information the chosenworld is loaded in Gazebo and the robots are spawned. Meanwhile a


GraphicalUser

Interface

Planner_1 Robot_1

Task

Plan,Current goal

Current goal

Pose

Planner_2 Robot_2

Task

Plan,Current goal

Current goal

Pose

Planner_N Robot_N

Task

Plan,Current goal

Current goal

Pose

Figure 6.9: The GUI starts for each robot planner node that subscribesthe user defined task and sends the synthesized plan back while thecurrent goal in the plan is send to the robot.

parameter file is generated for RVIZ which automatically adds topicsand models for visualization depending on selected agent models andnumber of agents. Through this RVIZ visualizes already the most im-portant topics for the agents and there is no need for the user to addthem manually. After everything is set up, the simulation is ready toget started.

6.5.2 Experiment

The robots in our experiments have their own on-board computer whichstarts the drivers of hardware on the robot and sends required infor-mation over wireless network to the users computer. While launchingthe drivers also two parameters need to be given to the robot in or-der that the on-board computer subscribes and publishes topics in theright namespace. One of the required parameters is again the robotname, which should match the name chosen in the GUI and the secondparameter is a boolean, set to True if the robot should use the motioncapture system for localization instead of the amcl-package.

Note if the localization is performed with the amcl-package, the workspaceshould be exact as possible as given in the loaded map, otherwise


the localization is not feasible. Moreover, the robots initial pose mustmatch the initial pose chosen in the GUI, since the amcl-package takesalso measurements of the robots odometry into account. The visual-ization of different topics is again performed with RVIZ.

6.6 Execution and runtime modifications

By pressing the button ”Start” the execution of the simulation or ex-periment starts and the first goal of the plan is send to the agents.The procedure for plan execution is given in Algorithm 9, where af-ter activation the first goal in plan R0 is send. During execution thealgorithm checks if the current goal is reached by calculating the erroramong current pose MP and goal pose. A goal pose is reached if theerror among MP and goal pose is small than goalTol.

Moreover, after the time period of Treplan we check if changes in thewFTS were applied and if with a replanning a more optimal plan canbe found. The used navigation controller from the move base-packageaborts the navigation if the current goal is not reachable (e.g. occu-pied by another robot). However, the goal might be occupied only fora short time period and the robot should restart its navigation con-trol after the current goal is again reachable. Therefore, Algorithm 9checks if the robot is moving by keeping track of its current pose MP .The costmap used by the navigation controller is cleared and the goalresend if the robot was not moving longer than a predefined time tol-erance timeTol.


Algorithm 9: Plan execution by hybrid controlInput: Robot pose MP(t), plan Rt, boolean active, Ξt, ℵt, ζt, Γ t,

Ct, AP tΨ

1 if active then2 Send first goal q′g in R0 to navigation controller3 MP = MP(t)

4 t = t

5 treplan = t

6 while active do7 if (t− treplan < Treplan) and (|Ξt|+ |ℵt| > 0) then8 Rt+ = OptPlan(Atp)9 Rt+ = RunTemp(Atp, Rt+ , Γ t, AP t

Ψ , Ct)10 treplan = t

11 end12 poseError = getPoseError(MP(t), q′g)13 if poseError < goalTol then14 Send next goal q′g in Rt

15 if q′g ∈ ζt then16 removePropostion()17 end18 end19 if MP 6=M P(t) then20 t = t

21 end22 if (t− t) > timeTol then23 call Service clearCostmap()24 resend current goal q′g25 end26 MP = MP(t)

27 end28 end

The whole structure of the program is given in Figure 6.10.The data logger can be activated and deactivated by the button

”Record rosbag”.


GraphicalUser

Interface

Soft NBA

FTS

Hard NBAProduct BA

Initial Plan

Updated Plan

Next Move

1 2

3

4

56

7 4

8

9

Temporary Task

Figure 6.10: Data flow among Graphical User Interface and plannernode. Input data comes from the GUI, shaped in the rounded rect-angular; internal variables are in rectangles; output data is in paral-lelogram; numbers on the arrows are algorithms: 1. LTL-formulasto Buchi automaton; 2. automaton intersection by Definition 3.1.5; 3.product automaton by Definition 3.1.6; 4. plan synthesize by Algo-rithm 3 of [16]; 5. transition system update by Algorithm 4; 6. productautomaton update by Algorithm 6 of [16]; validate and revise plan byAlgorithm 5; 8. replanning for temporary task by Algorithm 3; 9. planexecution by Algorithm 9.


Figure 6.11: The dialog window for defining φnew. The task φtemp =

� (r03 ∧ � (r05 ∧ � (r04))) shall be done in a desired time Td = 60.0s.

6.6.1 Temporary task

While an agent is executing its plan, it might receive temporary tasks,as introduced in chapter 4. A new temporary task can be send to theagent by clicking on the button ”Temporary task”. A new dialog win-dow will pop up, where the task φnew can be defined. On the top ofthe new dialog window φtemp can be defined and beneath the desiredtime Td,new to finish the new task can be inserted. With the button ”Addeventually” the task sequence is extended and with the button ”Removeeventually” the task sequence is shortened. If everything is definedproperly the temporary task φnew is send to the agent by pressing thebutton ”Send temporary task”. The button ”Cancel” closes the dialogwithout sending any temporary task.

6.6.2 Workspace update

The button ”Change FTS” allows the user to change the wFTS whileagents are executing their plans. A dialog window pop up similar tothe window for selecting the wFTS in section 6.2. The difference isthat if the user applies changes in the wFTS the GUI sends a Senset

message to the agents as defined in chapter 5. By receiving the Senset

message the agents run Algorithm 6 to revise or replan the plan.

Chapter 7

Case studies

In this chapter three case studies are presented. The first case studyis about a single robot receiving multiple temporary tasks during itsplan execution. The second case study is about changes in the wFTS,while executing a temporary task. The last case study is an experimentin the Smart Mobility Lab (SML) with three agents applying collisionavoidance.

7.1 Hotel scenario: Temporary tasks

In this case study we assume that we are in a hotel environment. Thehotel has a bar r01, a lounge r03, four bedrooms {r13, r17, r18, r26}and three trash rooms {r05, r11, r25}. The map and the chosen wFTSis illustrated in Figure 7.1. We assume that our robot has the capa-bilities to vacuum clean the floor, deliver drinks from the hotel bar tobedrooms and it can remove trash from bedrooms.

Let us consider a scenario, where the agent has the task to vacuumclean the corridors, but is not allowed to enter the lounge area, becausethis would disturb the customers. While the robot is vacuum cleaning,it is ready to receive a drink delivery task to a bedroom, or a trash re-moving task. If a customer in a bedroom orders a drink the robot goesto the hotel bar to get the drink and then moves to the bedroom. Whena bedroom requires trash removing, the robot goes to the bedroom topick up the trash and unloads the thrash in a trash room.

With this scenario we evaluate the theory and algorithms from chap-ter 4. The agent will receive four temporary tasks during the executionof its initial task and adapts its path to fulfill all tasks.

63

64 CHAPTER 7. CASE STUDIES

Figure 7.1: Blue: Hotel bar; red: Hotel lounge; green: trash rooms;purple: bedrooms.

The LTL-formulas for the initial vacuum cleaning task is then:

φhard = �¬ lounge

φsoft = (� � r27) ∧ (� � r06) ∧ (� � r21)

The initial pose of the robot is r28 next to the hotel bar. The initialsynthesized path τ 0 is

τ 0 =(r28)(r27)(r24)(r22)(r10)(r08)(r06)(r08)(r10)(r12)(r14)

(r16)(r19)(r20) [(r21)(r20)(r19)(r16)(r14)(r12)(r10)(r22)

(r24)(r27)(r24)(r22)(r10)(r08)(r06)(r08)(r10)(r12)(r14)

(r16)(r19)(r20)(r21)]ω,

(7.1)

which is basically a surveillance task among the poses r06, r27 and r21.During the initial path execution the robot receives a drink order frombedroom r17, while the robot is on the edge between r24 and r22. Therobot’s trajectory before receiving the first temporary task is shown inFigure 7.2. The received temporary task is:

φ1 = (φtemp,1 = � (drinks ∧ � r17), Td,1 = 300s, T0,1 = 105s)

CHAPTER 7. CASE STUDIES 65

Figure 7.2: In black the trajectory until the robot receives the first tem-porary task φ1 at time t = 105s.

The desired time span to deliver a drink to a bedroom is 300s andthe task was received at time T0,1 = 105s.

After receiving a new temporary task the robot adapts its plan, suchthat it delivers first the drink and then continues with its initial task.The updated path from its current pose r22 is then

τ t =(r22)(r24)(r27)(r02)(r01)(r02)(r27)(r24)(r22)(r10)(r12)

(r14)(r16)(r17)(r16)(r14)(r12)(r10)(r08)(r06)(r08)(r10)

(r12)(r14)(r16)(r19)(r20) [(r21)(r20)(r19)(r16)(r14)(r12)

(r10)(r22)(r24)(r27)(r24)(r22)(r10)(r08)(r06)(r08)(r10)

(r12)(r14)(r16)(r19)(r20)(r21)]ω,

(7.2)

where the blue path accomplishes the new temporary task.While the robot is executing its current task, the robot receives a

second temporary task φ2. The second temporary task is again a drinkorder, but this time from bedroom r26. So the second temporary taskis:


The desired time span to deliver the drink is again 300s and thetask was received at time T0,new = 176s. At this time the robot is on the


Figure 7.3: Black: The trajectory t < T0,1; Blue: The trajectory T0,1 <

t < T0,2 until the robot received φ2.

edge between r02 and r27, coming back from the hotel bar, where itpicked up the drink for room r17. The trajectory before receiving φ2 isshown in Figure 7.3. The new path from r27 after receiving the secondtemporary φ2 task is then

τ t =(r27)(r02)(r01)(r02)(r27)(r24)(r26)(r24)(r22)(r10)(r12)

(r14)(r16)(r17)(r16)(r14)(r12)(r10)(r08)(r06)(r08)(r10)

(r12)(r14)(r16)(r19)(r20) [(r21)(r20)(r19)(r16)(r14)(r12)

(r10)(r22)(r24)(r27)(r24)(r22)(r10)(r08)(r06)(r08)(r10)

(r12)(r14)(r16)(r19)(r20)(r21)]ω,

(7.3)

where the path in cyan is to finish the second temporary task φ2 andin blue the first temporary task φ1. One can see, that the robot accom-plishes first the second temporary task φ2. Therefore, it goes back tothe hotel bar to pick up the second order of drinks and does then thedeliveries.

At time t = T0,3 = 293s the robot receives the task φ3 to remove thetrash in bedroom r13 in a desired time of 600s. Therefore, the thirdtemporal task is:

φ3 = (φtemp,3 = � (r13 ∧ � trash), Td,3 = 600s, T0,3 = 293s)



t < T0,2; Cyan: The trajectory T0,2 < t < T0,3 until the robot receivedφ3.

At time t = T0,3 the robot has already finished temporary task num-ber 2 and is on the edge between r22 and r10, as shown in Figure 7.4.The path is updated to

τ t =(r10)(r12)(r14)(r16)(r17)(r16)(r14)(r12)(r13)(r12)(r10)

(r11)(r10)(r08)(r06)(r08)(r10)(r12)(r14)(r16)(r19)(r20)

[(r21)(r20)(r19)(r16)(r14)(r12)(r10)(r22)(r24)(r27)(r24)

(r22)(r10)(r08)(r06)(r08)(r10)(r12)(r14)(r16)(r19)(r20)

(r21)]ω,

(7.4)

One can see, that the agent first fulfills the delivery task φ1 (blue), sinceit was received earlier and has shorter desired time span to fulfill thetask than φ3. Afterwards the robot removes the trash from bedroomr13 (green).

The last temporary task of the scenario is send when the robot is onthe edge between r06 and r08, as shown in Figure 7.5. The forth taskφ4 is the same as the first task, but received at time t = T0,4 = 473s:




t < T0,2; Cyan: The trajectory T0,2 < t < T0,3; Green: The trajectoryT0,3 < t < T0,4 until the robot received φ4.

The path is updated to

τ t =(r10)(r22)(r24)(r27)(r02)(r01)(r02)(r27)(r24)(r26)(r10)

(r11)(r10)(r08)(r06)(r08)(r10)(r12)(r14)(r16)(r19)(r20)

[(r21)(r20)(r19)(r16)(r14)(r12)(r10)(r22)(r24)(r27)(r24)

(r22)(r10)(r08)(r06)(r08)(r10)(r12)(r14)(r16)(r19)(r20)

(r21)]ω,

(7.5)

Note that the hard-LTL-specifications are satisfied. The robot makesits way through the corridor, even though it would be a shorter wayfor the robot to reach the hotel bar through the lounge to pick up adrink. Given that, one can see that the specifications are also satisfiedfor temporary tasks. The trajectory is shown in Figure 7.6. The simu-lation is stopped after the robot finished task φ4.



t < T0,2; Cyan: The trajectory T0,2 < t < T0,3; Green: The trajectoryT0,3 < t < T0,4; Magenta: The trajectory T0,4 < t after receiving φ4.

7.2 Hotel scenario: Temporary task with workspaceadaption

In this scenario we consider the same workspace as in our previousscenario in section 7.1. In addition the initial task stays also the samewhich is to vacuum clean the corridors and never enter the hotel lounge.In this scenario is shown that a temporary task will also be accom-plished after changes in the wFTS were applied. Since we have thesame initial task as in section 7.1, also the initial path τ 0 is the same asgiven in eq. (7.1). The robot starts its plan execution and shortly after,it receives a temporary task φnew to remove trash from bedroom r13.When the robot received the temporary task it was between pose r27

and r24 at time t = T0,new = 43s and the received temporary task isgiven as

φnew = (φtemp,new = � (r13 ∧ � trash), Td,new = 600s, T0,new = 43s)

and the path is adapted to


τ t =(r24)(r22)(r10)(r12)(r13)(r12)(r10)(r11)(r10)(r08)(r06)

(r08)(r10)(r12)(r14)(r16)(r19)(r20) [(r21)(r20)(r19)(r16)

(r14)(r12)(r10)(r22)(r24)(r27)(r24)(r22)(r10)(r16)(r19)

(r08)(r06)(r08)(r10)(r12)(r14)(r20)(r21)]ω,

(7.6)

where the blue path in τ t is the path to finish the temporary task φnew.While the robot is on the edge between r22 and r10 it receives the in-formation, that the entrance to trash room r11 is blocked. The infor-mation is given as Senset, where no new poses are added or removed,but E¬ contains the edge among r10 and r11. Since this edge is in thecurrent path, the robot applies a replanning which results in

τ t =(r10)(r12)(r13)(r12)(r10)(r22)(r24)(r25)(r24)(r22)(r10)

(r08)(r06)(r08)(r10)(r12)(r14)(r16)(r19)(r20) [(r21)(r20)

(r19)(r16)(r14)(r12)(r10)(r22)(r24)(r27)(r24)(r22)(r10)

(r16)(r19)(r08)(r06)(r08)(r10)(r12)(r14)(r20)(r21)]ω

(7.7)

One can see that the robot found his path to another trash roomr25, since trash room r11 is blocked. Note the replanning was appliedimmediately after receiving the information about the wFTS change,since through the wFTS update the path became invalid. When therobot reaches the bedroom r13, it receives the information that thetrash room r11 is accessible again. The robot is receiving a Senset

information, where E contains the edge between r10 and r11. Therobot’s path is not adapted immediately, since the current path is stillvalid. As introduced in Algorithm 9 the robot checks if the workspacehas changed frequently after a period of Treplan. In this case Treplanis Treplan = 10s. So after the time period Treplan the robot starts a re-planning of the current path, since the wFTS has changed. After thereplanning the robot’s path is

τ t =(r12)(r10)(r11)(r10)(r08)(r06)(r08)(r10)(r12)(r14)(r16)

(r19)(r20) [(r21)(r20)(r19)(r16)(r14)(r12)(r10)(r22)(r24)

(r27)(r24)(r22)(r10)(r08)(r06)(r08)(r10)(r12)(r14)(r16)

(r19)(r20)(r21)]ω.

(7.8)


Figure 7.7: Blue: Hotel bar; red: Hotel lounge; green: trash rooms;purple: bedrooms.

Through the replanning the path of robot is reoptimized and itspath is again to unload the trash in trash room r11. The whole robottrajectory is shown in Figure 7.7. The Simulation is stopped when therobot reaches pose r06.

7.3 Experiment: Multiple robots with colli-sion avoidance

The experiment was carried out in the Smart Mobility Lab at KTH. Theagents are three TurtleBots, where all TurtleBots use an Astra Orbbecdepth camera1 for localization. The three TurtleBots have different on-board computers. The first TurtleBot is equipped with a Jetson TX12,the second TurtleBot uses a laptop with 2.60GHz quadro core andTurtleBot three has an Intel-Nuc3 as on-board computer. On the on-board computer of each TurtleBot the Kobuki base and camera driversare running. Information are send over a 2.5GHz wireless network to

1Product webpage: https://orbbec3d.com/product-astra/2Product webpage: https://developer.nvidia.com/embedded/buy/

jetson-tx13Product webpage: https://www.intel.com/content/www/us/en/

products/boards-kits/nuc/mini-pcs.html

https://orbbec3d.com/product-astra/

https://developer.nvidia.com/embedded/buy/jetson-tx1

https://developer.nvidia.com/embedded/buy/jetson-tx1

https://www.intel.com/content/www/us/en/products/boards-kits/nuc/mini-pcs.html

https://www.intel.com/content/www/us/en/products/boards-kits/nuc/mini-pcs.html


Figure 7.8: Map with the wFTS for the experiment. The rectangle hasthe size 4m× 6m.

the desktop PC. The workspace for the experiment is a simple rectan-gle (4m× 6m) with a wFTS shown in Figure 7.8.

TurtleBot1 has initial pose r06, TurtleBot2 has initial pose r09 andTurtleBot3 has initial pose r02. In this experiment we considered onlyhard-task specifications, which are:

TurtleBot1 performs a surveillance task among pose r06 and r03 andhas to avoid pose r02. The LTL-formula for the task is then:

φ1 = (� � r06) ∧ (� � r03) ∧ (�¬ r02)

The initial path for TurtleBot1 is then synthesized as:

τ 01 = (r06)(r05)(r04) [(r03)(r04)(r05)(r06)(r05)(r04)]ω

TurtleBot2 performs a surveillance task among pose r09 and r01 andhas to avoid pose r02. The LTL-formulas for the task is then:

φ2 = (� � r09) ∧ (� � r01) ∧ (�¬ r02)


τ 02 = (r09)(r08)(r07) [(r01)(r07)(r08)(r09)(r08)(r07)]ω


TurtleBot3 performs a surveillance task among pose r01 and r03. TheLTL-formula for the task is then:

φ3 = (� � r01) ∧ (� � r03)


τ 03 = (r02)(r01) [(r03)(r02)(r01)(r02)]ω

One can see, that the three paths of the TurtleBots have some over-lapping goal poses. If two TurtleBots have the same goal pose at thesame time, a collision avoidance maneuver is executed. Moreover,during the experiment a human walks in the workspace and the Turtle-Bots avoid the human, while executing their path plan. A video ofthe experiment can be found under https://www.youtube.com/watch?v=CftcY3nMpn0&t=169s.

https://www.youtube.com/watch?v=CftcY3nMpn0&t=169s

https://www.youtube.com/watch?v=CftcY3nMpn0&t=169s

Chapter 8

Conclusion

In this thesis the state of the art theory for robot motion planning un-der LTL-formulas is extended with temporary tasks. A frame work fortemporary tasks was presented, which allows to send LTL-formulas assequence of eventually to an agent. Moreover, a new cost function wasprovided which allows to give different temporary tasks different pri-orities. In addition, a Graphical User Interface for robot motion plan-ning is provided. The user is able to define weighted finite transitionsystems by clicking in a map and save it for later use. Different agenttypes can be added and for each agent, its task specifications can beadded in a convenient manner. The presented GUI allows also to sim-ulate changes in the workspace. The algorithms were validated withsimulations and experiments in the Smart Mobility Lab. We showedthat the agents are able to execute their motion plan successfully whileapplying collision avoidance among other agents or even to avoid hu-man beings. Furthermore, the agent were able to detect if the currentgoal was occupied. The agent was waiting until the current goal wasagain accessible to continue its motion plan.

75

Chapter 9

Future work

The current state of the Graphical User Interface allows the user toperform basic robot motion and task planning. However, a fully robotaction model as introduced in [28], where the wFTS is extended to afull robot action model. This action model allows to include specificactions of the robot in the task and motion planning. This would be anuseful extension of the current GUI.

Another feature to add to the GUI would be the global task plan-ning as shown in [17], where a global task is defined for a multi-agentsystem and then the global task is distributed to the agents in smallerlocal tasks.

Future work for online task adaption could be to extend the currentframe work, which allows only LTL-formulas as sequence of eventu-allies. More useful is if the robot is able to receive all different kindof LTL-formulas which can be defined by the introduced semantics insection 3.1.2. However, by including all semantics in online plan adap-tion needs to find a efficient way to update the non-deterministic Buchiautomaton and the the product automaton in a second step.

77

Bibliography

[1] G. E. Fainekos, H. Kress-Gazit, and G. J. Pappas, “Temporal logicmotion planning for mobile robots,” in Robotics and Automation,2005. ICRA 2005. Proceedings of the 2005 IEEE International Confer-ence on. IEEE, 2005, pp. 2020–2025.

[2] ——, “Hybrid controllers for path planning: A temporal logic ap-proach,” in Decision and Control, 2005 and 2005 European ControlConference. CDC-ECC’05. 44th IEEE Conference on. IEEE, 2005,pp. 4885–4890.

[3] M. Kloetzer and C. Belta, “Ltl planning for groups of robots,” inNetworking, Sensing and Control, 2006. ICNSC’06. Proceedings of the2006 IEEE International Conference on. IEEE, 2006, pp. 578–583.

[4] P. Tabuada and G. J. Pappas, “Linear time logic control of discrete-time linear systems,” IEEE Transactions on Automatic Control,vol. 51, no. 12, pp. 1862–1877, 2006.

[5] M. Kloetzer and C. Belta, “Automatic deployment of distributedteams of robots from temporal logic motion specifications,” IEEETransactions on Robotics, vol. 26, no. 1, pp. 48–61, 2010.

[6] H. Kress-Gazit, G. E. Fainekos, and G. J. Pappas, “Where’s waldo?sensor-based temporal logic motion planning,” in Robotics andAutomation, 2007 IEEE International Conference on. IEEE, 2007,pp. 3116–3121.

[7] ——, “Temporal-logic-based reactive mission and motion plan-ning,” IEEE transactions on robotics, vol. 25, no. 6, pp. 1370–1381,2009.

[8] T. Wongpiromsarn, U. Topcu, and R. M. Murray, “Receding hori-zon control for temporal logic specifications,” in Proceedings of the

79

80 BIBLIOGRAPHY

13th ACM international conference on Hybrid systems: computationand control. ACM, 2010, pp. 101–110.

[9] S. L. Smith, J. Tumova, C. Belta, and D. Rus, “Optimal path plan-ning under temporal logic constraints,” in Intelligent Robots andSystems (IROS), 2010 IEEE/RSJ International Conference on. IEEE,2010, pp. 3288–3293.

[10] ——, “Optimal path planning for surveillance with temporal-logic constraints,” The International Journal of Robotics Research,vol. 30, no. 14, pp. 1695–1708, 2011.

[11] A. Ulusoy, S. L. Smith, X. C. Ding, and C. Belta, “Robust multi-robot optimal path planning with temporal logic constraints,” inRobotics and Automation (ICRA), 2012 IEEE International Conferenceon. IEEE, 2012, pp. 4693–4698.

[12] M. Guo, K. H. Johansson, and D. V. Dimarogonas, “Revising mo-tion planning under linear temporal logic specifications in par-tially known workspaces,” in Robotics and Automation (ICRA),2013 IEEE International Conference on. IEEE, 2013, pp. 5025–5032.

[13] M. Guo and D. V. Dimarogonas, “Reconfiguration in motion plan-ning of single-and multi-agent systems under infeasible local ltlspecifications,” in Decision and Control (CDC), 2013 IEEE 52nd An-nual Conference on. IEEE, 2013, pp. 2758–2763.

[14] M. Guo, K. H. Johansson, and D. V. Dimarogonas, “Motion andaction planning under ltl specifications using navigation func-tions and action description language,” in Intelligent Robots andSystems (IROS), 2013 IEEE/RSJ International Conference on. IEEE,2013, pp. 240–245.

[15] M. Guo and D. V. Dimarogonas, “Distributed plan reconfigura-tion via knowledge transfer in multi-agent systems under localltl specifications,” in Robotics and Automation (ICRA), 2014 IEEEInternational Conference on. IEEE, 2014, pp. 4304–4309.

[16] ——, “Multi-agent plan reconfiguration under local ltl specifica-tions,” The International Journal of Robotics Research, vol. 34, no. 2,pp. 218–235, 2015.

BIBLIOGRAPHY 81

[17] P. Schillinger, M. Burger, and D. V. Dimarogonas, “Decompositionof finite ltl specifications for efficient multi-agent planning,” inDistributed Autonomous Robotic Systems. Springer, 2018, pp. 253–267.

[18] P. Schillinger, M. Burger, and D. V. Dimarogonas, “Multi-objectivesearch for optimal multi-robot planning with finite ltl speci-fications and resource constraints,” in Robotics and Automation(ICRA), 2017 IEEE International Conference on. IEEE, 2017, pp.768–774.

[19] M. Guo, S. Andersson, and D. V. Dimarogonas, “Human-in-the-loop mixed-initiative control under temporal tasks,” arXivpreprint arXiv:1802.06839, 2018.

[20] J. Sola Ortega, “Quaternion kinematics for the error-state kf,”2016.

[21] C. Belta, A. Bicchi, M. Egerstedt, E. Frazzoli, E. Klavins, and G. J.Pappas, “Symbolic planning and control of robot motion [grandchallenges of robotics],” IEEE Robotics & Automation Magazine,vol. 14, no. 1, pp. 61–70, 2007.

[22] C. Baier and J. Katoen, Principles of Model Checking. MIT Press,2008.

[23] M. Kloetzer and C. Belta, “A fully automated framework forcontrol of linear systems from ltl specifications,” in InternationalWorkshop on Hybrid Systems: Computation and Control. Springer,2006, pp. 333–347.

[24] G. E. Fainekos, A. Girard, H. Kress-Gazit, and G. J. Pappas, “Tem-poral logic motion planning for dynamic robots,” Automatica,vol. 45, no. 2, pp. 343–352, 2009.

[25] D.-T. Lee and B. J. Schachter, “Two algorithms for constructing adelaunay triangulation,” International Journal of Computer & Infor-mation Sciences, vol. 9, no. 3, pp. 219–242, 1980.

[26] P. Gastin and D. Oddoux, “Fast ltl to buchi automata translation,”in Computer Aided Verification, G. Berry, H. Comon, and A. Finkel,Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2001, pp.53–65.

82 BIBLIOGRAPHY

[27] A. Ulusoy, S. L. Smith, X. C. Ding, C. Belta, and D. Rus, “Opti-mality and robustness in multi-robot path planning with tempo-ral logic constraints,” The International Journal of Robotics Research,vol. 32, no. 8, pp. 889–911, 2013.

[28] M. Guo and D. V. Dimarogonas, “Bottom-up motion and task co-ordination for loosely-coupled multi-agent systems with depen-dent local tasks.” in CASE, 2015, pp. 348–355.

TRITA TRITA-EECS-EX-2018:407

ISSN 1653-5146

www.kth.se

Documents

A Graphical User Interface for the Hybrid Control of Multi ...1249463/FULLTEXT01.pdf · Medan en agent utfor sin r¨ orelseplan, kan¨ andringar¨ i det viktat ﬁnita overg¨ angssystemet