SCIENCE CHINAInformation Sciences

July 2018, Vol. 61 079201:1–079201:3

https://doi.org/10.1007/s11432-017-9336-x

c© Science China Press and Springer-Verlag GmbH Germany, part of Springer Nature 2018 info.scichina.com link.springer.com

. LETTER .

Multi-robot coordinated exploration of indoor

environments using semantic information

Guangsheng LI1*, Wusheng CHOU1,2,3 & Fang YIN2

1School of Mechanical Engineering and Automation, Beihang University, Beijing 100191, China;2State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, China;

3Beijing Key Laboratory of Advanced Nuclear Materials and Physics, Beihang University, Beijing 100191, China

Received 8 August 2017/Accepted 30 November 2017/Published online 13 June 2018

Citation Li G S, Chou W S, Yin F. Multi-robot coordinated exploration of indoor environments using semantic

information. Sci China Inf Sci, 2018, 61(7): 079201, https://doi.org/10.1007/s11432-017-9336-x

Dear editor,Multi-robot exploration of indoor environments isa fundamental problem in mobile robotics [1]. Inthe context of multi-robot exploration, the mainchallenge is in achieving coordination among therobots such that they can be better distributedover the environment for simultaneously explor-ing different regions and avoiding interference withother robots [2, 3]. To date, many approacheshave been proposed to address this issue. How-ever, most of these approaches are only focused onthe applications that assume zero prior knowledgeabout the environments, and very limited studiesconsider the additional background knowledge orany assumptions about the environmental struc-ture. In general, indoor environments constructedby humans often contain certain structures, whichare presented in terms of typical semantic conceptslike corridors, doorways, offices, seminar rooms.By taking advantage of such rich semantic infor-mation, robots can perform exploration tasks moreefficiently.

In this study, we present a novel frontier-basedexploration strategy to coordinate multiple mobilerobots for exploring indoor environments, whichaim to minimize the time needed to explore thewhole environment. Our coordination technique ismotivated by the studies of [4,5], in which Ref. [4]gives a rigorous proof that by maximizing the ex-

pected utility and by minimizing the potential foroverlap in information gain, targets can be ap-propriately assigned to the individual robots, andRef. [5] shows that the supervised learning ap-proaches could be used to estimate the backgroundknowledge about environment structure and incor-porate it into the target assignment procedure.

Proposed multi-robot exploration strategy. Toassign appropriate target frontiers for the individ-ual robots, we present a decision-theoretic explo-ration approach for explicitly coordinating the mo-bile robots based on the iteratively evaluating anumber of target frontiers according to the costsfor moving the target frontiers and their utilitiesduring the target frontier assignment procedure,which determines where the robots would movenext. Furthermore, we take into account the se-mantic information of target frontiers and inte-grate this knowledge into the utility functions tohelp the robots obtain a higher reward for explor-ing non-corridor places, such as offices, seminarrooms, and study rooms. As a result, the robotshave higher priority to thoroughly explore theserooms and will not have to return to the previouslyexplored places, which leads to a shorter overallexploration trajectory and thus reduces the totalexploration time.

Figure 1 illustrates the schematic view of ourproposed multi-robot exploration system, which

*Corresponding author (email: liguangsheng10@163.com)

Li G S, et al. Sci China Inf Sci July 2018 Vol. 61 079201:2

Semantic classification of indoor places Estimating semantic

classification

Candidate target frontiers determination

Indoor environment

DBSCAN

Semantic classification

of candidate target

frontiers

55

55

224

55

55

C1:96@55×55

C2:256@27×27

27

27

27

27

13

13

C3:384@13×13

13

13

13

C4:384@13×13

13

13

13

13

C5:256@13×13

13

13

F1:4096

F2:4096

8 8

2048 2048

8

C2:22555

5

55

55555555555555

2244

11

11

33

33

33

33

33

33

33

11

11

55

55

33

20444442044444 20444442044444

Stride of 4

Kernel

Doorway

Office Study room

CorridorSeminar room

Great hall

Laboratory

Drinking room

Semantic classification

of robot’s current

position

Camera

LIDAR

Semantic label

(probability distribution)

Frontier generation Candidate target frontiers

The frontier-based multi-robot target assignment strategy

Hidden Markov model

(HMM )

Predicting

Utility function

Utility

Revenue function U

s

Wireless networkWireless networkWireless network

Robot1Robot2

Robot3

Grid map of indoor environment

Robot2

Semantic parameter

γ

Explored space

Unexplored space

Frontiers

Robot

Wall

F2F3 F4

F5 F6

F7

F8

Cost function

(a)

(b)

(c)

(d) (e)

(f)

(g)

CNN-based classifier

Ucost( )utility cost

1

=argmax γ=

⋅ −

∑m

j

U U U

Figure 1 (Color online) Illustration of the architecture of our multi-robot exploration system. (a) Typical indoor envi-ronment with different semantic classifications; (b) Kinect and laser range scanner; (c) CNN-based classifier for semanticclassification of indoor places; (d) candidate target frontiers decision-making process with VDBSCAN; (e) hidden Markovmodel for estimating the semantic classification; (f) our proposed target frontier assignment strategy; (g) the generatedgrid-map.

primarily consists of four phases outlined below.

(1) Candidate target frontiers determination.To limit the number of target frontiers and gener-ate the representative candidate target frontiers,varied density based spatial clustering of appli-cations with noise (VDBSCAN) algorithm [6] isapplied to generate frontier clusters according tothe frontier distribution density. By using DB-SCAN, the contiguous or neighboring frontiers canbe identified or grouped into the same frontier clus-ter, and the centroid of this frontier cluster is con-sidered as a candidate target frontier. The detailson the determination of candidate target frontierscan be found in Appendix A.

(2) Semantic classification of indoor places. Forthe semantic classifications of the indoor placeswhere the robots are located, a classifier trainedwith convolutional neural network (CNN) is ap-plied to determine the semantic types based onvisual observations (panoramic scans) and returnthe semantic labels during exploration. In thisstudy, we consider the indoor environment witheight place classifications, the typical instancesof which are shown in Figure 1(a). The detailsabout the CNN-based classifier are presented inAppendix B.

(3) Estimating the semantic classification ofcandidate target frontier. The CNN-based clas-sifier is only able to classify a single observation ofa robot’s current position and without taking intoaccount the previous classification results. Whenrobots move through an environment, however, thesemantic classifications between nearby positionsmay change according to a certain sequence andthe transition is most likely due to the structureof man-made environments. Therefore, speculat-ing about the semantic classification of the nexttarget frontier by taking advantage of the spatialdependency of the previous semantic classificationresults can lead to a high classification accuracy.

To this end, we introduce a hidden Markovmodel (HMM) for estimating the semantic type oftarget frontier as suggested by Rottmann et al. [7].In HMM, we maintain a posterior Bel(Lx) aboutthe semantic type Lx of the place x where therobot is currently in, which can be expressed as

Bel(Lx)

= α ·P (zx|Lx)·∑

Lx−1

P (Lx|Lx−1)·Bel(Lx−1), (1)

where α is a normalizing constant, zx is the clas-sification output of the CNN-based classifier from

Li G S, et al. Sci China Inf Sci July 2018 Vol. 61 079201:3

the place x, P (zx|Lx) is the observation model andP (Lx|Lx−1) is the transition model. These twomodels are detailed in Appendix C.

(4) Candidate target frontier assignment for co-ordinating multi-robot exploration. The cost ofreaching a target frontier is given by the minimumtraveling distance from the robot’s position to thetarget frontier, which is defined as

Ucost(i, t) = Distance(i, t), (2)

where Distance(i, t) indicates the Euclidean dis-tance between the target frontier t and the roboti. In this study, we compute the optimal path us-ing the A* algorithm.

To estimate the utility of a target frontier, weuse the same technique described in [4], whichcomputes the expected utility based on distanceand visibility. The utility of the target frontier tnwith respect to the robot i is defined as follows:

Uutility(i, t) = Uinitial(i, t)−n−1∑

j=1 j 6=i

Pv(d), (3)

Pv(d) =

1−d

R max, if d < R max,

0, otherwise,(4)

where R max is the maximum measuring distanceof the range sensor, d is the Euclidean distance ofthe robot and its target frontier, and Pv(d) is theprobability that frontier t at a distance from targetfrontier t can be observed by other robots.

Furthermore, to integrate the semantic infor-mation into the target frontier assignment proce-dure, we give a higher priority to the target fron-tiers that are located in non-corridor places. Inour current implementation, all non-corridor tar-get frontiers receive a γ times initial utility com-pared to the corridor target frontiers. Therefore,the robots will prefer the target frontiers in non-corridor places and eventually make a thoroughexploration of these places.

Accordingly, the target frontier assignment thatintegrates semantic information for robot i can bedescribed as follows:

U(i, t)

=argmax

m∑

i=1

(

n∑

t=1

(γ ·Uutility(i, t)−Ucost(i, t))

)

, (5)

where γ is the semantic parameter, m and n arethe total number of robots and candidate targetfrontiers, respectively.

By trading off the costs of the target frontiersand their utilities on a global scale, different tar-get frontiers are assigned to the individual robots,such that they are better distributed over the envi-ronment and are not focused on the same regions,

thereby allowing the robots to act in a coordinatedmanner. In addition, we compute a new assign-ment whenever a robot has reached its assignedtarget frontier or whenever the time elapsed aftercomputing the latest assignment exceeds a giventhreshold. The resulting target frontier assign-ment algorithm is given in Appendix D.

Experimental results. We implemented ourmulti-robot exploration system under the robotoperating system (ROS), and conducted a seriesof experiments on real robots. Because of spacelimitations, the detailed results are presented inAppendix E.

Conclusion and future work. This studypresents a frontier-based multi-robot explorationstrategy based on CNN and HMM techniques toaddress the problem of exploring an indoor envi-ronment with mobile robots. The feasibility andefficiency of our presented method is illustrated byreal-world experiments. Our future work will fo-cus on learning semantic labels in an unsupervisedmanner, so that the system might be able to de-termine the semantic types of different places onits own.

Acknowledgements This work was supported by

National Natural Science Foundation of China (Grant

No. 61633002).

Supporting information Appendix A–E. The

supporting information is available online at info.

scichina.com and link.springer.com. The supporting

materials are published as submitted, without type-

setting or editing. The responsibility for scientific ac-

curacy and content remains entirely with the authors.

References

1 Stachniss C. Robotic Mapping and Exploration.Berlin: Springer, 2009

2 Stachniss C, Mozos O M, Burgard W. Efficient explo-ration of unknown indoor environments using a teamof mobile robots. Ann Math Artif Intel, 2008, 52:205–227

3 Fang H, Lu S L, Chen J. New advances in complex mo-tion control for single robot systems and multi-agentsystems. Sci China Tech Sci, 2016, 59: 1963–1964

4 Burgard W, Moors M, Stachniss C, et al. Coordinatedmulti-robot exploration. IEEE Trans Robot, 2005, 21:376–386

5 Obwald S, Bennewitz M, Burgard W, et al. Speeding-up robot exploration by exploiting background infor-mation. IEEE Robot Autom Lett, 2016, 1: 716–723

6 Liu P, Zhou D, Wu N J. VDBSCAN: varied densitybased spatial clustering of applications with noise. In:Proceedings of the International Conference on ServiceSystems and Service Management, Hong Kong, 2007

7 Rottmann A, Mozos O M, Stachniss C, et al. Semanticplace classification of indoor environments with mobilerobots using boosting. In: Proceedings of the 20thNational Conference on Artificial Intelligence, Pitts-burgh, 2005. 1306–1311