Automatic ground truth generation for image sequences

International Journal of Electronics and Communication Engineering & Technology (IJECET), ISSN 0976 –

6464(Print), ISSN 0976 – 6472(Online), Volume 5, Issue 9, September (2014), pp. 24-34 © IAEME

24

AUTOMATIC GROUND TRUTH GENERATION FOR IMAGE SEQUENCES

Shivank Bhagotra

Electronics & Communication Engineering, BKBIET, Pilani, Rajasthan, India

ABSTRACT

This paper presents an approach for generating efficient Ground Truth for image sequences

related to Traffic Signals and Direction Signs by making suitable annotations around the area of

interest and saving corresponding information in the form of XML files. This feature can be readily

used in the cars to detect the speed limit signals and automatically optimize their speed in case of

driverless cars. The Graphic User Interface (GUI) has been designed in GUIDE Matlab and the

Image Database has been provided by the MIPS-MIAM laboratory, Ensisa, Mulhouse, France.

The entire project deals with making a driverless car wherein the real time images as

perceived by the camera placed in front of the car are processedand annotations are formed around

Direction signs and Traffic Speed Limit Signals. The specifications such as Content,

Representativeness, Occlusion, Image Identifier, and Uncertainty get stored in the form of XML

files. This stored data is processed and subsequent changescan be made in the speed, direction of the

car.

Keywords: Annotation, Content, GUIDE, GUI, Ground Truth, XML Files.

1. INTRODUCTION

The annotation tool has been adapted from the software developed at University of Bonn,

Germany, as a general annotation tool. The objective of the project deals with the transformation of

the Tool from the ‘Annotation Tool’ to the ‘Traffic Annotation Tool’ which requires deep

understanding of the Annotation tool environment and the ways callbacks are behaving to make

changes in the tool. Objectives to be achieved are:

i. Addition of static textbox Content and Popup menu associated with it.

ii. Writing the Callback for the popup menu ‘content’ to make it work like other popup menus.

iii. Changes in the popup menu ‘Source’.

iv. Changes in the popup menu ‘Class Name’.

INTERNATIONAL JOURNAL OF ELECTRONICS AND

COMMUNICATION ENGINEERING & TECHNOLOGY (IJECET)

ISSN 0976 – 6464(Print)

ISSN 0976 – 6472(Online)

Volume 5, Issue 9, September (2014), pp. 24-34

© IAEME: http://www.iaeme.com/IJECET.asp

Journal Impact Factor (2014): 7.2836 (Calculated by GISI)

www.jifactor.com

IJECET

© I A E M E



25

2. ANNOTATION TOOL

Image Annotations generally refer to highlighting features, details, or points of

interest within an image. Annotation Tool is a Matlab based image Annotation Tool developed at

University of Bonn for the Etrims image database and this particular Annotation Tool has been

modified to make it suitable to annotate images from the database (Traffic Signs) related to our

project.

We are working upon the Annotation tool version 1.7 as the latest version of the Annotation

tool which is 2.4 had some bugs indeed, the annotations are not being saved properly in the XML

files and the generated files had some errors as all class names were being shown up in the file.

The Annotation Tool originally employed LabelMe Matlab toolbox developed at the Massachusetts

Institute of Technology (MIT) for the LabelMe image database.Here is an example of Annotation

Tool interface where an image from the Etrims Image database is being annotated:

Fig 1: Screenshot of the Annotation Tool

2.1. SETUP

i. Download the Matlab toolbox LabelMe and add it to the Matlab search path, e.g.

• C:\LabelMeToolbox

ii. Download the annotation tool and add it to the Matlab search path, e.g.

• C:\AnnotationTool

iii. Create home folders for your image data and annotations e.g.

• C:\AnnotationTool\home-images

• C:\AnnotationTool\home-annotations

iv. An annotation tool comes along with three text files two of which should contain home folder

paths. Write the image data home directory and the annotations home directory in the

corresponding text files:

• C:\ AnnotationTool\home-images.txt

• C:\AnnotationTool\home-annotations.txt

v. Now we create directories with your image datasets, e.g.:

• C:\AnnotationTool\home-images\image-dataset1



26

• C:\AnnotationTool\home-images\image-dataset2

vi. And copy your image data in this e.g.

• C:\AnnotationTool\home-images\image-dataset1\my-image.jpg

It is obligatory to create the image dataset subdirectory as this folder name will be stored with the

annotation.In Matlab, we can now run AnnotationTool.m after the first image is labelled, the

Annotation is stored in an XML file of the same name as the labelled image. The XML file with

annotation is saved in the annotations home directory of the same name as the image dataset

directory of the labelled image, e.g.

• C:\AnnotationTool\home-annotations\image-dataset1/my-image.xml

3. IMAGE ANNOTATION

3.1. LABELLING IMAGE DATA

Looking at the Data panel in the upper right corner of the (Fig 2).Opening an image dataset

folder by pressing the button Open Image Folder.Then selecting an image in the listbox underneath.

We may now specify the Image sourcein the Current image panel. In case the popup menu does not

offer a relevant option, we may specify an alternative source of the image by choosing the option

‘other’.

After pressing the annotate button on the New Annotation panel the tool will switch to

annotation mode where only image labelling using the mouse is allowed. Annotate by pressing the

left mouse button and clicking in the image area. Pressing the right mouse button will finish the

object labelling and will close the polygon (Fig. 2). If we press the Annotate button now, the

previous label will be erased and we may label the object again. Pressing the right mouse button

without having labelled anything will just cancel the annotation mode. The zoomfeature will ease the

annotation of smaller objects.

Fig 2: Annotating an image object from our database

3.2. OBJECT ANNOTATION

Having labelled an image we may now specify its class, degree of its occlusion,

representativeness and Content (modification) (Fig 3). It is possible to specify an alternative object

class name by choosing the option ‘other’…in the Object class pop up menu. The Annotation tool



27

comes along with three text files, one of which, called object-classes.txt contains a list of object class

names. We can simply extend or change the classes by simply editing the text file.

The next value to be specified is the annotation uncertainty: it is an estimate, in pixels, of

how precise the labelling is. A set of green and red concentric circles is drawn in the middle of the

new object label. A diameter of the red circle in pixels corresponds to the value chosen in the

Annotation Uncertainty popup menu.

And the last value to specify is the Content which has been added by me in the panel, where

the user can enter the value of the speed sign when it is being annotated and the same will appear in

the information about the annotation in the XML file generated. Having specified all the required

information we may add the annotation to the Current annotationslistbox by pressing the Add Object

button (Fig 4). A login that should specify the labeller and the place of annotation is required when

adding the first object. This information is added to the annotation of every labelled object.

(a) (b) (c)

Fig 3: (a) Specifying the Image Source, (B) Class of the Image Object, (C) Degree of Occlusion and

Content

After adding the new annotation to the pool of the Current Annotationwe have the possibility

of deleting it by pressing the Delete Last button.It is only the last added annotation that the labeller is

allowed to erase. Pressing the Save Annotationbutton will save all current annotations to a file.

(a) (b)

Fig 4: (a) Adding an annotated image object to current annotations, (b) A login specifying the

labeller and the place of annotation is required when adding the first object



28

4. ANNOTATION TOOL TRANSFORMATION

• When I added an additional Content Popup menu, as it would be useful for the tool to

have to annotate images from our database which consisted of Speed Limit Signals,

Guide was not generating a corresponding Callback in the Annotation Tool for the

Content Popup menu even after being saved with the same filename. Then after searching

over the internet, the problem was solved as there was a change to be made in the settings

of the Guide.

Fig 5: Changes to be made for Callback

After adding the desired uicontrol to the Guide environment, we need to go to tools>GUI

options (Fig 5). After this, we need to select the radio button for GenerateFIG-file and M-file

(Fig 6).

Fig 6: Selecting Radio button for both files

After the following changes are made, corresponding callbacks for the added uicontrols will

be generated in the .m file of the tool. Now changes can be made in the Callback related to the

uicontrol to make them function accordingly.

• The abling and disabling of the Content popup menu should be the same like the other popup

menus from the tool, meaning it should get activated when annotation is over as the right



29

mouse button is clicked. This was initially not working as there were errors in the Matlab

command window when the code was being run. It was due to static number of properties to

be activated or deactivated fixed to 20 which was prohibiting any addition done to the code

(Fig 7).

Fig 7

Then we set a global variable defines as ‘Handles.NUMCONTROLS’ and set its value to 21

and then we were able to make popup menu ‘Content’ enabled like other popup menus (Fig 8).

Fig 8

Therefore, in order to add any uicontrol to the Annotation Tool code we need to increase the

value for the ‘Handles.NUMCONTROLS’ which is a global variable we have defined at the starting

of the code and make corresponding changes in the code.

Following the above changes, corresponding changes have been made in the subsequent code

and Callback for the content to make it work like the other popup menus and get the value of

‘Content’ in the XML file which is generated after the annotation is saved (Fig 9).

Fig 9



30

5. TRAFFIC ANNOTATION TOOL

Several changes have been made in the Annotation Tool to make it ‘Traffic Annotation Tool’.

• Changes in the popup menu ‘SOURCE’. Change in the source of the image can be made by opening the Property Inspector of

the popup menu ‘Source’and modifying the String section.

(a) (b)

Fig 10: (a) Annotation Tool (b) Traffic Annotation Tool

As we can see modification has been done in the popup menu ‘Source’ according to our

project (Fig 10).

• Changes in the popup menu ‘Class name’. The Annotation Tool comes along with three text files, one of which, called ‘object-

classes.txt’ contains a list of object class names. We can make changes to the set of classes by

simply editing the text file.

Fig 11: Command given in the code to read the text file and load it into the popup menu

As we can see modification has been done in the popup menu ‘Class Name’ according to our

project (Fig 12).

(a) (b)




31

• Adding an additional Static Textbox ‘Content’ and popup menu associated with it. Static textbox ‘Content’ and corresponding popup menu can be added to the interface

when editable guide window for Annotation Tool is opened by the guide(AnnotationTool)

command in the command window (Fig 13).

(a) (b)


• Writing the Callback for the popup menu ‘Content’ to make it work like other popup

menus. Several additions to the code have to be done for adding the Content popup menu and

making it work like the other popup menus. This section has been explained in the later part of

the report (See Fig 9).

Therefore, all the changes have been made successfully and the Annotation Tool is

working correctly. The XML files generated for the annotations have the additional tag called

‘Content’ which specifies the value of the speed limit sign in addition to the Class name,

Occlusion, Representativeness and Annotation Uncertainty.

6. RESULTS

By the end of the project we have following Ground truth which is generated for an

annotation when it is saved and can be retrieved from the XML file generated.

• Image identifier We can determine that from the XML file where it is stored as the filename.

• Position of the top left corner of the elements

We can determine that by having a look at the XML file generated as it contains the coordinates

the points plotted.

• Resolution(in pixels) Image size (Pixel Dimension) as is in the axis where the image is loaded:

Width: 640 pixels

Height: 480 pixels

Therefore, the resolution of the annotation made can be determined by the axis and the

coordinates we have in the XML file.

• Type of Traffic Sign We can determine the type of traffic sign we are annotating as it is loaded in the Class names

Popup Menu and that similar will be reflected in the XML file whether it is a speed limit sign,

danger sign, no parking sign, etc.



32

• Annotation Specifications We have been able to determine Occlusion, Representativeness, Annotation Uncertainty and Content

for a particular traffic signal.

• Creation of an XML File.

Command used = ‘writexml’.It transforms a struct or struct array into an xml file.

Filename = ‘name of xml file’. V= struct variable containing fields of type char or numeric.

Writexml (filename, xml) will generate an .xml file.

All the results obtained can be seen in this example of an annotated image (3257.jpg) from

our database.

Fig 14: Image annotation by the Traffic Annotation Tool

All the ground truth generated for a particular image by the Traffic Annotation Tool can be

seen over the screen once the annotation is saved (Fig 14). The corresponding XML generated is for

the above image and all the results obtained are well depicted in the XML file and are highlighted.

<annotation>

<filename>3257.jpg</filename> <folder>image-dataset2</folder>

<sourceImage>ENSISA_UHA</sourceImage> <rectified>0</rectified>

<viewType>terrestrial</viewType>

<scale>n/a</scale>

<imageWidthHeight>640 480</imageWidthHeight> <transformationMatrix>n/a</transformationMatrix>

<object>

<name>speed limit signals</name> <objectID>73552514801738</objectID>

<occlusion>60</occlusion>

<representativeness>40</representativeness>



33

<uncertainty>n/a</uncertainty>

<Content>50</Content> <deleted>0</deleted>

<verified>0</verified>

<date>18-Oct-2013</date>

<sourceAnnotation>Shivank@UHA</sourceAnnotation>

<polygon>

<pt> <x>577.1029</x># position of the top left corner

<y>223.9076</y>

</pt> <pt>

<x>577.8747</x>

<y>264.8098</y>

</pt>

<pt>

<x>537.7443</x>

<y>264.8098</y>

</pt>

<pt>

<x>537.7443</x>

<y>224.6794</y>

</pt>

<pt>

<x>578.6464</x>

<y>223.1359</y>

</pt>

</polygon>

<objectParts>n/a</objectParts>

</object>

</annotation>

7. CONCLUSION

In this paper, Transformation of the Annotation Tool to the ‘Traffic Annotation Tool’ has

been discussed. The steps to make the transition have been discussed in detail citing all problems

encountered while transforming the code. The corresponding XML file generated contains all the

desired ground truth as our objective in the beginning of the project. Further changes and adoptions

can be made in the code for the Annotation Tool to redesign it according to need of the project. This

work can be further enhanced by designing an interface to incorporate the ground truth generated to

efficiently modify the speed of the car.

8. ACKNOWLEDGEMENT

It gives me immense pleasure to express my deepest sense of gratitude and sincere thanks to

my highly respected and esteemed Mentors Prof. Jean-Philippe Lauffenburger and Jeremie Daniel

(Ecole Nationale Supérieure d’ingénieurs Sud Alsace (ENSISA), MIPS-MIAM Laboratory) for their

valuable guidance, encouragement and help for completing this work.

International Journal of Electronics and Communication Engineering & Technology (IJECET), ISSN 0976

6464(Print), ISSN 0976 – 6472(Online), Volume 5, Issue

REFERENCES

[1] Annotation Tool Documentation:

[http://www.ipb.uni-bonn.de/html_pages_software/annotation

[2] @techreport{annotation-tool

Author = "Kor{\v c}, F. and Schneider, D.",

Title = "Annotation Tool",

Number = "TR-IGG-P-2007-

Month = "June",

Year = "2007",

Institute = "Dept. of Photogrammetry, University of Bonn",

URL = http://www.ipb.unibonn.de/html_pages_software/annotation

P-2007-01.pdf

[3] Bryan C. Russell, Antonio Torralba, Kevin P. Murphy,

‘LabelMe: a database and webbased tool for Image Annotation’

Computer Vision, 2008.

[4] For further information on Etrims image database:

[http://www.ipb.uni-bonn.de/projects/etrims_db/]

[5] Matlab tutorials can be viewed here:

[http://www.mathworks.com/products/matlab/videos.html]

[6] Detailed tutorial on Guide for beginners can be

[http://www.mathworks.com/videos/creating

[7] Abbass.A.Khorsheed, Hussein Chible, Giorgio Giacinto

De-Noising for Image Annotation

Electronics and Communication Engineering &Technology (IJECET), Volume

2014, pp. 113 - 118, ISSN Print: 0976

BIOGRAPHY

Shivank Bhagotra

his Bachelors in Electronics and Communication Engineering from B.K. Birla

Institute of Engineering and Technology, Pilani. He has been a Research Intern

at MIPS-MIAM Laboratories, ENSISA, Mulhous

associated with IEEE for the past 3 years and currently acting as the IEEE

Mentor BKBIET Student Branch.

International Journal of Electronics and Communication Engineering & Technology (IJECET), ISSN 0976

6472(Online), Volume 5, Issue 9, September (2014), pp. 24-34 © IAEME

34

Annotation Tool Documentation:

bonn.de/html_pages_software/annotation-tool/

tool-2007,

v c}, F. and Schneider, D.",

-01",

Institute = "Dept. of Photogrammetry, University of Bonn",

http://www.ipb.unibonn.de/html_pages_software/annotation-tool/publ/Korc

Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, William T. freeman

‘LabelMe: a database and webbased tool for Image Annotation’ for International J

For further information on Etrims image database:

bonn.de/projects/etrims_db/]

Matlab tutorials can be viewed here:

om/products/matlab/videos.html]

Detailed tutorial on Guide for beginners can be viewed on Math Works Website:

[http://www.mathworks.com/videos/creating-a-gui-with-guide-68979.html]

Abbass.A.Khorsheed, Hussein Chible, Giorgio Giacinto and Ammar J. Fattah, “Automatic

or Image Annotation using Latent Semantic Analysis”, International

Electronics and Communication Engineering &Technology (IJECET), Volume

118, ISSN Print: 0976- 6464, ISSN Online: 0976 –6472.

Shivank Bhagotra was born in Jammu, India in 1993. He is currently pursuing



MIAM Laboratories, ENSISA, Mulhouse, France. He has been


Mentor BKBIET Student Branch.


© IAEME

tool/publ/Korc-TR-IGG-

T. freeman MIT, USA

for International Journal of

viewed on Math Works Website:

nd Ammar J. Fattah, “Automatic

International Journal of

Electronics and Communication Engineering &Technology (IJECET), Volume 5, Issue 1,

was born in Jammu, India in 1993. He is currently pursuing



e, France. He has been
