Systems 40Nexys2_rm

8/13/2019 Systems 40Nexys2_rm

1/6

FPGA & DSP infrared image processing module for people andobjects detection

Snejana Pleshkova,Department of Telecommunications

Technical UniversityKliment Ohridski, 8 Sofia

[email protected]

Abstract: Infrared image processing systems are used in the beginning only as visualization systems for observation of people and objects in the night or as night vision systems. Next in these systems are included the blocks and algorithmsfor thermal image processing and these systems became the base of the todays infrared image processing systems.There are many areas of infrared image processing systems applications like industrial control, medical, military and

police systems, etc. One of the topic applications of the infrared image processing systems now are the surveillance andobservation infrared system. The goal of this article is to describe the development and of a module with FPGA (FieldProgrammable Gate Array) and DSP (Digital Signal Processor) for infrared image processing in infrared surveillanceand observation system for customs control and combating terrorism. It is proposed the structure of the module suitablefor capturing and processing thermal images from an infrared image sensor. Some results of testing the proposed FPGAand DSP infrared image processing module are presented with an algorithm developed for people or objects detectionin thermal images.

Key-Words: - infrared visual systems; infrared image processing; FPGA and digital signal processors; people andobjects detection

1 Introduction

Infrared vision systems are the information systemsused in military, police custom traffic control, industrial

and other specific applications for collecting and processing visual information from infrared images [1,2]. They are composed usually with a infrared videocamera [3] for capturing the still or moving thermalimages. The means or tools for development the methodsand application algorithms for infrared captured image

processing can be an appropriate hardware or softwarefor implementation, testing and execution of thesemethods and algorithms [3, 4]. Most of these tools workwell for developing and testing tasks, but for real

practical implementation of the developed methods andalgorithms is necessary to transfer these algorithms inthe special developed for the appropriate task fixed or

programmable modules.The goal of this article is the development of such

module with FPGA (Field Programmable Gate Array)and DSP (Digital Signal Processor) specially designedfor infrared image processing in infrared surveillanceand observation system for customs control andcombating terrorism.

2 Infrared Image Processing Module

Archtecture in a Thermo Vision System

for Objects structure of an intelligentMultimedia Detection A. The place of the of FPGA and DSPs

Infrared Image Processing Module

There are thermo visual systems with image processing modules and embedded digital signal processors for real time object detection [5]. From theschema block of such system (Fig. 1.) is possible todefine or determine in the general case the place of the

proposed infrared image processing module as a part ofan infrared visual system.

IR camera 1 IEEE 1394

IR camera 2

IR camera 3

IEEE 1394

IEEE 1394

PHYLayer

LinkLayer

DigitalSignal

Processor

LCDDisplay

FPGA

EthernetOr

WirelessConnection

Real Time Thermo Vision Module

HOSTController

USB

LAN

WLAN

Figure 1. The schema block of a thermo visual systemwiith embedded digital signal processor for real time objectdetection

Recent Researches in System Science

ISBN: 978-1-61804-023-7 253


2/6

The important requirements in such existingexamples of thermal vision systems are the real timeexecution of the algorithms for infrared image

processing in some practical applications such as objectsor people detection and tracking.

Here in this article the goal of development of aninfrared image processing module is something different- to give the priority of FPGA (Field Programmable GateArray) over DSP (Digital Signal Processor) in thecreating, testing and practical implementations ofalgorithms for infrared image processing tasks.

The differences in implementation of the infraredimage processing algorithms between FPGA (FieldProgrammable Gate Arrays) and DSP (Digital SignalProcessor) are the following:

- the design style in Field Programmable Gate Arrays(FPGA) is more deeply directed to hardware style of

design, opposite to Digital Signal Processors (DSP),where development of the algorithms is realized as a program for the chosen signal processor;

- the development of image processing modules withField Programmable Gate Arrays (FPGA) isaccomplished with standard VHDL or Verilog [7]languages for programmable logic arrays design;) manywell developed image and signal processing libraries forimplementing processing and calculation blocks in FieldProgrammable Gate Array;

- there are for Field Programmable Gate Arrays(FPGA) s (FPGA), which are very easy to use in special

cases of infrared image processing modules.The above mentioned differences between FPGA

(Field Programmable Gate Arrays) and DSP (DigitalSignal Processor) can be used in design of an infraredimage processing module, proposed in this article. Thegeneral requirements in this case are to design theinfrared image processing module with effective usingof the hardware and software resources of FieldProgrammable Gate Arrays (FPGA) and to communicatewith the added Digital Signal Processor (DSP), usingtheir capabilities of fats calculations as a supplementary

possibility to extend the processing speed.

B. The structure of the proposed FPGA and DSP Infrared Image Model

The structure of the proposed infrared image processing module is show on Figure 2.

It can be seen from Figure 2, that these requirementsare satisfied, because the FPGA (Field ProgrammableGate Arrays) is presented as central part of the proposedstructure of the module for infrared image processing.

Infrared ImageSensor

ExpansionInterface

DSP

FPGA

Pixel Data

I2C

Clk

Data

JTAG

EEPROM

EmbeddedTCP/IP

Interface

RJ-45

Wi-FiDSP

Interface

FPGAExternalMemory

DisplayInterface

DSPExternalMemory

LCDDisplay

Figure 2. The structure of the proposed

infrared image processing module

The FPGA (Field Programmable Gate Array) isconnected with the Infrared Image Sensor. This allowsFPGA to receive the Pixel Data of the captured imagesfrom the Infrared Image Sensor and also to control and

adjust the Infrared Image Sensor via I2C Bus.The FPGA is connected also to the DSP (Digital

Signal Processor) via DSP Interface to realize thecommunication with the added Digital Signal Processor(DSP). It is possible to transfer the captured infraredimages as Pixel Data or as intermediate processingimages from the FPGA to DSP. The DSP can preparesome infrared image processing operations or algorithmsand then transfer back the results to the FPGA.

The FPGA (Field Programmable Gate Array) andDSP (Digital Signal Processor) both have the externalmemory block named in Fig.1 as FPGA ExternalMemory and DSP External Memory, respectively. It is

possible to use these two external memories asadditional memory locations to storing input,intermediate or output infrared images in execution ofthe algorithms for infrared images processing in theFPGA or DSP blocks of the proposed module forinfrared image processing.

The rest of the blocks shown in Fig. 1 and included inthe proposed module for infrared image processing arenot directly connected with the executed algorithms forinfrared image processing, but are closely connected andnecessary for the module control of operation, testingand their connections and interfacing with other pars of ainfrared or thermo visual system with a concrete


ISBN: 978-1-61804-023-7 254


3/6

practical application. These additional block shown inFig. 1 in the proposed module for infrared image

processing are the following:- Display Interface for connecting the LCD Display or

monitor useful for input, intermediate or output infraredimages visualization in time of development, testing andexecution of the algorithms for infrared image

processing in the proposed module;- Embedded TCP/IP Interface with wired or wireless

capabilities and two standard connectors RJ-45 for wiredlocal area network and Wi-Fi for wireless networks, bothallow the proposed infrared image processing module tosend or receive the infrared input, processed or outputvisual information with establishing the connectionswith the other parts in an infrared visual system;

- JTAG (Joint Test action Group) interface forstandardized [8] test procedures in the proposed module

for infrared image processing;- EEPROM block for collecting the initial data for all blocks in the proposed module and especially for FPGAand DSP initialization starting an algorithm for infraredimage processing and storing the current data inEEPROM finishing the work of the algorithm in the

proposed module and using the same stored initial datain the next start procedure;

- Expansion Interface for future addition of thesupplementary blocks to the developed module forinfrared image processing.

C. The Implementation of Spartan -3E Xilinx FPGA in Infrared Image Processing Module

The proposed structure of the infrared image processing module, shown in Figure 2, is realized withthe popular and used in many signal and image

processing applications Xilinx FPGA Spartan 3E [9].The main connections for this concrete implementationof the Spartan 3E FPGA in the proposed infraredimage processing module are presented in Figure 3.

Figure 3. The Implementation of Spartan -3E Xilinx FPGA inInfrared Image Processing Module

The capabilities of using the Spartan 3E FPGA ininfrared image processing algorithms and applicationsare the following [10].

- Real-Time Pixel-by-Pixel Processing andTransformation;

- Logical Complexity - around 5,000 Slices or More;- Embedded Memory for Image and look up Table

Storage.With these capabilities of Spartan 3E FPGA is

possible to realize the following real time characteristicsof the proposed infrared image processing module:

- real time capturing of thermal images fromstandard infrared cameras;

- real time interfacing between FPGA, DSP (DigitalSignal Processor) and infrared camera;

- real time infrared image processing in FPGA andDSP (Digital Signal Processor);

-

real time visualization of input, intermediate andoutput infrared images.

D. The development of an algorithm for testing the proposed structure of the module with implementation ofSpartan -3E Xilinx FPGA in infrared image pocessing

for object detection

The structure of the proposed module for infraredimage processing, show as block schema in Fig. 2 and asan implementation with the Spartan 3E FPGA, shownin Fig. 3, is tested with an algorithm for object detection

in infrared images.The development of the test algorithm is first createdand simulated as Matlab Simulink model shown inFigure 4.

Figure 4. The Simulink model of the algorithm for testing thecapabilities for infrared image processing of the proposedstructure of the module with Spartan 3E FPGA


ISBN: 978-1-61804-023-7 255


4/6

The infrared or thermal image data are presentedinside this block in form of binary image data formatand are shown as first block Read Binary File inFigure 4.

The test algorithm performs the infrared image processing as a filter labeled in Figure 4 as General FIRBlock. The coefficients of the FIR filter are presented asfeatures in form of rectangles, are collected as templatesand are also show in Fig.ure 4 as Read Binary File 1,Read Binary File 2 and Read Binary File 3,respectively for each of the chosen in [11] features:vertical (ftv.bin), horizontal (fth.bin) and diagonal(ftd.bin) in form of rectangles.

In Figure 4 is shown the name of input infrared imageloaded from the database with collected infrared images.The infrared input image chosen from the database inthis testing example is shown as the block Thermal

Image 1.bin in Figure 4.The processing operations are presented as the threeSimulink blocks 2-D Correlation, which performmatching in form of two dimensional correlation

between chosen for testing Thermal Image 1 and each ofthe features fth and ftd, respectively. In these 2-DCorrelation blocks is applied the following equation tocalculate the two dimensional cross-correlation ( ) ji,C

between the matrix of the tested thermal image and eachof the matrix of the features in form of rectangles ftv,fth and ftv:

( ) ( ) ( )

=

=

++=1

0

1

0

,.,, y x N

m

N

nk mk jnimnm ji FTTIC (1)

for 3,2,1;10;10 =++ k N N j N N i x f x y f y k k

,

where( ) jik ,C is two dimensional cross-correlation for each

of three features ( )3,2,1=k ; x N and y N - horizontal and vertical dimensions of

thermal image matrix mTI , respectively; x

f k N and y

f k N - horizontal and vertical dimensions ofthe feature matrix k FT , respectively;

3,2,1=k - index of three features matrix 21 , FTFT and 3FT for features ftv, fth and ftd,respectively;

The outputs of three blocks 2D Correlation containthe results of two dimensional cross-correlation in formof matrices ( ) jik ,C for 3,2,1=k . From these threeoutputs of the blocks 2D Correlation the values in

matrices ( ) jik ,C are estimated in three Simulink blocks

named Maximum , to calculate the maximal valuesmaxk ft in each of three matrices ( ) ( )3,2,1,, =k jik C :

( )[ ] ji ft k k ,maxmax C= (2)for 3,2,1=k .

The calculated in blocks Maximum maximalvalues max2

max1 , ft ft and

max3 ft correspond to

determination existence of features for vertical ftv,horizontal fth and diagonal ftd properties in testedThermal image 1

Three outputs values max2max

1 , ft ft andmax3 ft from the

corresponding outputs Val of the respective blocksMaximum are merged using a Simulink block formultiplexing (the vertical black bar with three inputs inFig. 4). The joined information for the positions of the

maximal values max2max1 , ft ft and max3 ft determines andguarantee with some supposition and probability thatfrom these joined feature positions is possible to locatethe position of the objects in tested thermal image. Tomark the places of these possible object positions isadded in Figure 4 the block Constant to define theinitial characteristics of Rectangle (rw, rh) withcorresponding dimensions rw and rh for width andheight, respectively. The initial values or rw and rh areadded in a two input multiplex block in Fig. 4 with the

joined feature max2max

1 , ft ft andmax3 ft position

information. The output of the multiplex block isconnected to the input P of the block Draw Shapes inFigure 4. To the input I of this block is entered the visualinformation for the tested Thermal Image 1. The finalresult of the operation of this block is to draw therectangles with the appropriate dimensions (width rwand height rh) in the possible places to mark detection ofexistent objects.

To reach this final operation and to achieve a goodexactness of the important objects detection in testedthermal images are prepared the suitable operations for

estimation of joined featuremax

2

max

1 , ft ft andmax

3 ft todefine an object with some preliminary set forms anddimensions.

This is a very important and necessary processing stepto eliminate from the list of possible detected objects theobjects with small or large dimensions (exceeded the

preliminary set dimensions) or the objects with anunusual or unexpected form. These restrictions dependfrom the application of the thermo visual system andhere in this article are defined for the goal of detectionthe objects hidden in dress or baggage of people.

The results of detected objects separated and markedwith the appropriate rectangles in the tested thermal


ISBN: 978-1-61804-023-7 256


5/6

image are directed to the Output of General FIRBlock, shown in Figure 4.

3 Test and Results

The described in Figure 4 Simulink model of thealgorithm for objects detection in infrared images isimplemented in the proposed FPGA and DSP infraredimage processing module. The tests are performed withthe infrared images collected in an infrared imagedatabase. The infrared images are prepared withinfrared camera type Flir [4, 12] and the preprocessing ofthese images is made with Flir software [13]. One ofthese test images used as input infrared image is shownin Fig. 5. The image contain an object (ring) hiddenunder the gloves of a person. The goal of the algorithmfor processing of this infrared image is to find andseparate the object existing in the input infrared image.

Figure 5. An example of infrared image containing an object(ring) hidden under the gloves of a person

The result output infrared image, shown in Fig.6,demonstrate the correct work after execution of thealgorithm implemented in the proposed FPGA and

DSP infrared image processing module.

Figure 6. The output infrared image show the correctobject detection (ring separated with a rectangle) after

processing with the algorithm for object detectionimplemented in the proposed FPGA and DSP infrared image

processing module

The view of the FPGA block used in the proposed FPGAand DSP module is shown in Figure 7 [14] and the view ofthe DSP block used is TMS320C6416 [15], shown in Figure 8.

Figure 7. The view of the FPGA block used in the proposed FPGA

Figure 8. The view of the DSP block used in the proposed FPGA


ISBN: 978-1-61804-023-7 257


6/6

In the Table 1 are listed some of the achievedimportant characteristics of the proposed FPGA modulefor infrared image processing.

Table 1

Property SpartanX3S1500 4

Slides 4.379

Logic 28% of Logic40% of Multipliers

Memory 17% of Memory

ClockFrequency 75 MHz

AcknowledgmentThis work was supported by National Ministry of

Science and Education of Bulgaria under ContractDDVU 02/04/2011: Thermo Vision Methods andRecourses in Information Systems for Customs Controland Combating Terrorism Aimed at Detecting andTracking Objects and People.

References[1] Lebold J. Infrared Thermography and Distribution

System Maintenance Electricity Today, Volume 3. 2008, 18-19

[2] Coon D. D. and Perera A.G. U. Spectral informationcoding by infrared photoreceptors. International Journal ofInfrared and Millimeter Waves Volume 7, Number 10, 1571-1583

[3] FLIR Application Book. FLIR Company 2010[4] FLIR Infrared Cameras. http//www.flir.com/[5] Al.Bekiarski, Sn.Pleshkova, L. Taneva. Thermo

vision system with embedded digital signal processor for realtime objects detection, The 4th International Congress onImage and Signal Processing, 15-17 October 2011, DonghuaUniversity , Shanghai, China (to be published)

[6] IEEE Standard VHDL Language. Reference Manual.IEEE Std 1076, 2000 Edition (Incorporates IEEE Std 1076-1993 and IEEE Std 1076a-2000)

[7] IEEE Standard Verilog Hardware DescriptionLanguage. IEEE Std 1364-2001 (Revision of IEEE Std 1364-1995)

[8] IEEE Standard Test Access Port and Boundary-ScanArchitecture IEEE Std 1149.1-2001 (Revision of IEEE Std1149.1-1990)

[9] Spartan-3E FPGA Family: Data Sheet. Xilinx.DS312 (v3.8) August 26, 2009

[10] Tusch M. High-Performances Image Processing withFPGAs. CEO Apical Limited. 2010

[11] Sn.Pleshkova, Al.Bekiarski. Algorithm of feature

estimation for real time object detection in thermal images,The 4th International Congress on Image and SignalProcessing, 15-17 October 2011, Donghua University ,Shanghai, China (to be published)


ISBN: 978-1-61804-023-7 258

Documents

Systems 40Nexys2_rm