Latest Developments in Image Processing on JET

by Andrea Murari1, J.Vega2, T.Craciunescu3, P.Arena4, D.Mazon5, L.Gabellieri6, M.Gelfusa7, D.Pacella6, S.Palazzo4, A.Romano6, J.F.Delmond8, A. De Maack9 , T.Lesage8

12

4

5

3

9

87 University of Rome

“Tor Vergata”

6

CODAS: Raw Data

Total Raw data: a record Total Raw data: a record of almost 35 Gbytes per of almost 35 Gbytes per shot has been reached shot has been reached

which keeps JET increase which keeps JET increase in stored information in in stored information in line with the Moore law. line with the Moore law.

JET Database exceeds JET Database exceeds 100 Terabytes100 Terabytes

About 50% are About 50% are images images

Cameras: Visualization

In total more than 30 In total more than 30 cameras operational cameras operational

((PIW protectionPIW protection). ). New visualization tools New visualization tools are indispensable for are indispensable for the analysis (PinUp) the analysis (PinUp)

A new specialist is A new specialist is rostered in the control rostered in the control

room: the VSO (Viewing room: the VSO (Viewing Systems Officer)Systems Officer)

Goals of Imaging in JET

Goals of imaging:

o Imaging of the IR emission from the wall for portection and physics studies

o Imaging of edge instabilities (ELMs, MARFEs etc) for phyics and to assess their effects on the wall.

o Overview of the general discharge behaviour

Issues of Imaging in JET

Issues posed by the exploitation of images:

o Information retrieval (discussed in detail last meeting)

o Image registration (vibrations and interference)

o Integration of models (see V.Martin Talk)

o Real time identification of events

o Extraction of quantitative information for physics studies (see T.Craciunescu Talk)

Mathematical indicators

• 8 different mathematical indicators for vibration detection have been investigated:

• Normalized cross-correlation• Shannon entropy• Tsallis entropy• Renyi entropy• Alpha entropy• Shannon mutual information• Tsallis mutual information• Renyi mutual information

Entropy

Mutual information

Normalized cross-correlation

Normalized cross-correlation

Shannon Entropy Shannon Entropy

Tsallis entropy / Sq entropy Tsallis entropy / Sq entropy

q : degree of non-additivity q : degree of non-additivity

Equal when q 1

pi : probability of finding the system in each possible state i (or residual i)k : Total number of possible states(or number of possible residuals)

Additive and Non additive entropy

Shannon entropy is additive because it assumes that there are no correlations between the systems being added

Tsallis entropy is not additive because it can take into account these correlations.

Tsallis entropy is not additive. For a sum of two systems A1 and A2

Tsallis entropy is finding many applications from statistical mechanics to signal processing, image processing etc

Sq (A1 + A2 ) = Sq (A1) + Sq (A2) + (1-q) Sq (A1) Sq (A2)

In the case of camera movements, the difference between two frames presents long range correlations

These long range correlations, which are less pronounced, in case of objects moving in the still field of view of a camera, can be emphasised by the proper selection of q in the Tsallis entropy.

Application of Tsallis entropy to image registration

Shannon Entropy : 0 Sq entropy : 0 Shannon Entropy : 0 Sq entropy : 0 Shannon Entropy :

0.61 Sq entropy :

3.16

Shannon Entropy : 0.61

Sq entropy : 3.16

Shannon Entropy : 0.81

Sq entropy : 3.99

Shannon Entropy : 0.81

Sq entropy : 3.99

Background

Matrix

Background

Matrix

Object Matrix q=0.1 Object Matrix q=0.1

Shannon Entropy : +0.23

Sq entropy : +0.83

Shannon Entropy : +0.23

Sq entropy : +0.83

Red: Tsallis entropy versus row shift

Blue: Shannon entropy vs row shift

Mutual information

Renyi definition

The Wide Angle Camera KL7 provides a view of the main vessel in the IR

•The Camera seats at the end of and endoscope with many optical components whose position is not monitored• No reliable reference points in the field of view

Image registration: diagnostic

All the major typical events are included

Plasma current between 2 and 3.5 MA Toroidal field between 1.9 and 3.4T

Statistics of frames observed in JET

• A database of 69 videos and almost 40000 frames has been analysed manually to determine the cases with movements.

Comparison Entropies The vertical lines indicate the period with vibrations

Comparison Mutual Informations

Statistics: Threshold• Method: determination of a threshold discriminating

between the frames with and without movements

No mouvement

No mouvement

MouvementMouvement

Succes Rate: OverviewConclusions

Threshold % of good results

Frame where no movement is wrongly detected

Frame where movement is wrongly detected

Normalized cross-correlation 0.94 71.66 14.84 3.78

Shannon entropy 1.6 84.17 15.35 0.48

Shannonmutual information 0.62 78.09 0.47 21.44

Tsallis entropy 25 86.19 6.66 7.15

Tsallismutual information 0.58 79.98 0.48 19.54

Renyi entropy 8 84.70 15.14 0.16

Renyimutual information 1.28 79.80 2.58 17.62

• The result is that entropy of Tsallis is the best among the other entropies.• The mutual information with Tsallis definition is the best definitions among from the definition of mutual information and NCC.

Success Rate: missed and false alarms

86,19%

6,66%7,15%

Tsallis entropy analysis

Correct analysis

Frame where no mouvement is wrongly detected

Frame where mouvement is wrongly detected

False alarmsMissed alarms

Succesfull identifications

Registration: Method Comparison

• A synthetic videos has been shifted by 10 rows and then two of the best indicators have been tried to register it.

Shift

Application to video 73851, frame 786

• Frame 786 is chosen among frames with vibrations. The result of the Tsallis mutual information, which is shown below, is the matrix must be shift by two rows leftwards.

Verification

Mean(value of pixel)=1.3864

Mean(value of pixel)=1.2788

• Subtraction of the frame affected by the movement and the reference frame before and after the registration shows a clear improvement. More effective in the main chamber because the divertor is affected by ELMs

Image Analysis: Hot spot detection

The white areas represent the potential hot regions, parts of the wall which reach a to high temperature.

11,300 frames have been analysed manually

A C++ algorithm to be run on a serial machine has been developed to automatically identify the hot spots (100% success rate in terms of image processing not physics)• Infrared Wide Angle

View: Size of IR images: 496x560 pixels Assumption: the

temperature map provided is correct

Reference serial algorithm: computational time

• For traditional serial algorithms, the computational time depends on the content of the image. A potential problem for real time applications

Computational time versus number of white pixels

Computational time evolution during a discharge

• Array of cells– Information for each cell:

• State (mapped to greyscale value)• Input• Output (dependent on state)

– Each cell is connected to a set of neighbours (usually belonging to a 3x3 square)

– A state equation defines the time evolution of the cell:

Cellular Nonlinear Networks

ijjiSlkC

kljiSlkC

klijij zulkjiBylkjiAxxrr

),(),(),(),(

,;,,;,

where xij is the state of the cell, ykl the output and ukl the input.

• CNNs are a new computational paradigm. If supported by an adequate memory they have the same computational power of Universal Turing machines but with the benefit of parallelism.

• A, B: feedback and input synaptic operators– They define how the state evolves and how neighbour

cells influence it.

– For image processing, they define the kind of filter implemented by the CNN, and are usually 3x3 matrices

a-1,-1 a-1,0 a-1,1

a0,-1 a0,0 a0, 1

a1,-1 a1,0 a1,1

ijjiSlkC

kljiSlkC

klijij zulkjiBylkjiAxxrr

),(),(),(),(

,;,,;,

• zij is a bias constant.

• The set (A, B, z) is called a template. Nonlinear (morphological) operators can be implemented

yi-1,j-1 yi-1,j yi-1,j+1

yi,j-1 yi,j yi,j+1

yi+1,j-1 yi+1,j yi+1,j+1

Summation of dot products

Cellular Nonlinear Networks

1. Directed Growing Shadow

• This template create “shadows” from white pixels by increasing the objects. The template was customized so that the main direction of growth is horizontal.

This template allows merging small close regions – this corresponds to the clustering operation of the serial algorithm.

To be classified as hot spot

To be eliminated

2. ConcaveFiller

• The ConcaveFiller template is applied in order to avoid that the following shrinking phase might separate the regions unified by DirectedGrowingShadow.

S.Palazzo, A.Murari et al REVIEW OF SCIENTIFIC INSTRUMENTS 81, 083505 2010

3. Object Decreasing

• Object Decreasing is applied in order to rescale the objects back to their original size, while keeping the merge regions united.

• Object Removal allows to remove “small objects”

How to implement different processing algorithms to different parts of the images?

Space-varying CNNs

• The implementation approach is based on the definitions of regions in the input image.

• The image is divided into a grid of rectangular cells (regions), by specifying the coordinates of the grid’s rows and columns.

• Each region is then assigned its own sequence of templates, which can differ from other regions in terms of number of templates to be applied, number of iterations or templates’ coefficients. Mathematics already developed.

• The total computation time will depend on the longest template sequence among all regions.

CNN implementation on FPGA

Core array architecture A core takes as input a stripe of the

image (or the output of the upper-row core) and computes the next iteration.

All cores in a column process the same part of the image.

All cores in a row execute the same iteration (on different input stripes).

Parallelism is provided by adding columns to the array – that is, by dividing the image into more parts, to be independently processed.

Hot spot detection• The new algorithm divides the image

into different number of regions on which it is possible to:– Apply customized temperature thresholds,

for example a higher one in the bottom-left divertor’s region.

– Apply region-specific template sequences, in order to improve the global detection accuracy.

Deterministic computational time Implementation with FPGA using cores Total computation time with a 100 MHz clock and 1 column of cores:

106 ∙10 ns = 10 ms → Maximum frame rate: 100 fps

It is possible to increase the frame rate by adding parallelism, i.e. more columns in the core array architecture. With a 10-column core array, the computation time is reduced to 1 ms, and the maximum input frame rate becomes 1000 fps.

Conclusions

o Bidimensional measurements are the new frontier in plasma physics (they are a step forward comparable to profiles)

oVideos contain a wealth of information which can give a very significant contribution to both the understanding of the physics and the real time control of fusion plasmas (including protection)

o Image manipulation: many tools are on the market but they are not always exactly what is needed and therefore significant level of development is required

Apha entropy

Tsallis definition

Results• The figure below shows the Iα entropy. This

entropy does not provide coherent and understable results so it will not be used in the following.