Automatic Feature Generation for Endoscopic Image Classification

20 April 2023

Chair for Computer Aided Medical Procedures & Augmented Reality

Department of Computer Science | Technische Universität München

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu

Automatic Feature Generation for Endoscopic Image Classification

Ulrich Klank

CAMP | Department of Computer Science | Technische Universität München | 20 April 2023 2


Endoscopic Image Classification

Synchronization of surgeries by phases is essential for Workflow Project1

Analyzing endoscopic images supports the classification of the current state into a phase.

1 A. Ahmadi, T. Sielhorst, R. Stauder, M. Horn, H. Feussner, N. Navab -Recovery of surgical workflow without explicit models



Task

How can these images be classified into surgical phases?

Or if they can’t be classified well enough, which features can be added to the rest of the information?



Definition of a Feature

Feature as representation of discriminative information Numerical vector calculated out of the

image Reduction of the dimension

Features Basic: Gradient/Color/Motion Histogram Integral image1

Phase concruency2

Gradient Histogram

1 Viola, Jones- Robust Real-time Object Detection

2 Kovesi - Image Features from Phase Congruency



Feature Space

With a limit to algorithms with an output smaller or equal than the input we get a number of injective function about:

depending on the dimension of the input vector d and the number of different values c which are possible in the input.

Exploration of this space with Genetic Programming1 A method to explore a high dimensional space Fitness function influences the resulting features

1 Feature Extraction from Multiple Data Sources Using Genetic Programming - John J. Szymanski,* Steven P. Brumby, Paul Pope, Damian Eads, Diana Esch-Mosher,Mark Galassi, Neal R. Harvey, Hersey D.W. McCulloch, Simon J. Perkins, Reid Porter,James Theiler, A. Cody Young, Jeffrey J. Bloch and Nancy David

(cd)cd( cd¡ 1)

2



Feature Generation by Genetic Programming12

Mutation

EvaluationCode

1 Wolfgang Banzhaf, Peter Nordin, Robert E. Keller, Frank D. Francone - Genetic Programming: An Introduction

2 Koza - Genetic Programming



Implementation Steps

Definition of a programming language (more) Designed for recombination and mutation

Designing a Framework to run programs Translation of basic features in the new language Evaluation of the fitness of programs Recombination and Mutation



An Example as Proof of Concept

Implementation of a color histogram in the new language

Test on an easy sample, 6 images(10*10*RGB, jpeg) in 3 classes:

Dimension of input here (cd): 76.800



Evaluation of the Example

The Fitness Function is the core of the system




Comparison of several Programs Pi

P1P1 P2

P2…




A random selection of the labeled input Ik

several from of each class

P1P1 P2

P2 …

I1 I2 I3




Execution of the program with every selected input Ik

As result there are output vectors:

P1P1 P2

P2 …

I1 I2 I3

o11;o12 ¢¢¢o23




Compare each pair of output vectors of one Program, by the distance

P1P1 P2

P2 …

I1 I2 I3

dik;l =

P lenm=0

(oi k [m]¡ oi l [m])2

(joi k [m]j+joi l [m]j)2

o11;o12 ¢¢¢o23




Select the maximum of these distances, only considering pairs of different classes:

P1P1 P2

P2 …

I1 I2 I3

o11;o12 ¢¢¢o23

dik;l =

P lenm=0

(oi k [m]¡ oi l [m])2

(joi k [m]j+joi l [m]j)2

Now we have two classes k, l which our feature separates

argmaxkl(dikl jk:class 6= l:class)



Evaluation of the example

Subtract the distances in images of the classes k or l

P1P1 P2

P2 …

I1 I2 I3

o11;o12 ¢¢¢o23

dik;l =

P lenm=0

(oi k [m]¡ oi l [m])2

(joi k [m]j+joi l [m]j)2 f i = dikl ¡ di

k k +dil l

2argmaxkl(di

kl jk:class 6= l:class)



3.1 Results in this Example

< 64 >

< 72 >

< -16 >

< 72 >

< -16 >

< -16 >

NewProgram



Fitness in the Example:

Input Program was calculating a Color Histogram (Output length

512 in ~2000 steps), resulting fitness: ~ 1.2 % The new generated program calculated one number (in 30

steps) and got a fitness of: ~ 98.8 %

P1P1

P2P2



Evolution Technique used in the Example

Selection of the two best Program and generation several children by these two methods:

One parental program: Mutation Adding of random commands Removing a single commands1 Duplicating an existing command Switching commands

Two parental programs: Breeding Mixing both, preferring blocks of a specific size Adding random commands in the gaps between

P1P1

P11P11

P1P1 P2

P2

P12P12



Conclusion

The concept seems to work Adaptation to workflow project’s data can start



Steps to adapt it to the Workflow project

Adapt fitness function (more) Change Genetic rules (more) Optimize the framework Generate C code out of the intermediate

language



Any Questions?



Appendix



Planned Improvements in the Genetic rules

Probability to survive should depend on the fitness of all other existing programs

Even a program with a low fitness should have a small change to survive

Also the reproduction should not be restricted to the best program

Back



Fitness

Actual fitness expresses: Linear separation of two classes

Further improvements: Static Analysis of Code1

Semantic Checks to prevent running in infinite loops to prevent the execution of programs not using the input …

Other Classifiers to evaluate the output

1 Colin G. Johnson-Deriving genetic programming fitness properties by static analysis

Back



Specification of the language

Assembler style with some high level functions

1 Opcode and 3 Arguments for one command

High level examples: Reading of the input and its dimensions For loops

Adding of more functions possible

Back



Scheme for the programming language

OPCode 1.Parameter

2. Parameter

3. Parameter

010

For

counter to stack

0

From

Constant

100

To

Constant

65

End of the loop

Constant

123

Read pixel

Value to stack

-1

x

From stack

20

y

Constant

…

…



Generated Program (without nops)

INPUTDIMENSION 0 0 4595026

SAVE 0 -1 0

INPUTDIMENSION 1 0 0

SAVE 23367 -1 0

FOR 0 -1 37

LOAD 4 0 0

LOAD 1 0 0

SAVE 5 -1 0

LOAD 5 0 0

POP 0 0 -1

READINPUT -1 -1 -1

LOAD 5 0 -1

POP 0 0 -1

PUSH 64 0 -1

SAVE 3 -1 0

PUSH 8 0 0

MULTIPLY 0 0 0

LOAD 3 0 0

PLUS 0 12604 0

PLUS 0 6457 0

PLUS 31190 0 0

JUMP 55 0 0

DIVIDE 0 0 0

DIVIDE 0 0 0

PUSH 32 0 0

LOAD 1 -1 0

LOAD 2 0 0

RETURN 0 0 0

JUMP 0 0 0

FOR 8 520 58

Documents

Automatic Feature Generation for Endoscopic Image Classification