Christophe Gauge

Gan

non

Uni

vers

ity

Dep

artm

ent o

f Com

pute

r and

Info

rmati

on S

cien

ce

Advisor: Dr. Sreela Sasi

Automated Colorization of Grayscale Images

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

IntroductionImage Colorization

WHAT:Adding color

to monochrome

images

WHEN: Performed

since the early 20th century

WHY: Improve visual

appeal of illustrations

HOW: A painstaking and subjective

manual task

2

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Introduction (contd.)

Digital Image Colorization

Automation of colorization

Improve visual appeal of images

Color accuracy, finer details

Add relevant information to images

Make images more understandable

3

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Introduction (contd.)

Applications of Image Colorization

Applications

Homeland Security

Satellite Imaging

Old photos and films

Medical Imaging

Video compression

4

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Colorization Techniques

Scribble-based colorization

User add color scribbles to image to be colorized

laborious, time-consuming, subjective, and painstaking manual

task.

Example-based colorization

automation by extracting colors from

sample image

results can vary depending on example

image chosen

+ = + =

Previous ResearchImage Colorization

5

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Current ResearchProcess Workflow

Texture-based Segmentation

ImageImageSample

Image

Feature Extraction

Color Descriptors

Texture Descriptors

New Grayscale

Image

Texture-based Segmentation

Feature Extraction

Texture Descriptors

Texture Matching

Colorization ProcessDatabase

6

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Image Segmentation

Image segmentation:

•Is the partitioning of an image into homogeneous regions based on a set of characteristics.

•Is a key element in image analysis and computer vision.

7

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Image Segmentation (contd.)

Clustering:

•Is one of the methods available for image segmentation.

•Is a process which can be used for classifying pixels based on similarity according to the pixel’s color or gray-level intensity.

8

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Image Segmentation (contd.)

Despite the substantial amount of research performed to date, the design of a robust and efficient clustering algorithm remains a very challenging problem

9

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Color-based Image SegmentationComposite Image

10

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Color-based Image SegmentationComposite Image with salt & pepper noise added

11

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Texture-based Image Segmentation 12

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Workflow ProcessTexture-Based Image Segmentation

Original Image

Filtered Image

Filtered Image

Filtered Image

Filtered Image …

Feature Image

Feature Image

Feature Image

Feature Image …

Feature Image

Blobs

Gabor Filters

Energy Computation

Segmentation

Add, mean smoothing, normalization

13

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

14Image Segmentation

Multi-Channel Filtering - Gabor Transform

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Previous Research (contd.)

Texture-Based Segmentation

15

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

16Image SegmentationNormalized Sum of Gabor Responses

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Current ResearchProcess Workflow

Texture-based Segmentation

ImageImageSample

Image

Feature Extraction

Color Descriptors

Texture Descriptors

New Grayscale

Image

Texture-based Segmentation

Feature Extraction

Texture Descriptors

Texture Matching

Colorization ProcessDatabase

17

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Previous Research (contd.)

Clustering and Feature Extraction

18

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Previous Research

•The K-means algorithm has been used for a fast and crisp “hard” segmentation. •The Fuzzy set theory has improved this process by allowing the concept of partial membership, in which an image pixel can belong to multiple clusters. •This “soft” clustering allows for a more precise computation of the cluster membership, and has been used successfully for image clustering and segmentation.

19

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

•The Fuzzy C-means clustering (FCM) algorithm [1] is a widely used method for “soft” image clustering. •However, the FCM algorithm is computationally intensive. •It is also very sensitive to noise because it only iteratively compares the properties of each individual pixel to each cluster in the feature domain.

Previous Research (contd.)

[1] James C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms. New York: Plenum, 1981.

20

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Image SegmentationModified Fuzzy C-means Clustering

21

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Previous Research (contd.)

Fuzzy C-means clustering (FCM) Algorithm

22

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Previous Research (contd.)

FCM Pseudo-code

Step 1 Set the number c of clusters, the fuzzy parameter m, and the stopping condition ε

Step 2 Initialize the fuzzy membership values µ

Step 3 Set the loop counter b = 0

Step 4 Calculate the cluster centroid values using (3)

Step 5 For each pixel, compute the membership values using (4) for each cluster

Step 6 Compute the objective function A. If the value of A between consecutive iterations < ε then stop, otherwise set b=b+1 and go to step 4

23

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

[2] Stelios Krinidis and Vassilios Chatzis, "A Robust Fuzzy Local Information C-means Clustering Algorithm," Image Processing, IEEE Transactions on, pp. 1-1, 2010.

Previous Research (contd.)

Modified Fuzzy C-means clustering with Gki factor

In order to improve the tolerance to noise of the Fuzzy C-means clustering algorithm, Krinidis and Chatzis [2] have proposed a new Robust Fuzzy Local Information C-means Clustering (FLICM) algorithm by introducing the novel Gki factor.

The purpose of this algorithm is to adjust the fuzzy membership of each pixel by adding local information from the membership of neighboring pixels.

24

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Previous Research (contd.)

Modified Fuzzy C-means clustering with Gki factor

Sliding window of size 1 around the ith pixel

The Gki factor is obtained by using a sliding window of predefined dimensions:

25

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Previous Research (contd.)

Modified Fuzzy C-means clustering with Gki factor

The Gki factor is calculated by using the following equation:

26

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Current AlgorithmModified Fuzzy C-means clustering with novel Hik factor

This algorithm is further improved by including both the local spatial information from neighboring pixels and the spatial Euclidian distance of each pixel to the cluster’s center of gravity.

In this research, the algorithm is also extended for clustering of color images in the Red-Green-Blue (RGB) color space.

27

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Current Algorithm (contd.)

XX

X

pidi1

di2 di3

c1

c2

c3

Illustration of the new Hik factor displaying the spatial Euclidian distance to the center of gravity of each cluster

28

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Current Algorithm (contd.)

Process Workflow

Customize Parameters

Calculate cluster membership values

Compute Gki

Readjust membership values

Compute Hki

Compute objective function

Defuzzification and clustering

-

Image

Calculate cluster centroid

29

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Current Algorithm (contd.)

Modified Fuzzy C-means Clustering30

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Simulation and ResultsSynthetic Grayscale Test Image

31

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Natural test image

FCM segmentation with 5 clusters

FCM segmentationusing the modified FCM algorithm

with 5 clusters, Gki window=1 and Hik

Simulation and ResultsNatural Test Image

32

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Simulation and ResultsSynthetic Grayscale Test Image

Synthetic 4-color test image with added salt and pepper noise

FCM clustering FCM clustering with Gki window=1 and with Hik

FCM clustering with Gki window=5 and with Hik

33

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Synthetic 4-color test image with added salt and pepper noise

FCM clustering FCM clustering with Gki window=1 and with Hik

FCM clustering with Gki window=5 and with Hik

Simulation and ResultsSynthetic Color Test Image

34

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Image SegmentationClustering Demo

35

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Modified Fuzzy C-means ClusteringSummary

•In this research, the FCM with the Gki factor is modified using the Hik factor, and the algorithm is extended for the clustering of color images.

•The use of the sliding window in the Gki factor improves the segmentation results by incorporating local information about neighboring pixels in the membership function of the clusters. However, this resulted in a significant increase in the number of calculations required for each iteration for each pixel, and can be given by

36

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Modified Fuzzy C-means ClusteringSummary (contd.)

•By combining the Gki and the Hik factors, this modified FCM algorithm considerably reduced the number of iterations needed to achieve convergence. The tolerance to noise of the Fuzzy C-means algorithm is also greatly increased, allowing for an improved capability to obtain coherent and contiguous segments from the original image.

37

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Modified Fuzzy C-means ClusteringSummary (contd.)

•However, because of the radial nature of the spatial Euclidean distance to the cluster’s center of gravity, this new method may not be as effective for images containing circular shapes, or for images where the cluster’s center of gravity are close to each-other.

•In this research, the FCM is extended for the clustering of color images in the RGB color space. The effectiveness of this algorithm may be tested for images in other color spaces also.

38

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Current ResearchProcess Workflow

Texture-based Segmentation

ImageImageSample

Image

Feature Extraction

Color Descriptors

Texture Descriptors

New Grayscale

Image

Texture-based Segmentation

Feature Extraction

Texture Descriptors

Texture Matching

Colorization ProcessDatabase

39

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

40

Sample Color Images

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

41Image SegmentationNormalized Sum of Gabor Responses

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Image SegmentationFeature Extraction

42

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Image SegmentationFeature Extraction (contd.)

43

Blob Filtering for color and texture extraction.

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

44

Texture and Color database

Image SegmentationFeature Extraction (contd.)

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

45Current ResearchProcess Workflow

Texture-based Segmentation

ImageImageSample

Image

Feature Extraction

Color Descriptors

Texture Descriptors

New Grayscale

Image

Texture-based Segmentation

Feature Extraction

Texture Descriptors

Texture Matching

Colorization ProcessDatabase

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

46Grayscale Image Processing

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

47Current ResearchProcess Workflow

Texture-based Segmentation

ImageImageSample

Image

Feature Extraction

Color Descriptors

Texture Descriptors

New Grayscale

Image

Texture-based Segmentation

Feature Extraction

Texture Descriptors

Texture Matching

Colorization ProcessDatabase

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

48Previous ResearchVisual descriptors

• Visual descriptors are descriptions of the visual features of the contents of images.

• They describe elementary characteristics such as the shape, color, and texture.

• MPEG-7 is a multimedia content description standard. It was standardized in ISO/IEC 15938 (Multimedia content description interface).

• This description is associated with the content itself, to allow fast and efficient searching for material that is of interest to the user.

• MPEG-7 is formally called Multimedia Content Description Interface. Thus, it is not a standard which deals with the actual encoding of moving pictures and audio, like MPEG-1, MPEG-2 and MPEG-4. It uses XML to store metadata.

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

49Previous ResearchVisual descriptors

http://chatzichristofis.info/?page_id=213

The Img(Rummager) application was developed in the Automatic Control Systems & Robotics Laboratory at the Democritus University of Thrace-Greece.

The application can execute an image search based on a query image, either from XML-based index files, or directly from a folder containing image files, extracting the comparison features in real time.

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Previous Research (contd.)

Content-Based Image Retrieval

50

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

MPEG-7 EHD Fuzzy Spatial BTDH ADS

51Previous Research (contd.)

Content-Based Image Retrieval

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Image Descriptors used:

MPEG-7 Homogeneous Texture Descriptor: Edge Histogram Descriptor (EHD).

CCD for Medical Radiology Images: Brightness and Texture Directionality Histogram (BTDH)Fuzzy rule based scalable composite descriptor (BTDH) is a compact composite descriptor that can be used for the indexing and retrieval of radiology medical images. This descriptor uses brightness and texture characteristics as well as the spatial distribution of these characteristics in one compact 1D vector. The most important characteristic of the proposed descriptor is that its size adapts according to the storage capabilities of the application that is using it. This characteristic renders the descriptor appropriate for use in large medical (or gray scale) image databases.

Simulation Results (contd.)

Content-Based Image Retrieval (CBIR)52

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Query image

MPEG-7 EHD Fuzzy Spatial BTDH ADS

Result Matching color Result Matching

color Result Matching color

Simulation Results (contd.)

Content-Based Image Retrieval (CBIR)53

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

54Current ResearchProcess Workflow

Texture-based Segmentation

ImageImageSample

Image

Feature Extraction

Color Descriptors

Texture Descriptors

New Grayscale

Image

Texture-based Segmentation

Feature Extraction

Texture Descriptors

Texture Matching

Colorization ProcessDatabase

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

The RGB color space is defined by the three chromaticities of the red, green, and blue additive primaries, and can produce any chromaticity that is the triangle defined by those primary colors.

The YCbCr color space is used in video and digital photography systems. • Y is the luma (luminance ) component and • Cb and Cr are the blue-difference and red-difference

chroma components.

Simulation Results (contd.)

Image Colorization55

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

56Image from Wikipedia Simulation Results (contd.)

Image Colorization

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Simulation Results (contd.)

Colorization57

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Conclusion and Future Work

•New and innovative method•Automating example-based colorization •Combines several state-of-the-art techniques

•Reasonably accurate results were obtained•Several of the steps require custom parameters •computationally very intensive

•Texture retrieval needs improvement•Complex textures containing multiple colors•Anisotropic diffusion for preserving strong edge information

•Combining these techniques in order to automatically colorize grayscale images is a viable option

58

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

Conclusion and Future Work (contd.)

•Images segmentation and clustering methods computationally very intensive, Processing time for each 600x450 sample color image took 20 minutes on a quad-core Intel 2.6 GHz processor.•Texture retrieval methods still need to be improved for scale and rotation invariance•Store more complete color descriptors to accommodate more complex textures containing multiple colors. •Anisotropic diffusion could also be used to smooth the Gabor response images while preserving strong edge information.

•Testing conducted as part of this research proved that the ability to combine these techniques in order to automatically colorize grayscale images is a viable option.

59

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

References

[1] Anat Levin, Dani Lischinski, and Yair Weiss, "Colorization using optimization," ACM Transactions on Graphics, vol. 23, no. 3, p. 689–694, 2004.

[2] R. Irony, D. Cohen-Or, and D. Lischinski, "Colorization by example," in Eurographics Symposium on Rendering, 2005, p. 277–280.

[3] Ashikhmin M., Mueller K. Welsh T., "Transferring Color to Greyscale Images,".

[4] X., Wan L., Qu Y., Wong T., Lin S., Leung C., Heng P. Liu, "Intrinsic colorization," ACM Trans. Graph., vol. 27, no. 5, p. 152, 2008.

[5] Malik J. Perona P., "Preattentive texture discrimination with early vision mechanisms," J. Opt. Soc. Am. A, vol. 7, no. 5, May 1990.

[6] A. K. Jain and F. Farrokhnia, "Unsupervised texture segmentation using Gabor filters," Pattern Recognition, vol. 24, no. 12, pp. 1167-1186, 1991.

[7] Seo Naotoshi, "Texture Segmentation using Gabor Filters," University of Maryland, College Park, MD, Project ENEE731 , 2006.

[8] Xiaoming Hu, Xinghui Dong, Jiahua Wu, Ping Zou Junyu Dong, "Texture Segmentation Based on Probabilistic Index Maps," in International Conference on Education Technology and Computer, 2009, pp. 35-39.

60

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

References (contd.)

[9] X Muñoz, J Freixeneta, X Cufı a, and J Martı a, "Strategies for image ı ́ ı ́segmentation combining region and boundary information," Pattern Recognition Letters, vol. 24, no. 1-3, pp. 375-392, January 2003.

[10] James C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms. New York: Plenum, 1981.

[11] Chuang Keh-Shih, Tzenga Hong-Long, Chen Sharon, Wu Jay, and Chen Tzong-Jer, "Fuzzy c-means clustering with spatial information for image segmentation," Computerized Medical Imaging and Graphics, vol. 30, no. 1, pp. 9-15, January 2006.

[12] Zhou Huiyu, Schaefer Gerald, Sadka Abdul H., and Celebi M. Emre, "Anisotropic Mean Shift Based Fuzzy C-Means Segmentation of Dermoscopy Images," IEEE Journal of Selected Topics in Signal Processing, vol. 3, no. 1, pp. 26-34, February 2009.

[13] Stelios Krinidis and Vassilios Chatzis, "A Robust Fuzzy Local Information C-means Clustering Algorithm," Image Processing, IEEE Transactions on, pp. 1-1, 2010.

[14] Gauge Christophe and Sasi Sreela, "Automated Colorization of Grayscale Images Using Texture Descriptors and a Modified Fuzzy C-Means Clustering,“ Journal of Intelligent Learning Systems and Applications (JILSA), Vol. 4 No. 2, 2012, pp. 135-143, DOI: 10.4236/jilsa.

61

Auto

mat

ed C

olor

izati

on o

f Gra

ysca

le Im

ages

62

Questions?