Download pdf - Ph.D Dissertation Defense Slides on Efficient VLSI Architectures for Image Enhancement Techniques

IntroductionMotivation & Objectives

ContributionsConclusions & Future Scope

Efficient VLSI Architectures for ImageEnhancement Techniques

Ph. D Dissertation Defense

M. C. Hanumantharaju - 1DS07MEN02Research Scholar

Dr. M. RavishankarResearch Advisor

Department of Information Science and EngineeringDayananda Sagar College of Engineering, Bangalore-560078

March 7, 2014

M. C, Hanumantharaju, Ph.D Dissertation Defense on Development of VLSI Architectures for Image Enhancement Techniques: slide 1/146



Efficient VLSI Architectures for ImageEnhancement Techniques

Ph. D Dissertation Defense

M. C. Hanumantharaju - 1DS07MEN02Research Scholar

Dr. M. RavishankarResearch Advisor

Department of Information Science and EngineeringDayananda Sagar College of Engineering, Bangalore-560078

March 7, 2014




Outline

1 Introduction

2 Motivation & Objectives

3 ContributionsAROF Architectures and Its FPGA ImplementationAdaptive Color Image Enhancement using GMFGaussian Image Enhancement: Algorithm &Architecture

4 Conclusions & Future Scope




Image Enhancement : An Introduction

Image processing2-D signal processingImproves characteristics, properties and parameters

Image enhancementKey step in image processingModifies the attributes of an imageMakes it appropriate for analysis, diagnosis, and display.

Some of the image enhancement applications includeSharpening: improves car license plate numberContrast enhancement: medical image enhancement.Edge enhancement: enhances objects in aerial image.

The realm of image enhancement wraps upReconstruction & RestorationFilteringSegmentationCompression & Transmission




Image enhancement : An Example

(a) Original Image (b) Enhanced Image




Challenges

Image enhancement algorithms have numerous parame-ters to specify and that needs to be adjusted to obtainsatisfactory results.

Lack of integrated algorithms.

Presently, image enhancement research demands betterreconstruction of high quality images than possible withavailable researcher methods.

Image enhancement algorithms depends on the input im-ages instead of adapting to its local features.

Limited speed achieved in software implementation sinceimage enhancement algorithms consists of large array ofdata.




Choice of Implementations

General Purpose Processors(GPPs)

Flexible.

Technology limits the pro-cessing speed.

Limited performance.

Instruction sets are notsuitable for fast processingof high resolution images.

GPP instructions are se-quential and hence systemthroughput decreases.

Digital Signal Processors(DSPs)

Improvement over GPPs.

Falls between GPPs andASICs.

Inadequate pipelining andparallel processing.

Fixed architectures thatlimits the performance.

Parallel operation is possi-ble with multiple DSPs.




Choice of Implementation Cont.,

Application Specific ICs(ASICs)

Fast & efficient.

Fixed circuit.

Large time to market.

High cost, except for largevolume commercial appli-cations.

No optimization.

Field Programmable GateArrays (FPGAs)

High throughput.

Dynamically reconfig-urable.

Massive pipelining andparallelism.

Cost effective.

Attractive choice for real-ization of DIP algorithms.

Present Research Work uses FPGAs forImplementation




Motivation

The motivation behind this work is to bring out the fea-tures in the image that are not clearly visible owing todifferent illumination conditions.

Current market demands better reconstruction of highquality images than is possible with currently availableresearch outputs.

Limitations of image enhancement schemes: difficult totune parameters, deficit of integrated algorithms, lack ofquantitative standard, dependence on inputs instead ofadapting to local features.

Software implementation : inadequate speed.

Hardware implementation of image enhancement algo-rithm is in great demand for applications such as medical,forensic and surveillance etc.




Objectives

Development of efficient VLSI architectures for image en-hancement algorithms.

Design & simulate the algorithm using software approach(C or Matlab).

Test the algorithm for images having different environ-mental conditions.

Realize the algorithm using HDL (Verilog or VHDL).

Verify both software & hardware implementation results.

Evaluate the efficiency of the algorithm using performancemetrics such as PSNR, contrast, luminance, IEF and waveletenergy etc.

Compare proposed approach with other existing methods.




Hardware Design Flow




AROF Architectures and Its FPGA ImplementationAdaptive Color Image Enhancement using GMFGaussian Image Enhancement: Algorithm & Architecture

Adaptive Rank Order Filter (AROF)

Non-linear filter.

AROF is a powerful technique for denoising an image cor-rupted by salt & pepper noise or impulse noise.

Impulse noise is often introduced into digital images dur-ing image acquisition or Interference during transmission.

AROF not only adapts filter output but also window size: iterative algorithm.

AROF window expands : All Pixels within the currentwindow are noisy or median itself is noisy.





Adaptive Rank Order Filter : Flow Chart





Noisy Lena (90% Noise Level) and Restored Image

Figure: First Image : Lena Image with High Noise Density (90% Salt &Pepper Noise) Second Image : Restored Lena Image using AROF.





Related Work

Andreadis et al.1 proposed FPGA implementation of real-time adaptive image impulse noise suppression. However,the system slows down for highly corrupted images.

An efficient hardware implementation of weighted medianfilter using cumulative histogram proposed by Fahmy etal.2 reduces impulse noise satisfactorily. However, hard-ware complexity is high for smaller window size.

1I. Andreadis, G. Louverdis, ”Real-time Adaptive Image Impulse Noise Sup-pression”, IEEE Tran. on Instrumentation and Measurement, Vol. 53, Issue 3,pp. 798-806, 2004.

2S. A Fahmy, P.Y.K. Cheung and W. Luk, ”Novel FPGA-based implementa-tion of median and weighted median filters for image processing”, InternationalConference on Field Programmable Logic and Applications, pp. 142-147, 2005.





Related Work Cont..,

Meena et al.3 proposed a optimized architectures for rankOrder filter. However, optimizations are done for sortingnetwork with out considering noise levels in an image.

Chih et al.4 proposed an efficient denoising architecturefor impulse noise removal in images. Although, decisiontree based approach used in this scheme is effective forhardware implementation, technique may not provide sat-isfactory results for images corrupted with high noise den-sity.

3S. M Meena and K. Linganagouda, ”Implementation and Analysis of Opti-mized Architectures for Rank Order Filter”, Journal of Real Time Image Pro-cessing, Vol. 3, Issue 1-3, pp. 33-41, 2008.

4Chih-Yuan Lien, Chien-Chuan Huang, Pei-Yin Chen and Yi-Fan Lin in IEEETran. on Computers, Vol. 62, No. 4, pp. 631-643, April 2013.





Advantages of Proposed Method

The proposed AROF reduces impulse noise withoutdegrading image information.

The reconstructed images using AROF provides bettervisual quality than possible with Adaptive Median Filter(AMF).

AROF consists of sliding window, sorting network,median selection etc. Therefore, the proposed AROF isbest suited for FPGA implementation.

AROF has better performance than the AMF when thenoise density is moderate or high.





Adaptive Rank Order Filter : Illustration





Adaptive Rank Order Filter : Illustration Cont..,















Top Level Module of AROF





Detailed Architecture of AROF





3× 3 Sliding Window Example





Detailed Architecture of 3× 3 Sliding Window















Noise Detection Unit and Sorting Network Modules

(a) Impulse Noise Detector (b) Nine Element Sorting Module





Details of Nine Element Sorter





Details of Nine Element Sorter





Architecture of Compare & Swap Module

(c) Top Architecture of Compare &Swap

(d) Detailed Architecture of Com-pare & Swap





Sorting Networks : Comparison for 9 Elements

Bubble sort : 36 comparators, 14 parallel Operations.

Bose Nelson sort : 27 comparators, 11 paralleloperations.

Hibbard sort : 27 comparators, 12 parallel operations.

Bitonic sort5 : 28 comparators, 8 parallel operations.

Batchers merge exchange sort6 : 27 comparators, 8parallel operations.

Optimal sort : 25 comparators, 8 parallel operations(proposed).

5Zdenek Vasicek and Lukas Sekanina, ”Novel Hardware Implementation ofAdaptive Median Filters”, in proceedings of 11th IEEE Workshop on Design andDiagnostics of Electronic Circuits and Systems, pp. 1-6, April, 2008.

6Baddar, Sherenaz W. Al-Haj, and Kenneth E. Batcher, ”The AKS SortingNetwork,” Designing Sorting Networks, Springer New York, pp. 73-80, 2011.





Matlab Simulation Results of AMF and AROF for”Lena” Image

Figure: First Row: Original Images with Noise Level 20%, 40%, & 60%,Second Row: AMF Reconstructed Images, Third Row: AROFRecontructed Images





Matlab Simulation Results of AMF and AROF for”House” Image

Figure: First Row: Original Images with Noise Level 20%, 40%, & 60%,Second Row: AMF Reconstructed Images, Third Row: AROFRecontructed Images





PSNR & IEF Computation

The Peak Signal to Noise Ration (PSNR) & Image EnhancementFactor (IEF) are computed as follows:

PSNR = 10 log10

[2552

MSE

]MSE = 1

M×N

∑Mi=1

∑Nj=1

[I (i , j)− I (i , j)

]2

where E(x, y) is the enhanced gray element at position (x, y), I(x,y) is the original gray element at position (x, y) and, p and q denotethe size of the gray image.

IEF =∑M

i=1

∑Nj=1[n(i ,j)−I (i ,j)]2∑M

i=1

∑Nj=1[f (i ,j)−I (i ,j)]2

where n(x, y) is the noisy image, I(x, y) is the original image andf(x, y) is the reconstructed image.





Quality Assessment using PSNR (dB) and IEF forLena and House Image

Figure: First Row: PSNR for Lena & House Image, Second Row: IEFfor Lena & House Image





ModelSim Simulation Results : Validity of Input





ModelSim Simulation Results : AROF Pixel Output





Timing Diagram for Illustrating the PipeliningOperation of AROF System





RTL View of the Top Module AROF System





Zoomed View of the Top Module

Figure: U1: Impulse Noise Detection, U2: Sliding Window, U3: SortingNetwork, U4: Median Computation, U5: Delay Unit and U6: OutputSelection





FPGA (XC5VLX50-1FF1153) Device Utilization





Timing Summary

The timing summary for the AROF system as reported by Xilinx ISEtool is as follows:

Speed Grade : -1

Minimum period: 4.392ns (Maximum Frequency:227.668 MHz)

Minimum input arrival time before clock: 1.154ns

Maximum output required time after clock: 4.101ns

Clock period: 4.392ns (frequency: 227.668MHz)

Total number of paths / destination ports: 50736 / 5200

Delay: 4.392ns (Levels of Logic = 5)

Total number of paths / destination ports: 8 / 8

Offset: 1.154ns (Levels of Logic = 1)





Matlab & Hardware Results of ”Butterfly” Image

Figure: First Row: Original Images with Noise Levels 20%, 40%, &60%, Second Row: Matlab Reconstructed Images, Third Row:Hardware Recontructed Images





Matlab & Hardware Results of ”Traffic Signal”Image

Figure: First Row: Original Images with Noise Levels 20%, 40%, &60%, Second Row: Matlab Reconstructed Images, Third Row:Hardware Recontructed Images





Quality Assessment using PSNR (dB) and IEF forLena and House Image

Figure: First Row: PSNR for ”Butterfly” & ”Traffic Signal” Image,Second Row: IEF for ”Butterfly” & ”Traffic Signal” Image





Timing Summary

Table: Comparison of Present Implementation of AROF withAnother Implementation

Parameter Present Implementation Benkrid7

Picture Size (Pixels) 1600× 1200 512× 512

Frame Rate 118 25

7K. Benkrid, D. Crookes, and A. Benkrid, ”Design and Implementation ofa Novel Algorithm for General Purpose Median Filtering on FPGA’s”, In IEEEInternational Symposium on Circuits and Systems, (ISCAS2002), Vol. 4, pp.425-428, 2002.





Publication Details

An Efficient VLSI Architecture for Adaptive Rank OrderFilter for Image Noise Removal, in the International Con-ference on Signal Acquisition & Processing (ICSAP2011),Singapore, 26-28 Feb, 2011.

A Novel FPGA Implementation of Adaptive Rank OrderFilter for Image Noise Removal, In an International Jour-nal of Computer and Electrical Engineering (IJCE), Vol.4, No. 3, June 2012.





Adaptive Color Image Enhancement using GMF

HSV color space is adapted since it separates color fromintensity.

The contrast of color image is enhanced by providing highfrequency spatial information from the saturation compo-nent into luminance component.

Local correlation is computed for luminance enhancementusing Geometric Mean Filter (GMF).

Saturation component is enhanced by stretching its dy-namic range.

The hue component of HSV is preserved in order to avoidcolor distortion or shifting.

Sobel operator is used in order to smooth edges.





Related Work

Jayanta Mukherjee et al.8 proposed enhancement of colorimages by scaling DCT coefficients.

Advantages

Chromatic components are processed along withluminance.Visual quality is improved by reducing halo artifacts.

Disadvantages

This scheme violate gray world assumption.Improves the quality of an image at the cost ofcomputational complexity.

8Jayanta Mukherjee and Sanjit K. Mitra, ”Enhancement of Color Images byScaling the DCT Coefficients,” IEEE Transactions on Image Processing, Vol. 17,No. 10, pp. 1783-1794, 2008.





Related Work Cont.,

Gang Song et al.9 proposed adaptive color imageenhancement based on human visual properties in HSVspace.

Advantages

Image enhancement is based on arithmetic mean &variance computation.Smooths local variations in an image.Noise is reduced to some extent.

Disadvantages

Results in blurring effect.Image details such as sharpness and edges are notsatisfactory.

9Gang Song and Xiang-Lei Qiao, ”Adaptive Color Image Enhancement basedon Human Visual Properties,” in 3rd IEEE Conference on Industrial Electronicsand Applications (ICIEA 2008), pp. 1892-1895, 2008.





Related Work Cont.,

Tsai et al.10 proposed fast dynamic range compressionwith local contrast preservation algorithm.

Advantages

The contrast enhancement operation in this scheme isachieved by adaptive intensity transfer function and linearcolor remapping techniques.Processes 30 frames per second with the resolution of640× 480 pixels.

Disadvantages

However, there is still considerable room for optimizationof the design based on Verilog coding since the authoruses C++ for the design entry.

10Chi-Yi Tsai, ”A Fast Dynamic Range Compression with Local ContrastPreservation Algorithm and its Application to Real-Time Video Enhancement,”IEEE Transactions on Multimedia, Vol. 14, Issue 4, pp. 1140-1152, 2012.





Related Work Cont.,

Ming et al.11 has developed high performancearchitecture for enhancement of the video streamcaptured under non-uniform lightning conditions.

AdvantagesThe RGB to HSV color space conversion adapted in thisscheme use log-domain in order to avoid complex divisionprocess.Processes 30 frames per second with the resolution of640× 480 pixels.

DisadvantagesColor space conversion and image enhancement operationincreases the latency of the system.The HSV to RGB converter modules developed to achievereconstructed RGB images is not efficient fromcomputation point of view.

11M. Z Zhang, M. J Seow, and V. K Asari, ”A High Performance Architecturefor Color Image Enhancement using a Machine Learning Approach,” in Interna-tional Journal of Computational Intelligence, Vol. 2, Issue 1, pp. 40-47, 2006.





Related Work Cont.,

Stefano Marsi et al.12 proposed illumination reflectancevideo enhancement based on FPGA implementation. Al-though, this scheme reduces halo artifacts, large sized fil-ter module, multiplier module and divider block exploitedin this work increases the computation complexity.Faming et al.13 proposed a new pixel based variationalmodel for remote sensing multi-source image fusion usinggradient features. Visual inspection of the reconstructedimage reveals distortion in the spectral information whilemerging the multi-spectral data.

12Stefano Marsi and G. Ramponi, ”A Fexible FPGA Implementation forIlluminance-refectance video enhancement,” Journal of Real-Time Image Pro-cessing, pp. 1-13, 2011.

13Guixu Zhanga Faming Fang, Fang Li and Chaomin Shen, ”A VariationalMethod for Multi-source Remote-Sensing Image Fusion,” in International Journalof Remote Sensing, Vol. 34, Issue 7, pp. 2470-2486, 2013.





Algorithm Steps

Step 1 : Read the color image.

Step 2 : Transform the image from RGB to HSV colorspace.

Step 3 : Separate composite HSV into individual hue(H), saturation (S), and value (V) components.

Step 4 : Enhance value and saturation componentsadaptively based on geometric mean filter.

Step 5 : Preserve hue in order to avoid color distortion.

Step 6 : Combine separated components of H, S and Vinto composite HSV.

Step 7 : Transform back HSV to RGB color space.

Step 8 : Display the enhanced color image.





Proposed Enhancement Equations

Local Geometric Mean for Value or Luminance is given by :

Vw = 1mn [

∏i ,j∈wV (i , j)]

1mn where m, n represents window

size, V is the luminance,Vw is the geometric mean forluminance.Local Geometric Mean for Saturation is given by :

Sw = 1mn [

∏i ,j∈wS(i , j)]

1mn where m, n represents window

size, S is the saturation,Sw is the geometric mean forsaturation.Local Variance for Value or Luminance is given by :

σ2v (x , y) =

∑i ,j∈w

[V (i , j)− Vw (x , y)

]2where x, y is the

center pixel within the window.





Proposed Enhancement Equations Cont.,

Local Variance for Saturation is given by :

σ2s (x , y) =

∑i ,j∈w

[S(i , j)− Sw

]2

Local Correlation Coefficient for luminance and saturation isgiven by :

ρ(x , y) =∑

i,j∈w [V (i ,j)−Vw ][S(i ,j)−Sw ]√σ2

v (x ,y)σ2s (x ,y) where ρ(x , y) is an

adaptive measure for luminance enhancement.

T1(x , y) = K1

[V (x , y)− Vw (x , y)

]T2(x , y) = K2

[S(x , y)− Sw (x , y)

]ρ(x , y) where K1 and K2

are constants which is assumed as 2.






New Luminance Enhancement with Saturation Feedback isgiven by :

Venh(x , y) = V (x , y) + T1(x , y)− T2(x , y)

where Venh(x , y) is the enhanced luminance.

The saturation enhancement is given by :

Senh(x , y) = [S(x , y)]γ

where S(x , y) is the saturation component, γ is the stretchcoefficient which determines the degree of saturationenhancement, Senh(x , y) is the enhanced saturationcomponent.






The proposed method has low artifacts or noise.

Better contrast & Improved sharpness.

Our method avoids color distortion since hue is preserved.

Enhanced images are richer in color, have more clarity andbetter visual effects.

Provides a general framework for ”Cross-Component Un-Sharp Masking (USM)”, which is a color enhancementstrategy that may be applied to components from anycolor space.





Block Diagram of the Proposed AGMF

Figure: Block Diagram of the Proposed Adaptive Color ImageEnhancement Method based on Geometric Mean Filter





Flow Chart for RGB to HSV Color Space Conversion

Figure: Flow Chart for RGB to HSV Color Space Conversion





Flow Chart for HSV to RGB Color Space Conversion

Figure: Flow Chart for HSV to RGB Color Space ConversionM. C, Hanumantharaju, Ph.D Dissertation Defense on Development of VLSI Architectures for Image Enhancement Techniques: slide 62/146




Top Architecture of RGB-HSV Color SpaceConversion

(a) Signal diagram for HSV to RGBColor Space Converter

(b) Signal diagram for RGB to HSVColor Space Converter





Architecture for RGB to HSV Color Space Converter

Figure: Detailed Architecture for RGB to HSV Color SpaceConverterM. C, Hanumantharaju, Ph.D Dissertation Defense on Development of VLSI Architectures for Image Enhancement Techniques: slide 64/146




Architecture for HSV to RGB Color Space Converter

Figure: Detailed Architecture for HSV to RGB Color SpaceConverterM. C, Hanumantharaju, Ph.D Dissertation Defense on Development of VLSI Architectures for Image Enhancement Techniques: slide 65/146




Architecture Details of AGMF

Figure: Detailed Signal Diagram of Adaptive Color ImageEnhancement based on Geometric Mean Filter





Architecture for Geometric Mean Filter Module





Architecture for Geometric Mean Filter ModuleCont.,





Architecture for Histogram Equalization





Matlab Simulation Results of ”Tractor” Image

Figure: (a) Original Image (b) Histogram Equalized Image (c) ImageEnhanced using Gang et al. Method & (d) Proposed Method





Matlab Simulation Results of ”Office” Image






Matlab Simulation Results of ”Girl” Image






Performance Comparison





ModelSim Simulation Results : Validity of RGBInput





ModelSim Simulation Results : Validity of theEnhanced Output





RTL Top View of AGMF





RTL Detailed View of RGB to HSV Color SpaceConverter





RTL Detailed View of HSV to RGB Color SpaceConverter





FPGA Resource Utilization for RGB to HSVConverter





FPGA Resource Utilization for HSV to RGBConverter





FPGA Resource Utilization for Adaptive GeometricMean Filter based Color Image Enhancement





Timing Summary for RGB to HSV Conversion

The timing summary for the RGB to HSV Color Space Converter asreported by Xilinx ISE tool is as follows:

Speed Grade : -10

Minimum period: 29.730 ns (Maximum Frequency:33.635 MHz)

Minimum input arrival time before clock: 3.971 ns

Maximum output required time after clock: 8.047 ns

Clock period: 29.730 ns (frequency: 33.635 MHz)

Total number of paths /destination ports:158142043944387 / 824

Delay: 29.730 ns (Levels of Logic = 35)





Timing Summary for HSV to RGB Conversion

The timing summary for the HSV to RGB Color Space Converter asreported by Xilinx ISE tool is as follows:

Speed Grade : -10





Total number of paths /destination ports: 4867 / 727






Timing Summary for AGMF Design

The timing summary for the AGMF Design as reported by Xilinx ISEtool is as follows:

Speed Grade : -10





Total number of paths /destination ports: 187392 /18456






RGB to HSV and vice versa: Software Approach

Figure: First Row : Original Image, Image in HSV Space, RestoredImage from HSV Space. Second Row :Original Image, Image in HSVSpace, Restored Image from HSV Space.





RGB to HSV and vice versa: Hardware Approach

Figure: First Row : Original Image, Image in HSV Space, RestoredImage from HSV Space. Second Row :Original Image, Image in HSVSpace, Restored Image from HSV Space.





Matlab & Hardware Results of ”Nature” Image

Figure: First Row: Original ”Nature” Image, Photoflair Software En-hanced Image (PSNR:28.12 dB), Image Enhanced using Histogram Equal-ization (PSNR:29.01 dB) Second Row: Matlab Reconstructed Image us-ing Proposed AGMF (PSNR:30.12 dB), Verilog Reconstructed Image usingProposed AGMF (PSNR:30.98 dB)





Matlab & Hardware Results of ”Big Ben” Image

Figure: First Row: Original ”Big Ben” Image, Photoflair Software En-hanced Image ((PSNR:27.68 dB), Image Enhanced using Histogram Equal-ization (PSNR: 28.3 dB) Second Row: Matlab Reconstructed Image us-ing Proposed AGMF (PSNR:29.07 dB), Verilog Reconstructed Image usingProposed AGMF (PSNR:29.16 dB)





Conclusion

Adaptive color image enhancement in HSV color spacebased on Geometric Mean Filter was presented.

Luminance Enhancement is achieved using Saturation feed-back.

Geometric mean filter offers better reconstructed imagequality compared to arithmetic mean filter.

Experimental results presented shows that the color im-ages enhanced by the proposed algorithm are clearer,more vivid and more brilliant.

Performance of the proposed algorithm validated by ap-plying luminance, contrast and PSNR.

The FPGA implementation of AGMF processes images ofsize 1600× 1200 pixels at 121 frames per second.





Publication Details

Adaptive Color Image Enhancement based on Geomet-ric Mean Filter, In Proceedings of International Confer-ence on Communication, Computing and Security (ICCCS2011) NIT, Rourkela, India, Feb 12-14, ACM, 2011.

A Novel Reconfigurable Architecture for Enhancing ColorImage Based on Adaptive Saturation Feedback, In the In-ternational Conference on Advanced Information and Mo-bile Communication (AIM 2011), pp. 162-169, Nagpur,Maharashtra, India, April 21-22, Springer, 2011.

A Novel FPGA Implementation of Adaptive Color ImageEnhancement based on HSV Color Space, In the Inter-national Conference on Electronics and Computer Tech-nology (ICECT 2011), pp. 162-169, Kanyakumari, India,08-10 April, IEEE, 2011.





Gaussian Image Enhancement : Introduction

Retinex is a popular image enhancement method for bridg-ing the gap between images and the human observationof scenes.

Retinex Algorithm was Proposed by Edwin Herbert Landin 1986.

Proposed Gaussian based color image enhancement tech-nique is the modified version of NASA’s Retinex Algo-rithm

Retinex is a model of lightness and color perception ofhuman vision.

Retinex is an adaptive imaging algorithm.





Gaussian Image enhancement : An Example

Figure: Original Image and Enhanced Image using Proposed Method





Related Work

D. J Jobson et al.14 proposed a color image enhancementbased on multiscale retinex for bridging the gap betweencolor images and Human observation of scenes.

AdvantagesGood dynamic range compression and color constancy.Reconstructed images are favorable for Human visual per-ception and improve contrast.

DisadvantagesThis method fails to produce good color rendition for aclass of images that contain violations of the gray worldassumption.Unable to remove Halo artifacts completely.Many parameters were assumed such as alpha, beta, gain,Gaussian scales etc.

14D. J Jobson, Z. Rahman and G. A Woodell, ”A Multiscale Retinex forBridging the Gap Between Color Images and the Human Observation of Scenes”,in IEEE Transactions on Image Processing, Vol. 6, pp. 965-976, 1997.





Related Work Cont.,

Digital Signal Processors (DSPs)15 has been used for theimplementation of image enhancement algorithms.

AdvantagesImproved efficiency compared to general purpose comput-ers.

DisadvantagesOnly marginal improvement has been achieved since paral-lelism and pipelining incorporated in the design are inade-quate.This scheme uses optimized DSP libraries for complex op-erations and does not take full advantage of inherent par-allelism of image enhancement algorithm.The enhancement of 25-30 frames per second of large sizevideo frames with 1024 × 768 pixel resolution is still notpossible with DSPs.

15D. J Jobson, G. D Hines, Z. Rahman and G. A Woodell, ”DSP Implemen-tation of the Retinex Image Enhancement Algorithm”, In Visual InformationProcessing XIII, Proc. SPIE, Vol. 5438, pp. 13-24, 2004.





Related Work Cont.,

The neural network based learning algorithm16 provides anexcellent solution for the color image enhancement withcolor restoration.

Advantages

The hardware implementation of the algorithm parallelizesthe computation and delivers real time throughput for colorimage enhancement.

Disadvantages

The window related operations such as convolution, sum-mation and matrix dot products in an image enhancementarchitecture demands an tremendous amount of hardwareresources.

16M. Z Zhang, M. J Seow, and V. K Asari, ”A High Performance Architecturefor Color Image Enhancement using a Machine Learning Approach”, In Interna-tional Journal of Computational Intelligence Research-Special Issue on Advancesin Neural Networks, Vol 2, Issue 1, pp. 40-47, 2006.





Related Work Cont.,

Hiroshi Tsutsui et al.17 proposed an FPGA implementa-tion of adaptive real-time video image enhancement basedon variational model of the Retinex theory.

Authors claimed that the architectures developed in thisscheme are efficient and can handle color picture of size1900× 1200 pixels at the real time video rate of 60 fps.

However, the computational cost of the algorithm de-pends on the number of processing layers while the maxi-mum layers and iterations used are 5 and 30, respectively.

Also, authors have not justified how high throughput hasbeen achieved in spite of time consuming iterations to thetune of 30.

17Hiroshi Tsutsui, H. Nakamura, R. Hashimoto, H. Okuhata, and T. Onoye,”An FPGA Implementation of Real-time Retinex Video Image Enhancement”, InIEEE World Automation Congress (WAC), pp. 1-6. 2010.





Related Work Cont.,

Abdullah M. Alsuwailem et al.18 proposed a new approachfor HE using FPGAs.

Although efficient architectures were developed for HE,the reconstructed images using this scheme are generallynot acceptable.

The HE process loses the details such as edges, contrastand leads to over enhancement of noise in images.

The enhancement approach using adaptive HE schemealso fails to produce satisfactory results since the processenlarges the contrast of background noise while lesseningthe exploitable signal.

18Abdullah M. Alsuwailem and S. A Alshebeili, ”A New Approach for Real-time Histogram Equalization using FPGA”, In IEEE Proceedings of InternationalSymposium on Intelligent Signal Processing and Communication Systems (IS-PACS2005), pp. 397-400, 2005.





Proposed Gaussian Based Image EnhancementMethod

Original image (which is of poor quality and needing en-hancement) is read in RGB color space.

The color components are separated followed by the con-volution with 3×3 Gaussian kernel in order to smooth theimage.

Logarithmic operation is accomplished in order to com-press the dynamic range of the image and to improve lowintensity pixel values.

Gain/Offset adjustment is done in order to translate thepixels into the display range of 0 to 255.

The No. of scales, scales, gain and offset do not vary fromone image to another. This implies that the algorithm iscanonical.






Depending on circumstances, the proposed methodcould achieve :

Sharpening : Compensates for the blurring introduced byimage formation process.Color constancy processing : Improves consistency ofoutput as illumination changes.Good dynamic range compression and color renditioneffect.Canonical constant : independent of inputs.General enhancement algorithm for all types of pictures.Provides satisfactory results for bi-modal pictures.





Flow Diagram of Proposed Enhancement Method





Gaussian Kernels: 3× 3 kernel and 5× 5 kernel

116

1 2 1

2 4 2

1 2 1

1273

1 4 7 4 1

4 16 26 16 4

7 26 41 26 7

4 16 26 16 4

1 4 7 4 1

Figure: Gaussian Kernels: 3× 3 kernel and 5× 5 kernel





Proposed Enhancement Equations

The two dimension (2D) Gaussian function is defined by

g(x , y) = 12πσ2 e

− x2+y2

2σ2

The Gaussian convolution matrix is given by

G (x , y) = I (x , y)⊗ g(x , y)

Mathematically, 2D convolution can be represented as

G (x , y) =∑M

i=1

∑Nj=1 I (i , j)× g(x − i , y − j)

The convolution operation for a mask of 5× 5 is given by

P(x , y) =∑4

i=0 Wi×Pi∑4i=0 Wi






The logarithmic processing on a 2D image is carried out by using

the following Eqn. GL(x , y) = K × log2 [1 + G (x , y)]

This gain/offset correction is accomplished by using Eqn. givenbelow:

I ′(x , y) = dmaxGLmax−GLmin

[GL(x , y)− GLmin]

where dmax is the maximum intensity, which is chosen as, 255 foran image with 8-bit representation, GL(x , y) is the log-transformedimage, GLmin is the minimum value of log transformed image,GLmax is the maximum value of log transformed image, I ′(x , y) isthe enhanced image and, x and y represent spatial coordinates.





Block Diagram & Signal Description for the TopModule of Gaussian Based Image EnhancementSystem

Block Diagram Signal Description

Signals Descriptionclk This is the global clock signalreset n Active low system resetrin [7:0] Red color componentgin [7:0] Green color componentbin [7:0] Blue color componentro [7:0] Enhanced red color componentgo [7:0] Enhanced Green color componentbo [7:0] Enhanced Blue color componentpixel valid Valid signal for enhanced RGB pixel





Detailed Architecture of Gaussian Based ColorImage Enhancement System





Top Architecture of Serpentine Memory

Signal Diagram Signal Description

Signals Descriptionclk Global clock signalreset n Active low system resetpixel in [7:0] R/G/B Input pixelwindow valid Valid signal for sliding windoww11 [7:0] to w15 [7:0] First row pixel valuesw21 [7:0] to w25 [7:0] Second row pixel valuesw31 [7:0] to w35 [7:0] Third row pixel valuesw41 [7:0] to w45 [7:0] Fourth row pixel valuesw51 [7:0] to w55 [7:0] Fifth row pixel values





Detailed Architecture of Serpentine Memory System





Top Architecture of 2D Gaussian Convolution

Signal Diagram Signal Description

Signals Descriptionwindow valid Valid signal from sliding windowW11 to W15 First row pixel valuesW21 to W25 Second row pixel valuesW51 to W55 Fifth row pixel valuesG11 to G15 First row Gaussian kernel valuesG21 to G25 Second row Gaussian kernel valuesG51 to G55 Fifth row Gaussian kernel valuesconv out [7:0] Gaussian convolved output pixelsconv valid Valid signal





Detailed Architecture of 2D Gaussian ConvolutionProcessor for R/G/B Color Channels





Architectures of Adder, Multiplier and Logarithm

Adder & Multiplier ModuleLogarithm Module





Detailed Architecture of 24-bit Unsigned Adder





Detailed Architecture of Pipelined Multiplier Design





Results and Discussions

Simulation Tool : Matlab Ver. 7.6.0.324 (R2008a).

Resolution of Test Images : 640× 480 Pixels (VGA),800× 600 Pixels (SVGA), 1024× 768 Pixels (XGA).

Test Images :http://dragon.larc.nasa.gov/retinex/pao/news/http://visl.technion.ac.il/1999/99-07/www/http://ivrg.epfl.ch/index.html

Performance Metrics : Contrast Enhancement,Luminance Enhancement, Peak Signal to Noise Ratio(PSNR).

Histogram : Plotted to show Pixel Distribution.


http://dragon.larc.nasa.gov/retinex/pao/news/

http://visl.technion.ac.il/1999/99-07/www/

http://ivrg.epfl.ch/index.html




Comparison of Matlab Reconstructed Pictures UsingImage Enhancement Algorithms: ”Trees” Image

Figure: First Row:Original Image, Histogram Equalization, NASA’sMSRCR. Second Row:Chih et al. MSRCR, Proposed Method





Comparison of Matlab Reconstructed Pictures UsingImage Enhancement Algorithms: ”Palette” Image






Comparison of Matlab Reconstructed Pictures UsingImage Enhancement Algorithms: ”House” Image






ModelSim Simulation Waveforms for InputtingImage Data





Waveforms of Sliding Window Module for one of theR/G/B Color Components





Waveforms for Gaussian Convolution Output at21090 ns





Starting of Enhanced Pixel Data





Waveforms for Ending of Reconstructed Pixels at13,31,990 ns





Processing Time Report for the Top Design Moduleof Gaussian Based Image Enhancement System

Module Clock Cycles Required to Process each pixel dataSliding Window 1029Gaussian convolution 23Logarithm 1gain/offset correction 7





Timing Diagram for Illustrating the PipeliningOperation of the Proposed System





RTL View of the ”Gaussian IE”

RTL View of the Top Module

aZoomed RTL View

aNote: U1: Red Color,U2: GreenColor, U3: Blue Color Component Pro-cessors





Zoomed View of U1 or U2 or U3 Module





FPGA Resource Utilization for Gaussian Based ColorImage Enhancement Design





Matlab and Verilog Reconstructed ”Tree”, ”House”and ”Color Palette” Images

Figure: First Column: Original Images, Second Column: MatlabReconstructed Images, Third Column: Hardware Reconstructed Images





Matlab and Verilog Reconstructed ”Couple”, ”DarkRoad” and ”Memorial Church” Images

Figure: First Column: Original Images, Second Column: MatlabReconstructed Images, Third Column: Hardware Reconstructed Images





Performance Evaluation : Approximate WaveletEnergy Metric





Performance Evaluation : Detailed Wavelet EnergyMetric





Conclusion

Gaussian based color image enhancement algorithm wasdesigned and architectures were developed.

The proposed method is efficient from computation pointof view as compared to other researcher methods.

The visual quality of reconstructed pictures and image en-hancement achieved for various test images are comparedusing wavelet energy metric.

Color image enhancement system is implemented on Xil-inx Virtex-II Pro XC2VP40-7FF1148 FPGA device and iscapable of processing high resolution videos up to 1600×1200 pixels at 117 frames per second.

RTL compliant Verilog coding of our system fits into asingle FPGA chip with a gate count utilization of about321,804.





Publication Details

Design of Novel Algorithm and Architecture for Gaussianbased Color Image Enhancement, in International Con-ference on Advances in Computing, Communication andControl, Mumbai, India, 18-19 Jan, 2013.

Design and FPGA Implementation of a 2D Gaussian Sur-round Function with Reduced On-Chip Memory Utiliza-tion, in International Conference on Advances in Com-puting, Communication and Informatics (ICACCI2013),Mysore, India, 22-25 Aug, 2013.

A Novel Full-Reference Color Image Quality AssessmentBased on Energy Computation in the Wavelet Domain”,In Journal of Intelligent Systems, Vol. 22, No. 2, pp.155-177, May 2013.




Conclusions

The AROF have been used in this work to remove impulsenoise since AROF has better filtering properties comparedto AMF.

The core modules of the AROF system, namely, slidingwindow, impulse noise detection, sorting network, mediancomputation and output selection were realized using Ver-ilog for ASIC/FPGA implementation.

A Novel algorithm for color image enhancement basedon AGMF is proposed with the architecture developmentsuitable for FPGA/ASIC implementations.

The Verilog codes for the functional modules of AGMF,namely, RGB to HSV, histogram equalization, value com-ponent enhancement, HSV to RGB converter etc. havebeen developed and successfully simulated.




Conclusions Cont.,

Design of new algorithm and architectures for Gaussianbased color image enhancement system for real-time ap-plications has been presented.

The Gaussian color image enhancement functional mod-ules, namely, serpentine memory, 2D Gaussian convolu-tion, logarithm base-2 and gain/offset correction were re-alized using Verilog conforming to RTL coding guidelinespractised in Industries.

The designs presented exploits high degrees of pipelin-ing and parallel processing in order to achieve real timeperformance.

Quality assessment of image enhancement algorithms arebased metrics: CEP, LEP, PSNR, and WE etc.




Future Scope

The work presented throws open a number of work thatmay be undertaken by researchers in the near future.

The design of the proposed AROF is modular and flexible,and therefore, it can be upgraded to accommodate newmodules, both present and future, without appreciableincrease in hardware.

The functional modules of AROF, AGMF & Gaussian im-age enhancement residing in FPGAs presently can be re-placed by ASIC resulting in more compact, low power,high speed and cost effective system suitable for volumeproduction.

A medical image enhancement technique based on retinexcan also be designed and implemented on FPGA/ASIC bymodifying the algorithms and architectures.




VLSI Image Processing Groups in ForeignUniversities

Dr. Vijayan K. Asari, Old Dominion University, Norfolk, USAhttp://www.ece.odu.edu/~vasari/

Dr. Ryan Kastner, University of California, Sandiego http://cseweb.ucsd.edu/~kastner/main

Dr. Junguk Cho, University of California, Sandiego http://cseweb.ucsd.edu/~j10cho/index.html

Dr. Venkatesan Muthukumar, University of Nevada Las Vegas,USA http://www.ee.unlv.edu/~venkim/index.html

Dr. Ming Z. Zhang, Old Dominion University, Norfolk, USAhttp://caprolibra.com/Prfdex.html

Dr. Sudha Natarajan, NTU, Singapore http://www.ntu.edu.sg/home/sudha/


http://www.ece.odu.edu/~vasari/

http://cseweb.ucsd.edu/~kastner/main

http://cseweb.ucsd.edu/~kastner/main

http://cseweb.ucsd.edu/~j10cho/index.html

http://cseweb.ucsd.edu/~j10cho/index.html

http://www.ee.unlv.edu/~venkim/index.html

http://caprolibra.com/Prfdex.html

http://www.ntu.edu.sg/home/sudha/

http://www.ntu.edu.sg/home/sudha/



VLSI Image Processing Groups in Indian Universities

Dr. Swapna Banerjee, Dept. of EE, CAD and VLSI Laboratory,IIT Kharagpur, India.

Dr. Nitin Chandrachoodan, Dept. of EE, VLSI Laboratory, IITMadras, India.

Dr. S. Srinivasan, VLSI Laboratory, Dept. of EE, IIT Madras,India.

Dr. V. Kamakoti, Reconfigurable and Intelligent Systems Engi-neering Group (RISE Laboratory), Dept. of CSE, IIT Madras,India.

Sanjay Sing, Scientist Fellow, IC Design Group, CEERI, Pilani,India.

Dr. S. S. S. P Rao, Dept. of CSE, IIT Bombay, India.




VLSI Image Processing Journals

Elsevier Journal on Microprocessors and Micro-systems.

Springer Journal of VLSI Signal Processing Systems for Signal,Image and Video Technology.

IEEE Transactions on Very Large Scale Integration (VLSI) Sys-tems.

IEEE Transactions on Circuits and Systems for Video Technol-ogy.

IEEE Journal on Computer Architectures for Intelligent Ma-chines.

Journal of Circuits, Systems and Computers.




VLSI Image Processing Industries




Image Processing Books




VLSI Signal Processing Books




Refrences

Mohd Firdaus Zakaria, Haidi Ibrahim and Shahrel AzminSuandi, ”A Review: Image Compensation Techniques”, Pro-ceedings of Second International Conference on Computer En-gineering and Technology (ICCET-2010), 16-18 April, 2010.

C. Iakovidou, V. Vonikakis and I. Andreadis, ”FPGA implemen-tation of a real-time biologically inspired image enhancementalgorithm”, Journal of Real Time Image Processing, Vol. 3, No.4, pp. 269-287, 2008.




Refrences

Ming Z. Zhanga,Ming-Jung Seowa, Li Tao and Vijayan K.Asari,”A tunable high-performance architecture for enhance-ment of stream video captured under non-uniform lighting con-ditions”, Journal of Micrprocessors and Microsystems, Vol. 32,Issue 7, pp. 386-393, 2008.

Hiroshi Tsutsui, Hideyuki Nakamura, Ryoji Hashimoto, Hi-royuki Okuhata and Takao Onoye, ”An FPGA Implementationof Real-Time Retinex Video Image Enhancement”, Proceedingsof World Automation Congress (WAC), pp. 1-6, 19-23 Sept,2010.




Refrences

D. J Jobson, Z. Rahman, and G. A Woodell, A Multiscale retinexfor bridging the gap between color images and the human ob-servation of scenes, IEEE Transaction Image Processing, Vol. 6,No. 7, pp. 965-976, July 1997.

Xinghao Ding, Xinxin Wang, Quan Xiao, ”Color Image En-hancement with a Human Visual System based Adaptive Filter”,Proceedings of International Conference on Image Analysis andSignal Processing, April, 2010.

Hongqing Hu and Guoqiang Ni, ”The improved algorithm forthe defect of the Retinex Image Enhancement”, Proceedingsof International Conference on Anti-Counterfeiting Security andIdentification in Communication (ASID), pp. 257-260, July,2010.
