Multimedia Services:Image

Sep-2015

Dani Gutiérrez PorsetAssociate Professor

Communications Engineering

Eman ta zabal zazu

2 2Multimedia Services:

Image

Thanks, Licences and Tools

● Thanks to people and organizations who took or take part in free software and free knowledge projects, specially Wikimedia Foundation and KDE

● This presentation is licensed as CC BY-SA 3.0 EShttp://creativecommons.org/licenses/by-sa/3.0/es/

● This presentation has been made with KDE, LibreOffice, Inkscape, Gimp, Chromium, Firefox

3 3Multimedia Services:

Image

Sources and References

● Images from Wikimedia Foundation, if not referenced other source. Logos and trademarks belong to respective organizations

● Texts:

– Wikipedia pages and referenced articles and material– “Guide to Voice and Video over IP” - Sun, Mkwawa, Jammeh,

Ifeachor– “Video over IP” - Wes Simpson– “Computer Networking, a top-down approach” - Kurose, Ross

4 4Multimedia Services:

Image

Index

● Introduction● Color● Raster images● Compression● Image files and formats

5 5Multimedia Services:

Image

Visual perception in humans

● Images and colors are sensations in our brain● Photoreceptors in eye:

– Rods, for brightness– LMS cones for color:

● S (short wavelength = blue)● M (medium wavelength = green)● L (long wavelength = red)

Introduction

6 6Multimedia Services:

Image

Sensitivity of human eye/brain

● More sensible to brightness than color● Less sensitive to the higher spatial frequency components● Less sensitive to quantizing distortion at high luminance

levels● Not all theoretical colors can be perceived (“imaginary

colors“)● More sensible to green. Actually, color sensitivity depends

on bright

Introduction

7 7Multimedia Services:

Image

Image Devices andCorrections (1/2)

● Devices:– Input: camera, eye,...– Output: monitor, projector, printer,...

● Gamut: which colors can be perceived or generated by a device

● Corrections:– For Brightness: Gamma: Mathematical function to consider human

perception of brightness. Relates Luma and Luminance– For Color: ICC profile: color space that characterizes colors in input

or output device, and uses a Profile Connection Space

Introduction

8 8Multimedia Services:

Image

Image Devices andCorrections: Workflow (2/2)

● In capturing devices, e.g. camera, when saving an image:– If raw format is used no correction applied– If non-raw format (e.g. JPEG), image is saved with

gamma encoded or compressed

● Each display device has a gamma value to decode gamma compressed images

Introduction

9 9Multimedia Services:

Image

Types of Digital Still Images

● Raster or Bitmap: represented by a matrix of pixels1

● Vector: represented by vectorial elements: points, vectors (strokes or paths: bézier curves), text,...– No quality loss if size or rotation altered.– for 2D and 3D. E.g.

● 2D: SVG, MathML, POV-Ray● 3D: Blender .blend, CAD .dwg,...

● Possible to mix raster and vectorial in some formats, e.g. SVG, WMF, EMF, PDF, EPS, Postscript, SWF (Shockwave Flash, includes animations,...)

1: All of the next slides in this presentation refer to raster images

Introduction

10 10Multimedia Services:

Image

Applications of Raster Images

● Icons: PNG, BMP, SVG● Web: JPG, PNG, GIF. Streamable using

progressive (interlaced) rendering● Cameras: RAW, EXIF● Graphic Edition: formats with lossless

compression, (proprietary formats)● Print Publishing: TIFF, PDF

Introduction

11 11Multimedia Services:

Image

Pixel Resolution

● For Surface processing devices (e.g. camera, display)

● Concept: pixel dimensions or Width x height pixels● Units: pixels (or megapixels,...)● For an image file it's the real size in pixels

Introduction

12 12Multimedia Services:

Image

Standards forPixel Resolution

● CIF (and variations)● SDTV (Standard Definition TV)● HDTV (High Definition TV): 1080p, 1080i, 720p● 4K family● 8K and Ultra HD TV● VGA, XGA,...

http://en.wikipedia.org/wiki/Graphics_display_resolution https://en.wikipedia.org/wiki/Common_Intermediate_Format https://en.wikipedia.org/wiki/List_of_common_resolutions

Introduction

13 13Multimedia Services:

Image

Aspect ratio

● Ratio of width to height pixels expressed as A:B● Some frequently used e.g. 4:3, 16:9 (HDTV), 21:9● Types:

– SAR: Storage Aspect Ratio– DAR: Aspect Ratio for the Display– PAR: Pixel Aspect Ratio. PAR = DAR / SAR

● Scaling, cropping and vertical/horizontal bands are used to adapt SAR to DAR

Introduction

14 14Multimedia Services:

Image

Spatial Resolution

● For line processing devices (e.g. scanner and printer)

● Concept: granularity in a line● Units: ppi (pixels per inch) or dpi (dots per inch)● For an image file the number of dpi or the real size

in metric units are just metadata

Introduction

15 15Multimedia Services:

Image

Introduction to Color

● Color spectrum (increasing wavelengths):

● Cases:– Device independent models– Device/eye dependent models

Color

16 16Multimedia Services:

Image

Color models and Color spaces

● Color model: mathematical model that uses tuples to reference colors– e.g. RGB, CMYK, YUV, HSV, LAB,...– Conversions between models with matrices

● Color space: organization of colors to represent colors precisely in terms of perception, coming:– From a model transformation for a color model, e.g.

RGB: sRGB, Adobe RGB,...– From a list of colors, e.g. Pantone System

● Profile Connection Space: reference color space used for conversions : CIELAB and CIEXYZ

● Terms “Model” and “Space” sometimes mixed

Color

17 17Multimedia Services:

Image

RGB Color model

● Red, Green, Blue. RGBA: 4th component A = Alpha● Additive model● Device dependent● Uses: electronic displays● Cube● Problems:

– Not efficient with “real-world” images– Not efficient for some image processing, e.g. modify intensity

Color

18 18Multimedia Services:

Image

CMY, CMYK Color model

● Cyan, Magenta, Yellow● Subtractive model● Device dependent● Derived from old RYB● Uses: printing processes, with BlacK

Color

19 19Multimedia Services:

Image

Y'UV, YUV, YCbCr, YPbPr,.Color models

● Components:– Non-color info (black, gray, white info). Terms:

● Luminance: measured. Symbol: Y● Luma: perceived. Symbol: Y'

– Chroma or Chrominance. Symbol: C. Similar concepts:● U, V: color information● Pb, Pr: Chroma for blue (B − Y) and for red (R − Y). For analog encoding (analog component video)● Cb, Cr: Idem for digital encodings● I, Q: In-phase, Quadrature, from rotation of U, V

– Sometimes YUVA: 4th component A = Alpha

● Uses: video systems (Y'UV in PAL, SECAM, YIQ in NTSC)● Advantages:

– Compatible with black and white analog TV– Some Chroma info can be discarded (better perception than RGB with higher compression)

Color

20 20Multimedia Services:

Image

HSV, HSL, HSB, HSIColor models

● Components:– Hue: distinguishes wich colour is– Saturation: intensity or colorfulness– 3rd component: Value, Lightness

(from black to white), Brightnessand/or Intensity

● Cylindrical coordinates. Same RGBcolor model represented in other way

● Uses: by digital artists. More intuitive representation for attributes recognized by human vision

Color

21 21Multimedia Services:

Image

LAB Color model

● LAB or La*b*:– L: Lightness– A, B: Chromacity, in color-opponent dimensions

● Device independent● Designed to approximate human vision● Wider gamut than RGB and CMYK and human eye. More data per

pixel required● Uses:

– Conversions from RGB to CMYK– Interchange, device-independent format

Color

22 22Multimedia Services:

Image

Color depth

● Number of bits to indicate the color:– For a single pixel: Bits per pixel– For each color component: “Bits per channel”, “Bits per color” (RGB,

YCbCr, CMYK) or “Bits per sample”. Alpha channel if transparency is used

● Representations:– Indexed or palette– Direct color:

● Number of bits depending on color space: 8, 15, 16, “True” color = 24, “Deep” color (30, 36, 48) bits. e.g. RGBA 32 bits

● Type e.g. floating values for HDR

Color

23 23Multimedia Services:

Image

Indexed Colors

● Better for images with low number of colors or large areas with solid colors, e.g. cartoons

● Types of palette:– Adaptive: optimized for each image, contains the most

frequent colors in image– Master: contains a miniaturized version of full RGB colors– ...

● Dithering algorithms are usually needed to transform from Direct (e.g. RGB) to Palette schema

Color

24 24Multimedia Services:

Image

Lossless compression methodsCompression

RLE or Run-lengthencoding

Adaptive dictionary

Deflation = combinationof Lz77 and Huffman

Chain codes

TIFF, BMP,...

LZW GIF, TIFF

PNG, TIFF

Monochrome images

25 25Multimedia Services:

Image

Lossy Image compressionand Human Perception

http://en.wikipedia.org/wiki/Image_compression

To relative differences betweendarker tones than lighter tones

To brightness informationthan color information

To green than blue and red

Gamma correctionto convert luminance in luma

Chroma subsampling

Luma with more G info than R, B:Y, Cb, Cr

To spatial low frequencytransitions than high freq

Quantization matrixfor DCT

Human eye is more sensitive... Method to apply:

Compression

26 26Multimedia Services:

Image

YCbCr and Chroma sub-sampling

● Lossy compression method based on Y'CbCr that takes less chroma samples than luma samples (horizontally and/or vertically), based on human visual perception

● Taking a region of w x 2 pixels, the notation is w:h:v– w: pixel width of the region. Generally w=4– h: number of Cb or number of Cr samples in the 1st row of the region– v: number of chroma changes samples between 1st and 2nd row of the region– Exception: not valid for 4:1:0 (region of 4 pixels height)

● Internal packaging in a file or stream: distinct dispositions, e.g. in Y'UV420p (p=“Planar”) first one component for all pixels, second other component for all pixels,...

http://en.wikipedia.org/wiki/Chroma_subsampling

http://en.wikipedia.org/wiki/YUV#Y.27UV420p_.28and_Y.27V12_or_YV12.29_to_RGB888_conversion

Compression

27 27Multimedia Services:

Image

Examples ofYCbCr and Chroma sub-sampling

Compression

28 28Multimedia Services:

Image

Other Lossy compression methods

● Palette and color quantization for reduced color space (GIF, PNG)

● Transform coding, ej. DCT (JPEG), Wavelet (JPEG 2000, DjVu)

● Fractal compression

Compression

29 29Multimedia Services:

Image

Structure of Image files

● Structure:– File metadata, e.g. Pixel resolution. If chroma subsampling there

maybe distinct pixel blocks– Possible file chunks and chunk metadata– Info for each pixel, packed in pixel by pixel schema or in channel

(planes) schema:● Color: finite or ~ infinite number of colors. Special cases: BW, gray● Alpha channel: Transparency (0%) - Opacity (100%)

● Image File size = f (chunks, resolution, color, compression)http://en.wikipedia.org/wiki/Image_file_format

http://en.wikipedia.org/wiki/Comparison_of_image_file_formats

Image files and formats

30 30Multimedia Services:

Image

Image Formats for Internet

● Supported natively in browsers:– Mandatory: GIF, JPEG, PNG– Some browsers: SVG, PDF,...

● Interlacing: incremental decoding in browsers, so they render progressively images showing first a degraded version that changes to final image:– Better for human perception. Targeted mainly to slower communications– Formats:

● GIF or PNG: images saved with "interlaced" option● JPEG: images saved with "progressive" option

Image files and formats

31 31Multimedia Services:

Image

JPEG compression and file format

● Theorically available compression types:– Lossy compression in three phases (see appendix)– Lossless coding, but not supported in most products

● Modes:– Baseline– Progressive: Data is compressed in multiple passes of progressively

higher detail. Progressive rendering in browsers

● Can embed an ICC color profile● Does not support transparency nor animation● Variations: JPEG 2000 (wavelets compression), JPEG XR,...

Image files and formats

32 32Multimedia Services:

Image

JPEG Encoding flow (1/3):Chroma subsampling

● 1.a Convert from RGB to Y'CbCr● 1.b Apply chroma subsampling, from 4:4:4 to

typically 4:2:2 or 4:2:0 [1st compression, lossy]

Image files and formats

33 33Multimedia Services:

Image

JPEG Encoding flow (2/3):Discard high-frequency variations

● 2.a Split the result image in blocks of 8x8 pixels for luma. MCU (Minimum Coded Unit) or macroblocks = blocks for luma and chroma: 8x8 (if 4:4:4), 16x8 (if 4:2:2) or 16x16 (if 4:2:0)

● 2.b Transform to frequency domain with DCT applied to each macroblock

● 2.c Quantize frequency components. One matrix for luma and one for chrominance

● 2.d Discard high-frequency coefficients due to human perception. This gives the bigger or smaller compression [2nd compression, lossy]

Image files and formats

34 34Multimedia Services:

Image

JPEG Encoding flow (3/3):Final entropy encoding

● 3. Compress the resulting 8 x8 blocks in zig-zag with entropy encoding [3rd compression, lossless]

The matrix from 2.d is like: Path for compression in 3:

(Logo in FFmpeg library: )

Image files and formats

35 35Multimedia Services:

Image

PNG file format● Lossless compression: filter + Deflate algorithm● 3 Type of images: Palette-based, Grayscale, Direct color RGB[A]● Color depth: bits/channel x no. of channels (Gray, R,G,B,A)● Bits/channel: 1, 2, 4, 8, 16 bits/channel (if alpha 64 bits/pixel), e.g.

PNG8: indexed with 8 bits, PNG24: RGB 8 bits/channel, PNG32: PNG24+Alpha● RGB space color● Chunks:

– Criticals: Header (width, height, color type, bit depth), Palette, Data– Others: ICC profile, gamma, text,...

● Interlacing: 2-dimensional, 7-pass● Does not support animation

Image files and formats

36 36Multimedia Services:

Image

TIFF file format

● Can be seen as a file format, and as a container for lossy and lossless data

● Much more powerful and complicated than PNG or JPEG● Applications:

– Base for lot of raw proprietary formats in cameras– Professional image editing

● Allowed compressions:– Lossless: CCITT Group 3 and Group 4 bi-level, LZW– Lossy: JPEG

● Color spaces of images: RGB, CMYK, YCbCr,...

Image files and formats

37 37Multimedia Services:

Image

Formats for image metadata

● Metadata: date and time, geolocation, data about the image (e.g. orientation),...

● Formats that specify how metadata is inserted into standard formats (e.g. JPEG, TIFF,...):– EXIF: Used in digital cameras and scanners– XMP– IPTC IIM

Image files and formats