ICIP2016 image compression grand challenge

23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona

OVERVIEWANDBENCHMARKINGSUMMARYFORTHEICIP2016COMPRESSIONCHALLENGE

Evangelos Alexiou ,IreneViola,LukasKrasula,ThomasRichter,TimBruylants,AntonioPinheiro,KarelFliegel,MartinRerabek,Athanassios Skodras,PeterSchelkens andTouradjEbrahimi

AcontributionfromQualinet toJPEGcallforinformation

23rdInternationalConferenceonImageProcessingSeptember25– 28,2016,Phoenix,Arizona


Background

• Callforinformationonstillimagecoding– IssuedbyJPEG committeeonFebruary2015– Broadscopenotonlylimitedtocompressionefficiency• Newimagingmodalities (morethan8-bit,HDR,…)• Features (scalability,randomaccess,…)• Characteristics (complexity,latency,…)

– AfirstresponseproducedduringPCS2015– Bothlossy andlossless

• ICIP2016FeatureEvent– EvaluationofcurrentandfutureImagecompressiontechnologies

• Thiscontributiononlyfocusesoncompressionefficiencyofconventionalimagesinlossy and losslesswithouttakingintoaccountothercriteria(features,complexity,delay,etc.)• Objective andsubjective evaluations inlossy casecarriedoutbyQualinet– VUB/iMinds (Belgium)– UBI(Portugal)– CTU(CzechRepublic)– UniversityStuttgart(Germany)– UniversityPatras(Greece)– EPFL(Switzerland)

• Lossless evaluations carriedoutbyUniversityofStuttgart(Germany)

2

http://www.jpeg.org

http://www.qualinet.eu


Testmaterialinlossy evaluations• Contents:7 (1training+6test):– Resolutions - 800x1152or800x1280dependingoncontent– Subjectiveevaluationsoncroppedversionstofitdisplay– Objectivemetricsperformedonthecroppedversions

• Stimuli:– Originalimages– Compressed/decompressedimageswith10codecs• JPEG(default)• JPEG(PSNR)• JPEG (visual)• JPEG2000(PSNR)• JPEG2000(visual)• JPEGXR(444)• JPEGXR(420)• HEVC (SCCext.)• Daala• WebP

• 8bitratesforobjectivemetrics:– 0.25,0.5,0.75,1,1.25,1.5,1.75and2bpp

• 4bitratesforsubjectiveevaluations:– 0.25,0.5,0.75and1bpp or0.75,1,1.25and1.5bpp dependingoncontent

training

3

bike cafe honolulu

p08 p26 woman


Testmaterialinlosslessevaluations 4

RGB,444,24bpp


Croppedimagesusedinlossy evaluations

5

p08honolulu

bike cafe


Croppedimagesusedinlossy evaluations

6

p26 woman


Codecsusedinlosslessevaluations 7

1. JPEGXRLossless2. JPEG2000lossless3. FLIF (submittedtoICIP2016GrandChallenge)4. JPEGLSpart-1withoutcolortransformation5. JPEGLSpart-2withlosslesscolortransformation6. PNG7. WebP lossless8. JPEGXTpart8losslesswithresidualcoding9. JPEGXTpart8losslesswitharithmeticcoding10. JPEGXTpart8withlosslessDCT11. JPEG-1lossless12. JPEG-1losslesswitharithmeticcoding13. JPEGXTpart8,residualcodingwithoptimizedHuffmancoding,progressivemode14. JPEGXTpart8,losslessDCTwithoptimizedHuffmancoding,progressivemode.15. HierarchicalJPEGwiththeinitialpassaDCTcodingandthesecondlevelofthepyramidlosslesscodingand

optimizedHuffmancoding16. Sameas15,butwitharithmeticcodinginsteadofHuffmancoding17. Sameas15,exceptthatthefirstlevelofthepyramidisadownscaled¼andprocessedbytheDCT.compressed

withDCT,thenupscaled,andtheresidualiscompressedbythepredictivemodeofJPEG.18. Sameas17,butwitharithmeticcoding.


Codecsusedinlossy evaluations

• JPEG(default):IJGimplementationofJPEGwithdefaultoptions(i.e.AnnexKquantizationsettings,420colorsubsampling,baselineconfiguration) – “cjpeg” withoutfurtheroptions.

• JPEG(PSNR):JPEGXTreferencesoftwareusingJPEGXTpart1baselinecompressorcompatiblewith10918-1.PNSRoptimizedversion:-oz-h-qt 1–v– Enabling:444subsampling,flatquantization matrix,optimized Huffman coding,progressive scan order.

• JPEG(visual): JPEGXTreferencesoftwareusingJPEGXTpart1baselinecompressorcompatiblewith10918-1.Visuallyoptimizedversion: -oz-h-qt 3-v-s1x1,2x2,2x2– Enabling: 420subsampling,ImageMagick quantization matrix,optimized Huffman coding,progressivecoding.

8



• JPEG2000(PSNR):JPEG2000compressorusingAccusoft softwarewiththefollowingcommandlineoptions: -lo -as-cn 1– Enabling: strict rateallocator,5decomposition levels,one layer,noprecincts,notiles. lossy 5/3and not9/7wavelet.

• JPEG2000(visual):JPEG2000compressorusingAccusoft software.Visuallyoptimizedversion:-lo -as-cn 1-w 1000– Enabling: features identical to JPEG2000(PSNR),but with visualweighting.

• JPEGXR(444):JPEGXRreferencesoftwarewiththefollowingoptions: -fYUV444-l1–d– Enabling: 444chromasubsampling,oneleveloverlap,derivedchromaquantization.

• JPEGXR(420):JPEGXRreferencesoftwarewiththefollowingoptions:-fYUV420-l2–d– Enabling: 420chromasubsampling,twoleveloverlap,derivedchromaquantization.

9



• HEVC(SCCext.):HEVCwithScreenContentCodingextensionconfiguredtothemainintra-profilefor420chroma input,withthefollowingcommandlineoptions: -ccfg/encoder_intra_main_scc.cfg --InputChromaFormat=420--ProgressiveSource --FrameOnly -cf 420--FrameRate=30--FramesToBeEncoded=1--QuadtreeTULog2MaxSize=5--GOPSize=1--IntraPeriod=1--ConformanceWindowMode=1--AdaptiveQP=1--RateControl=0--TransquantBypassEnable=1--CrossComponentPrediction=0--ColourTransform=0

• Daala:mozilla Daala withdefaultconfiguration.Nofurtheroptions.

•WebP:GoogleWebP (partofWebM)withthefollowingcommandlineoptions:-m6-partition_limit 50-af

10


Subjectiveevaluationmethodology

• SubjectiveevaluationmethodologybasedonITU-TP.910• ACR-HR:AbsoluteCategoryRatingwithHiddenReference• Randomization ofpresentationorder• 5-level discretescale:bad,poor,fair,good,excellent• 10codecstestedfortheirsubjectivequality–10(codecs)x6(images)x4(bitrates)+6(originals)=246stimuli

• 21naïvesubjects participatedinVUB,UBIandEPFL labs• Eachsubject completed 3sessions of80stimuli(circa15minpersession,30minbreak)• Shorttraining forbad,fairandexcellentqualityillustrations• Display:AppleMacBookProRetina15inchorequivalent• Typicalofficeenvironment

11

Stimulus1

Vote1

Stimulus80

Vote80

time

Training


Objectiveevaluationmetrics

• PSNR–Widelyusedqualitymetricinimageprocessingcommunity.–PerformedforbothYchannelandRGB.

• SSIM:StructuralSimilarityIndex–Meanofsimilaritybetweenanimageundertestanditsreferencebasedonstructuralinformation.

•MSSIM:MultiscaleStructuralSimilarityIndex–MultiscaleversionofSSIM.

• FSIM:FeatureSimilarityIndex–BasedonSSIM.–Addsacomparisonoflow-levelfeaturesetsbetweenthereferenceandthedistortedimages.– analyzesthehighphasecongruencyextractinghighlyinformativefeaturesandthegradientmagnitude,toencodethecontrastinformation.– ThisanalysisiscomplementaryandreflectsdifferentaspectsoftheHVSinassessingthelocalqualityofanimage.–PerformedforbothYandCchannels.


Objectiveevaluationmetrics

• HDR-VDP2.2: HighDynamicRangeVisibleDifferencePredictor–CalibratedmetricdevelopedforHDRimages–Considersalight-adaptivecontrastsensitivityfunction,astherangesoflightadaptationcanvarysubstantially.– Includesaspecificmodelofthepointspreadfunction(PSF)oftheeyeoptics,ashumanopticallensflarecanbeverystronginhighcontrastHDRcontent.– Thefront-endamplitudenon-linearityisbasedonintegrationoftheWeber-Fechnerlaw.– Takesintoaccounttheangularresolution.–Usesamulti-scaledecomposition.–Aneuralnoiseblockisdefinedtocalculateper-pixelprobabilitiesmapsofvisibilityandthepredictedqualitymetric.

• CIEDE2000:Colordifferencemetric– Includesweightingfactorsforlightness,chroma,andhue(liketheCIE1976L*a*b*perceptualspace).–Alsoincludesfactorstohandletherelationshipbetweenchroma andhue.

• VIF:VisualInformationFidelity–Analysesthenaturalscenestatistics.–UsesanimagedegradationmodelandtheHVSmodel.–BasedonthequantificationoftheShannoninformationpresentinboththereferenceandthedistortedimages.


14

Thankyouforyourattention!Resultsofevaluationswillbepresentedattheendofthissession!!!


SUMMARYANDNEXTSTEPSFORTHEIMAGECOMPRESSIONCHALLENGE

Evangelos Alexiou,IreneViola,LukasKrasula,ThomasRichter,TimBruylants,AntonioPinheiro,Karel Fliegel,MartinRerabek,Athanassios Skodras,PeterSchelkens andTouradjEbrahimi

AcontributionfromQualinet toJPEGcallforinformation

23rdInternationalConferenceonImageProcessingSeptember25– 28,2016,Phoenix,Arizona


Overview

• ICIP2016FeatureEvent– EvaluationofcurrentandfutureImagecompressiontechnologies

• 7Contents usedinlossy evaluations(1training+6tests)– Resolutions800x1152or800x1280

• 10codecs testedinlossy evaluations(anchorsandproponents)• 8 bitrates forobjectivemetrics– 0.25,0.5.0.75,1,1.25,1.5,1.75,2bpp

• 4bitrates forsubjectiveevaluations– 0.25,0.5,0.75,1bpp or0.75,1,1.25and1.5bpp

• SubjectiveassessmentprotocolbasedonITU-TP.910forlossy evaluations– ACR-HR:AbsoluteCategoryRatingwithHiddenReference– 21naïvesubjects inVUB,UBIandEPFL– 3sessionsof80stimuli– 5-level discretescale–Outliersdetectionbasedon boxplotalgorithm– Display: AppleMacBookProRetina15inch– Typicalofficeenvironment

• Objectiveevaluation inlossy caseusing9metrics– Cross-checked between CTU,UniversityStuttgartandUniversityPatras

16

http://www.jpeg.org

http://www.qualinet.eu


Testimagesinlossy evaluations 17

training

bike cafe honolulu

p08 p26 woman


Objectiveevaluation:PSNRY results18


Objectiveevaluation:PSNRRGB results19


Objectiveevaluation:SSIMresults 20


Objectiveevaluation:MSSIMresults 21


Objectiveevaluation:FSIMY results22


Objectiveevaluation:FSIMC results23


Objectiveevaluation:HDR-VDP-2.2results 24


Objectiveevaluation:CIEDE2000results 25


Objectiveevaluation:VIFresults 26


Outlierdetection

• StatisticalanalysisusingBoxplot method.

• Foreachtestcondition,anoutlier isdefinedasadatapointthatislocatedoutsidetheinterquartilerange i.e.abovetheupperquartileorbelowthelowerquartileofthedistributionofthescoresmultipliedby1.5.

• Ifthesamesubjectisidentifiedasoutlierinmorethan20% ofthetestconditions,thecorrespondingscoresarediscarded.

• Nooutlierswerefound.

27


Subjectiveevaluationresults 28

bike.ppm

0 0.5 1 1.5bitrate [bpp]

1

1.5

2

2.5

3

3.5

4

4.5

5

MO

S

JPEG (default)HEVC (SCC ext.)JP2K (PSNR)JP2K (visual)DaalaJPEG XR (444)WebPJPEG (PSNR)JPEG (visual)JPEG XR (420)

cafe.ppm

0.5 1 1.5 2bitrate [bpp]

1

1.5

2

2.5

3

3.5

4

4.5

5

MO

S


bike cafe



honolulu.ppm


1

1.5

2

2.5

3

3.5

4

4.5

5

MO

S


p08.ppm


1

1.5

2

2.5

3

3.5

4

4.5

5

MO

S


honolulu p08



p26.ppm


1

1.5

2

2.5

3

3.5

4

4.5

5

MO

S


woman.ppm


1

1.5

2

2.5

3

3.5

4

4.5

5

MO

S


p26 woman


Statisticalsignificancetest

•One-tailedWelch’st-testwithnullhypothesis at5%significancelevel.• andareMeanOpinionScoresforaspecificcontentcompressedataspecificbitratewithcodecsand•Whenthenullhypothesisisrejected,thealternativehypothesisindicatesthat,accordingtoMOS,thefirstcodecissignificantlybetteratthe5%level.• Thesetestsareperformedconsideringallcombinationsofcodecs:10x10matrix.• Foreachpairofcodecs,wesumuptheresultsforeverycontentatagivenqualitylevel:scale0-6.•Onematrixforeachqualitylevel:4figures.

31

H0 :m1 ≤m2

m1 m2c2c1


Codecassessmentbasedonsubjectivetests 32

• Bitrate B1 correspondstothelowestquality• Bitrate B4 correspondstothehighestquality

Significant difference matrix for bitrate B1

JPEG (default)

HEVC (SCC ext.)

JP2K (PSNR)

JP2K (visual)

Daala

JPEG XR (444)

WebP

JPEG (PSNR)

JPEG (visual)

JPEG XR (420)

JPEG

(def

ault)

HEV

C (S

CC

ext

.)

JP2K

(PSN

R)

JP2K

(vis

ual)

Daa

la

JPEG

XR

(444

)

Web

P

JPEG

(PSN

R)

JPEG

(vis

ual)

JPEG

XR

(420

) 0

1

2

3

4

5

6Significant difference matrix for bitrate B2

JPEG (default)

HEVC (SCC ext.)

JP2K (PSNR)

JP2K (visual)

Daala

JPEG XR (444)

WebP

JPEG (PSNR)

JPEG (visual)

JPEG XR (420)

JPEG

(def

ault)

HEV

C (S

CC

ext

.)

JP2K

(PSN

R)

JP2K

(vis

ual)

Daa

la

JPEG

XR

(444

)

Web

P

JPEG

(PSN

R)

JPEG

(vis

ual)

JPEG

XR

(420

) 0

1

2

3

4

5

6


Codecassessmentbasedonsubjectivetests 33

• Bitrate B1 correspondstothelowestquality• Bitrate B4 correspondstothehighestquality

Significant difference matrix for bitrate B3

JPEG (default)

HEVC (SCC ext.)

JP2K (PSNR)

JP2K (visual)

Daala

JPEG XR (444)

WebP

JPEG (PSNR)

JPEG (visual)

JPEG XR (420)

JPEG

(def

ault)

HEV

C (S

CC

ext

.)

JP2K

(PSN

R)

JP2K

(vis

ual)

Daa

la

JPEG

XR

(444

)

Web

P

JPEG

(PSN

R)

JPEG

(vis

ual)

JPEG

XR

(420

) 0

1

2

3

4

5

6Significant difference matrix for bitrate B4

JPEG (default)

HEVC (SCC ext.)

JP2K (PSNR)

JP2K (visual)

Daala

JPEG XR (444)

WebP

JPEG (PSNR)

JPEG (visual)

JPEG XR (420)

JPEG

(def

ault)

HEV

C (S

CC

ext

.)

JP2K

(PSN

R)

JP2K

(vis

ual)

Daa

la

JPEG

XR

(444

)

Web

P

JPEG

(PSN

R)

JPEG

(vis

ual)

JPEG

XR

(420

) 0

1

2

3

4

5

6


Testmaterialinlosslessevaluations 34

RGB,444,24bpp


Losslessevaluationresults 35

6

8

10

12

14

16

18

Lossless compression

bpp

JPEG-XR

JPEG-2000 FL

IF

JPEG-LS

JPEG-LS2PNGWebP

JPEG-LL

JPEG-LLA

JPG-SRM

JPG-PRED

JPG-ACPRD

JPG-ACP

JPG-ACPLJPG-H

JPG-ACH

JPG-H1

JPG-ACH1


Conclusions 36

• HEVC andDaala oftenoutperformothercodecsinbothobjectivemetricsandsubjectiveevaluationsinlossy case–Daala performs bestinimagescontainingfaces

• JPEG,JPEG2000and JPEGXRperformwellin higherbitratesbasedonPSNRRGB metricinsometestedimagesinlossy case

• JPEG2000(PSNR)exhibitsgoodcolorrenditionbasedonCIEDE2000metricinseveraltestedimagesinlossy case

• JPEG(visual)performwellinhigherbitratesinlossy case

• FLIF inaverageproducesbestlosslesscompression performancewhencomparedtoallalternativestestedfortheimagestested


Nextsteps

• Thereisevidencethatsignificantimprovementsincompressionefficiencycanbeobtainedusinglateststateoftheartinlossy andlossless cases

• Furtherevaluationsareneededtobetterquantifythecostofsuchhigherefficiencyincompressionintermsofrequiredresourcesandotherfeatures(delay,etc.)

• Conclusionsregardingcompressionefficiencyneedtobeverifiedusingalargerdatasetandthroughmoreextensiveevaluationscampaignssuchascrowdsourcing

• Keepinmindtheseresultscompareencodersandnotcodingalgorithms(inparticulardecoders!)

•Manyofthealgorithmsundertestareindevelopmentandtheirperformancecanstillimprove

37


38

Thankyouforyourattention!

Education

ICIP2016 image compression grand challenge