Multimedia (Social Forensics)

Multimedia Forensics:

discovering the history of

multimedia contents.

Prof. Sebastiano BattiatoDipartimento di Matematica e Informatica,

Università di Catania

Image Processing LAB – http://iplab.dmi.unict.it

[email protected]

2nd Meeting EU IAI – Interpol Headquarter (Lyon) – October 2016

http://iplab.dmi.unict.it/


Multimedia

Forensics

- Source identification

- Integrity verification/tampering detection

Techniques from multimedia forensics merely provide a way to

test for the authenticity and source of digital sensor data. In this

sense is not about analyzing the semantics of digital or

digitized media objects.


Original File: Special Cases

• Recapture: create a fake and then take a

picture with the camera we want to

pretend the picture was taken with

• Staging: the image file is authentic, but

the content has been staged

In these cases an authentic file does not

imply an authentic content.

Multimedia Forensics


Multimedia Forensics (in practice)

• Source Identification

• Integrity/Authenticity

• Enhancement/Restoration

• Interpretation and Content Analysis– Plate Recognition

– Dynamic Reconstruction (car crashes, etc.)

– Antropomethric issues

– …


“Forensics Image (Video) analysis is

the application of IMAGE SCIENCE

and DOMAIN EXPERTISE to interpret

the content of an image or the image

itself in legal matters” (SWGIT –

www.fbi.gov)



Recent documents:

• 2016-02-08 SWGDE Best Practices for Photographic Comparison for AllDisciplines

• 2016-02-08 SWGDE Image Processing Guidelines Version1.0

• 2016-02-08 SWGDE Proposed Techniques for Advanced Data Recoveryfrom Security Digital Video Recorders v1-1

• 2016-02-08 SWGDE Training Guidelines for Video Analysis, ImageAnalysis and Photography V1-1

https://www.swgde.org/

https://www.swgde.org/documents/Released For Public Comment/2016-02-08 SWGDE Best Practices for Photographic Comparison for All Disciplines

https://www.swgde.org/documents/Current Documents/2016-02-08 SWGDE Image Processing Guidelines

https://www.swgde.org/documents/Current Documents/2016-02-08 SWGDE Training Guidelines for Video Analysis, Image Analysis and Photography_v1-1

https://www.swgde.org/documents/Current Documents/2016-02-08 SWGDE Training Guidelines for Video Analysis, Image Analysis and Photography_v1-1

ISO Guidelines


Fantasy/Fiction


CSI Effect



Esper Blade Runner



Reality


I Need That Plate! No Way...

I Need That Plate! No Way...

Boston Marathon

“The FBI, reportedly has more than 2,000 agents looking at the publicly

available evidence,”

Challenging Problems

Prof. Sebastiano Battiato – CF 2015-2016


(source Interpol)

Multimedia Forensics is based on the idea

that inherent traces (like digital fingerprints)

are left behind in a digital media during both

the creation phase and any other

successively process.


• Example:• Forensic analysis of a smartphone: which pictures have been generated

on the device and which ones have been generated by other devices

and sent by messaging application or saved from the internet

• We can identify:• Type of device

• Maker and model

• Specific exemplar

Camera BallisticsWhich Device Has Created This Picture?

Device Identification

Model Identification

http://snapsnapsnap.photos/how-does-the-iphone-6-camera-compare-to-previous-iphone-cameras/

Camera Identification

Source Identification Noise Based

Sensor output carries not only pure signal

but also various noise components. Sensor

noise model could be used as a

representative feature for cameras.


PRNU as a camera fingerprint

PRNU Estimation


Sensor Identification Using

Pattern Noise


[Lukas2006] J. Lukas, J. Fridrich, and M. Goljan, “Digital Camera identification from sensor pattern noise” IEEE

Transaction Inf. Foren.Sec. Vol. 1, 205–214 (2006).

Sensor Identification Using

Pattern NoiseThis method provide good results, and is

quite reliable also using:

–images with different level of JPEG

compression (low, medium)

–images processed using point-wise operator

such as brightness/contrast adjustment or

gamma correction.

–images acquired by two cameras of the same

brand and model.


Integrity: What is a Forgery?• “Forgery” is a

subjective word.

• An image canbecome a forgery

based upon the

context in which

it is used.

• An image altered for fun or someone who has taken an badphoto, but has been altered to improve its appearancecannot be considered a forgery even though it has beenaltered from its original capture.


Altering Images

The concepts have movedinto the digital world byvirtue of digital camerasand the availability ofdigital image editingsoftware

The ease of use of digital image editing software, which doesnot require any special skills, makes image manipulation easyto achieve.

circa 1860: This nearly iconic portrait of U.S. President Abraham Lincoln is a composite of Lincoln's head and the Southern politician John Calhoun's body.


Who Cares?

media


Who Cares?

geopolitics…

…and political propaganda2nd Meeting EU IAI – Interpol Headquarter

(Lyon) – October 2016



Advertisement

More (and more) examples

Photo Tampering through History

http://www.fourandsix.com/photo-tampering-history/

Photoshopdisaster

http://www.photoshopdisasters.com/


http://www.fourandsix.com/photo-tampering-history/

http://www.photoshopdisasters.com/

Image Editing

Malicious image editing alters the image semantic

content, mainly:

Adding information

Removing information


Piva 2013

Image Editing• Splicing (two images)

– Also called cut and paste, compositing

– Used to add information

• Cloning (single image)

– Also called copy and paste, copy move, region duplication

– Used to add or remove information

– Can be exact, or the clone can be resized, rotated…)

• Inpainting (kind of intelligent clone)

– Seam carving, content aware resize, content aware fill, content dependent crop

– Used to remove information

• Retouch (local editing)

– Dodge and burn, healing tool…

• Image enhancement/filtering

– Histogram equalization, contrast enhancement, median filtering, denoise, smooth…

• Image editing (geometric transformation)

– Resize, crop, zoom, shear


How To Authenticate An Image?

• Visual Inspection

• File AnalysisFile Format and Structures

Metadata (EXIF)

Compression Parameters (Quantization

Tables)

• Global AnalysisPixel and compressed data statistics

• Local AnalysisFinding inconsistencies of pixel statistics

across the image

Image Forensics Methods

Passive Methods: Using the alterations ofthe underlying statistics produced by digitalforgeries on an image:

PHYSICS BASEDCAMERA BASED

PIXEL BASEDGEOMETRIC BASED

FORMAT BASED2nd Meeting EU IAI – Interpol Headquarter (Lyon) – October 2016

Physics-BasedLighting inconsistencies can used for revealing traces of

digital tampering.


Camera-Based

INTERPOLATION

LENS CFA SENSOR

POST PROCESSINGDIGITAL IMAGESTORAGE

Processing and Storage

ORIGINALIMAGE

Acquisition


Types Of Analysis: Signal Level

Based on statistical features of pixel values; need good quality image

• Clone detection

– Cloned image blocks

– Similar couples of key points

• Resampling detection

– For resize, rotate, but also when splicing or cloning

• Enhancement Detection

– Specific for algorithms (median, histogram equalization, color adjustment)

• Seam carving detection

• General intrinsic footprints

• Inconsistencies from acquisition and coding fingerprints

– CFA, PRNU, DCT, ELA…


Format-BasedJPEG compression engine

(for both luminance and chrominance channels):

the input image ispartitioned into 8x8non-overlapping blocks


Format-BasedJPEG forgery engine

2nd Meeting EU IAI – Interpol Headquarter


Periodic artifact introduced by Double JPEG quantizations (2)

A. C. Popescu and H. Farid, Statistical tools for digital forensics, in Proc. 6th Int. Workshop Information Hiding, Berlin,

Germany, 2004, pp. 128–147, Springer-Verlag.

Z. Lin, J. He, X. Tang, and C.-K. Tang, Fast, automatic and fine-grained tampered JPEG image detection via DCT

coefficient analysis, Pattern Recognition, vol. 42, no. 11, pp. 2492–2501, Nov. 2009.

If q2<q1, then n(u2) =0 for some u2, hence the histogram related to the double

quantization can show periodically missing values. On the contrary, if q2>q1 the

histogram can have some periodicity in terms of peaks and valleys pattern.



THE TYPICAL PIPELINEFOR A COPY-PASTE

OPERATION

+

=

original image

QF(1) = q1

resulting image

QF(3) = q3

2nd image

QF(2) = q2

duplicating

resizing



F. Galvan, G. Puglisi, A. R. Bruna, S. Battiato, First Quantization Matrix Estimation from Double Compressed JPEG Images, IEEE Transactions on Information Forensics and Security, 2014.


Image Alignment for Tampering Detection


S. Battiato, G. M. Farinella, E. Messina, G. Puglisi - Robust Image Alignment for Image

Authentication and Tampering Detection – IEEE Transactions on Information Forensics & Security,

Vol. 7 – Issue 4, pp. 1105-1117, 2012.


http://revealproject.eu/

http://www.rewindproject.eu/

http://maven-project.eu/#_=_

https://s-five.eu/

The final draft of the FIVE Best Practice

Manual is publically available from

December 8, 2015 ("October/DIWG2015

version"): DRAFT_BPM_FIVE_20151009

https://www.s-five.eu/sftp/BPM/DRAFT_BPM_FIVE_20151208.doc

Use Cases




Image Manipulation: Case “Mozzarella Blu”



Evidence on the web


Forgery on Biomedical Images

Corriere della Sera – Ottobre 2013


Forgery on Science

“What’s in a picture? The temptation of image manipulation.,” J. Cell Biol., vol. 166, no. 1, pp.

11–5, Jul. 2004.2nd Meeting EU IAI – Interpol Headquarter (Lyon) – October 2016


Current Trends And

Challenges

Current Trends: Point&Shoot

and Share…


Future of Imaging

Nikon

Sharing

The Social Picture

S. Battiato, G. M. Farinella, F. L. M. Milotta, A. Ortis, L. Addesso, A. Casella, V. D'amico, G. Torrisi, The Social Picture, ACM International Conference on Multimedia Retrieval 2016


Social (Multimedia) Forensics

• Image and Video Phylogeny

ReVeal project

Social (Multimedia) Forensics

• Uploading an image on a Social Network

- The process alters images

- Resize

- Rename

- Meta-Data deletion/editing

- Re-Compression

- NEW JPEG file Structure

M. Moltisanti, A. Paratore, S. Battiato, L. Saravo - Image Manipulation on Facebook for Forensics

Evidence – ICIAP 2015, LNCS 2015;

O. Giudice, A. Paratore, M. Moltisanti, S. Battiato - A Classification Engine for Image Ballistics of

Social Data – (Arxiv 2016 No. 1699257) http://arxiv.org/abs/1610.06347

Social (Multimedia) Forensics (2)

• Uploading an image on a Social Network- The process alters images

- Each Social Network Service do different alterations

Resized

Proportionally

Squared

Image





- Each Social Network Service makes differentalterations





- Each Social Network Service makes different alterations









Social Network

Fingerprint

on Uploaded

Images




- Alterations are dependent to uploading client


• Social Image Ballistics (recover image history)

Uploaded images

dataset



Social Altered image dataset- 10 Social Platforms

- Facebook, Google+, Instagram, Flickr, Tumblr,

Twitter, Imgur, Tinypic, Telegram, Whatsapp

- 2720 JPEG Images representing different

subjects (natural, indoor, outdoor)

- Dataset available at: http://iplab.dmi.unict.it/DigitalForensics/social_image_forensics/



- On which Social Network was uploaded image I?

IGiven a JPEG image I, the Social Image Ballistics task has the objective of

defining:

1) if there is a compatibility between the non-related JPEG elements of I

(i.e. filename, EXIF data) and the processing pipeline of SNSs;

2) if there is a compatibility between the JPEG elements of I and the

processing pipeline of SNSs;

3) which SNS is compatible with the JPEG elements of the image, with a

certain degree of confidence, and what is the uploading source in

terms of operating system (OS) and application. Input Image






I

Input Image

Feature

Extraction






I

Input Image

Feature

Extraction




• Social Image Ballistics (recover image history)- On which Social Network was uploaded image I?

Representation of whole Dataset






Input Image

Feature

Extraction

• DQTs coeffs

• Image Size

• # EXIF

• # JPEG Markers

Anomaly

Detection

The Anomaly Detector excludes images not processed

by Social Network Platforms

Given a Similarity measure between features extracted

from images:

It is possible to build a distance matrix D of size N×N

where the element dij is equal to the distance

between the images Ii and Ij.

The Anomaly Detector is then defined as:




Input Image

Feature

Extraction

• DQTs coeffs

• Image Size

• # EXIF

• # JPEG Markers

Anomaly

Detection

SNS

Classification

Upload Client

Classification

Output: Not in our dataset

The image probably is not altered by a SNS

Image does not come from considered platforms










Conclusions

• Multimedia Forensics is now a

consolidated field but new intriguing

challenges emerge every day.

• Among other current trends include:

– Big Data analysis (e.g. Social Network) by

«deep» paradigm?

– Advanced Video Synopsis (First-person-

Vision)

– Semantic Exploitation of user-generated

content


Surveys

• Matthew C. Stamm, Min Wu and K. J. Ray Liu, Information

Forensics: An Overview of the First Decade (2013), in: IEEE

Access, 1(167-200)

• Alessandro Piva, An Overview on Image Forensics (2013), in:

ISRN Signal Processing, 2013 (Article ID 496701, 22 pages)

• C. Baron - Adobe Photoshop Forensics – Sleuths, Thruts, and

Fauxtography – Thomson Course Tehcnology - 2009


On line Resources• Tutorial by Prof. Hany Farid - Digital Image Forensics:

lecture notes, exercises, and matlab code for a survey

course in digital image and video

forensics. http://www.cs.dartmouth.edu/farid/downloads/tut

orials/digitalimageforensics.pdf

• SOFTWARE: Amped5, Authenticate, Adroit, Four&Six,

Izitru, Ghiro, …


http://www.cs.dartmouth.edu/farid/downloads/tutorials/digitalimageforensics.pdf

Other related works

• Wang W. and Dong J. and Tan T.: Exploring DCT CoefficientQuantization Effects for Local Tampering Detection, IEEETransactions on Information Forensics and Security, 9, 10,1653–1666, (2014)

• Liu Q. and Sung A.H. and Chen Z. and Chen L.: ExposingImage Tampering with the Same Quantization Matrix,Multimedia Data Mining and Analytics, 327–343, (2015)

• C. Pasquini, F. Perez-Gonzlez, Giulia Boato: A Benford-Fourier JPEG compression detector. ICIP 2014:

• C. Pasquini, G. Boato, F. Perez-Gonzlez Multiple JPEGcompression detection by means of Benford-Fouriercoefficients. WIFS 2014

2nd Meeting EU IAI – Interpol

Headquarter (Lyon) – October 2016

Main Scientific PublicationsM.Moltisanti, A.Paratore, S. Battiato, L. Saravo - Image Manipulation on Facebook for

Forensics Evidence – ICIAP 2015, LNCS 2015;

F. Galvan, G. Puglisi, A. R. Bruna, S. Battiato, First Quantization Matrix Estimation from

Double Compressed JPEG Images, IEEE Transactions on Information Forensics and

Security, 2014

S. Battiato, G. M. Farinella, E. Messina, G. Puglisi - Robust Image Alignment for Image

Authentication and Tampering Detection – IEEE Transactions on Information Forensics

& Security, Vol. 7 – Issue 4, pp. 1105-1117, 2012.

S. Battiato, G. M. Farinella, G. Puglisi, D. Ravì – Aligning Codeboooks for Near

Duplicate Image Detection – Multimedia Tools and Applications - Springer 2013.

S. Battiato, G. Messina - Digital Forgery Estimation into DCT Domain - A Critical Analysis

- In Proceedings of ACM Multimedia 2009 - Workshop Multimedia in Forensics - Bejing

(China), October 2009.

S. Battiato, G.M. Farinella, G.C. Guarnera, T. Meccio, G. Puglisi, D. Ravì, R. Rizzo - Bags

of Phrases with Codebooks Alignment for Near Duplicate Image Detection – In

Proceedings of ACM Multimedia – Workshop Multimedia in Forensics, Security and

Intelligence (MiFor 2010) – Florence (Italy), October 2010;


IISFA memberbook• S. Battiato, G. Messina, R. Rizzo - Image Forensics - Contraffazione Digitale e

Identificazione della Camera di Acquisizione: Status e Prospettive - Chapter in IISFA

Memberbook 2009 DIGITAL FORENSICS - Eds. G. Costabile, A. Attanasio - Experta, Italy

2009;

• S. Battiato, G.M. Farinella, G. Messina, G. Puglisi - Digital Video Forensics: Status e

Prospettive - Chapter in IISFA Memberbook 2010 DIGITAL FORENSICS - Eds. G. Costabile,

A. Attanasio - Experta, Italy 2010

• S. Battiato, G.M. Farinella, G. Puglisi - Image/Video Forensics: Casi di Studio - Chapter in

IISFA Memberbook 2011 DIGITAL FORENSICS - Eds. G. Costabile, A. Attanasio - Experta,

Italy 2012.

• S. Battiato, M. Moltisanti – Tecniche di Steganografia su Immagini Digitali – Chapter in

IISFA Memberbook 2012 DIGITAL FORENSICS - Eds. G. Costabile, A. Attanasio - Experta,

Italy (2013)

• S.Battiato, F. Galvan, M. Jerian, M. Salcuni - Linee Guida per l'autenticazione Forense di

Immagini – Chapter in IISFA Memberbook 2013 DIGITAL FORENSICS - Eds. G. Costabile,

A. Attanasio - Experta, Italy (2013)

• S. Battiato, A. Catania, F. Galvan, M. Jerian, L.P. Fontana – Acquisizione ed Analisi

Forense di Sistemi di Videosorveglianza - Chapter in IISFA Memberbook 2014 DIGITAL

FORENSICS - Eds. G. Costabile, A. Attanasio - Experta, Italy 2015


Sicurezza e Giustizia• S.Battiato, F. Galvan - Introduzione alla Image/Video Forensics - Sicurezza e

Giustizia - Numero I/MMXIII - pp. 42-43 – 2013.

• S.Battiato, F. Galvan - La Validità Probatoria Delle Immagini e dei Video-

Sicurezza e Giustizia - Numero II/MMXIII - pp. 30-31 – 2013

• S.Battiato, F. Galvan - Ricostruzione Di Informazioni 3d A Partire Da Immagini

Bidimensionali - Sicurezza e Giustizia ( n.IV_MMXIII ) – 2014

• S.Battiato, F. Galvan - Verifica dell'Attendibilità di un Alibi Costituito da

Immagini o Video - Sicurezza e Giustizia - Numero II/MMXIV - pp. 47-50 – 2014.

• Rundo, E. Tusa, S. Battiato - Medical Image Enhancement nei Procedimenti

Giudiziari Medico-Legali in ambito Oncologico - Sicurezza e Giustizia - Numero

I/MMXVI - pp. 53-56 - 2016

• - See more at: http://www.sicurezzaegiustizia.com/


Prof. Sebastiano Battiato

Dipartimento di Matematica e Informatica

University of Catania, Italy

Image Processing LAB – http://iplab.dmi.unict.it

[email protected]


http://iplab.dmi.unict.it/

mailto:[email protected]

Main Contacts

Further Info

Image Processing Lab

Università di Catania

www.dmi.unict.it/~iplab

Email

[email protected]


Presentations & Public Speaking

Multimedia (Social Forensics)