Unsupervised Automation of Photographic Composition Rules Serene Banerjee and Brian L. Evans

Unsupervised Automation of Photographic Composition Rules

Serene Banerjee and Brian L. Evanshttp://www.ece.utexas.edu/~bevans/projects/dsc/index.html

Computer Engineering AreaDept. of Electrical and Computer EngineeringThe University of Texas at Austin

1/19/2004 Automation of Composition Rules 2

Motivation Problem: Amateur photographers often take

low-quality pictures with digital still cameras Personal use Professionals who need to document

(e.g.. realtors and architects) Goal: Automate photographic composition rules

and find alternatives to the picture being acquired Analyze scene, including detection of main subject Develop algorithms to automate rules

Main subjectcropped

Too muchbackground


Solution Solution #1: Automatically detect main subject

Independent of indoor/outdoor setting or scene Low implementation complexity, fixed-point computation

Solution #2: Automate a few photograph composition rules Rule of thirds for placing the main subject Simulated background blur for motion pictures or depth-of-field

Following rule-of-thirds Blur background for action pictures


1: Main subject2: Lenses3: CCD4: Imaging device5: Raw data

Digital Still Cameras Converts optical image to electric signal using charge

coupled device (CCD) Software control

Zoom Focus, e.g. auto-focus filter Shutter aperture and speed White balance: Corrects color distortions

Settings that can be controlled (with added hardware) Camera angle Aspect ratio: Landscape or portrait mode

Produces JPEG compressed images


Main Subject Detection Methods Two differently focused photographs [Aizawa, Kodama, Kubota; 1999-2002]

One has foreground in focus, and other has background in focus Significant delay involved in changing the focus

Bayes nets based training [Luo, Etz, Singhal, Gray; 2000-2001] Bayesian network trained on example set and tested later Training time involved: suited for offline applications

Multi-level wavelet coefficients [Wang, Lee, Gray, Wiederhold; 1999-2001] Expensive to compute and analyze wavelet coefficients

Iterative classification from variance maps [Won, Pyan, Gray; 2002] Optimal solution from variance maps and refinement with watershed Suitable for offline applications involving iterative passes over image


Proposed Main Subject Detection User starts image acquisition Focus main subject using auto-focus filter Partially blur background and acquire resulting picture

Open shutter aperture (by lowering f-stop) which takes about 1 s Foreground edges stronger than background edges

While acquiring user-intended picture, process blurred background picture to detect main subject Generate edge map (subtract original image from sharpened image) Apply edge detector (Canny edge detector performs well) Close boundary (e.g. gradient vector flow or proposed approximation)


Symmetric 3 x 3 sharpening filter For integer and , coefficients are

Integer when dropping 1/(1 + ) term Fractional when -1 – 2 and 1/(1 + ) is power-of-two

Generate edge map Subtract original image from sharpened image Main subject region now has sharper edges

Generate Edge Map

++

++-

f(x,y)g(x,y) fsharp(x,y)Smoothing

filter

0),(

and ),,(*),(),(),,(),(),(

where,),(detect Want to

yxf

yxgkyxfyxfyxfyxfyxg

yxg

sharp

smooth

1

111

)1(1

fsmooth(x,y)

k+

Model for an image sharpening filterSharpening filter


Boundary Closure Gradient vector flow method [Xu, Yezzi, Prince; 1998-2001]

Compute gradient Outer boundary of detected sharp edges is initial contour Change shape of initial contour, depending on gradient Shape converges in approximately 5 iterations Disadvantage: computationally and memory intensive

Approximate lower complexity method Select leftmost & rightmost ON pixel and make row between

them ON Can detect convex regions but fails at concavities


Automation of Rule-of-Thirds

Goal: Center of mass of the main subject at 1/3 or 2/3 of the picture width (or height) from the left (or top) edge

Solution: For n-D, define function that attains minimum when center of

mass placed as desired and increases otherwise Shift picture so that minimum is attained

Implementation: For 2-D, sum of Euclidean distance from the 4 points Measure which of the 4 points is closest to the current position

of the center of mass Shift picture so that the rule-of-thirds is followed


Simulated Background Blurring Goal: Filter the image background and add artistic effects

keeping the main subject intact Solution:

Original image masked with detected main subject mask Region of interest filtering performed on masked image Possible motion blurs

Linear blur: subject or camera motion Radial blur: camera rotation Zoom: change in zoom

Applications Enhance sense of motion where the main subject is moving Digitally decrease the depth-of-field of the photograph


Proposed Module

Measure how close rule-of-thirds

followed

Auto-focus filter

Lower f-stop for blur

Filter to generate edge map

Detect sharper edges

Close boundary

Original Image

Automate rule-of-thirds

Simulate background blur

Binary Main Subject Mask

Generated Picture with Rule-of-Thirds

Generated Picture with Blur


Implementation Complexity

Number of computations and memory accesses per pixel Main subject detection: convolution with symmetric 3x3 filter, edge

detection, approximate boundary closure Rule-of-thirds: center of mass (1 division, 4 compares) , shift pixels Background blurring: convolution with symmetric 3x3 filter

Digital still cameras use ~160 digital signal processor instruction cycles per pixel

Processing step Multiply-Accumulates /pixel

Comparisons/pixel

Memory accesses/pixel

Main subject detection 18 4 10Rule of thirds 2 1 1 or 3Background blurring 9 4


Results (1)Original image with main

subject(s) in focusDetected strong edges with proposed algorithm

Detected main subject mask

Rule-of-Thirds: Main subject repositioned

Simulated background blur














Conclusion Developed automated low-complexity one-pass method for

main subject detection in digital still cameras Processes picture taken with blurred background All calculations in fixed-point arithmetic

Automates selected photographic composition rules Rule-of-thirds: Placement of the main subject on the canvas Simulated background blur: motion and depth-of-field

Applications: digital still cameras, surveillance, constrained image compression, and transmission and display

Copies of MATLAB code, poster, and paper, available athttp://www.ece.utexas.edu/~bevans/projects/dsc/index.html

Documents

Unsupervised Automation of Photographic Composition Rules Serene Banerjee and Brian L. Evans