Efficient  Gradient-‐Domain  Composi4ng  Using  Quadtrees  

By  Dus4n  Shean  and  Vera  Dadok  

What  this  Produces  

Background  A  QuadTree  is  a  tree  data  structure  that  has  exactly  four  children.      

Pros  of  Gradient  Composi4ng  

•  You  can  merge  together  images  without  seams  as  seen  on  Monday  

Cons  of  Gradient  Composi4ng  

•  Gradient-‐domain  are  slow  

•  Memory  Constraint:  Takes  O(n)  where  n  is  the  number  of  pixels  in  the  image    

This  Paper  

•  An  approxima4on  algorithm  •  Produces  a  photo  that  looks  the  same  but  in  reduced  4me  and  memory  

•  Takes  O(p)  where  p  is  the  total  length  of  the  seams  between  image  regions  

•  Reduces  to  sqrt(n)  

What  our  eyes  no4ce  

What  our  eyes  no4ce  

This  Paper  

•  Takes  advantage  of  the  human  eye  system.  •   Performs  slow  changes  in  luminance  to  avoid  the  eye  no4cing  seams  

•  Takes  high  frequency  ar4facts  and  spread  into  low-‐frequency  over  the  image  

This  Paper  

•  A  color  composite  and  its  gradient  tend  to  be  smooth  

•  Smoothness  calculated  prior  to  running  algorithm  

•  Solves  the  problem  in  a  reduced  space  by  assuming  certain  regions  of  the  solu4on  are  smooth.  

Important  Math  

•  x  is  the  n  rgb  pixels.  A  has  at  most  2  non  zero  values  per  row  

•  Apply  AT  to  both  sides.  AT  Ax  is  equivalent  to  applying  the  Laplacian  to  x  

This  Paper’s  Math  

 •  b-‐Axo  is  zero  for  any  pixel  not  adjacent  to  a  boundary  

•  x  =  xo  +  xd.  xo  is  the  ini4al  image  guess,    xd  is  the  difference  between  the  ini4al  condi4on  

Algorithm  Steps  

•  Input  Stage  –  Builds  the  quadtree  •  Compute  –  Traverse  the  quadtree  once  •  Output  –  Find  the  solu4on    

Input  Stage  

Input  Stage  with  boundaries  

Input  Stage  

•  Quadtree  is  constructed  from  the  input  image  •  Quadtree  will  only  have  non-‐zero  Laplacian  numbers  at  pixels  adjacent  to  a  seam.  

•  Once  quadtree  is  constructed  you  can  get  rid  of  the  image  

Computa4on  Stage  

Computa4on  Stage  

•  Transform  problem  into  a  reduced  space.  S  is  an  m  x  n  transforma4on  matrix.  

•  Our  equa4on  becomes:  

Computa4on  Stage  

•  Can  acquire  informa4on  in  1  quadtree  reversal  

•  Computes  the  following  and  stores  to  save  4me  

Matrix  

•  Solves  uses  Cholesky  factoriza4on  •  Cholesky  is  like  Gaussian  Elimina4on  •  Uses  triangular  matrix  •  Matlab  code  for  Cholesky  

Output  Stage  

•  The  interpolated  solu4on  xd  is  added  to  the  ini4al  composite  x0.  

•  Op4miza4on:  can  be  performed  independently  on  subregion  4les  

Output  Stage  

Why  it  looks  so  good?  

•  The  per-‐pixel  Root  Mean  Square  error  is  in  the  hundredths  

•  Using  a  0-‐255  color  range  makes  a  pixel  at  most  off  by  2  to  the  real  result  

Empirical  Data  

Relevance  of  this  Algorithm  

Relevance  of  this  Algorithm  

 Is  part  of  the  Auto-‐Blend  Layers  and  Photomerge  

features  in  Adobe  PhotoShop  

Limita4ons  

 •  Efficiency  depends  on  the  smoothness  factor  

Ques4ons?  

Efficiency  

–  In  what  cases  will  this  method  not  be  as  efficient  as  predicted,  O(p)  where  p=n.5?  •  Hint:  region  shape  macers.      

– Why  do  we  expect  O(n.5)  for  most  case?  •  Hint:  how  do  we  choose  the  boundaries?  

Efficiency  

–  In  what  cases  will  this  method  not  be  as  efficient  as  predicted,  O(n.5)?  •  Hint:  region  shape  macers.  •  Boundary/perimeter  length  influences  efficiency        

– Why  do  we  expect  O(n.5)  for  most  case?  •  Hint:  how  do  we  choose  the  boundaries?  •  Graph  cut  algorithms,  which  are  used  for  finding  the  seams  between  images,  typically  include  a  cost  term  for  length.  

vs  

Limita4ons  

•  What  other  limita4ons  exist  for  this  method?  

Limita4ons  

•  What  other  limita4ons  exist  for  this  method?  – Same  limita4ons  as  in  regular  Poisson  cloning  techniques:    •  If  colors  or  contrasts  are  extremely  different,  results  may  not  look  good  

 

‘Smoothness’  and  membrane  equa4on  

•  More  about  ‘smoothness’  – Smoothness  criteria  is  somewhat  unclear  from  the  ar4cle    

–  In  this  ar4cle,  ‘smooth’  =  regions  of  the  image  where  Laplacian  =  0.    •  Recall  that  the  solu4on  to  a  problem  like  this  (Laplacian  =  0,  with  boundary  condi4ons)  is  the  membrane  equa4on.  

Rela4onship  to  membrane  equa4on  

•  Recall  membrane  equa4on  is  the  solu4on  to  

 •  Associated  with:  

Source:  hcp://groups.csail.mit.edu/graphics/classes/CompPhoto06/html/lecturenotes/10_Gradient.pdf  

Membrane  equa4on  ßà  Smoothness!  

Source:  hcp://groups.csail.mit.edu/graphics/classes/CompPhoto06/html/lecturenotes/10_Gradient.pdf  

Quadtree  is  an  approxima4on  of  the  solu4on  to  the  membrane  equa4on  

Piecewise  linear  planes  (each  plane  has  a  constant  gradient)  

Source:  hcp://groups.csail.mit.edu/graphics/classes/CompPhoto06/html/lecturenotes/10_Gradient.pdf  

Discussion:  future  work  •  What  are  some  other  possible  applica4ons  of  this  efficient  approxima4on  method  

beyond  gradient-‐domain  composi4ng  of  panoramas?  –     –     –     –     –       

•  What  is  main  limita4on  on  extending  to  other  problems?  –     

Discussion:  future  work  •  What  are  some  other  possible  applica4ons  of  this  efficient  approxima4on  method  

beyond  gradient-‐domain  composi4ng  of  panoramas?  –  Reflec%on  removal  with  flash/no-‐flash  pairs:  –  Shadow  removal  –  Reproduc4on  of  a  photographic  look  –  Collage  instead  of  panorama  –  Video  (octrees)    

•  What  is  main  limita4on  on  extending  to  other  problems?  –  Need  smoothness  condi4ons  to  be  met  somehow.    In  this  paper,  you  need  xo,  ini4al  condi4on,  to  be  

available  that  gives  you  a  transformed  linear  system  that  has  a  sparse  residual.  

Source:  hcp://www.umiacs.umd.edu/~aagrawal/sig05/Gradient_Projec4on.html  

and  your  ideas!  

Discussion:  future  work  •  What  are  some  other  possible  applica4ons  of  this  efficient  approxima4on  method  

beyond  gradient-‐domain  composi4ng  of  panoramas?  –  Reflec4on  removal  with  flash/no-‐flash  pairs  –  Shadow  removal  –  Reproduc4on  of  a  photographic  look  –  Collage  instead  of  panorama:  –  Video  (octrees)    

•  What  is  main  limita4on  on  extending  to  other  problems?  –  Need  smoothness  condi4ons  to  be  met  somehow.    In  this  paper,  you  need  xo,  ini4al  condi4on,  to  be  

available  that  gives  you  a  transformed  linear  system  that  has  a  sparse  residual.  

Source:  hcp://agarwala.org/efficient_gdc/random/simplecomp.png  

and  your  ideas!  

Discussion  Ques4on:  Mixed  Gradients  

To  apply  this  quad-‐tree  method,  what  would  ini4al  condi4on  be?    

Source:  hcp://groups.csail.mit.edu/graphics/classes/CompPhoto06/html/lecturenotes/10_Gradient.pdf  

Discussion  Ques4on:  Mixed  Gradients  

•  In  mixed  gradients,  you  want  the  final  gradient  to  be  a  combina4on  of  the  gradient  of  the  original  and  the  final  –  (Grad(xoriginalImage)  +  Grad(xsourceImage))/2  

•  Therefore,  use    –  Ini4al  condi4on=  (xoriginalImage  +  xsourceImage)/2