Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
(표지)
Master’s Thesis
Real-time Intrinsic Image
Decomposition using Reconstructed
Indoor Scene for Dynamic Relighting
Yoonji Choi
Department of Electrical and Computer Engineering
The Graduate School
Sungkyunkwan University
(내표지)
Real-time Intrinsic Image
Decomposition using Reconstructed
Indoor Scene for Dynamic Relighting
Yoonji Choi
Department of Electrical and Computer Engineering
The Graduate School
Sungkyunkwan University
(심사청구서)
Real-time Intrinsic Image
Decomposition using Reconstructed
Indoor Scene for Dynamic Relighting
Yoonji Choi
A Master's Thesis Submitted to the Department of
Electrical and Computer Engineering
and the Graduate School of Sungkyunkwan University
in partial fulfillment of the requirements
for the degree of Master of Science in Engineering
October 2018
Approved by
Professor Sungkil Lee
(인정서)
This certifies that the master's thesis
of Yoonji Choi is approved.
Committee Chair : Jaepil Heo
Committee Member 1 : Jinkyu Lee
Major Advisor: Sungkil Lee
The Graduate School
Sungkyunkwan University
December 2018
- vii -
Contents
List of Tables ······································ ii
List of Figures ······································ ii
Abstract ······································ 1
1. Introduction ······································ 2
2. Related Work ······································ 6
2.1 Inverse Rendering ······································ 6
2.2 Intrinsic Image Decomposition ······································ 7
3. Algorithm ······································ 10
4. Data Organization ······································ 13
4.1 3D Reconstruction
using multiple view images······································ 13
4.2 Light source Estimation ······································ 13
5. Image Model ······································ 15
5.1 Realtime Performance ······································ 15
5.2 Inaccurate Reconstruction ······································ 16
6. Texture Flatten ······································ 21
6.1 Problem of Approximating
Reflection light color······································ 21
6.3 Texture filtering
to approximate reflection light color······································ 22
7. Inverse Rendering ······································ 23
7.1 Inverse Rendering for Direct lighting ······································ 23
7.2 Inverse rendering for indirect lighting ······································ 24
8. Results ······································ 26
8.1 Performance ······································ 26
8.2 Implementation in Rendered Scene ······································ 27
8.3 Implementation in Real-world Scene ······································ 30
9. Conclusion ······································ 32
10. Limitations and Future Work ······································ 34
References ······································ 36
Korean Abstract ······································ 40
- i -
Table 1. ······················································· 27
Fig.1. ······················································· 12
Fig.2. ······················································· 14
Fig.3. ······················································· 18
Fig.4. ······················································· 20
Fig.5. ······················································· 29
Fig.6. ······················································· 31
List of Tables
List of Figures
- 1 -
Abstract
Real-time Intrinsic Image Decomposition using
Reconstructed Indoor Scene for Dynamic Relighting
We present a first approach real time intrinsic image decomposition of indoor
scene. Improvement of multiple view 3D reconstruction make image-based
modeling available. However, shading included reconstructed texture restrict their
utility. Therefore we present intrinsic image decomposition method to manipulate
lighting using reconstructed model. Our methods aim at removing shading from
the indoor scene reconstructed by commercial multi-view 3D reconstruction
software. We apply inverse rendering to estimate reflectance relative to
reconstructed geometry information. Relighting with image-based modeling using
our method do not lose simplification of lighting. Real-time performance is
obtained by applying Phong reflection model and Light Propagation Volume
(LPV) used for real-time rendering. Ambient value for indirect lighting
approximation is added based on the implementation to avail our method with
the inaccuracy geometry. Our method approximates reflection light color using
bilateral filter and acquires the high quality estimation by removing indirect
lighting. Since we perform light source estimation based on reconstructed model,
do not require special equipment to measure lighting condition.
Keywords: Intrinsic Image Decomposition, Inverse Rendering, Relighting,
Real-time rendering
- 2 -
1. Introduction
3D reconstruction has been done using depth, camera or special hardware.
But recently, the improvement of 3D reconstruction, 3D geometry can be
reconstructed by multiple view image. Therefore, 3D reconstruction using
multiple view images make commercial application software available without
special hardware [1-4]. 3D reconstruction methods have improved accessibility to
image-based modelling [5] and reduce the cost and effort for 3D modeling.
However the utility of reconstructed models is restricted. The cause is the
shading include in texture captured under fixed lighting. Usage of model with
shading information is restricted in programs that often manipulate lighting
condition, such as games and movies. Removing shading from image methods
have been introduced to utilize the reconstructed model in program frequently
change lighting condition. In the case of 3D reconstruction for an object, it is
possible to position the object in a special device that manipulate the light
condition for capture and minimize the effect of shading. However 3D
reconstruction for environment is not possible, since there scale and fixed
lighting fixtures.
Objects in the real world are recognized and captured by the interaction of
reflectance and light. Assuming all surfaces in the scene has Lambertian
surface, pixel color can be defined by the product of shading and reflectance.
Therefore, estimating reflectance of object by removing shading from images is
- 3 -
possible. Intrinsic image decomposition aim to separate material
properties-reflectance component and shading component, and facilitates object
recognition, image editing and lighting condition editing.
Intrinsic image decomposition is an ill-posed problem since we should
estimate two unknowns based on given single image [6-9]. To solve ambiguity,
several approaches require additional information. They require multiple view
images taken under fixed lighting conditions [10-13], images taken under various
lighting condition[14], and images with depth information [14-15].
Intrinsic image decomposition for entire space is studied only a small
number, and which is also limited to outdoor [10-11]. In outdoor case, it is
assumed that sun only exists the lighting condition is changed with the
movement of the sun. Due to the assumption, the feature of light source
unchanged and, the ambiguity of the intrinsic image decomposition. However, in
indoor case, The lighting fixtures are fixed and it is difficult that changing the
lighting condition by manipulating the light.
Most Intrinsic image decomposition methods take long computation time.
Recently, real-time methods [17-19] presented, however, it is still a small
number and do not consider geometry information. And most estimations are
performed on a single image or object. A few methods presented about entire
scene, which is limited to the outdoor [10-11] and do not provide realtime
performance.
- 4 -
We introduce a first attempt to remove shading from the texture of the
reconstructed indoor model in real-time. We use multiple-view images taken
fixed lighting condition, and perform automatic 3D reconstruction using
commercial software. Our method aims at unconstrained lighting control using
reconstructed models of multi-view 3D reconstruction. Intrinsic image
decomposition methods for image editing are not suitable for relighting based on
image-based modeling, since they perform decomposition regardless of geometry
information. Our method estimates reflectance relative to degree of geometry
reconstruction for relighting and keeps the benefit of image-based rendering,
simplification of lighting. We do this by using light source estimation and
inverse rendering. Since we perform light source estimation based on
reconstructed model, we do not require special hardware to approximate lighting
condition. Estimated light source is regarded as point light to handle the indoor
scene that has various lighting fixtures that is hard to define the feature.
We achieve real-time performance by applying real-time rendering method for
inverse rendering and dividing lighting as direct lighting and indirect lighting.
Phong reflection model [20] is used for direct lighting and Light Propagation
Volume (LPV) [21] is used for indirect lighting. All of the estimated light source
is regarded as light source for direct lighting, However threshold is used for
indirect lighting to avoid useless computation. And considering the imperfect
reconstruction using commercial software without special hardware, ambient
value that adjust brightness is applied. Indirect lighting require the reflection
light color that present the color of surface where the interaction occur. We
perform texture flatten using bilateral filter to avoid iteration for estimating
- 5 -
reflection light color.
- 6 -
2. Related Work
Intrinsic image decomposition we proposed is related to the inverse rendering
that estimate the unknown information using rendering equation and intrinsic
image decomposition for image editing. Our method aim to enable manipulating
the lighting condition using image based model without high cost light
computation due to the reality from photographs, so our method more similar
with the inverse rendering that use geometry information. Since the intrinsic
image decomposition for image editing do not consider the rendering, and aim to
perfect separation of reflectance from images to edit the colors and recognize
objects.
2.1 Inverse Rendering
Inverse rendering approaches render backward from photographs to scene
components such as geometry, reflectance and lighting to estimate unknown
data. Therefore, reflectance can be estimated when the geometry and lighting
information is exist. Our method perform intrinsic image decomposition related
to the degree of geometry reconstruction. Therefore, inverse rendering, estimate
the reflectance based on geometry and lighting is close to our method.
Estimating reflectance using inverse rendering requires geometry and lighting
information. Improvement of 3D reconstruction make estimating geometry
- 7 -
information possible. Depth camera[18-19], IR sensor [22], multiple images or
other special hardware used to reconstruct geometry. Lighting is measured
through the environment map taken with reflective sphere [10-11]. Some
approaches manipulate lighting conditions during taking pictures[13-14].
Real-time intrinsic image methods using inverse rendering are introduced
based on Shape from Shading (SfS) [23] that recovers geometry and reflectance
using shading information. SfS is estimate the shading using depth information,
and refine the shading information by iteration. Real-time methods based on SfS
[18-19] use the video stream as input data and refer previous frame information
to enhance accuracy. They are only limited to object not full scene and require
depth camera.
Some approaches manipulate the light and take pictures from different
lighting condition [13-14]. However this approaches is difficult to utilize to entire
scene. In outdoor case, taking picture under the various lighting condition by the
time passes is possible, however that is very time consuming and can not be
adopted for indoor scene.
Intrinsic image decomposition methods for space [10-11] are limited to
outdoor scene. They assume sun is the only light source and other lightings
come from sky light, and use reflective sphere to measure the lighting condition.
And they take long computation time.
- 8 -
2.2 Intrinsic Image Decomposition
Many methods for image editing were presented to decompose images into
reflectance and shading without using geometry reconstruction. Though lots of
approaches are proposed, intrinsic image decomposition for image editing can not
be adopted to image based model. Since they can not preserve the simplification
of lighting, the benefit of image based modeling from photographs.
The Retinex algorithm classifies edges of grayscale image into shading or
reflectance based on the assumption that shading has gradual and reflectance
has sharp variation. Many approaches based on this assumption [24-25] were
presented, but Retinex based algorithm occurs error where shading dramatically
change like hard shadow. Methods using Assumption that neighboring pixels
with similar chromaticity have the same reflectance is presented [8, 26]. Other
approaches such as clustering [27] and additional optimization [9] to improve the
accuracy were attempted.
Despite of many approach, intrinsic image decomposition remains ill-pose
problem. To solve this ambiguous, many methods use various additional
information such as images taken under different light condition [12-13], depth
information [14-16], user assisted [9] and video stream [17].
Intrinsic image decomposition is almost require long computation time, since
they adopt iteration to improve the accuracy. Generally, as the number of
- 9 -
iteration is increase, the accuracy of the intrinsic image decomposition is
increases, however the performance decrease.
Real-time intrinsic image decomposition for image editing method [17] was
proposed based on the video stream. This method also need iteration to improve
the quality, even can obtained real-time solution using GPU optimization. And
this approaches use the feature of video stream, by using the information that
the previous frames have, which complement the lack of informations and some
case computation can be replaced by using result of the previous frames.
However, most intrinsic image decomposition methods still require long
computation time.
- 10 -
3. Algorithm
In this paper, we present a first approach to real time intrinsic image
decomposition of indoor scene. For our purpose, we obtain geometry and texture
by 3D reconstruction using multiple view images. Images are taken from
different viewpoints under fixed lighting condition. In this process, the model
includes light fixtures should be reconstructed. Since reflectance is estimated by
dividing the input texture by shading, we approximate shading using light
source information and reconstructed geometry. The light source is estimated
based on reconstructed model and we can obtain the color and position
information.
We divide the lighting into direct lighting and indirect lighting to obtain high
performance. Direct light can be calculate directly using geometry information
and result of light source estimation. However, indirect lighting is occurred by
interaction of light and object’s multiple reflection and lighting color is affected
by reflection point. Therefore, we should consider reflectance of reflection point.
However, our algorithm aim to estimate the reflectance of object. The reflection
light color can not be obtain directly. Generally, indirect lighting require iteration
to calculate the effect, which drop down the performance. Therefore, it is not
suitable for real time rendering. We apply the filtering on the reconstructed
model texture and approximate the object’s reflectance and adopt real time
rendering method that do not perform iteration.
- 11 -
The result of direct lighting and indirect lighting is combined and we
estimate the reflectance by dividing texture into lighting estimation result. The
method we present is aim to diffuse objects and, do not consider the specular
reflectance. Therefore, we assume all materials in the scene have Lambertian
surface and do not consider shadows. Fig. 1 shows overview of our algorithm.
- 12 -
Fig. 1 Overview of
the intrinsic image process proposed
- 13 -
4. Data Organization
4.1 3D Reconstruction using multiple view images
We aim to intrinsic image decomposition about indoor space 3D model that is
reconstructed without special device such as depth camera. We use multiple
view images and reconstruct the entire indoor scene using commercial software.
Our method can operate for 3D reconstruction indoor space model and the light
fixture emit lighting must be reconstructed.
3D reconstruction using multiple view image can eliminate some specular
effect due to the process combine the image process. Even though out method
do not consider the specular, due to the 3D reconstruction using multiple view
image process, some specular effect is removed. And the reconstructed model is
more suitable to operate our method.
4.2 Light source Estimation
Our method even do not require the hardware estimate the lighting condition
such as the sphere ball to capture environmental map. And unlike the outdoor,
lighting condition is changed according by time passes, lighting fixtures in
indoor scene is fixed and hard to manipulate the lighting. Therefore, We adopt
- 14 -
light source estimation based on 3D model texture. Our method target the entire
indoor model, so the light fixture that emit the light can be reconstruct when
we perform 3D reconstruction. Our method is available when the model
reconstructed entire scene include the light fixtures. The position information in
the 3D model estimated light source can be obtained using texture coordinate
and vertex information.
Indoor environment have various light fixtures with different characters, we
cannot define the feature of light source. So we assume the light source
estimation result as numerous point light. As the result, We can obtain point
light’s color and position information. To enhance accuracy, before the light
source estimation LDR (Low Dynamic Rage) texture is converted to HDR (High
Dynamici Rage) texture. Figure 2. represent the result of light source estimation
and 3D reconstructed model using multiple view images. the red dot represent
the estimated point light.
Fig 2. The result of 3D
reconstruction and light source
estimation. the red dot represent the
estimated light source.
- 15 -
5. Image Model
We perform shading estimation using reconstructed model and estimated light
source to estimation reflectance. Pixel color is defined as the multiple of
reflectance and shading, result of lighting. We estimate just light source
however, the shading is based on multiple interaction of lighting and the objects.
To make the problem simply, we separate the lighting as direct lighting and
indirect lighting like other real time rendering approach. Direct lighting is
calculated using the light source and geometry information. However, Indirect
lighting is more complex problem. We should consider the reflection occur the
surface of the objects, which means calculating indirect lighting require the
reflectance information. The reflectance is the goal of intrinsic image
decomposition. Therefore, to estimate the accurate reflection light color
computing lighting and estimating the reflectance, which take long computation
time, is performed repeatedly more than just calculating the indirect lighting. We
focus on the performance than the quality and adopt real time rendering method
that consider one bounce indirect lighting and propagate the one bounced result
and use filtered texture.
5.1 Realtime Performance
To achieve real time performance, we divide the lighting into direct lighting
- 16 -
and indirect lighting and adopt the realtime rendering method. Phong reflection
model is used to compute direct lighting. For indirect lighting, LPV do not
require multiple iteration is adopted. Following is our image formation model.
∈
cos ∈
∈
∙ ∈
cos∈
∈
∙ (1)
is the pixel at given image and is the reflectance of the object. Where
and are direct and indirect lighting, is the incidence angle. is
the direction vector from the point on the surface toward each light source, and
is the normal at this point on the surface. represents the attenuation of
light according to distance .
5.2 Inaccurate Reconstruction
The quality of multiple view images reconstruction without special hardware
is worse compare to using special hardware such as depth camera. Therefore,
- 17 -
some geometry informations are not reconstructed. Inaccuracies and missing data
of Reconstruct geometry make indirect lighting approximation inaccurate.
Inaccurate missing data make the lighting darker, which means reflectance is
estimated brighter. Fig. 3 shows the bad intrinsic image decomposition result
because of the missing data.
We use ambient value used in Phong reflection model to overcome the
limitation of reconstruction. Ambient value is applied to approxiate the indirect
lighting is applied with LPV in the image model. Below shows the modified
image model.
∈
∙ (2)
We perform an implementation to estimate the usability of ambient value and
the implementation confirmed that indirect lighting approximation using LPV
with ambient shows better quality. Fig. 2 illustrates our implementation result
involve the missing data from floor make the inaccurate reflectance estimation
result.
The ambient value is can be used to manipulate the brightness of entire
lighting. However, using the high ambient value, the lighting effect is too bright,
make the reflectance darker and estimation result shows worse quality.
- 18 -
Fig. 3 Results of intrinsic image decomposition and shading approximation
using LPV (left) and using LPV with ambient (Right)
to approximate indirect lighting.
Ambient value just manipulate the brightness of lighting, therefore, cannot
represent the effect of reflection light. Ambient value cannot replace the LPV
represent the indirect lighting involve the reflection lighting color and effect. We
perform an implementation to compare the estimation result of image model
using just ambient value and using ambient value and LPV together.
Fig. 3 shows the results of the reflectance estimation based on indirect
lighting approximation using ambient and LPV with ambient. Shading
- 19 -
approximation using ambient cannot represent reflection lighting. However,
shading Approximation based on LPV with ambient approximate reflection light
with color and removes shading from the drawer. Fig.4. shows the result of
implementation using ambient and LPV. Shading near by the drawer shows
yellow color and the intrinsic result of wall paper is more flatten due to
eliminating the shading color from texture.
- 20 -
Fig. 4 Results of intrinsic image decomposition using just ambient (left)
and using LPV and ambient (right) together to approximate indirect lighting.
- 21 -
6. Texture Flatten
Indirect lighting is the multiple interaction of light that reflected the objects
surface and other objects surface. Reflection light require the reflectance of
objects to represent the light color. However, the purpose of intrinsic image
decomposition is estimating the reflectance, represent the indirect lighting include
the reflectied light color is complex.
6.1 Problem of Approximating Reflection light color
Some approach[] do not consider the color of reflection light or redefine the
reflection light using iteration. estimating and redefining the reflectance approach
is used for inverse rendering and intrinsic image decomposition for image
editing both. In the first iteration, shading include the reflectance is eliminate
form texture, according the number of iteration shading value is redefined and
the genuine shading can be get rid of the texture. Increasing the number of
iteration usually makes the quality better however, makes the performance slow.
Even the computing the indirect lighting without estimate the reflectance take
almost computation time, therefore adopting iteration at inverse rendering cause
the serious performance degradation. In this paper, we apply texture filtering to
estimate the reflection light color to prevent the performance degradation.
- 22 -
6.2 Texture filtering to approximate reflection light color
Texture flatten is inspired by Retinex algorithm. Retinex algorithm studied
for intrinsic image decomposition for image editing is based on the assumption
that the changing of reflectance has sharp variation but the changing of shading
occurred by lighting is gradual. Inspired the assumption, we adopt Bilateral
Filter[] to estimate the reflection light color. Bilateral Filter is a de-noise
method, which preserve the edge of image. The changing of reflectance shows
the sharp variation can be considered as the edge, and the changing of shading
is blurred without preserve due to their smooth changing. Retinex based
algorithm do not work properly where the shading change dramatically such as
hard shadow. It is not problematic since, we do not consider the shadow for
estimating reflectance.
Texture flatten is applied to reconstructed texture, before computing indirect
lighting. Pixel color and geometry information is used to compute bilateral filter.
Bilateral filtering should load more than one element to preserve the edge,
therefore performance degradation arise. We make the operation computed by
GPU to improve the performance. Since our purpose is estimating the reflectance
based on geometry information, flatten texture is just used to estimate the
reflection light color.
- 23 -
7. Inverse Rendering
Inverse rendering is a method estimate an unknown data based on image
formation model and known data. In our case, reflectance is unknown data and
geometry is known by 3D reconstruction. And lighting can be estimated using
3D reconstruction, light source information, texture and image formation model.
The method we proposed use automatic 3D reconstruction software using
multiple view images taken under fixed lighting condition. If the geometry and
lighting is obtained, reflectance can be estimated dividing into shading.
7.1 Inverse Rendering for Direct lighting
We estimate the position and color information of light source by light source
estimation [29] that estimate area light as point light. Each lighting fixture has
their own feature, and we cannot define that, therefore we assume the estimated
light as point light. Light source estimation result a large number of point, since
the lighting fixtures form real-world have area. All of the estimated point lights
are used to compute the direct light. Light attenuation real world light have can
not be estimated form the light source estimation. Therefore, we should
manipulate the attenuate value by heuristic.
- 24 -
7.2 Inverse rendering for indirect lighting
Flatten texture and estimated light source is used to compute indirect
lighting. We adopt LPV at our image model and prevent performance
degradation by iteration. LPV generate reflectance shadow maps (RSM) [30]
approximate the reflection light color and position. RSM project the scene from
light position and assume the surface projected that light camera as reflection
point where the first reflection light is occurred by lighting fixture. RSM is
generated from light fixture, since LPV only consider the first reflection and
propagate the effect to adjacent space. At that time reflection light color is
estimated using texture correspond to reflected point.
As a result of light source estimation [29], area lights are estimated a large
number of point lights and RSMs are generated for each point light to compute
LPV. Computing direct lighting using dense point light can improve the shading
result, however in case of indirect light computing using LPV is different. RSMs
for approximating reflection light generated by adjacent point light have the
same information about reflection light, since the projection result is similar.
Thus using all of them does not affect the quality of lighting approximation and
just takes long computation time. Therefore, we adopt threshold to prevent
generate RSM of adjacent point lights. Below demonstrate the complete inverse
rendering equation. ′ is light source used to generate RSM.
∈
∙ ∈′
(3)
- 25 -
The final shading is computed by combining indirect lighting and direct
lighting. Reflectance is obtained by dividing texture into shading result. The
result does not show complete separation of reflectance, because the lighting is
computed using the geometry information. The genuine texture, texture filter is
not applied, is used for dividing to preserve the small detail the advantage of
image based modeling.
- 26 -
8. Results
We implemented our method in a rendered scene and a real-world scene.
The rendered scene is designed to only occur diffuse reflection to and Mitsuba
renderer[31] is used to make bedroom scene. We used 70~80 multiple view
images to reconstruct the rendered scene. The real world data, dressing room
scene, is reconstructed using 70~80 images captured by general camera of
IPhone 6S. Images taken under fixed lighting condition, and the indoor scene
model that includes lighting fixtures is reconstructed. We used Autodestk Recap
(http://recap360.autodesk.com), commercial software, for automatic multiple -view
3D reconstruction and acquired proxy geometry and an atlas texture.
8.1 Performance
We performed our method on Intel(R) Core(TM) i7-5960X CPU, GeForce
GTX 1080 Ti GPU. The rendered scene (Fig. 5) took computation time for
bilateral filter computation time 6.31ms, inverse rendering 29.75ms and total
36.06ms. The real-world data (Fig. 6) took bilateral filter computation time
6.52ms inverse rendering 29.36 and total computation time 34.91ms. We
confirmed that our method shows real-time computation time based on
implementation.
- 27 -
Table 1. The implementations computation
time(ms) for bilateral filter and inverse
rendering.
8.2 Implementation in Rendered Scene
We implement our method in the rendered scene that designed all objects
have Lambertian reflectance. Fig. 5 shows reconstruction result, shading
approximation and reflectance estimation.
We can verify the quality of our method in the area where the variance of
shading is large (yellow box). In case of the wall close to drawer (redbox), the
shading containing the color of indirect lighting is removed from the wallpaper.
As the result, we can ensure that indirect lighting approximation using bilateral
filter improves the quality of estimation. However, the inaccuracies of
- 28 -
reconstruction and lighting approximation without considering the characteristics
of light sources make inaccuracies of estimation at some area. Bed (blue box)
shows the inaccuracy of estimation arose from wrong reconstruction. Ceiling and
wall estimated different color despite they have same reflectance (green box).
The cause is wrong shading approximation without considering the features of
light.
- 29 -
Fig. 5 Results of intrinsic image decomposition using virtual
environment designed to cause diffuse reflection only. From left,
3D reconstruction result, shading approximation and intrinsic
image results are shown.
- 30 -
8.3 Implementation in Real-world Scene
We performed intrinsic image decomposition using real world data and
demonstrate our method can be applied in real-world.
Fig. 6 shows reconstruction result, shading approximation and reflectance
estimation. We can see the quality of our method in the area where the
variance of shading is large (red box). Removing Shading by indirect lighting
operated in dressing table (yellow box) improved quality of estimation. However,
similar to Fig. 4, inaccurate estimation is performed by the inaccuracies of
geometry, which is shown from switch (blue box). Unlike our assumption, the
real world has various reflective properties, which led inaccurate decomposition.
We can see that specular reflection from the dressing table (green box).
- 31 -
Fig. 6 Results of intrinsic image decomposition using real world
data. From left, 3D reconstruction result, shading approximation
and intrinsic image results are shown
- 32 -
9. Conclusion
We proposed an intrinsic image decomposition method that enables to
manipulate lighting condition using reconstructed model by removing shading.
Our method performed reflectance estimation relative to geometry, which did not
lose simplification of lighting, benefit of image-base rendering.
Our method aim to reconstructed indoor space model using commercial soft
ware without special hardware. we should consider the indoor space's feature
that hard to manipulate the light condition to estimate the lighting condition.
Therefore, we adopt light source estimation based on reconstructed model
texture without hardware such as sphere ball to capture environment map. And
we can estimate the color of light source since we use the texture for
estimation. Our method is easily utilized reconstructed model by multi view
images taken fixed lighting condition for relighting. All estimated light source is
regarded as point light to mimic various lighting fixtures characters.
We achieve real time performance using real time rendering techniques and
dividing lighting as direct lighting and indirect lighting for inverse rendering.
Phong reflectance model and LPV is adopted to image model, and we added
ambient in our image formation model based on the implementation, and obtain
better quality of estimating reflectance using inaccurate geometry using
automatic 3D reconstruction. All of the point light is used for computing direct
lighting and computing indirect light use portion of point light to void
- 33 -
computation about adjacent light source. Bilateral filter is applied to texture to
approximate reflection light color and avoid iteration.
We implemented our method at rendered scene only diffuse reflection occur
and real word scene captured using commercial camera and confirmed
reflectance was estimated considering the geometry reconstruction. And even
though we consider diffuse reflection only thanks to 3D reconstruction using
multiple view images, some specular effect is remove.
- 34 -
10. Limitations and Future Work
Indirect lighting approximation generated unnecessary RSMs and overhead
caused, therefore, user should intervene to adjust the threshold. It can be solved
by merging adjacent point lighting using light position information and make
one RSMs for a light fixture.
we adopted texture flatten to estimate reflection light color. It is meaningful
that can eliminate the effect of indirect involve reflection light color. However
model texture has shading, which makes reflection light color brighter or darker.
It can be overcome some iteration. However the method we proposed choose
performance rather than quality for real-time estimation.
Our estimation assumed all objects has diffuse surface, therefore the specular
reflectance can not be estimated. We could overcome the limitation using
multiple view reconstruction. Since the 3D reconstruction replace the missing
data caused specular effect using multiple view images. And we did not
consider the shadow since, the shadow does not have information to estimate
the reflectance where shadow is drawn. It can be overcome by using the
adjacent pixel to fill the spot where the shadow exist.
We can not measure the quality of method because of the lack of reference.
In real world, we can not extract the reflectance of object, therefore, some
- 35 -
reference to measure the quality of intrinsic image decomposition is presented.
However the references are limited to intrinsic image decomposition for image
editing that do not consider the geometry information, which is not available to
our method.
Estimation inaccuracies caused by result of light estimation that regard lights
as point lights. This limitation can be resolved by using use-assistance input
about light characteristics. Our method was not work automatically since the
LPV, light power and attenuation for lighting require adjustment by heuristic.
The shading computation can be replace by other method such as path tracing
or virtual point lighting that guarantee better quality, however require long
computation time. According the intrinsic image decomposition purpose, the
method can be changed.
- 36 -
References
[1] Snavely, Noah, Steven M. Seitz, and Richard Szeliski. "Photo tourism:
exploring photo collections in 3D." ACM transactions on graphics (TOG). Vol.
25, No. 3, pp. 835-846, ACM, 2006.
[2] Goesele, Michael, et al. "Multi-view stereo for community photo collections."
IEEE 11th International Conference, pp. 1-8. 2007
[3] Fuhrmann, Simon, Fabian Langguth, and Michael Goesele. "MVE-A
Multi-View Reconstruction Environment." GCH, pp. 11-18, Oct. 2014.
[4] Jancosek, Michal, and Tomás Pajdla. "Multi-view reconstruction preserving
weakly-supported surfaces." 2011 IEEE Conference, pp. 3121-3128. June 2011.
[5] Debevec, Paul E., Camillo J. Taylor, and Jitendra Malik. "Modeling and
rendering architecture from photographs: A hybrid geometry-and image-based
approach." Proceedings of the 23rd annual conference on Computer graphics and
interactive techniques, pp. 11-20, 1996.
[6] Bonneel, Nicolas, et al. "Intrinsic decompositions for image editing."
Computer Graphics Forum, Vol. 36. No. 2. May 2017.
[7] Tappen, Marshall F., William T. Freeman, and Edward H. Adelson.
"Recovering intrinsic images from a single image." Advances in neural
information processing systems, pp. 1367-1374, 2003.
[8] Shen, Li, Ping Tan, and Stephen Lin. "Intrinsic image decomposition with
non-local texture cues." IEEE Conference, pp.1-7, 2008.
[9] Shen, Jianbing, et al. "Intrinsic image decomposition using optimization and
user scribbles." IEEE transactions on cybernetics, Vol. 43, No. 2, pp.425-436,
- 37 -
2013.
[10] Duchêne, Sylvain, et al."Multi-view intrinsic images of outdoors scenes with
an application to relighting." ACM Transactions on Graphics, Vol. 34, No. 5,
pp164:1-16, Oct. 2015.
[11] Laffont, Pierre-Yves, Adrien Bousseau, and George Drettakis. "Rich intrinsic
image decomposition of outdoor scenes from multiple views." IEEE transactions
on visualization and computer graphics, Vol. 19, No. 2, pp. 210-224, Feb. 2013.
[12] Weiss, Yair. "Deriving intrinsic images from image sequences." Computer
Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International Conference, pp.
68-75, 2001.
[13] Matsushita, Yasuyuki, et al. "Estimating intrinsic images from image
sequences with biased illumination." European Conference on Computer Vision,
pp.274-286, 2004.
[14] Hudon, Matis, et al. "Shape and Reflectance from RGB-D Images using
Time Sequential Illumination." VISIGRAPP (3: VISAPP), pp.534-543, 2016.
[15] Hachama, Mohammed, Bernard Ghanem, and Peter Wonka. "Intrinsic scene
decomposition from rgb-d images." Proceedings of the IEEE International
Conference on Computer Vision, pp. 810-818, 2015.
[16] Chen, Qifeng, and Vladlen Koltun. "A simple model for intrinsic image
decomposition with depth cues." Computer Vision (ICCV), 2013 IEEE
International Conference, pp. 241-248, 2013.
[17] Meka, Abhimitra, et al. "Live intrinsic video." ACM Transactions on
Graphics (TOG), Vol. 35, No. 4, pp. 109:1-14, July 2016
[18] Wu, Chenglei, et al. "Real-time shading-based refinement for consumer
depth cameras." ACM Transactions on Graphics (TOG), Vol. 33, No. 6,
- 38 -
pp.200:1-10, Nov. 2014.
[19] Guo, Kaiwen, et al. "Real-time geometry, albedo, and motion reconstruction
using a single rgb-d camera." ACM Transactions on Graphics (TOG), Vol. 36,
No. 3, pp. 32:1-13, Jun. 2017.
[20] Phong, Bui Tuong. "Illumination for computer generated pictures."
Communications of the ACM, Vol. 18, No. 6, pp. 311-317, Jun. 1975.
[21] Kaplanyan, Anton, and Carsten Dachsbacher. "Cascaded light propagation
volumes for real-time indirect illumination." Proceedings of the 2010 ACM
SIGGRAPH symposium on Interactive 3D Graphics and Games, pp. 99-107,
2010.
[22] Choe, Gyeongmin, et al. "Refining geometry from depth sensors using IR
shading images." International Journal of Computer Vision, Vol. 122, No. 1, pp.
1-16, Mar. 2017.
[23] Horn, Berthold KP, and Michael J. Brooks. "The variational approach to
shape from shading." Computer Vision, Graphics, and Image Processing, Vol.
133, No. 2, pp. 174-208, Feb. 1986.
[24] Land, Edwin H., and John J. McCann. "Lightness and retinex theory."
Journal of the Optical Society of America, Vol. 61, No. 1, pp. 1-11, 1971.
[25] Horn, Berthold KP. "Determining lightness from an image." Computer
graphics and image processing, Vol. 3, No. 4, pp. 277-299, Dec. 1974
[26] Shen, Hui-Liang, et al. "Chromaticity-based separation of reflection
components in a single image." Pattern Recognition, Vol. 41, No. 8, pp.
2461-2469, Aug. 2008.
[27] Garces, Elena, et al. "Intrinsic images by clustering." Computer graphics
forum, Vol. 31, No. 4, pp. 1415-1424, Jun. 2012..
- 39 -
[28] Tomasi, Carlo, and Roberto Manduchi. "Bilateral filtering for gray and color
images." Sixth International Conference, pp. 839-846, 1998.
[29] Debevec, Paul. "A median cut algorithm for light probe sampling." ACM
Siggraph 2005 Posters, pp. 66, 2005.
[30] Dachsbacher, Carsten, and Marc Stamminger. "Reflective shadow maps."
Proceedings of the 2005 symposium on Interactive 3D graphics and games, pp.
203-231, Apr. 2005.
[31] W. Jakob. (2010). Mitsuba renderer [Online]. Available:
http://www.mitsuba-renderer.org .(downloaded 2018, Jan. 17)
- 40 -
논문요약
실내 환경 복원 모델을 이용한
동적 조명 적용을 위한 실시간 재질추정 기법
성균관대학교
전자전기컴퓨터공학과
최윤지
3D 기하 복원 기술의 발전으로 다시점 이미지와 상용 프로그램을 이용한 영상
기반 모델링의 접근이 용이해졌다. 하지만 복원된 모델의 텍스처에 포함된 음영으
로 영상 기반 모델의 사용은 한정되어 있다. 따라서 우리는 복원된 모델을 이용한
동적인 조명 조작을 위한 재질추정 기법을 제안한다. 본 논문이 제안하는 기법은
실내 환경을 복원한 영상기반 모델을 목표로 하며, 영상 기반 모델은 다시점 이미
지를 이용한 상용 프로그램으로 제작된다. 우리는 기하의 복원 정도에 비례한 재질
추정을 위하여 역렌더링 기법을 적용하여 본 논문이 제시하는 재질 추정을 수행한
모델은 영상 기반 모델의 장점인 조명 적용의 간략화를 잃지 않는다. 실시간 성능
은 실시간 렌더링 기법인 퐁 반사 모델과 광전파볼륨을 적용하여 달성하였으며, 사
용 프로그램을 이용해 복원한 기하의 불완전성을 고려하여 앰비언트 값이 간접조명
근사에 포함되었다. 간접조명 근사 시, 반사광 색상 근사를 위하여 양방향 필터가
적용되었으며, 간접 조명의 색상을 포함한 음영제거로 보다 높은 품질의 재질추정
을 수행할 수 있었다. 본 논문이 제안하는 기법은 텍스처 기반의 광원추정으로 조
명 측정을 위한 특수 장치를 요구하지 않는다.
- 41 -
주제어: 재질추정, 역렌더링, 재조명, 실시간 렌더링
(표지측면)