Socialized Mobile Photography: Learning to Photograph With Social Context via Mobile Devices

  • View
    218

  • Download
    5

Embed Size (px)

Transcript

  • 184 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 16, NO. 1, JANUARY 2014

    Socialized Mobile Photography: Learning toPhotograph With Social Context via Mobile DevicesWenyuan Yin, Student Member, IEEE, Tao Mei, Senior Member, IEEE, Chang Wen Chen, Fellow, IEEE, and

    Shipeng Li, Fellow, IEEE

    AbstractThe popularity of mobile devices equipped with var-ious cameras has revolutionized modern photography. People areable to take photos and share their experiences anytime and any-where. However, taking a high quality photograph via mobile de-vice remains a challenge for mobile users. In this paper we investi-gate a photography model to assist mobile users in capturing highquality photos by using both the rich context available frommobiledevices and crowdsourced social media on the Web. The photog-raphymodel is learned from community-contributed images on theWeb, and dependent on users social context. The context includesusers current geo-location, time (i.e., time of the day), and weather(e.g., clear, cloudy, foggy, etc.). Given a wide view of scene, our so-cialized mobile photography system is able to suggest the optimalview enclosure (composition) and appropriate camera parameters(aperture, ISO, and exposure time). Extensive experiments havebeen performed for eight well-known hot spot landmark locationswhere sufficient context tagged photos can be obtained. Throughboth objective and subjective evaluations, we show that the pro-posed socialized mobile photography system can indeed effectivelysuggest proper composition and camera parameters to help theuser capture high quality photos.

    Index TermsCamera parameters, mobile photography, socialcontext, social media, view enclosure.

    I. INTRODUCTION

    T HE recent popularity of mobile devices and the rapiddevelopment of wireless network technologies haverevolutionized the way people take and share multimediacontent. With the pervasiveness of mobile devices, more andmore people are taking photos to share their experiences usingtheir mobile devices anytime and anywhere. Market researchindicates that more than 27% of photos were captured bysmartphones in 2011, while the number was merely 17% in theprevious year [1]. The booming development of built-in mobilecameras (such as the advanced eight megapixel resolution andthe large aperture) has triggered a trend that may lead tomobile cameras replacing traditional handheld cameras.

    Manuscript received October 09, 2012; revised February 08, 2013 and May18, 2013; accepted June 27, 2013. Date of publication September 25, 2013;date of current version December 12, 2013. This work was supported by NSFGrant 0964797 and a Gift Funding fromKodak. Part of this work was performedwhen the first author visited Microsoft Research Asia as a research intern. Theassociate editor coordinating the review of this manuscript and approving it forpublication was Dr. Vasileios Mezaris.W. Yin and C. W. Chen are with the State University of New York at Buffalo,

    Buffalo, NY 14260 USA (e-mail: chencw@buffalo.edu).T. Mei and S. Li are with Microsoft Research Asia, Beijing 100080, China

    (e-mail: tmei@microsoft.com).Color versions of one or more of the figures in this paper are available online

    at http://ieeexplore.ieee.org.Digital Object Identifier 10.1109/TMM.2013.2283468

    However, mobile cameras cannot guarantee perfect photos.Although mobile cameras harness a variety of technologiesto take care of many camera settings (e.g., auto-exposure) forpoint-and-shoot ease, capturing high quality photos is still achallenging task for amateur mobile users, not to mention thoselacking photography knowledge and experience. Therefore,assisting amateur users to capture high quality photos viatheir mobile devices becomes a demanding task. While mostexisting research has predominantly focused on how to retrieveand manage photos on mobile devices [2], [3], or how to adaptmedia considering unique characteristics of mobile devices[39], there have been few attempts to address this topic before.To obtain high quality photos, various types of commercial

    software such as Photoshop have been developed for post-pro-cessing to adjust photo quality. However, most of them are de-signed for desktop PC, which require intensive computation. Al-though there exist some mobile applications for image post-pro-cessing, they are only able to conduct simple operations, such ascropping and contrast adjustment. These post-processing toolscannot always fix poorly captured images. For example, the in-formation lost in an over-exposed image is usually unrecover-able by any post-processing technique. Therefore, it is desirableto assist mobile users in obtaining high quality photos while thephotos are being taken. For example, if we can suggest the op-timal scene composition and suitable camera settings (i.e., aper-ture, ISO, and exposure time) based on the users current con-text (i.e., geo-location, time, and weather condition) and inputscene, the users ability to capture high quality photos via mo-bile devices will be improved significantly.On the other hand, computational aesthetics of photography

    have emerged as a hot research area. It aims to automatically as-sess or enhance image quality with computational models basedon various visual features, such as lighting (e.g., light, color,and texture) and composition (e.g., the rule of thirds). A com-prehensive survey on computational aesthetics can be found in[5]. However, aesthetic assessment or enhancement methods inexisting works are also designed for desktop PCs, and focus onevaluating or enhancing image quality as post-processing tools.Moreover, they only apply some simple and general photog-raphy rules to the aesthetics evaluations. For example, whenassessing the photo composition, they tend to place objects ac-cording to the rule of thirds or put the horizontal line lower in theframe, according to the golden ratio [6]. In many cases, insteadof applying these simple rules slavishly, professional photogra-phers usually adapt the composition and the camera exposureparameters for photographing, e.g., the scene and the lightingconditions. Therefore, the existing general photo aesthetics as-sessment and enhancement approaches are far from enough toguide mobile user to capture high quality photos on the fly.

    1520-9210 2013 IEEE

  • YIN et al.: SOCIALIZED MOBILE PHOTOGRAPHY: LEARNING TO PHOTOGRAPH WITH SOCIAL CONTEXT VIA MOBILE DEVICES 185

    Based on the above analysis, it is desirable to develop anintelligent built-in tool to assist mobile users in capturing highquality photos. Since composition and exposure parameters(i.e., aperture, ISO, and exposure time) are two key factorsaffecting photo quality [7], the system should be able to pro-vide suggestions to mobile users in terms of these two aspects.Therefore, we propose an intelligent mobile photographysystem, which can not only assist mobile user to find good viewenclosures, but also help them set correct exposure parameters.

    A. Challenges and Opportunities

    Despite recent development on computational aesthetics, in-telligent mobile photography still remains a challenge becauseno unified knowledge can always be applied to various scenesand contexts. Various composition rules and exposure prin-ciples apply adaptively and flexibly when capturing differentobjects or scenes with diverse perspectives and arrangementsunder various lighting conditions. Professional photographerstake years of training to obtain sufficient photography knowl-edge. They usually skillfully adopt the domain knowledge tocapture scene under different conditions. Photographers areartists whose knowledge is difficult to model and representwith simple rules. The descriptions on the complexity of photocomposition and camera settings are given in Section III.Therefore, assisting mobile users to take professional photosunder various shooting situations is a challenging task.Fortunately, the availability of rich context sensors on mo-

    bile devices and the explosion of images associated with theircapture contexts, camera parameters and social information onsocial media community bring us good opportunities to solvethis challenging problem. On mobile devices, the GPS and timeof the camera can be obtained and some other photography re-lated context information like weather condition at the shootingtime can be further inferred. On the other hand, a large volumeof photos on social media websites are associated with meta-data such as GPS and timestamp, as well as EXIF includingthe camera parameters. Moreover, in the social media commu-nity, the photo quality can be implicitly inferred from numberof views, number of favorites, and comments, or even explicitlyobtained from ratings. Despite some noise in the media meta-data, the aggregated photographs captured nearby containingthe same scene with their metadata from the media commu-nity can provide significant insight into relevant photographyrules in terms of composition and exposure parameters for mo-bile photography assistance.For example, as shown in Fig. 1, when capturing a photo

    for Statue of Liberty from the perspective as shown in theinput wide-view image, using the content and context of theinput image, photos with the same content from similar per-spectives and their associated social information which can re-flect their qualities can be crowdsourced. By analyzing the com-position and their aesthetic quality of these crowdsourced im-ages, we find that pictures with the Statue on the right third lineare usually highly preferred; hence the optimal composition ofthe input image can be inferred. Moreover, with the input timeand weather condition, we can also estimate the optimal cameraexposure parameters from the crowdsourced photos with sim-ilar content and lighting conditions. Here, the first two crowd-sourced similar photos with high ratings were captured under

    Fig.