The University of Manchester Research

Impact of reviewing lifelogging photos on recallingepisodic memoriesDOI:10.1145/2968219.2968562

Document VersionFinal published version

Link to publication record in Manchester Research Explorer

Citation for published version (APA):Agroudy, P. E., Machulla, T., Rzayev, R., Dingler, T., Funk, M., Schmidt, A., Ward, G., & Clinch, S. (2016). Impactof reviewing lifelogging photos on recalling episodic memories. In UbiComp '16 Proceedings of the 2016 ACMInternational Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (pp. 1014-1019).(UbiComp '16 Adjunct). Association for Computing Machinery. https://doi.org/10.1145/2968219.2968562Published in:UbiComp '16 Proceedings of the 2016 ACM International Joint Conference on Pervasive and UbiquitousComputing: Adjunct Publication

Citing this paperPlease note that where the full-text provided on Manchester Research Explorer is the Author Accepted Manuscriptor Proof version this may differ from the final Published version. If citing, it is advised that you check and use thepublisher's definitive version.

General rightsCopyright and moral rights for the publications made accessible in the Research Explorer are retained by theauthors and/or other copyright owners and it is a condition of accessing publications that users recognise andabide by the legal requirements associated with these rights.

Takedown policyIf you believe that this document breaches copyright please refer to the University of Manchester’s TakedownProcedures [http://man.ac.uk/04Y6Bo] or contact uml.scholarlycommunications@manchester.ac.uk providingrelevant details, so we can investigate your claim.

Download date:18. Jun. 2020

https://doi.org/10.1145/2968219.2968562https://www.research.manchester.ac.uk/portal/en/publications/impact-of-reviewing-lifelogging-photos-on-recalling-episodic-memories(39f6eecb-d811-474e-92a7-6b8e82781f1e).html/portal/sarah.clinch.htmlhttps://www.research.manchester.ac.uk/portal/en/publications/impact-of-reviewing-lifelogging-photos-on-recalling-episodic-memories(39f6eecb-d811-474e-92a7-6b8e82781f1e).htmlhttps://www.research.manchester.ac.uk/portal/en/publications/impact-of-reviewing-lifelogging-photos-on-recalling-episodic-memories(39f6eecb-d811-474e-92a7-6b8e82781f1e).htmlhttps://doi.org/10.1145/2968219.2968562

Impact of Reviewing LifeloggingPhotos on Recalling EpisodicMemories

Passant El.AgroudyTonja MachullaRufat RzayevTilman DinglerMarkus FunkAlbrecht SchmidtUniversity of StuttgartStuttgart, Germany�rstname.lastname@vis.uni-stuttgart.de

Geoff WardUniversity of EssexColchester, UKgdward@essex.ac.uk

Sarah ClinchUniversity of LancasterLancaster, UKs.clinch@lancaster.ac.uk

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor pro�t or commercial advantage and that copies bear this notice and the full citationon the �rst page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior speci�c permission and/or afee. Request permissions from Permissions@acm.org.Ubicomp/ISWC’16 Adjunct , September 12-16, 2016, Heidelberg, Germany' 2016 ACM. ISBN 978-1-4503-4462-3/16/09...$15.00.DOI: http://dx.doi.org/10.1145/2968219.2968562

AbstractPhotos are a rich and popular form for preserving memo-ries. Thus, they are widely used as cues to augment humanmemory. Near-continuous capture and sharing of photoshave generated a need to summarize and review relevantphotos to revive important events. However, there is limitedwork on exploring how regular reviewing of selected pho-tos in�uence overall recall of past events. In this paper, wepresent an experiment to investigate the effect of regularreviewing of egocentric lifelogging photos on the formationand retrieval of autobiographic memories. Our approachprotects the privacy of the participants and provides im-proved validation for their memory performance comparedto existing approaches. The results of our experiment area step towards developing memory shaping algorithms thataccentuate or attenuate memories on demand.

Author KeywordsEpisodic memory; lifelogging; video summaries; photo re-viewing; recall; egocentric photos; retrieval induced forget-ting; memory augmentation

ACM Classi�cation KeywordsH.5.m [Information interfaces and presentation]: Miscella-neous

http://dx.doi.org/10.1145/2968219.2968562

MotivationHuman memory is naturally volatile and prone to errors[11]. As a part of their evolution, humans attempted to pre-serve their memories using different forms like writing anddrawing. Pictures have been a popular medium startingfrom pharaonic drawings on the walls of temples until cin-ematic visions of a brain-embedded chip that auto-recordsthe life as a video1. Pictures are strong memory cues be-cause they hold condensed and rich details about the con-text of a situation [8]. They support reminiscence, re�ection,search and recall of missing details.

FREQUENT TERMS

Episodic memory is thememory of the autobiograph-ical events that occurred inthe past and can be statedexplicitly [13]. (E.g.: Remem-bering your emotions duringa past trip when you were inhigh school)

Reminiscencea is theact of recalling past experi-ences / a mental impressionretained or revived

Re�ection a is a thoughtoccurring in considerationor meditation. It meansthinking critically about pastevents.

aFrom www.dictionary.com

Filtering photos here refersto removing sensitive or lowquality photos then selectingimportant ones.

Reviewing photos hererefers to looking at all thephotos or a selected subsetof them.

Therefore, numerous commercial platforms have been de-veloped to assist users in capturing and presenting mem-ories through photos. Small and aesthetic wearable cam-eras like Narrative clip 2 and iON Snap Cam 3 focus on notmissing a moment by continuously capturing a �rst-person-view of your life every 30 seconds. The result is massiveamounts of photos (ca. 1200 per day). On the other hand,social networks 4 like Facebook and Instagram empowerusers to preserve a selected set of highly relevant memo-ries shared among a group of persons.

The downside of the abundant affordable capturing de-vices, sharing and storage platforms is the crippling largedatasets. Filtering and reviewing the produced photos is atime consuming and overwhelming task. Thus, the infor-mation retrieval community intervened with solid efforts toautomate the selection and presentation of important pho-tos. For example, Gurrin (E.g.: [9]) and Doherty (E.g.: [4])have done extensive work on developing selection algo-rithms for key photos and implicit detection of events andactivities. On the other hand, Girgensohn (E.g.: [7]) and

1 Final cut movie (http://www.imdb.com/title/tt0364343/)2 http://getnarrative.com/3 https://usa.ioncamera.com/snapcam/4 www.facebook.com , www.instagram.com

Byrne (E.g.: [2]) worked on �nding presentation techniquesfor the generated summaries.

The results of this research were adopted and comple-mented by the commercial lifelogging platforms to createuser experiences that support reminiscence and re�ection.For example, Facebook offers an On this day applica-tion that reviews events from previous years that occurredon the same day. The narrative clip offers an on-demandtimelapse video of the photos taken within a day. Also, it au-tomatically clusters the dataset into relevant moments afterremoving low quality pictures.

Despite the popularity of these applications and the abun-dance of the photo-reviewing tools, there is limited researchusing real-world datasets on how the reviewing processshapes our episodic memory (E.g.: Studies using Sense-Cam like [5] and [12] ). A possible reason is the challengingand unstandardized nature of lifelogging / memory exper-iments. Privacy concerns about the lifelog photos compli-cate the recruitment of participants as the photos exposesensitive information. Moreover, there are legal concernsabout sharing the lifelogs with researchers because of un-consenting individuals in the photos. Consequently, groundtruth for the recalled events is usually missing. Therefore,most well-established frameworks do not require the vali-dation of the narrated events by the participant and operateunder the assumption that the provided data is correct (e.g.,cognitive interviews [6]). Consequently, these frameworksprovide only loose guidelines for extracting memories. Ad-ditionally, they require consensus regarding the scoringsystem as well as cross-rater validation by multiple experts.

www.dictionary.comhttp://www.imdb.com/title/tt0364343/http://getnarrative.com/https://usa.ioncamera.com/snapcam/www.facebook.comwww.instagram.com

VisionWe present an approach to investigate the effect of review-ing lifelogging photos on the formation and retrieval of au-tobiographic memories. It uses personal lifelogs, protectsparticipants’ privacy and obtains subjective validation of therecalled events.

This work is part of the European Union project RECALL5.We aim for creating ubiquitous technologies that augmentthe human memory. A common approach to memory aug-mentation is to create an easily searchable and reachableexternal memory prosthetic that records everything. In con-trast, we envision improving the human memory withoutneeding external memory prosthetics.

In the following sections, we will describe the research aimsand the design of a planned study. In this study, we predictthat the reviewing process will improve the recalling of re-viewed events. However, it could also impede the retrievalof unreviewed events as the life of the person is crystal-lized around the selected photos by the algorithms. Theresults will provide deeper insights into the role of photo-selection systems in shaping our memories and thus per-ception about life. It also aligns with RECALL’s ultimategoal of building context-aware systems that can attenu-ate/accentuate memories on demand.

Research AimsWe want to capture potential improvement or deteriorationin recalling past events. We will explore the following ques-tions:

FREQUENT TERMS

Retrieval Induced Forgetting(RIF) refers to the forgettingof some information due totrying to remember otherrelevant information. Whilethe existence of this phe-nomenon is well supportedby laboratory studies, itsreproduction in real-worldsettings is still subject toon-going researcha.

a http://recall-fet.eu/2015/07/recall-at-sarmac-2015/

Figure 1: We predict thatparticipants will choose Pic 2 asa representative for Meeting Markevent because it is closer to thereviewed photo.

1. Does photo reviewing affect the amount of recalledevents? The results will indicate the viability of usingauto-generated photo reviews as a memory aid toenhance selective recall. We expect that the sheer

5 http://recall-fet.eu/

number of recalled events from the reviewed days ishigher than the unreviewed days.

2. Does photo reviewing affect the quality of the recalledevents? The focus here is on the amount of recalleddetails about each event. We will examine if the re-viewing promotes recalling richer details about theevents.

3. Does photo reviewing affect the recall of close un-reviewed events? Closeness in this context is thetemporal proximity. There are two potential and op-posite effects for the reviewing process. The �rst isbetter recall of unreviewed events if they are relevantor interconnected with reviewed events. The secondone is worse recall of unreviewed events due to theRetrieval Induced Forgetting phenomenon [1]. Thiseffect is relevant because it implies that enhancingthe memory for some photos (i.e. events) could in-hibit remembering other events. This poses an ethicaldilemma about the implications of deliberate memoryshaping tools as discussed in [3].

4. Are the reviewed photos recognized better than theunreviewed photos? We expect that the reviewedphotos are better recognized. Moreover, we predicta tendency to favor the reviewed photos as repre-sentatives to the recalled events. We also expect atendency to choose representative unreviewed pho-tos that are temporally close or spatially similar to thereviewed photos (See Figure 1).

5. Are the recalled events following a pattern? We willexplore the data to identify if the free recall is basedon a heuristic (E.g.: chronological activity-driven re-call). This will enable us to understand more abouthow the memory works in a natural setup.

http://recall-fet.eu/2015/07/recall-at-sarmac-2015/http://recall-fet.eu/2015/07/recall-at-sarmac-2015/http://recall-fet.eu/

MethodologyWe will conduct the study, using a within-study design on 15participants for 4 weeks each. The participants are mainlyuniversity students. Our system is composed of a wearablecamera with automatic timelapse mode (E.g. narrative clip),a mobile GPS logger (E.g.: GPS logger mobile app), a ded-icated laptop acting as a processing unit and a softwarefor analyzing, selecting and generating a review (summaryvideo). The extraction process of the key pictures is basedon the requirements elicited in [8]. The video duration isapproximately 2-3 minutes of key pictures. Figure 2 sum-marizes the morning routine of the participant. The partici-pants can only review a day using our auto-generated videofor three days per week. After each review session, theywill answer a short questionnaire to subjectively assess thevideo content.

Figure 2: Morning routine for theparticipants during the experiment.(1) The narrative clip camera isalways on, capturing a pictureevery 30 seconds unless covered.(2) The participant starts the GPSlogging explicitly on the smartphone. External GPS is used toovercome the proprietary locationformat of the camera and to avoiduploading the pictures to narrativecloud service.

By the end of each week, the participants are invited to thelab for an individual memory testing session. The session isdivided into four phases summarized in Figure 3:

Phase 1 - Recall events The target is to prompt the par-ticipants to recall as many events as they can fromthe previous week. They are given a �xed interval oftime (E.g.: 10 minutes, divided into 1-minute rounds)to recall past events. They note 1-3 words per eventas a cue to remember it during the session. This isan adaptation of the technique described in [10].

Phase 2 - Recall details and categorize events The par-ticipants are asked to recall speci�c details abouteach event (E.g.: time, location, . . . ) then submit arating of their certainty about each detail. They alsorate the importance of the event and tag it with prede-�ned activities (E.g.: Work, Relax, Family, . . . ).

Phase 3 - Select a photo per event and validate detailsEach participant views the photos taken through-out the week on a custom large-screen setup. Theypick a photo to represent each event. The systemautomatically asks them to rate the relevance of theselected photo to the closest two key pictures se-lected by the algorithm (before and after the selectedphoto). The participant also reviews his/her answersfrom phase 2 and validates them.

Phase 4 - Cluster photos to events Each participant re-views the weekly dataset of photos. (S)he groupsthem into distinct events and tags them with the sameprede�ned tags used in phase 2. Favourite photosare also marked. This phase is optional.

The timestamp of the selected representative photo foreach event (in Phase 3) enables us to identify if the photo(i.e. the event) was reviewed or not and infer it’s temporalrelation to the other events. Thus, we could answer ques-tion 1, 3 and 4. The number of entered details in phase2 with their validation from phase 3 enables us to answerquestion 2. Furthermore, the collected tags from phase 2and phase 4 enables us to answer question 5.

The proposed design protects the privacy of the partici-pants as the personal data (photos and GPS locations) arestored locally and are accessible by the participant only. Wecollect only the number of recalled events, photos’ names(timestamp format), ratings and miscellaneous computervision descriptors. Thus, the design offers a trade-off forquantifying abstract concepts, protecting privacy and lever-aging meaningful analysis.

Figure 3: Overview on the memory testing lab session.

SummaryWe propose an experiment to investigate the impact of re-viewing selected memories in the form of lifelogging photoson recalling past events. We predict that the reviewing willimprove the recall of reviewed events but impede the recallof unreviewed events. The results contribute to understand-ing the adverse effects of continuous lifelogging and review-

ing of memories. Moreover, they support the developmentof algorithms that control recall on demand.

AcknowledgmentsThe authors acknowledge the �nancial support of the Fu-ture and Emerging technologies (FET) programme with the7th Framework programme for Research of the EuropeanCommission, under FET grant number:612933 (RECALL)

REFERENCES1. Anderson, M., Bjork, R., and Bjork, E. Remembering

can cause forgetting: retrieval dynamics in long-termmemory. Journal of Experimental Psychology:Learning, Memory, and Cognition 20, 5 (1994), 1063.

2. Byrne, D., Lee, H., Jones, G., and Smeaton, A.Guidelines for the presentation and visualisation oflifelog content. In Proc. iHCI (2008).

3. Dingler, T., and Henze, N. That’s the dog from mywedding - algorithms for memory shaping. WorkshopPaper. Adjunct Proc. CHI Workshop on DesigningTechnology for Major Life Events, April 2014.

4. Doherty, A., Byrne, D., Smeaton, A., Jones, G., andHughes, M. Investigating keyframe selection methodsin the novel domain of passively captured visuallifelogs. In Proc. International Conference onContent-based Image and Video Retrieval (New York,NY, USA, 2008), CIVR ’08, ACM, pp. 259268.

5. Finley, J., Brewer, W., and Benjamin, A. The effects ofend-of-day picture review and a sensor-based picturecapture procedure on autobiographical memory usingsensecam. Memory 19, 7 (2011), 796807.

6. Geiselman, R., Fisher, R., MacKinnon, D., and Holland,H. Eyewitness memory enhancement in the policeinterview: cognitive retrieval mnemonics versushypnosis. Journal of Applied Psychology 70, 2 (1985),401.

7. Girgensohn, A. A fast layout algorithm for visual videosummaries. In Multimedia and Expo, 2003. ICME ’03.Proceedings. 2003 International Conference on (July2003), vol. 2, pp. II7780 vol.2.

8. Le, H., Clinch, S., Sas, C., Dingler, T., Henze, N., andDavies, N. Impact of video summary viewing onepisodic memory recall: Design guidelines for videosummarizations. In Proc. CHI (New York, NY, USA,2016), ACM, pp. 47934805.

9. Lee, H., Smeaton, A., O’connor, N., Jones, G., Blighe,M., Byrne, D., Doherty, A., and Gurrin, C. Constructinga sensecam visual diary as a media process.Multimedia Syst. 14, 6 (Dec. 2008), 341349.

10. Moreton, B., and Ward, G. Time scale similarity andlong-term memory for autobiographical events.Psychonomic Bulletin & Review 17, 4 (2010), 510515.

11. Schacter, D. The seven sins of memory, 2001.

12. Sellen, A., Fogg, A., Aitken, M., Hodges, S., Rother, C.,and Wood, K. Do life-logging technologies supportmemory for the past?: an experimental study usingsensecam. In Proc. CHI (2007), ACM, pp. 8190.

13. Tulving, E. What is episodic memory? CurrentDirections in Psychological Science 2, 3 (1993), 6770.

MotivationVisionResearch Aims