Storing The Whole Worlds Memories - Lancaster University The Whole Worl… · LifeLogging and Quantified self are described as the process of tracking personal data generated by our

Ben Taylor 33117535

1 | S t o r i n g T h e W h o l e W o r l d s M e m o r i e s

Benjamin David Taylor

Storing The Whole Worlds Memories

MSci Computer Science (with Industrial

Experience)

Submission Date - 20/03/15

Ben Taylor 33117535


I certify that the material contained in this dissertation is my own work and does not contain

unreferenced or unacknowledged material. I also warrant that the above statement applies to

the implementation of the project and all the associated documentation. Regarding the

electronically submitted version of this submitted work, I consent to this being stored

electronically and copied for assessment purposes, including the Department's use of plagiarism

detection systems in order to check the integrity of assessed work.

I agree to my dissertation being placed in the public domain, with my name explicitly included

as the author of the work.

Date: 20/03/15

Signed:

Ben Taylor 33117535


Abstract -

Human memory is fallible, and technology has advanced to a point where it is possible to gather

information on many aspects of a human life. This project aims to explore the different methods

of memory capture, storage and retrieval. A small application was produced based on research

around memory, and the tagging of images to improve memory recall. Results show image

tagging can be an effective aid, but more work would need to be done to reveal their full

potential.

Ben Taylor 33117535


Contents 1. Introduction ................................................................................................................................................................... 6

2. Background .................................................................................................................................................................... 6

2.1. Memory Augmentation ..................................................................................................................................... 7

2.2. LifeLogging/Quantified Self ............................................................................................................................ 8

2.3. Storing and Processing Memories ................................................................................................................ 9

2.4. Retrieving Memories ..................................................................................................................................... 10

2.5. File Systems ........................................................................................................................................................ 12

2.6. Memory Security .............................................................................................................................................. 13

3. Design ............................................................................................................................................................................ 14

3.1. System Evolution .............................................................................................................................................. 14

3.2. Decisions and Justifications ......................................................................................................................... 16

3.3. Overall architecture......................................................................................................................................... 18

4. Implementation ......................................................................................................................................................... 20

4.1. Language and Libraries ................................................................................................................................. 20

4.2. User Interface ..................................................................................................................................................... 21

4.3. Issues ..................................................................................................................................................................... 22

4.4. The System in operation................................................................................................................................ 23

4.5. Background Processes ................................................................................................................................... 32

5. Testing ........................................................................................................................................................................... 33

5.1. User Interface Testing .................................................................................................................................... 34

5.2. Background Process Testing ....................................................................................................................... 44

6. Evaluation .................................................................................................................................................................... 44

6.1. User Interface Evaluation ............................................................................................................................. 45

6.2. Survey Justifications ........................................................................................................................................ 46

6.3. Survey Results ................................................................................................................................................... 48

6.3.1. Tagging Evaluation .................................................................................................................................. 48

6.3.2. Searching By Tags .................................................................................................................................... 50

6.3.3. Which Data Is Most Helpful? ............................................................................................................... 51

7. Conclusion ................................................................................................................................................................... 54

7.1. Review Of Aims ................................................................................................................................................. 54

7.2. Suggested Revisions To Design/Implementation ............................................................................... 55

7.3. Future Work ....................................................................................................................................................... 56

8. Bibliography ................................................................................................................................................................ 56

9. Appendix ...................................................................................................................................................................... 58

9.1. Tagging Survey .................................................................................................................................................. 58

9.1.1. Screenshots of survey ............................................................................................................................ 58

Ben Taylor 33117535


9.1.2. Survey Questions ..................................................................................................................................... 62

9.1.3. Survey Results ........................................................................................................................................... 62

9.2. Project Proposal ................................................................................................................................................ 69

Link to Working Documents: http://www.lancaster.ac.uk/ug/taylorbd/index.html

Ben Taylor 33117535


1. Introduction Ubiquitous computing has grown to a point where there is now the potential to capture

many different aspects of a person's life. This has produced a movement known as LifeLogging,

or Quantified Self, where a person will wear technology to capture as much of their life as

possible. The technology they wear can vary from simple wearable cameras(Narrative Clip) to

exercise monitoring bands(Nike Fuel) and sleep monitoring equipment(Sleep Cycle Android

App). With the wide variety of data able to be collected, this technology is now able to support

memory augmentation.

It is clear human memory is inherently fallible, yet few people have tried to develop a

solution to aid memory recall and retention. This work will explore how technology can be used

to support memory augmentation, particularly the role software can play in achieving this goal.

The capture, storage and retrieval of personal memories will be investigated in detail, these

investigations will result in a piece of software being created which effectively performs as a

memory aid if successful. This project aims to explore different solutions to the end goals, and

apply the most suitable to the end system. The end goals of the project are as follows:

• Be able to capture and store memories

• Attach data to memories to aid memory recall

• Design an effective system to search memories

This report will cover the steps which were taken to produce and evaluate a system

which aims to meet the end goals. A brief overview of each section is given below.

Background - This section covers the research which has been carried out prior to

designing the system, it will also include explanations of terminology used and give general

background on this area of study.

Design - This section includes reasoning for design decisions, which will be based upon

research which was covered in the background. The evolution and final overall architecture of

the designed system will be covered, along with a description of each component.

Implementation - This section explains all language, library and toolkit use. An

overview of issues encountered along with possible solutions will also be included. This section

will be finished off with a run through of the created GUI.

Testing and Evaluation - This section explains how testing and evaluation was carried

out, and which tests were run on the system. Results from the testing will be discussed and

evaluation results will be analysed.

Conclusion - The conclusion wraps up the content contained of the report and gives an

overall review of the project. Possible improvements and further work will also be discussed.

2. Background Capturing and storing memories is not a new concept, it is an idea that has been around

for decades. One of the earliest ideas for such a system is known as the Memex, this was

proposed as a hypermedia system which could organise all of the knowledge a person might

gather during a lifetime (Manovich, n.d.). A user of Memex could create links between books,

journals and records etc. which would produce what was known as 'trails' (Gurrin, et al., 2014),

these trails could then be traversed by the creator to remember their thought processes or, by

others to see that persons thought processes. Since the first proposal of the Memex computing

has grown and improved, and memory capture has moved from knowledge to episodic memory.

Ben Taylor 33117535


2.1. Memory Augmentation

Memory augmentation is measured as a person's ability to retain information is

improved through the use of aids, it is also a method of improving a person's memory recall as

the human brain can be inaccurate and unreliable. There are different methods to augment a

person's memory, and these can all be implemented into a technological solution for memory

augmentation. When a study was conducted (Tatsuyuki, et al., 2006) it was found a

technological solution proved to be the best as it made the least mistakes, this shows that a

technological solution could greatly improve a person's memories if it was implemented

correctly.

Plenty of research has gone into human memory, how it works and what best helps a

person to remember. Before a technological solution can be created there are barriers which

must be overcome (Whittaker, et al., 2010), the biggest of which is people prefer physical

mementos over digital, it is not entirely clear why this is the case. This means that any system

which will be designed for memory augmentation will need to want to be used by the user

otherwise they may simply forget about it and not use it.

This is shown in a study (Petrelli, et al., 2008) in which parents with young children,

who also claim to actively curate their lives with digital memorabilia, were asked to give a tour

of the house. While giving the tour the parents were asked to indentify objects which triggered

important memories for them, and surprisingly all but one object across all families was a

physical mementos. This tour was later followed up by an interview in which the families were

asked if they had any digital memorabilia, they first denied they had any expect from digital

photographs, but when questioned further it was revealed they had a wide variety of digital

memorabilia.

It is unclear why digital memorabilia is so easily dismissed unless directly questioned

about. This could be because of digital invisibility, objects which are not seen as often are not as

easily recalled as memorable, but when the physical mementos were further investigated it was

revealed 25% of the physical objects where not in view. The further investigation also revealed

28% of the objects were functional and used every day, which then reinforces the memories

associated with them.

In a further study by (Whittaker, et al., 2010) 18 people were asked to wear a SenseCam

for 2 weeks, which collected thousands of images each day, to create a visual log of their daily

activities. Two months after this the same people were asked to recall events from a specific day

within their SenseCam use, the results revealed the collection of images were not as useful as

first assumed, in fact when compared to unaided memory the results were the same. The study

participants suggested the number of images were too overwhelming, which meant it was

difficult to get any information from them.

Perhaps one of the most important points to take from the studies performed by

S.Whittiker is known as Beyond Factual Recall, this is the point where memory is not simply the

retrieval of factual information, it is much more complex. To remember a event in a person's life

they must reminisce and reflect, this needs to be imitated in any digital tools created to support

memory.

This information clearly reveals more work will need to be input into a system based on

memory than simple searches of data. To improve methods of memory recall and augmentation

more information is needed than just a collection of images. This can go from organising the

Ben Taylor 33117535


data somehow, to attaching more data to the images to aid the memory in recalling events.

Methods such as this will be discussed in more detail later in the retrieval section.

2.2. LifeLogging/Quantified Self

LifeLogging and Quantified self are described as the process of tracking personal data

generated by our own behavioural activities, this covers activities such as eating, sleeping and

exercising (Krynsky, 2015). LifeLogging is not a very well known as a word, but it is in use by

more people than expected, this is because of the introduction of many probes into people's

lives, this type of probe normally comes under the type of Personal Biometrics. There are

however many different kinds of data which can be gathered for LifeLogging, each type

capturing a different part of a person's life.

Passive Visual Capture - This type covers devices which captures visual cues from a

person's life, it will include images and video, normally from the point of view of the user.

Devices which capture this type of data are usually small, wearable and always on. Examples of

devices with this type of capture are Narrative Clip and SenseCam.

Personal Biometrics - This type covers a wide variety of capture devices, in general

these devices are mostly used to create a quantified self. These are the most well know devices

as they are in commercial sale, and monitor activities such as sports, sleep and food intake.

Mobile Device Content - This is where a personal Smartphone is used to collect data.

This is the least obtrusive kind of probe as most people already carry a Smartphone. This will

gather data on people such as location, acceleration and movement. An example of this is the

Android App known as Saga.

Communication Activities - This type covers digital communications, it will passively

log e-mail, SMS, phone calls and social media. Tracking like this is unobtrusive, but all data has

semantic context so it can be hard to understand when reading the data back.

Data creation and access - This type covers a log of access data, e.g. Books and

webpages, this is not dissimilar from Memex. It will also track data which has been created by

the user. e.g. What they type into the computer.

All of these logging devices cover a different aspect of a person's life except from Mobile

Device Content, this is a special case. The number of sensors and battery power in a Smartphone

allow it to capture many different parts of a person's life, this can cover many of the different

types of LifeLogging tools. These devices all similar in one aspect, they are all small and

unobtrusive devices which means they are able to capture memories without much added effort

from the user.

Being able to collect so much data on a person opens up a lot of options for memory

capture and storage. Most of these methods have not been extensively tested for their ability to

aid memory recall, the most researched method of the above is visual aids. Previous work

emphasizes autobiographical memories are remembered through internal images (Conway,

1990) (Pillemer, 1998), past memory science experiments also show memories can be

spontaneously remembered based on a trigger such as an image (Hamilakis & Labanyi, 2008) .

Using this knowledge it would make the most sense to capture and store memories through

Passive Visual Capture. Not only could Passive Visual Capture be used but it could be

supplemented by personal images and video.

Ben Taylor 33117535


Using a passive camera can raise privacy issues, as it is always on some people are

uncomfortable knowing there is a camera watching them if they do not know what the images

will be used for. Google glass has faced similar issues with privacy (Thomas, 2015), many

businesses have banned the use of Google Glass on their premises to protect their customers

from what could be an invasion of privacy. Legally the use of Google Glass to capture memories

is fine, provided the captures are not shared and are only used for personal use. While these

captures are only memories from a person's point of view they are different from memories.

Recordings and images are a permanent memory until deleted, but typical memory is inherently

fallible and is so likely to forget without aids.

2.3. Storing and Processing Memories

Finding a way to capture memories overcomes the first barrier, but the bigger problems

arise when trying to store and then retrieve these memories back. Memories can be divided into

two types semantic and episodic (Gurrin, et al., 2014). Semantic memories are described as

knowledge, this type of memory can potentially be shared between all people e.g. The fact Paris

is the capital of France. Episodic memories are the more personal, each person has their own

account of events and no two people can have the same episodic memory, although they can be

similar, this type of memory is the main focus of this project. There is a huge assortment of data

which can be collected to capture memories, as discussed earlier video aids are best for memory

recall if enough is done the brain remember, for this reason this report will focus on the storage

and retrieval of visual memory aids.

Once a set of memories have been captured an effective method of storage and recall

need to be used, otherwise the devised system is unlikely to be used. All storage, processing and

retrieval of data needs to happen quickly to try and improve upon unaided memory, if no

improvement on unaided memory can be seen by the users they will revert to using their own

memories instead - seeing the system as unreliable. Evidence of this was shown in a study

(Whittaker, et al., 2010) where a group of people were given 3 methods of aid to remember a

conversation. The choices were a Dictaphone, a device known as ChittyChatty and pen and

paper, the final results shown the ChittyChatty was the most favoured followed by pen and

paper then finally the Dictaphone. This shows peoples tendency to revert to less complete

records for easier access, the Dictaphone easily produced a more complete record of the

conversation but its lack of easy access made it much less useful.

It has already been discussed how huge records of data do not promote effective

memory recall, so the raw data of what could be thousands of images each day is not going to be

useful in any memory system. This implies the data will need supplementing in some way and

organising, the best way of doing both of these will be a way which mimics the way memory

connects and organises data, as this will also improve how effective the data is when it is needed

for memory recall. One solution would be to impose organisation on users, but this would be a

mammoth task for a person using a device such as Narrative Clip, even people just using

personal images and videos are unlikely to organise their data (Whittaker, et al., 2010). This

means there will need to be some form of automatic organisation being carried out on the input

data, there are two choices for grouping visual memories simple chunking of data into some

length of time, or defining groups of data as 'events'. (Gurrin, et al., 2014)

The first of these two methods is the easiest, it simply requires some time and date

location on each input then grouping it together by 1 hour chunks for example. This is not the

most effective method to aid memory recall as a person can easily misremember the time and

Ben Taylor 33117535


date an event they are trying to remember happened. The chunks of data could then be

manually summarised by the user to improve effectiveness, but this has already been proven to

be a task people are unwilling to complete themselves.

The second method involves grouping the data together based on a unit of measurement

known as an 'event', an 'event' is a sequence of temporally related life log data over a period of

time. Events have had prior research and are much more useful for memory recall as they better

reflect the way the human memory groups memories together. This makes them ideal for

mimicking memory, but the detection of an event can be difficult by a machine which makes this

method difficult to use.

It has already been said that supplementing visual memories will aid storage, retrieval

and memory recall, and as people are unwilling to manually add this data automatic solutions to

this need to be looked at. While capturing images Narrative Clip tags each image with a

timestamp and location stamp, these are two simple pieces of data can be extremely helpful for

a memory system, in an ideal world all input data into a memory system would have this data

included, but this is not the case. Another method of adding meta data to visual memories will

need to be used, one which can automatically be added no matter the image, initially an idea

would be to use facial recognition but there is no guarantee a person will be in every captured

image and video. Extending on the idea of facial recognition, general tagging could be carried

out, but this would depend greatly on the ability of a tagging system to correctly recognise

objects, and for it to produce tags similar to what a person would expect.

A final point to consider would be using many different types of input, for example

combining visual memories with information collected from an Android App. This app would be

able to collect data such as time, location and movement, while all this data would be able to

enhance the visual memories actually being able to link the data temporally could prove to be a

challenge in itself as the visual data may not contain any time data.

2.4. Retrieving Memories

Retrieving memories can prove to just as challenging as storing them, especially when it

has to be competitive against the human brain. When storing memories the human brain will

store information more as a collage, than an organised list like a bookshelf in a library. All items

within this huge collage are then linked together via links (Mastin, 2015), in terms of computing

this sounds like a terribly inefficient way to re-access memories should they be needed again. In

the brain however this is very efficient, the brain is able to directly access these memories when

it wants to recall the information, whereas a computer system would need to search through all

possibilities until the wanted result is found.

This brings on the biggest question of this section of the report, how can a computer

system search a huge directory of memories and produce results in real time? All conventional

methods of searches are designed to search a set of data by traversing through it in order until

the correct result is found, and with a potential for so much data needing to be searched a better

result needs to be found. Just like in the storage section adding as much meta data as possible

will greatly improve the quality of memory retrieval. This is simply because there will be more

methods to search for the data by, while this does not solve the problem of sequential searching

is does aid in providing better results which are better related to the query.

One method to improve current search methods is to try and tailor them towards

memory instead of trying to redesign current storage and retrieval methods. In the same way

Ben Taylor 33117535


memory recall is more effective on already recalled events and important events, it is a good

idea to attempt to introduce that method into these searches (Whittaker, et al., 2010). Once

again to avoid a user having to tag all of their own data a automated system would need to be

put into place, one which can define an important situation, e.g. a meeting, over an much less

important one e.g. Watching TV. This is not a system which would be simple to implement and

so an easier method of automatically marking the data as more important is needed. A much

simpler method to approach this would mean assuming all memories are unimportant to begin

with. When any memory is accessed that memory should be marked as more important and

then placed into a separate data structure of more important memories, when any queries are

run later on this separate data structure will be searched first. If this system is implemented the

benefits would not be realised at first, but if it was used enough it would soon filter out all the

less important memories. Improving on this, temporally linked memories could also be

promoted, this acts similar to a cache, although it is much more permanent.

Once a search and retrieval method has been chosen, a way of easily displaying the data

which has been recovered needs to be decided upon. A series of uses and reasons for retrieving

memories needs to be considered, (Sellen & Whittaker, 2010) summarised these uses into,

what they call, the 5R's. Each of these 5R's address different access requirements in memory

data, the 5R's with a brief summary and example are below.

Recollecting - When a person would like to relive a past experience in order to

remember a piece of information e.g. recollecting a party to remember who they met while

there. In order to support this a system would need to be able to return information usually in

the form of a video or image. The retrieved information does not necessarily need to document

the event, it only needs to trigger the users of memory retrieval process.

Reminiscing - Similar to recollection, reliving a past experience, but in this case it is for

emotional or sentimental reasons. This type of memory requires a bigger overview of what

could be a large period of time, e.g. a holiday. An effective way of passing this information back

to the user will require summarising of large data into important chunks to support emotional

memory recall.

Retrieving(Information) - This is a much more specific form of recollection, where a

person is seeking a specific piece of information e.g. An address, phone number or document.

While the display of information like this would be simple, retrieving could prove to be difficult

as data may not be in plain text.

Reflecting - Linked more towards personal biometrics, it involves analysing large

quantities of data produced over a number of days. This is used to reveal patterns and to view

past experiences from a different perspective. Displaying this data would involve typical

summaries of large quantities of data, e.g. graphs and charts.

Remembering Intentions - Attempting to recall a person's prospective memory, rather

than their past. This is used to assist in remembering future plans, which could be relevant to

the current time e.g. Phone the doctor. This could prove to be the most difficult of all of the 5, as

capturing a person's thought process is an immediate challenge.

A final problem which would need to be tackled when retrieving memories is 'event

decay' (Doherty, et al., 2009). Event decay is based on the earlier discussed way of grouping

data known as the 'event', where an event is a sequence of temporally linked LifeLog data. Event

decay is a model where, as time passes a person will change the way in which they group events,

so actions which are seen as significant enough to become an event at one time may later be

forgotten about as it is no longer seen as significant. This provides an opportunity to reduce the

Ben Taylor 33117535


amount of data being stored as time passes, previous events could be combined thus freeing

space. The issue is doing this in such a way it does not summarise data too early, therefore

losing data the user still decides is useful, and when data is summarised, not shrinking it down

so much that important data can be lost when the user tries to retrieve it. To solve this problem,

although not ideal as it requires human confirmation, would be for the system to detect old

events i.e. 2+ years old, and suggest those events to the user to be combined, if the user still

finds these events significant enough to keep separate they can easily say so, otherwise they can

be grouped.

2.5. File Systems

File systems and memory usage is another section to cover when capturing memories.

Memory capture can produce huge amounts of data for every day they are being used, the

amount of data captured is dependent on the number of ways a user wishes to capture their

lives, but over time any method can produce large amounts of data, so it must be considered.

This section will be covering space usage of different types of memory collection and storage

architectures with their positives and negatives.

Data Type Volume per Year Memory Usage -

1 Year

Memory Usage - A

Lifetime

HD Video 5840 Hours 32.8TB 2.65PB

Standard Video 5840 Hours 17.1TB 1.42PB

Audio (Mono - 22KHz) 5840 Hours 227.8GB 19.4GB

Narrative Clip 700800 Images 634.92GB 52.7TB

Accelerometer (1Hz) 21 Million readings 0.05GB 4.25GB

Locations (0.2Hz) 3.9 Million GPS points 0.01GB 0.85GB

Reading Log User Dependant 1GB+ 85GB+

Table 2.5.1: An illustration of different types of data capture along with their quantities and

memory usage over periods of time. Volume per Year is based on 16hours a day being spent

capturing data, a lifetime is based on 85 years spent capturing data. (Gurrin, et al., 2014)

With such a wide variety of memory capture available, the memory usage per choice of

capture can vary greatly. Table 2.5.1 shows a brief illustration of memory usage per different

methods of capture, volume per year is based on 16 hours a day spent capturing data, being

captured 365 days a year. Memory usage based on a lifetime is an estimation early, it is not

unreasonable to assume memory usage of different data types could change, making these

upper estimates only. Along with memory usage improving, if following Moore's Law, what may

seem like a large amount of data usage for a lifetime i.e. 2.65PB, would soon seem very

insignificant by the end of the 85 years. With this knowledge the memory usage of capturing all

of this data is not a huge problem at this time, and as time progresses become much less of a

problem.

Storage space and memory usage is just a small part of storing memories, the data

structures and file systems for storing such data also need to be considered. Traditional

methods for storing large amounts of data have been to use a centralised data store, where all

data is stored in a central location, this is because these systems are much simpler to implement

quickly and effectively. These systems are not without their issues though, to store many users

personal data within a central store opens up vulnerabilities to large scale privacy violations via

Ben Taylor 33117535


inadvertent disclosures (Paul, 2010) and malicious attacks (Cheng, 2010). As to what could be

done with data retrieved from security breaches will be discussed later.

To tackle these big security issues one idea has predominately been accepted, to build a

true de-centralised distributed system, which limits the effects of security breaches while still

providing fast access times. One such project is called OceanStore (Kubiatowicz, et al., 2000),

this project aims to produce a system which is consistent, highly available and durable while

being built atop a infrastructure of untrusted servers. These goals are achieved by considering a

number of systems, including security, optimisation, distributed systems and location

algorithms.

Security - The system can be built on top of any number of servers, secure or not, this

creates a hugely flexible system where any server can join, therefore no extra infrastructure is

required to be built. This is achieved by not trusting any server with clear text, all data which

enters the system is encrypted.

Optimisation - In order to match efficiency levels of a typical centralised data store a

de-centralised system requires a high level of optimisation, for large distributed systems this

can be very difficult to do manually. In OceanStore automatic optimisation is achieved by having

a group of servers which monitor and optimise the system e.g. Moving a copy of regularly

accessed data closer to the user.

Distributed System - Any de-centralised data store will be, at its centre, a distributed

system. This means it will employ many of the principles of such a system, such as concurrency

and fault tolerance.

Location algorithms - With such a large amount of data and number of servers there

needs to be a way to find data quickly and efficiently. This is achieved through the use of a

probabilistic algorithm which will attempt to find the requested data on a server close to the

requester, A slower method is only used if the probabilistic algorithm fails.

2.6. Memory Security

When storing a person's memories security is of upmost priority. Memories are

personal and can be very private, so when a person chooses to use a system to help capture and

store their memories they need to know they are going to be secure. The earlier section

discussed what could be done when storing data to keep it secure, this section will focus on the

consequences of data being stolen, and how they can be manipulated to affect a person's way of

thinking (Davies, et al., 2014).

If a system or company is able to access a person's memories through a security fault

that person could be influenced to remember certain events in specific ways. This is possible if

current suggested psychology theories are true, that cued recall is able to both re-enforce and

attenuate memories. In practical terms this could mean an advertising company controlling

which memories you do and do not see, so are much more likely to remember positive times

with their product, while mostly negative times with a rival product. This type of memory

manipulation is the most concerning, it may not be possible for a person to tell if their memories

are being manipulated.

Furthermore in the future the most effective memory systems will not necessarily be

controlled by any one person, it would be a system which could source data from many places,

both personal and environmental. This would mean a system where people will share sensors,

Ben Taylor 33117535


e.g. a recording of a meeting. The security issues with a setup like this become more extensive to

avoid memory manipulation, how can a personal system detect manipulation before it has even

received the data. Before memories can even be sent to people in a shared system the first step

to overcome will involve deciding who deserves access to that data.

3. Design The design process for the system was undertaken in an unusual way. The design of the

system was solely based on research, there was no previous work to base this off, this made

designing the system more complicated. Instead of designing the system all in one go the design

was done step by step during implementation, this allowed the system to evolve based on the

tools which could be found and used. Designing the system this way meant a set of

requirements were not fully set out until the final design and implementation which was tested

and evaluated. This section of the report will explore how the system developed over time to

produce the final system, justification for the choices made during design will then be discussed.

To finish, this section will end with a explanation of the end system as a whole.

3.1. System Evolution

This section will cover how the system developed from a few simple components into

the final product. Only the steps in which the system changed will be covered in this section, the

detailed justifications for these choices are covered in the next section (Section 3.2.). Each step

of the process will be explained one at a time in the following paragraphs, until the end product

is achieved.

The system started out as a simple program which could read in live video from a

webcam while displaying and saving the video. This simple program allowed plenty of

experimentation to take place to see how videos could be manipulated, and how any of this

manipulation could help memory. This step also allowed knowledge to be gained about the

library which was used to manage the video input and output. At the end of this stage a simple

program had been made which took in live video, video chunking was then introduced to make

the video easier to review. Each chunk of the video would be saved into its own folder.

From here something had to be done with the video data to make it easier to access than

just a simple search. Many options were explored to add more data to the system automatically,

the added data would have to be automatic and able to be applied to every video. Following this

rule would provide another way of searching through videos which could be more direct than

simply looking through a list. Initial types of data to add involved location, timestamp, facial

recognition and giving each camera a unique ID. At this time the system was based on a series of

cameras all connecting to a central data store(See Figure 3.1.1), this data store could then be

accessed by people to review videos they were marked as being in. A system of this type could

always ensure the timestamp would be saved with each video, the camera ID and location could

also be hardcoded into each cameras program as they are all static. This then became a system

which allowed three more dimensions to search through collections of videos.

During the implementation of this idea it became apparent there was no easy way to

spot who, if anyone, was in each video. The aim of the project was to be non intrusive, so people

who wanted to use the system could not be asked to wear more gadgets to help the system. The

only other option in this case was to implement facial recognition, but even then it could not be

Ben Taylor 33117535


assumed a person's face could be seen just because they are in the frame. The current idea then

had to change for the system to be effective.

Figure 3.1.1: Layout of components during one of the stages of development. All cameras

connected to a data store which was then accessed by a user interface to review data.

Not all the ideas from the previous idea were thrown away, there was still a system

which was able to capture and chunk video. If this system was to evolve to become more

personal, one program for each user, more ways of video capture would need to be added. The

simplest way to allow more video content to be added would be for users to add any videos they

want, which would make the reach of the system much better. Allowing this would prevent the

certainty of timestamps and location data automatically being collected, only the camera ID

could be enforced, as a way to push some organisation onto the user. This meant a new type of

data needed to be considered, one which could automatically be applied to any input video.

From the background it was learnt people remember events internally as images, within

those images people recognise objects which they are able to name. Even if an event cannot be

fully remembered it is likely people are able to remember some aspects of their internal image,

but not enough to be able to search by factual terms e.g. Time/Date or location. If less factual

content was able to be added to each video automatically which worked closer to human

memory it would improve the system. This brought about the idea if image tagging, algorithms

which are smart enough to take an image and tag the content which is in it, this was the type of

data which was being searched for, it could always be applied as all that was required was an

image. Creating an algorithm which could tag videos was outside the scope of this project, so a

3rd party solution was found instead. The found solution was tested with a few images and it

was discovered to be fairly extensive, much more than expected, this then became the basis for

much of the project.

Applying this to the project meant altering the small camera program slightly, it was

adapted to take 5 screenshots throughout each chunk of a video. These five screenshots could

then serve as a sample of images from the video and could be sued to tag the video. At the same

time another program was developed which could read in users videos and split them into

chunks and collect screenshots for each chunk. These two programs were very similar at their

cores, but the way data was accessed and saved back was slightly different. The user videos

Ben Taylor 33117535


worked from an existent video which then had to be deleted once parsed, while the camera feed

just terminated the input to end the program.

Before any video content could be tagged a way of storing data which could be accessed

by each of the components needed to be developed. Without this each component was working

separately, which just formed different parts of the system. The quickest and most efficient way

of storing all this data was through a central database, this database served as the central point

for all the different components to share their data.

With the central database implemented a small program was written which would take

all content from the database and send it to be tagged. These tags would then be pulled back

into the system and saved into the database along with the confidence interval for each tag. At

this point the system was able to collect data but had no way of sending it back to the user, the

last component of the final system was then created, the user interface. At this point all

component had been defined and the system looked like Figure 3.3.1.

Based on the end system a set of requirements were created which would be used to

evaluate the system. The way these requirements were created is unconventional, as they are

usually defined before the system is designed and are used to design the system. Nevertheless

these requirements will still functional well when being used to evaluate the success of the

system. Each of the requirements have been given an ID, these will be used so each requirement

can easily be referenced back to later in the report.. The set of defined requirements are as

follows:

The System:

Functional

• (R1.1) SHALL allow data to be added to memories

• (R1.2) SHALL allow added data to be edited at the users discretion

• (R1.3) SHALL allow any data to be deleted at the users discretion

• (R1.4) SHALL allow multiple methods to search memory data

• (R1.5) SHALL automatically organise added memories

• (R1.6) SHALL allow any images and videos to be added

Non-Functional

• (R2.1) SHALL be able to store personal memory data

• (R2.2) SHALL promote memory recall as much as possible

• (R2.3) SHALL securely store all data to protect the privacy of users

• (R2.4) SHALL allow multiple methods of memory capture

• (R2.5) SHALL have a simple and intuitive user interface

3.2. Decisions and Justifications

The first design decision to consider was the types of data which would need to be

stored. This initial decision would make a huge impact on the effectiveness of the eventual

system, this has been shown in the background where research has proven different types of

data and they way it is displayed can have a effect on how well memories are remembered. With

this information in mind, and remembering autobiographical memories are remembered

through internal images (Conway, 1990) (Pillemer, 1998), it was decided the biggest method to

capture and store should be through passive visual capture.

Ben Taylor 33117535


Passive visual capture alone would not be sufficient to help memory recall (Whittaker,

et al., 2010), and would provide little aid in terms of textual data - making retrieving these

images back much more difficult. It is clear more data capture is needed in order to produce a

much more effective system, the problem would be ensuring this extra data is always there

when new data is added, otherwise the system will end up no different to a simple media

storage, there has to be something about the system which makes it appeal more towards

memory.

Typically the only data added to images and video would be a timestamp (time and

date), while this data would add some improvement to the retrieval and recollection of

memories it would not be sufficient. Two points of data stand out more than others to add to the

visual data, location and facial recognition. Location data is an immediate choice as many people

capture image and video through smart phones, which contain GPS which allows geotagging of

that data, there is however no promise all input data be geotagged, and as shown people loathe

to organise and tag their own data with location. This is where facial recognition comes in, this

is a system which can be automatic after initial set up, teaching the system the name each face

belongs to in a few photos. Facial recognition is an advanced technology, it is used in simple

security systems such as locking PC's so getting access to a library of some sort will not be able

to difficult.

While the previously mentioned data will improve the visual data, they are lacking in

one aspect - they cannot automatically be applied to all visual data, not all images will contain

people, and not all people wish to manually tag locations. This leads on to a further technology,

which is expanded from the facial recognition idea, automatic general image tagging. A

technology such as this is by no means perfected, but that does not mean it is not useful. Current

image tagging systems are not very specific, they tend to tag general names and colours such as

building, person and outdoors. While they may not immediately appear to be helpful the people

who use the system could adapt to begin to predict what memories would be tagged as based on

what they can remember.

If all of this data is combined it provides an adequate variation, which should be capable

of aiding in memory recall from a huge data store while still retaining the ability to spark a users

memory processes to help them remember. In terms of the 5R's (Sellen & Whittaker, 2010), this

would only cover the first two, recollection and reminiscing. The other 3, retrieving, reflection,

and remembering intentions, would be much more difficult to apply to the collection of data

which has been chosen to be used. While there is a possibility of reflection e.g. graphs of

common visited locations, implementing methods to capture the data necessary for retrieval

and remembering intentions could prove to be much more difficult.

Next, the method of data capture needed to be decided. The initial decision involved

having inputs which could be easily controlled to ensure the attached time and date data was

included, this meant limiting inputs to a connected real time camera. This is not an ideal

situation, as it greatly limits the level of memory capture the system would allow, the benefit of

ensuring the timestamp data is not beneficial enough to allow this limitation. Other methods

would need to be implemented, methods which allow more effective memory capture no matter

the location. Narrative clip is such a device, and it can easily be integrated into the system, it

only produces images which is a type of data the system aims to process, and all images

captured by the device is time stamped and geotagged - meaning those requirements are also

met. The final situation to consider, to make the system as openly useful as possible, is the cases

where a person does not, or chooses not to have access to a Narrative Clip. The Narrative Clip

can be seen as a controversial device in terms of privacy so it is not unrealistic to suggest there

Ben Taylor 33117535


are plenty of people who do not agree with their use, and the system must tailor toward these

cases also. The simplest solution to this is to give the user full control over what content they

choose to capture and then pass on to the system, this is done through personal videos and

images. Allowing the incorporation of personal visual capture also allows the system to include

captured memories before the system was designed, therefore increasing its usefulness.

As the system deals with memory a method of grouping data would be useful, to

promote effective recall and prevent an overwhelming number of images being produced or

long, hard to search videos. The two methods of grouping data, as discussed in the background,

are events and typical time chunking of data. It is unclear how a system would be able to group

data into events when a video is passed into it, so to keep the implementation of the project

quick the simpler time chunking method will be used. An optimal length of time will need to be

decided, one which makes video files small enough to be easy to search, while keeping them

large enough to avoid many videos of very similar data. The final settled on length was 5

minutes, this keeps the videos small enough for easy access.

Videos will be grouped into time intervals, but this does not benefit user recollection

through images. Grouping images by time interval could be a solution, but results of this would

not be effective in cases where constant images are being taken. If the only image input was

Narrative Clip, grouping by time would be the best solution, but as personal images can also be

used this could easily result in the user seeing no change in the way the images are organised.

To overcome this issue images will be accessed by the user in a slightly different way to videos,

whenever an image is directly accessed a selection of images which have similar tags will also

be displayed off to the side. The aim of this is to help the user find the correct image, in the cases

where they are looking at a similar but not the image they want, and to help a user see images

which they may have forgotten about, which will promote memory recall.

Finally, the cases where visual data was input into the system without meta data would

need to be taken into account. As the base method of organisation for the system would be

through timestamps a way of capturing timestamps for all input data need to be thought of. The

solution to this problem attempts to avoid requiring user input, but does not prevent it as user

input of a missing timestamp would be the best solution. Whenever visual data is input to the

system it will save the current date and time and use that as the data timestamp, there will then

be the option to change it later if the user wishes to. The reason for taking the current

timestamp is based on the hope that users of the system will be uploading images on a daily

basis if they aim to use the system effectively.

3.3. Overall architecture

The final overall architecture of the system will be described in the sections below. It

will start by showing the overall architecture with a brief explanation of how the system works

together, it will then go on to explain each component of the system in more detail. The

overview architecture of the system is shown in figure 3.2.1 below.

Ben Taylor 33117535


Figure 3.3.1: Showing the overall architecture of the designed system. Dashed arrows show

control flow, normal arrows show data flow and grey boxes show external devices and systems.

The central point in the entire system will be the database, this is where all data will be

stored, and the file paths to all images and video will be stored here as well. All static camera

feeds are passed directly into the system at real time and fed straight into the Camera Feed

Parser, this component the reads and saves the videos in the correct format. All user created

content, including narrative clip content, will be input into the system via the File Uploader.

Uploaded images do not need changing, but uploaded videos are then fed into the Video File

Parser which parses the videos in a similar way to the Camera Feed Parser, this means all input

videos are in the correct format. The final component is the Tagging Manager, this component

will take all images and videos which still require tagging and pass them to an external tagging

service. This has to be an external service as implementing such a system would be time

consuming, an external system will also have much more data available to it to produce an

effective system to start with. These produced tags are then passed back into the database and

linked with the corresponding data. Finally, when the user wishes to access their data they do so

through the user interface.

To start with the Camera Feed Parser will be explained. This component is designed to

take input from any camera connected to the computer running the designed system. The

parser will accept the input and split it into the desired 5 minute chunks, which were decided on

earlier, each chunk will be given its own position in the database - creating a number of separate

videos. Each of the separate videos will also have a couple of frames taken out at equal time

intervals, making 10 images per video, these images will also be saved into the database linking

them back to the video. The Video File Parser works in much the same way with one small

difference. It will operate by looking through the database to find any videos which have not

been marked as parsed, it will then take these videos and parse them in the same way as the

camera feed.

The main operation of the tagging manager is to create a connection between the

database and the external tagging system. This component will take all images and the captured

frames from each video and pass them to the external tagging system. If this system was to

operate in the real world an external tagging system could not be used as it would violate the

users privacy, as this is a study this is not necessary. For each image which the service tags the

tags along with their confidence values are simply added to the database, while being linked

Ben Taylor 33117535


back to their parent image. The process for tagging each video is slightly more complicated, to

prevent space being wasted with redundant data, which will soon add up with a large memory

system, the videos are tagged in a slightly smarter way. Each frame which was taken from the

video will be sent to be tagged, when the tags return they are all combined together, discarding

any copies of the same tag, the highest confidence interval for each tag is kept. As each video is

only 5 minutes long it is expected that many of the same tag will be returned. Once the set of

unique tags has been collected they are returned to the database and linked back to parent

video.

Once visual data has been captured, parsed and tagged it is ready to be accessed by the

user. The aim is to have the tagging manager and both parsers running constantly when they

are needed, these two could be lengthy processes and it would be best if they could not be

blocked by the user interface. This will mean implementing all database calls effectively so it is

not locked by any one process for too long. The user interface will allow all interaction with the

captured memory data, and it will attempt to relay the data back to the user in the most

effective way possible to promote reminiscence and recollection. The user interface will include

a display of all images and video, methods of searching for specific data and a way to upload any

files into the system. The biggest aim for the user interface is to make it as simple and intuitive

as possible while still relaying all information to make it an effective system.

4. Implementation The implementation of the system involved the use of a number of libraries to produce

both the back end data manipulation and to create a simple user interface. The implementation

time was split equally between both the back end and the user interface as both had challenges

to overcome. This section aims to highlight the way the system was implemented and why it

was done in that way. Starting with the choice of libraries, the user interface and the

encountered issues will be discussed before giving a walkthrough of the system.

4.1. Language and Libraries

Multiple languages were used to create the end system, they all met a different criteria

which were needed for different components of the system. Python was chosen to write the

'Camera Feed Parser', 'Tagging Manager' and 'Video File Parser' components (See Figure 3.2.1

for more details on components), python was also used to deal with all access to the central

database and handled all the data processing for the user interface. The design of the user

interface was written in html, based on a python server. The detailed reason for the use of each

language is covered below.

When choosing which language to write the back end system in there are a few choices,

the biggest ones which came to mind are; Java, C and Python. Python was chosen as it is a

portable, easy to use language which has a huge assortment of 3rd party libraries, something

which was needed during the system implementation. Python also has very useful and clear

documentation, this makes finding the solutions to any problems much easier. Code for python

is also very simplistic, a lot can be achieved with very little code, making it a simple yet powerful

language.

Although python is a malleable language there was not a library available which could

produce a user interface with ease, this meant looking elsewhere. The user interface was

created via html, this was chosen as there is also a number of libraries available to make this a

Ben Taylor 33117535


easier problem to tackle. Using html also benefitted from being able to use a python library to as

a back end server for the user interface, and as a templating system, avoiding the use of another

language, PHP.

A number of libraries were used in the implementation of the system. While this did

speed up the implementation for the most part, it did bring its own challenges with it. Using

these libraries meant learning how to use them to start with and then understanding any issues

and error messages when things did not work as expected. The used libraries and 3rd party

content which were used are as follows; Opencv2, SQLite3, Mako and CherryPy in python and

Twitter Bootstrap to create the user interface.

Opencv2 is described as, on its website1 as "OpenCV was built to provide a common

infrastructure for computer vision applications and to accelerate the use of machine

perception". The main use of Opencv2 in the system was in the 'Camera Feed Parser' and 'Video

Feed Parser'. This library was used to parse the video feeds and files and to pull frames out of

each video when they were required, readying the files to be tagged and then displayed.

Opencv2 was also chosen as it included the ability for machine learning, specifically object and

facial recognition. If the system was to be developed further this library will allow a much easier

implementation of its own facial recognition and image tagging via object recognition.

SQLite3 is a lighter version of SQL, it was chosen in this case as it is built into python,

meaning no extra installation. SQLite3 was used to control all database access and creation, it

was used in each component of the system, as they all worked from the database. The library

was pushed to it limits within the system, if any more work was to be done the database

management would have to be upgraded to SQL. SQL can better deal with the concurrency

issues which were starting to appear as the system was developed further.

The user interface was created using a combination of CherryPy, Mako and Twitter

Bootstrap. CherryPy was used to handle all page requests and to gather data to send to the user

interface. Mako was combined with HTML to create the user interfaces, it allowed the dynamic

creation of the interface depending on the callers request. Finally Twitter Bootstrap provided all

CSS files to help create a aesthetic user interface with little effort. Twitter Bootstrap and

CherryPy were chosen as they had simple and easy to understand tutorials. Mako was also

chosen because of its simplicity, as well as being able to replace PHP. These three libraries

combined produced a user interface which was simple and easy to use.

4.2. User Interface

Creating the user interface as a web application allowed the interface to be designed

step by step, thus allowing experimentation to take place during implementation. The final

design which was settled on was heavily influenced by webpages, as they are created using

HTML. When researching webpage designs a few points stuck out which were incorporated into

the final design; keep navigation either at the top or left side of the screen, keep content on the

screen as minimal as possible and keep a consistent design throughout.

Following these points the final design kept a consistent layout, when viewing the page

for each image and video they were designed in very similar ways, the biggest difference being

the main file on the page. Each list of items, e.g. search results and listing all images, was based

on the same python and HTML code which paginated all results into a consistent style. Finally

1 http://opencv.org/about.html

Ben Taylor 33117535


all forms which the user filled in to send data back to the server, such as tags, were all contained

within a panel to make it clear which fields and buttons were related to one another.

4.3. Issues

During the implementation of the system there were a few issues which needed to be

overcome, these issues were mostly technical bugs but they were not the only problems which

used up implementation time. The most time consuming of all problems occurred because of

lack of knowledge on the language or library which was in use, the biggest of the two being the

library use. The lack of use of HTML did slow down the implementation of the user interface, but

was not a major factor as there is plenty of documentation and help online and on the Twitter

Bootstrap website.

Opencv2 did cause a few problems during implementation because of bugs. The first

problem encountered was a bug where retrieving the frames per second of a live input failed

every time, because of this the frames per second has to be hardcoded into the program. This

will cause some issues as the feed coming in could be saved at a different frame rate as it's input

causing faster or slower feedback from the system. The live input has not been tested on a

variety of cameras so it is unclear how prevalent this error is.

There were also issues with reading in video from file to parse it. When capturing each

frame of a video from file Opencv2 returns a flag, true if a frame was retrieved and false if no

frame was. Building on this the system was implemented to accept an empty frame as the end of

the video file. When testing the system with input from a video file errors were returned and the

operation never completed. On further investigation it appeared the capture flag was always set

to true, so the program was attempting to capture frames which were not there. To solve this

issue the number of frames of the current video were loaded up and the program was stopped

just before it reached the end to prevent this error. This issue was never discovered when

testing with a live camera feed because the program was always terminated before the camera.

It isn't clear how this will affect the live camera feed. The camera feed is saved as it is read in, it

is assumed any errors that occur because of a camera being disconnected will just throw and

error and end the program, this is not the ideal solution and will need improving.

The final error which was encountered while using Opencv2 was not directly because of

the library but one of its dependencies. The image and video reading and saving depended on a

collection of codecs to encode and decode the data in the correct formats. The way the library is

installed when the .exe file is downloaded does not place codec file in a place it can be found by

the library. To solve this the file had to be moved and renamed so it could be found, this did not

allow the use of all codecs, some formats such as .mp4 were still unavailable when trying to use.

Other codec files were downloaded to attempt to solve this issue but the new files did not solve

the issue either. Further investigation is needed to solve this issue.

There were a couple of issues while using SQLite3 involving concurrency, this is the

reason an upgrade to SQL could be a better option should the system get any further

improvements. As the parsing and tagging of videos and images can take a lengthy amount of

time these processes were set to constantly run in the background. As these processes spent a

lot of time writing to the database they could lock it and prevent all other processes accessing

any data, which would then cause errors. This issue was most prevalent while the tagging

system was running, it would lock the database while waiting for a response from the tagging

service. While this choice of implementation was not ideal it was necessary to implement each

tag as an atomic operation to prevent any errors occurring. The first solution was to extend the

Ben Taylor 33117535


waiting time of any database accesses to around 30 seconds, this was plenty of time for the

locking process to release the desired resources. Further investigation revealed a better

solution, to instruct the database to use a write ahead log, this prevented any errors but it is

likely it slows down the data processing.

4.4. The System in operation

To give a feel for the user interface this section of the report will give a quick

walkthrough of a typical use of the system. The system will begin from an already used state to

give a better feel of how different types of data are used within the system while keeping

continuity in the walkthrough. A short description of each part of the user interface will be

accompanied by a screenshot of the page in the current state.

The walkthrough will step through a series of typical uses of the system. Typical uses of

the system include uploading more content into the system, searching for content through

various methods and updating the meta data which is saved with each image or video within the

system. Each of these typical uses will be shown in turn and a small explanation given along

with each page to give a better idea of what has been done at each stage, and why the page is

displaying the content it is.

Figure 4.4.1: Home page of the application

When the system is started up the user is first directed to the homepage(see Figure

4.4.1). This page is designed as a neutral page away from all content, marking it clearly as the

first page in the system. If more time could be spent on development this page would be

improved to become more welcoming and explain the system better. As it stands at the current

time the page points the user towards the navigation bar at the top of the page.

Ben Taylor 33117535


Figure 4.4.2: The initial state of the video page - before any video content has been added.

The video page (See Figure 4.4.2) currently shows the appropriate error message to

inform the user no content has been found yet. This could be for two reasons, either the user has

not input any video content yet as the message suggests or the content they have added is still

being processed and so is not ready to be displayed. A similar page is displayed for the images

when they are in a similar state.

Figure 4.4.3: The top half of the images page with uploaded content.

As there is currently image content uploaded to the server this content is all listed and

paginated to allow much easier access (See Figure 4.4.3 and 4.4.4). Each image can be clicked on

to display more information and to allow the user to edit data, this will be shown later. This

page was designed to give the user as much information as possible about the system state, so

the current number of images out of how many is shown. The video page also has a similar

layout when there is content uploaded, this will be shown later.

Ben Taylor 33117535


Figure 4.4.4: The bottom half of the images page with uploaded content.

Figure 4.4.5: The top half of an information page for a specific image, this is reached by clicking

on the image while it is shown anywhere in the user interface.

The information page can be seen in Figure 4.4.5 and 4.4.6, this page shows the image

and the tags which have been assigned to it. Displaying the tags gives the user a feel for how the

system is able to tag their photos and images, the aim is to help the user realise how the system

works so they can better devise their searches if they are able to. Further down the page the

option is given for the user to update metadata associated with the selected image, as an

example the tag 'Sleeping Hamster' will be added to show has the system updates the tags, the

updated page is shown in Figure 4.4.7. Adding more tags to an image is not intended for use on

all content within the system, it is there so the user can add any tags they think are missing on

their favourite content, making it easier for them to find.

Ben Taylor 33117535


Figure 4.4.6: The bottom half of an information page for a specific image, for more details see

Figure 4.4.5 description.

Figure 4.4.7: Shows how the image information page is updated once the user submits the form to

add their own tag(s), to see the page before form submission see Figure 4.4.5.

The upload page (See Figure 4.4.8) is used so the user can upload any of their own

content such as personal videos or images, images gathered from a Narrative Clip or similar

device or any other device which captures image or video content. This page only allows upload

of a single image of video at a time, which could be an issue if a user was to use a Narrative Clip.

To upload content the user simply enters the ID of the camera the content came from then

presses 'Choose File' which will bring up a file select option, this is shown in Figure 4.4.9. Once

the file has been selected the user presses the 'Upload Video' button to complete the upload. No

upload progress bar is shown directly, progress is only shown by the browser if it is supported.

Ben Taylor 33117535


Figure 4.4.8: Showing the upload page before any forms have been filled out.

Figure 4.4.9: The upload page as the user is uploading content. When 'Choose File' is pressed the

overlaying file select window is opened to allow the user to select the file they wish to upload. The

user interface is then updated as shown here.

Once content is uploaded into the system the background processes will pick it up and

process it into data ready to be displayed by the system. This means parsing any videos which

are too long, and automatically tagging all images and videos. These processes will be described

in more detail later, but they are carried out between the above upload and the displaying of the

uploaded content later.

Ben Taylor 33117535


Figure 4.4.10: This shows the top half of the video page once content has been uploaded. An

example of the page without any uploaded content can be seen in Figure 4.4.2.

Figure 4.4.11: This shows the bottom half of the video page once content has been uploaded.

After uploaded data has been added to the system and processed it is ready to be

displayed, this can be seen in Figure 4.4.10 and 4.4.11. As the figures show the video has been

split into 4 smaller chunks as was defined earlier, this is seen through the video

'Creation/Upload Time'. All the videos have the same value for this field - the time they were

uploaded at by default. These figures also show an example of how the system deals with

situations where there is not enough images to fill up all five slots on the page. The system will

display no images and mark all fields as N/A to make a clear distinction between an image not

found error and purposefully not displaying an image.

Ben Taylor 33117535


Figure 4.4.12: This shows the top half of the video information page. This page is accessed by

clicking on the respective image anywhere the video is listed

Figure 4.4.13: This shows the bottom half of the video information page.

As can be seen in Figure 4.4.12 and 4.4.13 the video information page is very similar to

the image information page, this is to keep the layout as consistent as possible. Similar to before,

when there is too few images to fill the similar videos box no image is displayed, this is not as

clear as before and it can just appear as there has been an error in the retrieval of the image. If

the user wishes to set the location of this video that is possible through the on screen form. The

updated page is shown in Figure 4.4.14, once the data is entered and 'Submit' is pressed the

page will update the data and reload instantly to show the change.

Ben Taylor 33117535


Figure 4.4.14: Showing the video information page once the Location data has been updated.

The final section of the user interface to be demonstrated is the search options. To allow

the searches to produce results capable of effectively demonstrating the system some of the

metadata has been updated. Some videos and images have had the tag 'Favourite' added to

them, just one example of how the user can use their own tags. Location data has been added to

some of the content. An example of a search is shown in Figure 4.4.15, and all the search options

in Figure 4.4.16. Each search allows the user to choose which content to search, videos or

images or both. To demonstrate the search the tag Favourite will be search for in images and

videos.

Figure 4.4.15: The top half of the search page, the user is about to search for a specific tag in

images and videos.

Ben Taylor 33117535


Figure 4.4.16: The bottom half of the search page. This shows all the options available to the user

to search the data.

Figure 4.4.17: The top half of the results page from the search displayed in Figure 4.4.15

The results from the search can be seen in Figure 4.4.17 and 4.4.18, to keep consistency

these are laid out in the same way as the video and image pages. In these two figures it can be

seen how the results are ordered chronologically, from most recent at the front to least recent.

These pages also allow the user to click on any image or video screenshot to get to the

corresponding information page.

Ben Taylor 33117535


Figure 4.4.18: The bottom half of the results page from the search displayed in Figure 4.4.15

4.5. Background Processes

During section 4.4 the background processes were mentioned. These processes take all

the raw input data and change them into a format which will be effective and will work with the

user interface. There is a window which the background processes are ran in, this produces

output to try and keep the user informed of the system state. The background processes in this

case will be 'Video Parser' and 'Tagging Manager', for more details on these components see

Figure 3.2.1.

Figure 4.5.1: The window showing the state of the background processes. This is the output which

is produced while the database is being polled to check for more data

When there is no data to be processed the program will poll the database to check if any

new content has appeared. By doing this any new content will be caught as soon as it is

Ben Taylor 33117535


uploaded and the database is updated to allow access to the data. Figure 4.5.1 shows an example

of the output which is produced during this time.

Once content is uploaded it is picked up by these processes, if video content is uploaded

it is first picked up by the 'Video Parser' before it can be tagged. Once being parsed the video is

able to be picked up by the tagger, along with any images which still need processing. Then

output produced one a video has been uploaded can be seen in Figure 4.5.2 and 4.5.3.

Figure 4.5.2: The output produced while the video parser is running.

Figure 4.5.3: The output produced while the tagging manager is running.

5. Testing This section aims to extensively test the functionality of the user interface and

background processes. As the system was being developed many tests were carried out to

ensure it was mostly functional, these tests are considered as white box tests. The white box

testing may not be as extensive as first thought, many assumptions could have been made

Ben Taylor 33117535


because detailed knowledge of the system was known. The following tests will aim to carry out

black box testing, this is not officially black box testing as knowledge of the inner workings of

the system are known. During testing however no assumptions will be made and everything will

be tested if there is a need to test it, the aim is to catch as many bugs in the system as possible.

5.1. User Interface Testing

To ensure the user interface is functioning as expected it was extensively tested. The

navigation and all input option were tested to try and reveal as many problems as possible. The

input testing covered situation where input should be accepted and tested cases where data

should be rejected to ensure bad data is not accepted into the system. Testing the navigation

was more complex, as the user interface is a web application this opened up navigation by a URL

as well, allowing more cases for errors.

First to be tested will be the overall navigation of the pages, this navigation is the same

on every page so once it is tested once it should not need testing on each page. Following this

the testing will move forward to test each page individually ensuring they all function correctly

and attempt to prevent wrong input where possible. Furthermore the expected functionally of

each form which can be submitted will be tested.

The format of each of the tests will be consistent, to keep them easy to read and

understand. Each table of test will have a small title describing which page or section it is

testing, a requirements ID will be added to this title for the requirements they relate to. First

each test case will have a short description describing what is being tested, followed by the

input used to carry out the test. Next the expected outcome will be noted followed by the actual

outcome, finally the decision on the passing of the test will be made and any necessary

comments will be added.

Navigation Testing

Test

ID

Description Input Expected

Output

Actual

Output

Pass/Fail

(Comments)

1. The user

attempts to

navigate to

the Videos

page

Button Press: Videos

(On navigation Bar)

The user is

directed to

the Videos

page

The user is

directed to

the Videos

page

Pass

2. The user

attempts to

navigate to

the Images

page


(On navigation Bar)

The user is

directed to

the Images

page

The user is

directed to

the Images

page

Pass

3. The user

attempts to

navigate to

the Search

page


(On navigation Bar)

The user is

directed to

the Search

page

The user is

directed to

the Search

page

Pass

4. The user

attempts to

navigate to


(On navigation Bar)

The user is

directed to

the Upload

The user is

directed to

the Upload

Pass

Ben Taylor 33117535


the Upload

page

page page

5.1. The user

attempts to

navigate to

the Home

page


(On navigation Bar)

The user is

directed to

the Home

page

The user is

directed to

the Home

page

Pass

5.2. Button Press: Videos

(On navigation Bar)

The user is

directed to

the Home

page

The user is

directed to

the Home

page

Pass

6.1. The user

attempts to

navigate to a

non-existent

page using

the URL bar

URL Bar: '

http://127.0.0.1:8080

/films'

A 404 error

page should

be shown.

A page saying

"The page

does not

exist" is

shown

Pass - A 404 error is

not directly needed

as an appropriate

error message is

shown.

6.2. URL Bar: '

http://127.0.0.1:8080

/films?ID=film5'

A 404 error

page should

be shown.

A 404 error

page is shown

Pass - The error page

does reveal some

code to the user

which should be

corrected.

Video List Testing

Test

ID


Output

Actual

Output

Pass/Fail

(Comments)

1. The user

attempts to

view all

uploaded

videos

Button Press: Videos The Videos

page will be

displayed and

a list of all

videos will be

shown

The videos

page is shown

and all videos

are paginated

and displayed

Pass - the results

have been paginated

to allow easier

searching

2.1. The user

attempts to

navigate to a

different

page in the

list

Button Press: 2 The second

page of

results will be

shown

The second

page of

results was

shown

Pass

2.2. From page 3 -

Button Press: Previous

The second

page of

results will be

shown

The second

page of

results was

shown

Pass

2.3. From page 1 -

Button Press: Next

The second

page of

results will be

shown

The second

page of

results was

shown

Pass

3. The user

attempts to

navigate to

the specific

page for a

video

Button Press:

Thumbnail of desired

video

The desired

video page

will be shown

The desired

video page

was shown

Pass - It may not be

entirely clear a user

is able to do this from

the layout of the page

4. The user

attempts to

URL bar: '

http://127.0.0.1:8080

A 404 error

page should

A page saying

"The page


not directly needed

Ben Taylor 33117535


navigate to a

non-existent

page

number

through URL

arguments

/video?page=115 ' be shown. does not

exist" is

shown

as an appropriate

error message is

shown.

Video Information Page (R1.1, R1.2, R1.3)

Test

ID


Output

Actual

Output

Pass/Fail

(Comments)

1.1. The user

attempts to

view the

information

page for a

video

The thumbnail for the

video is clicked on

from any display of it

The

information

page for the

video is

displayed

The

information

page for the

video was

displayed

Pass - There was a

error with the video

showing but all other

content on the page

works

1.2. The Id of a known

video is input into the

url bar as an argument

URL bar: '

http://127.0.0.1:8080

/videoShow?videoId=

GoKart%20GoPro_

2015-03-14%

2016.18.08_NKC%

202015%20Pre%

20Season%20Rac

e%20At%20Sutton%

20Elms.mp4.mp4_4'

The

information

page for the

video is

displayed

The

information

page for the

video was

displayed

Pass - There was a

error with the video

showing but all other

content on the page

works

1.3. The video of a non-

existent ID if input as

an argument

URL bar: '

http://127.0.0.1:8080

/videoShow?videoId=

Video5

A 404 error

page should

be shown.

An error page

was shown

with the

following

message ' No

video with

that ID exists '

Pass - The error

message is more

specific than

expected. Helping the

user know what is

wrong

2. The user

attempts to

navigate to a

similar

video

through the

similar

video side

bar

Button Press: The

thumbnail of the

desired video is clicked

The

information

page for the

desired video

is shown

The

information

page for the

desired video

was shown

Pass

3.1. The user

attempts to

add more

tags to the

video

Add a single tag

Tags: 'Driving'

Button Press: Submit

The tag will

be added and

the page will

be reloaded

to show the

user

The tag was

added and the

user page was

updated to

show the new

tag

Pass

3.2. Add Multiple tags The tag will The tag was Pass

Ben Taylor 33117535


Tags: 'Daytime;

favourite'

Button Press: 'Submit'

be added and

the page will

be reloaded

to show the

user

added and the

user page was

updated to

show the new

tag

3.3. Add multiple tags with

incorrect separator

Tags: 'Daytime,

favourite'


The user will

be informed

of the error

and be told

how to

correct the

mistake

The page is

updated and

the tags were

added

Failed - The user was

able to add the tags

incorrectly. This

appears as if it was

done correctly which

could cause further

problems

4.1. The user

attempts to

update the

date value of

the current

video

Update the date with a

correct value

New Date:

'15/12/2014'

The date will

be updated

and the page

will be

reloaded to

show the user

The page was

updated and

the new date

was added

Pass

4.2. Update with and

incorrect value

New Date:

'5481/1541258/566'


The page will

inform the

user the input

is wrong and

help them

correct it

The page did

not allow

input like this.

Pass - As incorrect

date is not allowed

the user does not

need to be told how

to correct the error

4.3. Update with a date

with a long year

New Date:

'12/02/458965'


The page will

inform the

user the input

is wrong and

help them

correct it

The page was

updated and

the new date

was displayed

Failed - The page

should not allow

digits longer than the

lengths it relies on.

The formation of

separating date and

time is affected when

displaying.

5. The user

attempts to

update the

location

value of the

current

video

New Location: 'Sutton

Elms'


The date will

be updated

and the page

will be

reloaded to

show the user

The page was

updated and

the new

location was

displayed

Pass - No error

checking can be

carried out. This field

is designed to accept

any string input.

6. The user

attempts to

delete the

current

video

While on the

information page for

the image -

Button Press: Delete

Video

The user will

be returned

to the video

list and be

informed the

video has

been

removed

The user is

returned to

the video

listing page

Pass - No

confirmation

message was shown

but the video is no

longer in the listing.

Ben Taylor 33117535


Image List Testing

Test

ID


Output

Actual

Output

Pass/Fail

(Comments)

1. The user

attempts to

view all

uploaded

images

Button Press: Images The Images

page will be

displayed and

a list of all

videos will be

shown

The Images

page is shown

and all videos

are paginated

and displayed

Pass - the results

have been paginated

to allow easier

searching

2.1. The user

attempts to

navigate to a

different

page in the

list

Button Press: 2 The second

page of

results will be

shown

The second

page of

results was

shown

Pass

2.2. From page 3 -

Button Press: Previous

The second

page of

results will be

shown

The second

page of

results was

shown

Pass

2.3. From page 1 -

Button Press: Next

The second

page of

results will be

shown

The second

page of

results was

shown

Pass

3. The user

attempts to

navigate to

the specific

page for an

image

Button Press:

Thumbnail of desired

image

The desired

image page

will be shown

The desired

image page

was shown

Pass - It may not be

entirely clear a user

is able to do this from

the layout of the page

4. The user

attempts to

navigate to a

non-existent

page

number

through URL

arguments

URL bar: '

http://127.0.0.1:8080

/images?page=115 '

A 404 error

page should

be shown.

A page saying

"The page

does not

exist" is

shown


not directly needed

as an appropriate

error message is

shown.

Image Information Page (R1.1, R1.2, R1.3)

Test

ID


Output

Actual

Output

Pass/Fail

(Comments)

1.1. The user

attempts to

view the

information

page for a

image

The thumbnail for the

image is clicked on

from any display of it

The

information

page for the

image is

displayed

The

information

image for the

video was

displayed

Pass

1.2. The Id of a known

image is input into the

url bar as an argument

URL bar: '

http://127.0.0.1:8080

The

information

page for the

image is

displayed

The

information

page for the

image was

displayed

Pass

Ben Taylor 33117535


/imageShow?imageId=

Narrative_2015-03-

08%2019.24.31_IMG_

20150305_164653.jpg

1.3. The video of a non-

existent ID if input as

an argument

URL bar: '

http://127.0.0.1:8080

/imageShow?imageId=

image5

A 404 error

page should

be shown.

An error page

was shown

with the

following

message ' No

image with

that ID exists '

Pass - The error

message is more

specific than

expected. Helping the

user know what is

wrong

2. The user

attempts to

navigate to a

similar

image

through the

similar

image side

bar

Button Press: The

thumbnail of the

desired image is

clicked

The

information

page for the

desired image

is shown

The

information

page for the

desired image

was shown

Pass

3.1. The user

attempts to

add more

tags to the

image

Add a single tag

Tags: 'Building'


The tag will

be added and

the page will

be reloaded

to show the

user

The tag was

added and the

user page was

updated to

show the new

tag

Pass

3.2. Add Multiple tags

Tags: 'Daytime;

favourite'


The tag will

be added and

the page will

be reloaded

to show the

user

The tag was

added and the

user page was

updated to

show the new

tag

Pass

3.3. Add multiple tags with

incorrect separator

Tags: 'Daytime,

favourite'


The user will

be informed

of the error

and be told

how to

correct the

mistake

The page is

updated and

the tags were

added

Failed - The user was

able to add the tags

incorrectly. This

appears as if it was

done correctly which

could cause further

problems

4.1. The user

attempts to

update the

date value of

the current

image

Update the date with a

correct value

New Date:

'15/12/2014'

The date will

be updated

and the page

will be

reloaded to

show the user

The page was

updated and

the new date

was added

Pass

4.2. Update with and

incorrect value

New Date:

'5481/1541258/566'


The page will

inform the

user the input

is wrong and

help them

correct it

The page did

not allow

input like this.

Pass - As incorrect

date is not allowed

the user does not

need to be told how

to correct the error

Ben Taylor 33117535


4.3. Update with a date

with a long year

New Date:

'12/02/458965'


The page will

inform the

user the input

is wrong and

help them

correct it

The page was

updated and

the new date

was displayed

Failed - The page

should not allow

digits longer than the

lengths it relies on.

The formation of

separating date and

time is affected when

displaying.

5. The user

attempts to

update the

location

value of the

current

video

New Location: 'Sutton

Elms'


The date will

be updated

and the page

will be

reloaded to

show the user

The page was

updated and

the new

location was

displayed

Pass - No error

checking can be

carried out. This field

is designed to accept

any string input.

6. The user

attempts to

delete the

current

image

The user is on the

information page for

the image

Button Press: Delete

Image

The user will

be returned

to the image

list and be

informed the

image has

been

removed

The user is

returned to

the image

listing page

Pass - No

confirmation

message was shown

but the image is no

longer in the listing.

Search Page Testing(R1.4)

Test

ID


Output

Actual

Output

Pass/Fail

(Comments)

1.1. The user

attempts to

search for

their

content with

the tag of

'Favourite'

Radio Button: 'Videos'

Tags: 'Favourite'


All videos

tagged as

favourite will

be shown

All video

results were

shown and

paginated

Pass - The results

were organised from

most recent to least

recent (This will be

the case with all

searches which pass)

1.2. Radio Button: 'Images'

Tags: 'Favourite'


All Images

tagged as

favourite will

be shown

All image

results were

shown and

paginated

Pass

1.3. Radio Button: 'Both'

Tags: 'Favourite'


All images

and videos

tagged as

favourite will

be shown

Combination

of results

from test 1.1.

and 1.2. were

shown and

paginated

Pass

2. The user

attempts to

search for a

tag which

doesn't exist

Radio Button: 'Both'

Tags: 'Asdfagfef'


No results

will be shown

and the user

will be

informed

there were no

An error page

appeared

saying 'No

results found

for your

search'

Pass

Ben Taylor 33117535


results

3. The user

attempts to

search for

multiple

tags at once


Tags: 'Favourite;

university'


All images

and video

containing

any of these

tags will be

shown

All results

from test 1.3.

were shown

with the

added

content which

contained the

extra tag

Pass - All radio

buttons were already

tested in this form

4.1. The user

attempts to

search for

content

which has a

date set

between

two chosen

dates

Radio Button: 'Videos'

Start Date:

'10/03/2015'

End date:

'13/03/2015'


All videos

with a date

set between

the given

values will be

shown

All the videos

with a date

between

those values

were shown

Pass


Start Date:

'10/03/2015'

End date:

'13/03/2015'


All images

with a date

set between

the given

values will be

shown

All the images

with a date

between

those values

were shown

Pass


Start Date:

'10/03/2015'

End date:

'13/03/2015'


The

combination

of results

from tests 3.1.

and 3.2. will

be shown

The

combination

of results

from tests 3.1.

and 3.2. was

shown

Pass

5.1. The user

attempts to

search for

dates where

there are no

results


Start Date:

'10/03/1000'

End date:

'13/03/1000'


No results

will be shown

and the user

will be

informed

there were no

results for the

search

An error page

was shown

displaying the

message

'There were

no results for

your search'

Pass


Start Date:

'10/03/2015'

End date:

'13/03/1000'


No results

will be shown

and the user

will be

informed

there were no

results for the

search

An error page

was shown

displaying the

message

'There were

no results for

your search'

Pass - These dates

have no overlapping

point. The user

interface should

prevent this or

inform the user what

has happened.

6.1. The user

attempts to

search for

content

marked at a

specific

location

Radio Button: 'Video'

Location: 'Sutton Elms'


All videos

marked with

a location of

Sutton Elms

will be shown

All videos

with the

location of

Sutton Elms

were shown

Pass




All images

marked with

a location of

Sutton Elms

will be shown

All images

with the

location of

Sutton Elms

were shown

Pass

Ben Taylor 33117535





A

combination

of results

from test 6.1.

and 6.2. will

be shown

A

combination

of results

from test 6.1.

and 6.2. was

shown

Pass

7. The user

attempts to

search for

content with

a location

that does

not exist


Location: 'sdfasdfasdf'


No results

will be shown

and an error

message will

be displayed

An error page

was displayed

with the

message 'No

results found

for your

search'

Pass

8.1. The user

attempts to

search for

content

from a

specific

camera

Radio Button: 'Video'

Camera ID: 'Phone'


All videos

from the

camera

'Phone' will

be shown

All videos

from the

camera

'phone' were

shown

Pass


Camera ID: 'Phone'


All images

from the

camera

'Phone' will

be shown

All images

from the

camera

'Phone' were

shown

Pass


Camera ID: 'Phone'


A

combination

of results

from tests 8.1.

and 8.2. will

be shown

A

combination

of results

from tests 8.1.

and 8.2. were

shown

Pass

9. The user

attempts to

search for a

camera ID

which does

not exist


Camera ID: 'asdafad'


No results

will be shown

and the user

will be

informed

there were no

results

An error page

was displayed

with the

message 'No

results found

for your

search'

Pass

Upload Page Testing(R1.6)

Test

ID


Output

Actual

Output

Pass/Fail

(Comments)

1. The user

attempts to

upload an

image file

Camera ID: 'Phone'

Image to Upload: (user

selects image from file)

Button Press: Upload

Image

The image

will be

uploaded and

the user will

be informed

The image

was uploaded

and the page

reloaded

Pass - There was no

message to say the

upload was

successful

2. The user

attempts to

upload a

non-image

file in the

Camera ID: 'Phone'


selects non - image

from file)


The user will

be told the file

they selected

was not an

image

The file select

restricted the

user to only

choosing

images to

Pass - No error

message is needed as

they error is not

possible

Ben Taylor 33117535


image

upload

Image upload

3. The user

attempts to

upload a

video file

Camera ID: 'GoPro'


selects video from file)


Video

The video will

be uploaded

and the user

will be

informed

The video

was uploaded

and the page

was reloaded.

The only

progress bar

shown was

provided by

the browser.

Pass - When

navigating away from

the page the user is

warned the uploaded

is still in progress.

Also there was no

message stating the

upload was

successful.

4. The user

attempts to

upload a

non-video

file in the

video

upload

Camera ID: 'GoPro'


selects a non-video

from file)


Video

The user will

be told the file

they selected

was not a

video

The file select

restricted the

user to only

choosing a

video

Pass - No error

message is needed as

they error is not

possible

5. The user

will attempt

to upload a

very large

file (1GB)

Camera ID: 'GoPro'


selects a large video

from file (1GB))


Video

The video will

be uploaded

and the user

will be

informed

The browser

displays a

web page

saying 'The

page is not

available - the

connection

was

interrupted'

Failed - The system

should allow large

files to be uploaded -

and if there is a limit

on file size a better

error message needs

to be shown. Current

error is not clear.

Overall the testing of the user interface was a success, this shows the white box testing

did manage to remove most of the bugs during implementation. There are a few tests which the

interface failed on, these were mostly minor issues but a couple of the issues were more

important. The minor issues involved not informing the user what was happening with the

system most of the time, this could prove to be more of an issue when the interface is evaluated

later as new users will not be as knowledgeable about the system as its creator.

Allowing the system to accept incorrect separators when adding tags to any content can

be a bigger issue than it appears. If the user was to add a list of tags separated by commas to the

list they would appear to be added separately when displayed, the system would store this as

one long string as it can only separate data by semi colon. When the user attempts to search for

the content using just one of the added tags they will get no results, as the tags do not exist

alone, and they will not understand why.

The final problem uncovered by the system is the issue with uploading content. The

process of uploading the image and video files was created to be simple, but the time taken for

uploads was not taken into account. This means the user is not kept updated about the progress

of the upload and then no confirmation once it is complete. Furthermore if a file is selected to be

uploaded to the system and it is too big for the system to accept the application simply crashes

and does not inform the user why it has crashed. This could further confuse the user as no

explanation is made to the user.

Ben Taylor 33117535


5.2. Background Process Testing

To ensure the background processes of the application were functioning correctly, as

with the user interface, they were all tested while being implemented. The testing of these

processes are not as extensive as the user interface testing as there is very little input and

output produced. Testing was mainly needed to ensure all parts of the processes functioned

properly alone, and then together once the entire system was put together. To test the 'Video

File Parser' and 'Tagging Manager' different file types and different sources of data were used.

Testing the 'Camera Feed Parser' proved to be more difficult as there were not many sources of

live input available to test.

First the 'Video File Parser' and 'Tagging Manager' were tested. To test these

components the most common file formats for each file type was used, the following sources

were used to define the most common file types: Image file types (Fulton, 2015), video file types

(Cassidy, 2015). First the Image file formats were tested, to do this one image was taken and

converted into four different file formats; .jpg, .tif, .png and .gif. These four images were then all

uploaded into the system and processed. All file formats were accepted besides the .tif

extensions, these file formats are not accepted by the server which automatically tags all the

images. The tagging system is implemented to delete any images and videos which cannot be

sent to be tagged as this is the basis for the entire system.

Next the video file formats were tested, four of the most common video file types were

chosen from the source (Cassidy, 2015). The chosen types to test the system were; .wmv, .mp4,

.avi and .mpeg. Once again, similar to the image testing, one video was chosen and this was

converted into all four file formats before being uploaded to the system and processed. All of

these file formats successfully uploaded to the system and were able to be processed and

tagged. The 'Video File Parser' is set to read in videos and output .jpg image files so if the videos

were going to cause any errors it would be when being read in and parsed. The 'Video File

Parser' is set to delete any video files it cannot read in so it does not cause any problems.

The 'Camera Feed Parser' was next to be tested. The best way to test this would be to

take multiple forms of live video input and feed it into the system, however this was not

possible. There was not multiple forms of live input available to be tested. This component is

implemented in a very similar way to the 'Video File Parser' so not many errors are expected to

be found, as no errors were found with that component. The 'Camera Feed Parser' has full

control of the image and video formats which it produces and the user has no control so no

issues with file formats are likely to occur.

Overall the background processes of the system preformed well and were able to cope

with most of the popular file formats. The only file format which could not be handled by the

system was because of third party software which is out of the control of this project. To solve

this the system would need to be developed further and have its own specialised tagging system

developed which can be better tailored towards its purpose. The system deals with any cases of

unsupported file formats by simply removing the files from the system, while this is not ideal, as

it does not inform the user at all, it is the best solution to quickly overcome the problem.

6. Evaluation To know how effective the system functions it needed to be evaluated by people outside

of the implementation. The usefulness of the user interface, the accuracy of the automated

tagging and the usefulness of the tags will all be taken into account during evaluation. Overall a

better evaluation could be produced if there was more time to conduct lengthy studies into the

Ben Taylor 33117535


use of each section of the system. This section of the report will be split into evaluating the user

interface, it will then go on to evaluate the tagging system and how useful it is in terms of aiding

memory recall.

6.1. User Interface Evaluation

As there was not enough time to conduct a full user study which would best relate to the

aims of the project, a full user study was not carried out. The best study to carry out would be a

lengthy study which allowed a number of people to use the system, during this study people

would capture, upload and review their memories and then give feedback on these areas. As this

was not possible a small study was carried out to get feedback on general improvements which

could be made. Expected responses include where the user interface is good and clear, also

points where the interface is not so clear and the user is not aware or are unsure of what the

system is doing. This section will be split into three parts, first how the evaluation took place

will be described, followed by the feedback which was given, from the feedback improvements

which could be made will be discussed.

The evaluation of the user interface was carried out after the testing of the user interface

which is earlier on this report, this gave a good direction to aim the evaluation to get some

feedback and suggestions. The study was carried out by asking the participants to navigate to

specific pages within the application, with an emphasis on the aspects which are known to need

improvements. As this evaluation did not involve the effectiveness of the search it did not

matter which content the participant tried to view. The participants were told as little about the

application functionality as possible to attempt to simulate the use of a real user.

To begin each participant was presented with the application on full screen, in the

background all the background processes were running. From here they were asked to navigate

to pages or attempt to achieve specific tasks, these were asked in a random order to prevent any

one action influencing the results of another. Participants were asked to navigate to each page,

the ones on the navigation bar and ones slightly harder to find. Tasks involved uploading and

image and a video, searching with specific terms, updating metadata and deleting content.

The results for navigation were mostly positive, this is likely to be because most of the

navigation choices are at the top of the page on a bar. Navigation to the information pages of

each image or video was a small issue as expected, the participant would navigate to the list of

content but be unsure how to progress from there. After some time the participants would try

clicking on the image in the list, this indicates this navigation option is not clear and the only

reason the participants were able to navigate there is because they were told that page exists.

When participants were asked to edit meta data or delete content they struggled with this task

if they had not previously been asked to navigate to an information page. The other problems

which were revealed were with the upload page as expected, for this page participants were

asked to upload an image, a video. When uploading the image the upload progress percentage

would appear, the participant would patiently wait for this to reach 100% and assume the

upload was complete. In the case of an image this was not a problem, once uploaded the content

is quickly managed and the user is redirected. When the video was uploaded however there

were more issues, the participant would notice the percentage increasing and wait, but assume

the upload was complete and they would navigate away from the page. Doing this means the

video is not processed properly and is not uploaded into the system.

From the results a few improvements can be made, some come from suggestions from

the evaluation participants. The first and most general improvements to be made include

Ben Taylor 33117535


creating more confirmation or not complete messages. This did not seem to be such of an issue

when updating meta data as the page changed instantly, this was more of an issue with

uploading and deleting content. Instead of simply redirecting the user to a different page they

should also be informed their content was deleted successfully. Furthermore the participants

said a good improvement would be including the progress of the entire upload process, not just

sending the data to the server. One suggestion which would be the best idea for uploading

would be for the system to send the upload data to the server and then allow the server to

process the data without the user waiting, allowing the user to carry on while the data is being

loaded into the system. With regards to navigation it was suggested a button would be added to

each image or video the lists, this button would say 'More Info' or something similar to help

better show the more info page is a possibility. This button would not replace the link within the

image, as some participants felt this was a good idea, it allowed a much bigger clicking option

should it be needed.

6.2. Survey Justifications

The ideal user study to use on the system would have been a lengthy study based on

user personal memories and then getting feedback on their opinions on the benefits and

drawbacks of the system. As there was a time constraint on this project this was not an option,

to resolve this issue a small survey was created, an example of this survey has been placed into

the appendix(Section 9.1.). This survey was designed to analyse how well the tagging system

was able to match the tags people would choose and the types of data people found useful when

relating the tags and data back to memory.

The survey was split into 3 main sections, hence the choice to have three pages. The first

page was based on how a user would manually tag different images and video if they had to do it

themselves, the second page aimed to find which tags a user would find most useful when they

were then to try and retrieve images and videos back. The third page was to find out more about

the users preferences if they were to try and remember an event, such as what data they would

prefer and which meta data they think would be most helpful.

Choosing the content for the survey was an important task, the aim was to try and relate

the survey to the uses of the produced system as much as possible. This still meant the content

in the survey could not be personal to any single person, which limited the usefulness of the

survey towards the actual system use. The four main expected content uses of the system would

be: uploading personal pictures of friends and family etc, uploading general images e.g. holiday

pictures of landmarks or scenery, uploading personal videos e.g. home movies and point of view

videos and feeding in video from a static camera e.g. someone setting up a camera in their home.

To give the survey better results similar content was used in pages 1 and 2, by doing this it was

revealed the different tags people expect based on what they are asked to do. Out of the four

main expected content uses only three were included, the static camera feeds were taken out as

the majority of these videos would be the same room with not much happening - so the

expected tags would be similar to images.

Based on the above knowledge the three types of content included in the survey were:

personal images, environment images and personal videos. As each of these types can take on

many forms it was important to choose content which were similar but were not the same to

see how the tags changed across scenarios. For the personal images two different images of

animals were chosen, this was to keep the images as neutral as possible, if people were included

and were recognised it could influence the results. Next the environment images needed to be

Ben Taylor 33117535


chosen, the two chosen images were InfoLab21 at Lancaster University and Lancaster Castle.

These two subjects were chosen as they were easily accessible and easy to recognise if a person

knew the area, they also had the benefits of being in two different scenarios. Finally the personal

images chosen were from a GoPro camera which was worn during a Go-kart race, these were

the only personal videos accessible, they also have the benefit of being from the wearers point

of view. Using a GoPro camera feed did prove to be an issue, from a less knowledgeable point of

view the two videos can appear to be very similar, to try and prevent this one video was from a

night race and the other from the day.

Once the content for the survey was chosen the wording of the questions to go along

with them had to be decided, the final questions can be seen in the appendix (Section 9.1.2.).

The questions had to be phrased so the user was aware the survey was about tagging images

and videos in relation to memory without giving so much information as to influence their

choices. All the images are videos were given general titles, e.g. image 1, image 2 etc., to prevent

any naming of the images causing influence while still being able to reference them. At the end

of the survey a few follow up questions were added to get some feedback from the user, these

questions were tailored towards the produced system to try find which information people will

find the most useful. Two of the questions were multiple choice, they both provided an 'other'

option, although this was never used. These two questions aimed to find which data people find

most helpful when remembering events, and were followed by text responses asking for

reasons for their choices during the quiz. These questions aimed to gain some understanding of

the choices people made when tagging and choosing their preferred data for memory recall.

Before the final draft of the survey was sent out it was first sent to one of the people

which supplied the images, they were asked to go through the survey and give feedback on how

easy it was to answer. The initial response was good, but the idea of tags was new to them and

they had no idea where to start. With this feedback in mind a few examples were added to each

question, all questions had the same examples to show the chosen examples were not specific to

each question. Each example was chosen carefully to show the different types of tags which

could be used other than just what was in the image, additional tags like colour and the time of

day were also viable tags.

The final step was choosing who to distribute the survey to. The audience of the

produced system is potentially anyone who wishes to use it, so there was to target audience to

try and give the survey too. The only people which were excluded from the survey were the

ones who helped supply the images of pets and the GoPro videos, this is because the content is

personal to them and no other survey takers. The survey was given to people both in and out of

Lancaster to ensure the choice of tags were not influenced too much by the knowledge of the

environment images.

The distribution of the survey was mainly done through online mediums. The link to the

survey was directly passed on to some people both in and out of Lancaster and they were then

asked to pass it on to others to try and spread the survey as much as possible. A link to the

survey was also posted on a online forum where it could be picked up by people from many

different backgrounds. It is not clear which of these methods of distribution were more

successful, but from a quick scan of the results it looks as if around half knew the area and the

other half did not, which is a good result. The turnout for the survey was not as high as hoped,

the final number of respondents was 13, while more would have been better, this number will

be sufficient to produce a variation of results.

Ben Taylor 33117535


6.3. Survey Results

The nature of the questions in the survey produced very long responses, when this is

combined with the number of questions the end results table becomes very large and difficult to

understand. To make the results easier to understand they have been split into 1 table for each

question on the first 2 pages, these tables provide a count of how many times each tag was

chosen. The third page of results are split apart as well, tables of quantities are placed where

they are applicable, otherwise a list of the given answers are supplied. The list of tags were

simplified in two ways, all counts ignored the case of the letters and the plural and singular

version of tags were added together. By doing this the list of results was significantly reduced,

this simplification will be taken into account when analysing results. These results can be found

on the appendix (Section 9.1.3.).

Analysing the results will be split into three sections. First the ability of the tagging

system to match people's expectations will be analysed, this is will be followed by an analysis of

the tags people expect to be the most useful. The final section will analyse the type of data which

people find the most helpful in remembering situations and why.

6.3.1. Tagging Evaluation

To evaluate the quality of the tags which are produced by automatic tagging the results

of the survey will be compared to the tags produced by the system. The tagging system will be

evaluated as a whole and individually, the individual evaluation is with regards to the first 3

questions of the survey. This includes the tagging of Image1, Image2 and Video 1.

Figure 6.3.1.1: Graph showing the quantities of common tags from image 1

First to be analysed is image 1, Figure 6.3.1.1 shows the most common tags and their

quantities which were produced b the survey. Many more tags were chosen, a total of 45 unique

tags for this image in total but most had only been chosen once. When the same image was run

through the tagging system a wide range of tags were produced, the two lists of tags only

crossed over for 5 tags, 4 of these which are in the set of most common. The tags which crossed

0

1

2

3

4

5

6

7

8

9

Most Common Tags for Image1

Ben Taylor 33117535


were; Building, Day, Glass, Modern and University, the last of these being the only one not in the

list of most common. It is worth noting, which was not realised until after the results came in,

one of the suggestions for tags in the questions was green. This could explain the high number

of suggested tags of Green when compared to any other colour. These results indicate the

tagging system can perform sufficient tagging on an image which displays an environment,

although they are not enough to completely forget about human additions.

Next Image 2 will be analysed, this was the image of a hamster. This image was chosen

as an example of a personal image. The most common tags suggested from the survey can be

seen in Figure 6.3.1.2., the total number of suggested unique tags for this image were 28, only

the tags were chosen twice or more are shown though. When this image was put through the

tagging system some very unexpected results were received e.g. Caucasian and Studio. From the

two sets of tags they crossed over in 4 places on the tags; cute, animal, hamster and rodent, all of

these tags are in the most common list. Reviewing these tags does reveal how many words can

be used to describe one aspect of the image, in the survey the suggestions of furry, fluffy and

furball were all suggested, but it can be argued these all relate to the tag 'Fur'. This suggests the

need for the system to be able to notice which tags could be similar. The close up personal

image did not perform as well as the environment image, this conclusion came about because of

the systems suggestions for very unrelated tags in this scenario and not the previous, indicating

its struggle to tag correctly.

Figure 6.3.1.2: Graph showing the quantities of common tags from image 2

Finally the first video of the survey will be evaluated for its tags. The most common tags

and their quantities can be seen in Figure 6.3.1.3., a total of 34 unique tags were suggested for

this video. Comparing the results could be done differently for the video as it is split into chunks

when it is uploaded into the system, this means it can be seen how well different parts of the

video are tagged. With that said, most of the chunks of video were given the same tags with one

or two changes, while two of the chunks often got very different tags such as; business, art and

financial. For the purposes of the evaluation of this video the anomalous results will be ignored

for now, as most of the time the tags are better, these results will not be forgotten about though.

Out of the two lists of tags they crossed over at four points, only two of these were in the most

0

2

4

6

8

10

12

Most Common Tags for Image2

Ben Taylor 33117535


common list for the video, the list of tags are as follows: car, people, racing and wheel. The tags

were not as successful in this case, only two were matched out of the most common, and one of

those tags was only chosen twice, the other tag however was one of the most common. This

again reveals how the need to notice the similarity between words would significantly improve

the system, if the results for race and racing were combined from the survey a total of 15 would

be achieved which would make that single automatic tag much more significant. From this data

alone is does not appear as if the tagging system can handle GoPro camera feeds well when

compared to users suggestions.

Figure 6.3.1.3: Graph showing the quantities of common tags from video 1

Overall the tagging system preformed well, but not well enough it could be used alone

and replace human tagging of the images, it can still perform as a very good addition to human

tagging though if it was left in its current state. Some very important points have been revealed

though to improve the system without changing the tagging system at all. The first improvement

would be producing a system which could ignore all capitalisation of letters so a search of

'RACING' would still be able to match 'racing' and so on, to further this the system should also

be able to spot plurals of words and be able to match that to the singular form also, even

pictures with multiple cars in some people still choose the singular form.

6.3.2. Searching By Tags

The original plan for the evaluation for this section was very similar to the previous. On

further inspection of the data however it is revealed the questions have been answered in a very

similar way to the previous 3 questions. So instead of analysing all three questions individually

and repeating the last section these three questions will be analysed as a whole. The reasons

behind these similarities will also be discussed in this section.

The questions on extensively tagging and most useful tags were both answered in very

similar ways, the expected result was less tags to be suggested for questions 4-6, in some cases

there were actually more tags suggested. Ten more unique tags were actually suggested in

0

1

2

3

4

5

6

7

8

9

10

Most Common Tags for Video1

Ben Taylor 33117535


question 4 than in question 1, three more tags were suggested in question 5 than in question 2

and only one more between question 6 to question 3. This could simply be the case where the

second group of images and video had more content in them to tag, but as there was a

consistent increase between the three groups of content this seems unlikely. There is also a

possibility the people who took the survey did not read the second lot of questions correctly and

assumed it was the same as the first, they also could have read the questions and not seen the

second case being significantly different to warrant any different way of tagging. After following

up with some of the known participants of the survey there was a mix of replies when they were

asked about this. Five of the participants were asked, 3 of them said, in their own words, they

did not think there was any significant difference between the two cases, while the other two

simply assumed they were the same question. When they were further asked why they assumed

the questions were the same they stated the first 3 questions were all very similar and the

second page had the same format. Although this was the case for some it was not for all,

suggesting people would try as many tags as possible to try and find their desired image or

video.

Overall the suggested tags preformed in much the same way as the automatic tags when

compared. Tagging the castle automatically did prove to be difficult, none of the most popular

tags were matched, the closest was cars with 7 suggestions and the automatic tag of motor

vehicle. This reinforces the idea knowledge of similar words will be needed by the system.

Besides this the second image and video were just as successful as the previous section of

analysis.

From this section of the survey a few considerations can be taken away. The first and

possibly most important of the two, people will try and use as many tags as possible to find their

content, this would mean designing the search algorithms to be much more complex. As they

are implemented they will just return any content which contains any of the tags being searched

for, this should be updated to allow a search using Boolean operators. Implementing Boolean

operators would significantly improve the results, especially in a situation where years of

content could be in the system. An example search could be 'Racing AND Driving NOT night',

from this the user could return all driven races which did not occur at night. A similar search is

nowhere near possible at the moment, and adding more tags to the search would only make

finding the desired content harder. The second note to make from this evaluation is the

similarity of some of the tags without being the same e.g. race, racer and racing. If an algorithm

was created which was able to spot similar words such as the above and be able to treat them as

one this would greatly improve the system. An algorithm like this would benefit the automatic

tagging system, it would not have to be as extensive, and it would also help memory recall, the

user would not need to remember the exact tag just similar ones to get the same content. Finally

the last addition which would further improve the searches, if a person was to search for the tag

of 'Dead tree' the system should split it up into the two separate words and search for 'Dead

AND Tree' as well as 'Dead tree'. This again allows the automatic tagging to be less specific

allowing more leeway before switching to a new tagging system. This could be further extended

to reverse the idea, so 'Dead AND Tree' would search for 'Dead tree' and 'Dead AND Tree' also.

6.3.3. Which Data Is Most Helpful?

The final page of the survey was designed around getting some feedback on the kinds of

data people found most helpful and reasons for their choice of tags. This section will go through

and analyse the results of each of the questions on the final page and try and relate them to

improvements which could be applied to the system.

Ben Taylor 33117535


The first question of this page, question 7 overall - 'Which information would you find

useful when trying to find an image or video from a vague memory?', gave multiple suggestions

and an 'Other' choice. The other choice was only used by one participant who suggested two

other choices; event and season, all of the results can be seen in Figure 6.3.3.1. Although people

would not be expected to know how useful tags are it was expected they would gain some small

knowledge during the survey and make an informed decision by this point. Almost all of the

participants said location data would be helpful to them, with tags coming up second and date

third. Date and time did not seem to have the expected relationship, it was expected that if one

was picked the other would too as they are closely related. The results show people do not find

the time information as helpful as first thought, one reason for this could be how specific it is, if

a person is trying to remember an event from their past they are unlikely to remember the exact

date, rendering the time useless.

Facial recognition is another type which scored lower than expected, it was thought

people would find this useful as they would remember who was with them when a memory

happened, and being able to search for all memories containing a certain person would help.

This could cause issues however, if a person was in an image but their face was not, so they

could not be recognised. Cases like this could cause issues if a person tried to search unless the

put the tags in themselves. The suggestion of events and seasons could be helpful, seasons could

be treated as a larger portion of time other than a specific date, thus allowing more leniency in

searches. Events are also a good suggestion, if the recognition of events was easier to spot they

would have been implemented into the system, so this suggestion shows an improvement like

this would be helpful. Finally tags received a good reception from this survey which indicates

people find this new type of organisation and search just as helpful, if not more than some other

types of metadata.

Figure 6.3.3.1: Graph of results from question 7 of the survey

Question 8 was based towards the produced system, the question was 'If you were

trying to remember something which would you find more helpful, multiple images, a video or

something different?'. It was followed by a question which asked the participants to explain why

they preferred this method, this revealed why people preferred one over the other. With this

0

2

4

6

8

10

12

14

Preferred Metadata

Ben Taylor 33117535


knowledge it would be possible to work on the drawbacks of each type and make the searching

and recollection of events better. The responses from the survey indicate neither videos or

images are preferred over the other with the responses coming back as 8 votes for images and 5

for videos. It could be argued a slight preference for images is shown but without more

respondents it is difficult to conclusively say. The reasons for the choices for each are similar,

the full responses can be found in section 9.1.3.

Generally images were chosen because they are easy to scan through, therefore

allowing easier access to the memory, or are easier to dismiss if they are similar but not the

memory they desire. One participant even suggested images are best because the brain

remembers memories as images, so the connection between the two would happen quicker. The

only reason people gave for choosing videos was for the added benefit of there being more

information, whether it be from audio output as well, or just because a video is a collection of

images. From this it can be taken that videos are mostly preferred because they provide extra

data and context, while images are preferred because of their ease of access. These were the

expected results, this is the very reason any videos input into the system are split into chunks of

5 minutes. Perhaps a better idea would be to allow the user more control over the size of the

chunks depending on the content being input, the original size of 5 minute chunks was decided

because long live video feeds were expected, but as the system evolves it seem reasonable this

such evolve too. The lack of use of the 'other' choice in this question indicates people find

images and videos to be sufficient methods of memory capture when combined with the added

meta data, a full user study would reveal if this was the case though.

The final question of the survey, 'How do you think the tags you chose will let you

distinguish one memory from another?', aimed to get some feedback on the reason people chose

the tags they did. With this information, if a bespoke tagging system was to be made, it would be

possible to produce much more effective tags to help the users. The full answers to this question

can be found in the appendix, Section 9.1.3. There was some consistency with the answers

given to this question, generally people chose tags for objects which they could easily see in

each. There was no mention of the colour tags in any of the answers to this question, it is

possible these tags were only chosen because they were given as a suggestion. Other answers

indicate people trying to get a balance between tags being specific towards each image or video

while still being memorable, this meant adding in the specific tags such as 'castle' but also

including 'building' and 'old' to give them other methods of accessing that memory through tags.

The same participant which also suggested event and season data earlier also tagged images

based on the weather when they were outdoors, as the weather is often a very memorable yet

general aspect of many outdoor memories. Overall it can be said the chosen tags all relate

directly to each image or video, with a mix of both general and more specific tags given to each,

it is this combination which will help people remember and retrieve the memories.

The evaluation of the user interface and tagging revealed some interesting points,

although more could be revealed if this was looked into more thoroughly and over a longer

period of time. It is clear there is still room for improvement with the user interface, but overall

the design of it was well received. The main points to take from the evaluation come from the

survey results, which indicate the best ways to improve the system to make the tagging better,

and improve the ability of the system to be a memory aid. Building a bespoke tagging system

would be an option, as it could evolve to a user's personal preferences for tags, instead of one

general system for everyone. Before considering this there are alternatives to help improve the

current system of tagging, these can be simplified down to 4 main points:

Ben Taylor 33117535


Treat Singular And Plural The Same - When a search for the singular version of a

word is sent, return results for the plural word as well. People will often search for the singular

term when there are multiple objects in the screen to attempt to match the tags. e.g. If searching

for 'Car' return results as if searching for 'Car' and 'Cars'.

Boolean Expressions For Searches - Current searches do not allow any amount of

complexity, introducing Boolean expressions will allow much more powerful searches to take

place. This is an especially important point for a system which aims to collect huge amount of

image and video data to search through.

Recognise Similar Words Or Phrases - Images and video can have ambiguous content,

and all people will have different ways of describing that content, the hope is all of the different

ways of describing the content will be similar. If a search for a race was to be conducted it

should be recognised how similar 'race' is to 'racing' and 'racer'.

Split Up Phrases Into Single Words - While automatic tagging it is unlikely to provide

results as specific as 'Large brick house', people are still likely to require searches like this.

Introducing Boolean expressions could improve this, but that still requires the user to separate

all searches with 'AND' e.g. 'Large AND brick AND house'. If the system would automatically do

this for the user, and do the reverse e.g. 'Green AND house' will also search for 'Green house', it

would make the system much easier to use and user friendly.

7. Conclusion The entire project as a whole has been successful, there have been limitations and there

is improvements which could be made, however given the limitations a lot has been revealed

and achieved. This conclusion will be split up into multiple sections, each discussing different

aspects of the project as a whole. The successes and failures will be discussed in relation to the

goals set out in the introduction, what could have been done differently if the knowledge which

was gained was known from the start and what has been learnt from doing the project.

Improvements to this project as well as how the knowledge gained can be built on in further

work could also be discussed.

7.1. Review Of Aims

The aims which were set out in the introduction were very general aims, this is because

the project was very exploratory, so a definitive set of requirements could not be defined at the

time. A better set of requirements to compare the success of the system to would be the set

which was defined in the design section of this document. These goals are based off the original

goals, they are just more specific towards certain aspects. First the system will be judged on its

functional requirements, and then the non functional.

The functional requirements could all be easily tested in the testing section of this

report, which they were, the only functional requirement which was not tested was R1.5, as this

was not a seen function of the system, but it was still achieved. Based on the results from the

testing of the system it can be said all the functional requirements of the system were met, there

were a few issues with the user interface discovered in this section but all the functionality was

there.

The non-functional requirements are much harder to test and prove, to decide if each

has been achieved it will be discussed and a reason will be given for the pass or failure. The first

Ben Taylor 33117535


(R2.1, The system shall be able to store personal memory data) and fourth (R2.4, The system

shall allow multiple methods of memory capture) non-functional requirements are the easiest

to judge. The produced system was able to provide inputs for any method of data capture

provided it was image or video, thus allowing multiple methods of memory capture, the system

was also able to store and playback this data, fulfilling requirement R2.1. The fifth requirement

(R2.5, the system shall have a simple and intuitive user interface) was harder to judge, based on

the feedback from the user interface evaluation the points made were mostly good. The user

interface could do with improvements to bring it up to a standard which will definitely pass this

requirement. The user interface was kept simple by only displaying necessary content on each

page, and was made intuitive by displaying almost all navigation option on every page, with this

in mind this requirement is passed, but improvements are definitely needed.

The last two non-functional requirements are harder to judge. The second requirement

(R2.2, The system shall promote memory recall as much as possible) could not be judged for a

pass or failure in the end. To judge this a lengthy memory study would need to be conducted

which asked people how helpful the user interface was, doing this could also reveal points for

improvement. The final requirement to discuss, R2.3 (The system shall securely store all data to

protect the privacy of users), could be seen in different ways. While no steps are taken to secure

the data which is stored by the system, it could be argued there is no need to take further steps

as no data ever leaves the computer the system is being used on. The way the system has been

produced it would be used a personal computer, and in those situations images and video are

rarely encrypted to begin with. If the system was to become an online collection much more

would need to be done to keep the data safe and secure. With the system in the state it is, it can

be said this requirement is also passed.

Overall the system achieved the goals which were set out at the beginning of the project,

there is still room for improvement though. One of the goals cannot be assessed at this point in

time so it is not clear how well the system preformed in that aspect, however some important

and useful data has be uncovered.

7.2. Suggested Revisions To Design/Implementation

This section will discuss any revisions or improvements which could be made to the

system, much of this will be based on the feedback received during the evaluation stage. The

main improvements were discussed at the end of the evaluation section, this will be a overview

of those improvements. Any improvements which could be made to the design stage will also be

discussed.

During the design stage many ideas were discussed, yet only few made it through due to

the time constraints. If there was more time more features would be added into the system,

features such as facial recognition and word clouds to help users visualise the list of tags easier.

Word clouds could also be combined with the confidence value of each tag to give the users a

much better idea of how well the tagging performs in different situations. Overall the best

features to add into the system would help communicate new and already existing information

to the users to help the effects of memory recall and to improve the system quality.

The improvements to the searching algorithm would be a great addition to the system.

Along with the discussed points it would be beneficial to allow searches across multiple fields

of data, this mean being able to search for specific tags while also limiting the dates and location

of the retrieved data. Implementing all of the discussed improvements in the evaluation plus the

Ben Taylor 33117535


point just discussed would greatly improve the usefulness of the system while making the

search for any content much easier.

If there was more time a bespoke tagging system would have been produced which

could evolve to a user's needs. The idea behind this would be the user 'teaching' the tagging

system how to tag certain objects in the same ways the user would recognise them. This idea

could also have been introduced with facial recognition, which would make the tagging of

images much more powerful. In general further investigation would have taken place to see

what more metadata could be added to each image and video to further improve memory recall.

Improving the user interface could also prove to be beneficial, adding in more messages

and confirmation to make sure the user knows when actions have been successful and

unsuccessful. The aim would be to push as much of the uploading of data processing behind the

scenes and not making the user wait around, further improvements would add in more

information about the system status. This would mean introducing messages which would

inform the user how much data is still being processed by the system, this aims to prevent long

processing times from confusing the user into thinking the data was never uploaded and

attempting to upload the data again.

7.3. Future Work

In the future this work cab be built upon to produce a much more powerful system. The

proposed system would be able to include many more of the aspects which were discussed

during the background section of this report. Future work on the system would mean improving

it based on the results from the feedback and with the ideas from the previous section. Once all

these improvements have been made a lengthy user study to evaluate how effective the types of

metadata are, how the user interface can help with memory recall and recollection and how

useful the search methods are when trying to remember information is.

Further work could also evolve this small system into a large distributed system which

can be accessed by many different users, who can also supply plenty of data to share.. This is an

ambitious leap to make from the current system, but the idea of a system like this would be able

to take advantage of current infrastructure, e.g. CCTV cameras if permitted, to help capture

memories in a whole new manner. Such a system would require much more thought about data

security and protection though, as memory manipulation could be a huge threat.

In conclusion the project has been successful, during the project plenty of skills were

learnt and developed which can then be taken on to further work. This project has taught the

need for plenty of relevant research before starting design, but also evolving the system as more

needs arise is also an option. This project has revealed a better understanding of how tags can

be used to enhance the ability to search through large amount of data, and how a search can be

designed to find a memory which the searcher cannot remember the full details about. Overall

the project has produced a good base from which more complex memory aid systems could be

built from.

8. Bibliography Berry, E. et al., 2009. The neural basis of effective memory therapy in a patient with limbic

encephalitis. Neurology, Neuro-Surgery, and Psychiatry with Practical Neurology, 80(3), pp. 582-

601.

Ben Taylor 33117535


Cassidy, k., 2015. Popular Video Formats, Players and Distribution Methods. [Online]

Available at: http://www.videomaker.com/article/14519-popular-video-formats-players-and-

distribution-methods

[Accessed 13 March 2015].

Cheng, J., 2010. Twitter gets government warning over 2009 security breaches. [Online]

Available at: http://arstechnica.com/tech-policy/2010/06/ftc-gives-twitter-a-wrist-slap-over-

2009-security-breaches/


Conway, M., 1990. Autobiogrpahical Memory.

Davies, N. et al., 2014. Security and Privacy Implications of Pervasive Memory Augmentation.

pp. 4-6.

Doherty, A., Gurrin, C. & Smeaton, A., 2009. An investigation into event decay from large

personal media archives. Proceedings of the 1st ACM international workshop on Events in

multimedia, pp. 49-56.

Fulton, W., 2015. Image file formats - TIF, JPG, PNG, GIF. [Online]

Available at: http://www.scantips.com/basics09.html


Gurrin, C., Smeaton, A. & Doherty, A., 2014. LifeLogging: Personal Big Data. Foundations and

Trends in Information Retrieval, Vol.8(1), pp. 14-17, 46-49, 70-75.

Hamilakis, Y. & Labanyi, J., 2008. Introduction: time, materiality, and the work of memory.

History and Memory, 20(2), pp. 5-17.

Krynsky, M., 2015. Lifelogging/Quantified Self | Lifestream Blog. [Online]

Available at: http://lifestreamblog.com/lifelogging/

[Accessed 25 February 2015].

Kubiatowicz, J. et al., 2000. OceanStore: An architecture for global-scale persistent storage.

Manovich, L., n.d. As We May Think. The New Media Reader, p. 35.

Mastin, L., 2015. Memory Recall/Retrieval - Memory Processes - The Human Memory. [Online]

Available at: http://www.human-memory.net/processes_recall.html


Paul, R., 2010. EPIC fail: Google faces FTC complaint over Buzz Privacy. [Online]

Available at: http://arstechnica.com/security/2010/02/epic-fail-google-faces-complaint-over-

buzz-privacy-issues/


Petrelli, D., Whittaker, S. & Brockmeier, J., 2008. Autography: What can physical mementos tell

us about digital memories. pp. 53-62.

Pillemer, D., 1998. Momentous events, vivid memories: How unforgettable mements help us

understand the meaning of our lives.

Sellen, A. & Whittaker, S., 2010. Beyond total capture: a constructive critique of lifelogging.

53(5), pp. 70-77.

Ben Taylor 33117535


Shakimov, A. et al., 2011. Vis-a-Vis: Privacy Preserving Online Social Networking via Virtual

Individual Servers. p. 1.

Stix, G., 2011. Photographic Memory: Wearable cam could help patients stave off effects of

impaired recall. Scientific American.

Tatsuyuki, K. et al., 2006. Ubiquitous Memories: a memory externalization system using physical

objects. Personal and ubiquitous Computing, 4(11), pp. 287-298.

Thomas, D., 2015. UK Watchdog raises privacy concerns over Google Glass. [Online]

Available at: http://www.ft.com/cms/s/0/3488b1d2-fd59-11e3-96a9-

00144feab7de.html#axzz3SmdW0fwE


Whittaker, S. et al., 2010. Socio-technical Lifelogging: Deriving Design Principles For A Future

Proof Digital Past. pp. 7, 11, 13-17.

9. Appendix

9.1. Tagging Survey

9.1.1. Screenshots of survey

Figure 9.1.1.1: Survey page 1 - top - first screenshot

Ben Taylor 33117535


Figure 9.1.1.2: Survey page 1 - second screenshot down

Figure 9.1.1.3: Survey page 1 - 3rd screenshot down

Figure 9.1.1.4: Survey page 1 - bottom - 4th screenshot down

Ben Taylor 33117535


Figure 9.1.1.5: Survey page 2 - top - 1st screenshot

Figure 9.1.1.6: Survey Page 2 - 2nd screenshot down

Figure 9.1.1.7: Survey Page 2 - 3rd Screenshot down

Ben Taylor 33117535


Figure 9.1.1.8: Survey Page 2 - Bottom - 4th screenshot down

Figure 9.1.1.9: Survey Page 3 - Top - 1st screenshot down

Figure 9.1.1.10: Survey page 3 - Bottom - 2nd screenshot down

Ben Taylor 33117535


9.1.2. Survey Questions

As the questions may be difficult to see in the screenshots they have been included

below with their question number.

1. Look at the above image (Image 1) if you were to tag this image, which tags would you

use? Try to be as extensive as possible. (Separate multiple tags with a comma) e.g

People, Day, Green

2. Look at the above image (Image 2) if you were to tag this image, which tags would you

use? Try to be as extensive as possible. (Separate multiple tags with a comma) e.g

People, Day, Green

3. Watch some of the video above (Video 1). Which tags would you attach to this video?

Try to be as extensive as possible. (Separate multiple tags with a comma) e.g People,

Day, Green

4. Look at the above image (Image 3). If you wanted to retrieve this image which tags

would you find helpful? (Separate multiple tags with a comma) e.g People, Day, Green

5. Look at the above image (Image 4). If you wanted to retrieve this image which tags


6. Watch some of the video above (Video 2). If you wanted to retrieve this video which tags


7. Which information would you find useful when trying to find an image or video from a

vague memory?

8. If you were trying to remember something which would you find more helpful, multiple

images, a video or something different?

9. Why would you find this the most helpful?

10. How do you think the tags you chose will let you distinguish one memory from another?

9.1.3. Survey Results

Tag Quantity

Building 8

Grass 7

Green 7

Modern 5

Glass 3

InfoLab 3

InfoLab21 3

Cloudy 2

Crossing 2

Day 2

Technology 2

Computing 2

Outside 1

Peaceful 1

Grey 1

Important 1

Green Building 1

Day Out 1

Meeting 1

Institution 1

Ben Taylor 33117535


Pathway 1

School 1

Square 1

Symmetrical 1

Geometric 1

Bland 1

Harsh 1

Grey 1

Paved 1

Architecture 1

Model 1

Laboratory 1

New Building 1

Blue 1

Orange 1

Windows 1

Colours 1

Brick 1

Lights 1

Business 1

Conservative 1

Lancaster 1

University 1 Lancaster University 1

Campus 1

Communications 1

Engineering 1

Table 9.1.3.1: Tally of results from question 1

Tag Quantity

Cute 10

Hamster 10

Animal 6

Furry 4

Pet 4

Fluffy 3

Sleep 3

Pink 3

Sawdust 2

Bedding 2

Bed 2 Wood Shavings 2

Rodent 2

Brown 2

Pink 2

Guinea Pig 2

Furball 1

Cage 1

Soft 1

Cosy 1

Childhood 1

Cuddly 1

Ben Taylor 33117535


Comfortable 1

Inquisitive 1

Yellow 1

Black 1

Woodchipping 1

Habitat 1

Gerbil 1

Care 1


Tag Quantity

GoKart 9

Racing 9

Driving 6

Race 5

Helmet 3

Sports 3

Green 3

POV 3

Compete 3

Track 3

Outside 2

Road 2

Exciting 2

Fast 2

Car 2

Concentrate 1

Winner 1

Loser 1

Kart 1

Motorsports 1

circuit 1

Camera 1

Qualifying 1

Racers 1

Safety 1

GoPro 1

Day 1

Fun 1

Recreation 1

Wheels 1

Raceway 1

Turning 1

Rushing 1

People 1

Crash 1

Tarmac 1

First Person 1


Tag Quantity

Castle 9

Cars 7

Grass 7

Ben Taylor 33117535


Trees 7

Building 4

Church 3

Green 3

History 3

Old 3

Old Building 2

Prison 2

Road 2

Tourism 2

Winter 2

Abbey 1

Ancient 1

Autumn 1

Branches 1

Brown 1

Cloudy 1

Cold 1

Creepy 1

Day 1

Dead Trees 1

Duke` 1

Dull 1

Gloomy 1

Green car 1

Grey 1

Grey Car 1

Hill 1

Interesting 1

Lancaster 1

Landmark 1

Landscape 1

Lime Car 1

Medieval 1

Monastery 1

Old Fashioned 1

Outside 1

Park 1

People 1 Place Of Interest 1

Royal 1

Scary 1

Scenery 1

Spooky 1

Summer 1 Tourist Attraction 1

Visiting 1

Wood 1

Stone 1

Turret 1

Brick 1

Overcast 1

Ben Taylor 33117535


Historic 1


Tag Quantity

Cage 11

Hamster 11

Cute 6

Green 5

Animal 3

Bars 3

Ginger 3

Rodent 3

Bedding 2

Guinea Pig 2

Blue 1

Curious 1

Fluff 1

Fluffy 1

Furry 1

Hamster Bedding 1

Happy Face 1

Mouse 1

Orange 1 Packing Peanuts 1

Shavings 1

Syrian Hamster 1

Tired 1

White 1

Bowl 1

Sleeping 1

Fat 1

Small 1

Bed 1

Gerbil 1

Pet 1

Care 1

Face 1


Tag Quantity

Night 9

Racing 7

Go Kart 5

Race 5

Endurance 4

Dark 3

Driving 3

Go Pro 2

Kart 2

Lights 2 Point Of View 2

Ben Taylor 33117535


Road 2

Sport 2

Track 2

Camera 1

Car 1

Competition 1

Competitive 1

Crash 1

Fast 1

Focus 1

Go-Kart 1

Helmet 1

Kart Race 1

Karting 1

Laps 1

Motorsport 1

Night Race 1

Night Time 1

Outdoor 1

People 1

Racers 1

Rushing 1

Safety 1

Speed 1

Tarmac 1 Warden Law 1

Fun 1


Info Quantity

Location 12

Tags 10

Date 8 Facial Recognition 6

Time 4

Event 1

Season 1


Type Quantity

Images 8

Video 5


Question 9 answers:

• (Images, the first response did not answer this question)

Ben Taylor 33117535


• (Images)Easier to remember

• (Images)If a video is being used to remembers something, most videos are short and

dont show alot of the particular event. Pictures make it easier as it has people in them to

help jog the memory

• (Videos)A video provides more information than images do

• (Images)Because you would instantly recognise the image from your memory

• (Images)Videos would take too long to scan through

• (Images)Images are easier to search through

• (Videos)gives a more detailed account of events

• (Images)The Brain remembers still images better

• (Images)Video takes time to watch

• (Videos)Both visual and audio triggers

• (Videos)A video can arguably capture more than still images, returning more valuable

information.

• (Videos)its mjust multiple images really, more context

Question 10 answers:

• They are specific towards the image or video. I chose them to relate towards the image

as much as possible

• I can visually see and recognise them.

• Specific descriptions such as autumn and weather descriptions like cloudy/raining help

differentiate days from the other because its a stand out factor of days out.

• Relating them to my own experiences and identifying the different aspects of the photo

that appeal to me and my response to the memory.

• Specific tags allow you to search through many variety of images and help you locate the

one you want, rather than go through every image,

• They're hopefully easy to remember, but are specific enough to narrow down search

results to a manageable level.

• They are specific towards each situation. I tried to avoid using just general tags by

adding more specific ones as well

• they visually describe the scene as i think i would remember them

• Different Backgrounds, Objects in photo

• Not sure

• Certain tags are only applicable to certain memories, if I wanted to recall a memory

where I visited a castle, the tags 'building' and 'old' would enable me to narrow down

the search for this particular memory.

• They include related words as well as the actual contents of the image/video. the

distinct difference between them, like the car to a hamster, thing to animal. i guess you

would relate to other things to like feelings which would effect everything too.

Ben Taylor 33117535


9.2. Project Proposal

Storing The Whole Worlds Memories Ben Taylor

Abstract

Human memory is fallible yet pervasive,

and technology has advanced to a point

where it is possible to store many aspects of

human activity. This project is based on the

idea of recognition rather than recall when

it comes to human memory and aims to aid

in remembering. The end product of this

project will be a prototype system which

has a chance to be extended into a whole

world implementation, so will take into

account huge amount of data storing and

searching. When evaluating the end product

it is hoped the system will be robust and

secure while having efficient storage and

querying of data.

1. Introduction

The main aim of this project is to produce a

prototype system of a memory store which,

while being able to be extended to suit the

needs of an entire planet, can store and then

display back memories. This prototype

would be based on concepts from

distributed file systems.

With the widespread introduction of many

different ways to capture memories such as

personal recordings etc. it has come to a

point where a system could be produced to

collate all of this data for each person

together in a kind of memory store. A

system such as this would need to offer

robust and secure storage while being able

to scale to the entire planet and still be able

to respond to queries in real time.

This report will be broken down into a few

different sections; background and the

proposed plan of work. The background will

help the understanding of the project, the

proposed project is a more in depth look at

the aims of the project and any problems

which will need to be overcome.

2.Background

Human memory is fallible yet pervasive,

and technology has advanced to a point

where it is possible to store many aspects of

human activity. This system would not be

used to replace any part of human memory,

but to enhance it as it would support the

idea of recognition rather than recall.

Recognition over recall is a very important

concept when it comes to humans and

remembering data. The human mind is built

in such a way which makes it much easier to

remember data when something triggers

the thought in the first place, this is the big

thought behind GUI menus, instead of

command line where the user is required to

remember commands to use the system.

People all over the world are out capturing

data about their lives all of the time on

personal recording devices and such. So

based on these two ideas it seem like a

logical step to combine them to produce a

system which would enhance a person's

memory through recognition using their

data captured. It could also be used to help

people who have conditions which cause

them to lose all or part of their memory, as

there is still a store of their memories

somewhere.

The closest thing to a system like this would

be social media sites, but even they are far

away from the proposed system. They are

similar in the aspect of collecting data on

people, and people posting up data on

themselves which is collected and stored by

Ben Taylor 33117535


the site owner. The big difference however,

while this data is stored it is not able to be

queried or viewed in any other form than

the one it is given in, so if it was written

down, it could not be combined with other

similar data to be viewed in a simpler way.

3.The proposed project

This project aims to collate data on people

and be able to store it in an efficient manner

which still allows quick access while having

the ability to scale to a whole planets needs.

3.1. Aims and Objectives

The main aims of this project are listed as

bullet points below:

• Store data in an efficient manner

• Allow fast real-time queries of data

• Securely store all data

• Produce a robust system

• Produce a simple system from the

complex idea.

3.2. Problems

When first thinking about how to create the

prototype a few problems came to mind

which would need to be solved in order for

the system to work to its full potential. They

are explained in the next few paragraphs.

The first problem to solve would be to

figure out how the system would be able to

take people's memories and store them in a

format which could be read back and

understood. This is because human memory

can store a large amounts of diverse

information, and this would need to be

translated into a form for the system to

understand, while not losing the key aspects

of the memory. Once this barrier has been

overcome an efficient way of storing all of

this data would need to be decided as in

practice there would be huge amounts of

data, and querying the data needs to be

efficient.

Secondly there would be the issue of how to

display the data in a way that is easily

intelligible. This would mean taking the

data and translating it back into a way for

people to understand, it would have to be

clear in what it is showing as to not cause

data to be remembered differently and

causing people to think the system is

unreliable.

The third problem would be finding a way

to implement searches of the data which are

fast yet provide the required results. This

would mean thinking very carefully about

how a user would search the system and

optimizing the search process as much as

possible. While much of the optimization

would not be needed in the prototype as it

is only a small system, the extension of the

project into the real world and across the

entire planet needs to be thought about.

The fourth and final problem would be

ensuring the system is covered with plenty

of security before any kind of testing is

carried out. This means thinking very

carefully about all aspects of the systems

components and how much security each

requires.

3.3. Methodology

When producing the system a lot of

emphasis will be placed upon designing and

optimizing the system. This will mean dry

running many of the algorithms and

ensuring they work before implementation.

There will also be a lot of thought put into

the best way to make the system secure and

robust, preventing any errors or

unauthorized access happening. This is

because the system could be dealing with

peoples personal data and would be

required by law to keep it safe. This would

mean a lot less time would need to be spent

implementing and testing/evaluating the

system.

Ben Taylor 33117535


4. References

[1] http://scc-intranet.lancs.ac.uk/StudentProjects/2014/Nigel/2014-049-220508