Instructions for preparing proposal Part B for Integrated Projects in the IST Priority

FP6-2004-IST-4

Spec. targeted research project

[SPARQLE]

Specific targeted research project

IST call 4

FP6-2004-IST-4

[Smart Personal Assistant to impRove Quality of Life for the Elderly]

[SPARQLE]

Date of preparation: [7th March 2005]

Participant no.

Participant name

Participant org. short name

1 (coordinator)

Thales Research and Technology (UK)

TRTUK

2

Centre for Communications Systems Research/I-Lab, University of Surrey (UK)

UniS

3

Dept. of Intelligent Systems, Institut Jozef Stefan (SI)

IJS

4

Commissariat à l'Energie Atomique (FR)

CEA

5

Kardiosis Cardiological Diagnostic Systems Ltd. (TR)

KCDSL

6

University of Dundee Queen Mother Research Centre (UK)

UoD

7

etc. (Check this participant numbering is reflected in the form A2 of each participant!)

Name of the coordinating person: [Tim Masterton]

e-mail: [tim.masterton@thalesgroup.com]

fax: [+44 118 923 8399]

Table of contents

[table of contents]

Proposal summary page

[Smart Personal Assistant to impRove Quality of Life for the Elderly]

[SPARQLE]

Strategic objectives addressed

IST-2004-2.4.8 Cognitive Systems

Proposal abstract

TJM - I've tried to modify this in line with your suggestions but I want it to match the Form A abstract, which is limited to 900 characters so we are still a bit over the limit.

SPARQLE aims to develop an always-on Smart Personal Assistant (SPA) for elderly citizens. The SPA will improve quality of life for the elderly by providing continuous access to medical assistance and social companionship at the same time as more efficient use of care resources. It will be designed to gather information from its surroundings using a network of sensors and audio-visual capture devices. It will also support communication with its user through a suite of easy-to-use interfaces. The important features of the SPA are that it will act autonomously on a wide range of sensory inputs (by using cognitive capabilities to build an understanding of the situation they represent) and provide assistance through a natural interface that finds high acceptability with its target users. It will also be possible to tailor the device to its user's requirements and access the communications infrastructure, either at the user's request or autonomously in the case of an emergency.

B.1 Scientific and technological objectives of the project and state of the art

Recommended length – three pages

(Describe in detail the proposed project’s S&T objectives. The objectives should be those achievable within the project, not through subsequent development, and should be stated in a measurable and verifiable form. The progress of the project work will be measured against these goals in later reviews and assessments. Describe the state-of-the-art in the area concerned and how the proposed project will enhance the state-of-the-art in that area.)

State of the art systems that are utilised for elderly care either provide simple event-triggered responses such as fall alarms or passively record medical information while depending on the evaluations and actions of other human beings. These health-oriented solutions are far from providing an alternative for the human-intelligence necessary for the care and guidance of the elderly.

This project aims to develop a Smart Personal Assistant (SPA) which will autonomously provide care, guidance and assistance to the elderly citizens in their everyday lives, as depicted in Figure 1. Such complicated tasks require high-level cognitive capabilities to evaluate the situation and to generate relevant and useful responses. The technological objectives associated with SPARQLE can be divided into five stages:

· Sensor networks and audio-visual (AV) devices - Obtaining data related to the health, environment and psychological status of the elderly through a wide range of sensors and audio and video capturing devices. While the aim is to capture as much information as possible, human factors will also be taken into account for the interface design. The ease with which the sensor subsystem can be tailored to the individual user will also be considered.

· Context extraction - Interpretation of the data gathered for speech recognition and detecting the short-term and long-term changes in the sensory data as well as salient aspects of the context for intuitive connotation.

· Cognitive Systems - Implementation of a high-level cognitive system that will carry out understanding, reasoning, and making autonomous decisions through advanced cognitive and artificial-intelligence methods, based on the context extracted from the sensory and AV information. The system will be able to accept information from multiple information sources, e.g. arbitraty number of sensors, and integrate it into one percieved/understood situation. It will evaluate the situation, compare it with previous experience and knowledge in the form of scenarios and ontologies and and decide on appropriate personalised actions to be taken. It will acquire situated knowledge through trial and error and learn from experience.

· Communicating - The results and recommendations need to be passed to the user or to remote helpers via network infrastructure. The user feedback should be on human terms, through synthesized speech as well as graphical annotation.

· Evaluating the solutions - Scientific improvements are required in the way that the vast test spaces of multi-sensor systems are explored for suitability. We intend to do this by building on the work done by one of our partners in developing the CAUTIC method of usability and acceptability analysis, which has a number of advantages for this type of development.

Figure 1 - SPARQLE System Overview

B.1.1 Example

To illustrate basic ideas of the SPARQLE, we present an example:

Martha, an 82 year old lady, comes into her living room where the PC with the SPARQLE system is located. The system recognizes her through camera and microphones and similarity with previous events at this hour and says through a speech system:

SPA> Dear Martha, good morning. Nice to see you again. I hope you slept well.

Martha> Yeah, yeah, it was lousy. I’m too old to sleep well.

(It is beyonf SPA’s capabilities to completely understand what Martha said, because she mumbles. However, using voice recognition, it concludes she has not requested anything of it, so it proceeds according to the standard morning scenario.)

SPA> Dear Martha, please take the usual pills. They are listed on my screen if you want to check.

Martha> Damm doctors, they won’t let me die. Well, if I have to…

(SPA recogniszs that she is taking pills and then preparing breakfast. It observes activities, stores short description in the log file as well as compressed sensor recording. When Martha has finished washing her dishes and walks around free-handed, SPA tries to communicate with her.)

SPA> Dear Martha, I hope that you enjoyed the breakfast. Your son has sent you an email. You also have five other mails waiting. Do you want me to read them to you as usual?

Martha> Heh.

(SPA does not understand and wants clarification.)

SPA> Sorry, I did not understand you. Please say Yes or No or click one of the two options on my screen.

Martha> How come you don’t understand me? I said Yes.

(SPA now understand that the answer is probably yes; since it matches Martha’s usual habits, it proceeds to read the emails. SPA understands that Martha is coperating and participating in these activities. During this action, SPA notices that Martha’s daughter is watching Martha through a camera and – according to instructions – reports.)

SPA> Dear Martha, your daughter Mary just started watching you through a camera. (SPA shows Mary on the screen.)

Martha> Oh, Mary, nice to see you. I haven’t heard from you in several days.

Mary> Mother, I talked to you yesterday. You forgot again!

Martha> You’re probably right! And if I won’t belive you , you’ll again show me SPA’s logs. (In a joking tone) Thanks a lot for such artificial intelligence!

Later in the afternoon, Martha falls asleep when watching TV. SPA is aware of her habit and does not react until Martha’s heart beat turns into unusual arrithmia. SPA now has several conflicting possible actions: to let Martha sleep on, to call for help or to try to communicate with Martha. Since none of the options wins in the conflict resolution, SPA waits for several seconds. During this period the heart continues to beat irregularly, but not dramatically so. As time passes, SPA proceeds with the wake-up scenario, which was not used before with Martha in such circumstances. SPA’s uderstanding is that the problem needs attention but that Martha still seems outside alarming state so no alarm or call for help is performed.

SPA> Martha, Martha, MARTHA. (each time SPA calls louder, but there is no reply from Martha, so SPA follows the predefined scenario and sets on alarm sounds. Martha wakes up, which SPA notices through voice and pictue.)

Martha> MMM, MHNMM.

SPA> Dear Martha, sorry, I did not understand. Your heart beat is slightly irregular and I am concerned for your health. Are you OK?

Martha> ANNN, AMMM…

(SPA now tries to downgrade the communication since the system fails to understand speech.

SPA> Dear Martha, I still sense some heart irregularity so it is important that we establish communication. Please say Yes if you feel OK or step to the computer and press any key or mouse button. If you need help, please say Help.

Martha> AHHH…

SPA> I am now calling for help. You can interrupt me by pressing any key or mouse button.

(SPA calls for help according to the general scenario for heart emergency, adapted especially for Martha since she is a heart patient. The SPA’s autonomous decision was made based on predefined scenarios, current situation and probability calculations including previous experience with Martha and incorporting time as an essential parameter.)

B.1.2 Sensor Networks

A device that aims to assist human intelligence should be able to collect wide range of information related to the health, environment and psychological status of its user. Sensors monitoring medical data such as body temperature, heart-beat rate, blood pressure, blood sugar level, etc, are already available and being used. However, while such sensors provide information about the health status of the elderly, they are not sufficient to evaluate the causes and effects leading to specific medical conditions. Moreover, such medical data gives no information about the mental and psychological status of the elderly and her/his surroundings. Therefore, such sensory information should be supported by speech recognition, AV content analysis, motion detection, proximity sensation, light-level detection, GPS-based localisation, etc.

SPARQLE will investigate the latest lightweight and wearable sensors and perform a thorough study of their interface and power requirements. A sample of available sensors will also be converted to wireless operation using an appropriate wireless technology. The interface design will be based on an assessment of the scalability of existing standards to further sensors. The documentation for this phase will demonstrate a migration path to sensors beyond those actually converted and will recommend enhancements to existing standards where required.

B.1.3 Context Extraction

A significant amount of pre-processing of the data from the sensors and AV capturing devices is required to reduce the amount of data fed to the cognitive system and translate it into terms that the cognitive system can handle. The term "context" is used to refer to the information passed from the initial processing to the cognitive system.

Context has several different meanings. However, the generally agreed primary set, which contains the location, time, identity and activity, can be used to deduce other types of contextual information through processing and interpretation. While the first two elements of this set are the easier to acquire, the latter two, namely identity and activity, require more attention.

Audio and video processing plays an essential role in the extraction of identity and activity information. Although there are available algorithms for identity detection using either speech or images, these need to be combined and improved to achieve robust and reliable operation under time-varying adverse conditions, which are likely to occur in everyday situations.

The detection of activity also requires separate and joint analysis of audio and video while being supported by the sensory information, such as the location and motion. Content analysis of audio is usually considered as an initial step of audio analysis to distinguish between speech, music, silence and environmental sounds. Distinguishing speech from other sounds and silence is essential to carry out proper segmentation prior to speech recognition. The recognition of environmental sounds for identification of its source and/or the cause is also among the current research interests. Environmental sounds, such as an alarm, door bell, barking dog, smashing glass, running water, foot steps, etc, carry intuitive connotation about the context.

The smart personal assistant needs the analysis and understanding of the context and environment autonomously. Therefore, video recording is an essential feature in addition to its audio recording capabilities, as the captured video also contains valuable information about the activity. However, the raw video input usually comprises a large amount of data when captured, which is not always significant without any further analysis of the content. Thus, a suite of video content analysis tools is needed to determine a region or an object of interest within the detected video scenes.

Key algorithms, such as object recognition, text detection, etc, will allow for the segmentation of the identified objects or regions of interest for the accurate analysis of the possible context that is being recorded. Furthermore, the object recognition and segmentation enables the tracking of important regions, which in turn provide the means for the necessary event detection and understanding of the context from the video input. In this way for instance, the recognition of the shape, colour or the name of a medicine box will trigger a warning if wrong medicine or time of the day is detected.

Thus, together with audio input processing, video content analysis provides a suitable mechanism for accurate focusing on the salient areas of the recorded content for better understanding of the context and/or environment. Audio can be used to guide video processing, and likewise video can lead the attention to the areas of focus in audio during content analysis. In the case of recording several channels of audio by a microphone array, it is possible to localise the source of a sound using acoustic source localisation strategies, which then enables automatic camera-aiming in the direction of the source.

B.1.4 Cognitive systems

An intelligent cognitive system will receive data from all the sensor sources and the context extraction processing and from environment, e.g. the web. These inputs will be processed by an array of intermediate-level cognitive agents, each suited to its particular type of data, whose task will be to prepare tha data in a uniform way for the higher levels of the congnitive system.

The system will constantly maintain a model of the current situation (a form of technical consciousness). Should the situation match the preconditions for one of several known scenarios, it will initiate the corresponding set of actions. Let us take as an example Martha, an elderly lady in the introductory section. It is morning and the agent responsible for interpretation of visual data reports that Martha has arrived in the living room. This matches the preconditions for the morning scenario: another agent, whose task is communication with Martha, prompts her to take her medications; the system then observes her making breakfast; after breakfast, it informs her of recieved e-mail and reads it to her is she so desires. Another example would be Martha being immobile and life-signs agent reporting unusual heart activity. This initiates medical emergency scenario: the systems first attempts to communicate with Martha; if she does not respond or if she says she does not feel well, remote communication agent calls Martha’s carers.

As evident from the example, the model will actually be a space of several types of agents (input, scenario-processing, output...) and facts (slightly similar to Minsky’s society of agents), coupled with additional knowledge in the form of ontologies. Such an approach is sufficiently modular to cope with different sets of available sensors and comminucation capabilities and offers flexibility beyond current systems, which operate in much more narrowly defined situations.

The cognitive system will be designed and implemented to cope with the aspects listed below and adapt the approach to interaction with the user and the environment accordingly. It will start out with parameters set by the user and carers and will be capable of adapting to the user:

· User's familiarity with the interface and connected technologies

· User's habits and routine

· User's preferences

· User's physiology

· User's emotional states

· User's contacts and their status and rights

Familiarity with the interface and connected technologies will be assessed by the responses to prompts from the SPA. For example, if a user fails to communicate successfully, the system will try to communicate in a gradually simpler ways; with time, it will learn which ways work best and try those first.

The user's habits and routines can be assessed by their movements and activities at different times of the day. Some information such as medication and exercise regimes may be recommended and pre-programmed. Others such as preferred times for intellectual or social activities may need to be learnt.

The user's preferences may include dietary, routine, contacts, information sources and interaction mechanisms. General cognitive models of the elderly will be adapted to particular persons, their needs and desires, technologically based on semantic Web approaches with extensive use of ontologies, and agent-based user profiles

The user's physiology will be monitored (potentially full time) by some of the sensors and so basic phases can be collected and compared with other aspects of the daily routine to identify patterns for later statistical analysis or the recognition of problems in real time.

The user's emotional state will be extracted from audio-visual input, possibly with some assistance from the medical sensors and communication with the system. Like the assessment of the user's physiology this can be stored or used for real time problem identification.

The user's favourite contacts will become evident from their actual communications and these can be built into meaningful and manageable data sets by the application of graph theory and clustering algorithms to the raw data at the lower data and agent technology for advanced functions. The access rights of various contacts to the user's information will probably need to be set up by the healthcare workers who configure the system in conjunction with the user and carers. However we should consider a means of allowing certain combinations of user and carers to modify access (subject to appropriate security checks, which may be based on the SPAs own view of the carers and helpers identity, built up from internet searches, voice and visual information i.e. intelligent authentication).

B.1.5 Human Computer Interface and Computer to Infrastructure Interface

What are the technological challenges?

Usable video for face to face communication that is portable/wearable.

Wearable audio capture arrays for noise cancellation, speaker recognition, audio object cancellation

In addition to the challenge of smaller or more capable displays/audio we need clever ways of using the sort of PDA type audio and video that we already expect. We want to make it easier for users AND carers to use the system.

Emotion recognition by natural language recognition in noisy / unpredictable environments

Video feature extraction in unpredictable environments

But both these are covered in initial processing above.

Adaptable network connection technology

Less confusing reiteration of the question or suggestion when the user doesn't respond or responds in an unexpected manner

Use of different modalities for different user situations,

Flexible use of communications channels (last two covered in impact section but this is technology)

Matjaz - these bits fit here but aren't really talking about technology can you reword?

The system will improve life of elderly by providing a simple integrated interface using audio and visual input and output adapted to each particular user. The primary input mode will be default voice recognition; if this proves to be insufficient and if the user is willing to train the system, secondary input mode, user-trained voice recognition, will be used; if the system is not able to understand the user at all, it will revert to the tertiary input mode, keyboard and mouse. For the most critical and straightforward functions such as calls for help, special-purpose buttons and similar devices can be used. The calls will be automatically diverted to desirable directions, e.g. neighbours, relatives, police, connected SPAs etc. Output will also be both voice and visual information on computer or TV screen, depending on the location and the users’s hearing/sight. In cases of voice communication, the system will repeat, if necessary, modified and simplified questions or replies if feedback from a particular elderly user will indicate possibility of misunderstanding.

Possibly make summary information available to relatives over a web based interface. Low data rate e.g. "The user is in supermarket X talking to friend Y in the till queue". Or "He left for the library half an hour ago after visiting Doctor Z".

Interfaces with infrastructure

Importance of emergency packets and the need to get through can use reliable transports such as TCP whereas streams can use UDP or similar to get the connection speed.

B.1.6 Evaluating the solutions

The main problem with evaluating acceptability and usability of multi-sensor equipment with multi-modal Human Computer interfaces and complex reasoning algorithms, is the potentially vast matrix of stimuli and possible outcomes that need to be tested. Also the testing required for each chosen set of stimuli and outcomes can be very complex when many measurements need to be combined into an overall assessment of acceptability or usability. They also become very lengthy when a large sample of users is required to incorporate the different attitudes to and capabilities with the new device or technology.

SPARQLE will use an adaptation of the CAUTIC system to address the concern of large user samples and complex tests.

The advantages of the CAUTIC system of usability and acceptability assessment are:

· With a small number of interviews, it provides evaluation results valid not only for the persons involved, but as well for different users of the same user profile.

· By its sociological approach of technology use, focusing individual users, the method brings a unique and new analysis of use, which comes as an essential complement to other evaluation methods.

· Besides evaluating qualities of existing products and services, it also highlights missing products and services. It indicates the necessity of new work organisations in order to improve all services in a site.

B.2 Relevance to the objectives of the IST Priority

Recommended length – three pages

(Describe in detail the manner how the proposed project’s objectives contribute to the scientific, technical, wider societal and policy objectives of the IST Priority as stated in this call.)

Strategic Objective

Addressed in this proposal by:

To develop artificial systems that can interpret data arising from real-world events and processes (mainly in the form of data-streams from sensors of all types and in particular from visual and/or audio sources)

This strategic objective is fully covered in this project. A network of sensors including audio and video capturing devices will provide data streams about the user and the environment. Detecting short-term and long-term changes in the sensory data and content analysis of the acquired information constitutes the pre-processing stage for the interpretation of data arising from real-world events. These include:

· Trend analysis for continuously monitored sensory data.

· Audio content analysis for distinguishing between speech, music, silence and environmental sounds.

· Recognition of speech and environmental sounds.

· Sound source localisation using microphone arrays.

· Video content analysis for object and text recognition as well as region of interest selection.

· Visual object segmentation and tracking.

· Motion, activity and event detection.

· Joint audio and video content analysis for automatic camera aiming and increasing the reliability and robustness of recognition algorithms.

Acquire situated knowledge of their environment; act, make or suggest decisions and communicate with people on human terms, thereby supporting them in performing complex tasks.

A system with high-level cognitive capabilities for reasoning and problem solving is foreseen in this project to support the elderly in performing complex tasks – the system will perform as a kind of intelligent assistant. The detailed information about the elderly and his/her environment acquired using a network of sensors and AV capturing devices will enable a thorough analysis of situations, including the users’ physical and mental state. This will help in acquiring situated knowledge by trial and error and learning from experience. Speech recognition, speech synthesis and visual guidance are also the key interfaces of the system for two-way flow of information through speech and thereby communication on human terms.

Focus is on research into ways of endowing artificial systems with high-level cognitive capabilities, typically perception, understanding, learning, knowledge representation and deliberation, thus advancing enabling technologies for scene interpretation, natural language understanding, automated reasoning and problem-solving,

robotics and automation, that are relevant for dealing with complex real-world systems.

The system will gather information from the environment and users through voice and vision. Apart from the signal level, the input will be interpreted in two additional steps: first at the intermediate level trying to prepare all information for the higher levels of the cognitive system, and finally in order to construct a model of the current situation, a form of technical consciousness or awareness. This model will be the basis for autonomous decisions. The interpretation of the model will be subject to modifications as the system learns user’s preferences and habits.

It aims at systems that develop their reasoning, planning and communication faculties through grounding in interactive and collaborative environments, which are part of, or connected to the real world.

The system will integrate multiple sources of information: real-time data from sensors, information from the Web, communication from third perties (for example the user’s friends and relatives). The system will also communicate with multiple parties: the user, his carers and possibly others.

These systems are expected to exhibit appropriate degrees of autonomy and also to learn through “social” interaction among themselves and/or through human-agent

Co-operation; in a longer term perspective, research will explore models for cognitive traits such as affect, consciousness or theory of mind.

The proposed SPA will perform sophisticated conversion of high bandwidth multimedia streams into low bandwidth streams of meaningful object-based content. It will also extract meaningful data from large data sets gathered from its own medical sensors and the wider infrastructure connected to its wireless channels. It will execute intelligent algorithms to assess the user's situation from these inputs and then make suggestions autonomously, tailoring the mode of interaction to the user's perceived situation. The cognitive functions will be based on a limited consciousness oriented towards practical tasks, where the mind will be perceived as a set of multiple agents, scenarios, and other forms of knowledge, incorporated into an advanced agent system, similar to the Minsky’s society of the mind.

Developing models and architectures for artificial cognitive systems, emphasising higher-level cognitive functions. It should yield new approaches towards

understanding and improving cognitive capabilities in artefacts and explore new methods of integrating these in complete artificial systems.

This project represents a complete cognitive system and will break new ground in the fusion of medical sensor information with standard time and location information and user emotional information (extracted from audio-visuals) to form an overall "awareness" of the user's situation.

Work is expected to be highly interdisciplinary, drawing on appropriate fields that contribute to cognitive science and cognitive engineering: artificial intelligence, computer vision and robotics, as well as relevant branches of mathematics (e.g. dynamical systems, information theory), the bio-sciences (e.g. neuroscience) and the humanities (e.g. linguistics, philosophy).

TRT(UK) provides project management, networking and communications engineers and video processing engineers. UniS provides audio-visual processing networking and communications researchers. IJS provides cognitive systems, machine learning and data mining researchers. CEA provides a human factors evaluation team and XXX (MINIATEC) (Philippe, Michel please can you write the correct sentence here). KCDSL provides medical sensor manufacturing expertise. YYY provides access to users and their associated health workers.

B.3 Potential impact

Recommended length – three pages

(Describe the strategic impact of the proposed project, for example in reinforcing competitiveness or on solving societal problems. Describe the innovation-related activities. Describe the exploitation and/or dissemination plans which are foreseen to ensure use of the project results. Describe the added-value in carrying out the work at a European level. Indicate what account is taken of other national or international research activities.)

In recent years, the elderly population has become a significant proportion of the general community. However, this fact is often overlooked when designing the environment and public services. Also, modern technologies are aimed mainly at the young and middle-aged. This creates an undesirable equality of opportunity between the older people and the rest of the society, as the elderly are more likely to encounter difficulties taking part in fast evolving lifestyles. The advantage of the system is that it will be able to gather information and perform communication with the users in several ways and at several cognitive levels, thus enabling on-the-spot solutions to problems with cognition or even impaired senses.

Older people can be easily confused by rapid technological progress. As a result, they become more indecisive when it comes to making even simple everyday life decisions. As the societal gap between the rest of the population and the elderly widens, they may retreat into seeking companionship only from others in the same age group and this may further increase their isolation.

A rapidly ageing population also has a serious impact on the health care expenses of governments and is exacerbated if the elderly are confined to their homes or caring institutions rather than being able to integrate with the rest of society to the fullest possible degree. To limit the costs and the workload of health care workers, remote health monitoring of the elderly is a much better option for preventive health measures. Therefore, it is imperative to cater for the potential requirements of the elderly as well as monitoring their health and changing habits.

To address these problems, this project proposes to provide the required assistance to the elderly to enable greater integration with the rest of society and thus improve their quality of life. The project aims to develop an always-on smart personal assistant (SPA) with high-level cognitive capabilities to interact both with its user and the environment.

The SPA will be designed to gather information from its surroundings using a network of sensors as well as a suite of audio-visual tools. It will then use advanced artificial intelligence and scene interpretation features to enhance its understanding of the situation, location and context. By analysis of the gathered information, the PA will make adequate verbal and/or audio-visual comments and suggestions. It will do this largely autonomously.

The SPA will collect information about the elderly user, user’s routines, physiological functions and health, emotional states, relationships, and surroundings based on the audio-visual, sensory and GPS (Global Positioning System) data. It will also enhance its knowledge and intelligence through the Internet browsing and user/specialist-given data and commands. In addition, the SPA’s cognitive features will enable simple audio-visual conversation with the person in the form of intelligent multiple-choice questions and straightforward answers based on the detected situation and environment. In other words, it will act as a virtual companion and sometimes as a consultant to the elderly. The context of the conversation will be augmented by the collected data through the audio-visual units and sensors. This will facilitate automated reasoning and problem-solving capabilities whilst providing useful feedback to the user. Consequently, the user will be informed of her/his conditions as well as the surrounding environment, and automatically provided with the necessary information about potential worries and possible questions.

The use of a natural audio-visual interface between the user and the SPA will provide a number of benefits for users who may not be familiar with current standards for human computer interaction. It will allow reassuring face to face dialogue with remote healthcare workers such as doctors, carers and family members. It will allow redundancy of communications channels if one of the user's senses becomes impaired during the normal daily routine. It will also provide an intuitive easy to learn interface for any casual assistance to which the user may have access, from members of the public they meet during the day. Modern processing power will also be used to ensure that the SPA can take adaptable approaches to the unexpected input or lack of input from the user or their carers. It will also allow flexible use of the communication channels available.

It is the unique combination of rich human interface, intelligent use of the sensors and interface, user-tailored algorithms, semi-autonomous operation and communication with the wider world that makes the device appropriate and economic for the target users.

In general terms, the SPA will comprise many advantages to the elderly, as it will:

· Improve their quality of life.

· Provide continuous health monitoring, which in return will improve their health status and reduce the health care costs.

· Help them make decisions and guide them when confused.

· Highlight pre-cautionary measures against unfamiliar and unfriendly locations, conditions and/or hazardous situations.

· Act as a virtual companion and consultant at times, thus providing social care and breaking the effects of boredom and loneliness.

· Assist them in outside world as well as in home environment to benefit from the developing technologies and fast advancing lifestyles.

I have made a start on this from Safak's introduction. The items I think we still need to cover are:

Innovation related activities

The strategic reason for introducing SPARQLE is to gain advantage in this sector. Generally, Europe is lagging behind U.S.A. in several attributes of information society, e.g. there is no European counterpart to Google. There is a large potential market for systems like SPARQLE, but current commercial and scientific systems offer only very limited funcionality.

Draw on or refer to the items in section B.1, Matjaz and Safak

Should this be about the impact of the innovation on the partners

Exploitation and dissemination plans - example section for TRT (UK) please add your own

TRT(UK) Exploitation

Thales Research and Technology (UK) Ltd is the UK laboratory within the corporate research centre of Thales and is a provider of knowledge solutions to the rest of the Thales group of companies. It is a centre of excellence for a variety of advanced technologies, including navigation, networks, security and virtual collaboration solutions, which it develops for the use of the group’s businesses and operating companies.

Thales would characterize its exploitation strategy as having two tracks:

· use of SPARQLE results to directly influence products and services where Thales already has a market position and thereby improve its competitiveness through novelty

· use of SPARQLE results to enable Thales to develop new products and services in areas where the company is not currently represented, but enhancing specialized technologies that already exist within Thales.

Thales will be able to use networking, navigation and portable terminal technologies in its mobile homeland security systems. Sensor integration techniques will also be appropriate for homeland security solutions such as port security monitoring of containers. User interface technologies will be exploited in the virtual collaborative systems work as well as the experience gained from new user acceptability test methods. We also see cognitive systems experience as useful to enhance our existing information extraction and management technologies again in the field of homeland security solutions.

UniS Exploitation

The University of Surrey and the Centre for Communication Systems Research in particular have an excellent track record in intellectual property exploitation. CCSR has a number of strategic partnerships with industrial and governmental bodies. Once the intellectual property rights for the research products have been secured, the accruing SPARQLE research technologies will be put forward for exploitation and/or licensing through our spin-off technology company MulSys Ltd. This will ensure maximum returns for both the University and the SPARQLE consortium whilst providing a pathway for the inclusion of the developed systems and algorithms in future European cognitive systems related technologies. In addition, the project results will be published in international journals, conferences, workshops and symposia to facilitate widespread dissemination of the results. Such activities will be particularly beneficial for both the consortium members and the members of other NoEs, IPs, STREPs and European public. Certain technologies, such as data fusion, audio and video processing for content analysis, and context understanding, will be applicable to other areas outside of the scope of the proposal. These other areas will be considered throughout the project to highlight further areas for research, and to find alternative exploitation paths.

IJS Exploitation

Department of Intelligent Systems at Jožef Stefan Institute has a long history of both cooperation with the industry and providing free services to the public. Speech and language technologies developed in cooperation with the Amebis company have been used in their commercial products. EMA, an employment agent developed at the Department, which was gathering over 90% of all nationally available jobs from various Internet sources and was able to both speak Slovene and translate information from English into Slovene and vice versa, was one of the ten most often visited sites in Slovenia. Another example is Govorec (Speaker), a text-to-speech system we donated to the Association of Blind and Short-Sighted. Along similar lines, we will provide some prototypes of the SPQRQLE system to medical and social care institutions. We will also attempt to find partners in the industry for commercial exploatation. Finally, as a scientific institution, we will disseminate the knowledge acquired through the project to the scientific community in the form of participation in conferences and publications in scientific journals.

CEA Exploitation

Philippe, Michel, Miguel please complete

KCDSL Exploitation

Kardiosis Cardiologic Daignostic Systems Ltd. located in Technopolis zone of METU University of Ankara provides research and product development capability to its parent company , Tepa Inc. Our explotation strategy is twofold and strongly associated with our expertise in biomedical signal processing and related ,software and hardware development:

1) To extend our research activities into developing wireless sensors for elderly and to integrate these sensors in to our existing systems.

2) To increase sensitivity and specificity of our diagnostic algorithms by the exploitation of the results of the cognitive systems research and multi-sensor data fusion .

Added-value of European level

Cultural differences in attitudes to the elderly between European member states

Differences in problems of old age across Europe

Maybe refer to section on partner specialities

Account of other research programmes

Refer to partner capabilities and networks of contacts

VISNET (Unis & TRTUK networking and security)

I-Lab and human factors

A paragraph each?

TJM - off the wall idea and maybe not practical in the time we have

Could we write a short story twice in the manner of a Turing test to give a flavour of how the dialogue with the machine would differ from that with a real human. Rather omits the visual aspects we are trying to convey (unless we annotate a picture). Only if it really brings out some of the benefits in a new and vivid way. Might just grab the attention of the reviewer - a story is always easier to read that a technical document. It almost needs a comic strip treatment

TJM - Maybe this should form the basis of the detailed plan in B.6

Initial user info gathering (3-6 months)

Cut-down initial solution demonstrating sensors, basic cognitive initial HCI and comms (18-24months)

Well defined sensor interface system and manageable/meaningful sensor data to cognitive interface

Better second version with navigation and improved cognitive/HCI (30-36 months)

I think I need to do this bit as I start to build up the tasks, work packages, schedule and costs.

Above is the central thema of our proposal.

Below are some more brainstorm ideas:

- In the most downgraded version the system will be able to provide specialised interface for elderly on standard PC connected to the Internet and without any sensors. It will have specialised knowledge about elderly (based on knowledge transformed into a system), each user (agent profiling), their senses, their cognitive abilities, will be able to learn and adapt to each user, preferences, needs, abilities (reliable AI and ML techniques). For example, if a user fails to communicate successfully, the system will try to communicate in a simpler way in several attempts and then learn how to improve communication next time. The system will be able to provide social contacts to friends, relatives and medical staff through emails, chat, telephones etc. So, a user will use these specialised interfaces for

a) normal use of computer and the Internet (most elderly have problems using computers and the Internet) (this means specialised interface for the elderly, based on a simple menu with only a couple of functions),

b) serving needs of users, either social kindness (as with pets, providing puzzles, games etc.) or checking timing of therapy, meals,

c) memorising future events (organiser), simple calculations etc.

The emphasis is not advanced functions, but on simpler functions that elderly will indeed be able to use. Most of the functions in this downgraded SPA are what normal people get from normal computers already, but elderly can't, and they need advanced computer systems that can constantly adapt to elderly abilities. So this system will have nearly all SW capabilities, but no additional HW. Brainstorm menu:

+ emails (submenu: relatives, doctors, friends, ...) (the idea here is that this is extremely simple menu that fulfilles all needs and anybody can perform it)

+ internet search (Google, normal Internet ...)(similar to above)

+organiser (specialized for elderly)(similar ...)

+calculations, other simple programs (similar ...)

+games

+therapy

+instructions (a menu of all instructions that elderly users have problems with,e.g. how to use mobile phone etc. These instructions can be textual or also graphical.)

+ normal computer interface

Maybe we should define these menus in reasonable detail in the proposal, so discussion with elderly is also needed in the brainstorming phase. Also, some case examples of communications showing advantages of our approach are also desired. This might be the actual winning point for acceptance of the project.

- In the middle versions there are some sensors, but typically cheap and available in normal PC shops, such as microfones, cams etc. The system then gets additional capabilities, but basically it performs the above menu with capability to talk and understand voice commands, e.g. voice searching the Internet - which we already have for normal users for Slovenian language and

there already exist solutions for English language).

- In the most advanced version (the most demanding for implementation) the system will integrate arbitrary number of sensors into fully aware system, which will be able to judge and act on its own, so it will have to be some form of an intelligent agent from the SW point. It will also have its own functions such as alarm capabilities in case of fire, accident etc. and will be able to call for help either to relatives, neighbours, police etc. It will also call doctors if therapy will not be performed correctly etc. The communication will be either visual or voice or through mouse or keyboard ...

B.3.1 Contributions to standards

Recommended length – one page

(Describe contributions to national or international standards which may be made by the project, if any.)

This project will help to standardise interfaces for user worn sensors and sensor processing algorithms by examining interfaces of the sensors already available and bringing a number of them together for connection to a common processing platform. This will allow common interface elements to be identified and handled in a standard manner. Various algorithms will be run on the platform to process data from different groups of sensors and these algorithms will use a common programming interface to access the sensors.

MPEG-7 is a standard for describing multimedia content, and therefore it provides audio and visual descriptors and description schemes that can be used for several purposes, such as content search, retrieval, analysis, etc. This project will address the understanding of specific contexts by analysing the gathered sensory (e.g. physiological data, GPS, etc) and AV information, and thus efficient content analysis tools will be developed. The research activities on the analysis of such content will also result in the provision and specification of several new descriptors and description schemes in the forms of important clues to the sensory and AV data. The aim will be to make contributions to possible MPEG-7 extensions in line with the accruing research technologies on such data descriptors developed in the different stages of the SPARQLE project.

Contributions to standards for the small scale end of scalable presence devices?

All - Any other ideas

B.4 The consortium and project resources

Recommended length – five pages

(Describe the role of the participants and the specific skills of each of them. Show how the participants are suited and committed to the tasks assigned to them; show the complementarity between participants. Describe how the opportunity of involving SMEs has been addressed. Describe the resources, human and material, that will be deployed for the implementation of the project. Include a STREP Project Effort Form, as shown below, covering the full duration of the project. Demonstrate how the project will mobilise the critical mass of resources (personnel, equipment, finance…) necessary for success; and show that the overall financial plan for the project is adequate.)

B.4.1 Overview of the consortium

The SPARQLE consortium draws on a range of companies from across Europe with expertise from the end user through the electronic equipment providers and the software developers (with specific video and audio processing, human factors and networking experience) to the cognitive systems that bring the overall project together.

The leading partners within this consortium have excellent track records in research and technology development, they also already have experience of running and collaborating in EU projects and will be able to pass valuable experience to those who are participating for the first time.

The geographical distribution of the consortium members will enable a balanced view of how the technology can be used to assist the elderly across Europe.

The consortium includes the Research and Development arms of one worldwide electronics equipment manufacturer, two public research institutes and two academic institutions. The sensor representative is a Small to Medium sized Enterprise (SME). The consortium has specialist skills in the areas of cognitive systems, image processing, human factors, Collaborative Virtual Environments (CVEs), fixed and wireless networking, network security issues and healthcare, management, technology and procedures. One of the universities is also represented by a unit that is dedicated to working with a panel of elderly users. One of the public research institutes has access to a group of care workers in the field of geriatrics.

The following table briefly summarizes the partners' expertise that is relevant for the SPARQLE project. A more detailed description of the companies and institutes involved can be found in section B.7.3.

Michel & team, I don't think I have a partner description for MINIATEC and areas of interest yet so please can you add something appropriate here.

Participant

Name

Type

Country

Expertise relevant for the project

P01

(CO)

TRTUK

IND

UK

UK part of the corporate research company for the Thales group with expertise in:

· EU Framework project management

· Quality of service, authentication, and information security issues over wide area networks and local area networks

· Navigation techniques

P02

UniS

UNIV

UK

Centre for Communication Systems Research (CCSR) has expertise in:

· Wireless networking, multimedia communications

· Speech and audio-visual signal processing

· Content adaptation, image segmentation

· Augmented and virtual reality technologies for interactive and collaborative environments

· AV feature extraction

P03

IJS

GOV

SI

· Matjaz - please complete

Department of intelligent Systems has expertise in:

· Artificial intelligence

· Cognitive Systems

· Speech technologies

P04

CEA

GOV

FR

User lab and MINIATEC IDEAS lab, with expertise in:

· CAUTIC system for acceptability and usability tests

· 3D sensors and microsystems (motion, sound, etc)

Philippe, Michel please check and augment if possible

P05

KCDSL

SME

TR

Kardiosis Cardiological Diagnostic Systems Ltd. Has experience in:

· Cardiological sensor development and manufacture

· Ecg signal processing

· Medical application software development

· Medical device design

P06

UoD

UNIV

UK

B.4.2 The role of the partners

Please can you all indicate provisionally which work you will be involved in and whether you can accept WP leadership as suggested. If you are OK with leading a work package please can you prepare a section like the example in B.6 for you package. Probably leave time scale off for now but try to estimate how much effort you need to put in.

The following table shows in brief the role of the partners in the project and their contributions to the SPARQLE work-packages, divided into two categories: The main efforts in partner’s own area of responsibility (bold) and contributions to other work packages (italic).

Participant

Role in the project

Contributions (main contributions highlighted)

P01

TRTUK

Coordinator

WP0 (Project management)

Leadership of WP5 (Security and Privacy)

Leadership of WP8 (Dissemination and exploitation activities)

System requirements & architecture contribution (WP1)

Contribution to navigation work (WP2)

Contribution to social factors testing (WP6)

Exploitation and dissemination within Thales group (WP8)

P02

UniS

Contractor

Leadership of WP2 (Sensor networks, GPS and communication technologies)

Leadership of WP3 (Data gathering/fusion, content processing and analysis of this content)

Contribution to data gathering (WP6)

Exploitation and dissemination through spin-off companies and publications (WP8)

P03

IJS

Contractor

Leadership of WP4? (Provision of high-level cognitive capabilities)

Contribution to Usability and acceptability tests (WP7)

Exploitation and dissemination through medical and social care institutions and scientific publications (WP8)

P04

CEA

Contractor

Leadership of WP1? (Scenarios, user requirements and socio-economic impact, HCI)

Leadership of WP7? (Usability and acceptability tests)

Contribution to Usability and acceptability tests (WP7)

Exploitation and dissemination within ?? (WP8)

P05

KCDSL

Contractor

Leadership of WP6? (Integration and build)

Contribution to Usability and acceptability tests (WP7)

Exploitation and dissemination within ?? (WP8)

P06

User Group

Contractor

Contribution to Usability and acceptability tests (WP7)

Exploitation and dissemination within ?? (WP8)

P07

Contractor

B.4.3 Key personnel, partner resources and background technology

The following table summarizes in brief the key personnel of the partners as well as the resources and the background technology, which the partners will bring into the project. A more detailed description of the key personnel’s expertise is given in the Annex in section B.7.3.

Participant

Key personal

Resources and background technology

P01

TRTUK

Tim Masterton

Chris Firth

TRT (UK) employs approximately 120 staff of which some 80% are science, maths or engineering graduates. The activities within TRT(UK) are subdivided into 4 labs; Navigation, Sensors & Signal Processing, Network Systems and Virtual Collaboration Systems. The Network Systems Group specialises in technologies associated with modern communication networks, security and vision processing.

P02

UniS

Ahmet Kondoz

Safak Dogan

Banu Gunel

I-lab: New multi-disciplinary research Lab at UniS with unique facilities for mixed reality visualization systems, high-end computing graphics-capable servers and wireless media technologies as well as audio and video capturing units and processing systems. The lab facilities also include access to a campus-wide WLAN as well as a wide variety of wireless channel simulators.

P03

IJS

Matjaz Gams

Matjaz - please complete

The Department of Intelligent Systems at the Jožef Stefan Institute consists of 20 researchers. The main research topics are artificial intelligence, machine learning, cognitive science, agent systems, man/machine studies, text mining, semantic Web, medical informatics, inductive logic programming, automated knowledge synthesis, qualitative modeling, decision support, genetic algorithms, knowledge based systems, heuristic programming, knowledge acquisition.

P04

CEA

Michel Ida

Phillipe Mallien

Philippe, Michel - please complete

P05

KCDSL

Oguz Tanrisever

Kardiosis Ltd. is the R&D arm of the TEPA Inc., a manufacturer of PC based ECG systems. Located in METU University TechnoPolis area, Kardiosis has more than 15 years of experience in medical device technology, particularly in cardiology. Capabilites include ECG signal analysis, analogue and digital design, firmware development and application software development in C# .Net.

P06

User Group

P07

STREP Project Effort Form

Full duration of project

(insert person-months for activities in which partners are involved)

Project acronym -

TRT(UK)

UniS

IJS

CEA

KCDSL

User Group

TOTAL PARTNERS

Research/innovation activities

WP name

WP name

WP name

etc

Total research/innovation

Demonstration activities

WP name

WP name

WP name

etc

Total demonstration

Consortium management activities

WP name

WP name

WP name

etc

Total consortium management

TOTAL ACTIVITIES

B.4.4 Sub-contracting

Recommended length – one page

(If any part of the work is foreseen to be sub-contracted by the participant responsible for it, describe the work involved and explain why a sub-contract approach has been chosen for it.)

B.4.5 Other countries

Recommended length –one page

(If one or more of the participants is based outside of the EU Member and Associated states, explain in terms of the project’s objectives why this/these participants have been included, describe the level of importance of their contribution to the project.)

B.5 Project management

TJM to consider tailoring this as it looks a bit heavy for 5/6 partners - low priority

Recommended length –three pages

(Describe the organisation, management and decision making structures of the project. Describe the plan for the management of knowledge, of intellectual property and of other innovation-related activities arising in the project.)

B.5.1 Project Structure

SPARQLE requires a number of different types of technology to be investigated, selected, extensively developed and then integrated into a working demonstration. An efficient, hierarchical work breakdown steers the project structure towards technology based work packages, so that low coupling is required between the major research and development efforts inside them. However there must also be a major effort involving all partners at the beginning of the project to agree more detailed aims for each partner than it is possible to present in this proposal. There must also be an intensive period of working together at the end to integrate a professional demonstration.

The need to co-ordinate meetings, reviews, audits, results, presentation and dissemination activities both between partners and between the project and the commission requires a separate block of effort for project co-ordination. User trials have been separated from the integration activity to allow the leadership of the two activities to be split if necessary.

This leads to the work package list given below:

These titles are different than in workpackage descriptions.

WP0 - Project management

WP1 - Scenarios, user requirements and socio-economic impact, HCI

WP2 - Sensor networks, GPS and communication technologies

WP3 - Data gathering/fusion, content processing and analysis of this content

WP4 - Provision of high-level cognitive capabilities

WP5 - Security and privacy

WP6 - Integration and build

WP7 - Usability and acceptability tests

WP8 - Dissemination and exploitation activities

B.5.2 Project Boards

SPARQLE will have two main control bodies. The Project Control Board (PCB) will be responsible for general project administration, contractual and financial issues, interfacing with the European Commission and information dissemination. It will be composed of the Project Co-ordinator and a representative from each of the participants.

The Technical Control Board (TCB) will be responsible for ensuring that the correct technical information flows occur between the participants to achieve the technical objectives and that technical progress is proceeding to plan. The TCB will advise the PCB if completion of the technical objectives requires changes to the project strategy. The TCB will be composed of a Technical Co-ordinator and the Work Package Leaders. The TCB will also be responsible for final approval of project deliverables.

B.5.3 Organisation Chart

Figure 2 shows the organisation chart for SPARQLE.

Typo: an!

Project

Co-ordinator

Participant

Representatives

Technical

Co-ordinator

PCB

TCB

Work Package

Teams

Work Package

Leaders

External

communication

with the

commission an

dissemination

B.5.4 Responsibilities

The Project Co-ordinator is responsible for:

· Day to day organisation of the project

· The first point of contact between the project and the outside world

· Organising and chairing the PCB meetings

· Execution of PCB decisions

· Ensuring that PCB meeting actions are recorded and circulated to the attendees and any other parties affected by the actions

· Organising the resolution of PCB disagreements which cannot be resolved by consensus

The Technical Co-ordinator is responsible for:

· Ensuring that the project's technical objectives are met

· Co-ordination of technical activities across the technical work packages

· Organising and chairing the TCB meetings

· Ensuring that TCB meeting actions are recorded and circulated to the attendees and any other parties affected by the actions

· Promoting visibility in the international forums

Work Package Leaders are responsible for:

· Initial detailed planning of their work packages

· The timely production and peer review of project deliverables

· Reporting work package progress to the TCB

Participant representatives look after their company's interests on the PCB and provide a balanced forum for resolving problems that span the consortium and the project.

Work Package Teams will perform the technical work under the direction of a work package leader.

B.5.5 Consortium Meetings

The PCB shall meet every three months in order to check and supervise the progress of the project. These meetings are expected to rotate around the partner sites to give all partners a view of the work in progress. Where practical these meetings will be timed to coincide with dissemination activities to minimise travel costs. Additional allowance has been made for 6 monthly technical meetings, which may be full meetings or a number of work package specific meetings. It is expected that most of the TCB work (e.g. reviewing) will be performed by E-mail. Extraordinary meetings will be handled by audio and video conferencing facilities wherever practical, and in extreme cases by moving the next planned consortium meeting.

B.5.6 Decision Making Process

The highest decision-making body within the project will be the PCB, constituted as described previously. Decisions will normally be made by seeking consensus. However, if consensus cannot be reached after a reasonable amount of time has been allowed for illustration and defence of conflicting positions, the Project Co-ordinator will be charged with the responsibility of reaching an equitable solution, to prevent deadlock in the project's operational progress.

B.5.7 Progress Reporting

Reporting will be by Bi-monthly Management Reports to be submitted by each partner to the Project Co-ordinator. The Project Co-ordinator will consolidate these into an overall Project Bi-monthly Management Report, which will be submitted to the EC. The reports will detail work in progress, milestones achieved, problems outstanding and actions for the following period. It will also identify proposed dissemination activities as they are decided.

B.5.8 Communication Flow

Outside the formal PCB and technical meetings, the consortium will usually communicate through electronic means. Mailing lists and a project web site will be set up with public and consortium private access. The consortium will make use of telephone and videoconference where practical to reduce the overheads incurred by physical travel.

B.5.9 Quality Management

The high quality of the output of the project (in terms of Deliverables, other project documents and software code) will be ensured using the in-house procedures of the participants, subject to the following basic minimum standards:

· A common Microsoft Word document template (or set of templates) is used for all deliverable documents to provide a consistent look and feel and reviewing practices

· A commercial version control tool is used to record deliverable document versions and software source code and configurations

· A defined coding standard is used for software produced

· A defined software development process is used for any new software components to ensure proper design, implementation, module test, integration, system test and documentation

At least one complete draft deliverable will be prepared, at least one month before the planned delivery date (apart from management reports) and distributed to all partners for review before a defined date. Approval of the final version will be by consensus of all partners by the defined date, before issue to the EC.

Documents other than deliverables, such as conference papers, presentations, etc., representing the results of project work will be provided for feedback and approval to all partners before being submitted for publication.

B.5.10 Confidentiality and IPR (External) Handling

Matters related to Confidentiality and IPR handling will be explicitly defined in the consortium agreement for the SPARQLE project.

Any material of a confidential nature supplied to the SPARQLE project will remain strictly for the information of the project members and it will not be disclosed to any other party without the explicit authorisation from the information owner.

Material obtained during the project and subject to confidentiality and IPR handling will be typically:

· Information or material from one project partner, which may be used by project partners only, but which remains the property of the originator. In this case, an agreement may be entered into with partner(s) for information needed for the project on a need-to-know basis.

· Information or material, which is jointly derived by the project partners during the project and which may remain confidential to the project partners as a project asset.

B.5.11 Management Plan for Knowledge and Intellectual Property

The project will enforce specific measures to manage knowledge and intellectual property generated during the project. The main vehicles for information distribution during the project will be:

· Internal Reports

· Deliverables

· Publications & Presentations

· Miscellaneous documents (e.g. White Papers, Website material).

While the Consortium Agreement will describe the precise procedures to be applied to control knowledge and IPR, the main features are described here.

Special care needs to be taken during the project to prevent the unintentional release of potentially patentable material, before patent approval procedures have been initiated. Consequently, the project will introduce two instruments to assist with the management of knowledge and IPR during the project:

· Publication Proposal Form

· Publication Register.

Researchers will be required to submit a Publication Proposal Form for approval prior to submitting a paper or presentation to an external publisher. The form will be submitted to the PCB and will provide brief details of the proposed disclosure. The PCB will consider if the disclosure might contain patentable material and will make a decision about how to protect the information. It is possible that the publication proposal will be denied or delayed until the patent application procedures have progressed sufficiently to protect the intellectual property. The PCB will reach a conclusion on publication requests within three weeks of receiving the Publication Proposal Form to prevent undue delay to researchers as they try to get their work published.

As part of its knowledge management, the project will ensure that a copy of all publications (Deliverables, Conference papers, White Papers, etc.) generated is kept by the Technical Co-ordinator. Generally, these copies will be accessible via the project website, subject to the copyright restrictions of the relevant publishers. Private Deliverables will only be available in the secure area of the website. The Project Co-ordinator will be responsible for keeping a register of all publications and their status.

B.5.12 Project Reviews

An extended project launch meeting will be held (2-3 days to develop a good working and personal relationship between the partners and allow time to thoroughly plan the exact direction which the individual partners will need to take to achieve the goals set out in this document.

A review of the requirements and design will be conducted before extensive development work is performed.

A review of the interim system will be conducted during or at the end of the user trials to consider any changes that are required to the system or testing methods that can better achieve the project objectives.

Peer reviews will be conducted on samples of new or heavily modified software to maintain high quality.

B.5.13 Risk Management

The main risk management strategy will be to perform two rounds of integration and test. It is important not to leave the individual technology strands to diverge for the entire project before trying to integrate them. The only way of ensuring that incompatible technology is not developed is to bring the strands together mid-project and make the important underlying functionality work in a controlled mini-demonstration with certain key features to test the underlying functionality. This corresponds to the modern practice of "spiral" (iterative) development in preference to a "waterfall" model. However, in a project of this size, no more than 2 iterations (interim and final) are justified.

Further specific risk issues are considered in section

B.6 Detailed implementation plan

This section describes in detail the work planned to achieve the objectives for the full duration of the of the proposed project. The recommended length, excluding the forms specified below, is up to 15 pages. An introduction should explain the structure of this workplan plan and how the plan will lead the participants to achieve the objectives. The workplan should be broken down according to types of activities: Research, technological development and innovation related activities, demonstration activities and project management activities. It should identify significant risks, and contingency plans for these. The plan must for each type of activity be broken down into workpackages (WPs) which should follow the logical phases of the project, and include management of the project and assessment of progress and results

The number of workpackages used must be appropriate to the complexity of the work and the overall value of the proposed project. Each workpackage should be a major sub-division of the proposed project and should also have a verifiable end-point - normally a deliverable or an important milestone in the overall project. The planning should be sufficiently detailed to justify the proposed effort and allow progress monitoring by the Commission – the day-to-day management of the project by the consortium may require a more detailed plan)

a) Detailed implementation plan introduction

(explaining the structure of this plan and the overall methodology used to achieve the objectives)

I will start to think about this but please let me know if you have any suggestions. I feel that we need to pull something together early (half way?) to check that we have all the right links and procedures in place. Perhaps one of Matjaz's low end solutions from the brainstorm. Mainly HCI and not many sensors? Comms and no nav?

Dissemination & management will be span tasks

Requirements up front

We need sensor interface evaluation early on if we are to provide the standardisation I am claiming. May need interface boxes to translate from proprietary to a common standard that we choose or propose. May need a software layer or configuration tool to assist with the interface.

User tests to the middle (HCI & comms) and end (Cognitive & nav)

Other tasks in the middle

We need some ideas about the platform

PDA/Laptop based ? (maybe a backpack for demonstration)

Safak what sort of AV quality do you envisage?

Sensor interfaces (hardware in computing platform?) wireless?

Comms technology WLAN/cellular?

Need plug-ins for medical sensors, nav, comms

Oguz, can you indicate what sort of interfaces are used for medical sensors in general

Despite the technological developments of sensing and monitoring devices, issues related to system integration, sensor miniaturization, low-power sensor interface circuitry design, wireless telemetric links and signal processing have still to be investigated. General interface requirements for medical sensors are mostly dependent on the bandwidth requirements, power requirements and typical usage scenario of the specific sensor.

When the sensor is part of an embedded microcontroller system, a suitable serial interface (rs232, I2C, SPI, microwire, OneWire) or a parallel interface can be selected that matches the bandwidth and power requirements of the sensor. For an ECG application, the acquisition device itself can be named as a complete sensor. In this case Bluetooth, usb or rs232 data interfaces can be used.

Sensors detail

State-of-the-art medical sensors tend to be bulky, uncomfortable to wear and need to be wired to battery packs and monitoring equipment, which may also be bulky, and in some cases will need to be attached to the mains as well.

GPS and cellular communications devices are already small enough to be wearable but the challenge here is to provide continuous access at low power while the user is on the move through various types of wireless coverage and radio propagation conditions.

GPS and low cost attitude sensors will be used to determine the user's location and orientation. This has the obvious benefits of allowing the user, the SPA and if necessary other networked support workers to track the user. However, it can also be used to assist the cognitive system with viewpoint, scenery, and communications options aspects of understanding the user's situation. Again these sensors will need to be low power, lightweight and simply connected to the SPAs sensor interface bus.

An IP based communications protocol will be used to keep the SPA (and the user if required) aware of the available communications channels and those that have been recently visited.

In conditions of stress the user will probably become static thus easing the communications problem and allowing other help to be involved in the loop e.g. a helper's using a number from the user's address book to contact help via their own mobile phone. This allows us to demonstrate benefits and start to explore cross network hand-offs with out the absolute need for it to be fully successful for a useful system. This fact makes this an ideal exploratory application for mobile monitoring and care.

Oguz - Please can you build a workpackage around your suggested sensor work but I think that we need to add something to it to about standardisation of sensor interfaces in general to improve the scalability of the platform, contribute to European and global standards and provide exploitation possibilities. Some general description is needed here with the detail in a work package description like the sample further down.

An interface between the sensor and PDA can either be wireless (Bluetooth, infra-red, etc) or wired (USB, rs232).

As for a medical sensor, I propose a single channel ECG module with either USB or Bluetooth connection to PDA. This module will require firmware and hardware development on the module side. On the PDA or PC side we need to develop a device driver to communicate with USB or Bluetooth drivers, and an API to acquire and present signals on the PDA screen. Our development is mainly done on C# and visual studio.net environment.

An ECG module can help us to demonstrate following:

· Home Monitoring: Monitoring of heart activity when required by a physician

· Fitness Monitoring: An elderly can wear this device while doing exercise (walking, riding a bicycle, running on a treadmill, etc) and monitor his heart rate .Life threatening arrhythmias can be sensed by signal processing and the elderly can be warned acoustically and visually by the PDA to stop the exercise

· Cardiac Rehabilitation and Cardiovascular screening

· Can be a starting point for other sensors.

WP1

Requirement analysis and specifications for the ECG sensor

1

WP2

Analog input stage and embedded controller hardware design.

4

WP3

Selection of Bluetooth module and integration

4

WP4

Embedded firmware development including bluetooth

4

WP5

Device driver and API development for the mobile device

4

WP6

ECG application software development for the mobile device

6

WP7

Sample production

3

WP7

Tests and verification

2

Context extraction detail

Audio

Data

Classification

and

Segmentation

Source

Localisation/

Beamforming

Speaker

Identification

Speech

Recognition

Object

Detection

Video

DataSegmentation

Face

Recognition

Object

Recognition

Text

Recognition

Sensory

Data

Trend

Analysis

Activity

Detection

Position

Detection

Environmental

Data

Extraction

Context

Extraction

Cognition

Emergency

Shortcut

Audio control

Video control

Sensor control

Figure 3 - Context analysis and extraction

Cognitive Systems detail

SPA will enabling better use of technical services and devices by

· Performing specific functions on their own when necessary, or through the integrated interface under user demands, e.g. phoning doctor or sending an email.

· Enabling help for use of common technical devices such as TVs, mobile phones, stationary phones, kitchen devices such as microwave .., washing machines etc. Basic knowledge will be incorporated in advance, and specific adaptations will be made for example by some relative of elderly who will provide specific help for desired functions of a particular elderly. The system will enable simple and efficient exchange of knowledge about use of specific devices through SPAs that will communicate with each other through the Internet based on multi-agent approaches.

It will have specialised knowledge about elderly (based on knowledge transformed into a system), each user (agent profiling), their senses, their cognitive abilities, will be able to learn and adapt to each user, preferences, needs, abilities (reliable AI and ML techniques

SPARQLE will improve social life by enabling simple communication through phones, cameras. e-mails, through chats and other internet functions. These functions will be specific adaptable to allow different levels of intimacy to relatives, friends, e-mail users etc. For example, relatives could be given rights to survey specific sensors and input devices on their own, thus enabling safety control over home, devices and first of all elderly themselves. Major advantage in communication over existing ones will be interaction/communication through a broad spectrum of HW and different cognitive levels/functions. The system will enable communication even for physically and mentally heavily impaired persons as well as for normal elderly.

b) Work planning, showing the timing of the different WPs and their tasks

(Insert Gantt chart or similar)

c) Graphical presentation of the components, showing their interdependencies

(Insert Pert diagram or similar)

d) Detailed work description broken down into workpackages:

(Workpackage list, use Workpackage list form below)

Workpackage list

Work-packageNo

Workpackage title

Lead contractorNo

Person-months

Startmonth

Endmonth

Deliv-erableNo

TOTAL

(Deliverables list, use Deliverables list form below)

Deliverables list

DeliverableNo

Deliverable title

Delivery date

Nature

Disseminationlevel

(Description of each workpackage, use Workpackage description form below, one per workpackage)

Workpackage description

Workpackage number

0

Start date or starting event:

T0+?

Workpackage title Project management

Participant id

Person-months per participant:

Objectives

(Participants: TRT (UK), UniS, IJS, CEA, KCDSL)

· To manage the project to the agreed schedule

· To provide an interface to the EU and regular progress reports

Description of work

See section B.5 for detail

Deliverables

D1.1 (Month 3) Project management plan

D1.2 (Month 2,4 etc to the end) Bi-monthly progress reports

Milestones and expected result

MS1.1 (Month W)

MS1.2 (Month M)

Workpackage description

Workpackage number

1

Start date or starting event:

T0+?

Workpackage title Scenarios, user requirements and socio-economic impact, HCI

Participant id

Person-months per participant:

Objectives

Michel, Philippe please can you augment the WP description

Description of work

T1.1 Identification of Actual Users (T0+? to T0+?m):

As the term elderly is all inclusive, we need to categorise the users into groups for example by physical abilities, cognitive abilities, levels of health-care required, or even by specific ailments. These diverse groups are best graded by health care professionals. The target users need to be identified. Users of the SPA (depending on the design concept) may also include health carers and other persons in contact with the elderly although their context of use is different. Actual users must be recruited for the evaluation. Users’ needs and abilities will in return dictate the detailed components of the design. Users’ characteristics and needs studies will provide a starting point, through the prototyping stages, and up to the final delivery.

T1.2 Longitudinal survey of users’ activity patterns (T0+? to T0+?m):

A realistic longitudinal survey of users’ activity patterns (including the type, location, duration, travel mode, other involved persons, etc) will provide the data of existing activities which can be used to predict more accurately the types of activities that users are likely to engage in – a case of “conjecture and confirm” type of intelligent process.

Deliverables

D1.1 (Month X)

D1.2 (Month Y)

D3.3 (Month Z)

Milestones and expected result

MS1.1 (Month W)

MS1.2 (Month M)

Workpackage description

Workpackage number

2

Start date or starting event:

T0+?

Workpackage title Data gathering, fusion and compression

Participant id

Person-months per participant:

Objectives

(Participants: TRT (UK), UniS, KCDSL)

· To collate data through the use of sensors and AV capture devices, and devise data fusion techniques to refine the information in a meaningful way.

· To reduce the amount of the sensory and AV data for efficient storage and transmission purposes via compression.

Description of work

T2.1 Sensor Networks (T0+? to T0+??):

In this task, a number of off-the-shelf sensor units will be utilised to provide the SPA with the necessary inputs on several physiological data (e.g. body temperature, heart beat rate, blood pressure level, rate of breathing, etc) of an elderly user as well as the environmental parameters (e.g. humidity in air, outside temperature, quality of air, darkness/illumination, etc). Methods will be examined to link them together in the most effective way to form a network of sensors for enabling the optimum collection of such external data. The location/position of the user will be gathered through the GPS navigation system, which is classified as a part of the sensory information in this project. An automatic command and control mechanism will be devised to select the operation of the sensors autonomously with respect to their operation reliability, power consumption aspects, and adequacy to the required purposes. For this reason, feedback will be provided from the context detection and cognitive systems blocks, and a built-in functional test algorithm will allow for the timely operation of various sensors, so as to assist each other when a particular context is perceived.

T2.2 Audio-Visual Tools (T0+?? to T0+??):

This task will investigate various light weight and robust audio-visual capturing units that provide good quality data and have low power consumption. The quality of the captured audio and video carries great importance for content analysis. Therefore, a microphone array which enables post-processing will be utilised for capturing audio. The size and geometry of the array as well as the sampling rate and structure-dependent processing methods will be taken into account. For the selection of the video capturing unit, the frame size, capture rate and digital focusing, zooming and brightness adjustment functions will be some of the aspects to be considered.

T2.3 Data Gathering and Fusion using Sensors and AV Capturing Units (T0+?? to T0+??):

This task will perform capturing of the data using a suite of sensors and AV recording devices. The data capture will particularly focus on recording the important aspec