Upload
blaise-richards
View
229
Download
0
Tags:
Embed Size (px)
Citation preview
WP 7 - Regola
TASK 7.4 : Pilot Application Validation
D 7.4 Functional and Non-functionalEvaluation Criteria for C2-SENSE
Pilot Applications
WP 7.4 – Pilot application validation
In this task, the testing environment for the C2-SENSE pilot application, which will be
developed on top of the integrated C2-SENSE components, will be constructed. The
assessment criteria will be defined as test case scenarios. The required tests that will be
performed on the developed pilot application for the thoroughly examination of the system
will be defined. The realized system for the pilot application scenarios will be tested as a
whole in the test environment by the end users. The tests will be in parallel with the
deployment of the pilot application. The feedback from the users will be collected through
questionnaires which will be provided to the system developers to enhance the
functionalities of the C2-SENSE final product.
WP 7.4 - concept
• Verification : technical test works. The product is produced right.
• Validation: User approve the functions. It’s the right product
• Acceptance test: is the final test before a delivery of oa product or a service
Test case: How? Why? Wich purpose?
WP 7.4 - concept
VALIDATION ACCEPTANCE TEST
EU Research, Not a product
technical aspects difficult to evaluate
C2-SENSE VALIDATION
This is the challenge
WP 7.4 - Evaluation Team
• Domain and IT Expert (REGOLA)
• User operator ( Regione Puglia, variuous org.)
• Technical expert of C2-SENSE components (Lutech, SRDC, SAGEM)
WP 7.4 - Object of Evaluation
• FUNCTIONAL REQUIREMENT: what C2-SENSE DO
• NON FUNCTIONAL REQUIREMENT: (simplifying) How C2-SENSE do it? It’s a good solution?
WP 7.4 - Evaluation Process
InspecionAnalysis
QuestionnaireSCORE REPORT
EVALUATION
PILOT APPLICATION EXECUTION
Function evaluationNon functional evaluation
WP 7.4 - Functional Evaluation
FUNCTIONAL EVALUATION
WP 7.4 – Functional evaluation
CFC system
National wheather service
CFD system
Regional civil protection service: SOIR
Dewetra
SOIR SystemCP Aletring
System
CFD Sensor Monitoring
System
Regional civil protection service: CFD
Volunteer Organization
VOL System
National operation room system
Department of Civil Protection
Prov GIS
Province
POAR alerting system
Press Office Apulia Region
POAR system
CCS System
Prefecture
Fire Army System
Fire Army
Army System
Army
Citizen App
Citizen
GIS
GIS
GIS
GIS Healthcare
Information System
hospital
MED system
Medical Emergency Dep
COC System
COC (Municipality)
COC Aletring SystemGIS
GIS
Architecture of the pilot application
WP 7.4 – Functional evaluation
WP 7.4 - Functional Evaluation
The idea is: To evaluate C2-SENSE component indirectly (where possible) examining the effects on the user operation on the pilot applicatoin
WP 7.4 - Functional Evaluation - the scopes
Scope4.1.2.1 Situation Reporting4.1.2.2 Mission Plan4.1.2.3 Scheduling4.1.2.4 Resource Management4.1.2.5 Alert4.1.2.6 Hospital Communication4.1.2.7 Tracking of Citizens4.1.2.8 Sensor Management4.1.2.9 Enterprise User Authentication and Authorization (EUAA)4.1.2.10 Audit Trail and Node Authentication (ATNA)4.1.2.11 Emergency Situation Map4.1.2.12 Situation Analysis4.1.2.13 Permission
• Action Scope are the ‘functional area of investigation’ of the Functional Validation process
• The idea is to identify the ‘integration profile’ as Scope of investigation.
WP 7.4 - Functional Evaluation
Objective Evaluation(the system work as
espected)
Subjective Evaluation(operator approve the behaviour of the solution)
Global Evaluation
WP 7.4 - Functional Objective Evaluation
Objective EvaluationN.
Scenario Referenc
eDay Hour Step Case code scope
Institution / Agency /
Actor
Initiator System
Target systemAction simulated test modality evaluation Data structure/data type annotation Open Point to Deploy
1 0 0 AM0000 01 D0-AM0000-01 configuration SOIR SOIR System CP alerting system
preparation of the enviroment. CP receive alerting data form municipalicies
Idea: minimun 10 municipality, 2 prefectures. Several channel
• configure CP alerting system with alerting escalation plan of interested stackeolder (municipality, prefectures and organizations)• Query CP alerting system to check configuration settings
CP alerting system is configured Common NOWTICE instance?
2 0 0 AM0000 02 D0-AM0000-02 configuration Municipality COC System COC alerting system
preparation of the enviroment. COC configure massive alert for the population
Idea: minimun 10 municipality, 2 prefectures. Several channel
• configure COC alerting system with alerting escalation plan of interested stackeolder (volunteer organization)• Query COC alerting system to check configuration settings
COC alerting system is configured Common NOWTICE instance?
3 0 0 AM0000 03 D0-AM0000-03Resource
ManagementMunicipality COC System SOIR System
Communication is necessary because SOIR can involve volunteer services directly
• From SOIR System Send a list of possibly involved Volunteer organization to SOIR System
The data is sent and user receive feedback of delivery Wich system uses SOIR? Simulation UI?
4 1 1 PM0330 01 D1-PM0330-01Situation Reporting
National wheather
serviceCFC system
all interested system, mostly CFD
- makes weather forecast (Forecasts are published on a web site)- notify to other system that a new forecast is available.
human polling on national weather service website. The department pubblish data that CFD operator elaborates. This step will be executed by uman operation only. Then a bullettin is sent to se collaboration enviroment (weather forecast pdf)
The document is sent and CFD receive feedback of delivery in order to be sure the message was received by all the actor connected
Wich system uses CFD? Simulation UI?
5 1 1 PM0330 02 D1-PM0330-02 NONE CFD NONE NONE
• CFD personnel ask to add a new sensor in a specific place. So they define the type and location• Contact specialists to install new sensor• When the sensor is ready and working network sends data to CFD system
It is a step necessary to go further. This line in the sheet it is just a reminder. Sensor have to be installed defining: geografic coordinate, and sensor type, activation time
6 1 1 PM0330 03 D1-PM0330-03Sensor
ManagementCFD Sensor
CFD - Sensor data DBOTHER SYSTEM
The sensor starts to send value sensor data arrive to C2Sense collaboration enviroment and they are delivered to all interested party that are:- mapping tool: display position of the sensor and last value- CFD Sensor data DB
to define CFD tool to simuate this scenario
7 1 1 PM0330 04 D1-PM0330-04Sensor
ManagementCFD
CFD Sensor Monitoring
System (C2Sense)
CFD - Sensor data DB
• check sensor network each 30mins Using a tool (to define) CFD operator read last sensor data Sensor data have to be reguralrly updated to define CFD tool to simuate this scenario
8 1 1 PM0330 05 D1-PM0330-05Sensor
ManagementCFD CFD System
CFD Sensor Monitoring System
• query the Sensor Monitoring system about current data of the deployed sensor• the sensor reply to the request in realtime
Using a tool (to define) CFD query the sensor Last sensor data have to be dispayed on the screen to define CFD tool to simuate this scenario
9 2 1 PM0330 06 D1-PM0330-06Situation Reporting
CFD Dewetra CFC System
• Share bullettin with CFC Network The department pubblish data that CFD operator elaborates. This step will be executed by uman operation only. Then a bullettin is sent to se collaboration enviroment (weather forecast pdf)
The document is sent and CFD receive feedback of delivery in order to be sure the message was received by all the actor connected
to define CFD tool to simuate this scenario
10 2 1 PM0330 08 D1-PM0330-08 NONE CFD CFD System all interested system• publishes bulletin of regional criticity (operator of cfd does that) It is a step necessary to go further. This line in the sheet it is
just a reminder. Sensor have to be installed defining: geografic coordinate, and sensor type, activation time
Human operation of CFD operators its website. Nothing to evaluate
11 3 1 PM0330 07 D1-PM0330-07 AlertRegional
Responsible
Regional Responsible (president of
region)
CP alerting system
• Regional responsible receive the document from CFD system (case D1-PM0330-07) , declare alert state and contact CP alert system to alerts stackeholder• It send an alerting order message to CP alerting system
The regional responsible activate an alert escalation to the configured municipality. 'Regional Responsible' has not a system. From a 'Profile test interface' the initiator send a message to 'alert stackeholder'. The message can contain PDF document, text of wheater phenomena, criticity level foreach alert zone This 'profile test interface' needs to show te response answer from CP alerting system
The activation is enabled on the CP activation system and the responsible recive, on its ui, a feedback of the result
to define wich tool to simuate this scenario. The regional responsible have to receive the documentation (email?)
12 3 1 PM0330 08 D1-PM0330-08 Alert SOIR SOIR System CP alerting system
•Share alerting message to stakeholder previously configured.• Messages will be send via fax, sms, email to the defined stakeholder• After 30 minutes send a delivery report to SOIR system to highlight about not delivered messages
From a 'Profile test interface' the initiator send a message to 'alert stackeholder'. The message can contain PDF document, text of wheater phenomena, criticity level foreach alert zone This 'profile test interface' needs to show te response answer from CP alerting system
The execution of the alerting plan will happen with CP Alerting system (NowTice)
To define SOIR tool to initiate the alerting
13 3 1 PM0330 09 D1-PM0330-09 Alert SOIRCP alerting
systemSOIR System
CP alerting system produce a delivery report to the initiator Automatic. The alerting system perform this operation automatically
SOIR receive the alerting report. From nowtice To define SOIR tool to initiate the alerting
14 6 2 PM0400 01 D2-PM0400-01Alert
(Notification)SOIR SOIR System CP alerting system
• Send a message to CP Alerting system in order to inform people of Local Authority involved of bad wheather condition with the suggestion to apply the civil protection plan. • 20 minutes maximun to alert all local authorities
From a 'Profile test interface' the initiator send a message to 'alert stackeholder'. If possible the message must carry the emergency procedure aproved by Civil Protection. Alerting system can perform this operation only with selected media (eg email, fax…)
The execution of the alerting plan will happen with CP Alerting system (NowTice)
To define SOIR tool to initiate the alerting
High Level test case
SCORE organized per ‘Action scope’
WP 7.4 - Functional Objective Evaluation
Score are assigend according a pre-defined schema
score meaning0 Function not covered by the solution at all.
1Function covered minimally, but is not sufficient and cannot be accepted in the real world.
2
Function covered, but some critical/important issue was detected (Eg: cost too high, bad performance…). The IA can accept the issue temporarly with the promise of a fix.
3Function covered and it is coherent with the reference. Some issue was identified, but a user can accept the solution.
4Function covered, it is coherent with the reference and no issue was identified.
WP 7.4 - Functional Objective Evaluation
Report contain annotation and a
radar chart
REPORT
WP 7.4 - Functional Evaluation
FUNCTIONAL SUBJECTIVE EVALUATION
WP 7.4 - Functional Subjective Evaluation
The idea is: TO ASK to the operator a subjective evaluation of the soluiton, using a pre-defined questionnaire organaized by scope
BUT C2-SENSE is not a product: for this reason the interdisciplinary team is mandatory, in order to explain user to asnwer!!
Regola (Domain and IT expert) will drive the survey.
The scopes are the same of the functional objective evaluation
WP 7.4 - Functional Subjective Evaluation
Example of subjective evaluation survey
Scope Subjective evaluation criteria
4.1.2.1 Situation Reporting
- the solution handles all the information you need?- the solution reduce the operator’s work load?- the solution decreases your response time in emergency management?- the solution decreases the loss of information?- overall impression from 0 to 4.?
4.1.2.2 Mission Plan
- the solution handles all the information you need?- the solution reduces the operator’s work load?- the solution decreases your response time in emergency management?- the solution decreases the loss of information?- overall impression from 0 to 4? annotation: could be useful to verify with operator if C2-SENSE helps to share procedures or protocols among actors.
WP 7.4 - Functional Subjective Evaluation
TO EVERY QUESTION WILL BE ASSIGNED A SCORE USING A PRE-DEFINED SCHEMA!
score meaning0 I totally disagree - or – very bad 1 I disagree – or - bad2 Neutral 3 I approve – or - good 4 I totally approve – or – very good
WP 7.4 - Functional Objective Evaluation
Report contain annotation and a
radar chart
REPORT
WP 7.4 - Functional Evaluation
FUNCTIONAL GLOBAL EVALUATION
WP 7.4 - Functional Evaluation
Objective Evaluation(the system work as
espected)
Subjective Evaluation(operator approve the behaviour of the solution)
Global Evaluation
WP 7.4 - NON Functional Evaluation
NON FUNCTIONAL EVALUATION
WP 7 - NON Functional Requirement
Non Functional Requirement:AccessibilityAudit and controlAvailability (see service level agreement)BackupCapacity, current and forecastCertificationComplianceConfiguration managementDependency on other partiesDeploymentDocumentationDisaster recoveryEfficiency (resource consumption for given load)Effectiveness (resulting performance in relation to effort)Emotional factors (like fun or absorbing or has "Wow! Factor")Environmental protectionEscrowExploitabilityExtensibility (adding features, and carry-forward of customizations at next major version upgrade)Failure managementFault tolerance (e.g. Operational System Monitoring, Measuring, and Management)Legal and licensing issues or patent-infringement-avoidabilityInteroperabilityMaintainability
ModifiabilityNetwork topologyOpen sourceOperabilityPerformance / response time (performance engineering)Platform compatibilityPricePrivacyPortabilityQuality (e.g. faults discovered, faults delivered, fault removal efficacy)Recovery / recoverability (e.g. mean time to recovery - MTTR)Reliability (e.g. mean time between failures - MTBF, or availability)ReportingResilienceResource constraints Response timeReusabilityRobustnessSafety or Factor of safetyScalability (horizontal, vertical)SecuritySoftware, tools, standards etc. CompatibilityStabilitySupportabilityTestabilityUsability by target user communityUser Friendliness
WP 7 - NON Functional Requirement
ISO 25000 series give a working framework on NON Functional
Requirement
WP 7 - NON Functional Requirement
Intrinsic Qualities
Functional SuitabilityPerformance Efficiency
CompatibilityUsability
ReliabilitySecurity
MaintainabilityPortability
Usage Qualities
Effectiveness
EfficiencySatisfaction
Freedom from Risk (Safety)Context Coverage (Usability Scope)
External Qualities
Service CostVendor Risk Mitigation
Product Risk Mitigation
WP 7 - NON Functional Requirement
Was defined a detailed list of NON funcional requirement
Most of them will be evaluated thinking at the POTENTIAL C2SENSE final product
deployed in production mode.
WP 7 - NON Functional Requirement
EXAMPLEIntrinsic quality – Reliability – Recoverability
Definition : The degree to which, in the event of an interruption or a failure, a product or system can recover the data directly affected and re-establish the desired state of the system
will be evaluated: Recovery Time Objective (RTO): It’s the time to recover full funcions after disarter. It
must tend to 0 in Emergency management Recovery Point Objective (RPO): Max time from data producing and data backup. It
must be near to 0 in Emergency management High availability architecture ready The evaluation will be performed on all C2-SENSE components deployable in production mode on a possible production architecture (not that usesd in pilot application)
WP 7 - NON Functional Requirement
Evaluation -> SCORE- 0 : NFR not covered by the solution at all - 1 : NFR covered minimally but is not sufficient and cannot be accepted in the real
world- 2: NFR covered but some important issue was detected (Eg: cost too high, bad
performance, etc.). The IA can accept the issue temporarily with the promise of a fix.
- 3 : NFR covered and it is coherent with the reference. Some issue was identified but a user can accept the solution
- 4 : NFR covered, it is coherent with the reference and no issue was identified. Annotation: in order to identify issue it’s important to mix : IA operator’s point of view Technical point of view State of the art of IT solutions
WP 7.4 - Functional Objective Evaluation
Report contain annotation and a
radar chart
REPORTIntrinsic Functional Suitability
Intrinsic Performance Efficiency
Intrinsic Compatibility
Intrinsic Usability
Intrinsic Reliability
Intrinsic Security
Intrinsic MaintainabilityIntrinsic Portability
Usage Effectiveness
Usage Efficiency
Usage Satisfaction
Usage Context Coverage
Usage Service Experience
External Service Cost
0
2
4
NFR Score
Issue
WP 7.4 - ISSUE
Pilot application need to be well defined technically
WP 7.4 - ISSUE
Pilot application seems to be to complex and less effective Scope Defined
test note4.1.2.1 Situation Reporting 35 too much4.1.2.2 Mission Plan 10 4.1.2.3 Scheduling 1 4.1.2.4 Resource Management 14 4.1.2.5 Alert 13 4.1.2.6 Hospital Communication 2 4.1.2.7 Tracking of Citizens 5 4.1.2.8 Sensor Management 3 4.1.2.9 EUAA 0 correct- too technical4.1.2.10 Audit Trail and Node Authentication (ATNA) 0 correct- too technical4.1.2.11 Emergency Situation Map 5 4.1.2.12 Situation Analysis 7 4.1.2.13 Permission 0 configuration 2 correct- too technical
WP 7.4 - ISSUE
NON FUNCTIONAL REQUIREMENT MUST TO BE EVALUATED BY OTHER PARTNER
WP 7.4 - ISSUE
Thank you for your attention