Open Data Infrastructures Evaluation Framework using Value Modelling

Embed Size (px)


This is my presentation at HICSS 2014, on evaluation models for Open Data sites and portals, based on the work we are doing at the ENGAGE project

Text of Open Data Infrastructures Evaluation Framework using Value Modelling

  • 1. Charalabidis,Y., Loukis, E., Alexopoulos, H. University of the Aegean, Greece University of the Aegean Department of Information and Communication Systems Engineering

2. INTRODUCTION: THE OPEN /BIG DATA MOVEMENT IN THE BACKGROUND Governments are increasingly opening to the society important data they possess, in order to be used for scientific, commercial and political purposes. Initially a first generation of Internet-based open government data (OGD) infrastructures has been developed in many countries, influenced by the Web 1.0 paradigm, in which there is a clear distinction between content producers and content users. 2 3. A SECOND GENERATION OF OGD INFRASTRUCTURES Recently a second generation of more advanced OGD infrastructures is under development, which is influenced by the principles of the new Web 2.0 paradigm: elimination of the clear distinction between passive content users/consumers and active content producers They aim to support highly active users, who assess the quality of the data they consume and mention weanesses of them and new needs they have and often become data pro-sumers = both consumers and providers of data 3 4. THE NEED FOR AN EVALUATION METHOD The big investments in this area necessitate a systematic evaluation of these OGD infrastructures, in order to gain a better understanding and assessment of the multidimensional value they generate However, a structured and comprehensive evaluation methodology is missing. This method contributes to filling this gap. It presents and validates a methodology for evaluating these advanced second generation of ODG infrastructures, based on a value model approach, i.e. on the estimation of value models of these infrastructures from users ratings. 4 5. INTRODUCTION In particular: it assesses various measures of generated value by OGD infrastructures, structured in three layers (associated with efficiency, effectiveness and users future behavior), and also the relations among them, leading finally to the formation of a value model of the OGD infrastructure, which enables: a deeper understanding of the whole value generation mechanism of it and also a rational definition of IS improvement priorities 5 6. BACKGROUND / SYNTHESIS Literature Review IS EvaluationTAMIS Success ModelsE-ServicesScoping eInfrastructures StakeholdersData AcquisitionData ProvisionCommunication6 7. Research Streams Insights IS Evaluation ISs offer various types of benefits, both financial and non-financial, and also tangible and intangible ones, which differ among the different types of IS it is not possible to formulate one generic IS evaluation method, which is applicable to all IS a comprehensive methodology for evaluating a particular type of IS should include evaluation of both its efficiency and its effectiveness, taking into account its particular characteristics, capabilities and objectives 7 8. Research Streams Insights TAM (Technology Acceptance Model) identify the characteristics and factors affecting the attitude towards using an IS, the intention to use it and finally the extent of its actual usage perceived usefulness and perceived ease of use determine an individual's intention to use a system with intention to use serving as a mediator of actual system useIS Success Models IS evaluation should adopt a layered approach based on the above interrelated IS success measures (information quality, system quality, service quality, user satisfaction, actual use, perceived usefulness, individual impact and organizational impact) and on the relations among them 8 9. Research Streams Insights e-Services Evaluation frameworks that assess the quality of the capabilities that the e-service provides to its users frameworks that assess the support it provides to users for performing various tasks and achieving various objectives, or users overall satisfaction the above frameworks do not include advanced ways of processing the evaluation data collected from the users, in order to maximize the extraction of valuerelated knowledge from them 9 10. Our Evaluation Model Approach (a) Efficiency layer: it includes efficiency measures, which assess the quality of the basic capabilities offered by the e-service to its users. (b) Effectiveness layer: it includes effectiveness measures, which assess to what extent the e-service assists the users for completing their tasks and achieving their objectives. (c) Future behaviour layer: it includes measures assessing to what extent the e-service influences the future behaviour of its users (e.g. to what extent they intend to use the e-service again in the future, or recommend it to friends and colleagues). 10 11. Value Model Definition Data Provision Capabilities Data Search & Download Capabilities User-level Feedback CapabilitiesSupport for Achieving User ObjectivesEase of UseFuture BehaviourPerformance Data Processing Capabilities Data Upload CapabilitiesSupport for Achieving Provider Objecti.Provid-level Feedback CapabilitiesEfficiency LevelEffectiveness LevelFut. Behavior Level 11 12. Value Measures The total of 41 value measures (all layers) were defined where 35 for the 1st layer 14 common value measures 15 value measures for users 06 value measures for providers These value measures was then converted to a question to be included in questionnaires to be distributed to stakeholders A five point Likert scale is used to measure agreement or disagreement 2 Questionnaires have been formulated 12 13. Indicative Value Dimension 1st Level Ease of Use 1.1FriendlinessThe platform provides a user friendly and easy to use environment.1.2Learning EasinessIt was easy to learn how to use the platform.1.3AestheticsThe web pages look attractive.1.4Ease of performing tasksIt is easy to perform the tasks I want in a small number of steps.1.5Multilingual aspectsThe platform allows me to work in my own language.1.6PersonalizationThe platform supports user account creation in order to personalize views and information shown.1.7Support & TrainingThe platform provides high quality of documentation and online help. 13 14. Indicative Value Dimension 1st Level Data Processing Capabilities 7.1 Data EnrichmentThe platform provides good capabilities for data enrichment (i.e. adding new elements - fields)7.2 Data CleansingThe platform provides good capabilities for data cleansing (i.e. detecting and correcting ubiquities in a dataset)7.3 LinkingThe platform provides good capabilities for linking datasets.7.4 VisualisationThe platform provides good capabilities for visualization of datasets14 15. Indicative Value Dimension 2nd Level Support for Achieving User Objectives 8.1 ACC1I think that using this platform enables me to do better research/inquiry and accomplish it more quickly8.2 ACC2This platform allows me to draw interesting conclusions on past government activity8.3 ACC3This platform enables me to create successful added-value electronic services8.4 ACC4I am in general highly satisfied with this platform15 16. Application : The ENGAGE project OGD system to evaluated: ENGAGE - A new multicountry, multi-lingual open data infrastructure for researchers, available at Target user group: post-graduate students from TU Delft and Uaegean, trained in the platfom Method of user input: electronic questionnaires Number of valid questionnaire responses processed: 42 (when the paper was submitted, now more than 100)16 17. The ENGAGE SystemSocial sciences ICTNatural Sciences and EngineeringGovernance Policy ModellingLawProviding PSI to research communities and citizens in a personalised mannerSingle point of AccessUser groupsTailored data servicesData Service Provision InfrastructureCitizensResearch and IndustryGovernance and policy makingSearch and Navigation toolsKnowledge / Data MiningCollaboration / CommunitiesVisualisation - AnalyticsData analyticsCitizens and educationPersonalisationDirectory services and direct linking to data archivesCurating, Annotating, Harmonising , Visualising Data QualityData Curation InfrastructureGathering data from governmental organisations and systems (the Gov Cloud)Data LinkingKnowledge MappingSemantic AnnotationAutomatic curation algorithms AnonymisationPublic Sector Information SourcesPublic Organisations, Repositories, DatabasesHarmonisation 18. Value Model Estimation Algorithm Value Dimensions Internal Consistency ExaminationValue Dimensions Variables CalculationAverage Ratings CalculationValue Models ConstructionCorrelations EstimationRegression Models EstimationImprovement Priorities Identification 18 19. Data Provision Capabilities 3.03 Data Search & Download Capabilities 3.03 User-level Feedback Capabilities 2.97 Ease of Use 3.35Estimated Value Model 0.6390.760Support for Achieving User Object. 3.170.6510.6240.730Future Behaviour 3.190.379 0.735Performance 2.15 Data Processing Capabilities 3.27 Data Upload Capabilities 2.930.489 0.479 0.135 0.632Support for Achieving Provider Obj. 3.120.680 0.307Provider-level Feedback Capabilities 3.4419 20. R2 coefficients of second and third layer value dimensions regression models Regression Models SUO model (8 indep. variables)0.776SPO model (8 indep. variables)0.599FBE model (2 indep. variables)0.412FBE model (10 indep. variables)6-9/01/2014R20.647HICSS 47 - University of the Aegean20 21. Improvement Priorities Identification Such an OGD infrastructure value model, Enables the identification of improvement priorities, which are the first layer OGD systems capabilities that receive low evaluation by the users, and at the same time have high impact on higher layers value generation 22. Mapping for decision support Lower Ratings Group data provision capabilitiesHigher Ratings Group provider-level feedback cap.Lower Impact Group data provision capabilitiesHigher Impact Group data processing capabilitiesdata searchdownload cap.ease of useuser-level feedback capab.ease of usedata u