D3.1.5v2 Published learning resources and evaluation of REASE

Coordinator: Jörg Diederich (L3S)

Diana Maynard (USFD), Frank van Harmelen (VU), York Sure (UKARL), Sylvain Dehors (INRIA), Mustafa Jarrar (VUB), Martin Dzbor (OU),

Dnyanesh Rajpathak (OU), Marco Ronchetti (UniTn), Steven Willmott (UPC), Heiner Stuckenschmidt (UniMannheim)

Abstract.EU-IST Network of Excellence (NoE) IST-2004-507482 KWEBDeliverable D3.1.5v2 (WP3.1)

AbstractThis deliverable summarizes the activities related to populating REASE, the reposit-ory of EASE for learning units about Semantic Web topics, with learning resources, including the creation and evaluation of the new catalogue, a description of the pub-lished learning units, the quality management process, and an evaluation of the usage of the published learning units, based on user’s experience as gathered from a ques-tionnaire sent to all REASE users and a user study conducted with about 30 persons in a class-room scenario at three different locations.

Document Identifier: KWEB/2007/D3.1.5v2/v2.0Class Deliverable: KWEB EU-IST-2004-507482Version: v2.0Date: 25/01/2007State: FinalDistribution: Public

Copyright © 2006 The contributors

Knowledge Web Consortium

This document is part of a research project funded by the IST Programme of the Com-

mission of the European Communities as project number IST-2004-507482.

University of Innsbruck (UIBK) – CoordinatorInstitute of Computer Science,Technikerstrasse 13A-6020 InnsbruckAustriaContact person: Dieter FenselE-mail address: dieter.fensel@uibk.ac.at

École Polythechnique Fédérale de Lausanne (EPFL)Computer Science Department Swiss Federal Institute of TechnologyIN (Ecublens), CH-1015 Lausanne.SwitzerlandContact person: Boi FaltingsE-mail address: boi.faltings@epfl.ch

France Telecom (FT)4 Rue du Clos Courtel 35512 Cesson SévignéFrance. PO Box 91226Contact person : Alain LegerE-mail address: alain.leger@rd.francetelecom.-com

Freie Universität Berlin (FU Berlin)Takustrasse, 914195 Berlin GermanyContact person: Robert TolksdorfE-mail address: tolk@inf.fu-berlin.de

Free University of Bozen-Bolzano (FUB)Piazza Domenicani 3 39100 Bolzano ItalyContact person: Enrico FranconiE-mail address: franconi@inf.unibz.it

Institut National de Recherche en Informa-tique et en Automatique (INRIA)ZIRST - 655 avenue de l'Europe - Montbonnot Saint Martin38334 Saint-IsmierFranceContact person: Jérôme EuzenatE-mail address: Jerome.Euzenat@inrialpes.fr

Centre for Research and Technology Hellas / Informatics and Telematics Institute (ITI-CERTH) 1st km Thermi – Panorama road57001 Thermi-ThessalonikiGreece. Po Box 361Contact person: Michael G. Strintzis E-mail address: strintzi@iti.gr

Forschungszentrum L3SAppelstraße 9a30167 Hannover GermanyContact person: Wolfgang NejdlE-mail address: nejdl@l3s.de

National University of Ireland Galway (NUIG)National University of IrelandScience and Technology BuildingUniversity RoadGalwayIrelandContact person: Christoph BusslerE-mail address: chris.bussler@deri.ie

The Open University (OU)Knowledge Media InstituteThe Open University Milton Keynes, MK7 6AA United Kingdom.Contact person: Enrico MottaE-mail address: e.motta@open.ac.uk

Universidad Politécnica de Madrid (UPM)Campus de Montegancedo sn 28660 Boadilla del Monte SpainContact person: Asunción Gómez PérezE-mail address: asun@fi.upm.es

University of Karlsruhe (UKARL)Institut für Angewandte Informatik und Formale Beschreibungsverfahren – AIFBUniversität Karlsruhe D-76128 Karlsruhe GermanyContact person: Rudi Studer

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 2

E-mail address: studer@aifb.uni-karlsruhe.de

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 3

University of Liverpool (UniLiv)Chadwick Building, Peach Street L697ZF Liverpool United KingdomContact person: Michael WooldridgeE-mail address: M.J.Wooldridge@csc.liv.ac.uk

University of Manchester (UoM)Room 2.32. Kilburn Building, Department of Computer Science, University of Manchester, Ox-ford RoadManchester, M13 9PL United KingdomContact person: Carole GobleE-mail address: carole@cs.man.ac.uk

University of Sheffield (USFD)Regent Court, 211 Portobello streetS14DP Sheffield United KingdomContact person: Hamish CunninghamE-mail address: hamish@dcs.shef.ac.uk

University of Trento (UniTn) Via Sommarive 14 38050 Trento ItalyContact person: Fausto GiunchigliaE-mail address: fausto@dit.unitn.it

Vrije Universiteit Amsterdam (VUA) De Boelelaan 1081a 1081HV. Amsterdam The NetherlandsContact person: Frank van HarmelenE-mail address: Frank.van.Harmelen@cs.vu.nl

Vrije Universiteit Brussel (VUB)Pleinlaan 2, Building G10 1050 Brussels BelgiumContact person: Robert MeersmanE-mail address: robert.meersman@vub.ac.be

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 4

Work package participants

The following partners have taken an active part in the work leading to the elaboration of this document, even if they might not have directly contributed by writing parts of this document:CERTHEPFLFUBFUBerlinINRIAL3SNUIGOUUKARLUniLivUniMannheimUniTnUPCUPMUSFDVUVUBVUM

REWERSE:Stony Brook University New YorkUniversity of GöttingenUniversity Nova of LisboaLMU MunichLinköping UniversityUniversity of Turin

AgentLink:University of Southampton

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 5

Changes

Version Date Author Changes

0.1 28-10-2005 Jörg Diederich Initial version

0.2 5-12-2005 Jörg Diederich Completed first version without statistics

0.3 15-12-2005 Diana Maynard Made a few small edits

0.4 16-12-2005 Jörg Diederich Included comments from others, added published learning units and the access statistics

0.5 20-12-2005 Jörg Diederich Included further comments from the WP

0.6 22-12-2005 Jörg Diederich Minor fix from Luis+Sylvain

1.0 13-01-2006 Jörg Diederich Comments from Quality control

1.01 20-01-2006 Jörg Diederich Comments from Quality assurance

2.0pre1 4-12-2006 Jörg Diederich Updated data, included section about evaluation of topic hierarchy and evaluation of the question-

naires

2.0pre2 15-12-2006 Diana Maynard Included evaluation of questionnaire

2.0pre3 16-12-2006 Mustafa Jarrar, Jörg Diederich Included evaluation of topic hierarchy

2.0pre6 22-01-2007 Jörg Diederich Included comments from quality control

2.0final 25-01-2007 Jörg Diederich Included further last-minute comments + final pol-ishing

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 6

Executive Summary

This deliverable summarizes the activities related to populating REASE, the reposit-ory of EASE for learning units about Semantic Web topics, with learning resources. The number of learning resources published by KnowledgeWeb members or resulting from events organized by KnowledgeWeb increased from about 30 at the end of 2004 and about 50 at the end of 2005 to 81 at the end of 2006. Even though we concen-trated on the creation of material for industrial education in 2005, our focus for 2006 still was on such industrial materials, but also on publishing material different from ‘slides only’ resources. Specifically, we published a series of one hour lecture record-ings from the KnowledgeWeb summer school.

We also evaluated the REASE catalogue, comprising 58 categories, which was de-veloped in 2005 and refined in 2006 in a more general discussion of a Semantic Web Topic Hierarchy among KnowledgeWeb and REWERSE participants. Though the evaluation of taxonomies is in general a difficult task, we have found some evidence that our taxonomy is reasonable, based on both manual inspection and on statistics taken from publications in Semantic Web topics.

To control the quality of the published learning units, we have set up a list of quality guidelines, which have to be followed when publishing learning units. This is comple-mented by a quality management process which determines how the guidelines are ac-tually enforced. This year we implemented that quality management process involving the editorial board. However, the quality of the published learning resources is very high in general. Hence, not many actions had to be taken to enforce the quality guidelines.

Finally this deliverable comprises also an evaluation of the usage of REASE and the published learning units using

a log file analysis; the results of a questionnaire, sent to all 230 users of REASE (as of

September 2006); the results of a user study, conducted with more than 30 people at three

different locations in November 2006. Though the users in general were quite satisfied with REASE, they made a large num-ber of suggestions to improve REASE, which will guide our efforts related to REASE in 2007.

The main contributions in 2006 are: Additional 30 learning resources by KnowledgeWeb partners Evaluation and a first refinement of the Semantic Web Topic Hierarchy

(section 2.1.5) Update of REASE and refinement of the user interface Evaluation of REASE

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 7

Contents

1 Introduction.........................................................................................................9

2 The REASE catalogue........................................................................................9

2.1 The Semantic Web Topic Hierarchy..............................................................92.1.1 Overall Structure..................................................................................102.1.2 Foundations..........................................................................................102.1.3 Semantic Web Core Topics..................................................................112.1.4 Semantic Web Special Topics..............................................................122.1.5 Evaluation of the Semantic Web Topic Hierarchy...............................12

2.2 The REASE Catalogue.................................................................................163 List of Published Learning Units.....................................................................18

3.1 Overview and Statistics................................................................................183.2 The Learning Units in Detail........................................................................18

3.2.1 Material for industrial education..........................................................193.2.2 Full-course material..............................................................................293.2.3 Miscellaneous Modules related to Semantic Web Material.................323.2.4 Modules about Core Topics around Semantic Web.............................333.2.5 Modules about Special Topics around Semantic Web.........................383.2.6 Courses contributed by REWERSE.....................................................433.2.7 Courses contributed by external institutions........................................50

3.3 Evaluation.....................................................................................................514 Evaluation of REASE........................................................................................51

4.1 User-based evaluation of REASE................................................................524.1.1 General background.............................................................................544.1.2 Usability and Accessibility Issues........................................................574.1.3 Finding information..............................................................................594.1.4 Information Quality..............................................................................614.1.5 Providing information to REASE........................................................614.1.6 General comments................................................................................634.1.7 Discussion and Improvements.............................................................64

4.2 Evaluation based on the User Study.............................................................654.2.1 Motivation............................................................................................654.2.2 General data and experiences...............................................................654.2.3 The Scenarios.......................................................................................664.2.4 Scenario 1: Find material about ‘problem solving methods (PSMs)’. .664.2.5 Scenario 2: Find materials suited for an industrial audience, discussing the role of Human Language Technologies (HLT) in the Semantic Web tools and applications...........................................................................................................684.2.6 Scenario 3: Find materials containing definitions of reasoning tech-niques; in particular deduction, in the materials about Description Logic (DL) ."

684.2.7 Scenario 4: Find material and the categories associated with the mate-rial in the catalogue containing description of species or layers in Web Ontology Language (OWL)."...............................................................................................694.2.8 Assessing the content of REASE.........................................................704.2.9 Freetext comments...............................................................................70

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 8

4.2.10 Summary..............................................................................................724.3 Summary of Evaluation................................................................................73

5 Quality Guidelines and Procedures................................................................73

5.1 Technical Requirements...............................................................................735.1.1 Non-Proprietary File Formats..............................................................735.1.2 Uploading Material vs. Linking...........................................................745.1.3 Metadata...............................................................................................745.1.4 File Formats..........................................................................................745.1.5 Modularization.....................................................................................745.1.6 Questionnaire.......................................................................................75

5.2 Non-Technical Requirements.......................................................................755.3 Quality Management Procedures.................................................................75

5.3.1 Controlling Requirements Automatically............................................755.3.2 Controlling Requirements Manually....................................................75

6 Usage of Learning Resources.........................................................................77

6.1 General Usage of the REASE Web Pages...................................................776.2 Registrations on REASE..............................................................................776.3 Access to REASE Resources.......................................................................786.4 Most Popular Resources on REASE............................................................79

7 Summary and Future Work..............................................................................79

References................................................................................................................81Appendix...................................................................................................................82

Questionnaire...........................................................................................................82Evaluating REASE, the Repository of Semantic Web Learning Units.................82

General Questions....................................................................................................82Specific Questions Related to Finding / Downloading Material..............................83Specific Questions Related to Uploading Material..................................................84General comments....................................................................................................85User Study................................................................................................................86

Evaluating REASE, the Repository of Semantic Web Learning Units.................86

The REASE evaluation task.....................................................................................86Motivation............................................................................................................86Duration................................................................................................................86

Part I. Familiarization...............................................................................................86Part II. Material gathering........................................................................................87

Scenario 1.............................................................................................................87Scenario 2.............................................................................................................88Scenario 3.............................................................................................................89Scenario 4.............................................................................................................89

Part III. Assessment of material relevance and quality............................................91General comments....................................................................................................92

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 9

1 IntroductionThis deliverable is intended to document the work in the education area related to publishing educational material on REASE, the Repository of EASE for learning units1. It is an extension of D3.1.5, published a year ago, which reported about the fol-lowing issues:

Publishing more learning resources, especially ones for industry Extend the REASE catalogue to allow for a more effective search Creating guidelines and procedures for quality management Performing a first evaluation of the usage of REASE.

Besides updating the statistics about the usage of REASE, we have focused on the fol-lowing issues in the past 12 months:

Publish more learning resources, again especially ones for industrial education and those based on video recordings

Evaluate the REASE topic hierarchy Evaluate the REASE platform using two additional orthogonal

strategies: Send a questionnaire to the actual REASE users Conduct a user study in a controlled classroom environment

These activities will be reported in more detail in the following sections. We start with a description of the REASE catalogue and thereafter describe the published learning resources with the help of the catalogue.

2 The REASE catalogueAs described in D3.3.2v2, the REASE catalogue initially comprised only five cate-gories. However, when more and more learning resources were added, it became clear that this initial classification was no longer sufficient, so we initiated a general discus-sion about a Semantic Web curriculum, the so-called Semantic Web Topic Hierar-chy2 together with the NoE REWERSE, in order to align the REASE catalogue with the curriculum activities in REWERSE. In 2006, we moved the Semantic Web cur-riculum to the OntoWorld wiki, a Wiki system which is itself semantically enhanced. Specifically, we have included the Semantic Web curriculum itself into the Wiki cate-gorization scheme. In this way, other users of the OntoWorld wiki, eg. those using it to publish workshop descriptions, can use the curriculum for classification of any wiki pages. Furthermore, we made some minor modifications to the Topic Hierarchy, which is currently available in version 1.1.

2.1 The Semantic Web Topic HierarchyThe Semantic Web Topic hierarchy was developed jointly with REWERSE starting from the initial curriculum as discussed in the REWERSE deliverable E-D5, which it-self is based upon the ACM Computing Classification System3. As this ACM classifi-cation system is by far outdated (its last version is from 1998), the Semantic Web Topic Hierarchy also comprises topics which were not existent or relevant at the time 1 http://rease.semanticweb.org 2 http://wiki.ontoworld.org/index.php/Semantic_Web_Topic_Hierarchy 3 http://www.acm.org/class/1998/

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 10

of the creation of the ACM classification system. Specifically, we examined the ses-sion titles of the two major conferences in the area of Semantic Web, the International Semantic Web Conference (ISWC) and the European Semantic Web Conference (ESWC) from past years, i.e. 2001-2005.

2.1.1 Overall StructureThe structure of the curriculum is in general three-fold:

Foundations Semantic Web Core Topics Semantic Web Special Topics

This retains the overall top-level structure of the original initial version of the REASE catalogue. Such a backward compatibility is important as REASE is a running system in daily use: it enables an automatic reclassification of already existing material in REASE and does not require a time-critical manual intervention of the original provider. However, a manual reclassification of the material into the newer, more fine-grained categories was still necessary. In the following subsections, we provide more details on the three main categories of the curriculum. The topic hierarchy has evolved little to version 1.1 since the last version of this deliverable (e.g., we deleted the top-level category “Information Management” from the foundations as it was too general). Both versions can still be found at the Ontoworld wiki.

2.1.2 FoundationsOriginally, the foundations category comprised the subcategories `Logics’ and `Web technologies’. This was extended by many new categories to allow for a more fine-grained categorization and to integrate existing categories from the ACM classifica-tion system. A more detailed description of the curriculum can be found in the REW-ERSE deliverable E-D7.

Specifically, we added the following categories and sub-categories:

Knowledge Engineering / Ontology Engineeringo Methodologieso Ontology population / generationo Maintenance and versioning (dynamics)o Mapping / translation / matching / aligning (heterogeneity)o Validationo Interoperability / Integrationo Modularization and Compositiono Tools

Knowledge Representation and Reasoningo Logics:

Predicate Logic Description Logics F-logic Modal Logics First-order Logic Temporal Logic

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 11

o Logic Programming Horn Logic Datalog Prolog Hilog

o Reasoning Basic Web information technologies

o XML Namespaces Schema languages XML query and transformation languages XML programming techniques

o Web data integrationo Securityo Web serviceso Personalization techniqueso Web data extraction / information extractiono Architecture of Web Information Systems

Agents Natural Language Processing

An automatic mapping from the old categories was performed using: Logics Knowledge Representation and Reasoning | Logics Web technologies Basic Web information technologies

Learning units which were classified as ‘Foundations’ in general, were reclassified manually based on an individual inspection.

2.1.3 Semantic Web Core TopicsOriginally, the REASE catalogue contained the categories `Knowledge Representa-tion’, `Ontologies’, and `Semantic Web Technologies’. We extended this scheme to the following categories and subcategories, trying to align them also to the well-known Semantic Web Layer cake:

Resource Description Framework / RDFSchema Query and Update Languages

o Query Languageso Update Languages

Ontologieso Ontology representation / Ontology languages / OWLo Ontology Engineeringo Ontology Reasoners

Rules + Logico Rule languageso Rule Markupo Reasoning languageso Rule Reasonerso Integration of Rules and Ontologies

Proof

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 12

Security / trust / privacy Applications

o Knowledge Managemento E-Learningo Bioinformaticso Multimediao ehealtho ebusinesso Lawo Engineeringo eGovernment

The original categories were mapped as follows:o Knowledge Representation Foundations | Knowledge Representation and

Reasoningo Ontologies Ontologieso Semantic Web Technologies Resource Description Framework / RDF-

Schemao This one was manually post-processed as it did not always match.

Again, learning units that were classified as ‘Semantic Web Core Topic’ in general, were reclassified manually.

2.1.4 Semantic Web Special TopicsOriginally, there were no categories below this topic. We extended this significantly to capture current hot topics of Semantic Web research:

o Natural language processing / human language technologieso Social impact of the Semantic Webo Social networks and Semantic Webo Peer-to-peer and Semantic Webo Agents and Semantic Webo Semantic Grido Outreach to industryo Benchmarking and scalabilityo Design and test bed case studieso Semantic Web Services

A reclassification was not necessary since we kept the category ‘Semantic Web Spe-cial Topics’.

2.1.5 Evaluation of the Semantic Web Topic HierarchyTopic-based classifications are an important part of information retrieval. The topic hierarchy of REASE is intended to guide users to quickly and correctly find the learn-ing units they seek. Although we use several metadata elements to describe a learning unit, the topic hierarchy is seen as a formal description, and thus it enables simple rea-soning. For example, if ‘ontology mapping’ is a sub-topic of ‘ontology engineering’, then all learning units that are classified under ‘ontology mapping’ are also instances of the topic ‘ontology engineering’.

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 13

In order for the topic hierarchy of REASE to be effective, first, the learning units should be correctly classified. To meet this requirement, we engaged human experts to review and assure that (all and only) the relevant topics appear in the description of a learning unit. Second, the topic hierarchy itself should be expressive. This means that the hierarchy should be descriptive enough to indicate what a learning unit is about, and intuitive enough so that users can easily understand and use it. We have found that this requirement is indeed not an easy goal to achieve, because of its sub-jective nature. In the following we describe this difficulty and our approach to build and validate the REASE hierarchy.

Although topic classification is an old subject matter especially in the libraries world, it has always been a difficult issue. Topic classification can be viewed from a variety of perspectives ranging from the purely ad hoc and pragmatic to the purely philosoph-ical, see [WJ99].

As the term topic typically means an area of knowledge [WJ99], the problem of clas-sifying such knowledge depends on how mature the understanding of the area is and to what degree it is shared within a community. Some areas of knowledge change very rapidly, especially IT research topics. In addition, people with different back-grounds view topic classification differently [GMZ06]. This is due to the different ex-pectations of what activities in an area of knowledge are.

J. Doyle stated: “It became clear to me that the main organizing principle for in-dices, at least for most people, is sociological. That is, one structures the index not to reflect conceptual relations, but to reflect things like what populations of people like to work together, what do they think of as the current main topics of interest, etc.” [WJ99].

C. Welty added: “as a consequence of the social nature of topic organization, the stability of a particular section of the hierarchy seems to be directly tied to the field’s maturity.”

In short, topic classifications are subjective knowledge, they are mostly influenced by personal tastes, may reflect fundamental disagreements, and change rapidly.

With this in mind, our approach to build the topic hierarchy of REASE is guided by the following two aspects.

1. The topic hierarchy should be built by active and up-to-date researchers. As such researchers are supposed to be aware of the research directions and activ-ities in the semantic web area, we expect the topic hierarchy to be more stable and to reflect a relatively mature understanding of the subject matter.

2. It should build on existing classifications if available.

3. In order for the topic hierarchy to be intuitive and easy to understand by users, the labels in the topic hierarchy should be familiar and frequently used in the semantic web community.

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 14

Regarding the first aspect, the REASE topic hierarchy has been built by a group of ac-tive researchers in KnowledgeWeb and REWERSE. It has been discussed on the mail-ing lists of KnowledgeWeb and REWERSE, for over a year, and around 300 emails were exchanged about it. Furthermore, the researchers involved in this discussion originated from different backgrounds, such as: artificial intelligence, peer-to-peer, database, natural language processing, business informatics, etc.

The Semantic Web Topic Hierarchy is also built upon existing classifications, namely the ACM Computer Classification System as mentioned before and as explained in more detail in the REWERSE deliverable E-D7.

For the third aspect, we cross-validated our topic hierarchy with frequently used key-words from Semantic Web publications. For this purpose, we extracted the author keywords of all Semantic Web publications in the popular DBLP archive, performed a standard set of methods for cleaning them (e.g., stemming), and created a list of co-occurring keywords, sorted by the number of co-occurrences, restricting ourselves to those keywords that occur at least 5 times with Semantic Web (cf. Table 1)

Table 1: Co-occurring Keywords from Semantic Web publications in DBLPontology 95 Matchmaking 7web services 39 Personalization 6RDF 34 Ontology mapping 6OWL 19 Information integration 6XML 18 DAML+OIL 6metadata 16 e-Learning 6knowledge representation 12 Semantic Web Services 6information retrieval 11 Interoperability 6knowledge management 11 machine learning 6Agents 10 OWL-S 5Description Logics 10 User interface 5annotation 9 Web service composition 5P2P 8 RDFS 5Semantic annotation 8 Rules 5Information extraction 8 Automated reasoning 5Search 7 multimedia 5DAML 7 e-commerce 5

The REASE topic hierarchy comprises directly contains 22 out of the 34 terms in Ta-ble 1, namely ‘ontology’, ‘web services’, ‘RDF’, ‘OWL’, ‘XML’, ‘knowledge repre-sentation’, ‘knowledge management’, ‘agents’, ‘description logics’, ‘peer-to-peer’, ‘information extraction’, ‘personalization’, ‘ontology mapping’, ‘information integra-tion’, ‘e-Learning’, ‘Semantic Web Services’, ‘interoperability’, ‘RDFS’, ‘rules’, ‘(automated) reasoning’, ‘multimedia’, and ‘e-commerce’ (in our hierarchy called eBusiness). 4 out of the remaining 12 are very general including ‘metadata’, ‘annota-tion’, ‘semantic annotation’, and ‘user interface’. The two keywords about predeces-sor languages of OWL (‘DAML’ and ‘DAML+OIL’) can be easily classified into the category ‘Ontology Representation / Ontology Languages / OWL’ and are, hence, also included in the Semantic Web Topic Hierarchy. Three of the remaining topics are subtopics of ‘Semantic Web Services’, namely ‘matchmaking’, ‘OWL-S’, and ‘Web service composition’ They might be added to refine the Semantic Web Service cate-gory in the future.

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 15

As a result of this analysis, we considered including the topic ‘search /information retrieval’, and ‘machine learning’ as an application area in the Semantic Web Topic Hierarchy. This will be discussed in the upcom-ing months.

The remaining 50 terms in the Semantic Web Topic Hierarchy do not co-occur with ‘Semantic Web’ more than five times and are, hence, not listed in Table 1. This is mostly because they are very specific and typi-cally not used as author keywords (like the subtopics of ‘XML’) or very rarely only (like ‘Logic Program-ming’ co-occurring 4 times with ‘Semantic Web’). We will analyze the usage of the categories in REASE in more detail in Section 3.

We also developed the Semantic GrowBag approach [DBT06] to automatically find tag graphs, i.e. relations between topics (which are not necessarily hierarchical) from tagged object collections, for example, publica-tion databases annotated with author keywords. We used the same DBLP dataset, enhanced with author keywords to get a tag graph for ‘Semantic Web’ as shown in Figure 1.In this figure (generated using author keywords from publications in the period 2001-2005), ‘Semantic Web’ is depicted with a black background, and the main re-lated concepts (i.e. ‘ontology’, ‘web services’, ‘RDF’, ‘metadata’, ‘knowledge representation’, ‘description logics’, and ‘OWL’), as found from a co-occurrence analysis, are shown with a grey background. Arrows in general can be best described to mean ‘is related to and more specific’, but quite some of them are also ‘hierar-chical’ and actually mean ‘subsume’. The confidence in the automatically computed relations is shown with bold lines and two-headed arrows for strong confi-dence and with a dashed line for a weak confidence. More details about the GrowBag scheme can be found in [DBT06].

As a result, GrowBag confirms the Semantic Web Topic Hierarchy in the following aspects:

The topics ‘web services’, ‘description logics’, and ‘knowledge representation’ are all important, but not a subtopic of ‘Semantic Web’ (in the Semantic Web Topic Hierarchy they all belong to the ‘foundation’ part)

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 16

The topics ‘ontology’, ‘RDF’, and ‘OWL’ are all subtopics of ‘Seman-tic Web’ (strong confidence)

The topics ‘rules’, ‘reasoning’, and ‘Semantic Web Services’ are subtopics of ‘Semantic Web’ (weak confidence)

Further interesting findings (those involving strong relations) are: ‘Ontology’ subsumes ‘document classification’ (an application area ac-

tually missing in the Semantic Web Topic Hierarchy) ‘RDF’ is a subtopic of ‘metadata’ ‘ontology’ is a super-topic of ‘service

composition’ as well as ‘Web service’ is a super-topic of ‘service composition’. This is the connection between the tag graph of ‘semantic web’ and ‘web service’ (“semantic web ser-vice” does not take this role because authors do neither use the pair ‘se-mantic web’ and ‘semantic web service’ nor the pair ‘web service’ and ‘semantic web service’).

Apart from ‘document classification’ no further keywords are found in the tag graph of ‘Semantic Web’ which appear to be missing in the Semantic Web Topic Hierarchy. This will also be discussed in the next version of the Semantic Web Topic Hierarchy.

2.2 The REASE CatalogueWhile the Semantic Web Topic Hierarchy reflects, of course, a compromise among the different opinions within the Semantic Web community (e.g., some consider `nat-ural language processing’ as a foundational topic while others treat it as special topic), we had to generate an even more simplified version for technical reasons: the REASE catalogue, though customizable, can only handle up to two hierarchical levels at maxi-mum. This has also the advantage that the number of categories is more limited, so REASE users are not `lost’ in too many catalogue categories.As a result, we skipped the first-level hierarchy of `foundations’, `Semantic Web core topics’ as there sometimes also is no real distinction between them (there was, for ex-ample, quite some discussion during the creation of the topic hierarchy whether on-tologies are foundational or belong to the core topics). Furthermore, we ignored the subcategories of `Logics’, `Logic Programming’, and `XML’, since it was not ex-pected that learning material in REASE will deal specifically with one of the subtopics. Instead, it is expected that learning units in these topics give an overview, for example, on `Logics’ and discuss most of the sub-categories.

As a result, the REASE catalogue comprises the following topics:

Knowledge Engineering / Ontology Engineeringo Methodologieso Ontology population / generationo Maintenance and versioning (dynamics)o Mapping / translation / matching / aligning (heterogeneity)o Validationo Interoperability / Integrationo Modularization and Composition

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 17

Figure 1: Semantic Grow-Bag of 'Semantic Web'

o Tools Knowledge Representation and Reasoning

o Logicso Logic Programmingo Reasoning

Basic Web information technologieso XMLo Web data integrationo Securityo Web serviceso Personalization techniqueso Web data extraction / information extractiono Architecture of Web Information Systems

Resource Description Framework / RDFSchema Semantic Web Query and Update Languages

o Query Languageso Update Languages

Ontologies for the Semantic Webo Ontology representation / Ontology languages / OWLo Ontology Engineeringo Ontology Reasoners

Rules + Logico Rule languageso Rule Markupo Rule Reasonerso Integration of Rules and Ontologies

Proof in the Semantic Web Security / trust / privacy in the Semantic Web Semantic Web Applications

o Knowledge Managemento E-Learningo Bioinformaticso Multimediao ehealtho ebusinesso Lawo Engineeringo eGovernment

Semantic Web Special Topicso Natural language processing / human language technologieso Social impact of the Semantic Webo Social networks and Semantic Webo Peer-to-peer and Semantic Webo Agents and Semantic Webo Semantic Grid

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 18

o Outreach to industry4

o Benchmarking and scalabilityo Design and test bad case studieso Semantic Web Services

Of course, this catalogue will be subject to further changes, for example, to align it with the shared master activities in work package 3.2.

3 List of Published Learning UnitsThis section summarizes the learning units that have been published on REASE by the end of 2006.

3.1 Overview and StatisticsThe following figure depicts the number of learning resources available on REASE since it was put online in July 2004. In total 105 learning units were published on REASE, of which 81 were published by KnowledgeWeb partners. 30 of these learning units (37%) are especially suited for in-dustrial education. (as of 18th December 2006). Two main events can be identified, In October / November 2004, an initial set of learning units was published as a results of the first public announcement of REASE in October 2004. A second significantly large set of resources was added in June / July 2005 by the tutors of the REWERSE summer school, who were required to add their resources before the start of the summer school. More resources were added step by step at the end of 2005 as a results of further educational activities in Knowl-edgeWeb, such as the industry-education events (reported in D3.2.9). The REW-ERSE summer school slides of 2006 were added during the whole year 2006 while the KnowledgeWeb summer school recordings of 2005 and the presentations of the ESWC industrial day were added in December 2006.

3.2 The Learning Units in DetailIn addition to the improved catalogue, we provide a simple classification of the mate-rial here into the following categories:

Material for industrial education Full-course materials Miscellaneous modules related to Semantic Web Modules about core topics for Semantic Web Modules about special topics for Semantic Web

The material for industrial education was kept separate as this was identified as the main target audience, which was not sufficiently represented by the material available on REASE by the end of 2004. Full-course materials are listed separately since they typically cover a broad range of topics within the main topic ‘Semantic Web’. Finally, smaller modules are classified into those dealing with prerequisites, core topics, and special topics. We also present a list of those modules on REASE, which were pub-

4 We now have added a separate orthogonal classification for ‘industrial audience’ vs. ‘aca-demic audience’ to be able to create a separate REASE view on industrial / academic material.

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 19

lished by people from outside KnowledgeWeb (i.e. REWERSE and AgentLinkIII). A more detailed classification of all material can be found on REASE.

3.2.1 Material for industrial educationIn this section we summarize the material for industrial education, divided into two groups: Material with introductory topics or core topics (like ontologies, RDF etc.) and material about advanced topics from the top-level category ‘Semantic Web Spe-cial Topics’ (such as natural language processing).

Title Semantic Web Information DayAbstract The Information Day gives an overview of the fundamental concepts and

technologies of the Semantic Web. It enables you to incorporate the buzz-word "Semantic Web" into your lexicon. Furthermore it gives you an op-portunity to evaluate the meaning of the Semantic Web for your existing and future projects.

Provider Free University BerlinLanguage GermanURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID=lr-freea-

lnixon-1091099944332 Categories Ontologies for the Semantic Web, RDF/RDFS, Semantic Web Applica-

tions, Outreach to IndustryType PDFLength 1:30h

Title Semantic Web - Überblick und EinleitungAbstract Der Vortrag vermittelt einen Überblick über die grundlegenden Konzepte

und Technologien des Semantic Web. Sie werden dadurch in die Lage versetzt, das Schlagwort Semantic Web in Ihre Begriffswelt einzuordnen.

Provider Free University BerlinLanguage GermanURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-freea-

lnixon-1118761334299 Categories RDF / RDFS, Outreach to IndustryType PPT / PDFLength 0:45h

Title Modellierung mit dem Semantic WebAbstract Welche Sprachen sind vorhanden um inhaltliche Sachverhalte im

Semantic Web zu notieren?- RDF- RDF-Schema- OWL

Provider Free University BerlinLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-freea-

lnixon-1118761875283 Categories Ontology Representation / Ontology Languages / OWL, RDF / RDFS,

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 20

Outreach to Industry, ToolsType PDFLength 0:45h

Title Semantic Web TutorialAbstract The Semantic Web was designed by the World-Wide-Web inventor, Tim

Berners-Lee, to enrich the WWW with content-based descriptions in a way, such that finding and compressing information by machines be-comes a lot easier. Objective of this presentation is to provide an over-view over the most important methods and technologies such content-based, i.e. semantic descriptions of information in the web, which are es-pecially important for knowledge management applications.

The presentation is structured as follows: a general introduction to the problem, annotation languages (RDF), creation and use of ontologies, and finally applications.

The potential of Semantic Web technology especially for knowledge man-agement is relayed to the participants. The current state-of-the-art is shown as well as potential application areas and concrete applications. Using the products and reference applications of Semantic Web compa-nies, the presentation also shows how Semantic Web technology is al-ready in use in business.

Provider AIFB – University of KarlsruheLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-freea-

lnixon-1130411486152Categories Outreach to Industry, RDF/ RDFS, ebusiness, Knowledge ManagementType PDFLength 2:00hTitle RDF BriefingAbstract An introduction into RDF with a small discussion why the ontology lan-

guage OWL is neededProvider Vrije Universiteit AmsterdamLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-vrij-hol -

ger-1133369535466Categories RDF / RDFS, Ontology Representation / Ontology Languages / OWL,

Outreach to IndustryType PDFLength 0:45h

Title Ontology Engineering Best Practices - Building and Applying the SWRC Ontology

Abstract This short tutorial describes how the Ontology 'Semantic Web for Re-search Communities' has been built, including a set of design considera-tions and guidelines for (re-)using it. It also includes a set of application examples.

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 21

Provider AIFB – University of KarlsruheLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-lear-

diederich-1134387089110Categories Methodologies, Modularization and Composition, Ontology Engineering,

Outreach to IndustryType PDFLength 0:45h

Title A small tutorial on the Alignment APIAbstract A small tutorial for the Alignment API. It is a full hands-on session with

questions and answer that should lead the reader from the very beginning to familiarity with the Alignment API.

Provider INRIALanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

inri-euzenat-1162216435831 Categories Mapping / Translation / Matching / Aligning (Heterogeneity), ToolsType HTMLLength 3:00h

Title Exploiting large scale semantics on the webAbstract A talk providing a view on the evolution of the semantic web and illus-

trating the key tenets of the emerging new generation of semantic web ap-plications 

Provider The Open UniversityLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

theo-emotta-1162838575399Categories Architecture of Information Systems, Web Data Integration, Ontologies

for the Semantic Web, Resource Description Framework (RDF) / RDF-Schema, Semantic Web Applications

Type PDF

Title Human Language Technology for the Semantic WebAbstract This tutorial covers the use of Human Language Technology for the Se-

mantic Web and Web Services. It includes material on an introduction to Information Extraction, Evaluation, Language Engineering and Machine Learning approaches, Semantic Metadata Creation, and Language Gener-ation.

Provider University of SheffieldLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID=lr-usfd-di-

ana-1097059567085Categories Web Data Extraction / Information Extraction, NLP / HLT, Outreach to

IndustryType PDF, html, PPT

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 22

Length 4:00h

Title Perspectives for Semantic Web Applications in EuropeAbstract What are the perspectives for applications based on the Semantic Web in

European industry? On the basis of the work in KnowledgeWeb, we eval-uate the current state of play and how KnowledgeWeb will facilitate the industrial uptake of this new technology.

Provider Free University BerlinLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-freea-

lnixon-1118762437312 Categories Outreach to IndustryType PDF, PPTLength 0:30h

Title Practical Applications of Human Language Technologies for the Seman-tic Web

Abstract This 4-hour tutorial presented at the ACAI -05 Advanced Course in Knowledge Technologies SEKT Summer School covers the use of Hu-man Language Technologies for the Semantic Web and Web Services, fo-cusing particularly on practical applications. It gives some introduction to text mining and Information Extraction, and aims to show how such core technologies can be adapted to deal with the needs of the Semantic Web, by means of real-life examples and applications.

Provider University of SheffieldLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-usfd-di -

ana-1118919150028Categories Web Data Extraction / Information Extraction, Semantic Web Applica-

tions, NLP / HLT, Outreach to IndustryType PDF, PPTLength 4:00h

Title HLT and Knowledge Acquisition for the Semantic Web: A Hands On Tu-torial

Abstract The core of this tutorial covers HLT tools, followed by a number of ex-ample Semantic Web applications, built by non-specialist HLT re-searchers. It covers the use of (1) GATE tools for deriving web service ontologies from text; (2) Text2Onto, an HLT-based paradigm for ontol-ogy construction; and (3) research on automatic ontology population from text and massive semantic annotation.

Provider University of Sheffield, AIFB University of Karlsruhe, Vrije Universiteit Amsterdam

Language EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-usfd-di -

ana-1118922707478Categories Web Data Extraction / Information Extraction, Ontology Population /

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 23

Generation, Tools, NLP / HLT, Outreach to IndustryType PDF, PPTLength 4:00 h

Title Schema and Ontology MatchingAbstract We view Matching as one of the key operations for enabling the Semantic

Web since it takes two schemas/ontologies, each consisting of a set of dis-crete entities (e.g., tables, XML elements, classes, properties, rules, predi-cates), as input and determines as output the relationships (e.g., equiva-lence, subsumption) holding between those entities. In this tutorial we in-troduce, via examples, the schema/ontology matching problem and its ap-plication domains. We provide a detailed discussion of the techniques used for schema/ontology matching with the help of a classification of matching approaches. We overview state of the art systems in light of the classification presented, indicating which part of the solution space they cover. Finally, we outline future research directions and new scientific challenges arising in schema/ontology matching.

Provider University of Trento, INRIALanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-univ-

pavel-1121707366586 Categories Outreach to Industry, Mapping / Translation / Matching / Aligning (Het-

erogeneity)Type PDF, LatexLength 3:00h

Title Semantic Web Use CasesAbstract This will give an overview of typical business problems in different fields

and their potential solution through Semantic Web technologies. We illus-trate this through exemplary use cases collected by KnowledgeWeb and specify how through the co-operation between industry and research we can achieve successful technology transfer.

Provider Free University BerlinLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-freea-

lnixon-1129891830518Categories Outreach to Industry, Knowledge Management, Multimedia, eBusinessType PDF, PPTLength 1:00h

Title The Semantic Web and the Future of Social SoftwareAbstract Short introduction to the Semantic Web and how it can enhance social

software.Provider National University of Ireland, GalwayLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID=l r-natia-

jbreslin-1133280396675Categories Outreach to Industry Social Impact of the Semantic Web

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 24

Type PDFLength 0:30h

Title Blogging for Business: Syndication and RSSAbstract Short introduction to syndication and RSS at the "Blogging for Business"

event in Cork. http://www.itcork.ie/index.cfm?page=events&eventId=47Provider National University of Ireland, GalwayLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-natia-

jbreslin-1133201478703Categories Basic Web Information Technology, Outreach to IndustryType PDFLength 0:30h

Title Semantic Web Services: A state of the art reportAbstract Gives an overview about the most prominent approaches in the area of

Semantic Web Services.Provider Vrije Universiteit AmsterdamLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-vrij-hol -

ger-1133372701206Categories Semantic Web Services, Outreach to IndustryType PDFLength 1h

Title The Semantic Web: How to bring technology to the marketAbstract This is a one-hour video recording of the presentation of Richard Ben-

jamins at the KnowledgeWeb summer school 2005.Provider ISOCO, Spain (content) / University of Trento (recordings)Language EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1165414149785 Categories eBusiness, Outreach to IndustryType PPT, PDFLength 1:00h

Title Ontology Design Patterns and Problems: Practical Ontology Engineering using Protege-OWL

Abstract An introduction to ontology engineering issues, including: * upper-ontologies * using a reasoner at creation time - ontology normalisation * ontology patterns - value partitions and enumerations - n-ary relations * classes as values * part-whole relationships

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 25

* qualified cardinalityProvider University of ManchesterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

theu-nickdrummond-1134031414043 Categories Tools, Ontology Engineering, Outreach to Industry, ReasoningType PDF, PPTLength 3h

Title A Practical Introduction to Ontologies & OWLAbstract 3 part practical introduction to building OWL ontologies, including rea-

soning and common errors. Based on the ProtegeOWL tool, but much of the content is not specific to this tool

Provider University of ManchesterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

theu-nickdrummond-1134035410370Categories Tools, Ontology Engineering, Ontology Representation / Ontology Lan-

guages / OWL, Ontology Population / GenerationType PDF, PPT

Title Web Service Modelling eXecution environmentAbstract This tutorial explains how WSMX can address some issues relating to

web services, and gives examples in the domains of e-banking, govern-ment and telecommunications.

Provider National University of GalwayLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

natia-jbreslin-1135088696250Categories Web Services, Semantic Web Applications, eGovernment, Semantic Web

ServicesType PDFLength 0:50

Title Benchmarking Semantic Web technologyAbstract The Semantic Web technology needs to be thoroughly evaluated for pro-

viding objective results and to attain a massive improvement in their qual-ity in order to be consolidated in the industrial and in the academic world. This paper presents software benchmarking as a process to carry out over the SemanticWeb technology in order to improve it and to search for best practices. It also describes a software benchmarking methodology and provides recommendations for performing evaluations in benchmarking activities.

Provider Universidad Politecnica de MadridLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

univa-rgarcia-1165913049094

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 26

Categories Benchmarking and Scalability, Outreach to IndustryType PDFLength 1h

Title Use of Ontology for production of access systems on Legislation, Ju-risprudence and Comments

Abstract This presentation was given as part of the industrial day at ESWC 2006.

Wolters Kluwer Belgium publishes about specialized areas related to leg-islation, jurisprudence and doctrine. The paper reports on an effort to transfer knowledge, scattered over a divers set of classification, coding and index generation systems, into a central thesaurus system, modeled and controlled by an ongtoloy.

Provider L3S Research CenterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1166447475348Categories Ontologies for the Semantic Web, Knowledge Management, LawType PDFLength 0:30h

Title Know how to use Know-howAbstract This presentation was given as part of the industrial day at ESWC 2006.

For the integration of data that resides in autonomous data sources Soft-ware AG uses ontologies. Data source ontologies describe the data sources themselves. Business ontologies provide an integrated view of the data. F-Logic rules are used to describe mappings between data objects in data source or business ontologies. Furthermore, F-Logic is used as the query language. F-Logic rules are perfectly suited to describe the map-pings between objects and their properties. In a first project we integrated data that on one side resides in a support and on the other side in a cus-tomer information system.

Provider L3S Research CenterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1166447181344Categories Web Data Integration, Knowledge ManagementType PDFLength 0:30h

Title Semantic Web Technology RoadMap: the case of the KnowledgeWeb Technology Roadmap

Abstract This presentation was given as part of the industrial day at ESWC 2006.

The Technology Roadmap activities promote a discussion on (i) the cur-rent and future trends on semantic web tools and applications, (ii) general organizational needs (common markets and social drivers, user require-

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 27

ments, etc.), and (iii) technology locks between organizational/user needs and research activities. Some emerging challenges should be unveiled and recommendations should be provided. These results are disseminated widely in order to allow researchers and entrepreneurs to address their ac-tivities, in a mutually beneficial way.

Provider L3S Research CenterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1166445347789Categories Outreach to IndustryType PDFLength 0:30h

Title Knowledge Management in the Petroleum IndustryAbstract This presentation was given as part of the industrial day at ESWC 2006.

The AKSIO project is developing a process-enabled knowledge manage-ment system to support operations of offshore oilfields. The system will provide timely and con-textual knowledge for work processes. Experiences will be processed and annotated by experts and linked to various resources and specialist knowl-edge networks. AKSIO will allow discovery of experiences through the support of a domain ontology. Core functionality of the AKSIO system is provided by careful application of Semantic Web technology, including ontology-based annotation and contextual ontology driven retrieval of content.

Provider L3S Research CenterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1166445051656Categories Knowledge Engineering / Ontology Engineering, Knowledge Manage-

mentType PDFLength 0:30h

Title Integrated Access to Biological DataAbstract This presentation was given as part of the industrial day at ESWC 2006.

This use case reflects the research on different and innovative ways to handle biological data repositories by means of semantic and artificial intelligence technologies such as ontologies, intelligent agents, semantic grid, etc. The human genome sequencing has given rise to a great number of biological data repositories that once analysed will be very essential for the study of diseases, pharmaceutical research, new treatments and for the development of new bio products. The problem faced is the huge quantity and heterogeneity of this kind of data and the also huge number and diver-sity of ontologies defined to model biological data.

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 28

Provider L3S Research CenterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1166443995355Categories Web Data Integration, Mapping / Translation / Matching / Aligning (Het-

erogeneity), BioinformaticsType PDFLength 0:30h

Title Automating BPM with SWS TechnologiesAbstract This presentation was given as part of the industrial day at ESWC 2006.

In this paper, we aim to investigate how semantic Web services can im-prove standard business process management tools. Based on a standard SAP process in the area of logistics, we show how current approaches support business flexibility via manual modeling tools. Our application of semantic Web service technologies on top of today?s business process management tools enables the automation of major tasks of business process management.

Provider L3S Research CenterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1166444283598Categories eBusiness, Semantic Web ServicesType PDFLength 0:30h

Title Training Management System for Aircraft Engineering: indexing and re-trieval of Corporate Learning Objectp

Abstract This presentation was given as part of the industrial day at ESWC 2006.

Training management in a company may benefit of a better integration with competence management outcomes. This paper is about an initial ex-ploration of this proposal. It proposes a specific approach to support the indexing and retrieval of training courses with regard to the professions? target competences. This approach is grounded on Learning Object meta-data, and semantic web (SW) technologies enabling advanced search and reasoning on Learning Object description. We intend to implement it us-ing the KINOA prototype platform that contains an annotation editor and a semantic search server. The approach requires that a semantic Learning Object repository is built on several existing data sources. Standards from IEEE LOM and AICC are used as a starting point for the building of the semantic learning object repository and extended to fit with our needs and context.

Provider L3S Research CenterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1166444646149

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 29

Categories eLearningType PDFLength 0:30h

Title iPad: Semantic Laboratory NotebookAbstract This presentation was given as part of the industrial day at ESWC 2006.

The main raw product of biomedical research is the information contained in laboratory notebooks and the associated computer files of individual re-searchers. Most of the problems in managing bioresearch information downstream stem from the way this information is initially recorded and stored. Electronic notebooks based on traditional knowledge management approaches have not been widely adopted by bio-researchers ? the vast majority still use paper notebooks. We describe deployment of a software system based on the semantic tagging approach that successfully ad-dresses the key adoption problems. This case study also indicates fruitful directions for the future R&D.

Provider L3S Research CenterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1166443111973Categories BioinformaticsType PDFLength 0:30h

3.2.2 Full-course materialThis section summarized the material on REASE which covers full courses in academia that might be usable in part by industry. Title Semantic Web LectureAbstract This lecture comprises four modules, which are kept separately on

REASE.Introduction and Overview: This first module of the Semantic Web Lec-ture describes the background on WWW and Semantic Web and intro-duces several markup languages such as HTML and XML. Furthermore, cascading style sheets, XPATH, and XSL are described.Basic building blocks: This second module of the Semantic Web Lecture describes the Semantic Web components RDF, RDF Schema, OWL and gives a brief introduction to ontology engineering.Logics: This third module of the Semantic Web Lecture covers the logics layer of the Semantic Web. It gives an introduction to logical languages, rule systems and rule markup languages. It covers aspects of trust and pol-icy management in Semantic Web as well as Semantic Web Services.Adaptive Hypermedia Systems: This fourth module of the Semantic Web Lecture covers an example for an advanced topic in the area of Semantic Web: Adaptive Hypermedia Systems.

Provider L3S Research CenterLanguage English

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 30

URL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID=lr-lear-diederich-1095939128541http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID=lr-lear-diederich-1095948083855 http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID=lr-lear-diederich-1096016131071 http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID=lr-lear-diederich-1096017582439

Categories Basic Web Information Technology, XML, Ontologies for the Semantic Web, RDF/RDFS, Ontology Engineering, Logics, Security/Privacy/Trust, Semantic Web Rules + Logics, Rule Markup, Social Networks and the Semantic Web

Format HTML, PDFLength 4 x 8:00h

Title Knowledge Management and Retrieval with Ontologies and Topic MapsAbstract Ontology-based knowledge management (6 h), Topic Maps (1.5) and

Knowledge Retrieval (1.5)Provider AIFB, University of KarlsruheLanguage GermanURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-aifb-

ukarl-1097520909219 Categories Ontology Representation / Ontology Languages / OWL, Basic Web Infor-

mation Technology, Knowledge ManagementFormat PDF, PPTLength 9:00h Title Knowledge Management II: Tools and ApplicationsAbstract Case-based Reasoning (CBR), Community of Practice (CoP), Data

Warehouse, Geschäftsprozessorientiertes WissensmanagementProvider AIFB, University of KarlsruheLanguage GermanURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-aifb-

ukarl-1097521506331 Categories Reasoning, eBusinessFormat PDF, PPTLength 6:00

Title Knowledge Engineering applied to Semantic WebAbstract Complete course on knowledge engineering techniques and formalisms

including:- ergonomics and scenario-based specifications;- ontology life cycles;- knowledge representation formalisms;- semantic web formalisms;- evaluation techniques;- semantic search engines;

Provider INRIA

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 31

Language FrenchURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-inri-fa -

bien_gadon-1097853456105 Categories Basic Web Information Technology, Knowledge Engineering / Ontology

Engineering, Ontologies for the Semantic Web, RDF / RDFSFormat PDFLength 50:00h

Title Web-based Knowledge RepresentationAbstract The WWW offers a great opportunity for using well-established and new

knowledge representation techniques.The aim in using these is to make web pages intended for human users accessible for machines as well. Such a web would enable a set of intelligent services such as: search-en-gines, information filters, adaptive web-sites a.s.o. This course presents the technology that enables the new generation of the web. It presents knowledge modeling concepts (ontologies) and knowledge representation languages developed for the web (XML, RDF, OWL). We investigate the increasing expressiveness of these languages and point out issues for fu-ture research in this field.

Provider Vrije Universiteit AmsterdamLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-vrij-

frankh-1098889115195 Categories RDF / RDFSFormat PDF

Title Introduction to Description LogicsAbstract The main effort of the research in knowledge representation is providing

theories and systems for expressing structured knowledge and for access-ing and reasoning with it in a principled way. In this course we will study Description Logics (DL), an important powerful class of logic-based knowledge representation languages (see www.dl.kr.org). The emphasis will be on a rigorous approach to knowledge representation and building ontologies. After an original review of the relevant concepts on computa-tional logics, the course will start with an introduction to Object-Oriented representations in Information Systems and Artificial Intelligence, which serve as the main motivations for studying DL. DL will be introduced with its simplest formalization; the computational properties and algo-rithms of the so called structural DL will be analyzed. Then, the course considers propositional DL: we will study the computational properties and the reasoning with tableaux calculus. In the second part of the course, we will consider advanced topics such as the representation of knowledge bases and ontologies, and the connections of DL with Modal Logics and First Order Logic. The last module of the course will analyze the connec-tions of DL with database theory.

Provider Free University of Bozen-BolzanoLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-free-fran -

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 32

coni-1099402926874 Categories Logics, Knowledge Engineering / Ontology EngineeringFormat PDFLength 24:00h

Title CT433.iii: Advanced Topics in IT: Semantic Web and Semantic Web Ser-vices

Abstract This first half of this stream will introduce the Semantic Web and de-scribe the metadata and ontological structures that are being used to build it. The second half will focus on the application of Semantic Web Ser-vices technology to B2B integration, including state-of-the-art implemen-tations and standards.

The main topics are as follows: Motivation for the Semantic Web Semantic Web Aspects Metadata and Semantics Data and Metadata Markup Languages and Formats Metadata Annotation Tools and Techniques Ontologies and Schemata Information Integration Synergies, ROI and Impact of the Semantic Web Introduction to Semantic Web Services and B2B Integration History and Current State Technology Concepts, Functionality and Execution Model Architectures and Implementations Products and Standards

Provider National University of Ireland, GalwayLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-natia-

jbreslin-1133192611850 Categories Semantic Web Applications, Semantic Web InfrastructureFormat PDFLength 12:00h

Title Ontological Engineering and the Semantic WebAbstract This slideset contains the material used in the 30-hour seminar "On-

tologías y Web Semántica" taught by Oscar Corcho in Tandil (Argentina) in April 2006, as part of a postgraduate school from the ProTIC network. (http://www.exa.unicen.edu.ar/escuelapav/cursos/ontologias.htm). Al-though the seminar was taught in Spanish, the material is available in English. This slideset is an extension of a previous REASE learning re-source submitted by the same authors.

Provider Universidad Polytecnica de Madrid, University of ManchesterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-theu-ocorcho-

1161777588280Categories Knowledge Engineering / Ontology Engineering, Ontologies for the Se-

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 33

mantic Web, Semantic Web ApplicationsType: PDFLength 30:00h

3.2.3 Miscellaneous Modules related to Semantic Web MaterialThis and the remaining section cover smaller modules and tutorial, which have not been classified into ‘Semantic Web Special Topics’ and have not been classified into ‘ Outreach to industry’ or comprise a full course. In this section, we start with material not directly related to Semantic Web, but useful for background knowledge.

Title Introduction to XSLAbstract A short introduction to XSL and XSLTProvider University of TrentoLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-univ-

ronchet-1097778439781 Categories Basic Web Information Technology

Title Introduction to XMLAbstract Powerpoint presentation: a short introduction to XMLProvider University of TrentoLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID lr-univ-

ronchet-1097777977326 Categories Basic Web Information Technology

Title Introduction to Java tools for dealing with XMLAbstract Introduction to various Java APIs for manipulating XML data with SAX

and DOM, and to apply XSL transformations (TRax)Provider University of TrentoLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-univ-ronchet-

1097779082452 Categories Basic Web Information Technology

3.2.4 Modules about Core Topics around Semantic WebThis section covers all modules in REASE which deal mainly with core topics around Semantic Web.

Title Introduction to Knowledge-Level Models of Problem SolvingAbstract This is a 40 minutes powerpoint presentation introducing the basics of

knowledge-level models of problem solving. The presentation illustrates the evolution of knowledge-based systems from the early rule-based shells to the current architectures based on the distinction between generic tasks, problem solving methods, domain models and application-specific knowledge

Provider The Open University

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 34

Language EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-theo-

emotta-1097763040129 Categories Knowledge Engineering / Ontology Engineering, Methodologies

Title Classification Problem SolvingAbstract An analysis of classification problem solving using a knowledge-level ar-

chitecture for characterizing knowledge-based problem solvingProvider The Open UniversityLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-theo-

emotta-1097764942806 Categories Knowledge Engineering / Ontology Engineering, Methodologies Title RDF, Resource Description FrameworkAbstract Ce cours présente le langage RDF dans son utilisation dans le Web sé-

mantiaueProvider INRIALanguage FrenchURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID lr-inri-syl -

vain_d-1097824786423 Categories RDF / RDFS

Title User Models and User Modeling for Knowledge Management Systems: An ontology based User Modeling Approach

Abstract PhD defense, Liana Razmerita,3rd December 2003

Provider INRIALanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-inri-syl -

vain_d-1098105953207 Categories Knowledge Management

Title Methods and tools for corporate memoriesAbstract Introduce corporate memories and describe the Corporate Semantic Web

(CSW) Approach. Presented during a summer school.Provider INRIALanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-inri-syl -

vain_d-1098114616208 Categories Methodologies, Tools, Knowledge Management

Title Méthodes et Outils pour la Gestion des ConnaissancesAbstract DESCRIPTION Le cours traite les points suivants:

Définitions et Besoins industrielsTypologie des connaissancesModèles pour la gestion des connaissancesMémoire d’entreprise

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 35

Approche Web sémantique d’entrepriseExemplesConclusions

Provider INRIALanguage FrenchURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-inri-syl -

vain_d-1098117238321 Categories Methodologies, Tools, Knowledge Management

Title Description Logics for Conceptual Design, Information Access, and On-tology Integration

Abstract In the tutorial I will argue that good Conceptual Modelling and Ontology Design is required to support powerful Query Management and to allow for semantic based Information Integration. Therefore, the tutorial has been structured into three parts:* In the first part, an extended ontology language and a methodology for conceptual and ontology design will be introduced.* In the second part, the query management problem in the presence of the previously devised conceptual model will be considered: a global framework will be introduced, together with various basic tasks involved in information access.* In the last part, general issues about ontology integration will be pre-sented.

Provider Free University of Bozen-BolzanoLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-free-fran -

coni-1099402118641 Categories Knowledge Engineering / Ontology Engineering

Title A Review of Computational LogicsAbstract This learning resource provides a in-depth introduction into Propositional

Logic (Foundations and deduction) and into First Order Logic (Founda-tions and how to use it)

Provider Free University of Bozen/BolzanoLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-free-franconi-

1166031349119 Categories LogicsType PDF

Title Ontological EngineeringAbstract This tutorial presents the theoretical foundations of Ontological Engineer-

ing, describes the most outstanding ontologies that are currently available, and covers the practical aspects of selecting and applying methodologies, languages, and tools for building ontologies. This tutorial also aims at pre-senting commercial-oriented and research-oriented ontology-based appli-cations.

Provider Universidad Politecnica de Madrid

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 36

Language EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-univa-

asun-1099404115104 Categories Ontologies for the Semantic Web

Title OWL Tutorial: Introduction to Ontology Development and Protégé-OWLAbstract Extensive OWL tutorial materialsProvider The University of ManchesterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-theu-jeff-

1100715238891 Categories Semantic Web Special Topics

Title Introduction to Semantic Web Ontology LanguagesAbstract Tutorial, jointly created with Grigoris Antoniou, at the REWERSE Sum-

mer School 2005.Provider Free University of Bozen-BolzanoLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-free-fran -

coni-1122522631796Categories Logics

Title Motivation for fuzzy OWLAbstract Few slides motivating more fuzzy OWL reasoningProvider Vrije Universiteit AmsterdamLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-vrij-hol -

ger-1133370220395Categories Ontology Representation / Ontology Languages / OWL, Reasoning

Title Ontology mapping: a way out of the medical tower of Babel?Abstract Overview of existing approaches for ontology mappingsProvider Vrije Universiteit AmsterdamLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-vrij-hol -

ger-1133369895277Categories Mapping / Translation / Matching / Aligning (Heterogeneity), Ontology

Representation / Ontology Languages / OWL

Title Fundamental Research Challenges Generated by the Semantic WebAbstract A 1 hour video about the research challenges in Semantic WebProvider Vrije Universiteit AmsterdamLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-vrij-hol -

ger-1133383390634 Categories Ontologies for the Semantic Web, Knowledge Representation and Rea-

soning, Knowledge Engineering / Ontology Engineering, Semantic Web

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 37

ApplicationsType Video, PPT, PDFLength 1:00h

Title OWL: An Ontology Language for the Semantic WebAbstract Tutorial given at the Third KnowledgeWeb Summer School on Ontologi-

cal Engineering and the Semantic Web (SSSW '05)Provider The University of ManchesterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-theu-

seanb-1133441337998Categories Ontology Representation / Ontology Languages / OWLType: Video, PPT, PDFLength 1:00h

Title OWL Reasoning ExamplesAbstract A collection of on-line examples illustrating the effects of inference and

reasoning. Presented as hands-on material during the third Knowl-edgeWeb Summer School on Ontological Engineering and the Semantic Web (SSSW'05)

Provider The University of ManchesterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-theu-

seanb-1133441594714Categories Ontology Representation / Ontology Languages / OWL

Title Ontology Validation and EvaluationAbstract This is a one-hour video recording of the presentation of Aldo Gangemi at

the KnowledgeWeb summer school 2005. It comprises either the video synchronized with the slides (but requires Quicktime, hence Windows or MacOS, otherwise the slides have to be switched manually).

Provider L3S Research CenterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1165400925956Categories ValidationType: VideoLength 1:00h

Title Satisficing Ontology MappingAbstract This is a one-hour video recording of the presentation of Steffen Staab at

the KnowledgeWeb summer school 2005. It comprises either the video synchronized with the slides (but requires Quicktime, hence Windows or MacOS, otherwise the slides have to be switched manually). It provides an in-depth view with concrete example mappings while the presentation of Natasha Noy provides the general overview.˜

Provider L3S Research Center

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 38

Language EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1165406019525Categories Mapping / Translation / Matching / Aligning (Heterogeneity)Type: VideoLength 1:00h

Title Ontology Mapping and AlignmentAbstract This is a one-hour video recording of the presentation of Natasha Noy at

the KnowledgeWeb summer school 2005. It comprises either the video synchronized with the slides (but requires Quicktime, hence Windows or MacOS, otherwise the slides have to be switched manually). It provide a high-level overview on ontology mapping while the presentation of Stef-fen Staab provides more details and example systems.

Provider L3S Research CenterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1165405050164Categories Mapping / Translation / Matching / Aligning (Heterogeneity)Type: VideoLength 1:00h

Title Ontology Engineering MethodologiesAbstract This is a one-hour video recording of the presentation of Asun Gomez

Perez at the KnowledgeWeb summer school 2005. It comprises either the video synchronized with the slides (but requires Quicktime, hence Win-dows or MacOS, otherwise the slides have to be switched manually).

Provider Universidad Polytecnica de MadridLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1165395787366Categories Methodologies, Ontology EngineeringType: VideoLength 1:00h

Title Semantic web tutorial: RDF, RDFS and SPARQL using CORESEAbstract This semantic web tutorial gives a quick tour of RDF, RDFS, SPARQL

and Rules. It was designed as a hand-on-keyboard introduction to the ba-sics of RDF model, RDFS semantics for lightweight ontologies, SPARQL query language for RDF graph bases and production rules for knowledge factorisation in semantic web annotation bases.

Provider INRIALanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

inri-sylvain_d-1166175161614Categories Tools, Resource Description Framework (RDF) / RDFSchema, Query

Languages

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 39

Type: HTML

3.2.5 Modules about Special Topics around Semantic WebThis section describes the modules, which have been classified into ‘Semantic Web Special Topics”, but not into ‘Outreach to Industry’.

Title WSMO TutorialAbstract The tutorial is intended to disseminate the Web Service Modeling Ontol-

ogy WSMO to worldwide audiences interested in Semantic Web Services. IRS-III is the tool used in the hands-on session

Provider The Open University, DERILanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-theo-lil -

iana-1097851359341 Categories Semantic Web Special Topics, Semantic Web Services Title Distributed Artificial Intelligence and Knowledge Management: ontolo-

gies and multi-agent systems for a corporate semantic webAbstract This Ph.D. Thesis concerns multi-agents systems for the management of a

corporate semantic web based on an ontology. It was carried out in the context of the European project CoMMA focusing on two application sce-narios: support technology monitoring activities and assist the integration of a new employee to the organisation. Three aspects were essentially de-veloped in this work:the design of a multi-agents architecture supporting both scenarios, and the organisational top-down approach followed to identify the societies, the roles and the interactions of agents;the construction of the ontology O'CoMMA and the structuring of a cor-porate memory exploiting semantic Web technologies;the design and implementation of the sub-societies of agents dedicated to the management of the annotations and the ontology and of the protocols underlying these groups of agents, in particular techniques for distributing annotations and queries between the agents.Keywords: distributed artificial intelligence, knowledge management, corporate memory, ontology, knowledge representation, multi-agent sys-tems, semantic web, information retrieval.

Provider INRIALanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-inri-fa -

bien_gadon-1098109372460 Categories Ontologies for the Semantic Web, Basic Web Information Technology,

Knowledge Engineering / Ontology Engineering, RDF / RDFS, Semantic Web Special Topics

Title Knowledge Assisted Multimedia AnalysisAbstract This is a 3-hour powerpoint presentation introducing the basics in knowl-

edge assisted multimedia analysis. The presentation gives emphasis on the

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 40

knowledge representation infrastructure for semantic multimedia content analysis and reasoning. It also includes an overview of existing multime-dia analysis, annotation and search and retrieval methods.

Provider CERTHLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID lr-cent-vk -

papa-1098345323868 Categories Semantic Web Special Topics, Multimedia

Title Document Annotation Through Information ExtractionAbstract Tutorial presented at the Second European Summer School on Ontologi-

cal Engineering and the Semantic Web, 18-24 July 2004 - Cercedilla (Spain) , http://babage.dia.fi.upm.es/summerschool/

Provider University of SheffieldLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID=l r-usfd-fci -

ravegna-1099915337904 Categories Web Data Extraction, NLP / HLT Title Introduction to Multi-agent systemsAbstract Multi-agent systems have emerged as one of the most important areas of

research and development in information technology in the 1990s. A multi-agent system is one composed of multiple interacting software com-ponents known as agents, which are typically capable of co-operating to solve problems that are beyond the abilities of any individual member. Multi-agent systems are important primarily because they have been found to have very wide applicability, in areas as diverse as industrial process control and electronic commerce. This module will begin by in-troducing the student to the notion of an agent, and will lead them to an understanding of what an agent is, how they can be constructed, and how agents can be made to cooperate effectively with one-another to solve problems. The practical component of the module will based on the many Java agent frameworks currently available (e.g., the Java-based ``Jack'' programming language).

Provider University of LiverpoolLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-univb-

valentina-1099517115906 Categories Agents and the Semantic Web

Title Text mining and the Semantic WebAbstract This hour-long tutorial gives an introduction to text mining issues for the

Semantic Web, covering topics such as what text mining is, an introduc-tion to information extraction and how it can be adapted for the Semantic Web, evaluation and visualisation tools and techniques. It is intended pri-marily for undergraduate and postgraduate students, but could equally serve as a learning tool for researchers new to the area of Human Lan-guage Technology and the Semantic Web.

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 41

Provider University of SheffieldLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-usfd-di -

ana-1110385153182 Categories NLP / HLT

Title Automating Document Annotation using Human Language Technologies and Machine Learning

Abstract Tutorial given at the Third Semantic Web Summer School in Cercedilla, Spain, http://babage.dia.fi.upm.es/sssw05/10-16 July 2005

Provider University of SheffieldLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-usfd-

cirave-1122455322366Categories Web Data Extraction, NLP / HLT

Title Semantic Grid: A love match? or A marriage of convenience?Abstract This is a one-hour video recording of the presentation of Carole Goble at

the KnowledgeWeb summer school 2005. It comprises either the video synchronized with the slides (but requires Quicktime, hence Windows or MacOS, otherwise the slides have to be switched manually).

Provider University of ManchesterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1165389150620Categories Semantic GridType: VideoLength 1:00h

Title Natural Languages and Ontology LearningAbstract This is a one-hour video recording of the presentation of Roberto Basili at

the KnowledgeWeb summer school 2005. It comprises either the video synchronized with the slides (but requires Quicktime, hence Windows or MacOS, otherwise the slides have to be switched manually).

Provider L3S Research CenterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1165410214501Categories Web Data Extraction / Information Extraction, Ontology Population /

Generation, Natural Language Processing / Human Language Technolo-gies

Type: VideoLength 1:00h

Title Multiagent Systems: Past, Present, and FutureAbstract This is a one-hour video recording of the presentation of Mike

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 42

Wooldridge at the KnowledgeWeb summer school 2005. It comprises ei-ther the video synchronized with the slides (but requires Quicktime, hence Windows or MacOS, otherwise the slides have to be switched manually).

Provider University of LiverpoolLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1165412451032Categories Agents and the Semantic WebType: VideoLength 1:00h

Title Semantic Web ServicesAbstract This is a one-hour video recording of the presentation of John Domingue

at the KnowledgeWeb summer school 2005. It comprises either the video synchronized with the slides (but requires Quicktime, hence Windows or MacOS, otherwise the slides have to be switched manually).

Provider The Open UniversityLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lear-diederich-1165413392914Categories Semantic Web ServicesType: VideoLength 1:15h

Title Semantic Grid Tutorial: Semantic Sticky NoteAbstract This practical tutorial provides an overview of Semantic Grid concepts by

means of a hands-on example where students have to create sticky notes with RDF descriptions attached to them. This tutorial is based on the tuto-rial "How to Build a Service Using GT4", also known as BAS GT4 Sticky Note tutorial.

Provider University of ManchesterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

theu-ocorcho-1165594638147Categories Semantic GridType: HTMLLength 2:00h

Title Semantics-based Peer-to-Peer SystemsAbstract This presentation is about the basic concepts of Semantic Web, Peer-to-

Peer systems, and social networks. It also shows how to integrate Seman-tics into Peer-to-Peer networks and combine it with the social network concepts using the Bibster system for exchanging bibtex descriptions of publication records.

Provider University of KarlsruheLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 43

aifb-ysu-1165399369226Categories Ontologies for the Semantic Web, Peer-to-peer and the Semantic Web,

Social Networks and the Semantic WebpType: PDFLength 1:30h

Title Tutorial Series - Semantic Web ServicesAbstract This is a series of over 20 tutorials on Semantic Web Services that have

been presented from 2004 through 2006 in related international events. The purpose of each tutorial is to introduce into the field and provide a comprehensive overview on the WHY and HOW of semantically enabled technologies for automated Web service usage. The tutorials are struc-tured into 3 main sections: 1) introduction & SWS Frameworks (WSMO, OWL-S, etc.) 2) SWS techniques (discovery, composition, execution, mediation) & sys-tems (SWS execution environments) 3) Hands-On sessions with IRS & WMX

The material, software, and additional information are provided on the website (see link below)

Provider University of InnsbruckLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

unive-mstollberg-1157532228058Categories Semantic Web ServicesType: HTML

Title What you Mean is What you Watch: Multimedia and the Semantic WebHAbstract This tutorial covers the hot topic of multimedia and the Semantic Web

with focus on multimedia search engines, automatic semantic annotation of multimedia, and use of semantic web tools in the production of new media formats. The tutorial also covers some open-source tools, thus en-abling the participants to put their newly learned skills into practice. The material presented includes the latest research results from several Euro-pean projects on multimedia and semantically-enabled knowledge tech-nologies.

Provider University of SheffieldLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

usfd-diana-1152536381899Categories Multimedia, Natural Language Processing / Human Language Technolo-

giesType PDFLength 3:00h

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 44

3.2.6 Courses contributed by REWERSEThese courses are mainly related to rules, rule languages, the underlying logics, and personalization. Though they were not contributed by KnowledgeWeb partners, we list them here also to have a complete overview on the available material in REASE.

Title Rules and Ontologies in F-logicAbstract A brief introduction to F-logic and its use for ontology specification.

Slides of a lecture given at the Reasoning Web summer school, July 2005, Malta.

Provider State University of New York at Stony BrookLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-stat-kifer-

1118033616456

Categories Ontologies for the Semantic Web, Logics, Logic Programming, Rule Lan-guages

Title Knowledge-base Programming with Frames and LogicAbstract This is a tutorial on knowledge representation using the FLORA-2 sys-

tem. FLORA-2 combines F-logic, HiLog, and Transaction Logic in a powerful declarative language. More information as well as the system it-self can be found at http://flora.sourceforge.net/

Provider State University of New York at Stony BrookLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-stat-kifer-

1118035174076 Categories Logics, Ontologies for the Semantic Web, Logic Programming

Title Web and Semantic Web Query Languages: A SurveyAbstract This learning unit presents an overview on existing web and Semantic

Web query languages and presents some of them in more detail, namely XML, RDF and Topic Maps.

Provider LMULanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-lmua-bry-

1118475429024 Categories RDF / RDFS, Query Languages

Title Information Extraction for the Semantic WebAbstract Web Information Extraction and Integration: Introduction, Overview,

Case Studies and System Demonstration. Slides of a lecture given at the Reasoning Web summer school, July 2005, Malta.

Provider DBAI, Vienna University of TechnologyLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-dbai-

robert-1118694353138Categories Basic Web Information Technology, Web Data Extraction, Web Data In-

tegration

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 45

Title Personalization for the Semantic Web -Part II-Abstract Personalization is a process by which it is possible to give the user opti-

mal support in accessing, retrieving, and storing information, where solu-tions are built so as to fit the preferences, the characteristics and the taste of the individual. This result can be achieved only by exploiting machine-interpretable semantic information, e.g. about the possible resources, about the user him/herself, about the context, about the goal of the inter-action. Personalization is realized by an inferencing process applied to the semantic information, which can be carried out in many different ways depending on the specific task. The objective of this paper is to provide a coherent introduction into issues and methods for realizing personaliza-tion in the Semantic Web.

Provider Dip. di Informatica, Universita' degli Studi di TorinoLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-dipd-bal -

doni-1119445597087 Categories Personalization Techniques, Semantic Web Special Topics, eLearning

Title Evolution and Reactivity on the Semantic WebAbstract In this course, presented at the Reasoning Web Summer School, July

2005, Malta, we talk about foundations of evolution and reactive lan-guages in general, and then concentrate on some specific issues posed by evolution and reactivity in the Web and in the Semantic Web.

Provider F. Ciências Tecnologia, U. Nova Lisboa, University of GöttingenLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-fcin-jja-

1121071930599Categories Rule Languages, Update Languages, Logics

Title Personalization for the Semantic Web, Part IAbstract This module describes personalization techniques for WWW-based sys-

tems. Topics are user modeling, adaptive hypermedia, and Web mining-based personalization.

Provider L3S Research CenterLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-lear-

henze-1119511340834 Categories Personalization Techniques

Title Towards Types for Web Rule LanguagesAbstract Various schema languages have been introduced to describe

(classes of) Web documents (DTD, XML Schema, Relax NG). Wepresent mathematical treatment of their main features. We areinterested in the sets of documents a schema defines; such setswill be called types. Using a mathematical formalism makes itpossible to discuss chosen aspects of a schema language in aprecise and simple way. Otherwise they are hidden among

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 46

numerous details of a large and sophisticated schema language.

Our goal is typing of rule languages, more preciselyapproximately describing their semantics by means of types.Thus we are interested in formalisms for types that facilitateconstructing (efficient) algorithms performing those operationson types that are needed in type checking and type inference forrules.

Provider Linköping UniversityLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-link-

wlodr-1129138244147 Categories Rule Languages, Query Languages

Title Rational Agents in Logic Programming for the Semantic WebAbstract Talk at U. Linköping, September 29th, 2005

Overview of the Semantic Web The REWERSE project Overview of select LP features Dynamic Logic Programming Evolving Logic Programs Reasoning Integration Framework Semantic Web Application Left out LP features Project W4: Well-founded semantics for the World Wide Web

Provider LissabonLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

fcin-lmp-1135242087035Categories Tools, Logic ProgrammingType PPT, PDF, audio recordings

Title The Semantic Web from an Industry PerspectiveAbstract The penetration of Semantic Web technology in industry and in services

is progressing slowly but accelerating as new success stories are reported. In this paper and lecture we present ongoing work in the cross-fertiliza-tion between industry and academia. In particular, we present a collection of application fields and use cases from enterprises which are interested in the promises of Semantic Web technology. The use cases are detailed and focused on the key knowledge processing components that will unlock the deployment of the technology in the some selected application fields. The presentation also includes the current main milestones of the technology roadmap.

Provider France Telecom (for the REWERSE summer school)Language EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lmua-stheidmann-1165329017000h Categories Semantic Web Applications, Outreach to Industry

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 47

Type PDF (for industrial audience)Length 2:00h

Title BUSINESS RULES IN THE SEMANTIC WEBAbstract Introduction to the business rules standardization effort 'Semantics of

Business Vocabulary and Rules' and a comparison of the business rules approach with the Semantic Web approach to knowledge representation.

Provider LibrtLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lmua-stheidmann-11565152728089Categories Rule Languages, Integration of Rules and Ontologies, Knowledge Repre-

sentation and ReasoningType PDF (for industrial audience)

Title GALEN RevisitedAbstract The slides cover a lecture "Ontological and Practical Issues in using

a Description Logic to Represent Medical Concepts: Experience from GALEN". GALEN seeks to provide re-usable terminology resources for clinical systems. The heart of GALEN is the Common Reference Model (CRM) formulated in a specialised description logic. The CRM is based on a set of principles that have evolved over the period of the project and illustrate key issues to be addressed by any large medical ontology. The principles on which the CRM is based are discussed followed by a more detailed look at the actual mechanisms employed. Finally the structure is compared with other biomedical ontologies in use or proposed.

Provider University of Manchster (for the REWERSE summer school)Language EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lmua-stheidmann-1163408808058hCategories Semantic Web Applications, Bioinformatics, eHealthType PDF

Title SPARQLing QueriesAbstract The slides cover a lecture on "Querying the Web with SPARQL". The

lecture focusses on the SPARQL query language for RDF rather than on the SPARQL protocol for accessing Semantic Web data. The lecture cov-ers syntax and semantics of Core SPARQL (basic graph patterns) and the operations for algebraic manipulation of results.

Provider University of Manchster (for the REWERSE summer school)Language EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lmua-stheidmann-1163409019667hCategories Semantic Web Query and Update Languages, Query LanguagesType PDF

Title Ontologies and Text Mining as a Basis for a Semantic Web for the Life Sciences

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 48

Abstract Introduction into how rules, reasoning, ontologies and the web are used in bioinformatics. There are three parts: introdcution to bioinformatics incl. overview over some relevant tools and systems, alignment of ontologies, and finally bio-medical literature search with ontologies.

Provider University of Dresden, University of LinköpingLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lmua-stheidmann-1157021489793hCategories Bioinformatics, Ontologies for the Semantic WebType PDF

Title Integrating ontologies and rules: semantic and computational issuesAbstract We present some recent results on the definition of logic-based systems

integrating ontologies and rules. In particular, we take into account on-tologies expressed in Description Logics and rules expressed in Datalog (and its nonmonotonic extensions). We first introduce the main issues that arise in the integration of ontologies and rules. In particular, we focus on the following aspects: (i) from the semantic viewpoint, ontologies are based on open-world semantics, while rules are typically interpreted un-der closed-world semantics. This semantic discrepancy constitutes an im-portant obstacle for the definition of a meaningful combination of ontolo-gies and rules; (ii) from the reasoning viewpoint, the interaction between an ontology and a rule component is very hard to handle, and does not preserve decidability and computational properties: e.g., starting from an ontology in which reasoning is decidable and a rule base in which reason-ing is decidable, reasoning in the formal system obtained by integrating the two components may not be a decidable problem. Then, we briefly survey the main approaches for the integration of ontologies and rules, with special emphasis on how they deal with the above mentioned issues, and present in detail one of such approaches, i.e., DL+LOG. Finally, we illustrate the main open problems in this research area, pointing out what still prevents us from the development of both effective and expressive systems able to integrate ontologies and rules.

Provider LMU MunichLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lmua-stheidmann-1156405439112QCategories Ontologies for the Semantic Web, Semantic Web Rules + LogicsType PDF

Title Reasoning with Rules and OntologiesAbstract The slides cover a lecture on combining nonmonotonic rules and ontolo-

gies, fostering a systems combination paradigm. After posing the problem and a discussion of some general issues for combining rules as in logic programming and ontologies, a generic combination of rules under the Answer Set Semantics and ontologies in Description Logics is discussed, which are known as nonmonotonic description logic programs (dl-pro-

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 49

grams). Some applications of the formalisms in reasoning with defaults and incomplete information are presented. Finally, some related ap-proaches are briefly discussed.

The aim of this lecture is to raise awareness of the diffculties of the com-bination of rules and ontologies, and to present a practical approach which combines existing reasoners like DLV from the logic programming world and Racer from the Description Logic world. The lecture follows the more extensive tutorial paper "Reasoning with Rules and Ontologies" by Eiter et al., in Springer LNCS 4126, pp. 93--127.

Provider TU WienLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-lmua-stheidmann-

1156152623083Categories Ontologies for the Semantic WebType PDF

Title Rule Set and Ontology CompositionAbstract To master large rule sets in ontologies and other logic-based specifica-

tions, the ability to divide them into components plays an important role. While a naive approach treats the rule sets as black-box components and composes them via combinators, their relationships are usually so compli-cated that this approach fails to be useful in many scenarios. Instead, the components should be "opened" before composition. The paper presents several such gray-box composition techniques, namely fragment-based genericity and extension, inline template expansions, se-mantic macros, and mixin layers. All approaches help to structure large ontologies and rule-based specifications into fine-grained components, from which they can be built up flexibly.

Provider University of LinköpingLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lmua-stheidmann-1155892506133Categories Semantic Web Rules + LogicsType PDF

Title RDF Querying: Language Constructs and Evaluation MethodsAbstract This article is firstly an introduction into query languages for the Seman-

tic Web, secondly an in-depth comparison of the languages introduced. Only RDF query languages are considered because, as of the writing of this paper, query languages for other Semantic Web data modeling for-malisms, especially OWL, are still an open research issue, and only a very small number of, furthermore incomplete, proposals for querying Seman-tic Web data modeled after other formalisms than RDF exist. The limita-tion to a few RDF query languages is motivated both by the objective of an in-depth comparison of the languages addressed and by space limita-tions. During the three years before the writing of this article, more than three dozen proposals for RDF query languages have been published! Not only such a large number, but also the often immature nature of the pro-

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 50

posals makes the focus on few, but representative languages a necessary condition for a non-trivial comparison.

For this article, the following RDF query languages have been, admittedly subjectively, selected: Firstly, the ``relational'' or ``pattern-based'' query languages SPARQL, RQL, TRIPLE, and Xcerpt; secondly the reactive rule query language Algae; thirdly and last the ``navigational access'' query language Versa. Although subjective, this choice is arguably a good coverage of the diverse language paradigms considered for querying RDF data. It is the authors' hope and expectation, that this comparison will mo-tivate and trigger further similar studies, thus completing the present arti-cle and overcoming its limitation.

This learning resource consists of the slides for the aforementioned arti-cle, the actual article is published in a Springer tutorial volume (see con-tent for details).

Provider LMULanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lmua-timfurche-1154600021390Categories Query LanguagesType PDFLength 6:00h

Title Rule Modeling and Markup I + IIAbstract - What are rules good for?

- What is model-driven development ("MDA")? - Why should you model rules? - Can you use rules right away? - How to model rules with UML+OCL

Provider TU CottbusLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

lmua-stheidmann-1137658746573http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-lmua-stheidmann-1137659189770!

Categories Rule MarkupType PDF

Title Tool-Workshop Ontologie-Editoren: Werkzeuge für die Entwicklung und Pflege Semantischer Systeme

Abstract Als qualifizierendes Vertiefungsangebot zu Modul-06 (Semantic Models: Informationsintegration und Wissenmodellierung durch Ontologien) vermittelt Ihnen dieser Workshop Grundlagen des Ontology-Engineerings und den Gebrauch leistungsfähiger Editoren. Unter fachkundiger Anleitung führt Sie unser F&E-Experte Alois Reitbauer (profactor Steyr) durch ein Fallstudien-Projekt und zeigt, wie Strukturmodelle semantischer Netze durch formale Logik erweitert werden können.)

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 51

Provider Semantic Web school AustriaLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

sws-swschool-1162454669176Categories Knowledge Engineering / Ontology Engineering, Tools, Ontology Engi-

neeringType Educational Activity, classroom project

3.2.7 Courses contributed by external institutionsAs a result of the cooperation between KnowledgeWeb and AgentLinkIII, one course was also added from one AgentLinkIII partner.

Title OWL-S for AgentsAbstract This tutorial looks at the issues (and motivation) behind Semantic Web

Services from an agent perspective, and gives a brief overview of OWL-S.

Provider University of SouthamptonLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/srchDetailsLR?lrID= lr-univi-

caphreak-1118829328036Categories Semantic Web Services, Agents and the Semantic Web

Title Ontology EngineeringAbstract Some OWL examples and hints for constructing ontologies manually.Provider Vienna University of TechnologyLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

ivuo-lanzenberger-1163243407244Categories Ontology Representation / Ontology Languages / OWLType PDFLength 1:30

Title OWL - Web Ontology LanguageAbstract Web Ontology Language: OWL Lite & OWL DL BasicsProvider Vienna University of TechnologyLanguage EnglishURL http://rease.semanticweb.org/ubp/PUSH/search@srchDetailsLR?lrID=lr-

ivuo-lanzenberger-1163183005875Categories Ontology Representation / Ontology Languages / OWLType PDFLength 1:30

3.3 EvaluationIn total, there are 105 learning units available in REASE, from which KnowledgeWeb has contributed 81. 4 of these modules are in French, 6 modules in German, the re-maining ones are in English. As courses for industrial education were identified to be

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 52

highly important, we focused on publishing such material in the past month, as shown in the following figure:

Thus, the percentage of courses suited for industrial education has grown from less than 10% at the beginning of 2005 and 25% at the end of 2005 to 30% at the end of 2006. About 39% of those learning units that have been added by KnowledgeWeb people during the last 12 month, were tagged as suited for industrial education (12 out of 31).

Furthermore, the KnowledgeWeb learning material covers 39 categories from the 58 available in the REASE catalogue, an additional 10 categories are covered by the REWERSE units. This underlines that there is not much overlap between the REW-ERSE material and the KnowledgeWeb material and that they complement each other very well.9 of the categories in the REASE catalogue are currently not used. For some topics such as the ‘dynamics’ topic, the creation of resources is under way. Other empty top-ics actually occur because of minor errors in the design of the topic hierarchy. For ex-ample, there are two ‘security’ categories (one not being used), which we plan to merge in the next version of the topic hierarchy. Finally, some topics are rather ‘vi-sionary’ and while research has already started the topic is mature enough for teaching and, hence, learning material are not available (e.g., ‘Proof’).

4 Evaluation of REASEAs discussed in the previous deliverable D3.1.4, the development of REASE has reached a stage of maturity where it requires some formal evaluation of its quality and effectiveness. This evaluation needs to be carried out on various dimensions:

Does the existence of REASE fulfill a needed role? Is the quality of material in REASE of a suitable standard? Is there sufficient material in REASE to make it worthwhile? Are the mechanisms for finding information within REASE adequate? Are there mechanisms for providing material in REASE adequate? Is the usability of REASE acceptable?

We have therefore carried out a two-part evaluation: first, in the form of a question-naire sent to all users of REASE, and an analysis of their responses; and second, a task-based study completed by selected volunteers. The results are reported in the fol-lowing two sections.

4.1 User-based evaluation of REASEThe user-based evaluation of REASE addresses mainly the issues of usability, acces-sibility, and quality of material in the repository. It consists of a simple questionnaire sent to all registered members of REASE, that asks them about their general experi-ences of REASE, for example how easy they found it to use and about their general satisfaction. Out of 237 questionnaires sent out, we received a total of 68 responses, which is nearly 30%. Note that some questions have fewer than 68 responses as some people did not answer every question, similarly some questions have more than 68 re-sponses as some people gave more than one answer to questions 2 and 3 (this was an

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 53

error on our part as we did not intend this to be possible, but did not prevent this on the web form). In the case of providing more than one answer, it does not affect our statistical analysis greatly as they are not judgemental questions but simply questions about general background. The questionnaire itself can be found at the end of this de-liverable.

We have opted for a non-numerical scale because it is less confusing for users, and because we can still calculate the scores from the responses (e.g. we assign a score of 5 to "strongly agree" and a score of 1 to "strongly disagree"). Unlike the SUS ques-tionnaire, we do not alternate questions expecting positive and negative answers as this can easily lead to confusion on the part of the respondent and thus to them mark-ing the opposite of what they intend. For example if they have just answered the ques-tion "Do you find the website easy to use?" as "strongly agree", then they are tempted to reply "strongly agree" also to the next question "do you find the website unneces-sarily complex?", because the brain tends then to associate "strongly agree" with a very positive experience. In fact, where possible we try to avoid the use altogether of statements such as "strongly agree" and "strongly disagree", preferring instead to use statements such as "easy to use" and "hard to use" in order to minimise confusion. This is particularly important in the case of our target users who come from academic and industrial sectors from many different countries and many of whom will not have English as their first language (and in some cases, will only have a limited knowledge of English).

Table 1 shows a breakdown of the answers provided to each question, where a score of 5 is given to the highest answer (e.g. "very easy") and is always the first answer out of the selection provided, ranging down to a score of 1 for the lowest answer (e.g. "very difficult") and is always the last answer out of the selection provided. The table shows the counts for each answer, not the total scores, so for example 6 people an-swered "this is my first visit" to question 1, and 31 people answered "5 - 10 times" to the same question. Questions which are of a non-judgemental nature (such as the first 3 questions) are marked with an asterisk, i.e. these do not reflect a quality assessment of REASE and cannot be judged "good" or "bad". Numbers in bold represent the highest score for that question.

Score 5 4 3 2 1

Total

Q1* 6 31 18 13 0

68

Q2* 39 17 31 5 -

92

Q3* 11 15 32 10 3

71

Q4 24 26 14 2 0

66

Q5* 7 47 6 3 -

63

Q6 11 37 15 2 0

65

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 54

Q7 7 29 27 4 0

65

Q8 5 20 27 13 0

65

Q9 6 33 25 2 0

66

Q10 8 28 23 7 0

66

Q11 5 28 19 11 0

63

Q12* 37 8 17 - -

62

Q13 48 - 7 - 4

59

Q14 11 19 21 8 3

62

Q15 14 27 15 4 0

60

Q16 32 26 7 0 0

65

Q17 1 9 11 7 1

29

Q18* 8 5 6 6 4

29

Q19 1 8 15 4 0

28

Q20 12 12 5 0 0

29

Table 1. Answers to questionnaireTable 2 shows the counts for answers to each section, where as before, a score of 5 is the highest answer (e.g. "very easy") and is always the first answer out of the selection provided, ranging down to a score of 1 for the lowest answer (e.g. "very difficult" and is always the last answer out of the selection provided. It does not include the non-judgemental questions, since aggregating these makes no sense.

 

Table 2. An-swers by Section

In the following sections we analyse the answers in more detail.

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 55

Score 5 4 3 2 1 Total Usability 85 147 124 28 0 384 Informa-tion Find-

ing

53 55 57 19 3 187

Informa-tion Pro-viding

14 29 31 11 1 86

4.1.1 General backgroundWe were interested to see what type of people were using REASE, partly so that we could better analyse the results, and partly because we were just interested to see which kind of person REASE appeals to or may be useful for. So we asked some gen-eral questions about the users (questions 1-5 in the survey):

1. How many times (roughly) have you visited REASE up to now? 2. What is/was your primary purpose for visiting REASE? 3. How did you hear about REASE? 4. How likely are you to return to REASE in the future? 5. Which term best suits your primary job status?

In response to question 1, most users had visited REASE between 1 and 5 times (31 out of 68, or 45.6%), with 18 users having visited between 5 and 10 times, 13 users having visited between 10 and 50 times, and for 6 users it was their first time. There were no real "power-users" who had visited more than 50 times (not that we expected there to be, but it was worth asking just in case). As time goes on, we expect the num-ber of repeat visits to increase: it is encouraging at least that so few people had only visited once.

In response to question 2, there were many people who provided several answers. Clearly they did not understand the term "primary purpose" in the question correctly, and mentioned every purpose for visiting REASE rather than their main one. As dis-cussed earlier, we could have prevented this by making only one option available. As a result, we shall interpret the responses as if the question had been "why did you visit REASE?" The majority of people visited REASE to find specific material (39 out of 92 responses, i.e. 42.4%), while the next highest reason was for general browsing (31). 17 people visited REASE to upload material, while the remaining 5 people gave their reason as "other". the comments are more enlightening: reasons ranged from searching for specific material in preparation for a lecture, through general curiosity, to increasing their knowledge of the domain "Searching for the purpose of reviewing the domain(s) - so somewhere between general browse and specific search"

Question 3 asked users where they heard about REASE. The majority of people heard about it through Knowledge Web or REWERSE (32 out of 71, i.e. 45.1%), which is not that surprising since we would expect members of these networks to take a look at the repository and to register. 15 people came across it via a search engine, which is very encouraging, while 11 people heard about it from a colleague (also very encour-aging). Several students mentioned in comments that they had been recommended to visit the repository by their professor. 10 people heard about it via other means (for example one person heard about it at a conference), while 3 people could not remem-ber. When asked how likely they were to return to REASE again (question 4), 26 out of 66 people thought they would be quite likely (39.4%), while 24 people thought they would be very likely. Thus the combination of people quite or very likely to return was 50out of 66 people, or 75.8%, which is very encouraging. 14 people also thought they might possibly return, while only 2 people thought it was unlikely, and no one discounted the possibility completely. One user commented that "The problem is that

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 56

I have a large number of information resources that I use for my work and research. My memory for every single one of them is short: If I remember about the REASE then I definitely would return to it." Another user commented that "On Internet today many other sources of knowledge is available at a click and so REASE must clearly emerge as a (the) reference resources hub on SWS".

Finally in this section, we asked users about their primary job status. The vast major-ity described themselves as academics (47 out of 63, or 74.6%), while 7 described themselves as students, 6 as industrials, and 3 as "other" (but did not elucidate what this might be). While we expected a high proportion of academics, we expected a higher proportion of students, but it is possible that some PhD students described themselves primarily as academics. It is slightly disappointing that there were not more responses from industry, but perhaps this is more to do with the answering of the questionnaire than the actual number of industrials who visit REASE.

From the initial user registration on REASE, we also have some statistics about the origin of users (in terms of country and institution). Table 3 shows the numbers of registered users from various countries, sorted in descending order of frequency. We have highlighted in bold the European countries in the list: unsurprisingly, most users come from Europe, and there are many users from countries and institutions involved in Knowledge Web and REWERSE. Since the evaluation was carried out, we have an additional 20 users registered, mostly from the top 4 countries: Germany, Spain, UK and the US.

Country Number of Users % of Total Germany 32 12%

UK 27 11% Spain 18 8%

US 17 7% Portugal 15 6% France 14 6% Italy 9 4%

Austria 8 3% Brazil 8 3%

Greece 8 3% Ireland 7 3%

Netherlands 6 3% Romania 6 3% Sweden 6 3%

India 5 2% Unknown 5 2%

China 4 2% Argentina 3 1% Australia 3 1% Canada 3 1% Korea 3 1% Russia 3 1%

New Zealand 3 1% Belgium 2 <1%

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 57

Finland 2 <1% Malaysia 2 <1% Slovakia 2 <1%

Switzerland 2 <1% Yugoslavia 2 <1% Bulgaria 1 <1% Colombia 1 <1% Croatia 1 <1%

Czech Republic 1 <1% Guatemala 1 <1%

Luxembourg 1 <1% Malta 1 <1%

Mexico 1 <1% Pakistan 1 <1% Taiwan 1 <1%

Thailand 1 <1% Turkey 1 <1% Total 237

  Table 3. Registered Users by Country

Table 4 shows the institutions having 3 or more registered users. Most of these are members of KnowledgeWeb and/or REWERSE. Interestingly, the only non-European institution in this list is the Documentation Research and Training Centre at the Indian Statistical Institute, which has 3 users. There were also 13 individual users not affili-ated with an organisation.

Institution Number of Users Universidad Politecnica de Madrid 10

University of Sheffield 8 INRIA 7

L3S Research Center, Hannover 7 F. Ciencias Tecnologia, U. Nova Lisboa 6

Open University 6 Free University Berlin 5

University of Manchester 5 CERTH-ITI 3

"Al.I.Cuza" University, Iasi 4 AIFB, University of Karlsruhe 4

DERI, Galway 4 Linkoeping University 4

Indian Statistical Institute 3 University of Innsbruck 3

University of Trento 3   Table 4. Top Registered Users by Institution

4.1.2 Usability and Accessibility IssuesAs discussed in the previous deliverable D3.1.4, usability and accessibility issues are extremely important for websites, portals and software. Accessibility covers many dif-

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 58

ferent issues, for example: design and appearance (fonts, colours, spacing, layout), use of hypertext, navigation ease, mystery meat navigation, appropriate use of images, sound and colour, aesthetics, formats for interactive behaviour (form filling, check-boxes, radio buttons, etc.), alternative modes of use, keyboard shortcuts and other mouse alternatives, ability to modify the setup, speed of familiarity acquisition, com-pliance with existing standards and so on. In KnowledgeWeb deliverable D3.1.4 we performed an internal analysis of accessibility issues in the portal and solved the out-standing issues such as problems of fonts, colours, images, navigation etc. according to our own criteria. In this questionnaire we focused on asking the users for feedback on these issues. Because the questionnaire was designed to cover a wide range of top-ics (not just usability and accessibility), we focused the questions on some general us-ability issues, while allowing them to comment on any specific issues in more detail. For each of the following 5 issues, we asked the users to rate the repository in terms of very good, quite good, OK, quite poor, very poor.

6 Aesthetics 7 Layout 8 Ease of navigation 9 General usability (fonts, colours etc) 10 Speed

All these questions scored reasonable marks, with questions 6, 7, 9 and 10 all rated mainly as "quite good", and none rated at all as "poor". In fact, some of the highest overall scores in the questionnaire were for this section, which is encouraging. We also asked them in question 13 to rate the format of the material (in terms of "yes", "no" or "not sure"):

13 Was the format of the material suitable for your needs?

For question 6 (aesthetics), 37 out of 65 responses (57%) were "quite good", while re-sponses were "very good", 11 were "OK", and only 2 were "quite poor" (just over 3%). There were no real comments about this aspect (some comments were given in answer to this question but they actually related to other aspects). For question 7 (layout), 29 out of 65 responses (44.6%) were "quite good", while 27 responses were "OK", 7 were "very good" and again only 2 were "quite poor". The general consensus from the comments here was that the pages are too cluttered and that much of the information does not need to be there all the time. One comment was that: "I do not feel the homepage to be easily understandable for browsing and ob-taining information. I’d rather prefer a typical / standard website structure.". It is not clear, however, what exactly is meant by a "typical/standard website structure". An-other comment was that the "text on left column [is] too narrow and elongated.". We already tried to address these issues in the improvements made to the user interface at the beginning of November 2006.Question 8 (ease of navigation) was primarily rated only as "OK". 27 out of 65 re-sponses (41.5%) were "OK", while 20 responses were "quite good". 13 responses were "poor" (20%) and only 5 were "very good". However, this issue may have more to do with the search mechanisms and categorisation (as detailed in questions 11 and 14). When asking the question, we were more concerned about general navigation around the site than searching for material, but the problems with searching etc. may

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 59

have played a role in the responses given. For example there were some comments about the navigation tree and search mechanisms being hard to use, and that it was not obvious if you had actually found all the relevant information when searching. Question 9 (general usability) was also primarily rated as "quite good" (by 33 out of 66 users, i.e. exactly 50%. 25 users rated the speed "OK", while 6 rated it as "very good" and 2 rated it "poor". The only comment in this section was about popup block-ers (which many people have installed) preventing the user getting to the actual mate-rial unless they were disabled, which is annoying for the user. We can conclude that the general usability could probably be improved a little by the attention to some of the other aspects already mentioned, which we already tried to address in the latest user interface version. Without any more specific comments, it is difficult to know what problems people had and how we might improve things in this respect. Question 10 (speed) was also primarily rated as "quite good" (by 28 out of 66 users, i.e. 42.4%. 23 users rated the speed "OK", while 8 rated it as "very good" and 7 rated it "poor". According to the comments received, the slow speeds seemed to be associ-ated with the downloading of material, which (as one person said) is to be expected when file sizes are very large. Another person commented that the speed does vary a lot depending on time of day, etc. One suggestion was to try to speed up the step be-tween finding and previewing the material. We also asked the users directly (in question 13) if the format of the material was suit-able for their needs. The answer to this question was "yes" in 48 out of 59 cases (81.4%), with only 7 negative responses and 4 "unsure" responses. The comments made the responses a bit clearer. Several people only thought the format was suitable sometimes but did not specify what was wrong. One person commented that power-point slides on their own were not always that useful without the addition of notes and background information. Another comment was that some of the documents were in a format that made inclusion in other presentations difficult (though we would point out that this is not always a bad thing as it prevents people simply copying and pasting other people’s material and using it as their own). If we add together the scores for all the questions in this section, the most frequent re-sponse, with 147 out of 384 total answers, is "quite good" (score 4). This is depicted in Table 2 and in Figure 2.

Figure 2. Total scores for usability questions

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 60

4.1.3 Finding informationAn important aspect of REASE is how easy it is to actually find the information one is looking for, since the quality of the information, and in fact the existence itself of REASE is irrelevant if the information cannot be found easily. This issue can be di-vided into two parts: first, layout and navigation issues, and second, the design of the catalogue topic hierarchy. The first issue was largely discussed in D3.1.4, where we established (and rectified) a number of problems with the layout and navigation (such as clarity of the search boxes, design of linking mechanisms, etc.). The search mecha-nism itself is reasonably efficient. However, it was established that the original topic hierarchy was inadequate and to some extent misleading. We asked the following questions about finding information:

11. How easy was it to find what you were looking for? 12. Did you find other things of interest that you had not set out to find? 13 13. How easy was it to use the search mechanism?

Question 11 returned a majority of "quite easy" responses (28 out of 63, i.e. 44.4%). 19 people replied "OK", while 11 people found it "quite hard" to find what they were looking for, and only 5 people found it "very easy". Fortunately no one found it com-pletely impossible. One person commented that there are some resources only avail-able for registered users, but you cannot find this out unless you are already logged in. The suggestion was made that "it would be useful to have a pop-up window which tells you that the resource is in general available at REASE but only for registered users". This has been changed in the latest version of REASE where all resources are visible in the catalogue, and users are informed that they have to login when they ac-tually try to access the learning resource.

The search mechanism, on the other hand, was clearly more of a problem for many people, with the majority of people (21 out of 62), i.e. 33.9%, finding it only "OK", 19 people finding it "quite easy", 11 people finding it "very easy", and 4 people finding it "quite hard". There were several comments which made the problems clearer. For ex-ample, some people got errors when performing a search (though it is not clear whether this was through user error or a problem with the system); other people found the hierarchy very unintuitive and difficult to use. It was also suggested that a more semantic search was incorporated, such as making suggestions based on a user’s past history. We have already modified the catalogue in several ways, for example, adding an expand/collapse mechanism and avoiding that the currently selected topic is always shown on top of the catalogue.

While the responses to these questions are not negative overall, they still show that people cannot find things as easily as they might, and that the search mechanisms could definitely be improved. If we look at the overall scores for questions 11 and 13,we find that they are very similar (216 for question 11 and 213 for question 13), which is on the one hand surprising given that the number of people who replied "quite easy" to question 11 was significantly higher than the number who replied "quite easy" to question 14, but on the other hand unsurprising given that the two is-sues are strongly related.

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 61

If we add together the scores for questions 11, 12 and 14, the most frequent response, with 57 out of 187 total answers, is "OK" (score 4). This is depicted in Table 2 and in Figure 3.

Figure 3. Total scores for information finding questions

We also asked the users about "accidental information discovery" (question 12) as we were interested to know about the browsing potential of the site, where people find things of interest that they were not originally searching for. An example of this is Amazon’s "other people who bought X also bought Y" mechanism (and in fact this was explicitly mentioned in one comment). REASE has a similar mechanism "Users interested in this learning resource were also interested in the following learning re-sources:" which appears when a user looks at the page containing specific information about a certain resource. The answers to question 12 showed that a high proportion of users did indeed benefit from accidental discovery, with 37 out of 62 users answering "yes" to the question (59.7%), 8 users replying "no", and 17 users unsure (perhaps be-cause they were not sure if the material they found accidentally was useful, or because they were not sure if they had found the material accidentally).

4.1.4 Information QualityAs mentioned earlier, we wanted to know how users perceived the quality of the in-formation they found on REASE. Question 15 addresses this issue.

15. How would you rate the quality of the material you found?

The majority of answers to this question were "quite good" (27 out of 60, i.e. just un-der half the total answers), with 15 responses of "OK", 14 of "very good" and only 4 as "not as good as I would have liked". No one thought the quality was very poor. There were many comments about this topic, ranging from very complimentary to suggestions for improvement. It is clear, as one person said, that there is quite a wide range of quality in the resources, as there is only minimal quality control on the mate-rial uploaded. Some people liked the fact that there is material provided by PhD stu-dents as well as by more established academics, reflecting a wide coverage of topics. Another useful comment was that "perhaps more needs to be done to make it clear

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 62

that this is an educational database which has material NOT in other databases such as ACM Digital", which is indeed a very valid point.

4.1.5 Providing information to REASEAnother important aspect of REASE is specific to the provision of information. If the system is not conducive to users providing material with minimal effort and time, then they simply will not do so since there is little benefit to themselves apart from the wider dissemination of their work (and they will therefore find other methods of dis-semination). Much of the questionnaire relates to both uptakers and providers of mate-rial, but we devote a small part specifically to the issue of information provision. It is important to find a balance between ensuring that there is sufficient information about the material provided (so that the uptakers are suitably informed about what is avail-able and do not waste time looking at material which is irrelevant to them or not suit-able for their needs), and on the other hand ensuring that for the providers the process of uploading their material is as easy and streamlined an experience as possible.

16. How easy was the general process of uploading your material (from start to finish)?

17. When approximately was the last time you uploaded material? 18. How easy was it to use the classification system? 19. Would you recommend the site to other providers of material about Semantic Web topics?

The responses showed that while information providers would, on the whole, recom-mend the site to others, they found the process of uploading information quite tedious. 11 out of 29 users (40%) found the general process OK, with 9 users finding it quite easy, 7 finding it quite hard, 1 finding it very easy, and 1 finding it impossible. The main comments were that the process took too long and there was too much informa-tion to be filled in; several people reported that they had spent time filling in the infor-mation only for the system to fail and that they had to repeat the information filling process from the beginning. Clearly there is a difficult tradeoff here, with information providers wanting to provide less information about their resources, but information seekers wanting more information about resources to be available. Question 18 is another non-judgemental question which looks at information provider behaviour and provides some background information. The most frequent answer to "when did you last upload material?" was "less than 2 months ago" (8 users out of 29, i.e. 27.6%), with 6 for "more than 6 months ago", 6 for "more than a year ago", 5 for "more than 2 months ago" and 4 who couldn’t remember. This shows a fairly even split across the board, and we cannot deduce much from this answer since it is more likely that people responding to the questionnaire will have uploaded information more recently. Question 19 looks at the classification system from the point of view of an informa-tion provider. In contrast with an information seeker (who can get lucky by using the general search facility if he is not sure in which category he should look), an informa-tion provider needs to understand the classification system quite clearly in order to de-cide where to place his material. Most users found the classification system OK (15 out of 28 users, i.e. 53.6%), with 8 finding it quite easy, 4 finding it quite hard, 1 find-ing it very easy, and no one finding it impossible. The comments showed that people

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 63

found it harder to classify their material at first, when there was little material already in the repository that they could use as an example, but easier the more material was present. A few people did not find it very intuitive. We have adopted a more popular tree-like view of the topic hierarchy in the latest version of REASE to make the classi-fication process of uploaded material easier.The most encouraging thing about this section was that while they may not have been completely satisfied with the process of adding resources to the repository, most infor-mation providers would still recommend the site to other users (question 20), and in fact, some commented that they have already done so). 12 out of 20 users said they would be very likely to recommend the site to others (60%), the same number said they would be quite likely, while 5 said they would possibly recommend it, and no one said they would be unlikely to recommend it or would never recommend it, which is good news. If we combine the scores for questions 17, 19 and 20 (we ignore here question 18 since it is non-judgemental), the most frequent response, with 31 out of 86 total an-swers, is "OK" (score 4). This is depicted in Table 2 and in Figure 4.

Figure 4. Total scores for information providing questions

4.1.6 General commentsWe include here a selection of general comments about points not discussed else-where. Where possible, we respond to them.

"Repository was wonderful. But it would be more wonderful to have documents and tutorial which covers depth and breadth of the domain."

"The coverage needs to be improved (e.g. I was not able to find a good presentation anbout OWL-S)." This should happen as people continue to add material.

"It is not easy to judge the suitability for teaching university classes."

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 64

"In general, there are quite few presentations that really go into the technical details, most material is quite untechnical which makes it hard to find material for university courses." Presumably this is because many tutorials and presentations are quite broad in their coverage in order to appeal to a wider audience and to provide some overview of the area. Perhaps we should try to encourage addition of more technical material.

"...I felt that if authors can upload their related papers dealing with the tutorial on the specific topic, it will be helpful to understand the topic properly. Also if authors can upload (the audio recording of the lecture, if they have), it would be very helpful for the reader. Many a times, the presentations are not explicit/detailed enough to under-stand the topic." This is a nice idea, but in reality very few presenters have their lectures recorded. We have, however, included the material from the Knowledge Web Summer Schools for 2005 on REASE and are working to include the material from 2006.

"I am afraid I have very little feedback at this time, simply because I haven’t really used REASE yet. This will change over the next couple of month as I start preparing an MSc-level course on Ontologies in the Life Sciences."

"I’m not sure, but the REASE site includes so much active content that the contents of REASE may not be correctly indexed by search engines like Google. If that’s the case, it is a pity ;-)"

"So far I was satisfied with the functionalities of the repository."

"In nutshell I was proud to find this website."

"Are the resources semantically marked up? Search could showcase semantics too - e.g. I search for ’web services’ and get matches on every title with ’web’ and don’t get anything which mentions ’owl-s’ or ’wsmo’." This is something we are already considering, but have not implemented yet.

"Good idea but I hope that you have the resources to keep it up to date. Similar initia-tives failed to keep the initial momentum (cf. semwebcentral). What will happen after Rewerse und Knowledge Web are finished? It would be quite helpful if the EU would support in a NEW project an EXISTING platform." This should be covered by the establishment of the EASE foundation, which would maintain the repository as one of its missions.

"High quality contents, seminal works and top best contents etc. should be available or referred to promote REASE as THE (or one very best) Education portal on SWS." We could perhaps highlight some important material more prominently on the site, perhaps by addition of some specific highlights on the main page.

"Probably some form of exposing REASE content to the external world - not only via its own interface, but maybe some well-documented services, e.g. to search materials, to get rankings, etc. Then one can easily compile it into one’s own home page, e.g. for students’ reference? Another point would be to see more interaction on REASE, did

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 65

others find materials useful? What other materials they recommend to complement the stuff written in this particular one? Nothing fancy, Amazon-style ’people who buy X also buy Y’ would already be very useful. But as I said, nice work... and thanks for starting this collection of materials on the SW!" We already have the Amazon-style "people who buy X also buy Y" in place.

"Given that most resources are in English, it would be nice to have a default language or at least to have English very high up in the menu." This is being carried out, but is a non-trivial change to implement.

4.1.7 Discussion and ImprovementsWe have already implemented some changes based on the feedback from the ques-tionnaire, which were ready in time for the user study described in Section 4.2, and are planning other improvements.

According to the feedback from questions 7, 8 and 11 concerning layout, ease of navi-gation and difficulty of finding things, we have made the following improvements:

Fewer clicks to reach popular pages (reorganization of the menu) Login with email address as alternative to account name Separate filter for industrial resources in the catalogue Improved visualization of the catalogue (selected categories stays

within the catalogue; expand/collapse button to view the whole cata-logue; adaptation of catalogue to selected filters)

Better selection mechanism to classify resources while uploading

The problem about popup windows mentioned in question 9 is being carried out, so that popups will not occur any longer.

We plan to experiment with some query relaxation techniques for the topic hierarchy, in conjunction with work carried out in WP2.1 (Scalability), in response to comments about the search mechanisms. We are also investigating the possibility of including a full text index over the pdf files using e.g. Lucene.

Clearly the questionnaire has been very useful, not only in establishing the validity of the repository and highlighting much user satisfaction, but also in highlighting areas which need attention and suggesting new features and improvements to the existing repository, mechanisms, and materials contained within it.

4.2 Evaluation based on the User StudyIn contrast to the questionnaire sent to all REASE users, the general intention of the user study performed in November 2006 was to get feedback from those users who were currently working with REASE and who had never visited REASE before. In this way, we especially hoped to get feedback about functional deficiencies of REASE and actual problems with the platform (which the users having answered the question-naire may have long forgotten). We were aware that the participants of the user study were using REASE without any intrinsic objective (they were kindly ‘asked’ to partic-ipate) and tried to formulate the study such that no special prior knowledge about the

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 66

Semantic Web would be required. In summary, we expected them to issue much more criticism about REASE than the users having answered the questionnaire (because the latter may have forgotten the problems they had when they used REASE the first time), and also to offer more timely criticism since the questionnaire responders may also have reported on issues that had been fixed since their visit.

The user study comprised a series of four short and fairly simple tasks that involved using REASE and that should take no more than 45 minutes. The task had to be car-ried out by the users on their own with only one contact person to resolve any prob-lems. For the study, only two windows of a web browser were required, one for inter-acting with REASE, and one describing the task in the html page, which was also used to record the results of this study and to include general comments (see also the appendix).

4.2.1 MotivationREASE is a repository containing a range of materials related to teaching and learning about Semantic Web topics. Similar to other document management systems, REASE supports its users in gathering appropriate materials through a range of searching and browsing strategies. In this user study we were interested in the effectiveness and ef-ficiency of these data-gathering strategies in three different scenarios - these are de-scribed below in more detail.In addition to evaluating the data-gathering strategies, we also asked the users to look at how good and useful the data-gathering strategies are. In other words, we were also interested in the subjective opinion of the materials the users gathered in each scenario.Moreover, it was also crucial for us to measure whether REASE’s interface and de-sign were comparatively easy for the users who were new to REASE, in order to make an effective use of REASE.

4.2.2 General data and experiencesAfter having carried out a pilot study at OU with two people (which revealed a few small problems in the study itself and also small problems on REASE, that we cor-rected immediately), we conducted the user study at four different locations with four different partners: UPC (6 participants) OU (5 participants) L3S (12 participants) UniTn (12 participants)and received in total feedback from 35 participants (one additional partner had to cancel the study because of firewall problems).

We ran into several general problems: None of the participants had ever used REASE before and the time reserved in the

study to get used to it was found to be too short. This was especially true concern-ing the navigational characteristics of the REASE platform: REASE can become confused rather easily if users make extensive use of their browser’s back and for-ward buttons. For this purpose, only very few users could actually rate the content of the discovered resources (the last task of the user study).

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 67

The study at L3S suffered from the fact that all users were connected using one WirelessLAN access point, which made the server seem to be very slow (because of the download of many MB of material within a short time frame). Furthermore, the platform crashed in the middle of the study because of a too small java mem-ory heap parameter value (this problem never occurred before because never be-fore were15 users simultaneously accessing REASE). This error was fixed imme-diately afterwards, so that the study conducted at UniTn did not suffer from this problem.

The interface to provide feedback did not work with Internet Explorer 7.0 (for some reason, the results were not stored in the database after people pushed the ‘submit’ button). Hence, some participants could not submit their results (we had one more participant at OU and two more at L3S, for the others we could manu-ally submit their results to the server).

On average, the study participants spent 44 minutes (min. 13 min. / max 151 minutes ; this one user seemed to have true interest in REASE as he continued even after sub-mitting the results), as we told them to stop after 45 minutes even if they cannot com-plete the study. This was intended to limit the time spent by the volunteers on the study.

4.2.3 The ScenariosAfter spending 5 minutes to get familiar with REASE, the participants had to use REASE in four different straightforward scenarios. The first one was intended to use the keyword search for finding material about the topic ‘Problem solving methods’. The second was intended to use the REASE catalogue to find material about the topic ‘Human language technologies’. In the third scenario, users were asked to use the ‘ad-vanced search’ facilities of REASE in combination with the catalogue to find re-sources about ‘Description Logics’. In the fourth and final scenario, the participants should use their favorite search strategy to find material about ‘OWL’.

4.2.4 Scenario 1: Find material about ‘problem solving methods (PSMs)’

This scenario was intended to make people type in the term ‘problem solving meth-ods’ (or any other term they considered useful) into the keyword search box. We con-sidered the first two results of the query ‘problem solving methods’ to be highly rele-vant (as also indicated by the relevance score of higher than 34 from the underlying Lucene search engine, cf. Figure 5), because they contain the phrase ‘problem solving methods’ in the title (the remaining resources found were based on single word matches like ‘methods’ or ‘problem’).

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 68

Figure 5: Result of the query ‘problem solving methods’As a result, all but three users (91%) listed these two resources as the result (one user provided URLs outside REASE, probably using the browser’s Google search box in the upper right corner instead of the REASE search facility; one user provided only one result; one user retrieved the correct list of resources resulting from the query ‘problem solving methods’, but ignored our ‘best matches’ for whatever reason and presented a seemingly arbitrary list of results being less highly ranked by the Lucene search engine).

While the success rate in total was very high, there were many different strategies to get to the final results. Since we provided space for up to four result resources in the web-based form, the participants tried several alternative queries to better explore the resources available on the platform. They posed on average 2.4 unique queries about ‘problem solving / PSM’ with 82 queries in total, ranging from one query per user to 8 at maximum. 27% of the queries were about ‘problem solving method[s]’, where our two favourite resources were returned as the top two results. 22% were about “psm” / “psms” (the acronym of problem solving methods), leading to an empty result set on REASE. Another 9% of the queries were about “problem solving methods psm”, achieving also the two favorite results as the top 2 results. A further 7% of the queries used ‘problem solving methods’ in the advanced query dialogue (which they should have used in scenario 3 only). Furthermore, 6% of the queries used double quotes to find ‘problem solving method[s]’, which returned only one of the favourite resources. Another 5% were about ‘problem solving’. The remaining queries were combinations of the above, sometimes using advanced search already.

As a result, 10 users provided more than two resources to fill the provided four boxes (8 participants provided four results in total, 2 three results). 6 participants mentioned the third resource on the result list for the query ‘problem solving method’ as their an-swer, which was only found because of the term ‘method’ in the title. We assume that the other participants skipped this answer because of the low relevance as indicated by the low relevance value of “4” from the search engine (two users explicitly mentioned in the free text comments that they could not find more relevant results). A further 5

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 69

participants also provided the fourth learning resource in the result list, which is also rather irrelevant (score 3.1), probably just to fill the four available result fields.

4.2.5 Scenario 2: Find materials suited for an industrial audience, discussing the role of Human Language Technologies (HLT) in the Semantic Web tools and applications

In this scenario, the participants were asked to use the catalogue to find material about Human language technologies and tools/applications. We considered one resource to be very relevant here (being in both catalogue categories ‘human language technolo-gies’ and ‘Semantic Web applications’ and being suited for industrial education and being tagged as ‘tutorial’.). As a result, 33 of the 34 users (one had technical problems and dropped out from the study) found this resource and returned it as their first result.

However, a closer log file analysis revealed that, while users were able to filter for in-dustrial resources, only 4 users actually went directly to the two catalogue categories ‘human language technology’ / ‘Semantic Web applications’. Instead, 7 participants used the keyword search of REASE to finally find the material we were looking for, the remaining ones just searched the catalogue using the ‘search result list filter’ func-tionality, which worked well because the catalogue initially returns all resources as re-sults when no filters are applied. Actually, this way of searching REASE was never intended to be the major search option (but it seems that the users liked it when they looked for alternative ways of searching to fill the remaining text boxes).

As in the previous scenario, all users again provided further resources as results (again perhaps due to the fact that we provided them with room for three answers; two ex-plicitly mentioned in the free text comments that they consider only the first one to be highly relevant). Another 34 of the 89 overall answers were about HLT resources for industry, which were not tagged with ‘applications’ or ‘tools’, another 27 answers comprised two additional resources in the category ‘human language technologies’ which explicitly had the phrase ‘human language technologies’ in the title, but were not suited for industrial education). So the participants tried to relax on one of the three conditions ‘HLT’, ‘industrial education’ and ‘tools / applications’ in different ways to fill in the remaining fields in the online form.

4.2.6 Scenario 3: Find materials containing definitions of reason-ing techniques; in particular deduction, in the materials about Description Logic (DL) ."

The intention of this scenario was to let the participants use the advanced search facil-ities to check their usefulness. The participants posed 111 queries about ‘reasoning’, ‘deduction’, ‘description’, ‘logics’, and/or ‘DL’, of which 103 (93%) were using the advanced search facility. They did not have a preference for any of the terms, so the frequency distribution of the used query keywords was rather uniform. 82 (80%) of the advanced queries were using the catalogue category filter to find ap-propriate results. From the 267 category filters used (3.3 per query), 137 (51%) were for the catalogue entry ‘Knowledge Representation and Reasoning’ and its three subentries (“logics”, “logic programming”, and “reasoning”). A further 39 (15%) were for ‘Semantic Web Rules and Logics’ including its subcategory ‘reasoning lan-

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 70

guages’. Hence, most of the participants actually found those categories relevant for the given task.

From the 91 answers, 65 (71%) were modules from the available Description Logics resources, that score highest on the query ‘description logics’. Another 11 answers (12%) described the resulting top two resources of the query ‘reasoning’ / ‘reasoning deduction’, while 5 pointed to a resource that was returned only if the category ‘knowledge representation and reasoning’ was selected in the advanced search dia-logue.

In general, only about 25% of the advanced queries were using categories from the catalogue to filter results. This either means that REASE users can find what they are looking for even without the catalogue filtering (because there are not yet that many resources in REASE), or they don’t really know how to use the catalogue filtering mechanisms or cannot find it (as opposed to the participants of the user study who were explicitly asked to use it). We will think about improving the user interface for ‘advanced search’ in this respect.In summary, the results were not as uniform in this scenario because of the more com-plex task description that left room for several different queries, including the words ‘reasoning’ (techniques), ‘deduction’ ‘description logic’, and ‘DL’.

4.2.7 Scenario 4: Find material and the categories associated with the material in the catalogue containing description of species or layers in Web Ontology Language (OWL)."

This scenario was intended to find out what search mechanisms users prefer after hav-ing tried all of them in the previous scenarios. However, most people had problems with the formulation of the task and were looking for ‘species’ or ‘layers’ (for which there is nothing available in REASE): From 140 search requests (4 per user), 17% of the search requests were about ‘species’ and ‘owl species’, the latter delivering similar results than ‘owl’, but the relevance scores of the underlying Lucene search engine were rather low. 11% of the search requests were about ‘OWL’ (either using basic or advanced search), which returned all kinds of results as Lucene by default also returns partial matches, for example, on ‘knowledge management’ unless quotes are used to enclose the search keywords (which nobody did here). A further 6% of the queries were about ‘Web ontology language’. 86 queries (61%) were posed using the advanced search di-alogue, 47 (33%) of the advanced queries actually used the filter for catalogue classi-fication.

Regarding the returned results, we expected them to find the category ‘Ontology rep-resentation / Ontology Languages / OWL’ and select the most appropriate 3 resources from that category. From the 32 participants having completed this scenario, 18 found this category. Regarding the identified relevant resources, 4 actually returned the first one (the most recent one, which, however, did not have the term ‘OWL’ in the title and was considered not to be of high relevance).16 participants returned the second on that list (having ‘OWL’ in the title, the first relevant item), 11 the third one (also hav-ing ‘OWL’ in the title), and 19 the fourth one (being the third with ‘OWL’ in the title on that list). However, only 16 users in total clicked directly on the category ‘Ontol-

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 71

ogy representation / Ontology Languages / OWL’, where these resources are listed as mentioned above, and 10 of them got to the category by clicking on the category men-tioned in a summary page of another resource (hence, they were not browsing the cat-alogue as we expected).

As second result category, ‘Ontologies for the Semantic Web’ turned out to be most popular (provided by 7 participants), which is the parent category for the above ‘On-tology representation / Ontology Languages / OWL’. However, this category did not lead to new resources to be returned as selecting a top-level category always shows the resource in all child categories (hence, the resources in the category ‘Ontology representation / Ontology Languages / OWL’ were also listed when showing the cate-gory ‘Ontologies for the Semantic Web’). There are large variations in the remaining returned resources as people tried different strategies to fill in the remaining text boxes and 5 provided even answers being no category at all in the catalogue (so it was not clear to them what was meant with ‘category’). A further popular resource was the one matching on the query ‘layers’ (mentioned 9 times as result).

4.2.8 Assessing the content of REASEAs the final task, we asked the participants to look at the downloaded material from scenario 4. This turned out to be difficult as most users had problems with scenario 4 or ran out of time. Hence, only 18 users finally assessed the material and provided a quality indication. 13/51 users provided feedback about the top resource in ‘OWL’ (selected by in total 16 participants in the previous scenario). As a result, the rele-vance of this resource was judged on average as 3.4 (on a scale from 1-5, 5 being the best), the quality as 3.0 (three users could not access the resource). A further 10/51 users provided feedback about the second most popular resource in ‘OWL’. As a re-sult, the relevance of this resource was on average 4 (again on a scale 1-5), the quality 3.4. Overall, the relevance of the found resources was judged as 3.0 as well as the quality of the content. This was exactly the average of the evaluation scale; though we believe that our resource are of high quality, the problem here was that the task de-scription of scenario 4, for which the material should have been evaluated, was not clear enough to many users. Hence, there were too many uncontrolled side-effects which prevent us from making a statistically sound statement about the quality and the relevance.

4.2.9 Freetext commentsThe participants of the user study provided us with many valuable free text comments, which are summarized in this section.

Specific comments: The ‘back-button’ problem: Many participants complained about non-determinis-

tic query results, which obviously resulted from using the browser’s ‘back’ button. This confuses REASE and can lead to all kinds of problems. we will try to fix this, but it is a general problem of web application program-ming

“it's REALLY annoying that result lists use (sometimes) java script for linking in-stead of plain links- my usual tabbed browsing behavior doesn't work that way. I really hate it when systems do that”

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 72

This was introduced by intention as using several instances of REASE within a browser can also confuse REASE. Again we will try to fix this, but it may turn out to be impossible to do in the underlying web application framework

One user suggested to limit the search results and cut off the very irrelevant items. the problem is what is ‘irrelevant’ here. We think it’s better to provide irrele-vant results rather than showing an empty result set. An improvement might be to use semantic query relaxation but this require major changes in the platform.

Deselecting ‘educational activities’ and ‘educational events’ doesn’t work one of both has to be selected, but the user interface does not make this clear

The ‘search in result’ filter are not correctly reset after a new query (neither in the keywords search result page nor in the browse catalogue page) This can be fixed easily

‘OR’ is not available for querying this is supported by the underlying Lucene search engine, an explanation of the search parameters is missing and will be added

Searching for keywords should also return matching categories add a ‘search catalogue categories’ or provide a list of matched categories as search results

‘Predefined filters such as ‘learning resource type’, ‘target audience’, or educa-tional material type eased the search a lot.’

Search for sub-strings by default make search for acronyms very difficult (search for ‘DL’ or ‘OWL’. this should also be fixed by adding a link to the search syntax of Lucene (http://lucene.apache.org/java/docs/queryparsersyntax.html)

The list of language in te language selection dialogue in ‘advanced search’ is too large this will be restricted to the languages actually available in REASE

“Switching between the discipline selection window in ‘advanced search’ and the main window should be possible.” this has to be checked whether it can be done

“Difficult to find that ‘booking’ actually means ‘download’” change the name of the label

“No Register Button, login->apply is unintuitive” restructure the main page “when using search, it would be nice when the full filters would be displayed on

top of the result list (like in the category thingy)” should be added

“pop-ups are stupid” we are already trying to solve this

“its annoying that you can have only one browsing session (I like to use systems concurrently, using multiple pages at the same time)” this might turn out to be difficult to be changed in the underlying web applica-tion framework

“Browse catalogue was not useful for me. It was not user-friendly. Advanced search (when was working) was better.”

“I found the tool very useful, but better ranking strategies need to be deployed.“ In some cases the users used variance of the query such as Problem-Solving

Method, PSMs, PSM, among others. As a result in such cases the users either found a limited number of resources or in some other cases when the query was

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 73

quite different from the one that was mentioned in the task then no resources were retrieved. In order to handle different variance of queries there is a need to provide a more detailed analysis of the NLP techniques, and the tool can also make use of the external resources, such as stemming, synonym thesaurus / acronym matcher, or subscribe to external ontologies and knowledge-bases.

General final comments: “It's not a very friendly system. It's not easy to find what you want.” “We think it is a good way to centralized information and help people learn about

knowledge science.” “Its a nice way of collecting the material in a central repository...” “Idea is good. User needs quite some familiarity to use the system. Lots of op-

tions, but User Interface should be improved.” we will try to make another improvement of the user interface

“the search engine is good but it does not allow us to find learning materials with specific words.”

“For one file it was hard to retrieve the entire URL.; Very very distracting site; Too much information, too many ways to retrieve it. Just catalogue + search are enough; A nightmare to navigate it.” we will try to simplify the user interface

“Concerning the 3 search interfaces, I found them quite "well-designed" and the criteria for the advanced and the catalogue searches are quite relevant. However, I am not completely sure about the distinction of these two way of searching. More-over, if the design sounds correct, the tool in itself a rather difficult to use and I rapidly fall into a complete mess between the "go back", "cancel", "cross" and It was, at the end, simply impossible to use it without restarting the browser. Con-cerning the quality of the search results, my main concerns is that, when obtaining a results, it is not really clear "why" it have been selected (at least in the first screen): the place where the keywords are matched is not always explicit and some document does not match whereas it seems obvious that they should. It seems to be a good tool, but without these elements fixed, I would do better in doing Google.”

The provided results are relevant to the search. “Generally, the functionality of the tool are counter-intuitive. E.g. Searching and

browsing without filters should give the same results, but which is not the case.” this is because basic search searches in all metadata fields while searching the result list in the ‘browse catalogue’ page only search the corresponding metadata field (e.g., title). This has to be visualized somehow.

“Also, I later realised that the actual text of the resources seam not to be indexed, so the search is less effective.” we are already looking into this; however indexing Powerpoint / PDF can be-come very difficult and should remain an option (as the quality of metadata is typ-ically much higher).

4.2.10 SummaryWhile the user study has shown that users can find what we expected them to find, many users had problems to use the platform, which made an evaluation of the effi-ciency of the platform impossible. While their opinion about the relevance and quality

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 74

of the found learning material was rather neutral, the participants provided us with a long list of comments and suggestions to improve the platform. This makes the user study finally a success in the end, in spite of all the technical problems we encoun-tered. We are planning to address them to build an improved version of REASE dur-ing the remaining time of the project.

4.3 Summary of EvaluationThe evaluation based on the questionnaire basically revealed that those users who had an intrinsic motivation for using REASE were rather satisfied with the platform and the content, though they already found quite some improvements, which we already implemented in 2006. The evaluation based on the user study found many additional problems, partly because of the intense usage of REASE (many simultaneous partici-pants), partly because the participants immediately noted down their problems while the REASE users answering the questionnaire might have used it quite some time ago and might have forgotten about the problems they had in detail. We will use the feed-back from both evaluations to improve the platform in the future.

5 Quality Guidelines and ProceduresTo assure a high quality of the material stored in REASE, a review process is re-quired, especially since REASE is now moving more towards the public (we could as-sume a reasonable degree of quality for the material published from KnowledgeWeb / REWERSE partners up to now, but this will not necessarily be the case if people from outside both projects start uploading material). For this reason, we have set up a list of quality guidelines which are to be fulfilled before the learning material is finally ac-cepted to be published at REASE. This is necessary to ensure that REASE can achieve a high reputation in the area of ‘learning about Semantic Web’. The quality guidelines will evolve over time, so this section only describes the current state of the quality guidelines.The quality of each learning unit is related to two major areas: technical requirements and requirements regarding the content.

5.1 Technical RequirementsThe technical requirements define all issues which are not related to the content of a learning unit. Specifically, this comprises:

5.1.1 Non-Proprietary File FormatsTo ensure that learning units do not depend on specific applications to be able to use them, they should not be published in proprietary file formats. As an example, the very popular file formats for Microsoft Office applications are very difficult to read for users from other operating systems.Therefore, we require strictly that learning units must be provided at least in one non-proprietary format. However, we do want to keep the proprietary (source) formats ad-ditionally as many people work with them and reuse them for their own purposes (if the licence allows this).

Therefore, to support providing proprietary (editable) source files together with non-proprietary (read-only) versions, we have integrated an automatic conversion tool into

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 75

REASE (Linbox5). If the learning resource provider uploads their material in one of the Office file formats, they will be automatically converted into a PDF file and a se-lection between both is presented to REASE users, who want to access the learning resource.

5.1.2 Uploading Material vs. LinkingBasically, each learning material provider has the choice to either upload their mater-ial to the REASE server or to provide a URL to where the learning material is located.Providing a link basically has the potential advantage that updates are available in-stantaneously and automatically. However, it carries the risk that the material will not be available at all, for example, after a re-organization of the web server or if the pro-vider changes institution. Furthermore, it is not possible to automatically convert pro-prietary file formats (as mentioned in the section above). Therefore, we require that material is uploaded instead of providing a URL only, unless the material is itself in HTML.

5.1.3 MetadataTo implement a reasonable search service on REASE, it is essential that a sufficient number of metadata fields is specified for each resource. The main part of verifying this metadata is already done by the system. On the one hand, the REASE catalogue provides a classification into the most popular Semantic Web Topics, on the other hand the most important additional metadata fields apart from the classification are ‘mandatory’ in the sense that the system will not allow the user to complete the up-load of the material until the mandatory metadata fields are specified. However, if the metadata is to be described in free text, people might fill in wrong values such that a manual post-control of the metadata field is necessary.

5.1.4 File FormatsAs mentioned above, learning objects that are provided in an editable format (the source code) are highly valuable for persons who are teachers themselves. Such edit-able formats may also be valuable for EASE, for example, if they are only available to EASE members, generating a higher interest for EASE in this way. However, we do not force providers to upload their material in a source format as this might prevent too many people to use REASE at all to provide their learning units.

5.1.5 ModularizationThe utility of a learning resource also depends on its size. Oversized resources are dif-ficult to use for a potentially interested learner and they are difficult to classify ac-cording to the REASE catalogue. For example, if someone uploads a lecture on Se-mantic Web covering a 6-month-course at university, all topics can be associated with this course. To avoid this problem of too-common learning materials, we require that such material is to be broken into several subunits before it is published in REASE. As a rule of thumb, material that covers more than 12 hours is considered to be too long to constitute a single learning unit in REASE, but this has to be decided on a case-by-case basis within the quality management process as described below.

5 http://www.linbox.com/en/converter

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 76

5.1.6 QuestionnaireTo be able to get feedback from users of learning resources, REASE allows each pro-vider to attach a questionnaire to each learning unit. However, each provider has to in-dividually decide whether her material is associated with a questionnaire or not. We are currently not demanding that they do this as the questionnaire support of REASE is incomplete. However, we might change this in the future to get more feedback from REASE users.

5.2 Non-Technical RequirementsThe non-technical requirements are mainly related to the content of each learning re-source. We basically have to verify two issues:

o Relation to the Semantic Webo Quality of the actual content.

The first requirement is necessary to ensure that REASE keeps its focus on Semantic Web topics and the necessary basics to understand the Semantic Web. As an example, we are allowing material around the topics ‘XML’ (as RDF is often expressed in its XML variant), but a general tutorial about `HTML’ or `computer networks’ is out-of-focus.

5.3 Quality Management ProceduresQuality management in REASE is intended to ensure that all published learning units are in accordance with the above listed requirements. We can distinguish between automatically controlled requirements and those that have to be verified manually.

5.3.1 Controlling Requirements AutomaticallyThe fulfilment of the technical requirements is as often as possible ensured automatic-ally. For example, the most important metadata fields describing the learning units are mandatory such that REASE will not accept a new learning unit without these metadata fields being filled in. Furthermore, we implemented an automated conver-sion of the most popular proprietary formats (Microsoft Office) into the PDF format using the Linbox technology (http://www.linbox.com/converter).

5.3.2 Controlling Requirements ManuallyThis manual quality management process has to be effective and efficient. Therefore, REASE is required to support this process, which is already partly available. Each time an author publishes a new learning unit / updates an existing one, the adminis-trator of REASE has to approve the changes. In this manner, we can avoid the public-ation of low-quality material, which is not related to REASE at all. This is a sustain-able approach regarding the number of learning units and the expected low frequency of updates (which is different from other large-scale approaches, such as wikipedia, as REASE is only about a limited topic).

To ensure that the quality of the content of all Semantic-Web related learning units is high, we envision the following process:

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 77

1. The REASE administrator (currently Jörg Diederich, L3S) verifies the remain-ing technical requirements (those that cannot be validated automatically, or only with difficulty).

2. He also assesses the content of each learning unit to filter out the non-border-line cases. These include, on the one hand, learning units from Know-ledgeWeb partners or cooperating NoEs, which have a very high probability of being excellent and can thus be assumed to match the content requirements. On the other hand, the administrator can also easily filter out ‘spammers’, who try to use the platform for exchanging material completely unrelated to Se-mantic Web topics.

3. For borderline cases, we have installed an editorial board that will review the remaining units for their suitability to REASE in accordance with the quality guidelines. The current members of the editorial board are:

Holger Wache, VU (knowledge representation and reasoning: ontolo-gies, representation languages, reasoning techniques)

Diana Maynard, USFD (human language technology) York Sure, UKARL (ontology engineering, ontology management, se-

mantic web infrastructure) Lyndon Nixon, FUBerlin (materials for business professionals, multi-

media, Semantic Web services) Sylvain Dehors, INRIA (basic web information technology, ontologies

for the Semantic Web, Resource Description Framework (RDF) / RD-FSchema, e-learning)

Enrico Franconi, FUB (logics, Semantic Web languages) Martin Dzbor, OU (interoperability & integration, dynamics, tools, ar-

chitecture of information systems, personalization techniques, Se-mantic Web infrastructure/architecture, security/privacy/trust, informa-tion management)

John Breslin, NUIG (Semantic Web infrastructure, social networks in the Semantic Web)

Yiannis Kompatsiaris, CERTH (multimedia ontologies, semantic ana-lysis and reasoning of multimedia content, multimedia and Semantic Web)

Mustafa Yarrar, VUB (knowledge engineering / ontology engineering, knowledge representation and reasoning, ontologies for the Semantic Web, Semantic Web special topics)

Finally, some learning units are expected to be highlighted using some kind of ‘KnowledgeWeb certificate’, which can either be requested by other members of KnowledgeWeb (for example, if they have successfully used the learning unit for their own courses) or by other REASE users, who can express their opinion of the learning unit using the REASE feedback mechanism and rating scheme. This feedback mech-anism is currently, however, non-public and might be extended to become public.

Depending on the different communities represented in KnowledgeWeb (Description Logics, Ontology Engineering,…), we also envision recommendations for reading, which might be different depending on the community. These recommendations might be generated automatically / semi-automatically, depending on the advanced se-mantic platform for learning (ASPL), which will be developed in WP3.3.

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 78

6 Usage of Learning ResourcesIn this section, we report about the usage of REASE and the provided resources. The presented numbers are gathered from log files of the underlying web server and from the bookings and access information of the database, on which REASE is based.

6.1 General Usage of the REASE Web PagesThe usage of the REASE web pages since it went online in July 2004 is shown in the following figure (the statistics were taken on Dec-19 2006 from the web server log file excluding accesses from popular web robots and accesses from within the hosting domain of REASE):

The first public announcement of REASE was issued in October 2004, leading to an initial increase in the access statistics, because a first set of learning resources became available in November 2004. Whereas the number of accesses remained stable in the first half of 2005, it increased again in summer 2005, mainly because of the summer school activities of KnowledgeWeb and REWERSE. Especially, the teachers of the REWERSE summer school were required to upload their material before the summer school starts so that the students could access them from REASE directly. Finally, the usage of the REASE web pages increased again starting from October 2005. As an ex-ample, the REASE web pages were visited about 500 times from about 380 unique visitors in November 2005, downloading an approximate amount of 200 MB of data. Even though especially the increase in the number of non-unique visitors was partly caused by the evaluation activities in work package 3.3 (REASE is one service con-nected to ASPL-1, which was evaluated in November 2005 at USFD, OU, and Uni-versitatea "Al. I. Cuza" Romania as reported in D3.3.5), the main increase could not be associated with a single or few events. After the addition of several learning re-sources in the end of 2005, the general usage of REASE increased significantly at the beginning of 2006 with the usual decrease in the summer. The peak in November 2006 is again partly caused by the evaluation of the platform. More details are dis-cussed in the following sections about registered users and institutions and the actual access patterns of the learning material.

6.2 Registrations on REASETo access most of the material on REASE, users have to register first and specify (very few) information about their hosting institution (i.e., university or company and their country). The following figure depicts the number of registered users / institu-tions on REASE.

The first public announcement of REASE in October / November 2004 led to the reg-istration of users and institutions from KnowledgeWeb mainly. The second peak in June 2005 is mainly caused by the fact that REASE was used to distribute the learning material for the REWERSE summer school as mentioned above. The increase in No-vember 2005, however, is not dominated by KnowledgeWeb or REWERSE activities, only 2 from the 12 additionally registered institutions were actually directly related to one of these NoEs. In 2006, the increase in the number of registered institutions con-tinued with a slight increase in the rate of change. The number of registered users also

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 79

increased and doubled compared with the end of 2005. However, about 50 addition-ally registered users are due to the user study in November 2006.

6.3 Access to REASE ResourcesREASE resources were accessed as shown in the following figure:

The peaks in October and November 2004 were caused by a few users who accessed quite a large set of learning units, obviously playing around with the platform. This included people from KnowledgeWeb or REWERSE, but also one person from out-side both NoEs. The peak in July 2005 could be because of the KnowledgeWeb and REWERSE summer schools, which took place at that time. The peak in November 2005 is partly (about 40 from the 85 accesses) caused by the evaluation activity of WP3.3. However, 39 accesses were from users all over the world (Malaysia, Ger-many, USA, France, Brazil, Canada, and Greece), who were definitely not involved in KnowledgeWeb or REWERSE! Accesses continued on a much higher base level in 2006 with the usual ‘summer break’ in June-August. As in 2005, the peak in Novem-ber is partly a result of the user study.

6.4 Most Popular Resources on REASEBased on the access pattern by REASE users, the following learning units are the 10 most popular ones on REASE:

1. Semantic Web Lecture – Logics (L3S)2. Ontological Engineering (UPM)3. Semantic Web Tutorial (UKARL)4. Fundamental Research Challenges generated by the Semantic Web (VU)5. Semantic Web Lecture – Basic Building Blocks (L3S)6. OWL – Web Ontology Language (TU Wien)7. Semantic Web Lecture - Introduction and Overview (L3S)8. Information Extraction for the Semantic Web (TU Wien, REWERSE)

Web and Semantic Web Query Languages: A Survey (LMU, REWERSE)9. Semantic Web Use Cases (FUBerlin)

Most of the material has been available since November 2004, though there are some notable exceptions:

Both REWERSE courses (rank 8) were uploaded in June 2005 The Semantic Web Tutorial (rank 3) was uploaded in October 2005 The ‘Fundamental Research Challenges’ resource is a video of the summer

school presentation of Frank van Harmelen and was uploaded in November 2005

The ‘OWL – Web Ontology Language’ resource was viewed preferentially for one question in the user study (it was uploaded in November 2006).

While analyzing why the “Semantic Web Tutorial” become third most popular we noted the following:

It is the only English material on REASE providing an introduction to Seman-tic Web for people from industry

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 80

It is on rank 6 on Google for the search ‘Semantic Web Tutorial industry’ (probably because the KnowledgeWeb portal is on rank 5 for the Google query ‘Semantic Web industry’). [both ranks validated on 2005-12-19]

Of course, becoming the second most popular resource is only possible because the absolute number of bookings in REASE are still not very high (about 36 for the most popular resource).

7 Summary and Future WorkIn summary, the following main contributions were made regarding the activities re-lated to REASE:

A new more fine-granular classification system in the REASE catalogue, com-prising 58 categories.

81 learning units from KnowledgeWeb partners available on REASE (an addi-tional 31 compared to December 2004).

Increase of the percentage of KnowledgeWeb learning units for industrial edu-cation from 30% to 37%.

A detailed evaluation of REASE based on a questionnaire sent to all REASE users and a classroom-style user study with different user groups

A detailed description of the quality management process and a first set of quality guidelines to be enforced by the process.

An evaluation of the usage of REASE which shows a promising increase in usage during 2006 the past two month from users outside the KnowledgeWeb / REWERSE context.

The discussion about the REASE catalogue took place between KnowledgeWeb and REWERSE partners and finally merged into a global discussion of a Semantic Web Topic Hierarchy, which is described in more detail in the REWERSE deliverable E-D7. This topic hierarchy was also used as basis for the shared master curriculum (D3.2.4). Furthermore, the topic hierarchy was also included in the Semantic Web Re-search Community (SWRC) Ontology, (release 'swrc-swtopics'), see http://onto-ware.org/projects/swrc/), which itself is already in use for several different purposes such as project portals or Semantic Web applications such as bibster6. It was also used for organizing Semantic Web conferences like the ESWC (session organization).The topic hierarchy as well as the REASE catalogue will be subject of a constant evo-lution since the research area ‘Semantic Web’ is also subject of such evolution. Specifically, we will include feedback from other usage of the topic hierarchy (e.g., in the shared master curriculum) to improve the classification and we will use the result of the above evaluation to build the next version of the topic hierarchy.

Future work regarding REASE comprises the following issues: Publish more learning units, again focused on material for industrial educa-

tion, but also trying to fill those categories in the topic hierarchy, which are not covered yet by existing material

Continue to publicize REASE and recruit new users Continuous monitoring of the quality management process and application to

new resources Improve REASE based on the outcome of the evaluation.

6 http://bibster.semanticweb.org

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 81

References

[WJ99] Welty, Chris and Jessica Jenkins. 1999. An Ontology for Subject. J. Data and Knowledge Engineering. 31(2)155-181. September, 1999. Elsevier.

[GMZ06] F. Giunchiglia, M. Marchese and I. Zaihrayeu, "Encoding Classifications as Lightweight Ontologies", accepted for publication in Journal of Data Semantic, vol-ume VIII, Springer, Berlin, Germany, 2006.

[DBT06] J. Diederich, W.-T. Balke, and U. Thaden, „The Semantic GrowBag Demonstrator for Automatically Organizing Topic Facets”, SIGIR06 Workshop on Faceted Search, Seattle, USA, 2006.

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 82

AppendixIn this appendix, we provide the questionnaire sent to all REASE users in October 2006 and the form used for the user-study in November 2006.

Questionnaire

Evaluating REASE, the Repository of Semantic Web Learning Units

REASE, the Repository of Semantic Web Learning Units, has been set up in the con-text of the European Research Projects Knowledge Web and REWERSE to provide a single platform with high-quality learning materials around the topic 'Semantic Web' (More benefits of REASE).

We kindly ask you as a user of REASE to fill in the questionnaire below. Your feed-back is very helpful for us to improve the quality of REASE.

This is an anonymous submission.

General Questions

How many times (roughly) have you visited REASE up to now?

This is my first visit   1-5 times   5-10 times   10-50 times   more than 50 times  

Remarks: What is/was your primary purpose for visiting REASE?

Searching for specific material   Uploading material   General Browsing  

Other  

Remarks: How did you hear about REASE?

Recommendation of colleagues   Search Engine   From the KnowledgeWeb /

REWERSE mailing list   Other   Can't remember  

Remarks: How likely are you to return to REASE in the future?

Very likely   Quite likely   Possibly   Unlikely   Never  

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 83

Remarks: Which term suits best your primary job status?

Student   Academic   Industrial   Other   

Remarks: Please rate the following aspects of the site: General look and feel (aes-thetics):

Very good   Quite good   Ok   Quite poor   Very poor  

Remarks:

Layout:Very good   Quite good   Ok   Quite poor   Very poor  

Remarks:

Ease of navigation:

Very easy   Quite easy   Ok   Quite hard   Very hard  

Remarks: General usability (fonts, co-lours etc.):

Very good   Quite good   Ok   Quite poor   Very poor  

Remarks:

Speed:Very fast   Quite fast   Ok   Quite slow   Very slow  

Remarks:

Specific Questions Related to Finding / Downloading Material

If you have used REASE for finding and downloading material:

How easy was it to find what you were looking for?

Very easy   Quite easy   Ok   Quite hard   Very hard / impossi-ble  

Remarks: Did you find other things of interest that you had not set out to find?

Yes   No   Not sure  

Remarks:

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 84

Was the format of the material suitable for your needs?

Yes   No   Not sure  

Remarks: How easy was it to use the search mechanism?

Very easy   Quite easy   Ok   Quite hard   Very hard  

Remarks: How would you rate the quality of the material you found?

Very good   Quite good   Ok   Not as good as I would have liked  

Very poor  

Remarks: Would you recommend the site to other people looking for material about Se-mantic Web topics?

Very likely   Quite likely   Possibly   Unlikely   Never  

Remarks:

Specific Questions Related to Uploading Material

If you have used REASE for uploading material to REASE:

How easy was the general process of uploading your material (from start to fin-ish)?

Very easy   Quite easy   Ok   Quite hard   Very hard / impossible (had to give up)  

Remarks: When approximately was the last time you uploaded material?

Less than 2 months ago   More than 2 months ago   More than 6 months

ago   More than 1 year ago   Can't remember  

Remarks: How easy was it to use the classification system?

Very easy   Quite easy   Ok   Quite hard   Very hard / impossible (had to give up)  

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 85

Remarks: Would you recommend the site to other providers of material about Semantic Web topics?

Very likely   Quite likely   Possibly   Unlikely   Never  

Remarks:

General comments

Feel free to write any general comments (e.g. missing features you would like to see):

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 86

User Study

Evaluating REASE, the Repository of Semantic Web Learning Units

REASE, the Repository of Semantic Web Learning Units, has been set up in the con-text of the European Research Projects Knowledge Web and REWERSE to provide a single platform with high-quality learning materials around the topic 'Semantic Web' (More benefits of REASE).

The REASE evaluation task

Thank you for participating in this user study. The study comprises a series of short and fairly simple tasks that involve using the Repository of Semantic Web Educa-tional Materials (REASE). You will be carrying the tasks on your own, but feel free to ask the study facilitator if there are any issues.

During the study you only need two windows of a web browser, one for interacting with REASE, and one to read this page and fill in the results of this study. When you feel your work on each scenario is concluded, please record the materials you gath-ered electronically in this document, together with a brief explanation of why they are included and/or why you gave them a particular ranking score.

Motivation

REASE is a repository containing a range of materials related to teaching and learning about Semantic Web topics. Similar to other document management systems, REASE supports its users in gathering appropriate materials through a range of searching and browsing strategies. In this user study we are interested in the effectiveness and effi-ciency of these data-gathering strategies in several different scenarios - these are de-scribed below in more detail.

In addition to evaluating the data-gathering strategies, we would be grateful if you also spent some time on the second part of the user study, which looks at how good and useful the data-gathering strategies are. In other words, we are also interested in your subjective opinion of the materials you gathered in each scenario.

Duration

The entire user study comprises three parts and it should not exceed 45 minutes of your time. In addition to this time, you will be asked to fill in a short questionnaire, which may take up to 15 minutes at the end of the user study.

Part I. Familiarization

Expected duration: up to 5 minutes for this part

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 87

This first part is optional; if you are familiar with REASE, you can skip it. However, please, make sure that the REASE demands, e.g. on cookies and popups (popup-blockers must be disabled), are satisfied.

Caveat: Do not use Internet Explorer 7 for filling in this form as submitting the re-sults does not work! Also, do not use the browser's back/forward navigation buttons as this can confuse REASE. To go back there is always a 'cancel/go back' button or you can alternatively press the 'x' button in the upper right corner below the search box.

The purpose of this part is to introduce REASE to you and show its functional fea-tures relevant to carrying out the tasks in this user study. We will show you the regis-tration and login to the REASE repository, which is necessary in order to access the content of all learning materials without any restrictions. Also we point you to several search and browse functionalities you need later.

Please copy the email address with which you registered in REASE into the following

text-box:

(This is used only to be able to sort out multiple submissions and distinguish between the different groups doing this study. According to the REASE privacy policies, this email address will not be used for anything else.)

Part II. Material gathering

Expected duration: around 15 minutes for this part

The purpose of this part is to use four different strategies to access material in REASE in order to retrieve a set of materials that might be relevant to the scenarios given be-low. Please note that in each scenario you will need to "translate" the request into a search query or another action applicable to REASE. A concept that is relevant to each scenario is always underlined.

Each scenario has some time estimates, if feel you cannot complete the scenario within the recommended time limit - e.g. you have only two materials but we asked for four, move on to the next task, and note that you ran out of time. You are not go-ing to be disqualified or penalized.

Scenario 1

[Please spend max. 2 minutes on this scenario]

Please use the REASE search interface to retrieve a list of learning materials where you may expect to find some information on the concept given in the sentence below. Please copy the URLs of two (2) to four (4) materials from the retrieved list into the following table:

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 88

"Find materials about Problem Solving Methods (PSMs)"

URLs of found materials

You can add more comments in the following text-box:

Scenario 2

[Please, spend max. 3 minutes on this scenario]

Imagine you are preparing a seminar for an audience comprising professionals from an industrial organization, to whom you want to explain various principles of the Se-mantic Web. Please use the `Browse Catalogue' functionality and then apply appro-priate filtering criteria in REASE to retrieve a list of learning materials where you may expect to find some information suitable for the industrial audience and related to the concept(s) given in the sentence below.

From the retrieved list, copy the URLs of three (3) materials (preferably tutorials) into the following table (note that you may need to consider variants of concepts in the question):

"Find materials discussing the role of Human Language Technologies (HLT) in the Semantic Web tools and applications"

URLs of found materials

You can add more comments in the following text-box:

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 89

Scenario 3

[Please spend max. 5 minutes on this scenario]

Imagine you intend to write a review of a particular Semantic Web topic. For that pur-pose, please use the `Advanced Search' functionality of REASE, choose the appropri-ate discipline(s) in the discipline classification, and find a list of learning materials considering different aspects or views on a topic given in the sentence below.

Please retrieve the list mentioned above and copy the URLs of two (2) to four (4) ma-terials from this list into the following table.

"Find materials containing definitions of reasoning techniques; in particular de-duction, in the materials about Description Logic (DL) ."

URLs of found materials

You can add more comments in the following text-box:

Scenario 4

[Please spend max. 8 minutes on this scenario]

Please use any combination of REASE Search functionality and its Catalogue func-tionality to find categories (topics) of learning materials where you may find different views on the concept mentioned in the sentence below.

"Find material and the categories associated with the material in the catalogue containing description of species or layers in Web Ontology Language (OWL)."

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 90

Find two (2) categories that contain materials about the given concept and note them down in the following text-boxes. Then, retrieve the actual learning ma-terials.

Category 1:

Category 2:

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 91

Please list the URLs of three (3) materials for each category you selected in the previous step in the following table.

URLs of found materials (Category 1):

URLs of found materials (Category 2):

Finally, download three materials of your choice from those you listed in the previous table to your computer. You will use these materials later in Part III of this user study.

You can add more comments in the following text-box:

Part III. Assessment of material relevance and quality

[Expected duration: around 15 minutes for this part]

The purpose of this part is to reflect on the results of Scenario 4, where we asked you to download a few learning materials on your computer. Please open the downloaded documents (e.g. in Adobe Acrobat Reader or PowerPoint Viewer), and scan briefly their content.

We do not ask you to read the documents in depth; only locate the part (e.g. a page or paragraph) that you feel covers the "definition of Web Ontology Language (OWL) species" by using search functions available in Acrobat Reader or PowerPoint Viewer.

Once you located the part you feel is relevant, express your subjective opinion about the relevance to the search query of that section or paragraph by selecting the radio button that best describes your opinion on the "Relevance" scale provided below. Please, do not spend too much time on reading and assessing the materials. If you can-

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 92

not find the paragraph relevant to the scenario within (say) 5 minutes, simply select the radio button "cannot process".

Please also make a quick subjective judgement about the technical quality of the material. Your "first impression" is entirely sufficient for this purpose. Please, select the radio button that best describes your opinion on the "Quality" scale provided below.

Material 1: URL:

Relevance:

Cannot process   Absolutely irrelevant   Rather irrelevant   Moder-

ately relevant   Rather relevant   Very relevant  

Remarks:

Quality:

Cannot tell   Very low quality   Rather low quality   Reasonable  

Rather high quality   Very high quality  

Remarks: Material 2: URL:

Relevance:

Cannot process   Absolutely irrelevant   Rather irrelevant   Moder-

ately relevant   Rather relevant   Very relevant  

Remarks:

Quality:

Cannot tell   Very low quality   Rather low quality   Reasonable  

Rather high quality   Very high quality  

Remarks: Material 3: URL:

Relevance:

Cannot process   Absolutely irrelevant   Rather irrelevant   Moder-

ately relevant   Rather relevant   Very relevant  

Remarks: Quality: Cannot tell   Very low quality   Rather low quality   Reasonable  

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 93

Rather high quality   Very high quality  

Remarks:

General comments

Feel free to write any general comments:

KWEB/2007/D3.1.5v2/v2.0 25/01/2007 94