Institutional Repositories Workshop DISpace 1.0 Benoit PAUWELS Université Libre de Bruxelles (ULB) Brussels

Institutional Repositories Institutional Repositories WorkshopWorkshop

DISpace 1.0

Benoit PAUWELSUniversité Libre de Bruxelles (ULB)

Brussels

DISpace 1.0DISpace 1.0

1. Why do we need a Nereus XML Schema?

2. Why is DSpace not (sufficiently) interoperable with EO?

3. How can I make my DSpace system EO interoperable within one month?

4. Can we support « complex object » formats in DISpace?

5. Is DISpace interoperable with RePEc, OAISTER, NEEO, …?

EO general infrastructureEO general infrastructure

DSpace

EO HarvesterARNO

Fedora

Eprints

SRU

iPort

APA references

End User

Data Providers Service Provider

OAI

Q.Service = Q.Service = f(Q.Metadata)f(Q.Metadata)

AU: Capron, Henri^ Cincera, Michele

TI: Industry-university S&T transfer: Belgian evidence on CIS data

BC: Brussels economic review, 46(3)

PD: 2003

Service Provider

Authors: Capron, Henri; Cincera, Michele

Title: Industry-university S&T transfer: Belgian evidence on CIS data

BibCit: Brussels economic review, 46(3)

Pub date: 2003

Data Provider

Capron, Henri, & Cincera, Michele (2003). Industry-university S&T transfer: Belgian evidence on CIS data. Brussels economic review, 46(3).

APA references -- QuaLevel 6

Quality of service -- level 6

Prof. Dr. Michele Cincera



Department of Applied Economics (DULBEA)



Author: Capron, HenriAuthor: Cincera, Michele, 124468


BibCit: Brussels economic review, 46(3)

Pub date: 2003

Data Provider

AU: Capron, HenriAU: Cincera, Michele, 124468


BC: Brussels economic review, 46(3)

PD: 2003

Service Provider

Prof. Dr. Michele Cincera






JTitle: Brussels economic reviewVolume: 46Issue: 3

Pub date: 2003

Data Provider

AU: Capron, HenriAU: Cincera, Michele, 124468


JT: Brussels economic reviewVO: 46IS: 3

PD: 2003

Service Provider

Department of Applied Economics (DULBEA)


DSpace

EO HarvesterARNO

Fedora

Eprints

SRU

iPort

APA references

End User

EO

DP, exchange, SP formatsDP, exchange, SP formats

RePEc

National Repository of Academic Output

DSpace

DSpaceDSpace

DSpace

EO HarvesterARNO

Fedora

Eprints

SRU

iPort

APA references

End User

EO


RepEC


Who will do the mapping?

DSpace

EO HarvesterARNO

Fedora

Eprints

SRU

iPort

APA references

End User

EO


RepEC



DSpace EO Harvester

• choose appropriate internal metadata-format

– high quality service …– of several service providers

• map to exchange format …– of several SP

Data Provider Service Provider

DP internal metadata-format

DP mapping

Exchange metadata-format

SP mapping

SP internal metadata-format

• publish exchange metadata-format• choose appropriate internal metadata-format

– quality of metadata should be retained • map from exchange format• build service(s) based on internal metadata-

format

EO exchange metadata-EO exchange metadata-formatformat

• Let’s say we go for the international standard QDC

<record>

</record>




Pub date: 2003

QDC + at minimum one title and the (co-)authors


• QDC + at minimum one title and the (co-)authors

<record>

<dc:contributor> Capron, Henri </dc:contributor> <dc:contributor> Cincera, Michele </dc:contributor>

<dc:title> Industry-university S&T transfer: Belgian evidence on CIS data </dc:title>

</record>




Pub date: 2003

• EO : we want to produce publication lists per author+ unique identifier per author


• QDC + at minimum one title and the (co-)authors + unique identifier per author

• DP can deliver « author unique id » in various formats:– (124468)– |124468– [ uniqueid: 124468 ]

Impose format: rewrite QDC XML Schema

<record>

<dc:contributor> Capron, Henri </dc:contributor> <dc:contributor> Cincera, Michele (124468) </dc:contributor>


</record>




Pub date: 2003


• Nereus QDC XML Schema

• EO: we want nice APA style structured bibliographic citations impose format: “Guidelines for Encoding Bibliographic

Citation Information in Dublin Core Metadata” (MIMAS) OpenURL 1.0 ContextObject

<record>

<dc:contributor> Capron, Henri </dc:contributor> <nereus:author id=“124468”> Cincera, Michele </nereus:author>


</record>




Pub date: 2003


• Nereus QDC XML Schema + OpenURL 1.0 ContextObject<record>



<dcterms:bibliographicCitation> info:ofi/fmt:kev:mtx:ctx… &rft_val_fmt:journal &rft.btitle=Brussels economic review &rft.volume=46 &rft.issue=3 </dcterms:bibliographicCitation>

</record>




Pub date: 2003

• BUT: certain bib. cit. metadata doesn’t map to OpenURL ContextObject– Book chapter: authors/editors of

book– Part of research report: sponsor,

authors/editors of report, volume– …


• In general: imposing QDC + OpenURL ContextObject without additional guidelines (profile) results in: – unstructured and incomplete metadata at the SP– low(er)-quality services

• EO: { Nereus QDC + OpenURL ContextObject + Nereus profile } is not good enough– certain metadata fields for certain document types don’t

find their place– EO service could be more ‘user-friendly’ with the

introduction of complex object metadata structures NEEO

• Need for a common standard for – bibliographic metadata

• QDC + EPrints Application Profile? • MODS?

– complex object metadata structure• METS? MPEG21/DIDL?

• Which one to choose?– some variant of QDC– some flavour of MARC (MARC21, UNIMARC,…)– MODS– …

• Can be whatever but– should be able to describe all types of academic output– high enough granularity and well-defined semantics so that

mapping to the different exchange formats is possible without loss of quality

– define metadata structure, irrespective of services !– object file related metadata !– follow standards !

• Unfortunately– a lot of IR softwares do not support granular, semantically well-

defined metadata formats

DP metadata-formatDP metadata-format

• DSpace uses QDC as internal metadata format; this is not good enough for high(er)-quality services (like EO)

• Cincera, Michele, 1960, 1965

• Brussels economic review 46, 122

DISpace DP formatDISpace DP format

<dc:contributor> Capron, Henri</dc:contributor>

<dc:contributor> Cincera, Michele, 124468<dc:contributor>

<dc:title> Industry-university S&T transfer: Belgian evidence on CIS data</dc:title>

<dcterms:bibliographicCitation> Brussels economic review 46(3)</dcterms:bibliographicCitation>

<dcterms:issued> 2003</dcterms:issued>

Some other examples:

• Bibliographic citation for book chapter <dcterms:bibliographicCitation>

The national innovation system of Belgium. Capron,H. ; Meeusen, Wim. Berlin Springer-Verlag 2000, 73-100. 790813087

</dcterms:bibliographicCitation>

• Bibliographic citation for book chapter <dcterms:bibliographicCitation>

Croissance et convergence économique des régionsthéorie, faits et déterminants. Beine, M. ; Docquier,

F.Bruxelles De Boeck Université 2000, 345-384. 2804133435


• Bibliographic citation for a conference contribution <dcterms:bibliographicCitation>

Proceedings of the international seminar on exchange oftechnology and know-how, 13-15 October 1999. Prague

1999 </dcterms:bibliographicCitation>


• add granularity: split up DSpace QDC fields into subfields

• define semantics of fields and subfields for each document type:– based on ISO-690 / Z44-005 – irrespective of aimed-at services– see document

« DISpace @ ULB: Input template fields, QDC fields and

subfields -- Version 1.0 – March 2006 » _


http://www.bib.ulb.ac.be/RDIB/DISpace/DISpaceQDCfields.doc

• Supported document types (15):– book– bookitem– article– proceedings– conference lecture– unpublished communication– unpublished theses and dissertations– unpublished research report– part of an unpublished research report– working paper– patent– interview – emission– web site– bibliography– course – sound – video – image – database – software -

others


• Example – bookitem:<type>|atype-level1|btype-level2|ctype-level3</type>

<title>|amaintitle|bsubtitle</title>

<contributor>|aname|=DAI</contributor>

<dcterms:issued>|adate|tpubstatus</dcterms:issued>

pubstatus = { ULB4PUB | ULB2BPUB | ULBPUB | ULBNPUB }

<dcterms:bibliographicCitation>

|amaintitle|bsubtitle|hauthors|eedition|upublisher

|cplace|vvolume|ppages|wcollection|icollectionnumber|sisbn




<dc:contributor> Capron, Henri</dc:contributor>

<dc:contributor> Cincera, Michele, 124468<dc:contributor>

<dc:title> Industry-university S&T transfer: Belgian evidence on CIS data</dc:title>

<dcterms:bibliographicCitation> Brussels economic review 46(3)</dcterms:bibliographicCitation>

<dcterms:issued> 2003</dcterms:issued>

• With subfields + definition of semantics


<dc:contributor>|aCapron, Henri</dc:contributor>

<dc:contributor>|aCincera, Michele|=124468<dc:contributor>

<dc:title>|aIndustry-university S&T transfer|bBelgian evidence on CIS data</dc:title>

<dcterms:bibliographicCitation>|aBrussels economic review|v46|i3</dcterms:bibliographicCitation>

<dcterms:issued>|a2003|tULBPUB</dcterms:issued>

<dcterms:bibliographicCitation>|aBrussels economic review|v46|p122</dcterms:bibliographicCitation>

<dcterms:bibliographicCitation> |aCroissance et convergence économique des régions|bthéorie, faits et déterminants|hBeine, M|hDocquier, F.|cBruxelles|uDe Boeck Université|p345-384|s2804133435


• generic solution:– all DISpace fields are subfielded – a subfield is denoted through the ‘|’ character followed by 1

character – every DISpace field has at least an ‘a’ subfield; this subfield

doesn’t have to be explicitly entered in the DISpace field:• American journal of sociology|v12|i3|p123-345|d2004|s1234-5678

• |aAmerican journal of sociology|v12|i3|p123-345|d2004|s1234-5678

– every DISpace field can have a different list of valid subfields

– all subfields can be repeated within a DISpace field

DISpace internal record format can be extended:– new QDC fields can be defined (through new qualifiers)– additional subfields can be defined


• We have:– DISpace « QDC+subfields » format– Nereus XML Schema + Nereus profile

• Next step:– Map (crosswalk) between the two metadata

formats

DISpace Nereus crosswalkDISpace Nereus crosswalk

DISpace Nereus crosswalkDISpace Nereus crosswalk<dc:contributor>

|aCapron,Henri</dc:contributor>

<dc:contributor>

|aCincera,Michele|=124468<dc:contributor>

<dc:title>|aIndustry-university S&T transfer|bBelgian evidence on CIS data</dc:title>

<dcterms:bibliographicCitation>|aBrussels economic review

|v46|i3</dcterms:bibliographicCitation>

<dcterms:issued>|a2003|tULBPUB</dcterms:issued>

<record>



<dcterms:bibliographicCitation> info:ofi/fmt:kev:mtx:ctx… &rft_val_fmt:journal &rft.btitle=Brussels economic review

&rft.volume=46 &rft.issue=3 </dcterms:bibliographicCitation>

</record>

• oaicat.propertiesCrosswalks.nereus_qdc=org.dspace.app.oai.NereusQdcCrosswalk

• NereusQdcCrosswalk.javadispace_dc = item.getDC();dispace_citation = dispace_dc.getCitation();eo_bibcit.append(‘&rft.jtitle’,dispace_citation.getSubfield(‘a’));eo_bibcit.append(‘&rft.volume’,dispace_citation.getSubfield(‘v’))

;eo_bibcit.append(‘&rft.issue’,dispace_citation.getSubfield(‘i’));eo_bibcit.append(‘&rft.pages’,dispace_citation.getSubfield(‘p’));

DISpace Nereus crosswalkDISpace Nereus crosswalk

DISpace EO OAI setDISpace EO OAI set• DSpace collections

– all items in DSpace reside in at least one collection– items can reside in more than one collection– every DSpace collection == OAI set

• Problem: – EO items reside in different collections– all of these collections can contain non-EO items

• Solution: – create ‘virtual collection’ for EO items– copy appropriate items into EO collection– appropriate items?

• all items written by authors that participate in the EO project

• <dc:contributor>|aCincera,Michele|=124468<dc:contributor>

• obtain item through Lucene search on DAI of author

DISpace EO OAI setDISpace EO OAI set

Collection 1

Collection 2

Collection 3 EO Collection1

2

3

45

DISpace

1 2

3 4

5

EO Harvester

6

7

8

OAI

EO non-EO

DISpace EO OAI setDISpace EO OAI set• script: map-items• Java program: ItemMapperManager.java• configuration file: itemmapper.xml

<virtual-collection> <collection-handle>2013/2269</collection-handle>

<collection-description>Economists Online</collection-description>

<lucene-query>((author:124468))</lucene-query> <lucene-query>((author:341421))</lucene-query> <lucene-query>((author:562814))</lucene-query> <lucene-query>((author:410846))</lucene-query> <lucene-query>((author:649475))</lucene-query> <lucene-query>((author:1077459))</lucene-query></virtual-collection>

DISpace bulk ingestDISpace bulk ingest• bulk upload and update of bibliographic metadata

and object files in DISpace– researchers/departments already have their CV/publist in

an MS-Excel or MS-Access database– bulk manipulation of bibliographic metadata

• load/update offline collection into a DISpace collection

• offline collection:– unique collection name item: unique ‘offline’ item id– TAB delimited file (export from Excel or Access)

• should respect specific structure– object files

• file naming convention == f(offline item id)

directory on DISpace server

DISpace bulk ingestDISpace bulk ingest

Example of offline collection ‘mc’ with 3 items; 2 items with FT

mc.txtmc-0001 Firms’ productivity growth and R&D

spillovers|ban analysis of alternative technological proximity measures Cincera, Michele|=649475Article|bArticle dans une revue|cAvec comité de lecture

Economics of innovation and new technology|d2004|v14|i7|s10438599

en ULB2BPUB sciences économiques et de gestion

mc-0002 …mc-0003 …

/mc/ mc.txt mc-0001.pdf mc-0002.pdf

DISpace bulk ingestDISpace bulk ingest• Perl script: dspace_upload.pl• configuration file: config.xml

<collections> <collection> <name>mc</name>

<id>2013/645</id> </collection></collections>

• Run script: perl dspace_upload.pl –n mc adds/updates items

mc-0001, mc-0002, mc-0003 in DISpace collection with handle

2013/645

1. Find an IT person with Java experience2. Define (sub-)communities + collections3. Define document types and data dictionary

– datadict.xml– doctypes.xml

4. Install DISpace Java software– Load and manipulate datadict and doctypes:

• LoadDSpaceConfig• DataDict.java• DocTypes.java

– Manipulate subfields:• DCValue.java

– Lucene indexing:• Item.java

– WebUI rendering:• ItemTag.java• ItemListTag.java

5. [ Bulk ingest – Offlinecollections ]6. Itemmapper7. Nereus OAI crosswalk

All DISpace source code and documentation is available on http://www.bib.ulb.ac.be/RDIB/DISpace/technical.html _

EO DISpace starters kitEO DISpace starters kit

http://www.bib.ulb.ac.be/RDIB/DISpace/technical.html

DISpace bitstream featuresDISpace bitstream features EO exchange metadata format = Nereus QDC

DISpace bitstream featuresDISpace bitstream features EO exchange metadata format = Nereus QDC

DISpace bitstream featuresDISpace bitstream features

Problems (for the SP):– <dc:identifier> : ambiguous way of pointing to

object file(s)– Service at the mercy of each data provider’s

capacity to come up with a comprehensive jump-off page

– Redundancy of screens (same metadata)– Non-localized links in data provider’s jump-off

page

EO exchange metadata format = Nereus QDC

http://bib17.ulb.ac.be:8080/dspace/handle/2013/2801




http://nereus.uvt.nl/iPort?previous=7&sessionid=311206707570&request=full_record&position=2&domain=Dome




DISpace bitstream featuresDISpace bitstream features EO exchange metadata format = MPEG21/DIDL


• MPEG21/DIDL– Digital Item Description Language– language that permits to describe

complex objects• bibliographical metadata• object files metadata:

– Location– Format/Size– DRM– Description of content


<record> <nereus:author id=“ulb.ac.be.649475“> Cincera, Michele </nereus:author> <dc:title> Brain Drain and MNEs </dc:title> <dc:type xsi:type=NEREUSType> Part of book – chapter </dc:type> <dc:identifier> http://bib17.ulb.ac.be:8080/dspace/handle/2013/2801 </dc:identifier></record>

EO exchange metadata format = Nereus QDC


<record><container> ## for bibliographic metadata

<nereus:author id=“ulb.ac.be.649475“> Cincera, Michele </nereus:author> … <dc:identifier> http://bib17.ulb.ac.be:8080/dspace/handle/2013/2801 </dc:identifier>

</container><container> ## for object file metadata

<objfile><location>http://bib17.ulb.ac.be:8080/dspace/bitstream/2013/2801/1/

text.pdf</location><description>Full document</description><format>application/pdf</format>

</objfile>

</container> </record>

EO exchange metadata format = MPEG21/DIDL

DISpace bitstream featuresDISpace bitstream features• Solution in DISpace:

– We need a possibility to define multiple characteristics of a bitstream (object file)

– DSpace comes with one « description » metadata field per bitstream

– DISpace: • add granularity to this field through the introduction of

subfields + semantics • per document type• currently: 3 features

– subfield a -- general description of content of bitstream– subfield s -- accessibility: { ULBINTERNET | ULBINTRANET |

ULBINVISIBLE }– subfield v -- versioning: { ULBPREPRINT | ULBPOSTPRINT |

ULBPUBPRINT }• example: |aChapter1|sULBINTRANET|vULBPUBPRINT• XML config file: bsfeatures.xml


DISpace submission DISpace submission interfacesinterfaces

• Need to reflect granularity and semantics of metadata fields and subfields in the DISpace submission interfaces:– Ease of submission process for the researcher self-archiver– Guarantee quality of metadata

Journal title


• Submission interface is completely configurable through XML configuration file « doctemplates.xml »:– submission template per document type (#15)– mandatory | optional– repeatable | non-repeatable– instructions / information texts– mapping with fields / subfields in database– define « validators / processors » per input

element– types of input elements:

• text / textarea / select / constant / helper • size of text(area) boxes / list of valid values / value of

constant


<dspace-doctemplate> <internal_doctype>article</internal_doctype> <input_areas> <input_area name="title">

<status>M</status> <help_txt>Enter the main title of the item.</help_txt> <repeatable> <yn>N</yn> … </repeatable> <map element="title" qualifier=""/> <inputelems> <inputelem>

<id>maintitle</id> <status>M</status> <map subfield="a" /> <input_type>text</input_type> <hint_text><i>Main title</i></hint_text> <text_size>90</text_size> <text_max_size>90</text_max_size>

</inputelem> <inputelem>

<id>subtitle</id> <status>O</status> <map subfield="b" /> <input_type>text</input_type>

…</inputelem>

…


<input_area name="publicationstatus"> <map element="date" qualifier="issued"/> <inputelems>

<inputelem> <id>pubstatus</id>

<map subfield="t" /> <input_type>select</input_type> <select_vlist>-1,ULB4PUB,ULB2BPUB,ULBPUB</select_vlist> <select_dlist>,Submitted for publication,To be published,

Published </select_dlist>

</inputelem>. . .


• « helper » input elements

<inputelem> <id>fullname</id> <status>M</status> <input_type>helper</input_type> <helper>

<name>inputAuthors</name> <datas> <data><id>last</id>

<map subfield="a" /> </data> <data><id>uniqueid</id> <map subfield="=" /> </data>

</datas> …


« helper » input elements

• dedicated JSP • permits to generate vocabulary-controlled, well-formed metadata

in the submission interface• experimental: AJAX technology (DWR Java open source library)• currently:

– author/DAI lookup• based on list of potential ULB authors: name + local unique

identifier (DAI)• non-ULB co-authors are added to this list by submitter

– journal/ISSN lookup• based on official ISSN database• plans:

– extend with Sherpa/Romeo information– use Z39.50 access

• planned:– date helper– ontology helper


• DWR allows code in a browser to use Java functions running on a web server just as if it was in the browser

• Java is fundamentally synchronous where Ajax is asynchronous. So when you call a remote method, you provide DWR with a callback function to be called when the data has been returned from the network.


• submission workflow:– choose document type and collection– metadata template (with appropriate

helpers) for document type can be filled out

– (optionally) bitstreams are attached to the item

– last verification– confirm deposit

DISpace - other featuresDISpace - other features• DISpace collection structure

• main, special, special purpose• DISpace = institutional repository + institutional

bibliography• submitter can edit/delete his items

• SRU/SRW access• seamless integration into (library and other) portals

• Authentication• integration with university’s authentication mechanism

• To come: Web-based export toolkit• permits researcher to create publication lists « à la

carte »• reference format, filters on document types and

metadata fields, sort, output format (XML, CSV, HTML,…)

DISpace 1.0• on the researcher’s desktop 1/2007

DISpace 1.0DISpace 1.0

1. Why do we need a Nereus XML Schema?

international standards and profiles do not fulfill

2. Why is DSpace not (sufficiently) interoperable with EO?

the internal record structure of DSpace is not granular enough

3. How can I make my DSpace system EO interoperable within one month?

use the DISpace EO starters kit

4. Can we support « complex object » formats in DISpace?

yes; bitstream features

5. Is DISpace interoperable with RePEc, OAISTER, NEEO, …?

very probable; generic and extensible solution

Documents

Institutional Repositories Workshop DISpace 1.0 Benoit PAUWELS Université Libre de Bruxelles (ULB) Brussels