Digital Curation for the Big Data Sciences 大数据科研中的数字保存张智雄中国科学院国家科学图书馆

Digital Curation for the Big Data Sciences

大数据科研中的数字保存

张智雄中国科学院国家科学图书馆

提纲 Digital Curation 的兴起 Digital Curation 是什么？ Digital Curation 和 Preservation 不

同？大数据科研带来的 Digital Curation 挑

战、问题及应对措施结语




1 、 Digital Curation 的兴起 Data Deluge

1 、 Digital Curation 的兴起 From Data Deluge to D

ata Curation Philip Lord, Alison Mac

donald, Liz Lyon, David Giaretta

The Digital Archiving Consultancy Limited and the Digital Curation Centre

1 、 Digital Curation 的兴起 The Digital Curation Centre 成立

在 e-Science Core 项目的支持下， DCC 于 2004 年 3 月 1 日成立

总部位于 Edinburgh 的 National e-Science Centre

University of Edinburgh (lead,Informatics, Law, Information Services and research institutes)

University of Glasgow (HATII and Information Services)

UKOLN, University of Bath Council for the Central Laboratory of the Resea

rch Councils (CCLRC)

1 、 Digital Curation 的兴起会议期刊

International Digital Curation Conference ， Bath ， Sep. 29 - 30, 2005

8th International Digital Curation Conference, Amsterdam, 14 - 17 January 2013

DigCCurr 2007 、 DigCCurr 2009 、 DigCCurr 2013 An International Symposium on Digital Curation （ April 18-2

0, 2007 ） Digital Curation Practice, Promise and Prospects （ April 1-3,

2009 ） Chapel Hill, North Carolina, United States Public Symposium, 2010-2013

International Journal of Digital Curation 2006 开始 http://www.ijdc.net/ Vol 8, No 1 (2013)

1 、 Digital Curation 的兴起以 Curation 命名的机构

The Greek Digital Curation Unit (DCU) at the Athena Research Centre(2007)

UC3 ， University of California Curation Center (2010)

The Digital Research and Curation Center at The Johns Hopkins University’s Sheridan Libraries

The University of Toronto’s iSchool established The Digital Curation Institute （ 2010 ）

Purdue University Library’s Distributed Data Curation Center (D2C2) （ 2009 ）

......

1 、 Digital Curation 的兴起与 Curation 相关的教育培训

DigCCurr I (2006-09) ， DigCCurr II (2008-13) School of Information and Library Science (SILS) University of North

Carolina at Chapel Hill,NARA Preserving Access to Our Digital Future: Building an International Di

gital Curation Curriculum. Extending an International Digital Curation Curriculum to Doctoral S

tudents and Practitioners International Data curation Education Action (IDEA) Working Gr

oup Developing an International Curation and Preservation Training and

Education Roadmap Education for Digital Stewardship: Librarians, Archivists or Curators

?" Masters Programme in Digital Curation, Luleå University of Tech

nology IFLA, 2011“ Education for Digital Curation” Board on Research Data and Information

Symposium on Digital Curation in the Era of Big Data:Career Opportunities and Educational Requirements

1 、 Digital Curation 的兴起相关技术工具

Data Asset Framework (DAF) enumerating and auditing data holdings

DRAMBORA self-assessment of possible risk

TRAC Trustworthy Repositories Audit & Certification, Criteria a

nd Checklist Digital Preservation Suite

preservation plans DROID

identifies file formats ......



战、问题及应用措施结论

2 、 Digital Curation 是什么先说一下数字保存（ Digital

Preservation ）数字是一把的双刃剑

优点方便易用、可复制、易传输、大量

携带 ... 问题

脆弱性删除、盗取、修改、失真 ....

依赖性技术、系统、标准、软件、上下

文（元数据）、组织、经济 ... 飞速退化性（ obsolescence）

媒体、硬件、软件、格式 ...

2 、 Digital Curation 是什么 Digital Preservation

1996 年 5 月 1 日 , 成为重要关注内容 Preserving Digital Information: Report of t

he Task Force on Archiving of Digital Information

Commission on Preservation and Access Research Libraries Group. Inc. (RLG)

目标 : “continued access indefinitely into the future o

f records stored in digital electronic form.”

http://www.clir.org/pubs/reports/pub63/reports/pub63watersgarrett.pdf

2 、 Digital Curation 是什么 21 世纪初数字保存 (DP) 已经成

为数字图书馆的一个重要领域主要研究内容

保存策略和方法、保存元数据、存储体系、保存仓储、保存工作流、 Web 存档、保存信息模型

主要标准规范：开放档案信息系统（ OAIS2002) 、

主要数字保存系统和服务体系 e-Depot DIAS, NDIIPP, LOCKSS, Portico, CDL DPR ，

FCLA DAITSS......

2 、 Digital Curation 是什么为什么还会出现 Digital Curation?

已经数字保存已经有两个接受的术语了数字保存（ Digital Preservation ）数字存档（ Digital Archiving ）

为什么还要提出 Digital Curation ？ Digital Curation 是什么？与 Digital Preservation 有什么不同的思路和方法？

2 、 Digital Curation 是什么 Digital Curation ：被创造的新词 Digital Data Curation Task Force

Report of the Task Force Strategy Discussion Day

Tuesday, 26th,November 2002,Centre Point, London WC1 ， January 2003

e-Science Curation Report Data curation for e-Science in the UK: a

n audit to establish requirements for future curation and provision ， 2003

JCSR(the Joint Information Systems Committee’s Committee for the Support of Research ， JISC 研究支持委员会 )

2 、 Digital Curation 是什么 Digital Data Curation Task Force

由 Tony Hey ，当时 JCSR 的主席召集目标：明确和构建英国原始研究数据的 Curation 战略会议日期 2002 年 11 月 26 日 The application of the term “curation” is new,

and in several ways the meeting found itself grappling with questions of scope, with frequent overlap with questions relating to digital preservation.

It did not reach a definition of the term.

2 、 Digital Curation 是什么 Digital Data Curation Task Force

What is curation? Dr John Taylor, Director General of the Research Councils Tony Hey, distinguish the actions involved in caring for digita

l data beyond its original use, from digital preservation. Seamus Ross, “curation in the museum sense” covers three

core concepts: conservation, preservation and access Alison Allden “curation” implied in an active management

of information, involving planning. re-use of data is a core issue. If data is to be re-used, then it needs special treatment

Rolf Apweiler, curation is when people add value to data Jeremy Frey, curation is research work in itself - managing, i

mproving, enhancing data

2 、 Digital Curation 是什么 e-Science Curation Report

“curation” 来源于 “ curator” somebody who keeps something for the publi

c good, whose value often needs to be brought out by the curator.

两个重要特点 more support for explicit policies with regard

to data sharing digital curator is store-keeper ， but he shoul

d take an active role in promoting and adding value to his holdings


此前 “curation” is commonly used to refer to th

e work done on genomic and proteomic databases, annotating and managing annotations

现在 It covers a wider context than just archiving; it embraces the care of the record within scientific context and environment


Working definitions Curation: The activity of, managing and promoting the use o

f data from its point of creation, to ensure it is fit for contemporary purpose, and available for discovery and re-use. For dynamic datasets this may mean continuous enrichment or updating to keep it fit for purpose. Higher levels of curation will also involve maintaining links with annotation and with other published materials

Archiving: A curation activity which ensures that data is properly selected, stored, can be accessed and that its logical and physical integrity is maintained over time, including security and authenticity

Preservation: An activity within archiving in which specific items of data are maintained over time so that they can still be accessed and understood through changes in technology


That the objective of digital curation of primary research data is

to keep data which is valuable, potentially valuable or which is required to be kept;

and in such a way that it is accessible and usable by others (while observing relevant restrictions), that its value is maintained and, where possible, enhanced;

and that this activity and service should be provided at affordable and justifiable cost.

2 、 Digital Curation 是什么 JISC通讯定义

JISC circular 6/03 (Revised), July 2003 The term ‘digital curation’ is increas

ingly being used for the actions needed to maintain and utilise digital data and research results over their entire life-cycle for current and future generations of users.

2 、 Digital Curation 是什么 DDC定义 1

DCC Approach to Digital Curation, 15 Aug 2004

curation : general term - taking care of things

data curation : looking after and adding value to data

digital curation : looking after and somehow "adding value" to digital data. This probably implies creating some new data from the existing, in order to make the latter more useful and "fit for purpose".


DCC Charter and Statement of Principles

What is digital curation? Digital curation is maintaining and adding

value to a trusted body of digital research data for current and future use; it encompasses the active management of data throughout the research lifecycle.

http://www.dcc.ac.uk/about-us/dcc-charter/dcc-charter-and-statement-principles


Digital curation involves maintaining, preserving and adding value to digital research data throughout its lifecycle.

The active management of research data reduces threats to their long-term research value and mitigates the risk of digital obsolescence. Meanwhile, curated data in trusted digital repositories may be shared among the wider UK research community.

As well as reducing duplication of effort in research data creation, curation enhances the long-term value of existing data by making it available for further high quality research

http://www.dcc.ac.uk/digital-curation/what-digital-curation


DCC Briefing Papers Digital curation is the management and preservation of digit

al data over the long-term. All activities involved in managing data from planning its cre

ation, best practice in digitisation and documentation, and ensuring its availability and suitability for discovery and re-use in the future are part of digital curation.

Digital curation can also include managing vast data sets for daily use, for example ensuring that they can be searched and continue to be readable.

Digital curation is therefore applicable to a large range of professional situations from the beginning of the information life-cycle to the end; digitisers, metadata creators, funders, policy-makers, and repository managers to name a few examples http://www.dcc.ac.uk/resources/briefing-papers/introduction-curation



战、问题及应用措施结论

3 、 Curation 和 Preservation不同？

JISC Preservation 和 Curation 对比 JISC Digital Preservation briefing paper

Digital preservation actions and interventions ensure continued and reliable acc

ess to authentic digital objects for as long as they are deemed to be of value.

Digital curation maintaining and adding value to a trusted body of digital inf

ormation for future and current use; active management and appraisal of data over the entire life

cycle. builds upon the underlying concepts of digital preservation emphasising opportunities for added value and knowledge t

hrough annotation and continuing resource management. http://sitecore.jisc.ac.uk/publications/briefingpapers/2006/pub_digipreservationbp.aspx

3 、 Curation 和 Preservation不同？ ARL 的两者对比

New Roles for New Times: Digital Curation for Preservation, March 2011

Digital curation refers to the actions people take to maintain and add value to digital information over its lifecycle, including the processes used when creating digital content.

Digital preservation focuses on the “series of managed activities necessary to ensure continued access to digital materials for as long as necessary.”

intersection of these actions, digital curation facilitate the preservation.


Digital Curation: The Emergence of a New Discipline 中的对比 digital preservation efforts originally focussed on ensuring that mate

rial survived technical obsolescence and organisational mismanagement. Preservation implied a passive state, where material would be mothballed in an inaccessible “dark archive”, with only a few authorised users, to ensure that it retained its integrity and authenticity

ensuring that digital material is managed throughout its lifecycle so that it remains accessible to those who need to use it. Metadata is used to both improve accessibility and discoverability; and to control authentication procedures, creating audit trails to ensure that material cannot be accessed or altered by those not authorised to do so. Digital material is actively preserved, used and reused for new purposes, creating new materials. This is Digital Curation: the management and preservation of digital material to ensure accessibility over the long-term

3 、 Curation 和 Preservation不同？应对的问题不同

Preservation 应对技术退化和组织失效

Curation From Data Deluge to Data Curation, Data v

olumes, complexity of the data itself

3 、 Curation 和 Preservation不同？行动的目的不同

Preservation 以数据的生存为目的保证数据完整性、可信赖、真实性

Curation 以数据能够被科研利用为目的实现数据管理并使数据增值

3 、 Curation 和 Preservation不同？达成的目标

Preservation使数据可访问、可理解、可应用

Curation 对数据的整个生命周期进行管理，包括数据的创建和在旧数据之上新生成的新数据，实现数据利用和再生

3 、 Curation 和 Preservation不同？为什么人服务？

Preservation 为了未来后世能够利用

Curation 为了当前和未来可用

3 、 Curation 和 Preservation不同？行为模型

Preservation OAIS参考模型

Curation DCC Curation Lifecycle Model

3 、 Curation 和 Preservation不同？ OAIS参考模型

6 项功能活动、 3类信息包、 3种角色


DCC Curation Lifecycle Model Full Lifecycle Actions

Description and Representation Information Preservation Planning Community Watch and Participation Curate and Preserve

Sequential Actions Conceptualise Create or Receive Appraise and Select Ingest Preservation Action Store Access, Use and Reuse Transform

Occasional Actions Dispose Reappraise Migrate

3 、 Curation 和 Preservation不同？活动参与成员

Preservation 数据提供者、数据保存者、受权使用者

Curation 数据创造者、数据提供者、数据存档者、数据消费者

3 、 Curation 和 Preservation不同？保存的周期

Preservation从数据提供开始，一直到所要求的未来时段，保证数据生存

Curation从数据的产生开始，数据整个生命周期，中间有丢弃

1 、从数字保存到数字保管数据应用范围

Preservation受权访问

Curation 数据共享、数据重用

3 、 Curation 和 Preservation不同？思路方法

Preservation迁移、仿真

Curation creation and management add value to generate new sources of infor

mation and knowledg

3 、 Curation 和 Preservation不同？保存中的主观能动性

Preservation Preservation implied a passive state

Curation Digital material is actively preserved active management of data throughout th

e research lifecycle. active management and appraisal of data

over the entire life cycle.

3 、 Curation 和 Preservation不同？保存的地方

Preservation inaccessible “dark archive”

CurationOpen Trusted Repositories




4 、 Digital Curation 挑战 e-Science Curation Report



4 、 Digital Curation 挑战 Data Tsunami 、 Data deluge 、超规模数据

CERN （欧洲核能研究组织） ESA （欧洲航天局）

未来数据规模将更大，数据增长将更快天文观测数据

Sloan Digital Sky Survey ， 2008 年的前 10 年，产生 25 terabytes 数据

2014 ， Large Synoptic Survey Telescope每晚 20 terabytes

2019 年， Square Kilometre Array radio telescope将产生 50 TB 已处理的数据，如果以裸数据为计，每秒 7000TB

4 、 Digital Curation 挑战 Big Data——>big data science

“ 大数据科研”的时代已经来临不仅限于大装置或部分领域的科学大数据科研是一种新的科学发现范式

Data-intensive Science ， Data-intensive Discovery

存在于所有科研领域观测、试验和计算机产生数据日益增长的价值不论是物理科学、人文科学，还是社科科学。

4 、 Digital Curation 挑战 Data as the Infrastructure

European Union “In a sense, the physical and technical infrastructu

re becomes invisible and the data themselves become the infrastructure a valuable asset, on which science, technology, the economy and society can advance”

GRDI2020 项目将构建 Research Data Infrastructure 促进数据管理系统、数字图书馆、研究图书馆、数字仓储、工具及研究团队的集成，

第 8框架项目将 data e-infrastructure 作为优先领域（ Horizon 2020 programme ）

4 、 Digital Curation 挑战大数据科研带来了一系列新的问题和挑

战数据政策问题保管规划问题保管可靠可信问题保管内容揭示问题保管技术框架问题保管仓储系统问题 ......

4 、 Digital Curation 挑战数据政策问题（ Data Policy ）

公共资助的研究数据，如何来保存和利用总体上趋向开放获取和应用 OECD

Principles and Guidelines for Access to Research Data from Public Funding （ 2007 ）

Research Councils UK (RCUK) Common Principles on Data Policy ， 2011

美国 Open Data Policy-Managing Information as an Asse

t ,2013,5,9

4 、 Digital Curation 挑战保管规划问题

问题 Open Data 是否意味着发布所有数据？保存的具体目标是什么？ everything should be preserved ？

相关项目 MaRDI-Gross project

Managing Research Data Infrastructures – Big Science DMP planning within big science developing a toolkit to provide guidelines on the applicatio

n Plato is a well-established tool for systematic prese

rvation planning

4 、 Digital Curation 挑战保管可靠可信问题

Data Seal of Approval (DSA) Audit and certification of trustworthy

digital repositories (ISO 16363) Criteria for Trustworthy Digital Archive

s (DIN 31644)

4 、 Digital Curation 挑战保管内容揭示问题

Large-scale content profiling for preservation analysis

C3PO （ Clever, Crafty Content Profiling of Objects ）

SCAPE Data Curation Profile

Purdue University Libraries

4 、 Digital Curation 挑战保管技术框架问题

在大数据科研环境下，通常的技术框架不能解决相关的问题

如何快速有效地实现保存分析？ MapReduce方法的应用

欧盟 FP7 的 SCAPE(Scalable Preservation Environments) 项目

英国 Research Data Management Infrastructure (RDMI) Projects

4 、 Digital Curation 挑战保管仓储系统问题

Supporting the preservation lifecycle in repositories, or2013.net

4 、 Digital Curation 挑战保管仓储系统问题

基于当前工具的大规模保存仓储生命周期框架




5 、结语好的科研需要好的数据保证为了让使能够带来 Nobel Prices 的科学

数据不丢失，需要有效实现数据的保存 Digital Curation 是积极实现数据保存管理的重要手段

在大数据科研环境下， Digital Curation 还面临着很多关键的挑战

希望更多的人关注和研究这一重要问题

欢迎批评谢谢报告中引用的所有资源的作者欢迎各位专家的批评指正！

张智雄[email protected]

Documents

Digital Curation for the Big Data Sciences 大数据科研中的数字保存 张智雄 中国科学院国家科学图书馆

Digital Curation for the Big Data Sciences 大数据科研中的数字保存张智雄中国科学院国家科学图书馆