SHARE Notification Service Development Update from the Center for Open Science

Center for Open Science

CommunityMetascienceInfrastructure

http://osf.io

ModularReliableFlexibleScalable

COS-SHARE Development Team

Brian Geiger Chris Seto Coral Sheldon-Hess

Erin Braswell Fabian Von Feilitzsch

COS-SHARE Developer Alumni

Casey Rollins Erica Baranski Faye Huynh Lauren Revere Ginny Huang

Michelle Yao Peter Fan Saman Ehsan Xander Herrick

Gather Notify

Providers Consumers

How are we getting data?Application

Programming

Interface

GET https://frogworld.com/api/frogs

Request Response

[{

name: Kermit,color: green,type: felt

},{

name: Travis,color: green,type: tree

}]

POST https://osf.io/api/share

Request Response

{title: Easy Being Green?,contributors: [

Kermit,Travis

],source: frogworld,id: 10.100/frogworld.102

}

Success

Resource 1 Resource 2

{title: Easy Being Green?,contributors: [

Kermit,Travis

],source: frogworld,id: 10.100/frogworld.102

}

{title: Easy Being Green?,contributors: [

Kermit,Travis

],origin: frogworld,id: 10.100/frogworld.102, description:

Exploring greenness.}

Resource 1 Resource 2

{title: Easy Being Green?,contributors: [

Kermit,Travis

],source: frogworld,id: 10.100/frogworld.102

}

{name: Easy Being Green?,contributors: [

Frog Scientists Intl,Amphibians United,

],origin: frogworld,doi: 10.100/frogworld.102

}

Resource 1 Resource 2

{title: Easy Being Green?,contributors: [

Kermit,Travis

],source: frogworld,id: 10.100/frogworld.102

}

{title: Easy Being Green?,source: frogworld,id: 10.100/frogworld.102

}

Resource 1 Resource 2

{title: Easy Being Green?,contributors: [

Kermit,Travis

],source: frogworld,id: 10.100/frogworld.102

}

{title: Easy Being Green?,contributors: [

No Contributors],source: frogworld,id: 10.100/frogworld.102

}

Gather Notify

Providers Consumers

Request

GET

http://repository.stcloudstate.edu/do/oai/?

verb=ListRecords

&metadataPrefix=oai_dc

&from=2014-10-02T00:00:00Z

Request Response

GET

http://repository.stcloudstate.edu/do/oai/?

verb=ListRecords

&metadataPrefix=oai_dc

&from=2014-10-02T00:00:00Z

<record><date> … </date><title> … </title><contributors>

<name>...</name><name>...</name>

</contributors><description>

…</description><id>...</id>

</record>

Provider:stcloudstate

Provider:stcloudstate 11:05

Provider:stcloudstate 11:05 GET

stcloud.edu/api/

Provider:stcloudstate 11:05 GET

stcloud.edu/api/

GETplos.org/api/

GETcolumbia.edu/api/

GETmit.edu/api/

GETvtech.edu/api/

GETdataONE.org/api/

Queue

Provider:stcloudstate 11:05 GET

stcloud.edu/api/

GETplos.org/api/

GETcolumbia.edu/api/

GETmit.edu/api/

GETvtech.edu/api/

GETdataONE.org/api/

Queue

worker

server image by RRZEicons

Provider:stcloudstate 11:05 GET

stcloud.edu/api/

GETplos.org/api/

GETcolumbia.edu/api/

GETmit.edu/api/

GETvtech.edu/api/

GETdataONE.org/api/

Queue

worker

worker

worker

worker

server image by RRZEicons

GETstcloud.edu/api/

Archive Response

Archive Data

Archive Request

GETstcloud.edu/api/

Archive Response

Archive Data

Normalize

Archive Request

<record><header>

<identifier>oai:repository.stcloudstate.edu:comm_facpubs-1001</identifier><datestamp>2014-10-01T17:54:11Z</datestamp>

</header><metadata>

<oai_dc:dc><dc:title>The Press as an Elite Power Group in Japan</dc:title><dc:creator>Akhavan-Majid, Roya</dc:creator><dc:description>

A lengthy field study in Japan using interviews and other sources and focusing on the nation's five leading national newspapers and Tokyo's major television stations finds evidence of much overlap between industry and the news media, through interlocking directorships and social club memberships

</dc:description><dc:date>1990-12-01T08:00:00Z</dc:date><dc:identifier>http://repository.stcloudstate.edu/comm_facpubs/3</dc:identifier><dc:source>Mass Communication Faculty Publications</dc:source><dc:publisher>the Repository at St. Cloud State</dc:publisher><dc:subject>Mass Communication</dc:subject>

</oai_dc:dc></metadata>

</record>

{ "source": "stcloud", "description": "A lengthy field study in Japan using interviews and other sources and focusing on the nation's five leading national newspapers and Tokyo's major television stations finds evidence of much overlap between industry and the news media, through interlocking directorships and social club memberships,", "contributors": { "given": "Roya", "family": "Akhavan-Majid", "middle": "", "email": "" }, "title": "The Press as an Elite Power Group in Japan", "dateCreated": "1970-01-01T00:00:00", "dateUpdated": "2014-10-01T17:54:11+00:00", "tags": [ "mass communication"], "id": { "url": "http://repository.stcloudstate.edu/comm_facpubs/3", "serviceID": "oai:repository.stcloudstate.edu:comm_facpubs-1001", "doi": "" }}

GETstcloud.edu/api/

Archive Response

Archive Data

Normalize

Send to OSF

Archive Request

Gather Notify

Providers Consumers

Notify

Consumers

OSF Metadata

OSF Metadata and Search APIs

Notify

Consumers

RSS

PubSubHubbub

ResourceSync

OSF Search Engine

OSF API

RSS

/rss/?q=open AND science AND source:scitech

PubSubHubbub

ResourceSync

/resync/resourcelist/?q=open AND science AND source:scitech

Inside the Notification Service

3:30 in this room

Recap of lessons learned

Today….

Over 20 content providers

Robust backend providing:➢ archive of all research release events➢ monitoring and extensive logging➢ recovery from service interruptions

Solid foundation to enable building of other services like curation and networking of research projects

More institutional repositories

Tests with complementary types of content:

Providers able to push content to SHARE NS

Coming soon...

Today Early 2015

Fall 2015

Beta release

Full release

20 + service providersRobust backendOutput formats

RSSResourceSyncSearch

More service providers Complementary dataRefined metadata schema and push of content

Try it: http://share-dev.osf.io

Follow on GitHub: https://github.com/CenterForOpenScience/SHARE

See more details:https://osf.io/29ypx/wiki/

[ Intentionally Left Blank ]

Inside the Notification Service

Elasticsearch

RabbitMQ

MongoDB/TokuMX

Celery Workers OSF App Servers

ScrAPI

Load Balancers

Caching Proxies

Raw

UserProvider Provider Provider

FluentdElasticsearch

Kibana

UserUser

● Reliable ○ Messages persist○ Redundant message storage

● Scalable○ easily clustered

● Supported○ Most used message broker

RabbitMQ

Celery● Asynchronous job execution● Scalable

○ Allows for any number of worker machines

● Fault resistant● Large community

○ Tight integration with RabbitMQ

MongoDB/TokuMX● Fast● Transaction-based● Improve document

storage● Document-level

locking

Elasticsearch● Flexible

○ Schema-less

● Built to be horizontally extensible

● Robust API

Fluentd● Most robust log

aggregation system● Converts log messages to

JSON for easy processing● Emphasizes simplicity

Sentry● No exceptions uncaught

● Real-time logging

● Easy triage

Kibana● Nice visualizations● Easy setup● Simple to use● Stats at a glance● In depth progress

monitoring

HarvestersGathering metadata from providers

RSS Feed

SHARE in the world

SHARE on GitHub

https://github.com/CenterForOpenScience/SHARE

Ways to access SHARE

Search API

Knockout - Observables

Knockout - Functions

HTML