FolderShare: Building a data sharing cloud on Drupal 8 for ... · Node : 25,639 (Drupal core 8.5.0)...

AmitChourasia,DavidNadeau&MichaelNormanSanDiegoSupercomputerCenter,UCSanDiego

Projectcode: dibbs.seedme.org/downloads

or

drupal.org/projects/foldershare

drupal.org/projects/smalldata

Trialwebsite: sandbox.seedme.org

Projectwebsite:dibbs.seedme.org

FolderShare:BuildingadatasharingcloudonDrupal8forresearchers

PresentaHonOverview1.  Background&moHvaHon2.  Architecture3.  SeedMe2plaPorm

FolderShare:VirtualfilesystemforDrupal

»  EnHtydatamodel&accesscontrol»  Filemanagement&security»  ViewsintegraHon»  UI&Commandplugins»  FileformaUers»  Webservices

SmalldatamoduleandAPI

–  VisualizaHon4.  Targetusers/Usecases5.  Screenshots6.   Demo

SeedMe1

a.k.a.StreamencodeexploreanddisseminateMyexperiments•  VideoencodingfromvisualizaDonimageswasthemainfocus

•  BasedonDrupal7•  InproducHonasPlaPormasaService(PaaS)

SeedMe2:Datasharingbuildingblocks

EvoluHonoftheoriginalSeedMeproject(Completerewrite)•  DataorganizaDon,sharinganddatamanagementisthemainfocus

•  IncorporatesuserfeedbackfromoriginalSeedMeproject•  BasedonDrupal8•  BuiltfordistribuHon(opensource)andextension

SeedMeProjects

SeedMe2’sfocus•  Enablerapidaccesstodatatoconsumabledataovertheweb

Mustbehandledbystockwebbrowser(Upload<2GBperfile)Mustbedisplayableonmanydevices(PhonetoPC)

•  RichpresentaDonSupportcustomizaHon/extension

•  DiscoverabilityIndexandsearchfiles,descripHons,etc..

•  ThreewayinteracDonWebbrowserCommandlineAPI

Datamanagementstumblingblocks

Transfer Storage CollaboraJon AutomaJonAccesscontrol

ButwhataboutPresentaJonandDiscovery?

Datamanagementstumblingblocks

Transfer Storage CollaboraJon AutomaJonAccesscontrol

Issuesduetocontentdispersion

DescripJoninsomeone’smindDataintheCloud Discussiononemails

ThreeD’s:Data,DescripGon,Discussion

FilesystembasedsoluDons

RelatedsoluHons

ContentmanagementsystembasedsoluDonsHubZeroFigShare

MiddlewareGlobusIRodsNEWT

SoNwarerepositoriesGitHubSVNCVS

FilehosDngClouddrives(Dropbox,etc)WebDAV

ToolsSCPFTP

LimitaDonsofexisDngsoluDons Lackextensibility Lacksupportforrichcontent(descripHon,discussion,etc…) Lackindependentdevelopersupport Lack3wayinteracHonviawebbrowser&commandline&API Resourcerestricted

Workflow

Update as desired

Description

CreateProject

Sharing

Add foldersView

SearchDownload

Web browser, Command line, REST or App

Sign In

1 2

3

4

Upload files

Users

Drupal8

Architecture

Modules

•  Virtualfilesystem•  Accesscontrol•  Hierarchicalstorage•  Commandplugins•  UIanddisplay•  Search/index•  Webservices

•  FileformaUers•  QuickVisualizaHon

Webserver(Apache+PHP)

Database(MySQL)

Webbrowser

RESTclients

Commandline

Smalldata(PHPlibrary)

Drupal8

Contributed

Modules

e.g.

Federated

AuthenDcaDon

viaOIDC

module

ProjectcontribuHons

Drupal(ContentManagementSystem)

•  Modularandextensiblearchitecture(1,000+communitycontributedmodulesforversion8)•  Widelyusedinindustry,academiaandgovernment(thirdmostpopularCMSonweb)•  Largedeveloper&supportcommunity(4,000contributorsforDrupal8core+thousandsmore)•  Securityadvisoryandupdatesforcoreandstablecontributed

moduleseverymonth•  VersaHledeploymentopHons(personalhosHng,insHtuHonalhosHng,cloudhosHng)

FolderSharemodule

•  Virtualfilesystem(fieldable):–  EnHtytype&API– Accesscontrols– Usagetracking–  Views,displays,breadcrumbs,forms–  PluginsforfieldformaUers,search,views,acHons,andqueueworkers

•  Configurablebysites–  e.g.Keywords,comments,flags,DOIs

•  Extensiblebydevelopers

Files&folders

•  Childrenpointtoparents– ParentIDsenablefastqueriesforallchildrenofafolder

– RootIDsenablefastqueriesforaccesscontrolsandbreadcrumbs

parenHdpointstoimmediateparent

rooHdpointstotopfolder

Accesscontrols

•  Drupalaccount-based•  Permissions+accesscontrollistontopfolders– Listofusersthatcanviewandauthor

•  TopfoldercontrolsenHrehierarchy– SimplerthandesktopOSes– Similartofilesharingservices– Fasttocheckaccess

Filestorage

•  Foldersonlyexistindatabase•  Filesdescribedindatabase&storedondisk

•  Diskdirectory!==folderhierarchy– BeUerforsecurityandloadbalancing– Fileshavegeneratednames

•  Avoidsnamecollisions•  Avoidscharactersetandnamelengthlimits

Views

•  Listpersonal,public,andsharedfiles&folders– Pages&embeddedviewsinfolderpages

•  Integrateddesktop-likeUI– Selectfilesandfolders– Thenchoosemenucommand

•  ThreeUIvariants:– NoscripHng– ScripHngbutnoAJAX– ScripHngwithAJAX

Plugins

•  FieldformaUers– Foldernames,enHtyreferences,MIME-typeicons

•  Search–  Indexandpresentresults

•  Queueworker– Updatefolderhierarchysizesinbackground

•  AcHons&customcommands– MenuUIitemstoadd,delete,etc

Totallinesofphpcode

Node:25,639(Drupalcore8.5.0)Foldershare:50,156(Alphaversion)

0

20,000

40,000

60,000

80,000

100,000

120,000

140,000

v1 v2 v3 v4 v5 alpha

Linesofcode

Release

Foldersharecodemain

PHP

Docs

0

2000

4000

6000

8000

10000

12000

v1 v2 v3 v4 v5 alpha

Linesofcode

Release

Foldersharecodemisc

JS

YML

CSS

TWIG

TXT

Codetrivia

FoldershareAPIDocumentaHon

SmallDataAPI&Module

•  Structureddataparsers&writers– Tables,trees,andgraphs–  JSON,CSV,TSV,TXT,etc.

•  FieldformaUers– Light-weightvisualizaHon– Lineplots,barcharts,piecharts,etc.

SmallData

FORADMINISTRATORS

FolderShareconfiguraDonAdminmenuStructure>FolderShare

FolderShareconfiguraDonFieldsManagefields,forms&display

ConfiguraHonlocatedinadminmenuStructure>FolderShare

FolderShareconfiguraDonFiles

StoragelocaHon&uploadrestricHons

FolderShareconfiguraDonInterface

Commandplugins

FolderShareconfiguraDonLists

ManagelisHngoffileandfolders

FolderShareconfiguraDonSearch(opDonal)

FolderShareconfiguraDonSecurity

ManagesharingcapabiliHes

FolderShareconfiguraDonWebservices(opDonal)

ManageRESTcapabiliHes

FolderShareRESTseWngsRequiresRESTUIcontributedmodule

ManageRESToperaHons

FolderShareUsageAdminmenuReports>FolderShareusage

FolderSharepermissions

FORUSERS

Topfoldersownedbyyou

Topfolderssharedwithyou

Publictopfolders

Menu

Theselistsdisplaytoplevelfolders

FoldersmayhaveadescripHon

MenuopHons–withnoselecHon

MenuopHons–withselecHon

Sharingformtorestrictaccess

SortablelisHngoffilesandfolders.

Differentusers

Breadcrumbsshowspath

EveryfolderandfilemayaddadescripHon.(FormaUedtextfieldakaBodyfieldinDrupal’sNode)

AddcustomsfieldssuchasComments,Keywords/tags,taxonomytoFolderShare.

TheFolderShare

enDtyisfieldable.

MenuopHonschangeonselecHon

Viewsubfolder

Breadcrumbsshowspath

VisualizaHonscanbeswitchedinteracHvelytodifferentcharttypes

QuickvisualizaHonofCSV&JSONfiles

SamplecommandlineinteracHonfoldershare --helpfoldershare --host http://demo.seedme.org --user dave --password ’cliRocks!'

help

SamplecommandlineinteracHonfoldershare --helpfoldershare --host http://demo.seedme.org --user dave --password ’cliRocks!'

help

ls --help ls / ls -l "/Classification Collection" ls -l "/Classification Collection/Preliminary Results"

mkdir --help mkdir "/test" mkdir "/test/data"

upload --help upload "plots/villi.png" "/test" upload "plots/composite.png" "/test” Upload "sample-small-data/OpenGL mesh memory use.csv" "/test/data" Upload "sample-small-data/Image classification breakdown schedule.json" "/test/data"

download --help download "/test" "/Users/amit/downloads"

rm --help rm -rf "/test”

SeedMe2–Targetusers/Usecases•  Researchers

–  CollaboraHonhubforaprojectorpersonalwebsite•  Customprojectrepositories

–  IncludeprojectspecificcustomizaHon(e.g.taxonomy,keywords)•  Developers

–  IntegrateyourscienHficapplicaHon•  Sciencegateways

–  Datasharing–  Datapublishing–  Dataescrowservice

•  CyberInfrastructureproviders–  OfferSeedMe2plaPormtoyouruserbase/communiHes

Realusagebyphysicists

Deployment/serviceopHonsDIY-Runyourowninstance(Yourbranding+domain)•  Onyourownhardware•  Cloudhosted

Provider/vendorrunsaninstance•  YourinsHtuHon•  CIcentersorITservices•  Commercialvendors

Cloudservice(Ourdomain&branding)

Explorerswelcome(webbrowserneeded)Trialwebsite:sandbox.seedme.org

Tryityourself!

DevelopersinvitedDownloadcode:h_ps://dibbs.seedme.org/downloads

Team

AmitChourasia,DavidNadeau,DmitryMishin&MichaelNormanSanDiegoSupercomputerCenter|UniversityofCaliforniaSanDiego

SeedMe

Acknowledgements

AllusersandapplicaHonintegratorsfortheirvaluablefeedback

NaDonalScienceFoundaDon

ThismaterialisbaseduponworksupportedbytheNaHonalScienceFoundaHonunderGrantNo.ACI-1235505andACI-1443083

"Anyopinions,findings,andconclusionsorrecommendaHonsexpressedinthismaterialarethoseoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheNaHonalScienceFoundaHon."

Talk to usamit sdsc.edu

Keen to learn about potential uses cases in your work

Tutorial: Thursday afternoonBuilding data centric website for research projects and research groups

Requirements

1.   Installdocker2.   Downloadstart/finalimages(~1.2GB)