iRODS: the use of rules and micro services for automatic data
conversion and signal pattern searching
Martyn Fletcher, Tom Jackson, Bojian Liang, Michael Weeks, Mark Jessop and Jim Austin.
Slide 28-Dec-2009
Contents
1. The iREAD project.2. iRODS overview.3. What are iRODS micro-services?4. What are iRODS rules?5. iREAD Demonstrations.6. Data conversion using iRODS.7. Pattern searching using iRODS.8. iRODS browser web client.9. Conclusions / experiences.10.Potential future work.
Slide 38-Dec-2009
The iREAD Project
• The iREAD (iRODS Evaluation and Demonstrator) Project:
– Funded by JISC: http://www.jisc.ac.uk/whatwedo/programmes/einfrastructure/iread.aspx
– All documentation, experiences, results and demonstrations are available on the public website see http://www.wrg.york.ac.uk/iread
– Demonstration of searching available for all at http://www.wrg.york.ac.uk/iread and on the White Rose Grid stand at 3:00 pm.
– Only part of the iREAD work is reported in this presentation.
– iRODS™- the Integrated Rule-Oriented Data System iRODS (DICE group at UNC Chapel Hill, University of California, San Diego. Enhancement of Storage Request Broker (SRB). http://www.irods.org
Slide 48-Dec-2009
iRODS overview
Virtual file system (layer over existing data and meta data)
Meta data(iCAT)
User
iRODSSystem
Disk, tape,etc.
iRODSServer
Disk, tape,etc.
iRODSServerDistributed
Enhancement over SRB: Rules and micro–service (within each server) can be used to automatically process data, make copies on other servers, enforce access controls, etc.
Slide 58-Dec-2009
What are iRODS micro services?• Written C.• Perform actions - executed as part of a rule.• Inbuilt micro services for accessing various parts of
the system:– create , delete, read files and collections (directories),
deny access to files, simple workflow constructs, etc. • Users can create their own micro services:
– Operate on data, run external web services, etc– We have created ones for searching, data conversion,
etc. e.g. msiSearch.• Adding or changing a micro service necessitates:
– Re-compilation of the iRODS installation.– A stop / start of the iRODS system.– This situation may change in future.
Slide 68-Dec-2009
What are iRODS rules?• A rule is a “grouping” of:
– Other rules.– Micro-services.
• Can be executed:– Automatically on events, time, upload, etc. – part
of the rule base (core.irb file).– By users on command from API or rule files.
• Can be added to a live iRODS system:– No need for a recompile and stop / start.
Slide 78-Dec-2009
What are iRODS rules (2)?• iRODS rules have the:
Action_Name|Condition|Workflow_Chain|Recovery_Chain • Action_Name is the name of the rule.• Condition is the condition for execution of the rule.• Workflow_Chain is a list of micro-services or other rules to be
executed.• Recovery_Chain is a list of recovery actions if any of the
Workflow_Chain fails.
Slide 88-Dec-2009
iREAD demonstrations• The iREAD iRODS demonstration setup is as
follows:– iRODS version 2.0.1.– The iRODS browser installation (part of the extrods
1.1.0.1 package) for connection to iRODS.– Apache web server 2.2.10 for the browser. – PHP 5.26 for the browser.
• Rules and micro-services have been added to iRODS as necessary.
• The browser installation and PHP service have also been modified for demonstration purposes.
Slide 98-Dec-2009
Data conversion using iRODS• Image conversion micro-services have been compiled
and installed (supplied with iRODS).• Additional skeleton micro-services have been included
for future conversion of Neurophysiology data formats. • Installed rules (acPostProcForPut) for post processing
of the put command.• The acPostProcPut rules – only one is executed (each
is a single line in the core.irb file):1. acPostProcForPut|$objPath like /tempZone/home/demo/files-and-conversions/*\.gif|
msiImageConvert($objPath,null,$objPath.jpg,null)|nop2. acPostProcForPut|$objPath like /tempZone/home/demo/files-and-conversions/*\.bmp|
msiImageConvert($objPath,null,$objPath.png,null)|nop3. acPostProcForPut|$objPath like /tempZone/home/demo/files-and-conversions/*\.mcd|
msiDataObjCopy($objPath,$objPath.ndf,demoResc,*Junk)|nop4. acPostProcForPut||nop|nop
Slide 108-Dec-2009
Pattern searching using iRODS (1)
• Used an existing remote search service developed during the DAME and BROADEN e-Science projects.
• Created micro services which access the web service.
• Created rule to run the search micro services.• Modified browser to add buttons, display
functions, etc. (Javascript / HTML.)• Used PHP API to run the rule.
Slide 118-Dec-2009
Pattern searching using iRODS (2)
Slide 128-Dec-2009
Pattern searching using iRODS (3)
• The pattern search rule: MyNewSearchTestRule||
msiDataObjUnlink(*resultsPath,*STATUS)##msiDataObjOpen(*queryPath,*QUERY_FD)##msiDataObjRead(*QUERY_FD,10000,*QUERY_BUF)##msiDataObjClose(*QUERY_FD,*junk)##msiNewRemoteMthSearch(*QUERY_BUF,*ClIP,*HtIP,*HtTime,*HtIndex)##msiNewRemoteGetResult(*ClIP,*HtIP,*HtTime,*HtIndex,*results)##msiDataObjCreate(*resultsPath,null,*RESULTS_FD)##msiDataObjWrite(*RESULTS_FD,*results,100)##msiDataObjClose(*RESULTS_FD,*junk)|nop##nop##nop##nop##nop##nop##nop##nop##nop
Slide 138-Dec-2009
Pattern searching using iRODS (4)• Micro services used (in following :
– msiDataObjUnlink - delete the existing results.txt file.– msiDataObjOpen - open the query.txt file. – msiDataObjRead - read the query.txt file. – msiDataObjClose - close the query.txt file. – msiNewRemoteMthSearch - invoke the search web service and
pass the query data for matching.– msiNewRemoteGetResult - poll the search service and when it
has finished search retrieve the results.– msiDataObjCreate - create a new results.txt file. – msiDataObjWrite - write the retrived results to the new results.txt
file.– msiDataObjClose - close the results.txt file.
Slide 148-Dec-2009
iRODS web browser client• All demonstrations available through the modified iRODS web browser
client accessible at http://www.wrg.york.ac.uk/iread.
Slide 158-Dec-2009
Conclusions / experiences (1)
• Rules and micro services can be run remotely on other servers (same / different zone) using the remoteExec micro service.
• Parallel execution / broadcast execution modes would greatly assist the use of iRODS for distributed operation.
Slide 168-Dec-2009
Conclusions / experiences (2)• Micro services (and rules) need to be installed
on every server where they need to executed.• Rules can become very long and unwieldy - plan
carefully - tool support would be good. • It would beneficial to be able to add micro
services to a live system without:– Recompiling the iRODS server.– Starting and stopping the server.
• Debugging:– Information not extensive e.g. a micro service can fail
producing a non informative messages on the command line and in the rodsLog.
Slide 178-Dec-2009
Potential Future Work (1)
• Provide automatic:– Generation of the search databases (as new
data arrives).– Search for known conditions in bulk data -
automatically.
• iRODS would provide benefits to the CARMEN neuroscience project to provide:– Automatic data conversion.– Service execution.
Slide 188-Dec-2009
The iREAD Project
• Work funded by JISC: http://www.jisc.ac.uk/whatwedo/programmes/einfrastructure/iread.aspx
• All documentation, experiences, results and demonstrations are available on the public website see: http://www.wrg.york.ac.uk/iread
• Demonstration of searching on the White Rose Grid stand (3:00pm today) and available for all at http://www.wrg.york.ac.uk/iread