Embed Size (px)
Citation preview
A middleware that allows to dynamically extend the pipeline with an isolated, language independent functionality that works Inline with PUT and GET
© IBM Corporation 3
Glossary
• A Storlet is a compiled code, deployed to Swift.• Storlets Compute Engine: a sandboxing technology, used for running storlets in a secured
and isolated manner inside Swift nodes.• Storlets Gateway: A compute engine specific implementation that glues together Swift with
the engine. The gateway implements a standard API for storlets invocations.• Storlets middleware: Intercepts storlets upload and execution requests and invokes the
storlet gateway via its standard API accordingly
© IBM Corporation 4
Glossary
• A Storlet is a compiled code, deployed to Swift.• Storlets Compute Engine: a sandboxing technology, used for running storlets in a secured
and isolated manner inside Swift nodes.• Storlets Gateway: A compute engine specific implementation that glues together Swift with
the engine. The gateway implements a standard API for storlets invocations.• Storlets middleware: Intercepts storlets upload and execution requests and invokes the
storlet gateway via its standard API accordingly
This session is about the storlets middleware and the gateway API
© IBM Corporation 5
Per-Storlet Daemon SbusPer-Storlet Daemon Sbus
High Level DesignSwift Node
Factory Sbus
'Host' ‘Per Swift Account Docker Container'
'Host'
Storlets Swift Middleware
DaemonFactory
StorletDaemon
Storlet
StorletDaemon
StorletStorlet
StorletDaemon
Per-Storlet Daemon Sbus
Docker Registry and Docker Image Management
Storlets Docker Gateway
© IBM Corporation 6
Storlet Upload
Proxy Tier
Storage Tier
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
PUT http://{host}/v1/{account}/storlet/{storlet_name}
© IBM Corporation 7
Storlet Upload
catch_errors
storlet_handler
proxy-server
Gateway
validate_upoad
authorize_storlet_execution
augment_storlet_request
invoke_get
invoke_slo_get
invoke_put
© IBM Corporation 8
Invocation on GET
GET http://{host}/v1/{account}/{container}/{object}X-Run-Storlet: my_storlet-1.0.jar
Proxy Tier
Storage Tier
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
© IBM Corporation 9
Invocation on GET
catch_errors
storlet_handler
proxy-server
Gateway
validate_upoad
authorize_storlet_execution
augment_storlet_request
invoke_get
invoke_slo_get
invoke_put
© IBM Corporation 10
The Storlet Gateway API• StorletGateway(account, conf)
• There is a different sandbox instance per account• conf is the compute engine configuration loaded and parsed once by the middleware and passed to
the gateway• validate_storlet_upload(request)
• Called upon a put to a container that is defined in the storlet middleware configuration as a storlet / dependency container
• authorize_storlet_execution(request)• Called before invoking storlet execution, allowing the gateway to enforce execution ‘access control’
• invoke_get(response) / invoke_slo_get(response)• The GET invocations are done with the response which carries the object data• Return a <metadata dictionary / data stream> pair, where the stream is wrapped with an ‘app_iter’
• invoke_put(request)• The PUT invocation is done with the request which carries the uploaded data
• augment_storlet_request(request)• Allows the proxy middleware add information to be consumed by the object server middleware
© IBM Corporation 11
Integration Aspects
• SLO – Placing the storlet_handler before the SLO middleware gives access to the full object on GET
• Ranges – interpreted as storlet input. For storlet output we may end up working hard to find out there is no such range
• EC/Encryption – make sure to locate the storlet_handler at the right place in the proxy pipeline and do all the computations on the proxy.
• Need a hint that the policy is EC• Etag – Viewing Etag as a property of an object kept in persistent storage, the Etag of data
returned from a GET + X-Run-Storlet is probably irrelevant, and should probably be filtered out
