Upload
snaplogic-inc
View
671
Download
11
Embed Size (px)
DESCRIPTION
In our bi-weekly TechTalks designed for SnapLogic partners and customers, our cloud integration experts talk about tips and tricks for the SnapLogic Elastic Integration Platform not covered in basic training. This week’s TechTalk focuses on scaling integrations with two basic approaches, and provides recommendations on using Child Pipelines when the work you want to parallelize is compute heavy, and using a Splitter and Join Snap when the work is computationally light. To learn more, visit: SnapLogic.com/techtalk
Citation preview
SnapLogic Best Practices:Tips and Tricks in 30 Minutes
Praneal NarayanJason Slater
2
Goals for the Tech Talk Series
1) Share Best Practices
2) Connect with the Cloud Integration Experts
3) Learn Tips and Tricks not Covered in Basic Training
Use the Chat Panels for Q&A
3
Agenda
Introductions– Praneal Narayan
– Jason Slater
Today’s Topic: 2 Approaches to Scale Your Integrations– Eliminate bottlenecks
– Improve pipeline performance
Wrap Up and Topics for the Next Session
4
Before We Get Started…..
Documentation
Download the following:– Snaplex
– JSDK
– Snap Packs
– Mac, Linux and Windows Installers
Community Board and Idea Exchange
Don’t forget to use our Developer Community
developer.snaplogic.com
5
Scaling Your Integrations
1) Child Pipelines– Split work into parent and child pipelines
– Distributes execution across multiple nodes
– Requires that you configure a ForEach Snap
2) Splitting/Joining data flow and (optionally) embedding a pipeline– Useful for performing things like multiple web service
calls in parallel
– Requires a Router Snap to fork data-flow
– Use the forked data, either directly using Snaps or by embedding a pipeline
Two approaches we’ll be talking about today…
6
Recommendation
Use Child Pipelines:– When the work you want to parallelize is compute
heavy
Use a Splitter and Join Snap:– When it’s computationally light
7
A Few Tips
✔ Parameters in child start with underscore (_) and you often need to use a Data Snap to copy it to a $ variable
With the Router Snap, use numeric ranges, hash character, etc.
Don't pass parameters to embedded pipelines, since they normally receive documents
Name each embedded pipeline instance so you can better understand run statistics later
Be aware of limitations such as throttling and concurrent requests
Be aware of mutually exclusive things you do in parallel pipelines, like writing to a file w/ overwrite option selected; DB is better approach.
✔
✔
✔
✔
✔
8
Connecting with the SnapLogic Team
@SnapLogic
Facebook/SnapLogic
www.SnapLogic.com
developer.snaplogic.com
9
Appendix – ForEach
Setting up your child pipeline using ForEach– Split work into parent and child pipelines
– Distributes execution across multiple nodes
– May alleviate CPU and NIC resource contention for computation-intense pipelines
– Pipelines have spin up/down overhead: may not be best approach for very short-lived pipelines
– Configuring the ForEach Snap• Choose the Snaplex: where child pipelines will run
• Synchronous vs. Fire-and-Forget
• Batch size for Synchronous Mode: how many child pipelines to run at a time
• Pass parameters from parent to child
10
Appendix – Router, Forked, Union
Use a Router snap to fork data-flow to:– Increase number of outputs to have more tines
– Choosing and using an expression for each tine of the fork
– Select First Match
– Inspect output of each tine to validate that your fork is working properly, given limited preview data
Use the forked data, either directly using Snaps or by embedding a pipeline– Depends on number of times, if you need to process the
forked data before/after
– If you modify an embedded pipeline's first or last snaps, you will need to re-add each instance to the parent pipeline
Use a Union snap to join the data– This merges the flow of documents back together