Upload
wso2
View
953
Download
0
Embed Size (px)
DESCRIPTION
Citation preview
Scalability, Availability & Management of WSO2 Carbon
Afkham Azeez
Director of Architecture
WSO2 Inc
1
About Me
• PMC member Apache Axis, Committer Synapse & Web Services
• Member, Apache Software Foundation
• Co-author, Axis2 Web Services
• Director of Architecture, WSO2 Inc
• Blog: http://blog.afkham.org
• Twitter: afkham_azeez
2
Agenda
• A brief look at the WSO2 platform
• Carbon clustering for scalability & availability
• WSO2 Elastic Load Balancer
• Worker node & management node separation
• Deployment synchronization
• Logging
• Lazy loading
3
WSO2 Offerings • WSO2 Carbon
• Full platform of servers for deployment on-premise, in private or public cloud
• Products share a consistent architecture and core platform services (e.g. logging, management, security, identity, caching) through OSGi and the “Carbon Core”
• Includes ESB, AppServer, Data Services, Governance, Identity, Business Process, and more
• WSO2 Stratos
• Platform-as-a-Service (PaaS) Foundation
• Supports running servers as elastic, metered, billed, multi-tenant with self-service
• Including all Carbon Servers, PHP, Jetty, and a growing list through a standard Cartridge model
• WSO2 StratosLive
• http://stratoslive.wso2.com
• WSO2’s Public PaaS
• An instance of Stratos running in the cloud with all Carbon Servers available
4
Consistent Architecture
• Carbon: A consistent set of class-leading enterprise servers
• The same products run either on-premise or in the cloud, single-tenant or multi-tenant
• Utilize the same Carbon core runtime for a seamless experience
• Stratos: A cloud platform for enterprise, hybrid and public deployment
• Extends the deployment to support full self-service, elastic scaling, metering and billing
• Supports Carbon and native server runtimes
• Including Java and non-Java servers such as Jetty and PHP
• Re-uses the same core Carbon architecture to offer core PaaS services including:
• Identity, Logging, File, Relational Storage, Column Storage, Code Deployment, etc
• Both projects share a common set of OSGi modules and a core runtime architecture
5
WSO2 SOA Platform
6
WSO2 Carbon
7
Carbon Clustering
• Membership types
• Static
• Dynamic
• Hybrid
• Membership modes
• Multicast
• Well-known address
8
Static membership
• Predefined members
• Other (non-predefined) nodes cannot join
M2 M1
M3 M4
Static group
N
9
Dynamic membership
• No predefined members
• Nodes can join & leave
M2 M1
M3 M4
Dynamic group
N Join
10
Hybrid membership
• Some predefined (well-known) members, and some dynamic members
• Nodes can join & leave
• Membership revolves around the static members
Hybrid group
N
M2 M1
M3 M4
Static members
M6 M5
M7
Dynamic members
Join (IP,
Port)
11
Multicast based membership management
N
M2
M1
M3
M4
Join (IP,
Port)
12
Well-known Address (WKA) based membership management
Hybrid group
N WK2
WK1
WK3 WK4
Static members
M6
M5
M7
Dynamic members
Join (IP,
Port)
Notify
13
Multicast vs. WKA
Multicast WKA
All nodes should be in the same subnet Nodes can be in different networks
All nodes should be in the same multicast
domain No multicasting requirement
Multicasting should not be blocked
No fixed IP addresses or hosts required At least one well-known IP address or host
required
Failure of any member does not affect
membership discovery
New members can join with some WKA
nodes down, but not if all WKA nodes are
down
Does not work on IaaSs such as Amazon
EC2
IaaS-friendly
Requires keepalived, elastic IPs or some
other mechanism for remapping IP
addresses of WK members in cases of
failure 14
Multicast vs. WKA – how to decide?
• Multicast
• Cluster is going to be setup in a network where multicasting is allowed
• WKA
• Cloud based deployment
• Members are distributed across datacenters & regions
• Multicasting blocked
15
Configuring Clustering
<parameter name="membershipScheme">multicast | wka</parameter>
Parameters related to multicast based membership discovery
<parameter name="mcastAddress">228.0.0.4</parameter>
<parameter name="mcastPort">45564</parameter>
<parameter name="mcastFrequency">500</parameter>
Parameters that describe the local member
<parameter name="domain">wso2.carbon.domain</parameter>
<parameter name="localMemberHost">127.0.0.1</parameter>
<parameter name="localMemberPort">4000</parameter>
<parameter name="properties">…
16
Configuring Clustering Static/well-known members <members> <member> <hostName>127.0.0.1</hostName> <port>4000</port> </member> <member> <hostName>127.0.0.1</hostName> <port>4001</port> </member> </members>
State replication
<stateManager class="org.apache.axis2.clustering.state.DefaultStateManager"
enable=”true">
17
HTTP Session Replication
• catalina-server.xml • <Cluster className="org.wso2.carbon.core.session.CarbonTomcatSimpleTcpCluster"/>
• <Valve className="org.wso2.carbon.tomcat.ext.valves.CarbonTomcatSessionReplicationValve"/>
• web.xml • <distributable/>
18
Elastic Load Balancer 2.0
• New sysadmin-friendly configuration language
• High performance PassThrough transport
• Tenant-aware load balancing
• Ability to dedicate clusters for tenants (private jet mode)
• Improved auto-scaler
• Separate IaaS-aware Cloud controller takes care of spawning new instances on different IaaSs
19
ELB Config
20
Passthru Transport Latency Additional time taken via router/ESB/ELB vs direct
-1.00
1.00
3.00
5.00
7.00
9.00
11.00
13.00
15.00
10 25 50 100 250 500
La
ten
cy (
ms)
Concurrent clients
Axiom
Binary
Relay
Passthru
21
Tenant-aware LB
22
Private Jet mode
• Analogy
• Economy class
• no SLA management, only elasticity
• Business class
• elasticity plus SLA guarantees
• Private Jet
• Guaranteed isolated VMs or machines for a specific tenant
• Still elastically scaled
Private Jet Mode
24
Auto-scaling
25
Management & Worker Node Separation
26
Management & Worker Node Separation
• Proper separation of concerns - management nodes specialize in management of the setup while worker nodes specialize in serving requests to deployment artifacts
• Only management nodes are authorized to add new artifacts into the system or make configuration changes
• Worker nodes can only deploy artifacts & read configuration
• Lower memory foot in the worker nodes because the management console related OSGi bundles are not loaded
• Improved security - management nodes can be behind the internal firewall & be exposed to clients running within the organization only, while worker nodes can be exposed to external clients.
27
Deployment Synchronization
• DepSync allows you to synchronize deployment artifacts across nodes in a cluster
• Also includes meta data synchronization
28
Deployment Synchronization
29
Management Node Config – carbon.xml
<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>true</AutoCommit>
<AutoCheckout>true</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>http://10.100.3.115/svn/repos/wsas</SvnUrl>
<SvnUser>wso2</SvnUser>
<SvnPassword>wso2123</SvnPassword>
<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>
AutoCommit/Checkout = RW
30
Management Node Config – catalina-server.xml
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol” port="9763” proxyPort="8300” …
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol” port="9443" proxyPort="8263” …
31
Worker Node Config - Create worker node
ant createWorker
Buildfile: /Users/azeez/software/wso2carbon-core-4.0.2/bin/build.xml
createWorker:
[input] You are about to delete all the front-end components from the server runtime. Do you really want to proceed? (y, [n])
32
Worker Node Config - Start worker node
./wso2server.sh -DworkerNode
33
WorkerNode Config – carbon.xml
• <!--
• Host name or IP address of the machine hosting this server
• e.g. www.wso2.org, 192.168.1.10
• This is will become part of the End Point Reference of the
• services deployed on this server instance.
• -->
• <HostName>appserver.wso2.com</HostName>
34
Worker Node Config – carbon.xml
<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>false</AutoCommit>
<AutoCheckout>true</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>http://10.100.3.115/svn/repos/wsas</SvnUrl>
<SvnUser>wso2</SvnUser>
<SvnPassword>wso2123</SvnPassword>
<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>
AutoCheckout = RO
35
Logging
36
Lazy Loading
• Four variants
• Lazy initialization
• Virtual proxy
• Value holder
• Ghost
• Two aspect of lazy loading in Carbon
• Lazy loading global configuration
• Lazy loading artifacts
37
Ghost deployment for services
carbon.xml - <EnableGhostDeployer>true</EnableGhostDeployer>
38
Lazy loading with Ghost Deployment
39
BAM 2.0
TODO
40
BAM 2.0
41
Summary
Availability Scalability Management
State replication Tenant partitioning
Private jet mode
Management nodes
Session replication Ghost deployment Logging infrastructure
Multiple load balancers with
keepalived or DNS RR
BAM 2.0 architecture Deployment synchronization
Autoscaling
Elastic Load Balancer
42
References
Information on tenant-aware load balancing http://sanjeewamalalgoda.blogspot.com/2012/03/tenant-aware-load-balancer-is-upcoming.html
http://sanjeewamalalgoda.blogspot.com/2012/05/tenant-aware-load-balancer.html
Information on long running performance http://kishanthan.wordpress.com/2012/06/27/lazy-loading-pattern-in-wso2-stratos
Lazy loading deployment artifacts http://blog.afkham.org/2011/11/lazy-loading-deployment-artifacts-in.html
Scaling Stratos http://srinathsview.blogspot.com/2012/06/scaling-wso2-stratos.html
http://blog.afkham.org/2011/09/how-to-setup-wso2-elastic-load-balancer.html
http://blog.afkham.org/2011/09/wso2-load-balancer-how-it-works.html
43
Auto-scaler service deployment http://nirmalfdo.blogspot.com/2012/07/autoscaler-service-deployment.html
Auto-scaler service http://nirmalfdo.blogspot.com/2012/07/wso2-autoscaler-service-part-i.html
Pass-through transport – performance comparison http://wso2.org/library/articles/2012/03/wso2-esb-message-transfer-mechanisms-comparative-
benchmarks/
Automatic failover for WSO2 ELB http://gonesimple.org/2012/09/24/automatic-fail-over-for-wso2-elb/
44
Questions?
45
http://www.flickr.com/photos/oberazzi/