Upload
bazaarvoiceengineering
View
748
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Nexus is Bazaarvoice's next generation cloud infrastructure built on top of Amazon Web Services. Nexus is highly available and resilient, built with best practices on top of services such as VPC, Autoscaling, ELB, Cloudformation, and more.
Citation preview
NexusBazaarvoice Cloud Infrastructure
Victor TracCloud Architect, Bazaarvoice
welcome to cloud computing!!1
BazaarvoiceNot bizarre boys
Austin-based company founded in 2005
SaaS serving software that collects and displays user generated content, crunches analytics, and extracts data.
Engineering offices in Austin, NYC, London, and San Francisco
Basic stats:Thousands of clients
Hundreds of millions of pieces of content
Hundreds of millions of unique visitors per month
Tens of billions of page-views per month
Edge Traffic
EC2, S3, VPC,Regions, Autoscale,
CloudFormation, ELB...?
Does this mean anything to you?
VPC & SubnetsVPC allows us to choose our internal IP space.
Default Route for All Subnets to IGW● Let's call these subnets all "Public"● Requires all instances to have EIPs before
talking to the internet● EIPs are a limited resource
Public: Default route via IGW
Private: Default route via instance(s) in Public Subnets
Advantage: Most instances in the private subnet can talk to the internet without dealing with an EIP.
Security Groups
Autoscaling
Elastic Load Balancing
● Only Round Robin and Sticky Sessions● Supports HTTP Response code or basic TCP connection Health Checks
{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "A text description for the template usage", "Parameters": { // A set of inputs used to customize the template per deployment }, "Mappings": { // Mappings match a key to a corresponding set of named values }, "Resources" : { // The set of AWS resources and relationships between them }, "Outputs" : { // A set of values to be made visible to the stack creator }}
CloudFormation Instance Example
{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "Create an EC2 instance running the Amazon Linux 32 bit AMI.”, "Parameters" : { "KeyPair" : { "Description" : "The EC2 Key Pair to allow SSH access to the instance", "Type" : "String" } }, "Resources" : { "Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "KeyName" : { "Ref" : "KeyPair" }, "ImageId" : "ami-75g0061f", "InstanceType" : "m1.small" } } }, "Outputs" : { "InstanceId" : { "Description" : "The InstanceId of the newly created EC2 instance", "Value" : { "Ref" : "Ec2Instance" } } }}
IAM and Console AccessSign-on Credentials
● IAM Console login○ Username, Password, and MFA Time token
Access Credentials● AWS has three API types: REST, Query, & SOAP. ● Each API uses one or more Access Credentials
○ Access Keys for REST and Query APIs○ x.509 Certificates for SOAP API○ EC2 KeyPairs for instance SSH authentication
In the beginning...
A Java application server + a MySQL Database
Scaled by adding in another application server.
Then we just duplicated this entire stack, giving us two "clusters".
Scaled more by adding more and more clusters.
Add in AWS
Decentralization --> Everyone goes fast!
Goals
Full control over AWS resourcesEC2 resources, Autoscale, ELB, S3, etc.
Team IsolationResources created by one team can only be modified/terminated by
that team
3rd Party SolutionenStratus, RightScale, asgard, etc
Good
● enStratus & RightScale provide cloud-agnostic
tools
Bad
● No AWS API Access
● No AWS CLI Tools & SDKs
● Locked into only supported services
Multiple Accounts
Good
● Provides for full resource control with direct API
access
● Protects teams from one another
● Allows for easy accounting on a per-team basis
● May make it easier for external auditors to
determine which teams have
"production" access
Bad
● Inter-team network communications can become
very complicated, relying
on VPN between VPCs ->
Reduced Reliability
● Management of networking is a possible bottleneck
● Shared resources may need to be redundantly built in
every VPC: LDAP, DNS,
Monitoring
Single Shared Account
Good
● Sharing of resources will be simple - just open
access via security groups
between teams
● Reliable networking between teams without need for VPN
● Possibly better performance due to fewer hops
● Certain resources can be shared: LDAP, DNS,
Monitoring, etc.
Bad
● No built-in protections between teams, even with
IAM
● Creates a centralized resource that someone has
to maintain
● Requires us to build tools to use long-term
Nexus, circa August 2012
In more detail...
Nexus is:
● AWS Infrastructure designed with best practices:
○ secure○ highly available○ multi-region○ repeatable
● Cloud building blocks and recipes for all of Engineering
● A Single Account Solution
Philosophy: Engineering teams at Bazaarvoice are free to choose
their own stack, but we want to make Nexus so compelling that it
is the default choice.
(some) Batteries Included
Included:● Bastion Hosts
● NAT Instances
● VPN Connectivity between Regions
● Internal DNS
● Monitoring*
● Centralized Logging*
● Services Discovery*
● Scripts & CloudFormation in Github to create ephemeral
VPCs that look like a
Managed Environment
Dev teams provide
anything required to
run their app, which
probably means:
● Puppet/Chef/etc
● Your actual app
● Deployment process
Nexus Regions
bas·tion (*/ˈbas-chən/*)
NAT Instances
Autoscaling with CloudFormation
Internal DNS
External DNS (Route53)
Records in the .nexus.bazaarvoice.com zone are for you to use.
Badger
Cabertoss
Conformity -> Measurable Efficiency
Limitations & Risks
● Danger! Single Shared Account○ You can wipe out all of a region with a
bad script.
● Single NAT per AZ○ Someone else downloading lots of data from
the internet will affect all other instances sharing your private subnet.
● Single VPN Instance per VPN Destination○ Similar to NAT problem, but worse. ○ Avoid VPN when possible○ If not possible, make your VPN dependency
resilient to lack of bandwidth and network blips
Nexus is a catalyst:
old and busted
waterfall
centralized development
8-10 week release cycle
monolithic app
mysql
solr
java
dev + ops
new and shiny
agile
distributed teams
release anytime
services oriented architecture
cassandra
elasticsearch
whatever
devops
Email: [email protected]: @victortrac
Thanks!