Stack Exchange Infrastructure - LISA 14

The Stack Exchange

InfrastructureVroom Vroom

inet.perf.profile

• SRE Generalist @ Stack Exchange

• @GABeech

• http://brokenhaze.com

• http://stackexchange.com

A brief Overview

• 560 Million Page Views a Month

• 34TB of Data transfered a Month

• 1665 rps (2250 peak) Across web Farm

• WISC(HER)

Our First Priority is

PerformanceNobody likes a slow site, least of all us.

When your site is slow people leave.

Make your site fast, and the people will stay

Good write up on moz.com:

http://moz.com/blog/site-speed-are-you-fast-

does-it-matter

The Performance

toolkit

• Mini Profiler

• OpServer

(https://github.com/opserver/Opserver)

• Client Timings

(http://teststackoverflow.com/)

Mini Profiler

OpServer

OpServer HAproxy

OpServer Redis

OpServer SQL

Client Timings

You can’t be fast if

you are not up

• Highly Redundant network

• Datacenter, ISP, Edge, Core, Server,

Port

Load Balencers

• HAProxy

• 2 Servers (Hot/Standby)

• Multiple Tiers (HAProxy Processes)

SSL Termination

• Terminated at LB

• Feature added to HAProxy 1.5

• See:

http://brokenhaze.com/blog/2014/03/25/

how-stack-exchange-gets-the-most-out-

of-haproxy/

Web Servers

• IIS

• 9 Production (2 Test/Dev)

• Dell R610’s

• 32GB Memory

• 2xE5-5640

Data Tier

• MS SQL Server

• 4 Servers

• 2 Always-On Clusters

• Each Cluster 1 RW, 1 RO

Caching Tier

• Redis

• 2 Servers

• Hot / Standby configuration

Tag Engine

• Our Special index of SO

• Tagging is hard

• Written by Marc Gravell

• http://blog.marcgravell.com/2014/04/technica

l-debt-case-study-tags.html

Elastic Search

• 203GB Index

• 3 Machines

• 42M searches/day

Deployment

• Git

• TeamCity

• Custom Powershell Scripts

So what does this get

you

• 52 ms homepage render time

• 33 ms questions page render time

Always See our

Performance

• http://stackexchange.com/performance

Thank YOU!

Contact:

@GABeech

george@stackoverflow.com

Office Hours:

Wednesday, November 12th

(today…)

2:00pm - 3:30pmLISA Lab