Designing and Delivering Scalable and Resilient Web Services

Ron JacobsSr. Technical Evangelist, Microsofthttp://blogs.msdn.com/rjacobs

Agenda

The Problem

A Possible Solution

New Possibilities

What to do next

Simple

Do the simplest thing that will possibly work

Scalability

Able to support the required quality of service as the system load increases

-Wiktionary

Typical Web Architecture

• Need to get routed to same machine (i.e. sticky sessions)Users

• Each machine round trips for data

• Data that is cached is stored in the memory of one server

Web Tier

• CPU and disk can get saturated due to traffic

• Service access is slow

Data Tier

Web Explosion

Database

IIS/ASP.NET Application

Web Site’s too

slow!!

Database is hot!!

Where did my

shopping cart go?

IIS/ASP.NET Application

IIS/ASP.NET Application

Servers are crashing

Services are slow

Agenda

The Problem

A Possible Solution

New Possibilities

What to do next

Data Near Processing

ASP.NET

Web ServiceDatabase

Smart Client Browser

Cache

CacheCache

CacheCache

Good but…

• Cache is scoped to machine / process– Machines die– Processes recycle

• Cache memory is limited

What if?

• You could have as much cache as you wanted?• You could share a giant cache across servers,

services and even clients?

• What if this was something you could simply add to the platform for 1free?

1Some features may require certain editions of Windows Server

Windows Server AppFabric

AppFabricCACHING

MONITORING

WORKFLOW HOSTING

SERVICE HOSTING

SCALE OUTHIGH AVAILABILITY MANAGEMENT

AppFabric Cache – Formerly known as Code Name “Velocity”

Unified Cache View

What is AppFabric Caching?

• An explicit, distributed, in-memory application cache for all kinds of data

Caching clients can be across machines or

processes

Clients Access the Cache as if it

was a large single cache

Cache Layer distributes data

across the various cache

nodes

AppFabric Cache

• No need for sticky sessionsUsers

• Cached data available to all serversWeb

Tier

• Multiple machines means scale and high-availability

• Lots of cache memory

Caching Tier

• Reduces load on database• Duplicate Service Calls

eliminated

Data Tier

Data Distribution - Partitioned Cache

… Web Tier

Cache TierCa

che

Ser

vice

Cach

e S

ervi

ce

Cach

e S

ervi

ce

A C EDB G IHF

Scale on Data Size - More machines => More memory to cache

Scale on Cache Throughput - More machines => keys distributed across

more machines => better throughput

ASP.Net App

Caching Client

ASP.Net App

Caching Client

ASP.Net App

Caching Client

AC ED BGI HF

Scale Test Output

Load

Throughput

Latency

1 Cache ServerAs load increases, throughput fails

to scale latency increases

Caching Tier

Add Second Cache Server

ThroughputLoad Max

Throughput increasesLatency

decreases

Caching Tier

Load

Latency

Add Third Cache ServerLoad

Throughput

Latency

Caching Tier

Associated Press

• Caches metadata and news

• Serves 16 million hits per day

• Increased the amount of cached data 6 times.

System.Web.Cache

AppFabric DataCache

Deployment

Cach

ing

Ser

vice

Cach

ing

Serv

ice

Cach

ing

Ser

vice

Configuration Store

Install AppFabric

Copy Client DLLs

… Application Application Application

<hosts> <host name="BL1CDB8083714“ cachePort="22233" cacheHostName="DistributedCacheService"/> ….. </hosts> <localCache isEnabled=“true" ../><security … />

Caching Access Layer Caching Access Layer Caching Access Layer.NET 3.5 SP1 OR .NET 4

Configure AppFabric

.NET 4

Update Web.config

Usage Pattern – Cache Aside (Explicit Caching)

// Read from CacheToy toyObj = (Toy) catalog.Get("toy-101");

Application

Cach

ing

Serv

ice

Database

// If Not present in the cacheif (toyObj == null){ // Read from backend.. toyObj = ReadFromDatabase(); // Populate Cache catalog.Put("toy-101", toyObj);

return toyObj;}

Caching Access Layer

Administration

• PowerShell cmdlets are used to administer the cache cluster

• Rich set of cmdlets for – Cache cluster management– Cache creation and monitoring

Hello AppFabric Cache

Demo

Using PowerShell

Remember – PowerShell can also be called from .NET Code!

AppFabric Cache Codeplex Tool

http://mdcadmintool.codeplex.com/

Demo

Security

• Domain Based Security Option– Domain Account / Local Account based Authentication– Only authorized servers can join the cluster– Only authorized clients can connect to the cluster

• Transport Level Security– Turn on/off Signing or Encryption

• Can turn off Cache Security– Use Firewalls, IPSec, VLANs to protect cache

grant-cacheallowedclientaccount  RedDomain\Machine1$ grant-cacheallowedclientaccount  RedDomain\John

AppFabric Caching Logical Hierarchy

• Host– Physical processes hosting AppFabric

Caching instance.• Named Caches

– Can span across machines– Defined in the configuration file

• Cache Item– Key, Payload (Object ), Tags, TTL,

Timestamps, Version• Regions

– Physically co-located Container of Cache Items

– May be implicit or explicitly created

Regions Region A

Key Payload Tags Key Payload Tags 121 xxxx “Toy” “Child”

123 yyyy “Toy” “Chair”..

Machine -> Cache Host -> Named Caches -> Regions -> Cache Items -> Objects

AppFabric Caching Service

Named Cache : Product Catalog

Named Cache : Electronics Inventory

AppFabric Caching Service

AppFabric Caching Service

AppFabric Caching Service

AppFabric Caching API// Create instance of cachefactory (reads appconfig)DataCacheFactory fac = new DataCacheFactory();

// Get a named cache from the factoryDataCache catalog = fac.GetCache("catalogcache");

// Simple Get/Putcatalog.Put("toy-101", new Toy("Puzzle", .,.));

// From the same or a different clientToy toyObj = (Toy)catalog.Get("toy-101");

// Region based Get/Putcatalog.CreateRegion("toyRegion");

// Both toy and toyparts are put in the same region catalog.Put("toy-101", new Toy( .,.), “toyRegion”);Catalog.Put("toypart-100", new ToyParts(…), “toyRegion”);

Toy toyObj = (Toy)catalog.Get("toy-101“,"toyRegion");

Access APIs – Tagging Items

Tag hotItem = new Tag("hotItem");

catalog.Put("toy-101", new Toy("Puzzle"), new Tag[]{hotItem}, “toyRegion”);

catalog.Put("toy-102", new Toy("Bridge"), “toyRegion”);

// From the same or a different clientList<KeyValuePair<string, object>> toys = catalog.GetAnyMatchingTag("toyRegion", hotItem);

Types of DataReference Activity Resource

Primary Read Only Read-Write Not shared Read-Write, Shared

Catalog Data Shopping Cart Auction Data/Seat Assignment

Web Tier

Distributed Cache

Shopping Cart

Grocery Catalog

Grocery Inventory

Grocery Shop

ApplicationApplication

AppFabric Caching Client

Reference Data – Performance• Catalog data doesn’t change often• Unnecessary network cost to access from different machines• Solution – Local Cache

Put(K2, v3)

Routing Table

Cache2Cache1

Primary for K2,V2

K2, V2

Primary for K1,V1

K1, V1

Cache3

Primary for K3,V3

K3, V3

AppFabric Caching ClientLocal Cache

Routing TableK2, V2

Get(K2)Get(K2)

K2, V3

Reference Data – Bulk Get

• Bulk Fetch from region– 200-300k ops per second– Fewer network calls

Catalog.BulkGet( new List<string>(){“toy-101”, “toy-

102”} , “toyRegion”);

Activity Data – Session Integration

…Session State

stored in AppFabric Caching

Cach

e S

ervi

ce

Cach

ing

Ser

vice

Load Balance RequestsNo more sticky routing

Cach

ing

Ser

vice

Scale your Session StoreDynamically

Highly Available

Drop in AppFabric Caching

SessionStoreProvider

Allows session state to be shared amongst multiple applications

Application

Caching Access Layer

Application ApplicationCaching Access Layer Caching Access Layer

<sessionState mode="Custom“ customProvider="SessionStoreProvider"><providers> <add name="SessionStoreProvider" type=“Microsoft.Data.Caching.DataCacheSessionStoreProvider, ClientLibrary“ cacheName="<YourNamedCache>"/></providers></sessionState>

ApplicationApplication

(K2, V2)

Cache2Cache1 Cache3

Primary for

Activity Data - Availability

Get(K2)

Primary for Primary for

K3, V3

AppFabric Caching ClientRouting Table

K2, V2

PUT

Secondary for

K2, V2

K1, V1

Secondary for

K3, V3

Secondary for

K1, V1

AppFabric Caching Client

Routing Table

K2, V2

Replication AgentK2, V2

Resource Data - Optimistic Locking

• GetCacheItem returns a version object• Every update to an object internally increments it's version• Supply the version obtained along with the Put/Remove• Put/Remove will succeed only if the passed in version matches the version

in the cache

Version Based Update

Time Client1 Client2 (Different Thread or process)

T0 CacheItem item = catalog.GetCacheItem(“PlayerRegion”, ”Zune”);

CacheItem item = catalog.GetCacheItem(“PlayerRegion”, ”Zune”);

T1 ((ZuneObject)item.Object).inventory --; ((ZuneObject)item.Object).inventory--;

T2 catalog.Put(“PlayerRegion”, “Zune”, item.Object, item.Version);

T3 catalog.Put(“PlayerRegion”, “Zune”, item.Object, item.Version);// Version mismatch// Client must retry again

Two clients access the same item

Both update the item

Second Client gets in first; put succeeds because item version matches; atomically

increments the version

First client tries put;Fails because the versions

don’t match

K1

Resource Data - Pessimistic Locking

• Take locks on non-existent keys– Allows you to co-ordinate calls for data

Client1: GetAndLock ("k1")

Client2: GetAndLock ("k1")

Client3: Get ("k1")

Regular Get succeeds

GetAndLock gets lock handle

Other GetAndLock on same item fails

Data Race

GET GETGET

Lock Non-Existent Key

GET/LOCK GET/LOCKGET/LOCKCa

che

Ser

vice

Cach

ing

Ser

vice

Cach

e S

ervi

ce

Composite Race

CALL WAITWAITCa

che

Ser

vice

Cach

ing

Ser

vice

Cach

e S

ervi

ce

Composite Race

PUTUNLOCK GETGET

Cach

e S

ervi

ce

Cach

ing

Ser

vice

Cach

e S

ervi

ce

Resource/Activity Data – Tracking Changes

• Cache Event notifications• Register on any client to notify changes• Batched Notifications

DataCache.RegisterCacheLevelCallback( int filter, DataCacheChangeCallback delegate);

DataCache.RegisterRegionLevelCallback( String region, int filter, DataCacheChangeCallback delegate);

DataCache.RegisterKeyLevelCallback( String region, String key, int filter, DataCacheChangeCallback delegate);

Application

Cache2Cache1

Primary for

K2, V2

Primary for

K1, V1

Cache3

Primary for

K3, V3

Scalable Notifications

AppFabric Caching Client

Routing Table

Register Notification for Key “K3"

Map Keys to Partition

Poll Required Nodes

Nodes Return List of Changes LSN Order

Partition: P2

Last LSN: 19

Call DelegateStore Last LSN

Change LogPartition P11 Add K22 Del

K32

Change Log

(Partition

P2)

18Del K32

19Del K43

Change Log33 Add K134 Del

K22

Agenda

The Problem

A Possible Solution

New Possibilities

What to do next

Data Center

Pre-Fetch

Hospital

Data Center

Pre-Fetch

HospitalRemote Clinic

Slow!!

WAN

Data Center

Pre-Fetch

HospitalRemote Clinic

Cach

e S

ervi

ce

WAN

Agenda

The Problem

A Possible Solution

New Possibilities

What to do next

Web Platform Installer

Select Enterprise

Install AppFabric

Install AppFabric

endpoint.tv

AppFabric on MSDN

http://msdn.microsoft.com/AppFabric

© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions,

it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.